Saturday, September 1, 2007

Celestial Music

Yesterday morning I woke up from one of those startlingly lucid dreams where I was playing a completed game design. I tell my wife that odd ideas are like exotic fruit and if you fail to jot them down, they will rot away, never to be tasted again. So here are my quickly captured notes of what is, literally, a dream game of mine. :-)

Imagine, if you will, a space strategy game. You start out with a single planet surrounded by hostile enemy planets. Your goal is to clear the map of enemies and create a galaxy spanning civilization. All pretty standard stuff.

The difference is that you fight and grow using music that you compose by building your empire. This is where it gets trippy.

Planets as sequencers
Watch this movie clip of a multi-touch music creation tool. It is wonderfully bizarre.

The basic gist, multi-touch madness aside, is that smart objects positioned on a flat screen can be used to create music. Link them together using a directed graph and you have a pretty competent music sequencer and sound effects generator.

The game design leap is that these objects can also be planets in a strategy game. Each planet has a different effect.

  • Forest planets are sources for sound effects
  • Water planets are filters for distorting a sound.
  • Cities planets are sequencers for taking a sound stream and playing it out over time.
Throughout the design, you’ll see that every element can be seen in two ways. One is as an element of a song. Two is as part of a game.

Linking planets
Each planet has a space port. You can drag a space lane from a space port to another nearby planet.

In the process, you connect a sound source to a sound processor. In very short order, just by linking up space lanes, you can create a giant sequenced sound machine. At the same time, you are directing trade and setting up your space empire.

Setting properties on planets

Each planet is a fanciful user interface that lets you adjust the properties of the filter.
  • Planets have a sun. You can adjust the time it takes for the sun to go around the planet in order to adjust the timing of how long a sound plays. In the game this is described as planetary engineering
  • Distance from the other planets effect the volume of the source sound effect.
  • City planets allow you to adjust the height of the cities, thus allowing you to adjust how you sequence a sound source.

Music = culture = resource
Each planet produces a sound. Some generate sounds. Others take in sounds and modify them. The stream of sounds produced by your planets is what most folks would call music. It also represents ‘culture’ in the game. Culture is the resource that makes everything in the game work.

Culture pools in planets that are the final destination of sound streams. They collect it in real time. Culture can be spent on upgrading your planets and creating war ships, defending against attacks.

The enemy does not sleep
You are not alone in the universe. There are other planets on the fringes of your empire. They send ships to destroy your civilized worlds and knock them back into desolate husks. They’ll disturb your carefully calculated rhythms, disrupt trade routes and generally cause your empire to devolve into chaos, then silence.

If a planet is attacked enough, it will be turned into a dead world, devoid of music.

Spending culture
Luckily, you can use your culture to both defend, fight back and ultimately conquer the enemy planets.

Some planets have industrial complexes. An activated industrial complex sucks in culture and converts it into ones of three main types of ships
  • Defenses: The planet builds a shield for fending off enemy attacks. It can take X damage per second. Defenses tend to beat equally matched ships.
  • Attack ships: The planet can send out a stream of attack ships. If the planet is poorly defended, the planet is slowly bombed into the stone age and reduced to a simple forest planet.
  • Ambassadors: Ambassadors convert dead planets into player planets.
At first you defend against attacks. Then you fight back. Eventually you conquer the enemies and make their worlds yours.

The benefits of conquest
Conquest is great. Capturing new worlds allow you to build more complex sequences of sound. This means more interesting songs. It also means that culture accumulates faster and allows you to expand your empire further.

Some planets have tidbits of plot, mysteries, single use powerups, treasures and other rewards. Conquest advances the player in the game.

Culture is multiplied by audience appreciation
An empire is a song. As a song it can be shared with your friends outside of the game. At any point, you can send a link to your empire to a friend. They can check it out and simply by the act of listening to your song, your empire gains marvelous bonuses.

If they like, your friends can rate the song. This gives you even more bonus culture. If they like the style of your songs, they can subscribe to your various empires. This gives you even more bonus. By sharing your creations with the outside world, you gain resources that let you advance your single player game.

A game that encourages the creation of great songs
The game is balanced to encourage this. During the early stages of creating your empire, there is enough culture in random combinations of planets to conquer a fair portion of the map. Eventually, you start running out of culture. You’ll come across powerful enemies that you cannot defeat unless you start sharing your songs with others.

Good songs, as judged by an audience of your peers, will gain you greater rewards than random garbled messes of sound. Judging music is hard for a computer. However, it is easyier for people. The design harnesses your friends as our AI judgment algorithm to encourage and reward the user to become a great composer.

Some maps can be explored and beaten if the player creates songs that are enjoyed by one or two people. Other maps require that the player create songs that are loved by hundreds.

The enemy as a tutorial
The map creation for the game is tricky since it serves two purposes. The first is to provide a challenge to the player. This is pretty straight forward level design and deals with choke points, resources, power ups etc.

The second, more devious goal, is to teach the player compositional structures. The enemy worlds are a nodal graph of a song just like the player’s worlds are a graph of a song. The level designer composes the enemy empire as a song.

All the various tricks of making a song are there for the player to see. The way that you distort a basic sound into something intensely cool. The way that you use a sequencer plus a snare sample to create the beat. All of it is visually laid out as a working model for the player.

When the player conquers the enemy worlds, they are essentially deconstructing the level designer’s song filter node by filter node. The player is then asked to put their newly acquired node to use. The easiest thing would be to replicate what was already done.

What we have is an experiential lesson in music composition masquerading as a game.
  • The player observes a functional model.
  • They dissect the model to understand its parts
  • They are required to reassemble the model and make it work again.
  • They are rewarded for making something better than the original.
I finally got around to reading Rules of Play by Katie Salen and Eric Zimmerman. The tome repeatedly emphasizes that you can analyze a single game through multiple perspectives. Celestial Music is a game that is two things simultaneously. It is a strategy and it is a music creation tool. It is also, by the fact that is both of these things, a system for exploring and learning music in a user friendly manner.

Games lubricate experiential learning about a system. Plunk an inexperienced person down in front of a piano and some sheet music and they will become frustrated. Sit that same person down in front of a game like Celestial Music and they will slowly learn. The result may not be Mozart, but it will certainly be music.

As I awoke groggily from my dream yesterday, I was left with the amazing memory of playing this quirky game. Sound and visuals flowed throughout the screen like some clockwork instrument pulse with life. There was no real distinction between building something beautiful and playing an enthralling game. Delightful.

Take care


Human computation
Humans are far better at some activities than computers. Luis von Ahn is a computer scientist at Carnegie Mellon that studies how to tap into the abilities of people to solve hard problems like image recognition or human identification. He uses games as his medium. The fact that people are better than computers at determining what is meaningful music is leveraged in the Celestial Music design.

The Rules of Play
I rather enjoyed this book's emphasis on the Magic Circle and how it interacts with culture at large.


  1. This is a fantastic idea, though the music could become monotonous in some situations if the designer is not careful, especially in a genre marked by strategic standstills. Perhaps the more a planet is being used/observed, the more integral its musical function becomes to your track? In this way, your familiar core planets could become the backing while the exciting fringes of your empire provide an exotic melody.

    If non-linear music in games interests you, I recommend trying the puzzle game Meteos for the DS. Its music is not groundbreaking, just a backing track with mood/embellishements added depending on your situation in the game. What's interesting about Meteos' music is just how effective it is: When I was playing it, I occassionally found myself inadvertantly delaying beneficial moves in the game to make sure that the resulting fanfare would sit nicely in the music, and I found that I would play significantly worse with the sound turned off.

  2. Very interesting. Two references that may be of interest to anyone researching a music/game hybrid:

    This is essentially the same thing you propose, minus the strategy game element. No game here, it's just a very simple, accessible way to mix tracks to make music, although it has some game-like properties (not surprising given who made it).

    "Everyday Shooter"
    This is a 2d overhead shmup where shooting enemies down plays music. It is simultaneously a twitch shooter, a musical instrument, and a music album (with each level roughly corresponding to a "track" on a traditional album).

  3. This reminds me a little of some work done by the supervisor of my final year project at University. In his case, it was an artificial life simulation. There were 'plants' and 'animals' (represented by basic geometric shapes) projected onto four pieces of glass arranged into a + shape, creating four regions. Each entity created a different sound tone. They also had a limited ability to listen to sounds in their proximity, so eventually would evolve different tones when looking fore a mate, eating and so on, assuming that those sounds improved their survivability.

    The trick was that there were sensors around each region. If people were in one of the four areas, then it would spawn more food for the animals in that region, the logic being that if the sounds being produced in one area were unpleasant, people would move away and things in that area would die off, while the area with the pleasant sounds would survive.

  4. Hallo,

    This reminds me of the story:

    Music of the Ainur

    from J.R.R Tolkien published
    in the Silmarillion.

    Its about the battle of constructive and destructive forces in a symphony about the creation of a world.

  5. Another quick thought re-reading this... wouldn't it make more sense if it was a moon rather than a sun orbiting the planets? Geocentrism has kind of gone out of fashion a while back.

    I'd really like to see this implemented in such a way that the result isn't just a cacophony.

  6. Music aside, this sounds like a religious game. You implement your religion/ideology and people adhere or not. You change the "tune" in order to satisfy different populations. This could be a ratings game as well, with TV or stuff like that. You must create the movie that will convince enough people that you are the good guy, while satisfying their inherent desire to be lead and their lust for bloodshed. One could call the game Spin.

  7. Man, You must have some really weird dreams if this is what pumps out of your subconscious at night.

    Tat aside I think it's a fantastic idea for a game, still needs work to sort out the balancing and fun factor (avoid what Caillan says about avoiding monotony.

    All in all, keep having those dreams!

  8. Brilliant! I'd love to see this done. It's great how it combines a game with a content-creating and sorting system, kind of like in that article "What is the Holy Grail in Online Games?"

    It's cool that you were able to experience this in a dream. My dream games are never very good ones. For example, I once dreamed of playing a cross country running game where you might turn down your running speed, leave the game, and then come back hours later. I suppose it could be an interesting experiment, but it's not exactly a marketable idea.

  9. hmmm....


    I'll see what I can do. :)

    Today's captcha word is "fuboqc", an acronym for "fucked up beyond ontologically quantifiable condition".

  10. wow - this is probably one of the cooles ideas I've seen in a long time.
    I've been investigating the possibility to use music in order to influence the gameplay. The fun thing is, that I've actually never thought about it like you did, with this approach. Two thumbs up :)

  11. I absolutely loooove this idea (since I love music and music games)! If this never becomes a game, I shall be very sad.
    Some little notes:
    - I agree with Simon that the suns should be moons
    - The width of the cities could be adjustable between maybe 4 thicknesses to allow for different note lengths (16th, 8th, quarter, whole) to allow for creation of more complex and interesting melodies
    - There should be some incentive to switch the planets you have around to change the music every once and a while. Maybe connect it with changing popularity, or changing of cultures over time.
    - One thing to consider is would you only hear your own planets music, or both yours and the enemy's, or would there be some balance between them.
    - Also maybe placing animals on the certain planets could effect the sound (more fish in the ocean, etc.) though maybe that would make it too complex. It would be nice if each planet had some type of adjustable element like the city has.

    Hope that all makes sense :)

    It's sorta neat because this (loosely) connects with a class I'm taking about culture and music.

  12. Hi,

    beautiful idea. I think the tricky part is designing it the way, that a good conquering strategy automatically creates nice music. This will be very hard (if not impossible). Simply composing a great song (without conquering other planets) is hard enough to do (look at all the crap on the radio).

    However: "impossible" sound like a challenge to me ;o). If you ever find some game developer attempting to realize this, I would volunteer with my partners to work on the musical side. Usually we score music for games or do interactive audio programming, but we always search for innovative ways of merging music and games. Your idea sounds like one! So let me know if you should need help on (designing) the musical side.


  13. You had me all the way up to defending and fighting enemy planets. You have a winner of an idea here and, having worked with reactivision, could see implementing this. But why muck it up with interplanetary star trekery.. Every game, like a story, needs an antagonist, but they don't need to be a focal point to a game and can still be rewarding (viva pinata sours anyone?).

  14. Thought I'd respond off the top of my head, 'cause it would be fun.

    I think the obvious connection to be made from this is the old Anime Robotech, where the use of 'culture' through music was a unique and demoralizing experience to the invading aliens. To them, this unique idea of music and passion was chaos to the overly bleak and controlled masses.

    Conceptually you could extrapolate the concept to the idea that you don't destroy the alien races but try to figure out how to evolve them to your own.

    Example: Through 'events' you have to determine by analyzing their reactions to your creations and finding the counterpart responses to shift them a level/phase at a time towards your own 'harmony'. This could be shown through color shifts in enemy worlds. A totally black and evil planet might take time to discover how to shift it to a dark blue, then into the red spectrum to get over some 'hurdle' in working on the blues, etc.

    The difficulty would be in determining what would be challenging yet interesting/easy enough for the average player to master. A booming percussive 'attack' might be replied with a cutting flute solo. Perhaps you have to work with simple themes and patterns first (as a more powerful initial attack would trigger the other planet's return attack response) to build up the momentum in order to get over the hurdle and block out the flute solo.

    This would involve a lot of player timing and simplify the selection of music to themes/power levels. The key to constructing your musical 'attack' then would fall on trying to follow the rhythm of each small piece and determine which note/point in the piece gives you the best values for a transition into the next one. You would likely start in easy scenarios with a few theme types and have to figure out how they relate together, building up in scenarios to more complex themes with numerous possible transitions to consider.

    Thus the black planet in question wouldn't respond if you played the percussive up slowly, building culture pressure against them. You'd have to switch to some other instrument type (violin/trumpet/whatever) to fool them and avoid that reaction, but the key would be to perform well enough that you'd switch back to a powerful percussive section correctly and drown out the retaliatory flute. Thus, they'd shift up another color notch towards yours and you'd have to repeat the process (though with some variation, likely, as it seems logical that using the same transitions over and over again would result in less musical power).

    One example of this in literature would be the first part of the Silmarillion, where music was the construtive force of the cosmos. In theory, everything that happened afterwards was spelled out in the fighting melodies between the original god-like beings.

    And yeah, I agree I play better games in my sleep often than in real life. :D

  15. Make the battles rhythm-based and call Toshio Iwai IMMEDIATELY!

    That, and keep blogging!

  16. About Celestial music, check out this: