In order to make this happen, we needed to create both a gameplay that lends itself to creative exploration and a musical generation system that will provide complex, generative, but reproducible audio. Each of these things is important for different reasons.
Complexity is perhaps the most obvious -- it is the idea that actions in the game which create music must also have enough depth and offer enough value that repetition will not get old. This is an odd science, because on one level, music is based on repetition and patterns, yet in games a repetitive sound can destroy an illusion -- or at least become annoying.
Additionally, the musical complexity must be someone limited, or else the audience will lose the connection between the action and the musical result. For example, if a certain tile triggers a sound when the player's character traverses it that could potentially get old very quickly.
However, if that tile emits a different pitch of the same sound each time, it will add a layer of musical complexity. The trade-off is a decrease in the connection between the action and the sound -- a balance needs to be struck!
Generative, or procedural, audio is a bit of a buzz phrase, and there are certainly a number of different definitions floating around the internet. Thinking of it in terms of generative music frames things a bit differently, relating it closer to the world of 20th century classical music composers like Xenakis and John Cage (and arguably, jazz composers like Charles Mingus).
In that world, the concentration can be on setting up a system within which a performer can explore and generate their own music. In many ways, it is about creating constraints and borders, a framework that allows an infinitely variable set of musical possibilities.
For example, in John Cage's pipe organ piece As Slow As Possible, the actual tempo it is meant to be played at is left undefined -- meaning performance times have varied between 20 minutes and 639 years. Cage's Fontana Mix takes this even further, as even the instrumentation is defined as "any number of tracks of magnetic tape, or for any number of players, any kind and number of instruments". The score is a series of transparencies and paper, which when layered create patterns that direct the players. A potentially infinite number of patterns can be created.
This may all seem rather academic, but it is important to note the basic parallels between this type of music composition (often called non-deterministic, or parametric) and games. Games and sports, of course, are the epitome of setting up a system and letting the players make their own way. Each play of the game will be slightly different, and in some cases different playthroughs will result in barely recognizable endings. It's no accident that many of these 20th century composers were fascinated by game theory.
Finally, reproducibility is essential in terms of rewarding the user. As described in the previous two paragraphs, music that is generated in a game needs to have depth, and the user needs to feel agency over its creation, but in order for something to be a game there needs to be some element of mystery. This mystery does need to be reproducible, however, or the user will not be able understand the patterns and play with them -- which is a major component of turning an object that makes noise into a musical instrument.
So how did we put all of these things into practice? Well, we made a game with dogs wearing hats. Pugs Luv Beats is a universal iOS music composition game that we released in December 2011. In the game, the player controls an alien breed of pugs. Once the masters of a wondrous and highly advanced civilization, these pugs are the victims of their own greed. They loved nothing more than to collect beats, which they cultivated with their special brand of "luv". But an ill-advised scheme to grow the biggest beat of all time spun wildly out of control, and their home planet was destroyed. The player must help the pugs to grow more beats so they can rediscover new planets, build houses, and recover their lost technology.
We are very proud of the game, particularly since we are a tiny operation of three people. Jon Brodsky handled all of the coding, using a combination of openFrameworks, Lua, and a custom build game engine called Blud. I did all of the audio using libpd and Pure Data, which I'll get to shortly. Sean McIlroy is the artist and designer in the team, and he did an amazing job developing the cute, original, and oddly poignant pug characters, as well as the whole gameworld and UI. This was an enormously important aspect -- as you'll soon see, Pugs Luv Beats is in many ways a music sequencer disguised as a game, however we wanted to make sure that it looked nothing like music software.
The gameplay itself focuses on the pugs. You start out with a single pug living in a house. You need to make the pug harvest beets, which you do simply by tapping a path for the pug to run. When the pug hits a beet, it creates a musical beat (ha!) and your beet count rises. More beets lets you buy another house, with another pug. Buying more pug houses lets you uncover more of the planet, and thus more beets. Collecting enough beets will also allow you to buy a brand new planet and start over again.
As you expand your universe, you will discover new terrains -- these new terrains will make different sounds, letting you grow your musical palette. However, these terrains will slow your pugs down -- that's why you need to find and equip your pugs with hats and costumes. You want to go faster on snow? Find a Santa hat! How about water? Shark fin!
This approach to the game design and structure means the game is effectively endless -- even once you have discovered all of the hats and costumes, you can continue to explore the universe and find new combinations of terrains. Each planet is also a separate musical world, so you can always start from scratch, or go back to your previous planets to remix and recompose -- all using costumed pugs.
Behind our aesthetic styling and collection game mechanic there is a generative music engine that puts all of this theory and design into practice. Let's delve into a bit.
We decided pretty early on that we wanted to use Pure Data to handle all of the audio. This was made possible thanks to the hard work of Peter Brinkmann and Peter Kirn, among others, who put together libpd (go buy the new book!) Pure Data is a free and open source graphical programming interface designed mostly for audio and music geeks who don't want to have to program everything by hand. I will go ahead and admit at this point that I fall into this category.