Creating great audio for games requires successfully combining content and code. Unfortunately, the sound designer/composer (hereafter referred to as "the composer") often has little control over how the code works and thus how the content will actually sound in the final game. This, in turn, reduces the potential of game audio. To remedy this situation, we need a better flow of ideas between the disciplines of the audio programmer and the composer. Even with great communication, however, the coder's time is often at a premium, which causes the audio to suffer as a result.
To solve this problem, we need ways for composers to be able to experiment with interactive audio without needing to wait for the coder's time. This is especially true for smaller game companies which do not have a dedicated audio coder or a refined audio toolset. By putting more control in the composer's hands, the composer can test out ideas independent from the coder. Likewise, the more the composer learns about the control structures and methods to control content, the better informed and effective he will be at creating content. Advanced users of Pure Data could even construct a platform independent prototype sound driver linked to the game via MIDI or TCP/IP.
In this article, we'll examine an audio prototyping tool which can be used to free the composer to experiment with interactive audio ideas on his own.
Benefits and Drawbacks
SampleBank.ogg - Plays simulated "footsteps" as generated in the SampleBankTester.pd patch [Figure 3]
Crowd.ogg - Crowd sound as the intensity rises and falls in the CrowdEngine.pd patch [Figure 10]
Speech.ogg - All five sentences are output from the StitchedSpeech.pd patch using synthesized speech as its source samples.
Music.ogg - AdaptiveMusic.pd patch playing example music. The input intensities and volumes can be seen on Chart 1.
Note: To play .ogg files you can use Winamp2.80 or better for PC or see http://www.vorbis.org/ for a list of more players and related information.
Before we begin, let us consider two hurdles to having the composer do the prototyping. First, there is a steep learning curve for the composer to learn what is, in essence, a new discipline. Many composers won't find this learning process rewarding enough to continue past the difficult initial learning phase. Second, there is a problem that's widespread in the game development business: audio often plays second fiddle in games to areas including graphics, artificial intelligence and physics. As such, the more complicated the control structures for audio become, the more team members from other areas of the game become concerned about losing precious CPU cycles to audio. Often the most interesting audio possibilities require too much CPU horsepower, so a balance must be struck. If the audio team can demonstrate the prototype to the rest of the team early in the development cycle, they can lobby for more resources before decisions are set.
If the audio team can demonstrate the prototype to the rest of the team early in the development cycle, they can lobby for more resources before decisions are set. In the end, if the composer is unable to learn the techniques for prototyping interactive audio, this task is likely better left to the coder. But even in that case, the goal is to have the composer drive the process.
Since the goal is for the composer to do the bulk of the prototyping, it is important to find musician-friendly prototyping tools. It is possible to use applications such as Reaktor by Native Instruments or AudioMulch by Ross Bencina, but these tools simply aren't flexible enough to implement more complex game audio behaviors.
What's Pure Data?
A good balance between flexibility and complexity for audio prototyping lies in graphical, object-oriented audio tools like Max/MSP (available only for the Macintosh) or Pure Data (available for Windows, Linux and Mac OS X). I will concentrate on Pure Data rather than Max/MSP in this article, although they are functionally very similar.
The major drawback to to Pure Data is that it is definitely less user-friendly (due in part to its cross-platform nature) than the commercially refined Max/MSP. However, with some extra effort by the user, I believe Pure Data is a better choice for game audio development. Did I happen to mention it was free?
Table 1. Pure Data Tips
and Tricks For Newbies
- Make sure to turn on the DSP, otherwise you won't hear anything.
- Use the SHIFT key when dragging on a number box to change it in fractional increments
- When working on a patch, it's often best to stay in edit mode and use the CTRL key to modify any parameters
- I still find symbols and lists confusing, so it's best to try to get these straight early on
- Use the right-mouse button on the canvas and select help to see a list of core Pure Data commands
- Use color-coding and clear arrangements to guide the user to elements which should be modified and hide parameters which shouldn't be modified in sub-patches
- Scan the web for new developments since extensions are being continuously added and modified
- Use comments and labels to keep things clear for yourself and others
- Save frequently and keep your archives organized for easy recovery
- Sequence events by using compound messages
- For easy reuse, make sure you label the inlets and outlets of abstractions and subpatches
- I commonly use an underscore at the start of my abstractions to flag them as being separate regular Pure Data objects
Pure Data (and its ilk) are interpreted languages which avoid the time spent recompiling when changes are made. They also allow the user to modify parameters and behaviors while the patch is running. This real-time feedback loop between modification and audition makes prototyping very rapid. However, nothing is perfect and there are a number of significant drawbacks which can cause problems during prototyping. One of the main problems with a modular programming environment such as Pure Data is that the order of operations, which one often takes for granted in a more procedural language such as C++, can become quite difficult with larger patches. I spent an inordinate amount of time trying to get the parameters to initialize properly with the "Adaptive Music" patch. Larger program patches also have a tendency to grow incomprehensible quite quickly (which I have tried to avoid in my example prototypes). However, through the diligent use of hiding complexity, splitting patches into logical subpatches and by making a clear user interface, this problem can be overcome.
To download Pure Data, go to http://pure-data.sourceforge.net/. When you begin learning Pure Data, you should have a look at the tool's official web site, http://pure-data.org/. The program provides documentation and tutorials in the "pd/doc" directory. Going through the tutorials is useful, but they can sometimes be confusing since they are often geared towards learning electroacoustic principles and the advanced uses of Pure Data instead of being geared towards the novice user. The official website provides links to current developments in Pure Data and to related externals (extensions which others have coded). Another good source for current information is the Pure Data newsgroup at http://www.iem.at/mailinglists/pd-list/.
The goal of this article is not to teach Pure Data, but to show how a knowledgeable user can create prototypes for video games. To become knowledgeable about Pure Data, it's best to read the official documentation, go through some of the tutorials, peruse the newsgroup and then dive in and start modifying the tutorial patches to find out how things really work. Once one is comfortable with the general workings of Pure Data, the example patches here can be explored and modified.
The Four Areas of Game Audio
Like film and television, game audio contains music, speech and sound effects. But games also require interactivity and real-time effects applied to the audio. When this occurs, game audio moves further away from its roots in linear media.
Highly adaptive audio is difficult to create, due to the increasing number of variables created between the content and the code. Adaptive music requires the composer to create content which seamlessly transitions between a multitude of game states. Similarly, complex multi-sample speech stitching can spiral into an almost endless number of possible sentences, and getting such a system to sound good can be overwhelming.
Besides music, speech and sound effects, games also use a fourth category of audio. This category of audio attempts to simulate highly complex sounds, such as the roar of an entire crowd or the sound of a racing car engine. These sounds are made up of many dynamic parameters which modulate the audio content in real-time. With physical modeling and other synthesis methods it may even be possible to synthesize the sound entirely. For the purposes of this article, I will use the term "engine" to refer to this category of audio. The difficulty for the composer, is that the content must react well to many different kinds and combinations of modulation to maintain the illusion of the object it is attempting to simulate.
With game audio platforms supporting audio resolutions similar to film, the question is no longer if we can produce audio which sounds good, but whether we can produce audio which reacts well and sounds good in all of the different states that the game player might trigger. In the following Pure Data section of the article, we'll examine four different prototype patches which address the issues of each of these four categories within game audio.
Using Patches In Pure Data
In the following section I introduce four patches which I put together to aid in audio prototyping. These patches that correspond to the four game audio categories outlined above. The patches are meant to be working prototypes which can be further modified to suit the individual needs of the composer, or just to start learning about Pure Data. The idea is to begin by playing around with the patches and make some sound, understand how they work and eventually start modifying them to suit your own individual needs. Alternatively, you can create new patches using these as a starting point.
Sample Bank Patch
One of the basic concepts when working with game sound effects is the notion of a sample bank. Usually one wants certain sounds to always stay loaded (e.g., short menu sounds), and to have groups of sounds relate to specific locations, levels, or items (e.g., vehicles or weapons). The following Sample Bank patch ,which I have prototyped, reads all the sound effects contained in a spreadsheet and assigns them default values which can be applied to default volume, envelopes and any other desired parameters.
For the first patch, I'll describe the Pure Data code in more detail than the other three patches. Rather than construct a large number of array "slots" which the sounds load into (and thus restrict the maximum number of sound effects), this patch creates the sound effects objects (data and associated operations) at runtime using the "; pd-" message at the bottom of the parent window. This method is very similar to the way the sounds might actually be loaded and created in a compiled object-oriented language such as C++ for the game.
Once the sound effect objects are created, they can respond to messages. For instance, they can be told to play as a one-shot sound or as a continuous loop as well as can change their pitch, and so on. Through the use of messaging, multiple effects can be triggered and the resulting mix can be auditioned. This use of triggering multiple sounds with a single message can be thought of as sound macros or soundtags which would often otherwise be difficult to create using a sample editor or sampler. This patch allows the composer to generate and control composite sounds, such as player collisions, breaking glass and other more complex sound events by triggering multiple samples to produce a varied result each time.
The "_playSample" abstraction begins by loading its respective sample into the sample data table. The size of the file is output from the "soundfiler" object which is converted to a frequency and sent to the "phasor~" object so that it will play back the sound in a loop at the correct rate. When an event is received the abstraction parses the event type and event parameter apart. If the event is "looping" it starts the "phasor~" object which generates a sawtooth wave of the correct frequency to modulate the sample playback position pointer in the "tabread4~" object to play the sample in a loop at the correct frequency scaled by the event parameter. If a "one-shot" event is received, then the playback rate is set to zero after the sample is played once using a "delay" object. If a "pitch" event is received, then the phasor's frequency is scaled by the parameter. Thus, sending "pitch 0.5" will cause the sample to play an octave lower and "pitch 2" will play the sample an octave higher.
Since the data and its related operations are created as a single unit (or "object" in object-oriented terminology), multiple objects must be created to trigger multiple versions of the same sound. The drawback is that creating multiple versions of the sample generates extra overhead in the spreadsheet. The upside is that the system avoids issues of voices, or confusion over which instance of the sound to modify once the sound is playing. For example, if we trigger multiple tire squeal sounds using the same sample and want to control each sample's playback pitch, we know which sample we are controlling since each sample object has a different name. An improvement could be to define the maximum number of instances of a given object (instance capping) and have Pure Data create the appropriate objects and index them automatically. When implemented, it also wouldn't make much sense to have multiple versions of the same data, so this would need to be optimized as well.
In the "Sample Bank Tester" patch, we can test one-shot, looping and pitch events. The "footsteps" bang causes the simulation of footsteps to be triggered. This is created by randomly triggering footstep samples with a small variation in delay time before the next sample is triggered. This can be heard in the footsteps.ogg file.
This patch allows the composer to easily change the triggering and modulation of a spreadsheet full of sound effects. Currently, the patch does not use a lot of the supplied parameters, including the "game area" column, which could eventually be used to calculate the memory usage for each bank.
If the game was able to output events that Pure Data could react to, such as TCP/IP, MIDI or some other protocol, this patch could be used to interactively change the behavior of the audio interactively. Likely there would be latentcy issues and other technical concerns but it would allow for multi-platform games to all trigger the same audio to make sure that the sound has been correctly coded on each platform. In essence, it could function as a highly flexible platform independent sound driver.