Interactive Music...er, Audio

Interactive Audio.

Say this in a crowd of developers and you'll most likely get one of these three responses:

Why is it that these two little words "interactive audio" seem to be near the top of so many developers bad_word.lst file? Doesn't it seem odd that in an industry called "interactive entertainment" one of the key components in every game lacks this interactivity? If you had the ability to add CD quality, interactive audio into your game wouldn't you want to do it? That question is akin to asking if you want tires to come with your new car. If interactive audio is something we want and need in games, then why is it in such limited use? It's not for a lack of desire; we know we need it. In my humble opinion the reason it's the exception instead of the rule boils down to one thing: Fear.

Where are we?

Until fairly recently audio has taken a back seat (the trunk may be more accurate) to other technologies in our industry. I don't believe it's because developers feel audio is unimportant necessarily, I hope by now we all understand the important role audio plays in the gaming experience but there was just never much focus directed towards the development of audio technology because of the PCs predominant use as a business machine. PC power and graphics technology has been driven by the business application need rather than entertainment value. But even the television, which is designed solely for its entertainment value, hadn't seen any significant improvements in audio technology until the 'home theater' craze. For years, our high-end stereo systems sat right next to the TV with little or no thought of interconnection. Music technology has been geared towards and driven by the aural experience alone and almost completely separated from the visual experience, even though we all love a good concert. It's fairly easy to see how audio ended up so far behind in its integration in a predominantly visual industry.

Fortunately for all of us, things have been changing and audio is being recognized as an integral part of the entertainment experience. There are now a plethora of companies working on audio hardware -- sound cards and high quality sound cards are very inexpensive and few PCs ship without one. But CD quality audio had previously required far too much storage and processing power to be used in mainstream games. Today, however, processing speeds, RAM prices, hard drive capacity and the proliferation of the CD-ROM drive, coupled with compression and streaming technologies have finally made the ability to include quality audio in games a reality. Unfortunately, audio technology is still behind the curve. Not the actual hardware technology itself, mind you, but the integration of the latest audio technology at a level commensurate with the rest of the industry technologies.

Actually, there is quite a bit going on in the audio industry. Manufacturers and sound designers are exploring the latest and greatest advancements like personal environmental audio settings, positional 3D and surround sound. These things are important but they are not advancing the core technology, they are only adding bells and whistles to the current technology. We now have the ability to add CD quality audio into our games, but we need that audio to be as interactive as the other pieces.

Indefinitus Definition
OK, so let's define what I mean by "Interactive Audio". I must preface this definition by telling you that this is what I perceive the term to mean as it pertains to the interactive entertainment industry. I do this because you won't find this term in a dictionary. In fact, even the word "interactive" is only listed as an adjective under "interaction".

Interactive audio is a technology designed to allow specifically created audio, placed in a given application, to react to user input and or changes in the application environment.
Sounds simple enough, eh? Perhaps it will make more sense in an example. Let's say you are developing a racing game where you drive through various cities. Generally you would have a particular audio track to represent each different city and various utility screens. These audio tracks play from start to finish and normally loop over and over as long as the user is present in that environment. With interactive audio you could have the music adapt to changes in the environment. Wouldn't it be better if while passing through China Town in San Francisco for instance, some ethnic instruments were added to the audio track and then removed as you leave that section of the city? Or maybe even transition the entire theme to one with an ethnic feel and then transition back as you leave that section of the city. How about decreasing the tempo and changing the instruments and style. You could go from a techno sound to a cool acid jazz as you exit the city and hit the freeway. In movies the music generally takes on a slightly different role (another difference between our industry and the movie industry which I'll expound on below). The intent is to create is a particular mood or atmosphere relevant to what's happening or what is about to happen. In a perfect situation we would build tension or suspense and then transition right into the event, guiding the emotion of the user. The ability to do these types of things and more and do them seamlessly -- this is what I mean by interactive audio. With the current way game audio works the audio changes abruptly, if at all, only at the event. The audio is incapable of being a vehicle to move the players' emotion. The capability of being a vehicle needs to be the next step in game audio advancement.

The term "Adaptive Audio" was coined some time in the not to distant past to describe a method of switching audio tracks in a similar fashion, but the problems associated with this method caused it's own demise. The first of which being the shear number of audio files needed to accomplish the task, which of course added to the space required to store them. Adaptive Audio requires constant loading and unloading of large audio files which causes a great deal of hard drive accessing, slowing the process down and causing stuttering of the game. Also, all of those large files being loaded and unloaded taxes the processor and memory, which the programmers object to since these resources are needed for so called "more important tasks" like, running the game engine and graphics. Can you blame them? Some might say that Adaptive Audio or even Reactive Audio is a better term for what I'm talking about, but I think that these labels do not properly convey the idea that we want the audio to not only react to a given situation or adapt to the changes in the environment but to also to give a portent of things to come. Since Adaptive Audio has been the label on a different technology, we should stick to interactive audio so as not to confuse things further.