I Heard You Calling

Synaesthesia has become a buzz word for some high-minded abstractionists in the games industry, but 3D games can co-opt the phenomenon for their own immersive purposes. Video games are particularly adept at connecting sound and visual motion into a way that can produce a real physical sensation in the player.

"We did a lot of real-time post processing with DSP filtering on sounds to convey distances and room-space perception. Each space the player is in has its own reverb type, to keep audiovisual context," said Guerrilla's Mario Lavin, sound director on Killzone 2.

"We also wanted to make the player feel the weight of their equipment, so all the character movements trigger randomized equipment rattles, belt clicks, clothing rustles, etc."

Sound can play a major part in building atmosphere and suggesting mood to the environment and story. In Halo 3: ODST, for instance, your player will wheeze in pain after just a couple shots, emphasizing the new vulnerability of the troopers in comparison to Master Chief.

In Darkside Chronicles, sound effects are used to insinuate short burst of narrative without forcing players to watch cutscenes or read text. In the opening chapter, set during the events from Resident Evil 2, sound effects are used to establish a mood of imminent danger and imply eerily recent disappearances.

"The police siren follows, and the players become aware that they are under desperate circumstances," said Noguchi. "A song being played at a nearby shop confirms that there was normal human life happening just moments before the zombie invasion occurred."

To conventional thinking, this is wasted effort and geometry; players aren't doing anything here, they're just moving through space. In reality, players create associations with, and assumptions about, the gameworld at all times. Even if you're not killing zombies or collecting ammo, you're still taking part in the experience, picking out audiovisual cues at every turn.

Sometimes a combination of visual and sound cues can be used to completely replace the HUD, letting players absorb all the critical information they need from the game world itself. "We spent a lot of time working on Faith's breathing, the sound of her clothing and footsteps, for instance. That can tell you so much about what speed you are travelling at, over what surface, or if your wall run is about to end," said O'Brien.

"In most games, footsteps are a pretty simple thing to add, but running and moving was so integral to Mirror's Edge that we had to create a huge library of footsteps and a system to manage them. We had them for different speeds, different surfaces, different landings; the list goes on and on. The breathing system was also key. During playtests we actually saw players starting to sync their breathing with Faith!"

Mirror's Edge

With different sounds competing for audio bandwidth, proper layering can be vital. "Our audio team spends a huge amount of time and iteration to make sure that not only are the individual sounds unique and reinforce the game atmosphere, but that they complement one another, so the player does not lose the individual sounds in a cacophony of sources," Retro's Kelbaugh told me.

In the movie industry, there's a saying that if you're having a problem with the third act of your screenplay, the real problem is in the first act. The analog for game development is in planning. "It might seem kind of obvious that the audio always comes last when you're building a game, and you tend to not have the time," said Visceral's Bagwell.

"It was really important to us to lock the game early enough to give the audio guys enough time to make the game really sing and make it sound amazing. If the level designers are still moving things around, or they're still moving the cameras around, you can't really set that up."

Having big budgets and long development cycles can definitely make it easier to focus on atmosphere and immersive flourishes, but immersion can still be accomplished with some rigorous upfront planning that identifies what qualities are most important to your game. Would you rather spend an extra two months prototyping or add an extra two levels to the end-game gauntlet?