Playing by Ear: Using Audio to Create Blind-Accessible Games

Navigating by Ear

Giving players enough cues to let them know where they are in the 3D world, but not so many that their ears are overwhelmed with sound, was our biggest challenge. There is a lot of information to convey:

The hardest pieces of information to convey audibly tend to be navigation issues that are nonexistent or trivial in a graphical interface, such as the location of the exits. For example, a simple solution for making exits easy to find in a graphical user interface is to make them look like familiar exits in the real world (such as doors and corridors).

Exits. Initially, we installed doors at all of the exits of each room in our level (gameplay occurs in an indoor environment). Our doors were of the electric Star Trek variety, automatically sliding open as you approach, so it seemed natural to have them emit an electrical humming noise. The idea was that when standing in the center of the room, you would be able to identify the exits just by the noises they were making; if you heard a hum to your left, you would know there was a door to your left.

That implementation was a complete failure. It was difficult to navigate out of any room with more than one exit, unless you cheated and peeked at the screen. If you stumbled around long enough you'd eventually hear a door make its opening "whoosh" sound, but even then it was hard to navigate through the doorway.

We tried several variations - we gave each individual door a slightly different sound, created doors that beeped instead of hummed, and tweaked attenuation parameters so you didn't hear doors until you were fairly close to them. None of them worked very well.

We finally realized that the doors weren't really what the player needed to hear. Blind players actually needed audio cues to navigate into the room or hallway that lay beyond the door. We ripped out the doors and installed air conditioning vents - point audio sources with a pleasant hum - into the center of each hallway, adjusting their volume and attenuation so that they could be heard down the length of the entire hallway and slightly into the adjoining rooms. We also modified the game engine so that walls occluded point audio sources.

After making those changes, finding exits by ear became easy. When you hear the air conditioning noise, you have an unoccluded path into the hallway. You then rotate yourself until the noise is coming from directly ahead (so it has the same volume in both ears). You can then simply walk forward into the hallway (see Figure 1).

	This point source audio entity, with an attenuation radius of r, cannot be heard by player 2 due to occlusion.

This same technique can be used to make it a little easier for sighted players to find hidden passageways and rooms in a level. Observant sighted players will notice a quiet hum coming from their left as they walk by a hidden exit and will use the audio to navigate their way into the hidden area. This is one case where blind players would have an advantage; hidden passages would sound the same as any other exit.

Footsteps, bumps, and scrapes. As in many games, we generate footstep sounds as the player walks. They are a tried-and-true solution to the problem of giving players feedback on whether or not they're moving and letting them know how fast they're moving. In fact, it would have been more work to make movement silent, since footsteps are hard-coded into the Quake 1 engine.
We have, however, extensively modified the audible movement cues to make blind navigation easier. Originally, the game engine played a simple "ugh" sound when the player walked into a wall. Sighted players can easily see if they've walked directly into a wall and are stuck, or walked into it at an angle and are sliding along it. To give blind players the same information, we adjusted the stereo pan of the "ugh, I ran into a wall" noise based on the angle at which they ran into the wall. If you run into the wall with your left shoulder, you hear it in your left ear; walk straight into a wall and you hear it equally in both ears. We also added a scraping noise to indicate that a player is moving forward but contacting the wall. The scrape sound is stereo-panned in the same way as the bump sound.

Supporting artificial stereo panning did require us to modify the game engine's sound code. We added an extra floating-point parameter (balance) to the play-a-sound function. Zero is the normal setting, which plays the sound using the standard 3D stereo spatialization algorithm. A value of -1 results in the sound occurring completely in the left ear, and +1 results in a sound completely in the right ear. Of course, values in between pan the sound from left to right.

Getting oriented. Letting players know which way they're facing is always a challenge when you allow unrestricted movement in a 3D world. As in many games, we simplify the problem by restricting movement to 2D movement along a ground plane. Therefore, the player's orientation can be described in north/south/east/west terms; players can't fly up and down. Even on a 2D plane, however, after a few turns down a series of passageways it is easy to lose track of which way you're facing. We've found that some of the same techniques help both blind and sighted players keep themselves oriented.

The most basic technique is to simplify level design. Unless getting lost is part of the game, avoid creating a maze of twisty passages, all of which look and sound alike.

Another technique we use is to build a consistent orientation cue into the 3D environment. For sighted players in an outdoor environment, that might be moss on the north sides of trees, or clouds in the sky that always move from west to east. In our case, we modified the hallway air-conditioning noises so that north-south-oriented hallways make a slightly different noise from east-west hallways.

We also bound a keyboard key to an audible compass. Pressing the key announces which way the player is facing, rounded to the nearest compass point (such as "northwest" or "south"). Implementing the audible compass was much easier than a graphical compass and gives the same information.

Audible objects. Besides exits and walls, the other objects that sighted players can see and that blind players need to hear are the other players (our game is multiplayer) and any object that can be picked up, be poked, or otherwise affect the gameplay.

The other players are easy to hear, since they're making footstep, wall-bump, and scrape noises as they walk around the level. We do implement special code so the artificial stereo panning of the bump and scrape noises (indicating the angle of impact) is only done for your own bumps and scrapes. As other players bump into walls, you hear their grunts as ordinary point sound sources, emanating from the point at which they hit the wall.

All of the other visible objects in our game are assigned a reasonable, regular idling sound so they are always audible. We've created a noisy, silly, fun environment, choosing objects that appeal to both the eyes and the ears. Walk around a level and you might hear chickens clucking, a grandfather clock ticking, and pigs squealing as they're picked up and thrown.

All of these noises are occluded by the walls of the level, which limits the number of sound sources audible at any one time and prevents blind players from trying to walk through walls to get to objects that they can hear but can't see. Sound occlusion is a wonderful thing. Our implementation silences sounds if the line segment from the center of the listener's head to the center of the sound source intersects any of the walls of the level. The result is not physically accurate, but works very well as a navigation aid and was easy to implement. Occlusion also prevents sighted players from becoming frustrated trying to find a path to an object that they can hear right next door.

Ambient noises. After working through the navigation considerations to allow blind folks to move around our 3D world, we then added audio decoration to make the world more interesting. We tried to give each part of the level a distinct character by adding audible landmarks. For example, you might hear the sounds of pots and pans clanking in the kitchen or hear a stately old grandfather clock ticking in the study. We added a generic arbitrary_noise object type to make it easy for our level designer and audio engineer to sprinkle interesting sound throughout the level.
We also implemented a quick and dirty form of environmental audio. If our minimum system requirements had allowed it, we would have used EAX environmental audio (more on that later).

	Room center entities create different auditory environments for each room.

Instead, we created room_center objects and placed them around the level (see Figure 2). They are simply invisible boxes that the level designers used to mark out the various rooms in the level. One of the attributes of the room_center object is footstepNoise. By using different footstep noises for different rooms, we give the impression of the player being in different environments. A carpeted study has quiet footsteps, while a kitchen has sharp, echoing footsteps. It was easy to modify the game engine's footstep-playing code to play the appropriate footstep noise depending on what room_center object the player is in.

Sounds other than footsteps should also be affected by the sonic properties of the room. That's easy for any audio sources that are part of the room; our audio engineer just "precompiles" the room's environment into the sound file. In our game, objects that can walk or be carried into rooms sound the same no matter where they are, which is unfortunate but not a huge problem.