Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Gamasutra: The Art & Business of Making Gamesspacer
Envisioning Our Interactive Audio Future
arrowPress Releases
July 28, 2021
Games Press
View All     RSS

If you enjoy reading this site, you might also want to check out these UBM Tech sites:


Envisioning Our Interactive Audio Future

July 3, 2013 Article Start Previous Page 3 of 3

Milestone 7: Bug Crunching

Back to work, I quickly navigate a user-interface terminal and instantiate a new area of the world to test in. A transition opens soundlessly in front of me and floods my senses with the sound of the city. I'm overwhelmed by the density of the experience: all oppression and intensity as drones weave their way in and out of a sympathetic embrace. This humming metropolis is a hive, a thrumming and humming frenetic audio vibration that typified 2012 Earth in all its glory. In a moment: speeding cars, trickling fountains, and skyscrapers resonating in an orchestration of the then-modern age.

Between the insistent air-conditioner rattle and intermittent elevated train clatter lies the golden age of information technology, an endless stream of activity amidst the rapid acceleration of interconnectivity. It's not long, as I cross the threshold of modernity, before I'm confronted with the endless saturation of sensual information. These simulated molecules caress my battered brain-stem in a dance of orchestrated input toward an inevitable overflow of stimulation. Whereas moments ago I was adrift in the tranquility of the rural countryside, I'm now firmly frenetic in the no-hold-back- all-attack modern age.

I engage a frequency analyzer that immediately projects a rainbow of visual analysis across every corner of the world. I can use this debug model to visualize the density of the frequency spectrum based on color as it emanates from every object, radiating a kind of ethereal cloud of color based on the sounds of the city. As cars speed by I see their unique voicing characteristics reflected in a wash of frequency analysis, the hue and saturation accentuating a buildup of tonality across the spectrum. I'm looking for an anomaly reported by early adopters of this simulation, some sort of unnatural resonance.

Loading a restore point, the world around me snaps into its prerecorded routine. It must have been a familiar scene: buildings reaching to the sky, transportation of all kinds treading familiar paths, a swarm of people vibrating with energy. Vehicles zipping around like oversized insects buzzing with momentum, intent on reaching their destination through a complex mechanical ballet of belching exhaust and the endless drone of so-and- so revolutions per minute. Each building resonating with a hum. From the depths of their mechanical rooms to the vibrations of people cascading throughout the interior, each building is like an oscillator in this polyphonic synthesizer of a city.

Milestone 8: Anomaly

It's the same on the sidewalks that flank every thoroughfare. Everybody on the street moving, seemingly in sync, toward an unseen destination. Each person moves lockstep in time with the heartbeat of the city; like a million tracks of kick drum mixed below the threshold of hearing, but undeniably propelling each step. The individuality of their concrete footfalls is lost in a sea of clear density.

Aside from the occasional amplitude increase, so far, the playback seems normal. I scrub through the capture to the moment specified in the report and all at once the world is saturated in a haze of pink, representing a frequency pileup. I switch to manual control and reverse to the origin of the anomaly. I peruse the data streaming in one of the debug windows in an attempt to isolate the source and immediately see the problem. I navigate to the source; it seems at first glance to be a discrepancy in the modeling. As I trace along a sinuous pulsating line that connects the extreme data- point to a mass of overlapping frequencies, I find myself standing low to the ground somewhere in the middle of the sidewalk.

I switch off the frequency analyzer and find myself face to face with a little girl sobbing uncontrollably. This explains it—the quality of our voice modeling always seems to break down during moments of extreme emotional response. Our models take into account the simulated size, shape, and composition of the simulated vocal chords and we've captured and re- created the entire range of motion, but there are extremes in every case that still require fine-tuning.

Milestone 9: Synthesized Soul

Of course we began by nailing down celebrity vocalizations first: These were the most profitable. As soon as an actress or actor achieved a certain status we rushed in to take a vocal snapshot which could be perpetuated beyond the peak of their profession. Coupled with a full body scan, procedural animations, and phoneme-based lip-syncing, we've had digital doubles as part of our experiences for a long time. There still exist moments when even our best technology cannot yet achieve the artistry present within a live body, though. At the end of the day it is still the performance, whether acted or authored, that has the potential to connect with the audience. It remains a testament to the human body- instrument that there are still secrets held within.

The girl seems to have quieted down and her face begins to scan the crowd frantically. My heart goes out to her as I remember that day so long ago when, as a parent, I lost someone dear to me, if only for a moment. Of course this is still only a simulation; none of this is happening, but I'm still here having a real reaction that I can feel in my chest as my breathing accelerates. It's more a result of my experience than any belief in this as "real." Regardless, I kneel down on the sidewalk in front of her and ask if she is lost. She looks in my face not knowing exactly what the answer is, but eventually nodding in acknowledgment. I begin to ask who she was with when, midsentence, the message is broken by a mother quickly lofting the little girl into her arms. Amidst a volley of hugs and exasperated condolences she's quickly whisked away through the bustling parade of lunchtime foot-traffic. I can see the little girl's smile through the sea of people as I return to the task at hand.

Scrubbing back in time to the original anomaly, now identified as the little girl's cries, I utilize the controls attached to the data-point for the girl's vocalizations. Soloing the sound of her voice, I begin to apply various smoothing algorithms. Without losing any of the ferocity or emotion, I methodically constrain the parameters that have caused things to become unnatural. The process takes just seconds, but in a simulation of this scope it could take hours to address every similar case. Instead, I apply the adjustments to the vocalization models to be used in the event of other such anomalies. While the changes don't go as deep as the simulation itself, they can be applied in real-time if the same behavior is found. If it happens often enough, the solution will be used to inform the next update to the vocalization model.

This period of Earth has become known as a "great turning point" in the evolution of life on the planet. People began to notice the changing environment. Even amidst the abstracted nature of the city, people were taking note of the fact that everyone had a role to play in preserving the planet. By the time this recognition spread, the focus had already shifted to worlds beyond the confines of a single planet. From there, it was a combination of exodus and slow decay.

I exhale and return to the task at hand, flipping to the next restore point that needs investigating. I'm faced with a wall of water, an undulating sea, and wind set to tear the roofs off the nearby houses. The howl and moan as the waves increase their amplitude approaching shore is unhinged in a moment of sheer sonic terror. I bring up my display, and prepare to orchestrate the power of nature.

Milestone 10: Ship It

Through a combination of fictional story and factual reference the idea of this story is to inspire the work being done to help envision the way forward for interactive audio. While some of these workflows and methodologies live strictly in the realm of science fiction, there are aspects that can be found running in simulations at universities today. This radical change in approach— from the standard sample playback methodology into a composite toolbox which incorporates extensible procedural, synthesis, and physical modeling techniques— is rapidly evolving within today's game industry toward a future hybrid model of dynamic sound. Due to increasing size, diversity, and complexity inherent in most games, it seems inevitable that sound needs cannot be met with sample-based content alone. I continue to look forward to the creative ways that sound can reinforce the perceived reality of an interactive experience by leveraging the inherent dynamism of simulations.

"Life's like a movie, make your own ending, keep believing, keep pretending." - The Muppets

Damian Kastbauer is lost at and on Twitter @lostlab.

Works Cited

  1. Noriko Kurachi -- Now Hear This
  2. Sounding Liquids -- Sound Synthesis from Fluid Simulation
  3. Efficient Numerical Simulation of Sound Propagation
  4. Precomputed Wave Simulation for Real-Time Sound Propagation of Dynamic Sources in Complex Scenes
  5. Motion-driven Concatenative Synthesis of Cloth Sounds
  6. Precomputed Acoustic Transfer: Output-sensitive, accurate sound generation for geometrically complex vibration sources
  7. RESound: Interactive Sound Rendering for Dynamic Virtual Environments
  8. Fast Modal Sounds with Scalable Frequency-Domain Synthesis
  9. Walter Murch, Dense Clarity –- Clear Density

Article Start Previous Page 3 of 3

Related Jobs

Sucker Punch Productions
Sucker Punch Productions — Bellevue, Washington, United States

AI Systems Designer
Xbox Graphics
Xbox Graphics — Redmond, Washington, United States

Senior Software Engineer: Performance Tooling
Xbox Graphics
Xbox Graphics — Redmond, Washington, United States

Senior Software Engineer: GPU Compilers
Insomniac Games
Insomniac Games — Burbank, California, United States

Senior Hard Surface Outsource Artist

Loading Comments

loader image