Well it's taken a while, but for some unknown reason I suddenly have the urge to write my first article. Possibly an opportunity to reflect on the last 17 or so years of my career, or perhaps the beginnings of an early mid life crisis, who knows? But I've started now, so feel compelled to see it through, and hope that by the time I've finished it has been a worthwhile experience, provoking meaningful discussion, developing new ideas, or even just hoping that someone manages to read the entire article agreeing with what I have to say along the way (no, not the bit about the mid life crisis). OK, even the agreeing part is optional.
Well here we are in 2009. Seems like only yesterday that I was loading up my copy of MED sound studio, ripping sounds from other Amiga games, and beginning to create my first tunes. It really couldn't get much better - four channels of 8 bit sound rocked! Way better than what those PC folk had to endure with their FM synthesis (if they were "lucky" enough to have one of those ADLib sound cards). And of course I felt I made the right decision buying an Amiga rather than an ST with its MIDI interface, much to the disappointment of my music teacher.
Step forward a few years and how things have changed. Potentially hundreds of channels, 16 bit resolution, surround sound, considerably more processing power and memory. Today it's not unrealistic to expect 20-30% of the available resources to be allocated exclusively for audio. OK, speaking as an audio guy we could always use more, but I'm certainly reaching the stage where I'm thinking that we need to take a step back and consider that more is no longer necessarily better. To coin a phrase, it's what you do with it that counts, and I think we're arriving at the stage where the limiting factor is no longer the technology.
Of course we all know how much our contribution helps bind everything together to create an experience that becomes greater than the sum of its parts. In the right situation the correct sound has the power to grab you by the proverbials, something which even the most advanced tilt-shifter-shader-doobry-whatsit can only aspire to. Audio has a greater power of suggestion and can be more effective in certain situations that a purely visual representation, and it's the use of audio rather than the technology that is the major contributing factor here.
I certainly agree with Jason Page's comments made in his AGDC 2008 "Next Gen Audio - Is That It?" talk where he cites the (over)use of footsteps in games and that "maybe we don't need to play everything all the time". I'll even admit that I've been guilty of this myself, and upon reflection I believe part of the reason for this is because it's become the expectation that we "need" footsteps. Can you imagine the reaction from the team if you said that you were omitting footsteps, or even that you were focusing less effort in that area, possibly only highlighting the important ones and creating an illusion for the rest of them? You'd be cast aside and thrown into a sound proofed room - oh yeah, you're probably already there, so hey, you've got nothing to lose!
Yet this is exactly the approach taken by Walter Murch when adding the footsteps to George Lucas's THX-1138 when he realised that our minds are able to track one or two people's footsteps without a problem, but are unable to keep track of any more, instead comprehending these group of footsteps as a single sound source (
http://www.transom.org/guests/review/200504.review.murch.html). And of course restricted use of sound can be a powerful tool - imagine how much more impact the scene of a SWAT team bursting out of the back of a van becomes when a sudden explosion of footsteps is added to the mix that was previously without any.
Then there's the desire to make the audio as realistic as possible which again may be fuelled by expectation rather than pure creative sound design. I'm longing for the day when I get the opportunity to create a totally off the wall, surreal gaming audio experience where nothing uses "real" sound effects, or being able to use silence to provide a juxtaposition with a highly active scene. I want to be able to recreate that feeling as a kid when I was playing with my toy cars, planes, trains, etc, making all the sounds with my own voice.
It's possible that we've been there already in the early days of game audio where synthesis rather than sampling was used, but imagine what levels of expression could be achieved with today's technology and resources? I remember an experiment that we carried out a number of years ago when I was at DMA Design working on Wild Metal Country where we replaced the "real" sound effects with recordings created using only our voices. It started out as a bit of fun, but we soon realised that it worked on a completely different level. It also re-enforced our stance that the implementation of the audio was as important as the audio assets themselves.
Perhaps this is one direction "next gen audio" should be heading, or at least a branch of it rather than becoming obsessed with creating the ultimate realistic gun sound, infinite palette of material specific footsteps, believable dialogue, etc. As we approach the days of almost limitless audio resources maybe we should be considering constraining ourselves to help focus on the most important thing - the actual audio design, rather than focusing on producing assets to fit with the contents of an Excel spreadsheet.
This "expectation" also exists in other areas of audio design. For example, given today's audio technology and resources is there really still a need for separate music and sound effects volume sliders? Aren't these legacy items from the days of potentially "repetitive" chip music? Not that some of this music wasn't awesome (I still enjoy listening to it today), it's just that it slowly began to lose its awesomeness after the 459th iteration.
Correct me if I'm wrong, but are we not creating a single audio experience rather than two distinct music and sound effects experiences? You can't adjust the music and sound effects volumes for a film, so why would you for a game? If you feel that you need to provide these volume controls does that not suggest that you perhaps are not confident enough with your mix and that it possibly needs further work?
On the subject of game music, I have to admit that I was bitterly disappointed at the inclusion of the system music player on the Xbox 360 which could override the in-game music. To me this was a big step backwards for game audio and demonstrated a lack of understanding with regards to the role of music from the people making those sorts of decisions. I'm speculating here, but I feel that it is most likely that these decisions were probably made by those not directly involved in audio development and were based on the findings of a number of focus groups.
But then you have to ask yourself, do you want to create your audio experience based on the results of a focus group or a page full of emotionless statistics, or are you brave enough to believe in "your" audio vision (is there a non-visual equivalent of audio vision - audiosion?)? The risk with these focus groups is that we could end up with every game sounding a different shade of magnolia. Be brave I say, create something unique, and believe in your vision even when those around you are unable to comprehend it. There's a reason why you do what you do and they don't.
Nice post. But I would also suggest re our last paragraph - tell that to the Design Leads, Directors, and those handing out the money :-)
I think innovative audio is around the corner; as games are critically praised for it, practitioners like you and I can use the critical reception as currency to encourage creative audio in our next title. It can't be footsteps and guns forever.
I've been coming to much the same conclusion over the past months. After spending over ten years figuring out how to get more, more, and more! data in; playing with more and more accuracy, it finally hit me that I could rarely answer the question: "Why is that particular sound playing right now?" And to my current way of thinking, if you can't respond with the exact effect that particular footstep or bird call is supposed to have on the listener, then it probably shouldn't be playing.
I believe the next step here, (and you allude to it) is the old film dictum that the dialog, effects and music should all be produced in such a way that they can carry the film on their own. The mixer then analyzes the moment and sets the levels appropriately. I've got a first pass working on an AI part that will allow real-time mixing, and I'm hoping it can be fully implemented on my current project.
You're right though about needing to convince the team. :-) I'm working on a proof of concept now where you only hear the music during part of a big gunfight--should be very interesting to see the reactions.
jeff
The same goes for racing and sports games: The music might just not be to my taste, and having the opportunity to turn it down without turning down any other sounds in the game is a welcome feature in those cases.
I wish I could independently control music and voice volumes in movies too. How often do we jack up the volume during dialogue scenes only to scramble to turn it down when stuff starts blowing up (not the least of which being our eardrums)?
The alternative is to put the subtitles up and leave the TV's volume at the usual output, but that shouldn't be.
It's understandable that audio artists are upset when their work is being perceived as "optional" to the game experience but if I don't like the audio, and if the game doesn't provide any options... I'll just mute the TV. But then, everyone loses.
I'm grateful when being offered the choice.
However, just as there are movies that are considered "works of art" there are also movies like Evil Dead that are basically just "mindless entertainment" in a good way. The same is true for video games.
Take a game like World of Warcraft, for instance. Being based around the notion of a monthly subscription, MMOs have lots of things that are basically just meant to take as much time as possible. Grinding, farming, leveling, etc would be examples of this. During these activities, where people are doing the same thing over and over and over and over it's impossible to make a game have that cinematic movie feel to it. I have lots of friends who play WoW and completely mute all the sound and run an MP3 player in the background and just listen to music while they mindlessly grind and farm or whatever they are doing.
At the end of the day it totally depends on the nature and purpose of the game in question.
And what project are you on that gives you 20-30% of available resources? Wow. I would sell a body part for that kind of resource allocation. I've never worked on a project where I was permitted to use more than 10% of the system RAM, for example. Usually it's closer to 5%.
Michael raised the point about convincing the Design Leads, Directors, those handing out the money, and I'm very familiar with having to deal with that situation. In my experience, Audio tends to be the one discipline that needs to work just that little bit harder to get the attention and understanding of these people and that's certainly something I've had to and continue to do on a daily basis. As I've found out simply "complaining" tends not to be very successful, however trying to put a positive spin on it and even getting to the point where you almost convince them that it was their idea in the first place can be more successful. Having both audio design and audio programming experience however has helped me a lot as being able to actually demonstrate new ideas has let these people hear instantly what I was trying to describe. Even if you don't have any audio programming experience, I'd say that having at least one dedicated audio programmer as part of the audio team rather than the programming team is essential today. Michael - I wish you every success and look forward to hearing your next projects.
Jeff - your proof of concept work sounds awesome and I hope to hear it soon. I considered something similar during the development of Crackdown where the triggering of the first gun shot was the trigger for some fight scene music. However in the end, mostly due to general in-development design changes this wasn't possible.
David - yes I agree that in some games I also feel after a while that I need to turn the music down or off, however maybe this is an example where the use of the music could have been considered more? In these games, what is the purpose of the music? Does it form an integral part of the audio experience, or is it simply an overlay to the actual gaming experience? Would we perhaps be less inclined to turn it off if it functioned in a similar way to the work that Jeff mentioned in his reply? In Crackdown for instance, we had the music acting more as a sound effect within the world, coming from cars and buildings and being filtered to make it sound so. As for the actual music content, personally, Mexican gangsta rap is not on my iPod, but it was an essential element in creating the Los Muertos identity.
Pierre - personally for me, although I'm constantly trying to push the audio experience, at the end of the day we're creating a single gaming experience where every component should work together to create that. I feel that the days of parts of the experience being optional are coming to an end, particularly when you consider the increasing budgets allocated to all areas including audio. Your comment about scrambling for the volume control when stuff starts blowing up is a valid one, but there are things we can do as developers to deal with this. In the past I've added an option to games which allows you to squash the dynamic range, bringing up quieter sounds so that there is no longer a large volume difference between the quietest and loudest sounds. The first title I did this in was Wild Metal Country in which we ended up calling the menu option - neighbors in / out ;)
John - I completely understand your point of view, particularly with the use of music in MMOs. From an audio developer perspective however I'd personally be looking to develop alternative uses of music in this case.
Luis - at the end of the day I'm always going to push for the best audio and sometimes I feel that some of these statistics are not helping to do that. It's also possible that these statistics can vary considerably depending on how the questions are asked.
Mark - I think it's just a matter of keeping on trying, making small advances with every title. As for the 20-30%, I know of a number of titles that use this. Yes, it possibly is the extreme at the moment, but it's becoming less so.
Once again, thanks for all the comments.