Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
Design With Music In Mind: A Guide to Adaptive Audio for Game Designers
View All     RSS
October 15, 2019
arrowPress Releases
October 15, 2019
Games Press
View All     RSS







If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

Design With Music In Mind: A Guide to Adaptive Audio for Game Designers


May 29, 2003 Article Start Page 1 of 2 Next
 

There is a growing trend away from detached "linear" scores towards music that is tightly integrated with, and relevant to, gameplay. Game designers are learning that music doesn't have to be merely a detached backdrop to the action on screen - it can ebb and flow, adding emotional depth and soul to scenarios and help maintain the suspension of disbelief that is so crucial for players. We now have the ability to craft scores that adapt to what is going on in the game, in real time.

In this article, I'm going to explain ways that game designers can work more closely with composers to achieve a more integrated soundtrack for games. This is important because music is currently underutilized in most games, allowing plenty of room for design innovation which translates to more game sales, while delivering a more meaningful player experience. The following sections will show you, the game designer, how to incorporate music into your game design from the onset of a project, and how to follow through from initial concept to final implementation, leading to a highly immersive game score.

"Adaptive audio" is a term used to describe audio and music that reacts appropriately to - and even anticipates - gameplay. The term "interactive audio" is also used in this way, but it is used more broadly, and it has been misused and overused.

An analogy I like to use when describing the benefit of adaptive music relates to game graphics. In a sense, linear music is to pre-rendered animation as adaptive music is to real-time 3D graphics. What did games gain from game-rendered art assets? The ability to view objects from any side or distance, and the flexibility to create a truly interactive game environment. These graphical advances give gamers a more immersive and controllable environment, and adaptive music offers similar benefits. Currently most game music is "pre-rendered" - mixed in fairly large sections prior to being put in a game. In contrast, adaptive music is "game-rendered" - musical components are assembled by the game as it is played. This flexibility allows adaptive music to sync up with the game engine and become more integral to the action on screen.

The Spectrum of Adaptability

Many degrees of adaptability exist in music. On one end of the spectrum is linear pre-rendered music, and on the other is music that is completely game-rendered. Where on the spectrum a score lies depends on the game at hand, and aesthetic decisions made by the composer and game designer. There are many options for combining small pre-rendered assets (such as wave files) with assets that tend to be more flexible (such as MIDI files). Even within a particular game, different degrees of adaptability are called for, such as linear cut-scenes versus the ever-changing game environment.

No longer are there excuses for using detached, linear game scores. Even licensed music can be arranged to function adaptively. Orchestras can be recorded in a manner that permits adaptive arrangements. If a composer simply supplies long, linear musical pieces for a game, that composer is not "scoring" the game; they are just providing music that is in the correct genre and style. Imagine if a film composer did the same thing - created music that had nothing to do with the images and action on screen. That composer would be fired! Scoring to picture is half the art in film composing, and the same applies to game scores. The difference is that scoring games is an emerging art and is a new skill for most composers, and in many ways is a more complex task due to its nonlinear nature.

Integrating Music Into Your Game Design

The time for a game designer to begin thinking about game music is the moment the initial design process begins. As you begin envisioning the style of gameplay and the environments in which that game will take place, think about the aural aspects of the game, too.

Music dramatically affects the tempo and pacing of gameplay. Game designers who are aware of this can use music to their advantage in moving the story and heightening the emotional impact. Music that is not directly connected to gameplay elements can distract the player and literally pull them out of the environment. This is due to a sense of "disconnect" that the player feels when the music is inappropriate to the scene. This is true of all genres of games, whether they are action/adventure games, real-time strategy games, puzzle games or another genre. Even when music is not a core aspect of the gameplay, it needs to be tied to that core and support it.

When a game designer starts to think about the look of a game, ideas about style, color, and lighting arise. The game designer imagines what the game world might look like, and the moods that the visual landscapes will inspire. The same attention should be given to the audio landscape of the game. Audio affects the mood of any game in both profound and subtle ways, and it is often the subtle nuances that make or break a game scores' effectiveness. The more clarity of intention a designer brings to bear on the sonic nature of the game, the more likely the score will support the game overall. A theater director once told me to treat the music as if it were another character on stage, adding its own personality to the performance, interacting with the other performers. This director understood the power of aural landscapes, and used them to propel his performances.

Another important aspect of understanding how audio affects an audience is to know that it often is working on a subconscious level. The visual aspect is often foremost in our mind. This does not mean that audio is not enhancing the player experience. On the contrary, audio has the opportunity to affect the experience in a stealth-like manner, and to affect how a user interprets the gaming experience. That is a powerful tool, and it is why focus groups inadequately measure the importance of game audio. Just as good audio can enhance a player's overall perception of the game, poor audio will drag that perception down.

An effective aural landscape enhances gameplay by reinforcing the overall mood and ambience of the game and accents important gameplay events. Music drives the pace and momentum of gameplay and supports the visual aspect of the game by complementing the art direction. Subtle mood shifts in the score flow with the storyline as dramatic shifts in intensity follow the action of the gameplay. Finally, seamless music transitions connect the various moods and intensities, thus supporting the game's continuity while keeping the player "in the game".

The Music Design Document

Creative collaboration between the game designer and game composer is too often neglected. A good composer brings more to the table than composition chops. That individual brings ideas about how the score can best support the game, stylistic ideas, dramatic techniques, and is aware of how, when, and where music is effective, and why. The game designer often has a broad vision for the music, and the composer can focus that vision, and find specific solutions to creative and technical issues that arise. The collaboration between the designer and composer can inspire both people, and often results in a creative feedback loop. The collaboration also ensures that the game score is relevant to the game as a whole.

Creating a "music design document" is one way of turning the creative vision into a technical solution. A music design document begins by imagining how you would like the music to behave in the context of your game, and asking yourself questions like these:

  • What parts of the game should have music?
  • What style of music is best for this game?
  • When should the music be ambient, and when should it be intense?
  • How would you like the music to transition?
  • Are character themes appropriate?
  • Are there aspects of the gameplay that would benefit from musical accents?

These are the types of questions that will lead to an effective music design.

The music design document can be part of the game design document, and like the game design document, it should evolve as your vision of the game solidifies. The music design document should guide the process, and codify the ideas of your creative vision. Collaborating with a composer on the music design is as important as collaborating with an art director on the visual direction of the game. The major sections of a music design document should include these headings:

  • Music direction. Codify the style of music, production values, range of moods, and so on.
  • Themes. Consideration of world themes, character themes, and other appropriate aspects of the game that will affect the music at a meta level.
  • Functionality and adaptability. Describe how the music will behave in the game, and the degree of adaptability that will be utilized.
  • Technical requirements. This should cover the tools and technology that will be used or built to bring the score to life.
  • Integration and implementation. Outline how the music system will communicate with the game engine, and the techniques to be used for music implementation.
  • Production process. This should include asset creation, game implementation, and QA.

Adaptive Audio Technologies

Because adaptive music is at a relatively early stage in its evolution, there aren't many ready-made solutions for creating an immersive game score. Thus far, many companies are blazing their own trails by building proprietary interactive music engines and defining functionality based on the needs of specific games. Even off-the-shelf solutions such as DirectMusic need careful integration in order to function well. Therefore it's important to begin thinking about technical solution to your music needs at the onset of your project.

There are many approaches to adaptive audio, and many more that haven't been thought of yet. Any of these ideas and approaches can be used individually or combined to suit the needs of the project at hand. Before a specific music system and engine can be chosen or created for your game, the needs of that technology must be decided upon. Here are some adaptive audio solutions that are already available:

Wave files. The choice between wave and MIDI files used to be an either/or affair, but now games can use the best of both these worlds. Pre-mixed wave files, despite their linear nature, offer high production values due to the fact that any compositional technique, tool, or instrument, can be captured and mixed into a wave file using state-of-the-art equipment. Additionally, wave files can be arranged into small flexible wavelets, boosting their adaptive potential.


No One Lives Forever 2

TRON 2.0 (PC-Monolith/Disney) and No One Lives Forever 2 (PC-Monolith/Fox-Sierra), used wave technology in a highly adaptive manner. Each music state was divided into individual wave files of 1 to 4 measures in length, allowing seamless transitions between music states. The beginning of each wave file represents a transition point where the music can move to the next music state. The short waves overlap each other, or dovetail, creating a natural release and reverb decay during the state transitions.

MIDI files. MIDI files offer even more flexibility than wave files because the data is more granular. Each musical note is a piece of data that can be manipulated or shaped to suit the score. The tempo, harmonic content, and orchestration, can be altered quickly and transitions between sections are easier to create than with wave files. This places MIDI further down the spectrum of adaptability than wave files. In years past, it was difficult to create high-quality MIDI scores. Since then, we've learned that creating custom instrument banks tailored to the music at hand is the key. Each platform has a format for custom instrument banks (e.g., DLS for PC and Xbox, VAB files for Playstation, and so on). Creating and utilizing custom instruments is a specialized skill, and it has been done poorly more often than skillfully. This is partly why MIDI scores get a bad rap in the game industry. All the interactivity in the world is meaningless if the instrument quality is sub-par. When done well however, MIDI files with custom sounds offer the most dynamic music experience available with current technology.

As mentioned earlier, the choice doesn't have to be either waves on one hand or MIDI on the other. Wave and MIDI files can be combined to get the best of both worlds. One approach could be to use wave files as the main bed of music, and use MIDI files as musical accents and/or themes that layer over the wave files. This would let you take advantage of the quick response and harmonic flexibility of MIDI.

No One Lives Forever (PC-Monolith/Fox), and The Mark of Kri (PS2-Sony) are examples of highly adaptive scores using custom instrument banks and MIDI files.


Article Start Page 1 of 2 Next

Related Jobs

Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States
[10.15.19]

Project Manager
Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States
[10.15.19]

Outsourcing Manager
Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States
[10.15.19]

Environment Artist
Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States
[10.14.19]

Animator





Loading Comments

loader image