The Miracle of Performance Modeling

What is…

Performance modeling is the process of synthesizing the stylistic elements that make up important human nuances in music for application in electronic music. Without these elements, an electronic music performance can sound contrived and even unpleasant. Performance modeling works by artificially rebuilding performance characteristics and reintroducing them into electronic music performances. The goal is to craft a realistic and believable sounding piece of music.

Electronic musicians employ software tools, recorded MIDI data, various synthesis techniques and an understanding of orchestration to manipulate various performance parameters. By modifying the various parameters associated with a musical performance an electronic music producer can greatly enhance the quality of their work. It isn't an easy task but thanks to advances in performance modeling tools technologies things are getting easier.

Performance parameters are the elements of an instrumental or vocal performance that musicians and engineers manipulate to introduce expressiveness and variation into their work. Some general performance parameters common to all instruments and performances include pitch (note number), timing (note time), overall volume (controller #7 - volume), instrument volume (controller #10 - expression), velocity and panning.

The more subtle and more often ignored performance parameters specific to particular instruments or family of instruments include chord note arpeggiation on guitar, breath pressure on a brass or woodwind instrument, or use of various pedals on keyboard instruments. There are also side effects that result from normal mechanical use of an instrument. For instance, using different microphone placement techniques, recordists can control the amount of pick and fret noise captured during a guitar performance. These cues add greatly to a performance's perceived realism.

Consider a strummed string instrument performance. Note articulation for chords played on a strummed string instrument differ from those performed on a keyboard instrument. In most cases a synthesized strummed guitar part played on a keyboard will not sound authentic when compared to a strummed guitar part that is played on a guitar style controller. It's true that you could invest in a MIDI guitar to assist you in developing truly convincing synthesized guitar performances. But what if you can't afford a MIDI at the moment? What if you don't play guitar? What happens if your primary instrument is percussion? What if you aren't an instrumentalist at all?

Now I know that all game audio pros are virtuosos on all instruments and have infinite budgets, but for argument sakes lets assume for a moment that MIDI guitar isn't an option for you. You can step record or worse draw the notes in by hand. You are inevitably going to spend hours tweaking note velocities and positions to get a semi realistic sound. This method isn't practical. It is time consuming, unintuitive and ultimately frustrating.

Hardware and software performance modeling tools offer electronic musicians an alternative to the tedium of manual editing. Their interfaces are more precise than random generation methods and more intuitive than slaving note after note with a mouse. In the body of this article we will view those various options. But first lets talk about the role of MIDI in all of this.

Why MIDI Still?

The demise of MIDI is one of those things is prophesized quite often. If we dig deep enough into Nostradamus's notes I am sure we will find something about MIDI and doomsday. I have news for the doomsayers. MIDI isn't going anywhere, in fact there have been some pretty big steps forward for MIDI recently with DLS-2 (see below) and XMF. MIDI is a cost efficient manner of delivering musical content in interactive media. MIDI is an indispensable tool for game developers because of its flexibility and small size. Digital audio is just too big for some projects and can get expensive. MIDI also works great as a basis for interactive music composition.

If MIDI has so much to offer then why are some people so anxious to see it go away? The answer is simple. In the wrong hands MIDI sounds really awful. Most lay people don't understand why MIDI can sound so bad. They hear a quantized sequence played through a 500K sound set and their immediate reaction is "Wow that MIDI thing sounds awful! It's not going in my game!" They don't realize the various factors that are involved in creating a realistic sounding piece of sequenced music.

Obviously part of the problem with bad MIDI is a poor quality sound source. This isn't as big of a roadblock as it was a decade ago thanks to drastic improvements in sample based synthesis and sound synthesis in general. This is especially true today with hardware sequences sporting 512Mb of sample RAM and software synthesizers allowing sample sets up to about 4Gb! As music memory budgets improve on consoles sequenced music, and more importantly interactive music, are going to level up.

The other deciding factor in the perceived realism of a sequence relates to the expressiveness of the performance itself. Step editing in a performance and running a humanizing function usually falls short of perfection in terms of creating a convincing performance. This method usually takes way too long to get the desired result, if that is possible at all. Professional electronic music composers need robust tools that allow them to model performance data in intuitive ways.

Hardware manufacturers have provided a number of useful and often times esoteric pieces of gear to help acoustic musicians translate their performances into digital data. Thankfully a number of software manufacturers have begun developing software that allows non-instrumentalists to model performance data using GUIs. Using these various tools electronic musicians gain new levels of control over the various performance parameters involved in music production.

Sound Synthesis

Sound generation methods for electronic music have come a long way since their inception. Both modeling and sample based sound generation give musicians flexible means for sound design and musical performance. The quality of a production's sound sources and the eloquence of their designs play a role of obvious importance in delivering professional level electronic music for interactive media.

There are plenty of professional grade multisamples and modeling synths on the market that can be manipulated to display truly expressive responses during musical performances. Hard disc streaming of samples allows for massive amounts of sample data and therefore plenty of opportunities to add additional performance modeling material. Physical modeling synths are great for reproducing the realistic aspects of an instrument's performance characteristics. These aspects are often capable of being modulated in real time as well making physical modeling a truly expressive and powerful sound generation method.

Unfortunately the benefits of large professional samples and powerful modeling synthesizers are beyond the scope of most interactive media productions. Audio being what it is in interactive media, we don't always have the resources or the tools to be picky with how our message based music gets reproduced. Modeling technologies are generally too tasking on CPUs to justify use in music production. Sample playback is often limited to sometimes obscene file sizes, and will be until a method that allows for larger samples to be used for playback (picture GigaSampler style disc streaming on the next-generation of consoles… or PCs for that matter!).

This is especially true for portable and mobile games. You can't fit a 6GB-woodwind bank into a cellular phone. And GBA doesn't have the power to dedicate to cycle-hungry physical modeling synths. Crude waveform generators and small samples are the sound sources we have to construct our masterpieces. That being said (and widely understood), interactive audio pros make due with what they have. Among the things we do have is DLS-2. DLS-2 (Downloadable Sounds Level Two) is an open standard sampling format based on Sound Font 2.0 that many game developers implement into their titles. DLS-2 provides synthesis tools like LFOs, LPFs, a six-stage envelope and the ability to layer voices.

I mention DLS-2 for two reasons. First off there are a number of sound library providers that develop for DLS-2. You can avoid the agonizing process of taking a masterfully recorded set of instruments and squeezing the life out of them so they can all fit into half of your 4Mb sound budget. Secondly, DLS-2 is supported natively in DirectX Audio. DirectMusic Producer has a built-in DLS-2 Sampler that you can use to edit or design your own banks. Remember DMP is free so check that off your budget list.

Without the use of some sampling standard like DLS-2 or proprietary synthesis tools, you won't have a method for molding your raw sample and synth data. For example, a modeled guitar performance triggering a singular recorded guitar note for the entire range of the instrument with no regard for velocity or other performance parameters won't sound very realistic. In fact it will sound quite bad. Compare that to a modeled guitar performance triggering a multisample that includes a layer of fret buzzing and a layer of string noise. This fret buzzing may be introduced when the sequence triggers predefined notes at certain velocity levels. The audio developer may script a behavior for introducing the string noise at random intervals.

These effects are subtle but add greatly to the perceived realism of the musical performance. Where is the power of music and sound design if not in its subtlety after all? You can't do all this without velocity switching, envelopes, algorithmic tools etc. Synthesis tools (filters, oscillators, envelopes, effects etc.) allow us to reproduce performance parameters integral to modeling realistic musical performances. LFOs, filters and the like can model performance nuances such as vibrato, tremolo, dynamics etc. Not only are they essential in producing appropriate samples for music data playback, they can be used for sound design as well (as in the case of cone filtering for achieving directionality effects in DS3D).

There is a greater potential for these and newer synthesis methods in interactive media as music and sound design become more interactive. As interactive music begins to resemble real-time performers, programmable/scriptable real-time synthesis controls will become quite valuable. This is something to consider when setting forth your tool requirements to your audio programmer.