Adaptive
Music
By
Andrew
Clark
Gamasutra
May
15, 2001
URL: http://www.gamasutra.com/resource_guide/20010515/clark_01.htm
What is music?
In a
freshman "Introduction to Ethnomusicology" course I attended at the
University of Toronto, the entire first class was dedicated to attempting to
agree on a definition of "music." Easier said than done! Luckily,
since this paper is a solo effort, I get to pick the definition without argument.
Music: structured patterns of sound in time.
What's in a
name?
In many
ways that was a terrible definition. For one thing, it relies heavily on the
fact that you already know what music is. (The word "music" is a symbol
with which you already have years of personal experience.) Really, its sole
purpose is to focus our attention on some fundamental elements of music, without
getting distracted with sticky issues like art, creativity, culture, human involvement,
and intent.
Sound Structures
in Music
Part
of the reason that music is so appealing is the number of levels of patterns
that are going on at once. Sound comes from very fast air pressure changes at
rates of roughly 60 to 23000 times a second. Phrases involve the arrangement
of these sounds on a much different temporal scale. The sounds become melody
and rhythm, and harmonic motion. Structure is a higher level organization of
phrases. (E.g. classical Sonata form, or the ordering of verses and choruses
in a pop song.) Very different parts of the brain process and appreciate these
different elements simultaneously when you are listening to music.
Equivalent
structures in language
A non-musical
structural equivalent of a sound would be a word in language, e.g. the word
"sound". The equivalent of tone (the quality or "color"
of a sound) might be font. This sentence is a good example of a phrase. This
paragraph could be thought of as a sub-structure, and the essay as a whole as
a much larger architectural structure or form.
Structures and Adaptability in Language
This article explores adaptive art in the context of language. This is because language is much easier to demonstrate and discuss in the written medium than music is! It should also allow a much wider readership. So, despite the title "On Adaptive Music," this article doesn't talk about music at all - except by analogy. Part Two will take the lessons we learn from this journey and apply them to actual adaptive music solutions.
An Adaptive
Poem
I used to write
video game music for a living. It was a very exciting job, but it also had its
share of frustrations. For instance, part way through one particular project,
I discovered that the lead programmer hadn't bothered installing a sound card
in his new computer. As one might expect, this was making it kind of hard for
him to test the interactive score I was writing. (True story! The next part
is made up though.)
I decided that it might be a good idea for me to write some interactive poetry
for him to work with temporarily. (We could even keep it as an option in the
release version of the game, for the hearing impaired.) Like the music, the
poetry would change in real time in order to reflect the changing action of
the game and to enhance the atmosphere. It would be of extremely high quality,
and interesting enough that the player wouldn't get sick of it after a mere
eight hours of game-play.
An adaptive
poem?!!
Writing adaptive poetry sounds impossible, doesn't it? Well, it's essentially
the same challenge that an adaptive music composer faces… but more on that
later. For now, let's try to prototype an adaptive poem and see what happens.
The first step will be nailing down the tone and style, and what we want the
piece to communicate. The following example is a linear poem that will help
us figure out what we want the interactive version to do. (The game, by the
way, is called Seal Hunter and features the heroic Captain Brave-O.)
Example: Linear Poetry
The cold dawn breaks on the arctic ice
Gleaming hard on a cruel device:
Captain Brave-O's clubbing hook
Dripping blood from the lives it tookThe evil seals are everywhere
Breathing our precious human air
Soon the air will all be gone
And no more human breaths be drawnThen: BAM BOOM BANG, KER-CHUNK, KER-SPLAT!
The hook is WIELDED like a BAT
CAPTAIN BRAVE-O's in the FRAY,
HORDES of EVIL SEALS to SLAY!
His breathing slows, the battle done
This war, he knows, is far from won
And though there's not a seal in sight
It's far too quiet to be all right
Adaptability
at the word level
The action and
mood of Seal Hunter can change quite suddenly. Captain Brave-O could be walking
around being brave and heroic one minute, and then diabolical seals could leap
out at him the next! Ideally, our accompanying poem should be just as dynamic.
Unfortunately, since the game player will be controlling Captain Brave-O's actions,
we have no way of knowing in advance what is going to happen, when. One way
to try to deal with this is to compose a poem that is flexible at the word level,
so that it can change moods at any time.
The next expample randomly selects words from one of four word lists. Each list
contains only words that are appropriate to a particular game mood. When the
action of the game changes, the source word list is also changed, thereby changing
the mood of the poem on the fly.
Example: Adaptive Word Texture
(The level begins as Captain Brave-O enters heroically. Poem mood: HEROIC) Captain Brave-O brave hero marching gleaming brave hero Captain brave adventure marching hook Brave-O brave brave hero (The landscape changes; seals are nearby. Switch poem to CREEPY) danger fear shadows lurking seal evil danger caution scary (Seal surprise attack! Switch poem to COMBAT) bam ker-chunk ker-splat hook bat fray slay bang ker-splat bat slay slay ker-chunk bam ker-splat slay slay slay (The last seal is subdued but the territory is still scary. Also, Captain Brave-O's health is low. Switch to CREEPY, INJURED combo) injured hurt danger fear injured lurking seal evil danger caution shadows
That was surprisingly effective and very dynamic; however, aesthetically it might be considered somewhat lacking. It was more like word texture than traditional poetry -- there was no real attention to rhyme or meter. Let's see what happens when we try a less random ordering of words in an attempt to address this:
Example: Adaptive Word Texture with Meter and Rhyme
(The level begins: HEROIC)
mighty gleaming courage gleaming
roaming courage hero marching
Brave-O mighty Captain marching
Gleaming courage courage roaming
Mighty courage Brave-O Brave-O
Gleaming mighty roaming hero
Courage Captain gleaming marching
Hero mighty --
(Surprise seal attack! COMBAT)
fray seal bam bat
slay slam spray splat
hook fray squeal slam
bat seal slay bam
fray slay fray seal
splat…
(The battle ends.)
The second example
was essentially the same as the first, except for two small changes. This time,
the word list for a particular mood consisted only of words sharing the same
foot. Additionally, I added special lists of rhyming pairs. Every fourth word
was selected from this list, in order to form rhyming couplets. The result was
a kind of pseudo phrase structure (but without any real sentence-level meaning).
This more structured version came at a slight cost. Notice that when the sudden
seal attack happened, it broke the rhyme and rhythm scheme of Brave-O's strut
poem. If we'd waited for the line to finish, the attack poetry would have been
late and much of the effect would have been lost. Having to make this kind of
compromise makes this solution less dynamic than pure word texture. On the other
hand, patterns of rhyme and meter do have an undeniable attraction.
Patterns
The human brain is fantastically adept at recognizing all kinds of patterns. The human brain is fantastically adept at recognizing all kinds of patterns. The human brain is fantastically adept at recognizing all kinds of patterns. (I bet you skipped at least that last sentence without even reading it.) Patterns in any medium involve some form of repetition, and have a certain aesthetic appeal. Unfortunately, with over-repetition comes boredom. Repetition provides unity and cohesiveness; variation provides interest. This is always a delicate balance!
 |
|
Adaptive
Word Texture with Meter and Rhyme
|
What is interesting about these examples is how important scale is when balancing cohesiveness and interest. For example, when your brain looked at the boring pattern, it saw repeating "t"s. Boring. When it looked at the less boring pattern, it saw the same repeating "t"s; however, its interest was held by the larger grouping patterns that began to emerge. Spend some time with each of the patterns and think about the meaning of "t" in each one. I'm serious! There is a lot to be learned from these simple examples.
Boring Adaptive
Poetry
Think way back to "Adaptive Word Texture" examples. Which adaptive
poem would get boring faster? Most definitely it's the "Adaptive Word Texture
With Meter and Rhyme" version. In this example, two patterns (meter and
rhyme) repeat constantly without variation. By the second couplet, the pattern
has become recognizable. By the third, it has become predictable. By the fourth,
it begins to become tedious.
One might argue that the "Adaptive Word Texture" poem's structural
organization is in fact more repetitious and therefore more tedious. (After
all, it's essentially just word, word, word, word, word…) Why isn't this
the case? Well, in the first version the simplicity of the pattern works in
its favor. It's unobtrusive -- it's basically a non-pattern. It doesn't distract
from the interesting unpredictability of what the next word might be. By contrast,
the higher level structural patterns in the second version engage the part of
the brain that pays attention to higher level structural patterns -- and becomes
bored with them. In this case, the (repetitious) structure dominates the (non-repetitious)
content. This unbalance is due in part to the fact that our content only works
at the word level, while the patterns (and the brain) are working at the phrase
level.
Adaptability
at the Phrase Level
The problem with
word textures is that they are extremely limited in terms of actually expressing
ideas. Compare "danger fear shadows lurking seal evil danger" (from
"Adaptive Word Texture") to "the enemy could attack at any moment."
The first "sentence" is a pseudo-random collection of symbols. The
second uses syntax to combine symbols into a coherent thought. It actually says
something. (And you thought syntax was only important for programmers!)
What happens when we try to write an adaptive poem that actually communicates
something at the sentence (phrase) level? A logical place to start might be
with a sentence texture. This will work in much the same way as the word texture
example, except that we will use four lists of complete sentences.
Example: Adaptive Sentence Texture
(SPOOKIE cue) This place is really scary. Is that shadow moving? It's really dark here. The enemy could attack at -- (COMBAT cue) Hot blood spurts like a fountain! The fighting is fast and furious! The violence is incredibly intense and visceral! Teeth are gnashing and flippers are slashing! The fighting is fast and furious! Seal bits are flying everywhere!
An important thing to consider is the production cost of sentence texture compared to word texture. A word list that is twenty elements long requires only twenty words. A sentence list that is twenty elements long requires more than one hundred words! Plus, the production of a sentence involves a process of composition.
Variety at the
sentence level requires a bigger budget than the same amount of variety at the
word level. On the other hand, a single sentence covers more "ground"
in terms of words and time. The "Adaptive Sentence Texture" example
used far fewer elements than the"Adaptive Word Texture" example, but
was just as long. In terms of pure variation over time, perhaps fewer sentences
are required.
Another issue is that complete ideas demand more attention from the player.
The danger is that, instead of enhancing game-play, we're going to start distracting
the player from it. There's an old saying among filmies: "The best soundtrack
is the one you never hear." (You should be feeling it instead.) The idea
is that if you notice the music, it's because the composer's done something
wrong. Well, the same goes for adaptive poetry. We want the player to respond
on an emotional level, not an intellectual one. Complete sentences are dangerously
high level.
Lastly, sentence texture is a lot less flexible than word texture. It takes
longer to state an idea than it does to state a word. Our "Adaptive Word
Texture" example could switch moods literally in mid-stream. However, when
dealing with adaptive structures at the phrase level, we're suddenly faced with
logistical and creative concerns about what to do when the game mood changes
in mid-sentence. How do we make a smooth and effective transition?
Transitions
That was a pretty
good transition, wasn't it? It was even at a higher structural level than mere
sentences, or even paragraphs. I changed sections almost seamlessly! OK, not
so impressive. I already knew where I was going, and timing my arrival wasn't
critical. So, I was able to compose my sections so that they fit together just
so.
In adaptive poetry, timing is critical. In order to enhance the mood, changes
have to happen with the mood. If cues are delayed, even by a second, they aren't
contributing to the emotional experience of a game event. They become their
own distinct event, separate from the associated game event - which the player's
brain will already have had time to process. Poetry mood changes have to happen
on time!
In the "Adaptive Sentence Texture" example, this wasn't a big deal.
When the seals sprung their surprise attack, it made sense to suddenly interrupt
the previous mood. Ending a combat mood might be more complicated, though. Imagine
that the sentence "When will the carnage end!?" is one of our combat
list elements. Imagine that Captain Brave-O strikes down his last opponent midway
through the playback of this particular sentence. We want to switch to victory
poetry, and we don't want to be late. We also don't want to be awkward. One
way to address this situation is to compose transition versions of the combat
sentence. If the mood changes mid-way through the statement, we continue smoothly
with the transition version.
 |
|
Adaptive
Word Texture with Meter and Rhyme
|
Immediately, the cost of this approach should be obvious. Each element of the sentence list will require similar treatment if the transition to victory is to be possible at any time. Additionally, transitions to other moods may be necessary. ("When will Captain Brave-O be deader than this?") On the other hand, it does what it's supposed to de very well. What more could we ask for?
Going for Broke
Example: Adaptive Sentence Texture with Meter, Rhyme, and Transitions
(HEROIC cue)
Captain Brave-O, he's our man
Bravest soul in all the land
Hard as steel and twice as smart
Tough as nails and…
(SPOOKY cue)
…knows these parts
Wow, it's really scary here:
Smells of fish and smells of fear
Is that a shadow moving there?
Damn seals could be…
(COMBAT cue)
…oh, beware!
Bam boom bang, ker-chunk, ker-splat!
Filthy seal, do you like that?!
Seal bits flying everywhere
In a frenzy, just don't care
When, oh when, will the…
(VICTORY cue)
…seals try again?
Victory is ours, my friends!
Knowing Your
Arsenal
As these examples
get progressively high-level in terms of structure, I am reminded a little bit
of the Seal Hunter weapon selection options. The UltraNinja2000 (a light,
fast club) is perfect for lightning strike strategies, but doesn't do very much
damage. The more expensive Crusher is slower and difficult to wield effectively,
but makes an impressive impact when it connects! Each weapon has its place,
and is useful in different game situations. In fact, choosing the right tool
for the job is a big part of winning the game.
The same thing is true of the different adaptive strategies that we have examined.
Each has advantages and disadvantages. There is no one perfect choice that is
right for every game context. The best approach is to use all of them as necessary,
applying the most effective solution on a case by case basis. "What do
we want to hit the player over the head with for this part?" Choosing the
right tool for the job is a big part of winning the game. It also helps to have
a secret weapon handy…
Adaptability
at the Letter Level
Those of you who
have been paying attention to the structure if this paper will have been expecting
either A) an example of adaptive poetry combining multiple approaches or B)
"Adaptability at the paragraph level." Instead, in a surprise move,
I am unveiling a secret weapon that's been hiding under our noses all along!
It's easy to overlook adaptability at the letter level because we don't normally
think of packing much useful meaning into a single letter. If we could, it would
offer incredible advantages in terms of flexibility. (Even faster than the UltraNinja2000!)
Example: Adaptive Letter Texture
(The level begins as Captain Brave-O enters heroically. Poem mood: HEROIC) thbifaarakopthbifaarako (The landscape changes; seals are nearby. Switch poem to CREEPY) pthbifaarakopibysal (Seal surprise attack! Switch poem to COMBAT) TLSWERIPIAMISFORAHACAA (The last seal is subdued but the territory is still scary. Also, Captain Brave-O's health is low. Switch to CREEPY, INJURED combo) uwocbrpuacvpitiaadb
Shockingly, that
little example of pure adaptive letter texture was packed full of meaning! How
could this be? Well, don't forget that the game is its own narrative. The player
already knows what's going on. Our job as adaptive poets is to merely to enhance
the mood of the story -- telling it is somewhat redundant. Still, adaptive letter
texture on its own is somewhat limited in terms of what it can express.
(HEROIC) The real power of adaptive letter texture is its ability to com-(CREEPY)-bine
with other techniques. It is fully com-(COMBAT)-PATIBLE
WITH ALL OF THE OTHER APPROACHES THAT WE
HAVE DISCUSSED. IT COULD even BE USED TO
add AN ADAPTIVE element to
PRE-COM-(CREEPY, INJURED)-posed
material… and even com-(NEUTRAL, INJURED)-pletely
unrelated prose!
Notice too that with this technique we've added an "analogue" element
to our mood swings. Instead of limiting moods to either ON or OFF, BLACK or
WHITE, we've added some shades of gray (and red…). For instance, the frequency
of red characters (or completely red words) can indicate the status of Captain
Brave-O's health meter, without interfering with the content or structure of
other levels.
Seal Hunter: The Final Battle
Now that I've introduced my secret weapon, I'll leave you with one final example. The final example demonstrates how one might combine some of these approaches into a complete adaptive game poetry solution. Don't forget that the purpose of this exercise has been to learn something about adaptability in structured art forms, so that we can apply it to making interactive video game audio soundtracks. Meanwhile, think about the example that follows. What techniques are being used? Why did the game's Poetry Director make the choices he did? How much would this cost to implement and produce?
Example: Seal Hunter, Level 10
Captain Brave-O's back again,
He'll kick tail in Level 10!
Go Captain Brave-O, Go!…
…
Tum te tum…
…
…
Tum te tum……
Tum te tum…
…
Tum-tum…
…
Tum-tum…
…
Tum-tum tum-tum…
…
Tum-tum tum-tum tum-tum tum-tum tum-tum tum-tum tum-tum tum-tum
SEAL ATTACK! SEAL ATTACK!
THIS ONE'S BIG BUT DON'T TURN BACK!
GO CAPTAIN BRAVE-O, GO!
BAM KER-CHUNK OW! KER-SPLAT HOOK bat FRAY SLAY BANG ker-splat OW! bat SLAY SLAY ker-chunk bam ker-splat SLAY slay slay
His breathing slows, the battle done
This war, he knows, is far from won
And though there's not a seal in sight
It's far too quiet to be all right…
…
Tum te tum…
Annotated Bibliography
Hofstader, Douglas R. Gödel, Escher, Bach - 20th Anniversary Edition: An Eternal Golden Braid. New York: Basic Books, 1999.
Hofstadter, Douglas R. Le Ton beau de Marot - In Praise of the Music of Language. New York: Basic Books, 1997. Both Hofstader works explore symbol and meaning, form and content. The former has cognition and self-awareness as its central theme, while the latter is about communication and relationships. Hofstader draws deep and engaging analogies between his themes and the systems and structures that he explores (math, art, and music in the former; poetry, translation, and language in the latter). Frequently (particularly in GEB), the very structure and/or form of his writing demonstrates the content - an impressive device that adds an extra dimension to his exposition. Both of these books contributed enormously to the form, content, and even tone of my little paper.
Jourdain, Robert. Music, the Brain, and Ecstasy - How Music Captures Our Imagination. New York: Avon Books, Inc., 1998. A very broad survey of the phenomenon of music, from the physical and biological to the psychological and the sublime. Not an academic reference work, it's very readable - yet obviously exhaustively researched across many disciplines. Especially interesting to me were the explorations of how different elements of music (particularly patterns at different structural levels) are processed in different parts of a listener's brain, and the incredibly complex relationships that are involved in even the perception of music. OK, put it like that and it sounds boring, but seriously - anyone who is even remotely interested in music or audio will thoroughly enjoy this book. I'm not kidding, buy it! The takeaway is universal, despite a fairly western-classical-tradition-centric focus. Room for a sequel I guess…
Schoenberg, Arnold. Fundamentals of Musical Composition. London: Faber and Faber, Ltd., 1970. You wouldn't think that one could learn how to compose from a book, but something special clicked into place for me with this one. Partly the timing was just right - years of intensive training probably had something to do with it… Whatever it was, I really did understand music differently after reading this book. Unexpectedly (considering Schoenberg's prominent role in the deconstruction of tonality), the book limits itself strictly to traditional classical composition. It doesn't discuss or challenge tonality at all; it merely explains how and why the structures work. Despite the specific focus, there are lessons to be learned here about cohesiveness of form, repetition, and variation that can be applied to structures in any genre (…any medium?). Kind of an enigma is that the structure of the book itself is confusing at some points almost to the point of loosing the point entirely. Also, unlike the other works cited in this bibliography, it's extremely dry - unless you're really interested in classical music theory. Nonetheless, for me it was a gem. Grab it… if you can find it!
About the Author
Upon finishing
his BMus (Composition) at the University of Toronto in '97, Andrew was snapped
up by Toronto-based game design company Pseudo Interactive (http://www.pseudointeractive.com).
While serving as Audio Lead and Lead Composer, he picked up some programming
chops one the side and ultimately took over as Audio Programmer as well. Andrew
left Pseudo Interactive in September '99 in order to spend his life savings
on a geographically challenged game audio consulting business, and to focus
on independent research and development of his live interactive music performance
software "CarpeDM." (An early prototype of this software was a highlight
of the MS DirectX Developers Day 2000 audio track.) Since May 2000, Andrew has
worked as an ERP solutions developer for FutureLink Canada (http://www.futurelink.com)
by day. By night, he is the super-DJ technomancer / crime-fighter known only
as "PIr8_x" (http://www.silvershard.com). His e-mail address is andrewclarkis@home.com.
Comments, criticisms, and random thoughts are welcome!
http://www.gamasutra.com/resource_guide/20010515/clark_01.htm
Copyright © 2004 CMP Media Inc. All rights reserved.