Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 25, 2014
arrowPress Releases
October 25, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Coding for Music
by Rich Vreeland on 06/02/13 09:09:00 pm   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Preface

Using code in games to actually create music, note by note, on the fly, is an underutilized idea that has potential to add huge value to games that aren't even focused on music, by offering an extra sense of authorship to the player's experience. From the simplest implementation, like the subtle musical sound effects in the menus of Super Mario Galaxy, to the generative music systems woven throughout Spore, the potential is broad and exciting.

 

Back in 2009, as an excuse to get better at coding and to make something in the process, I started exploring a game idea that involved a man walking around and licking snowflakes that made music. I created a bunch of sound assets of different tones, and based on what sounded good, I'd choose what asset to trigger next. Over time, I refined what I thought was a game into more of a music tool, adding lots of features such as chords, changing keys, different scales, and so forth. The following ideas all come from that tool, which is called 'January.' Check it out at http://january.cc.

 

Generating Something

So (maybe) you want to generate some music! For simplicity's sake, we'll create a program that uses the notes associated with the white keys on the piano, starting with the middle note of the keyboard, C3, and going up a single C Major Scale: C3, D3, E3, F3, G3, A3, B3, ending with the octave, C4. We'll get a series of new note choices by figuring out what the last note played was. The program then chooses one randomly, plays it, and stores it as the new last note played. This creates a process that can be repeated to randomly generate a curated musical sequence. */

 

// For the purposes of these examples I'm using Flixel and AS3, because it's what I'm familiar with.

 

/** The last note played, stored after each new note is played. */

public static var lastNotePlayed: Class;

 

/** Play a note! */

public static function playNote():void

{

var noteToPlay: Class = getNote();

 

// This is Flixel's function for playing sounds.

FlxG.play(noteToPlay);

 

lastNotePlayed = noteToPlay;

}

 

/** Get a new series of note choices, based on what the previous note was. */

private static function getNote():Class

{

var noteChoices: Array = [];

    if (lastNotePlayed == C3) noteChoices = [D3, E3, F3, G3, A3, B3, C4];

else if (lastNotePlayed == D3) noteChoices = [C3, E3, G3];

else if (lastNotePlayed == E3) noteChoices = [F3, G3, C4];

else if (lastNotePlayed == F3) noteChoices = [E3, G3];

else if (lastNotePlayed == G3) noteChoices = [C3, D3, E3, F3, A3, B3, C4];

else if (lastNotePlayed == A3) noteChoices = [G3, B3, C4];

else if (lastNotePlayed == B3) noteChoices = [G3, A3, C4];

else if (lastNotePlayed == C4) noteChoices = [C3, G3, B3];

else    noteChoices = [C3, G3, C4];

 

var choice: int = Math.round(Math.random() * (noteChoices.length - 1));

var note: Class = noteChoices[choice];

 

return note;

}

 

Generating Anything

The approach above works if you intend to stay in one scale, playing over one chord, but what if you want to have the generated notes adapt to different chords, scales, and eventually keys? To accomplish these things, a more sophisticated approach will be necessary. Storing your note choices as strings instead of using the sound classes themselves is one way to do this that will work. By separating out the logic choices from the actual sounds, it will be easier to adapt your system to new musical contexts.

 

A musician playing along to different chords in a single key is more often than not utilizing the various 'musical modes' of that key. A brief synopsis of what these modes are may be necessary if you're not familiar with the concept. The key of C Major is comprised of 7 modes, and they all use the white keys on a piano. Each mode is essentially it's own 7-note scale starting from a different note of the C Major Scale (C, D, E, etc.). By starting the scale from a different place, you get a different sound, and each of these are called modes. If you start on C, you're in Ionian Mode: C D E F G A B. If you start on D, however, you're still in the key of C, but you're in Dorian Mode (D Dorian): D E F G A B C. This concept continues on up the C Major Scale for E (Phrygian), F (Lydian), G (Mixolydian), A (Aeolian aka Natural Minor) and B (Locrian).

 

In order to make different musical choices based on what mode we're currently using, we'll need to come up with a modular system. Directly referencing the sound classes is no longer the best option, because the various modes, while possessing the same notes, tend to sound best when they use the notes in their own unique ways. A 'C' note is going to sound quite different depending on whether you're playing it over a C chord or a D chord. In order to accommodate this fact, the code below creates a unique object variable for each mode. We store that mode's position in the scale (ie. Ionian is the 1st mode, Dorian the 2nd, etc) and use a 2D array of strings called 'logic' to represent our potential note choices as scale degrees. A scale degree is the numerical representation of a note in the scale. For instance, in C Ionian, the 4th degree is F. They let us talk about a note in a musical key or scale without saying what the note actually is (ie. C, D, etc). Here is what our old C Major Scale would look like in this new form (Ionian), plus a new mode, Dorian:

 

public static const IONIAN: Object = { position: 1 };

 

IONIAN.logic = [ /* 00 one  */ ['two', 'thr', 'for', 'fiv', 'six', 'sev', 'one'],

/* 01 two  */ ['one', 'thr', 'fiv'],

/* 02 thr  */ ['for', 'fiv', 'oct'],

/* 03 for  */ ['thr', 'fiv'],

/* 04 fiv  */ ['one', 'two', 'thr', 'for', 'six', 'sev', 'oct'],

/* 05 six  */ ['fiv', 'sev', 'oct'],

/* 06 sev  */ ['fiv', 'six', 'oct'],

/* 07 oct  */ ['one', 'fiv', 'sev'],

/* 08 else */ ['one', 'fiv', 'oct']

];

 

public static const DORIAN: Object = { position: 2 };

 

DORIAN.logic = [ /* 00 one  */ ['thr', 'fiv', 'sev', 'oct'],

/* 01 two  */ ['fiv', 'six', 'sev'],

/* 02 thr  */ ['one', 'for', 'fiv', 'six', 'sev', 'oct'],

/* 03 for  */ ['fiv', 'six', 'sev', 'oct'],

/* 04 fiv  */ ['thr', 'six', 'sev', 'oct'],

/* 05 six  */ ['one', 'for', 'fiv', 'sev', 'oct'],

/* 06 sev  */ ['thr', 'fiv', 'six', 'oct'],

/* 07 oct  */ ['one', 'thr', 'for', 'fiv', 'sev'],

/* 08 else */ ['one', 'fiv', 'oct']

];

 

This is all fine and dandy, but these strings need to be hooked up to the sound assets in order to work. As a way of dealing with different modes, we can create a for loop that will fill a 'loadout' object with the proper notes, assigned to the proper scale degrees.

 

/** An array of the scale degrees, used as a reference for populating the loadout object. */ 

public static const DEGREES: Array = ['one', 'two', 'thr', 'for', 'fiv', 'six', 'sev', 'oct'];

 

/** An array of all the notes needed to potentially form an eight note scale in any mode (Ionian is C3 - C4, Dorian is D3 - D4, Phrygian's E3 - E4, etc.) */ 

public static const C_MAJOR: Array = [C3, D3, E3, F3, G3, A3, B3, C4, D4, E4, F4, G4, A4, B4];

 

/** The loadout object, to be filled with the notes of the current mode/key. */

public static var loadout: Object = {};

 

/** The last note played, stored after each new note is played. */

public static var lastNotePlayed: Class;

 

// For the sake of the example, let's say the current key is F# Major.

public static var currentMode: Object = DORIAN;

 

/** Play a note! */

public static function playNote():void

{

var noteToPlay: Class = getNote(currentMode); // Pass the current mode into getNote(), whatever mode you want it to be!

 

FlxG.play(noteToPlay);

 

lastNotePlayed = noteToPlay;

}

 

/** Get a new series of note choices, based on what the previous note was. */

private static function getNote(modeToUse: Object):Class

{

/** Whether we've found the last note played in the current loadout. */

var found: Boolean = false;

/** The note choices that we'll use to decide the next note to get. */

var noteChoices: Array = [];

/** Make sure that the current scale we're using to make logic decisions starts on the first note of the current mode. 

ie. in Ionian, loadout['one'] = C3, loadout['two'] = D3, etc.

in Dorian, loadout['one'] = D3, loadout['two'] = E3, etc. */

for (var i:int = 0; i <= DEGREES.length - 1; i++)

loadout[DEGREES[i]] = C_MAJOR[i + modeToUse.position - 1]; // - 1 is to account for starting at Zero.

/** Iterate through the loadout, find which is the last note played, then store the associated set of choices to noteChoices. */

loop: for (var j: int = 0; j < DEGREES.length - 1; j++)

{

if (lastNotePlayed == loadout[DEGREES[j]])

{  

noteChoices = modeToUse.logic[j];

found = true;

break loop;

}

}

/** If the last note played was not found in the loadout, use the 'else' choices from the logic array. */

if (found == false)

noteChoices = modeToUse.logic[08]; // [08], or the last set of arguments in the logic array, used like an else statement.

/** The numerical choice from the array of choices that we will use. */

var choice: int = Math.round(Math.random() * (noteChoices.length - 1));

/** The note that we will return to the playNote() function and eventually play. */

var note: Class = loadout[noteChoices[choice]] as Class;

return note;

}

 

The system above will work great for moving through various modes in a single key, but what if you want to move through the other 11 keys, too? The most straightforward solution would be to create arrays for all of the keys. You also would of course need to create assets for all the new notes that you are using.

 

// Arrays of all the notes in needed to potentially form an eight note scale in any mode, in any key. s = #, ie. Cs3 = C#3 

public static const C_MAJOR : Array = [C3, D3, E3, F3, G3, A3, B3, C4, D4, E4, F4, G4, A4, B4];

public static const Cs_MAJOR: Array = [Cs3, Ds3, F3, Fs3, Gs3, As3, C4, Cs4, Ds4, F4, Fs4, Gs4, As4, C5];

public static const D_MAJOR : Array = [D3, E3, Fs3, G3, A3, Cs4, D4, E4, Fs4, G4, A4, Cs5];

public static const Ds_MAJOR: Array = [Ds3, F3, G3, Gs3, As3, D4, Ds4, F4, G4, Gs4, As4, D5];

public static const E_MAJOR : Array = [E3, Fs3, Gs3, A3, B3, Ds4, E4, Fs4, Gs4, A4, B4, Ds5];

public static const F_MAJOR : Array = [F3, G3, A3, As3, C4, E4, F4, G4, A4, As4, C5, E5];

public static const Fs_MAJOR: Array = [Fs3, Gs3, As3, B3, Cs4, F4, Fs4, Gs4, As4, B4, Cs5, F5];

public static const G_MAJOR : Array = [G3, A3, B3, C4, D4, F4, G4, A4, B4, C5, D5, F5];

public static const Gs_MAJOR: Array = [Gs3, As3, C4, Cs4, Ds4, Fs4, Gs4, As4, C5, Cs5, Ds5, Fs5];

public static const A_MAJOR : Array = [A3, B3, Cs4, D4, E4, G4, A4, B4, Cs5, D5, E5, G5];

public static const As_MAJOR: Array = [As3, C4, D4, Ds4, F4, Gs4, As4, C5, D5, Ds5, F5, Gs5];

public static const B_MAJOR : Array = [B3, Cs4, Ds4, E4, Fs4, A4, B4, Cs5, Ds5, E5, Fs5, A5];

 

// For the sake of the example, let's say the current key is F# Major.

public static var currentKey: Array = Fs_MAJOR;

 

Then, you would adjust the for loop that fills the loadout to check for the current key's scale:

 

for (var i:int = 0; i <= DEGREES.length - 1; i++)

loadout[DEGREES[i]] = currentKey[i + modeToUse.position - 1];

 

And voila! With these systems in place, you now have access to all 7 modes in all 12 keys. That's 84 different scales! Other applications, such as chords, can be generated in a similar fashion, using 2D arrays to store chord choices and the loadout system as before to make sure you're in the right key and the right mode. You could also use a timer to fan out the playback of the chord notes, if you want to be fancy. The possibilities are endless!

 

Considerations

If you start to experiment and your generative music system gets more complex, you will probably run into some 'musical bugs'. Namely, you will find dissonances (notes that don't sound good together) and other unwanted events creeping up from time to time. You'll want to consider preventing certain sequences from happening, such as repeated notes, 'trills' (hearing the same note twice with another note in between), and if you start changing keys and scales on the fly, you may want to limit what kind of notes play around the time that those events happen. These kinds of problems can often be solved with conditional checks, often against the lastNotePlayed.  If you want to check for trills, you could create a second variable, secondToLastNotePlayed, and continuously log the last two notes. This could also open the possibility space for what kind of choices you make about which notes come next, if you're up to the challenge.

 

At the end of the day, the most sophisticated music generating code won't amount to much if you don't have someone musical onboard, using their ears, making decisions about what sounds good. Maybe that's you, or maybe not, but hopefully either way this will be a decent springboard for thinking about using code to generate music, instead of the other way around.


Related Jobs

Crystal Dynamics
Crystal Dynamics — Redwood City, California, United States
[10.21.14]

Audio Lead
The Odd Gentlemen
The Odd Gentlemen — Los Angeles, California, United States
[10.20.14]

Sound Designer
Digital Extremes
Digital Extremes — London, Ontario, Canada
[10.18.14]

Sound Designer






Comments


Antoine Ladouceur
profile image
Awesome experiment and gosh do I like January... so peaceful.

Rich Vreeland
profile image
thank you!

Ryan Christensen
profile image
Excellent, love the demo at january.cc, reminds me of alot of games at http://www.ferryhalim.com/orisinal/. I really dig both generative and reactive sound systems, reacting to gameplay and intensifying, chillaxing based on events in the game. At Cheyenne Mountain on Stargate Worlds the reactive sound system was one of the best parts of that project, it's sad it didn't get to market. We have some fun reactive sound in SupaSupaCross (http://supasupacross.com) that intensifies as the racer gets closer to the end (you can also add your own custom recordings or sounds to replace in game sounds that are mixed reactively) but can be taken farther in reacting to npcs or players/events around you.

Audio is sometimes tagged on at the end of game projects but reactive sound systems make sure it is part of the process earlier and generally make for a better gaming experience. Audio is 50% of the experience and gets 5% of the time unfortunately on game dev projects too often.

Christian Nutt
profile image
Oh man Orisinal. That brings me back.

Rich Vreeland
profile image
Cool! I love Orisinal! I was definitely inspired by that.

Joe Wade
profile image
This was a really cool article, thank you so much for this!

I decided to try and code it out myself and experiment with it, even though I have zero musical knowledge. Also - AS3 isn't my thing. So I decided to translate your example into C#, and thought I'd share.

I'm not going to fill up the whole comment with code, so just 4 quick pastebin links if anyone is interested.

For the top, simple example:
A Note.cs class - http://pastebin.com/guwAfG3H
and the rest of it in a class called CMajor.cs - http://pastebin.com/niikHD5k

For the next, more complicated part, it still uses the same Note.cs from above plus:
A Mode.cs class - http://pastebin.com/EtzZGNgp
and the rest of it in a class called CMajorModes.cs - http://pastebin.com/iRiA42Vv


I tried to do a pretty literal translation to keep it simple for a first pass. I'm going to continue to play with it though and clean it up some more. I'd appreciate any comments.

Thanks again, this was fun!

Rich Vreeland
profile image
Hey awesome! Yea, AS3 just happens to be the thing I learned in, so it was easiest for me :)

Definitely keep messing with it, would love to see how it unfolds! :)

Christian Coler
profile image
This is great! Can you give any more detail about why specific notes were chosen? For example, in the first bit of code, when F is the last note played, the only available notes to play next are E and G. is this a personal sound preference, or did you have some other way of determining how the list of available "next" notes should be limited?

Rich Vreeland
profile image
Thanks Christian!

Specific notes were chosen (and others were not), to intentionally limit the directions in which the music goes, and to give it a unique characteristic. The more options you give the system, the less predictable it becomes, and in my experience it is nice to find a good balance between a "style" and the unpredictability the system creates.

The way in which I determined which notes to use in my larger version of this system (at January.cc), was to sit at the piano and play various two note sequences that I was inputting into the system, to see what sounded best.

Another option would be to pick up some basic music theory, which helps with understanding when to choose thirds, fifths, fourths, etc.

Hope that helps! :)


none
 
Comment: