Interactive Music Sequencer Design
By Scott Patterson
Gamasutra
May 15, 2001
URL: http://www.gamasutra.com/resource_guide/20010515/patterson_01.htm

Interactive music sequencer design for games is a real mouthful -- or rather an earful -- and a pretty large topic to boot. I'll begin with a summary of the motivations for making interactive music and then get in to the design discussion. To make the design discussion a bit more manageable, I will assume a lot of familiarity with music and synthesizer details and put greater focus on design options and programming methods. First, I will talk about the design issues in making a computer music language. Second, I will consider the controls available with audio synthesis. Third, I will add the issues related to providing interactive music functions for game control. Throughout, I will focus on concepts that directly influence interactivity, and point out approaches that make the complexity more manageable and help to make implementation practical.

Motivations for Interactive Music

Most forms of electronic entertainment include music. Games, movies, and TV shows are seen as incomplete without music. We hear music at the introduction to news programs, on our mobile phone rings, in department stores, and at coffee shops.

Music is its own form of entertainment. We listen for styles, attitudes, technology, improvisation, composition, and skilled performances. Our memories associate music with past situations, friends, places. We associate music with love and hate.

Games include music for many reasons: To identify with a particular audience. To establish attitude, tension, and mood. To let them hear the pride and glory of success or shame and ridicule of defeat. To march them off to the drums of war. To take them to a magical place. To set them in the past. To set them in the future. To take them to alternate worlds. To bring them back to reality. Music instantly adds definitions and associations beyond what the visuals can do.

Games are interactive. This means a player has control over the game in some way and the game asks the player to interact in some way. This control and interaction are the basis for how a game becomes immersive and entertaining. The quality of control and interaction are the basis of successful games.

Therefore, it is natural to want to mix the immersive quality of control and interaction in computer games with the immersive qualities of music. How do we control music? How can we create musical interaction? This is the motivation for this article.

The reasons for developing your own interactive music sequencer code are the same as the reasons for any code development. You may want standard features across many platforms. There may not be systems available that meet your needs. You may want an implementation that you can optimize for delivering the particular features you need. You may want control over your own code to provide improvements, enhancements, and reliability in line with internal scheduling requirements.

Making a Computer Music Language

Event Blocks
Music can be described as events or commands occurring over time. We can build our music command language with "event blocks" composed of three elements: time, event type, and event details. The time is stored relative to previous events and is called a delta-time, the event type is identified with a number, and the event details are zero or more parameters that are defined by the event type.

Figure 1 - Event Block

With this basic design we can describe any groupings of events and time intervals. Building our computer music language is now a task of choosing how to store the delta-time, what event types, and what details are needed for each event type.

MIDI (Musical Instrument Digital Interface) Music
The MIDI specification has been around for a long time. Most composition and sequencing software provides compatibility with MIDI input and output ports, and these products also provide options to save and load MIDI files.

Since the MIDI specification defines a delta-time storage format and a useful set of event types and parameters, we should start with this model as a reference. This will make it easier to convert the MIDI language to our custom music language.

Keep in mind that our goal is to create music directly on a machine with audio capabilities, so we want to take what is useful from the MIDI design, but not restrict ourselves to its features. We are also likely to get music from composers in the MIDI file format, so we will want to convert the MIDI file music language to our own custom language.

MIDI Events
The MIDI specification defines several events called the channel voice messages, and there are also meta-events defined in the MIDI file format 1.0 specification. Some of the meta-events simply contain text string data. I summarize the MIDI events and meta-events we are interested in Table 1, Table 2, and Table 3.

Table 1 - MIDI Channel Voice Message

Table 2 - Basic MIDI Event Types

Table 3 - Text MIDI Event Types

Of the MIDI channel voice messages, the control change event is unique in that it specifies an additional event identifier, the controller ID. Examples of common controller ID numbers are 7, which refers to volume and 10, which refers to pan position. Many controller ID numbers are not commonly used and we can have musicians insert control change messages with controller IDs to mark the music in special ways.

The basic meta-events listed give us ways to get time organization details and tempo changes. The text meta-events give us ways to get custom data embedded in strings. Composers can put special track settings in the track name strings or special kinds of playback commands in lyric strings. These methods will save us from having to write custom tools for composers when deadlines are looming.

MIDI Channel and Tracks
The "channel" in MIDI channel voice messages refers to the fact that these messages are sent with a channel number from 1 to 16 (0 to 15). The channel number is used by MIDI devices to route commands. We are not routing commands to MIDI gear, so we won't talk about channels anymore. Type 1 MIDI files have any number of tracks where each track contains a series of event blocks. It is the track concept that is useful for our data organization.

Sequences, Tracks, Events, Instruments, and Voices
This is where we leave the world of MIDI and talk about our custom music sequencer language. A quick summary of the terminology used in the rest of this paper is in order: A Sequence is a collection of Tracks that run simultaneously. Each Track is a sequence of Events that control the current Instrument. Certain types of Track Events will turn on and off Voices of the current Instrument. Some types of Track Events will modify all active Track Voices. You can think of an Instrument as the default settings given to a Voice that is turned on with a NoteOn Event. I will present a list of class definitions that display this structure in the implementation section.

Event Type Possibilities
Table 4 lists many event type ideas so you can get a taste of the possibilities. There are a lot of details that could be discussed ast to how these event types might be implemented, but I will just mention the options here and move on to the other parts of this discussion.

Table 4 - Event Type Ideas

Sequencer Data Structures
The main data structures that we could use for our music sequencer are listed in Code Example 1.

Code Example 1 - Music Sequencer Data Structures

typedef list< Sequence * >  SequencePtrList_t;
typedef list< Track * >        TrackPtrList_t;
typedef list< Voice * >        VoicePtrList_t;

class MusicSequencer_t {
     MusicSequencerState_t  State;
     SequencePtrList_t           ActiveSequencePtrList;
     SequencePtrList_t           FreeSequencePtrList;
     TrackPtrList_t                  ActiveTrackPtrList;
     TrackPtrList_t                  FreeTrackPtrList;
     VoicePtrList_t                  ActiveVoicePtrList;
     VoicePtrList_t                  FreeVoicePtrList;
};

class SequenceState_t {
     Tempo_t    Tempo;
     Volume_t   Volume;
};

class Sequence_t {
     SequenceState_t State;
     SequenceState_t BeginState;               // Interactive feature
     SequenceState_t EndState;                  // Interactive feature
     SequenceInterpolator_t Interpolator; // Interactive feature
     TimeUnit_t         TimeElapsed;
     TimeUnit_t         TimeStep;
     CallbackFunc_t *pCallback;                 // Interactive feature
     TrackPtrList_t    TrackPtrList;
};

class TrackState_t {
     Volume_t     Volume;
     PitchBend_t PitchBend;
     Pan_t         Pan;
     Effect_t      Effect;
};

class Track_t {
     TrackState_t      State;
     TrackState_t      BeginState;               // Interactive feature
     TrackState_t      EndState;                  // Interactive feature
     TrackInterpolator_t Interpolator;      // Interactive feature
     Sequence          *pOwner;
     char                    *pEvent;
     Instrument_t    *pInstrument;
     VoicePtrList_t   VoicePtrList;
};

class VoiceState_t {
      SynthVolume_t   Volume;
      SynthPitch_t        Pitch;
      SynthPan_t          Pan;
      SynthEffect_t      Effect;
};

class Voice_t {
     VoiceState_t                CurrentState;
     VoiceState_t                BeginState;     // Interactive feature
     VoiceState_t                EndState;        // Interactive feature
     VoiceInterpolator_t    Interpolator;   // Interactive feature
     Track_t                        *pOwner;
     char                               nKey;
};

Event Data Structures
To implement the event type commands we can have the event type command numbers correspond to an array lookup that holds the relevant function pointer and the byte length of the event type and parameters. Code Example 2 shows this code.

The function pointers give us a quick way to get to the code associated with each event type. The byte lengths give us a quick way to step to the next event block.

Code Example 2: Event Type Data Structures

// Example Note Off Event Block
typedef struct {
     char nEventType;
     char nKey;
     // no release velocity
}NoteOff_EventBlock_t;

void NoteOff_Function( Track_t *pTrack )
{
      // the pEvent is pointing at our event block
     NoteOff_EventBlock_t *pNoteOffEB = (NoteOff_EventBlock_t *)pEvent;

     // walk through this track's voices and turn off
     // any that have pVoice->nKey == pNoteOffEB->nKey
}

     // Example Note On Event Block
     typedef struct {
     char    nEventType;
     char    nKey;
     char    nVelocity;
}NoteOn_EventBlock_t;

void NoteOn_Function( Track_t *pTrack )
{
     // the pEvent is pointing at our event block
     NoteOn_EventBlock_t *pNoteOnEB = (NoteOn_EventBlock_t *)pEvent;

     // try to get a voice from the free list or
     // try to get a voice from the active list if possible
     // if we have a voice, turn it on with the pNoteOnEB->nKey
     // and pNoteOnEB->nVelocity and other state information
}

enum enumEventType
{
     EVENT_TYPE_NOTEOFF,
     EVENT_TYPE_NOTEON,
     .
     .
     .
     EVENT_TYPE_COUNT
};

typedef void (*EventFuncPtr_t)(Track_t *);

typedef struct {
     EventFuncPtr_t pFunc; // pointer to command function
     int nLength; // byte length of command
}EventTypes_t;

static EventTypes_t aET[EVENT_TYPE_COUNT] = {
     { NoteOff_Function, sizeof(NoteOff_EventBlock_t) },
     { NoteOn_Function, sizeof(NoteOn_EventBlock_t) },
     .
     .
     .
};

Timing
Different computer systems will have different ways of providing timing callbacks or threads that wake up at specific intervals. I will simply assume that we can have a function called at a specific interval that I will call the "audio frame callback". We can think of the time between the callbacks as the audio frame. During each callback we need to update our notion of how much time has passed and we need to send out all of the commands that have "timed out".

In the sequence data structure listed in Code Example 1 there is the TimeStep quantity that should be set based on the tempo, the delta-time parts per quarter note, and callback timing. We add the TimeStep to the TimeElapsed on each audio frame to keep track of the time. Since these time parameters are in the sequence structure, we can only change the tempo for the whole sequence. If we wanted to change tempo for each track individually we could put these parameters and the tempo setting in the track structure.

• Game design will influence music design.
• Music design will influence music programming.
• Game design will influence game programming.
• Game programming will influence music programming.

http://www.gamasutra.com/resource_guide/20010515/patterson_01.htm

Copyright © 2004 CMP Media Inc. All rights reserved.