"One measurement is worth fifty expert opinions" - Howard Sutherland
Experts
and expertise are hard to come by, and experts like us are often
unreliable. There are often situations when there is no available
expertise, or situations that no one has experienced before. Even
worse, there are situations where experts disagree. In these
situations, people generally turn to measurement.
Most
trades consider measurement more important than expert opinion, or they
largely abandoned complete reliance on expertise several hundred years
ago. A casual look around you will reveal the extent that measurement
is used in society. Food is sold in precise quantities; clothing comes
in a series of formalized sizes. Politics and market research are
dominated by polls and the analysis of measured data. Architects,
engineers and nearly every other kind of designer use measurement and
agreed units as one of the foundations of their work.
Measurement
in any form must have been originally developed to determine absolute
values in situations where there was no independent expertise, or where
the experts disagreed. For instance, a measurement of area, using an
agreed system of units, is an ideal way to resolve disputes over land.
A formal measurement of volume or weight is a perfect way of ensuring
fair trade. The use of measurements of qualities like force and mass
has been standard in engineering projects for centuries. In short, the
use of measurement, agreed systems of units and number are some the
cornerstones of human civilization. The very first recorded human
writing - Mesopotamian Cuneiform - was initially used to track
measurements of traded goods like wheat and oil.
It’s
amazing given the importance of measurement in so many trades and other
aspects of our life that game design is in such a prehistoric,
pre-measurement state. For whatever reason, game designers - working in
a technological and logical field, hardly ever use measurement in their
work. In fact, I’d wager quite a few people reading this article cannot
even comprehend what we could measure and how it would help us. I’ve
tried to use measurement as much as I can in my work, and have often
found it invaluable.
In
this article I’ll deal with the some of overall concepts behind these
kinds of analytical processes, and I’ll detail some specific examples
that will hopefully help your work right away. Finally I’ll try to
examine how this kind of technique and philosophy could potentially
mature to help us in the future.
Human Imperfections
As
mentioned above, the key reason humans have developed measurement is
because people tend disagree on interpretations of objective reality.
Our senses are imperfect, having evolved for a simple hunter-gatherer’s
existence, and our brains are similarly ill-prepared for minute,
consistent and logical examination of things.
A
simple example; in my hobby as an amateur racing bicycle builder, I
often find myself trying to roughly determine the difference in weight
between two components, a few hundred grams apart. Anyone who has tried
to do this will agree that it’s almost impossible to judge without
resorting to mechanical means, however crude. The mind and body is
simply not equipped to make such a precise judgement, and as a result
we often make glaring mistakes. This imperfection is what led to the
development of a system of units of weight, and measuring equipment for
said units – scales. Scales use two technologies - Firstly an agreed
system of weights and measures, and secondly machinery to precisely and
consistently determine the measurement of those units.
Game
designers often find themselves in situations where they are as
clueless as I am when comparing the weight of bicycle parts. Examples
of this come up all the time when designers discuss games. An example:
GTA
games can be pretty frustrating, right? Missions are often poorly
balanced, requiring a trial-and-error playing technique, which is
rendered even more frustrating by the fact that upon failing a mission,
players are required to physically move the character across the map to
a specific location to re-trigger the mission (I’m so very glad that
the designers of Mercenaries fixed this with a "retry" UI option).
A couple of years ago I had a discussion with a designer about this aspect of GTA,
and he made a statement to the effect: "when you fail a mission it
takes about ten minutes to re-start it". Now I’m aware that people
often exaggerate in these types of statements, so I asked him to try to
guess how long it actually takes to get from the hospital or police
station in Vice City to an average mission location (they are evenly
spaced). He responded – "ok, exaggerations aside, maybe five minutes
average".
One
of my pet fascinations is the way people find it impossible to properly
determine the passage of time, so I decided to go home and measure
this value, rather than rely on the expert opinion of this professional
game designer. My quick investigations at home came up with an
approximate average value of 45 seconds to a minute for the
task that was guessed at five minutes. That’s a serious inaccuracy, and
one that I think anyone (me included) would be inclined to make.
So
it’s clear that our brains are great at doing some things (coming up
with creative ideas and ways to solve problems, team work, verbal
communication), and not so great at others (measuring the passage of
time in free-roaming gangster games, and the differences in weight
between expensive bicycle parts). Nearly every other field resorts to
measurement in situations where our expert judgement breaks down, or is
found lacking. As game designers, we are professionals, and because
precision, analysis, and logic are an important part of our work, why not measure?
|
|
 |
 |
 |
Grand Theft Auto: Vice City
|
Applications
In the face of any uncertainty as to what is actually possible to measure in a videogame, let’s look at some specific examples.
1. Time
Time
and timing is vital to most games. Action games specifically often
succeed or fail based on the precise timings of lower level player
actions like melee attacks, weapon reloading, and character
acceleration. I think it’s really important to measure your game and
the games of competitors to try to find the “sweet spots” in these
timings.
Some of you might have read an article I wrote for Develop
a couple of years ago in which I claimed to have found the perfect
timing for character jumps – at 0.7 seconds between hitting the button
and the character landing the jump. I determined this figure by taking
several games that were well received from a design point of view,
recording the jump times with a stopwatch, and finding the average
figure. As far as I know this discovery is still supported by the data,
as now I often informally time jumps in new games.
There
is huge room for research on this topic. I’m convinced that there are
“sweet spot” timings for almost any action in a game. I hope that
someday someone will accurately and comprehensively compile this kind
of data. They would be doing a huge service to game designers all over
the world. We’d no longer have to exercise our expert (or should that
be inexpert) opinions because the hard data would be there to override
our expertise. We’d have a manual, and the beginnings of a method.
2. Distance
Distance
is a critical component, particularly in the 3D action/exploration
games that designers love to make. The size of levels, the spacing
between events, and the range of ranged and melee weapons are all
critical to creating an entertaining experience.
The
problem is that it is often impossible to do precise measurements of
distance in competitor’s games. Obviously in your own project you can
use the tape measure tool in your 3D package/level editor, or refer to
data in a config file, but this isn’t possible in a retail game.
One
thing you can do in these circumstances is use a stopwatch to measure
distance by timing the duration of movements. For instance, it is
possible to determine distances fairly precisely in first person
shooter maps by measuring the time taken to move across a space, given
the consistent and smooth movement of the in-game avatars. A few years
ago I used this method as an amateur map maker for a Half-Life mod, Frontline Force.
Looking only at the most popular maps, I measured the distances between
spawn points and capture points. Spotting a pattern, I applied this
"sweet spot" data to my own unique map with immediately positive
results. The map "felt" good first time round.
The use of measurement gave me, an inexperienced Frontline Force
mapper, an immediate advantage. I was able to concentrate on the
creative side of map building and not worry about basic low-level
characteristics like the sizes and the distances used in its
composition.
3. Density
This
is an area that really deserves a whole separate article. It’s really
hard to measure density (meaning the density of in-game events) without
developing a specific unique method.
You
first of all need to define a new unit, because this is an area not
covered by existing research. An analogy would be the development in of
new terms and units like "Shot", "Scene", and "Mise en Scene" (the
latter two borrowed and adapted from theater) which enabled academics
to analyze film.
I recently outlined a technique and a proposed unit in an article for Develop
Magazine. Put simply, my system involves recording gameplay using a
video capture card, and microscopically analyzing that data in a video
editing package like Adobe Premiere, dividing and subdividing the data
with markers until you reach a level of subdivision which I call
"Primary Elements." These elements are basically the absolute simplest,
fastest, most minute player actions you can measure – actions like
"select unit," "steer 30 degrees left” or "reload." Once you have these
elements marked and recorded, the units defined, you can do qualitative
and quantitative analysis of them, and of the recorded gameplay as a
whole.
The
advantage of this system is that measuring things "per Primary Element"
as opposed to "per minute" or "per level" is universal across genres.
You can try to find patterns emerging in games that on the surface bear
no relation to each other. For instance, turn-based strategy games like
Civilization and Dance Dance Revolution have very
little in common using other measurement systems, but they both contain
Primary Elements, so using this system they can be measured in almost
the same way.
We
may discover in years to come that there is a "sweet spot" of Primary
Elements per "level/challenge/encounter," which we can directly apply
to level/content design in almost genre. We might find that people get
bored of playing games at a "sweet spot" of Primary Elements,
irrespective of the actual playing time. I’ve long thought this could
be one of the reasons why people play slower-paced games like Civilization
for hours and hours longer than action games – we like to see minimum
number of "actions" and "reactions" take place in the game before we
get bored or mentally tired. It just so happens that this takes five
hours in Civ and twenty minutes in FIFA Street.
4. Area
This
is one type of measurement that is actually quite fun. Nearly
everything we make as developers is eventually rationalized to 2D
screen space, so measuring and examining the way your game uses area on
the screen is really useful. The method I’ve used to do this involves
using an artist’s chinagraph pencil to physically draw on the screen -
marking the position and movement areas of in-game objects.
For
example, I’ve specifically used this technique to mark and measure the
extent of character movement in a third person game (meaning the total
area of screen space that the character uses as it moves). First I
marked the movement extent of the characters in various retail games,
like Mario 64, GTA3, and Maximo. Then, with
the marks still on the screen I looked at our in-development game. This
method made it clear how our character movement and use of screen space
differed from the conventions and standards used in other games, and it
also helped us spot mistakes and odd glitches in the way the character
and camera moved.
This
is a really useful example of a method that insures us against the
types of mistakes we make if we rely on solely expertise and personal
opinion. Sometimes it’s hard for a designer to judge even the simplest
of aspects of a game (like how much the character in the game moves on
the screen). With a few simple marks and measurements you have
incontrovertible data that has removed the need for opinion and
guesswork.
Hopefully
you’ve seen that I’ve scratching the surface of what’s possible to
measure in a game, and the potential for applying these methods and
others to your own work.
The Road Ahead
It’s
obvious why measurement is used as a tool by humans, and why it has
been so fundamental in the progress of mankind. People are imperfect,
but fortunately our cultural developments have included techniques and
machines to overcome these imperfections as much as we can. Human
society and science are founded on concepts and ideas made possible
partly by measurement and agreed systems of units. It is hard to
imagine how hard life must have been for mankind before they learned to
measure things. Except it couldn’t have been that hard, because that’s
how most game designers work today!
Of
course it’s easy to adhere to old habits and it’s easy to go along with
the flow and follow the inertia of old working methods. Sometimes it’s
also nice to feel like an expert and use gut judgement to make
decisions. I do it all the time – it’s often the most efficient
approach given the highly pressured and panic-stricken circumstances we
find ourselves working in. It is however an imperfect and un-optimized
way of working.
I feel there is a real need and a real opportunity
for game designers to begin to use mature analytical processes that
compliment and help us in our opinion-led, gut-reaction working
methods. Using measurement and hard data, we can do several important
things.
Firstly,
we can examine phenomena that are impossible or very difficult to judge
with our senses alone (like the timings of mission repetition in GTA,
for example), overcoming the inaccuracies and assumptions that we
habitually get wrong. Secondly, we can reinforce and qualify our correct
expert opinions, by supporting those all-important gut feelings with
proper data-gathering investigations (like my jump timing example).
Finally, we can also use this data to help persuade others that we need
to get certain things done. If you can show someone that you not only
think something is true, but that you can ultimately prove with data
that this is the case, then you will find yourself in a much stronger
position when it comes to arguing for a related task to be carried out.
In
an ideal future, measurement and analytical data-gathering techniques
will give us a proper academic foundation (a formal, data-driven
academia, not the horrible cultural-studies influenced rubbish
currently pervading videogame studies). Designers will be able share
data in repositories that are accessible to all, just like any other
logical and analytical field. Field manuals, dictionaries,
encyclopaedias and papers will be published on the subject. People will
be employed full-time as academic game analysts and surveyors. Our
craft and trade will move from a prehistoric, "feel" focused method,
governed by the gut-feelings of disagreeing experts to a mature, modern
state, where opinion and expertise are supported or disproved by
something close to absolute truth.
References
- “Mind your language - Unlocking the secret formula of game design” Develop magazine, August 2002
- “Elementary game design” Develop magazine, October 2004
_____________________________________________________