|
 Features
Beyond
Psychological Theory: Getting Data that Improves Games
Common
game design feedback systems and their limitations
There are
many feedback systems that designers use (or, in some cases, been subjected
to). Most designers, like authors, recognize that they need feedback on
their work in order to improve it-- few authors have reason to believe
that their work is of publishable quality without some revision based
on feedback. I'm going to list the feedback systems of which I am aware,
and discuss how good of a feedback delivery system it is. There are two
main categories of feedback loops: feedback from professionals in the
games industry, and from non-professionals (i.e., gamers). While these
sources obviously affect each other, it is easier to talk about them separately
Feedback
from Professionals in the games industry
There are two main sources of this kind of feedback:
- Feedback
from those on the development team. This is the primary source of feedback
for the designer--people working on the game say stuff like "that
character sucks" or "That weapon is way too powerful."
This system is useful because it ably suits criteria two through four
(the feedback is very timely, granular enough, and easy to get), but
still leaves the designer with a question mark on criteria one--how
many gamers will agree that that weapon is way too powerful
- Feedback
from gaming industry experts. Game design consultants ("gurus"),
management at publishers, game journalists, etc. can also provide useful
feedback. While their feedback can often meet criteria three (sufficiently
granular), criteria two (timely) is sometimes a problem--long periods
can go between feedback, and recommendations can come after you can
use it. And the designer is still left with questions about criteria
one (accurately represents gamers), although some could argue that they
may be more accurately representing gamers because they have greater
exposure to more games in development.
So while
feedback from professionals is the current bread and butter for most teams
and definitely nails criteria two, three and four, it operates a great
deal on faith and hope on criterion one--that the feedback from industry
professionals accurately maps onto gamers' opinions. The reason this assumption
is questionable is perhaps best illuminated by a simple thought experiment--how
many games do you think a typical gamer tries or sees in a year? How many
do you think a gaming industry professional tries or sees? They are probably
different by a factor of ten or more. Gaming industry professionals are
in the top 1 percent in knowledge about games, and their tastes may simply
be way more developed (and esoteric) than typical gamers' tastes. While
some professionals in the industry are probably amazingly good at predicting
what gamers will like, which ones are they? How many think they are great
at it, when others disagree?
So while
feedback from industry professionals is necessary when designing the game,
they may not be the best at evaluating whether gamers will like something.
In the end, they can only speak for themselves.
Feedback
from Non-professionals
Game teams are not unaware of the problem of their judgment not always
mapping onto what most gamers really want. Because of this, they often
try to get feedback from those who are more likely to give them more accurate
feedback, and the obvious people to talk to are the gamers themselves.
Some common ways that this is done are listed below, along with some analysis
of how good a feedback system it is according to the four criteria.
- News
group postings/Beta testing/fan mail. This is reading the message boards
to see what people say about the game. The main problem with this as
a feedback system is with criteria two (timely). The game has to be
able to be fairly far along (at least beta, if not shipped) in order
to get the games to people; typically, that feedback arrives too late
to make any but the most cosmetic of changes. Also, the feedback often
runs into problems of not being sufficiently granular to take action
on. ("The character sucks!") But at least this kind of feedback
is relatively cheap in both time and money.
- Acquaintance
testing. This is where you try to get people (typically relatives, neighbor's
kids, etc.) from outside the industry to play your game and give you
feedback. This feedback is often sufficiently granular and may be relatively
accurate, but it is often not that timely due to scheduling problems,
and can be costly in time.
- Focus
groups/Focus testing. This kind of feedback system is typically done
by the publisher, and involves talking to small groups (usually four
- eight gamers) in a room about the game. They may get to see or play
demos of the game, but not always. One typical problem with focus groups
is that often tend to happen very late in the process when feedback
is hard to action on (not timely) and not sufficiently granular. The
costs for focus groups can also be quite high.
This approach
has potential to be useful, in that it involves listening to gamers who
aren't in the industry. However, there are many pitfalls to this--It is
often dubious as to how accurately the feedback represents gamers due
to the situations themselves (only certain kind of people post messages,
people feel pressured to say positive things, the people running the test
often lack sufficient training in how to avoid biasing the participants,
etc.), and the relatively small number of people. How to minimize these
concerns and create a feedback system that works on all four criteria
is discussed in the next section.
______________________________________________________
 Designing
a better feedback system
|
|
|
|
|
