Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
September 30, 2014
arrowPress Releases
September 30, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


Design success means knowing what to do with feedback
Design success means knowing what to do with feedback Exclusive
February 6, 2012 | By Soren Johnson

February 6, 2012 | By Soren Johnson
Comments
    8 comments
More: Console/PC, Exclusive, Design



[Game designers "need to place faith in the design process itself, not in their inevitably doomed first draft," says former Firaxis lead designer Soren Johnson. (Originally printed in Game Developer magazine.]

"You have to design for success, plan for failure, and then know how to rebound from that failure. Our 1.0 version of AI War was successful but not wildly so. Little irritations added up and annoyed people enough that they stopped playing. When the public gets the game, they find problems that you never did, and you must devote time to fixing them. Once you do, the fans are happy, and the game becomes more successful than it would have been otherwise. You have to eat a lot of humble pie, but after awhile you get really used to that, and the stuff that used to give me a hard time emotionally when I was first starting out is just par for the course now. I don't even notice. It's just feedback, and whether it's viable or not determines if it goes in the 'yes' bucket or the 'no' bucket or the 'maybe' bucket."

- Chris Park, designer of AI War: Fleet Command, from the Three Moves Ahead podcast, Episode 37


To be a game designer is to be wrong. Ideas do not work out as planned. Certain mechanics prove tedious instead of fun. Players spend their time focusing on the "wrong" parts of the game. The original vision starts to slowly slip away as it come into contact with the gamer.

Furthermore, designers are often bombarded with suggestions - from the rest of the team, from vocal fans, from well-meaning friends, and from dreaded executive fly-bys. Faced with such a deluge, the natural instinct is often to defend the design. These suggestions are simply trials and tribulations to be overcome so that you, the designer, can continue making the game "your" way.

The problem is that processing feedback is a fundamental part of the game design process, as important as the original vision itself. Games are not static objects that can be observed or judged in a vacuum. Instead, they live in the minds of our players, which means that each one might experience the game in a different way.

Thus, designers need to be great listeners more than great persuaders. If a designer ever finds herself needing to explain why a player should be having fun when he is not, something has gone seriously wrong. Instead, designers must listen to players with a great sense of humility, with an understanding that this feedback is the only way to remove the fog which separates the game in the designer's head from the one that is actually playable.

Ultimately, games must speak for themselves, so designers need to learn not to rely on the crutch of their own enthusiasm and communications skills to sell their ideas. Dropping one's ego to learn from the criticism can be an emotional challenge for designers who identify too closely with their original vision. Instead, they need to place faith in the design process itself, not in their inevitably doomed first draft.

Listen to the Team?

Thus, gathering and assessing feedback is one of the crucial skills for a modern game designer. The first, and sometimes only, source of feedback is the team itself. Any committed development team should be ready and eager to play its own game and provide the designer crucial early feedback on what works and what doesn't. However, feedback from the team carries significant limitations.

To begin, the team lives with the game for months, probably even years, far beyond an average player's time with the game. As veterans of development process, they can play on auto-pilot, making themselves blind to unintuitive mechanics or confusing UI. Developers quickly lose the ability to see the game objectively, often believing that the game is either in better or in worse shape than it really is.

Further, team members are often hired for their special skills - 3D animation, sound design, network optimization - and not because of their passion for the product itself. They might forget some of the simple joys that new players will still experience when starting the game. More dangerously, they might burnout on the game altogether and begin to resent the intense demands of your most vocal community members.

Trust the Community?

The community, of course, can be an overflowing cauldron of ideas and suggestions. When developing Civilization III, our most active fansite presented us with "The List" - an exhausting, 200,000 word tome detailing their expectations for the upcoming game. Wading into the forums can be an overwhelming experience for most designers, requiring a thick skin as posters rip apart their development choices.

However, no one understands a game better than players who are dedicated enough to join a community and make the game a part of their social life. These gamers might play for hundreds of hours, gaining knowledge of mechanics and systems that elude even the designers themselves.

The challenge with forums is that what players say and what they actually do are often two different things. In a talk at GDC 2011, Ben Cousins described just such a situation with the free-to-play online shooter Battlefield Heroes. The game had not been generating enough revenue, so the team reworked the monetization system - making it harder for non-paying players to "rent" premium items and then beginning to sell these items directly for cash.

The change caused an uproar in the forums; within a week, a 4,000-post thread developed decrying the changes, with many veterans pledging to quit the game. Exacerbating the debate was the fact that Cousins had publicly declared years earlier that "we have no plans to sell weapons." The press picked up the controversy, leading to articles such as "Battlefield Heroes is Practically Ruined" on Kotaku.

The metrics, however, told a very different story; revenue tripled with no discernible decrease in active users. It is hard to tell if the posters who pledged to quit were actually lying or not, but they were clearly not representative of the average player. Cousins dug deeper and found that only 20 percent of players had ever visited the forums and that only 2 percent had ever actually posted a message.

Furthermore, compared with the silent majority, community members had a much higher conversion rate (27 percent vs. 2 percent) and ARPPU ($110 vs. $32). Thus, the thoughts expressed on the forums were an inaccurate and misleading representation of the player base's actual feelings. The posters were perhaps making threats to change the game as they wanted (to save themselves money) instead of revealing their actual beliefs.

The Heroes experience highlights the importance of metrics as a secondary source of feedback. Watching what players actually do can be as important as listening to what they have to say. Still, metrics have their limitations as no set of numbers is going to help the designer understand why people have stopped playing the game.

While measuring how often unit X is built instead of unit Y provides a valuable tool for balancing an RTS, it's not necessarily clear that making the two units equally viable will actually make the game more fun. Metrics are great at answering specific objective questions that require real data - what difficulty level is most commonly picked first? - but to learn whether a game is actually fun, the designer's only option is to find out what players are feeling by listening closely to what they are saying.

Find Your Voices

Thus, designers are left with the conundrum that their best source of feedback - the vocal fan community - is not only an unreliable source of information, but one that might be actively trying to mislead the developers. Perhaps most frustrating is the possibility that the more forum posters are aware that the team is listening, the more likely they are to lie to the designers to get what they want. MMO developers are familiar with the player type who will always argue that his or her character's class is woefully underpowered, all objective evidence to the contrary.

This problem can be handled with a more proactive approach to gathering fan feedback. Not all fans are the same - a precious few are able to see the forest and provide accurate feedback that speaks to the health of the game's overall experience. While developing Civilization IV, we cultivated just such a group of enlightened fans to provide feedback we could trust.

These players had a history of being reliable sources of information during the post-release development of Civ III, and we provided them with a special, private forum for direct communication with the team. This group became our primary source of feedback both before and after release, providing us with much greater certainty about which ideas were working and which ones were not; Civ IV would have been significantly different - and certainly worse - without their input.

These groups must be managed carefully, however, to prevent the members from developing a sense of entitlement or superiority over other players. For this reason, the group's existence should be, if possible, a closely guarded secret. Further, the developers must try their best to find a representative group of players, perhaps looking outside the forums for new members.

Listen Early, Listen Often

Another accurate way to gather feedback is with "Kleenex" testing, so named because players get access to the pre-release game once and are then thrown away. The valuable lessons here come from players' initial reactions to the game, before they become accustomed to UI holes or gameplay quirks. Valve famously runs these tests regularly by gathering up random players from local game stores.

However, depth testing is also important, which can only be achieved by giving players continual access to the game before release, to explore and experiment with the game's systems and mechanics. Big publishers often have trouble giving fans early access to their games, for fear of leaking cracked versions to pirate sites or spreading confidential information to rival publishers.

Indies developers actually have a big advantage here because their greatest danger is not security, but obscurity. Thus, many recent indies (Spelunky, Desktop Dungeons, The Wager) have released early versions of their games, generating both marketing buzz and valuable feedback.

Some indie games, such as Frozen Synapse, Minecraft, and Spy Party, have even generated revenue by selling access to these alphas. This option gives teams the chance to bootstrap their way along while also learning how the game performs in the wild, a great option to help fight the long odds that most indies face.


Related Jobs

Raven Software / Activision
Raven Software / Activision — Madison, Wisconsin, United States
[09.30.14]

Lead Engineer - Raven
Trendy Entertainment
Trendy Entertainment — Gainesville, Florida, United States
[09.29.14]

Tools Programmer
Respawn Entertainment
Respawn Entertainment — San Fernando Valley, California, United States
[09.29.14]

Senior Systems Designer
College for Creative Studies
College for Creative Studies — Detroit, Michigan, United States
[09.29.14]

Game Design Faculty










Comments


Tynan Sylvester
profile image
"To be a game designer is to be wrong." -Soren Johnson



Compares well with



"The first draft of anything is shit." -Ernest Hemingway



---



You've covered it well here, Soren. Let me just add a thought.



The more testing I do the more I feel like it's about watching more than listening. As soon as people open their mouths, they start editing their memories, rationalizing, forgetting and inventing details. Every bias in human judgment and communication appears.



In play, we're honest. That's why I love watching. Observing playtest after playtest, you start to develop a mental map of the possibility space that emerges from the game. To me that's the primary usefulness of playtesting. Metrics come in second; tester feedback a distant third.



This applies as much in depth testing as anywhere else. Want to understand the design of StarCraft II? I suspect it's more helpful to watch the GSL than to ask its players what they think (and not just because most of them only speak Korean).

Michael Parker
profile image
It really depends on what sort of level you are testing. There is a difference between "do players like the game as a whole?" and "Is enemy X too hard to kill?"



I think it's somewhere in between watching and listening. Typically I'll watch and try to work out why the player is behaving in a certain way, or having problems with a certain part of the game. Usually if you outright ask "why are you finding this enemy hard?", you aren't going to get much useful feedback about specifics - the HUD, or clarity of sound, or whether the lighting doesnt emphasise his animations enough, etc. However, if you point out the observed behaviour, and potential causes, i've found players can sometimes offer quite insightful comments once they understand the nuances you're looking at.



It's that tricky balance where you have to draw attention to multiple possibilities in a netural way without biasing the answer (very hard to do!)

Joe McGinn
profile image
You have to be a good listener, and I agree about the first draft will also die ... BUT you need to know HOW to listen. Which of those legions of suggestions should you say yes to? When is it time to dig in your heels, and stick to your guns?



A designer needs to understand why and where fun comes from, so that they can correctly judge if the current path still might be successful. Fun does not happen overnight yet the design may be sound. I've had features where the teams is all, "This is not fun, this is not fun, this is not fun!" Then one day we finish - or even just tweak - the audio-visual feedback of the feature to the player, and suddenly, "This is the best feature ever!" Few non-designers appreciate this and why should they? Design is not their job. But without this skill you will never make a great game, you will be lost in a sea of designed-by-committee suggestions than you can't say no to.

James Orevich
profile image
All of the above have great points (article included).



Good listening skills is a recurring theme in articles, blogs and industry.

However, as Tynan mentioned, memories can be faulty, subjective or even entirely fictional (psychologically proven).

This article does well to go beyond the listening phase and I appreciate the insight.

E McNeill
profile image
The best piece of playtesting advice I ever got (though I forgot from who) was to ask only three questions:

1) What did you like?

2) What didn't you like?

3) What confused you?



The answers to those questions are important, and everything else you can ignore. The idea here is that playtesters are not game designers. They're qualified to speak about their subjective experiences, but not at all about how to fix the game.



I think the reason this advice appeals so much to me is that it emphasizes humility while also bolstering the designer's sense of control. I don't feel buffeted by the fickle-yet-authoritative feelings of my audience. Instead I'm just collecting information and acting on it as I see fit. I can maintain my own vision and design goals, rather than feeling like I'm obligated to sacrifice them.

Peter Stock
profile image
I think I read those three questions in Jesse Schell's book. I agree that the players might be able to give a good indication of what they would like different, but not necessarily how best to achieve it.

Abel Bascunana Pons
profile image
Very interesting article. It's clear that a game designer must be a very sympathetic guy.



I do agree with Joe McGinn though, that's a game designer must display a sound judgement to determine if the game or its features go on the right track. He must go through an abstraction process after collecting all possible data to take a game design decision... and at the same time not forget the whole picture.

Rajveer Kothari
profile image
Great article! Am sharing this with my team back at work!



As a practice that I have followed on the games that I worked on, I quietly followed my inner feeling of how convincing, the game's turning out to be. While playing a feature that the Development team would have just worked on, I keep looking out for moments to make me go like "Yeaahh!!"



If THAT feeling doesn't come up, I usually walk up to the team and communicate what I find missing. As the feedback's implemented and I find a certain level of conviction there in it, I turn into a little kid, sharing the feature with as many people I can bump into on my floor!



A great read, thanks Soren!


none
 
Comment: