Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
April 23, 2014
arrowPress Releases
April 23, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM TechWeb sites:


When your player's opinions are crap
When your player's opinions are crap
August 27, 2012 | By Frank Cifaldi

August 27, 2012 | By Frank Cifaldi
Comments
    21 comments
More: Console/PC, Indie, Design



"I'm a big believer in the fact that opinions are really important, but people's justifications for why they hold opinions are basically crap."
- Game designer Simon Strange (from Atari's series of Godzilla fighting games and the upcoming spiritual successor Kaiju Combat) explains in an upcoming Gamasutra interview that you should always listen to your players...to a point.

Your players are good at knowing when they don't like something. That part comes naturally. But being able to explain exactly why they don't like it is a really unreliable data point.

"When someone says 'I don't like this,' that's really important and you have to believe them. But when someone says 'I don't like this because-,' you can often kind of ignore their 'because,' because they often don't have the data to understand what's going on," he says.

This bit of wisdom seems especially apt for the smaller developers among you, who might rely on verbal feedback more than on hard data. Always, always listen to your playtesters when they don't like something, but dig a little deeper than their own opinions to figure out why.


Related Jobs

Gameloft
Gameloft — New Orleans, Louisiana, United States
[04.23.14]

R&D Game Designer
SOAR Inc.
SOAR Inc. — Mountain View, California, United States
[04.23.14]

Game Designer/Narrative Writer
Giant Sparrow
Giant Sparrow — Santa Monica, California, United States
[04.22.14]

Game Designer
Blizzard Entertainment
Blizzard Entertainment — Irvine, California, United States
[04.22.14]

Starcraft II - FX Artist










Comments


Maria Jayne
profile image
Data can tell you many useful things, but it really can't track perception (unless you count players not doing something). Player feedback should never be dismissed out of hand, despite it's often exaggerated, empassioned and sometimes false rants, it can tell you things about your game that numbers cannot. The real challenge, is finding the constructive and well written feedback among the sea of vitriol.

Jesse Tucker
profile image
What a misleading title. Player's opinions aren't crap, although their proposed solutions based on those opinions can be. Still, I wouldn't go around calling people's ideas "crap."

Jesse Tucker
profile image
To go a little further, "Listen to the problem, not the solution." would be a much better mantra than "Players' opinions are crap."

Dustin Chertoff
profile image
By that token, the opinions of arrogant developers are also "crap." You know, unless they more clearly explain how to go about distilling player feedback into useful objective/subjective metrics.

Carlo Delallana
profile image
We are all susceptible to a variety of cognitive biases that cloud our judgement. The list is staggering (http://en.wikipedia.org/wiki/List_of_cognitive_biases). Imagine having to parse meaningful insights through this minefield of biases.

Would biometric data be the most unbiased source of user feedback? Our autonomic systems would appear to be immune from these cognitive biases.

Bart Stewart
profile image
That's a tempting thought as it sounds like it would get past the layers of "shoulds" that cloud the thinking of most people.

But while I could see it being good for collecting data on excitement levels (what I like to call the Experientialist/Kinesthetic mode of play), and maybe OK for security-oriented Achiever play, would that monitoring tech work for more abstract motivations like exploration/Simulationist play or story/people-focused Narrativist/Socializer play?

I don't know, but i wonder if the particular kinds of pleasure that come from simulationist and relationship play might happen at a different cognitive level than emotions that get expressed at an autonomic level.

It might be interesting to try to calibrate player comments by the main style of the game they're commenting on.

Kevin Alexander
profile image
It is a rather unpopular belief right now, but i kinda agree...

Theirs a list of low hanging fruit the average user can easily articulate about, and this is typically why we hear about them in reviews most prominently, despite the fact that they may not always be the most relevant "why" people don't like something regards to a particular loop, feature, level, or whatever. Nevertheless, it sometimes is the only thing anybody can positively put there finger on.

The safest bets to play are:

Camera's
Controls
UI (or lack there of)

If someone doesn't like something, and they cannot quite identify the problems (but are mildly adept at gaming), they'll start with one of those 3, and we as developers can occasionally get caught up in over iterating these systems that undoubtedly cannot ever be "perfect" while ignoring the fact that the problems just might go deeper.

No one can doubt the significance of focus testing, yet, I wonder sometimes if theirs just as many bad ways to use that technique as good.

Jonathan Jennings
profile image
yeah I know all to many times I have seen a reviewer or blogger comment about how something they didn't like be it a animation or the mechanics of movement and blame it on " the engine" i don't expect the common user to understand even a portion of what the game engine does but I know several times i have seen a reviewer criticize an engine when the problem was more along the lines of a badly implemented mechanic.

Simon Ludgate
profile image
I both agree and disagree with the sentiment expressed here.

I agree that it is DIFFICULT to explain the justification behind an opinion. The opinion itself is very easy to form, but the justification requires considerable cognitive work, therefore it seems that most people will forgo that work, supplying a simple (and possibly incorrect) justification.

However, I disagree that people are UNABLE to correctly extract, analyze, and verbalize the justification for their opinion. The process of constructing justifications forms the basis for the practice of philosophy, for example. As someone who devoted the many years necessary to earn a degree in philosophy, I object to the suggestion that someone should outright reject my justifications simply because those of many others aren't as well formed.

I think this leads to an important insight: if you want useful feedback on your game, it may behoove you to spend the extra resources to consult people with the capability to providing an insightful and in-depth analysis, rather than relying solely on the casual opinions of a mass of inexpensive users.

Thomas Happ
profile image
I was kind of hoping for a more in-depth analysis when I read the title. But I agree with the overall gist.

Joe McGinn
profile image
Yeah these 100-word non-articles., wtf Gamasutra? A little more meat on the bone please!

Michael Wenk
profile image
Whether or not the basis of an opinion is reliable or not is not relevant. At the end, what matters is the player doesn't like it, and will not spend money.

And I disagree that the basis is useless. Even if they're completely confused why they feel something, you can glean some useful info from it.

"When someone says 'I don't like this,' that's really important and you have to believe them. But when someone says 'I don't like this because-,' you can often kind of ignore their 'because,' because they often don't have the data to understand what's going on,"

That statement is just Simon Strange's ideology coming out. The big picture is often, hell, usually false. The issue here is the developer is using that ideology to cover up the fact that the game sucks.

Simon Strange
profile image
I'm a bit miffed that Frank is so obviously trying to turn my statement into a divisive hook.
The title very clearly implies that I think the opinions of testers are worthless - but the full quote clearly expresses that I think player feedback is vital - it is only their justifications for their opinions which are unreliable.

I'm also quite put out by the follow up in the teaser "game designer Simon Strange, who knows better than his playtesters why they don't like something." I don't mistrust their justifications because I have a better idea. Saying that I do just makes me out to be an ass.

I only hope that the full thing comes across as clearly as this article from theverge - which makes the same points: http://www.theverge.com/gaming/2012/3/14/2861998/gearbox-borderla
nds-testing

Ah well - here's hoping that all press is, indeed, good press.

sean lindskog
profile image
Ouch.

Yeah, this title hardly seems fair. The actual quote says "opinions are really important", which is obviously very different than "player's opinions are crap".

Svein-Gunnar Johansen
profile image
The title does seem a bit off with regards to what you are actually trying to say, and unfortunately not in your favor. It did however make me curious enough to go and read the full article, so... Maybe all press is good press :)

For the record, I agree that user testing is invaluable for finding the problems. Finding the solutions must however be done by the developer.

Michael Rooney
profile image
@Sean: That's not the whole quote. It's still totally out of context, but the quote is exactly what's bolded at the top of the page. I think this is the whole, in context, quote though:

"I'm a big believer in the fact that opinions are really important, but people's justifications for why they hold opinions are basically crap. When someone says 'I don't like this,' that's really important and you have to believe them. But when someone says 'I don't like this because-,' you can often kind of ignore their 'because,' because they often don't have the data to understand what's going on."

I think that's still controversial. Players almost always know why they don't like things and are usually very good at explaining why if you give them a minute to think about. The less controversial opinion is that their solutions are generally crap.

Pietro Guardini
profile image
Simon, as a Games User Researcher I've got your point right from the beginning, despite the misleading title - and most likely every other GUR guy reading this. Gamers' opinions and comments are the most direct way into their understanding of a game, and if they got it wrong, then something may be wrong with the game tutorial, interface, etc.

I've just playtested a rally racing title (what could go wrong with a racing game?), and still their comments even if naive and wrong are useful to improve the game experience. Example: you have pacenotes, "left 5, right 2" and they thought the number corresponds to the gear you have to use in that particular corner - which is wrong, since it's related to the degree and severity of bends. My feedback for designers is to explain somewhere (loading screens, tutorial, etc) how the pacenotes work.

Wrong opinions are valuable since you get to know what's wrong and how to fix it.

Randy Lambert
profile image
Moral and aesthetic judgement are largely similar processes, both neurologically speaking and in that the conclusion is reached almost instantly - you know if you like a painting, find something offensive, or are having fun playing a game very quickly. But the justifications for arriving at these conclusions are all just rhetorical smoke and mirrors that just present the illusion of reason.

Johnathan Haidt (who has a good TED talk and some good books) did a lot of work on moral and by proxy aesthetic, judgment. To prove out something he calls "moral dumbfounding" he came up with this brilliant moral scenario:

"Mark and Julie are a brother and sister vactioning in France. They have some wine, and drunkenly decide to have sex. They use two forms of contraception so there is no statistical risk of pregnancy. While both enjoyed it equally they decide to never to do it again and they never do. Their platonic relationship is strengthened from that day forward. Did they do anything wrong?"

Literally every single person agrees that it's wrong. But Haidt presses them to explain exactly why. The most common answer is the risk of genetically abnormal children with birth defects, to which he points out the contraception eliminated that risk entirely. But that doesn't change anyone's mind.

Haidt explains it like this: "So whatís really clear [is] people give a reason. When that reason is stripped from them, they give another reason. When the new reason is stripped from them, they reach for another reason. And itís only when they reach deep into their pocket for another reason, and come up empty-handed, that they enter the state we call ďmoral dumbfounding.Ē Because they fully expect to find reasons. Theyíre surprised when they donít find reasons. And so in some of the videotapes you can see, they start laughing. But itís not an ďitís so funnyĒ laugh. Itís more of a nervous-embarrassment puzzled laugh. So itís a cognitive state where you ďknowĒ that something is morally wrong, but you canít find reasons to justify your belief. Instead of changing your mind about whatís wrong, you just say: ďI donít know, I canít explain it. I just know itís wrong.Ē So the fact that this state exists indicates that people hold beliefs separate from, or with no need of support from, the justifications that they give. Or another way of saying it is that the knowing that something is wrong and the explaining why are completely separate processes."
--(http://www.believermag.com/issues/200508/?read=interview_haidt)

Ultimately there's a similar process that occurs in aesthetic judgment at well. We're all just basically confabulating "reasons" as to why we enjoy a work of art or a book or a video game. It's not useful data because it's prone to so many errors and is, according to Haidt, a completely separate mental process altogether.

[EDIT:] Just to make this point clearer - we might stumble upon a "correct" answer, depending on our expertise with a certain subject. Obviously an art history major might have more "valid" reasons (read: useful as feedback) why they find a particular piece appealing. But they arrive at this conclusion that they like it no faster than anyone else with no artistic training.

And unfortunately game players often mistake their expertise as an end user with that of a developer and propose really overly detailed or specific solutions that aren't useful. You don't often see someone who goes to a lot of movies feeling like they can tell a director what to do differently (besides George Lucas). This is because games are an interactive medium and that interaction leads players to feel like they share some authorship (look at the Mass Effect 3 ending blowout for example). But at the end of the day we're not selling the game we're making to our players - we're selling them an EXPERIENCE, which we all tend to confuse sometimes. The game we make SUPPORTS and ENABLES their experience, but they have partial authorship as interactive participants in the experience ONLY, not the game. You have to make a distinction between the two if you want to implement user feedback correctly.

There's a story that ran on The Verge about Gearbox's "Truth Team" (an in-house Focus Testing department that makes use of the local colleges, and incidentally the place where I made my first industry connections). It was run by a cool cool lady named Stephanie Puri who said of game testers "They speak about their experience, not what actually exists in the game."
--(http://www.theverge.com/gaming/2012/3/14/2861998/gearbox-borderla
nds-testing)

The article details how in one of the first areas of Borderlands, Skagg Valley, players complained about the number of enemies in the area and that the combat was slowing down their progress. They resolved this by DOUBLING the amount of enemies in the area and the testers loved it. They took an area where the players wanted to explore and the infrequent combat was an impediment, and turned it into a high-concentrated combat area where the exploration was just a bonus.

So... if anything I feel this article's title isn't harsh enough. Players aren't intentionally trying to mislead you, but they will if you take their afterthought justifications / explanations of your game's faults on face value. Feedback isn't so simple that you can simply get a 1:1 return on the objective truth about your game every time you ask a question. You have to understand people, have to understand feedback, and understand what it is that they really are trying to communicate beyond just simply what they fill out in a survey text field. And that's the difference between bonafide Holy Grail user research and "metrics". Leave the latter for marketers and advertisers.

Len Taguchi
profile image
This is a rather misleading title.

Simon is one of the very rare developers who braves the forums of his games. He has consistently listened, and always politely interacted with player feedback for the 10 years that I've known him online.

Even now he has invited people to become part of the design process for his latest game as is evidenced by the abundance of positive discussion on the Sunstone Games forums: http://kaijucombat.com/community/index.php

To imply that this man disregards player opinion is disingenuous to say the least. The title should read "Why your player's opinions really matter". Bad form, Mr. Cifaldi.

Aaron Casillas
profile image
A designer is like a doctor, you should hear your patients ailments and figure out the best remedy. Sometimes the patient will think it's their knee that's causing the pain in his lower back, when in actuality it's the shoes he's wearing.

David Boudreau
profile image
misleading title.


none
 
Comment: