Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
April 18, 2014
arrowPress Releases
April 18, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM TechWeb sites:


Opinion: Player Metrics Vs. The Vocal Minority
Opinion: Player Metrics Vs. The Vocal Minority
October 31, 2011 | By Daniel Cook




["The internet is a series of echo chambers where bias massively swamps any real signal about player behavior," says Spry Fox's Dan Cook, who questions the role of the squeaky wheel in the age of metrics. (Reprinted with permission.)]

I continue to be fascinated by the disconnect between vocal players / reviewers and player metrics.

The standard theory

Over on Play This Thing, a user made a comment about how vocal players are like vocal customers that complain about service at a restaurant. The silent majority may never complain, but are very likely to share the same issues as the complainer.

The theory goes that even though only one or two people are pointing something out, you should pay extra attention to these words of wisdom since they represent large customer service issues that threaten your entire business if left unchecked.

This model is referenced by both vocal players and by self professed critics in order to boost their importance and authority.

My experience

Yet repeatedly I keep running into situations where this doesn't hold true. A vocal sub-segment of the community claims that something is horribly broken. Yet when you run the metrics of player behavior or run surveys of the silent majority, the complaint is either not an important factor or is an attempt to manipulate the game in favor of a very specific minority voice.

Some examples (these have a lot more detail, but the gist will suffice)

Example 1: There was a virtual riot (complete with organized virtual protests) in Realm of the Mad God when we changed the range of projectiles.

Suddenly, players who could no longer kill enemies far off screen could, because projectiles went half as far. We did this because a group of players had decided to primarily play using the mini-map and onscreen tactics were going extinct. There were many complaints about how the game was completely broken.

Yet fun scores and retention increased. Within a week or two of the change, people had moved on to talking about something else.

Example 2: In Triple Town, experienced developers whom we show the game think that the turn system is a mistake. Traditional core players complain about the microtransaction system. Kindle players dislike the fact that this isn't a one time price for all you can eat.

So one of the very first things we did was test how big an impact purchasing turns had on the player experience. The simple answer is that it doesn't seem to alter short term retention or fun scores. We are still collecting long term retention data (the game just launched), but it is immensely obvious that the big huge overwhelming issues facing the game have nothing to do with the primary complaint of the most vocal players. Our biggest issue is the first 200 turn experience around the tutorial. This is something that not a single player or review has mentioned.

Example 3: In Realm of the Mad God, a small change was made to how a key for unlocking a special dungeon worked. This resulted in a massive fervor on the forums. A group of players was using these rare items as both a means of storing wealth and as a method of controlling who received rare drops. Since this group is the most dominant group on the forums, they created a large discussion about how the game was floundering.

Again, the metrics showed either no change or a positive change. On deeper investigation, the elite group essentially was using their economic power to dissuade the majority of the players from accessing top end content. There was indeed something very meaningful going on (players really want to self organize into smaller groups and control who is in the group and who is out of the group), but it was occurring in a dysfunctional fashion.

General themes

There are a few reason why the disconnect occurs:

Communities are internally consistent games independent of your game. The first is that the community is playing a game that isn't your game. Within a forum there are groups jockeying for power and influence. They'll say whatever it takes to advance their position independent of the actual situation in the game.

Sometimes this is an organized rhetorical attempt to gain an advantage. Other times it is the result of limited perspective. To the players engaged in the game of running a mini-mafia inside of Realm, they were playing the game as their friends and the rules of the system dictated. The fact that "newbs" suffered was unimportant since they weren't meaningful players of the game at hand.

Loss of old skills and knowledge or the requirement that players learn new skills always provokes an angry response.

Change that isn't merely the addition of new content is almost always seen in a negative light. Learning has a real associated cost in time and effort. Anytime a game takes away a perceived skill, players feel that their time spent within the game is invalidated. They have immense difficulty imagining a better future as a result of the change and instead focus almost entirely on the intense sense of loss.

The game is seen as an opportunity to promote an ideology. Being anti-"social games" has little to do with social games and almost everything to do with having an identity as a specific class of core gamer. How you buy and play games signals that you are part of an elite group and how you are not your Mom.

In this light, any game that fits within a rough stereotype is seen as an opportunity to repeat a variety of message points. For core gamers, you get repetitive digs that X game is "a Skinner box," "immoral," "not a real game," "taking advantage of players," etc. Whether these points fit the particular game at hand is rather unimportant.

Consider Hero Generations by Scott Brodie, a beautiful example of a painfully honest indie game trying to make the world a better place. Yet as soon as people see it is on Facebook, out come the various slurs that acts as a means of "dehumanizing the Other" and help a negative forum post cement the original poster's position in the tribe of inward-looking bigots.

So you get hilarious comments like this one which one moment praises the game and then immediately starts repeating negative memes about Farmville:
"the game is nice, but just like the other crappy fb games, it looses its nostaliga in minutes. in reality, you do the same thing all the time, it sucks, tedious, and has bo ending, just like the milking machine called farmville."

Needed: measurements of actual player behavior. The internet is a series of echo chambers where bias massively swamps any real signal about player behavior. The more specialized the community and the more that its interests are focused on activities other than the game at hand, the less meat there is in its mental models and proposed solutions.

This situation puts game critics at a huge disadvantage. Their natural authority is inherently low and their comments are deeply and systemically disconnected from any sort of objective reality.

I continue to listen to passionate players, but I more and more preference measurable and observable player behavior over rhetorical opinion. There are indeed moments of truth that lead to improvements in the game to be skimmed from the morass of disingenuous deceitful, warped, navel gazing and self-serving game commentary.

For example, the complaints about destroying Realm of the Mad God's accidental mafia helped crystalize thoughts about guilds. However, even in this case, the stronger signal was the behavior of the players, not the ranting itself.

Why are teams embracing design empiricism based off measurable player behavior? Because the alternative forms of critique offer so little that is meaningful or truthful.


Related Jobs

Telltale Games
Telltale Games — San Rafael, California, United States
[04.18.14]

Producer
Telltale Games
Telltale Games — 94903, California, United States
[04.18.14]

Technical Release Manager
Telltale Games
Telltale Games — San Rafael, California, United States
[04.18.14]

Sr. Platform Engineer (emphasis on Next-Gen and TV!)
Telltale Games
Telltale Games — San Rafael, California, United States
[04.18.14]

Senior Systems Engineer - Core Technology










Comments


Joe McGinn
profile image
Great read Daniel, really appreciate you backing your thoughts with real examples.

Josh Tolentino
profile image
Having empirical data to back up (or debunk) the complaints of the vocal minority is indeed a powerful tool for tweaking a game and ensuring it has a better future, but I don't think it's entirely wise to simply dismiss the opinions of the players out of hand on the grounds of their being "disingenuous, deceitful, warped, navel-gazing and self-serving. The vocal minority are often the players most passionately engaged with the game or the behaviors and meta-games born therefrom.



It's not that they're the ones to be kowtowed to; they're more often wrong than right for the reasons you stated above, but simply relying on cold metrics and working solely from THOSE is as potentially damaging to the foundation (or future) of a game.



Furthermore, the "silent majority" who do not complain, and that you find do not actually have a problem with what the vocal minority are complaining about, sometimes may simply be unaware that a flaw exists. To use an awkward metaphor, while the core, vocal complainers might often miss the forest for a tree, that doesn't mean the tree isn't diseased.

Daniel Cook
profile image
As a side note, there is also some great discussion on this essay happening over on G+

http://bit.ly/uZGed9



Here are a couple of additional notes I made there:





Re: Application to reviews

I think a lot of the standard model of listening to the complainers comes from the reality that most traditional businesses have poor insight into user behavior. Often any feedback is seen as better than no feedback. Much of the tradition of 'criticism' comes from a time where simply talking about a user experience in a meaningful manner was a surprisingly difficult feat. Online games are in a rather unique situation where the developers have the ability to collect detailed, statistically meaningful data about a wide range of activities. It is, in essence, a disruptive technology that fundamentally alters that nature of informed critique.





Re: Role of community management

There is a great opportunity for connecting qualitative community feedback with quantitative player behavior. Right now it takes some serious effort to tie a specific user's complaints into any sort of understanding of who that user is in the game. How long have they been playing? Do they fall into any predefined player categories? What is their ARPU? What does their friend network look like? Can I observe them playing?



Once you have this, you can take the next step and validate the issue. In some sense, there is the equivalent of a 'data validation' on community issues just as there are 'repro steps' on bugs. Instead of just reporting that the forums are going mad about something, I'd love to see a connection between recorded behavior and the complaint at hand. If this means community management becomes expert data wranglers, so be it.



A validated issue is more than just a method of filtering out garbage requests. By connecting data and an issue, you are much further along in identifying the root causes and then coming up with a solution that isn't merely a half-assed patch.





Re: What are fun scores?

Fun scores are a simple 1-5 rating by the player at a specific point in time. The question is "How fun is this game? (1= Not fun, 5 = very fun)".



I generally split groups into several cohorts of equal size that are surveyed independently of one another (so that the act of giving a rating previously doesn't pollute subsequent ratings.) One group is surveyed 5 minutes after a successful launch, the next group 15 minutes and so on up to several weeks or months.



I like time stamped fun scores since you can see retention and how average user ratings changes over time (there is a success bias here, you need to be aware of). Since you measure thousands of actual game players in the heat of the moment, the results tend to be much more representative than reviews or forum feedback.



The biggest factor behind why I use fun scores is that they are highly predictive of distribution success on Flash portals. A score over 4 means millions of people will play your game. A score of less than 4 means that mere thousands (or hundreds) will play. I generally measure how a game rates and then work on the problem areas until I hit my goal of 4 or above. (Note: Fun scores are not predictive of retention or ARPU.)



I'm using Net Promotor scores as well.



take care,

Danc.

Lars Doucet
profile image
Posted this on Google+, reposting it here:



This is a fascinating re-enactment of the age old Humanities "vs." Science debate. Those on the pro-science side generally appeal to all the cool gizmos their theories have led to, and the (mostly accurate/reliable) predictions they can make about cannonball trajectories, neutrinos, etc. Despite all the advances in science, though, I wouldn't say that Humanities or the "squishier" side of thinking is done, played-out, superstitious, or irrelevant, however. It has a value all its own that's difficult to quantify, and therefore, judging it by the standards of science, or mistaking it for science, will always make it look bad because it operates in a completely different way.



Practical application for our debate:

Treating "thoughtful game design" as a predictive tool, or using it as a replacement for an evidence-based method, will not get you useful results (ie, listening to one dude's opinion and considering it representative). On the other hand, using this anecdote as evidence that "thoughtful game design" is wrong, superstitious, played-out, etc, is equally wrong and misleading.



As always, you need a balance of both of these. I think when we get right down to it, at least in this crowd of designers, nobody's calling for the abolishment of one or the other, but I think criticism from the opposing camp naturally makes us take defensive positions. If you're slightly on the science-first side, hearing calls from the "anti-metrics moralists" (a hat I wear myself sometimes), makes you feel like you have to defend science from the book-burning hordes. If you're slightly on the humanities-first side, hearing people praise the righteous march of numbers and statistics makes you fear for the death of human compassion and take your stand at the battlements.

Robert Hewson
profile image
Indeed, I think metrics are often useful for reinforcing existing concerns and focusing in on specific area for improvement, but they have to be used sensibly. Then again it's possible to get hung up on data which supports (or seem to support, or you could argue supports) an concern you already have and ignore the fact that it's not flagged up by the vast majority.



As with everything balance, discussion and objective contemplation, as much as possible, is key.

Robert Hewson
profile image
Very interesting read, I think you do need to learn to read between the lines when analysing metrics. There will always be political and social aspects which skew the data and the specific examples you give show this.



I've been discussing something similar in my blog about the Darknet in game feedback system built into Hydrophobia Prophecy which releases on PSN this week: http://www.gamasutra.com/blogs/RobertHewson/20111031/8771/Darknet
__in_game_player_feedback_for_the_Net_Generation.php

Kostas Yiatilis
profile image
So game communities work just like real communities. Elite trying to retain their power.

Aleksander Adamkiewicz
profile image
I find the development/modification of games after metrics to be an offensive practice, no matter if the metrics come from direct user-feedback or statistics.



None of the methods actually tell a developer if the change is "good" only if it satisfied enough people.



This goes both ways, both with metrics and vocal minority.

Jeff Wesevich
profile image
I loved the outline of the "standard theory." I first saw rabble-rousers using that as a justification back in the late '80s on Compuserve... :-)

Ian Bogost
profile image
Dan, what if both conclusions are true, but they're true of different things. Why should it be necessary to confirm that what people do is unrelated to what they say or value?

Michael Joseph
profile image
This article is extremely biased and that's pretty ironic.



"This model is referenced by both vocal players and by self professed critics in order to boost their importance and authority."



So it's a model, it's a theory, it's used to boost importance and authority and because I can demonstrate that one random person's complaint or criticism is incorrect, I can invalidate all complaints and criticisms.



Or wait, do you actually still have to pay attention to criticisms and evaluate them? And how is that different than any time before?





"The first is that the community is playing a game that isn't your game. Within a forum there are groups jockeying for power and influence. They'll say whatever it takes to advance their position independent of the actual situation in the game."



We like the community until they say things we dont like and then they're a bunch of selfish power hungry liars? Of course we know that with communities all over the world you take the good with the bad. There's nothing wrong with having a nuanced view but your response time and time again in this article is to be completely dismissive and not acknowledge the need for balance.



"The game is seen as an opportunity to promote an ideology. Being anti-"social games" has little to do with social games and almost everything to do with having an identity as a specific class of core gamer. How you buy and play games signals that you are part of an elite group and how you are not your Mom."



More insults and generalizations? "They're elitists! I can prove it, some random person said some random thing about a game and he was wrong!" And although there's nothing wrong with promoting one's ideologies, surely it's not fair to say that all people who criticize social games are promoting an ideology.



p.s. I've always found it curious that people should try to use as a pejorative label the word "elitist".

Nestor Forjan
profile image
There's a pretty extreme jump in logic at play here, isn't there? Daniel seems to be assuming that a) there are no instances of vocal criticism based on actual issues or, at least, that there is no reliable way to separate both without looking at the metrics and b) that the metrics don't generate false positives like the verbalized feedback does.



Without direct feedback context, in my experience, "fun scores" and other arbitrary metrics defined during testing are just as unreliable as hand picked forum complaints. The more complex a game is, the more difficult it is to isolate "fun" (or any other quality metric) impact from in-game issues to the actual source. It is a very directly creative and artistic process guided by both qualitative and quantitative feedback to go from a player concern or a design issue to a content change, especially in very complex, large scope triple-A products. Either you painstakingly define all your variables and communicate them to the subjects in very narrow terms so you can parse them in a lot of detail... or you make use of our built-in ability to communicate, let the players speak and listen to what they have to say as well as collect measurable data.



So, while I would certainly not recommend designers to go find every single piece of criticism thrown at them in online forums, I don't appreciate the activist post promoting of "quanti" over "quali" or the self-defeating suggestion that because there is bad feedback on direct consumer feedback all of it should be ignored. And yes, it is fair criticism that this article is, in turn, a move in a metagame of metrics-based design versus other design models that is currently happening in the industry.

Zack Hiwiller
profile image
Fun scores and other similar metrics are like the Pepsi Challenge. They gauge "sweetness" not success.



Then again, I deleted Triple Town after the coin turn cost increased. I guess based on metrics I was the only one.



I remember when I was working on football games for a large developer (hint hint). We had vocal ten-plus year players that screamed and complained about the minutiae every day on the forums and called us all sorts of bad names. Nothing we ever did seemed to please them, so we marginalized them and internally called them "fanatics" and "forum crazies" so we could focus on what we would rather focus on rather than deal with the problems that they brought up. After all, they grudgingly bought the game every year so they couldn't hate it that much!



It's odd that many of us treat the user's collective preferences as sacrosanct and most interested individual user's preferences as baseless.


none
 
Comment: