Gamasutra: The Art & Business of Making Gamesspacer
The Metrics Aren't the Message
View All     RSS
September 18, 2014
arrowPress Releases
September 18, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
The Metrics Aren't the Message

March 11, 2013 Article Start Page 1 of 3 Next
 

Metrics can rule you -- but should they? The Workshop Entertainment's new design director and Free Realms veteran Laralyn McWilliams explains how a pivotal moment in her life showed her that overreliance on analytics and friction in social games isn't the answer.

This article isn't about whether free-to-play games are bad, or social games are evil. For the record, I don't believe either case is true.

This, like life and like metrics, is about evolution. It's about change. It's about picking new behaviors based on the results of previous behaviors. It's about how an understanding of the end game can change the way you play.

I'm making a note here: HUGE SUCCESS.

It started back in 2006, when I began working on Free Realms. There really weren't many free-to-play games outside of a few upstarts from Asia, so the whole space felt brand new. I was lucky enough to work with a great group of developers at Sony Online Entertainment with an unprecedented amount of experience in online games. As a team, we understood how to make a great MMORPG, but free-to-play was the Wild West.

After launch, we started gathering data and looking at metrics. Although tracking and logging player choices had been an important part of online games for years, we quickly realized we needed better information and we needed it more quickly. This kicked off a frantic but exciting year of post-launch changes, and metrics played the key role.

Metrics gave me information, but they also let players talk to me directly and honestly. This wasn't the "guess, ship, and pray" design process of console games. I wasn't making decisions in a void anymore. With a combination of understanding the game, tracking changes, watching metrics, and listening to players, we made significant improvements to the player experience. We also significantly improved the amount of money we made.

This was free-to-play at its best: happy players, happy development team, happy businesspeople and execs.

I started speaking about metrics at conferences. I talked about what an important design tool they provide, and how they can guide your decisions on a live game (or on the game before launch via usability testing). I shared some metrics about play patterns among casual players, including some findings that were genuinely surprising to the development team.

Then I entered the world of social games.

We do what we must because we can.

Development of social games revolves around three core concepts:

  • Metrics are the basis for all decisions.
  • Monetization is based on friction.
  • Metrics and monetization are tuned to optimize the output of whales.

A common way to look at metrics for online games is using a funnel. The typical funnel for user retention and monetization looks like this:

The funnel would have to be scaled larger than your monitor for you to actually see the whales down there at the bottom. That’s the size of a group they represent when viewed in context with the rest of the player base. 

For many companies, modern free-to-play design -- especially in social and social/mobile -- focuses on the whales. The game is tuned to please the whales, even though the personality that lends itself to the highest spend in a game is certainly an edge case.

The games aren't just tuned to please the whales, though: they're tuned to squeeze the maximum amount of revenue out of them. Since the monetization is based on friction -- on players paying to bypass elements that stop them from playing, completing, or enjoying aspects of the game -- squeezing the most money out of the whales means continually turning up the friction until you hit the "sweet spot" where they pay regularly.

For the good of all of us (except the ones who are dead).

With the increase in friction comes the increase of players who hit the wall where the session gets so short or the grind gets so tedious that they quit playing. Online games have always had a certain amount of players quitting (called churn). The churn in social games is tremendous because the friction curve quickly gets so steep it curves most players right out of the game.

I'd entered the world of social game design with strong experience in free-to-play design and as an outspoken advocate for metrics as a part of the design process. As I tried to understand social game "best practices," I watched as people made decisions based purely on metrics with no interpretation. I watched as they made changes that increased short-term revenue without regard to the churn consequences of those changes on the larger player base.

I continued to speak at conferences about the importance of metrics as an information source for designers, but when I met someone at a dinner and he said, "Oh, you're the metrics lady," I felt unsettled. I believed in metrics... as a source of information. I'd always said that metrics aren't the answer -- they're one step in discovering the answer. Yet I was seeing metrics used without any attempt to understand the long-term effects of change, or the emotional side of the play experience.


Article Start Page 1 of 3 Next

Related Jobs

Digital Extremes
Digital Extremes — LONDON, Ontario, Canada
[09.18.14]

UI ARTIST/DESIGNER
Cloud Imperium Games
Cloud Imperium Games — Santa Monica, California, United States
[09.18.14]

Technical Designer
Cloud Imperium Games
Cloud Imperium Games — Santa Monica, California, United States
[09.18.14]

Technical Designer
Pocket Gems
Pocket Gems — San Francisco, California, United States
[09.17.14]

Associate Product Manager






Comments


David Medlock
profile image
Its probably a good idea to define a word ( whales ) before you start using it.

Raph Koster
profile image
It is a widely used industry term for at least the last four years, brought over from the gambling industry where it has decades of usage. It would be like asking to define free to play.

That said, the graph illustrates it pretty well: a very high spending player, present in small quantities.

Lars Doucet
profile image
Brilliant article, measured, to the point, and clear, and a wonderful tie-in with personal experience.

Tom Smith
profile image
Well spoken, well reasoned. I now have somewhere to point people when I claim that analytics can be a valuable design tool despite the best efforts of modern social design to sully its good name.

Christopher Thigpen
profile image
I really appreciate the article. From my experience, business development tends to put the blinders on once metrics are instituted into the operations. To a fault, it almost sucks the creativity out of designers to implement new innovations, and seems to chain them to their initial designs (for good or bad) to improve upon. I have found that there are immediate disadvantages to allowing metrics to design the game experience. Use it as a tool, much like a painter utilizes a brush. That way the innovations continue to be organic.

Great article and wonderful comments.

Laralyn McWilliams
profile image
"To a fault, it almost sucks the creativity out of designers to implement new innovations..."

That's one of the core flaws of using metrics to drive all design decisions rather than as an information source. How can you innovate when decisions have to be backed by data showing success? That's especially true for innovation in monetization.

Anthony Albino
profile image
Wonderful and inspiring article. I had a similar feeling about social games, I decided to never work for a social games company again when I realized that the design model was antithetical to every game design practice I held. Your article also helped to soften me on the idea of the potential value behind metrics when used outside of old paradigms. Awesome stuff.

Jeff Alexander
profile image
This article is so very much not the unconstructive rant typical for this subject matter. Thank you for being a light above the crowd.

Christy Marx
profile image
Everyone working in games, especially social games, should read this and take it to heart. I've been saying it for the past two years, but this puts it into a powerful perspective. And not just about games.

Strength to you, Laralyn.

tony oakden
profile image
good article. well written and I agree with the sentiments. And good luck to you Laralyn.

Nicholas Heathfield
profile image
Good article and I can't help but wonder, but... isn't radiation one of the few significant things that cause cancer?

Laralyn McWilliams
profile image
Yes, it is. Head and neck radiation has other long-term effects too, like changing the structure of the jaw bone so it no longer heals itself. The side effects are manageable, though, and far better than the alternative treatment (because there isn't one... yet). :-)

Kevin Hogan
profile image
"These changes would get the small group of paying players to pay more but they would also increase the churn across the rest of the player base. I was adding elements to disrupt the play experience -- to knock the player out of the Zen state of connection with our game -- to get him to pay money.

These changes would almost certainly generate more revenue for the company."

Nice thoughtful article, but it seems that there was an unhealthy fixation with the ARPU metric in the example that you describe. The ARPU metric is easy to boost by introducing paywalls and aggressively flushing out non-payers though using such tactics does not generate more revenue for the company over time. A focus on LTV would have provided a better guide to generate real value.

I wish you the best in your recovery Laralyn.

Carlo Delallana
profile image
Thanks for sharing your fight with cancer Laralyn. I've lost loved ones to this disease and it always makes me happy to see someone winning the fight.

Metrics is something that every designer should embrace. Our design's are part of a feedback loop that is adjusted based on player behavior. We optimize for fun, engagement, and eventually monetization. We did this before there were spreadsheets of data when we would bring players in to test the game and transform their feedback into design updates.

On metrics, monetization, and design maybe one litmus test we can use is if we are comfortable being "whales" in our own game. Would I spend money on the game i'm making? Would I be comfortable exposing my design to another player and feel confident that they would not come to the realization that they were being exploited, that my design causes pain and lo-and-behold I also happen to sell the pain relief!

We have to start treating our players like we want to be treated. I think the dark side of metrics is that it quantifies user behavior to a point where we stop seeing the person behind the game. They've become numbers on a spreadsheet and I think this is what leads to dangerous and exploitative design. Because when you dehumanize someone it becomes easier to abuse them. Our current method of tracking user behavior does not allow us to empathize with our players.

Curtiss Murphy
profile image
"It's about how an understanding of the endgame can change the way you play."

When my daughter was born, the statistics weren't in our favor. "6 months" is what the doctors told us, because that's what his statistics had told him. But, we did everything in our power to fight, and fight and fight. Because the median is not the message.

Doctors and surgeries and nurses... Hope and struggle, year, after year. And after a while, I began to tire. After one of those surgeries, my daughter had a really rough recovery in the hospital. And, I remember bringing her back home and helping her get ready for bed. Maybe it was the bright pink canopy on her adorable 4-poster bed, or maybe it was the exhaustion. Whatever it was, as I was sitting there, my brain just sort of rebooted.

My thoughts became simple. I'm grateful! My daughter was alive. She had made it through another one! And when I tucked her in the next night, it happened again. And, that became my routine, night after night, for almost 10 years.

But, years later, when I went in to tuck my kids in, the kids said, 'Dad, we're too old to be tucked in'. And, as I closed the door, I couldn't help but smile. A quiet calm came over me and chills ran up and down my arms. I was happy. Despite everything, I had a good life.

This year, my daughter turns 18! She defied the statistics and continues to do so, every single day. And, as for me, well, I do things differently now. I make different kinds of products (e.g., Gratitude Habit). As Laralyn said, understanding the endgame changed the way I play and work.

Sebastian Coman
profile image
I agree with everything you said. Game design should be "creatively driven" and "data informed" (not the other way around!).

Data-driven social network games, designed by quants new to the industry (under pressure to deliver bookings), fail to engage and immerse audiences. Many social/casual games developers forgot that games have to be fun and that game design is an art. That mistake is the very reason I started making my first social/casual game. Let's give the facebook and casual audiences some immersive and nail-biting games! There are tons of "veteran casual gamers" who are ready for the quality experiences. So let's remove the bricks, pestering and grievances, and rather focus on sustainable creative experiences; letting the numbers fall where they may.

Robert Green
profile image
Great article, but there's one thing I take a little issue with, right near the end:

"Our endgame is a happy, consistent player who sees the game improving, the community growing, and positive ways for him to spend his money with us over the years."

I know we're talking primarily about social games here, but isn't expecting that people will actually want to play your game for several years more than a little ambitious? Not every game can be WoW, and probably shouldn't try to be.

jason bailey
profile image
This article frustrated me. As a 'metrics driven' social game designer, where I think it went awry was with the core assumptions that:
- Metrics are the basis for all decisions.
- Monetization is based on friction.
- Metrics and monetization are tuned to optimize the output of whales.

All three are dead wrong in my experience. And I hope that most of those who have built successful social and other F2P games would agree.
The success of the 'business' end of a game studio is based on monetization. No money, no payroll, no workers = no games. So monetization is obviously a necessary evil.

Monetization is based on having a LTV (Average Life Time Value of a player) that exceeds costs of running the game (development, servers, content creation, and customer acquisition). So you either need a massive revenue per user, or a lot of users. Preferably both!

The biggest factor in calculating LTV is retention. The longer a person plays for, the more money you will make on average. It generally costs a lot more to acquire and engage a new player than to just keep the ones you already have. So take good care of the eyeballs you have been blessed with, especially on platforms like FB and iOS where they can be obtained for free.

A game is a series of interesting choices (can't remember who said that first, apologies).

Providing of interesting choices with options to buy items of value will drive monetization. IMHO a shinier hat or 'cool' visual item is not value. Or at least not enough and only to that very very small % in the bottom of your funnel. Tools that help a little for a long time. Items that you worked hard to build over time. Limited time content. New content with a heathy heap of creativity and fun. Well balanced quests. This money. Stuff that I feel good about after I pay for it. Stuff I can't wait for more to be released of. That is value. Those are interesting choices in deciding which to pursue and build.

It isn't about juicing whales, it is about converting more players to payers. We have many whales in our games and we provide them with ample opportunity to shovel in as much cash as they damn well please. Sure this is important and pays well. But not nearly as well as the long tail of median payers. People who pay $10-$30 a month. Month after month after month. These are the players, behaviours and metrics to focus on. In our main Facebook title, we have converted more than 30% of our current DAU (about 100,000 a day) to payers. By providing value. Great content and a strong community. For years now. We have no endgame.

The 'friction' is the game. The challenges, the quests, the endless clicking. The balance of retention and monetization is key. Especially in a long term game where people engagements ebb and flow. Let them always feel like they are winning with their mad clicking skills, at the same time as only being 'a few more clicks' away from the next win. Providing a frictionless 'click here to win it all' is not full of awesome sauce. Provide interesting challenges with valuable rewards. But never forget a way to pay around them if you really want something but don't have time, skill, or patience. Even the purported 'kings' of evil in social games I would argue know this. There are many millions who have played their games for years having never payed a dime. A few, tens of thousands even, that have paid an absurd and embarrassing amount, but what buys their CEO a new private jet is the millions and millions who pay a reasonable amount.

Anyway... Metrics are king, but their point is to help you build a more engaging game people find value in and want to support. It seems like you were using these powerful tools for evil. Those poor whales.

Apologies. I seem to of written an article of my own here... Guess it is because I care. So do you, so as a fellow passionate game designers, thanks for sharing. But, respectfully, I challenge what you are saying and offer reams of data and sexy spread sheets with fancy colourful charts to back me up.

Robert Green
profile image
I think you were making a fairly good go of that until you got to this point:

"But never forget a way to pay around them if you really want something but don't have time, skill, or patience."

Time and patience are basically saying the same thing in this context - that the game involves long periods of time, and presumably not enjoyable time, because I doubt anyone has ever claimed not to have the patience to do something they enjoyed doing.

Look at the recent backlash against Real Racing 3 for an example. The monetisation strategy, which is fairly common in freemium games, is based almost entirely around completely unnecessary periods of waiting. Contrary to your comment, this friction is most definitely NOT the game; the game is about racing cars and by all accounts it's pretty good. But from RR2, which wasn't freemium, they added a number of delays (when repairing a car, when having a new car delivered, etc) which exist for the sole purpose of also being able to add a purchase to skip them. That's really the only reason for them to be there. That's why there have been console racing games for decades and no one ever thought to include these things.

Anthony Albino
profile image
That friction being the game, as you state, for social games is grinding. Grinding is generally a shallow game experience and has been commonly used to stretch out, and thus water down, many games. There's a big distinction between grinding (doing the same repetitive task over and over) and actually engaging a game that's deeper. One with interesting choices, as you point out, would be a deeper experience, and by the way was notably stated by Sid Meier.

The distinction can be subtle. One might suggest, for instance, that Tetris (based on my above assertion) is a constant grind because you're performing the same task. But this is not true, because the circumstances change with each play, and thus require a different choice and level of execution with each move. This is very different from, say, mindlessly harvesting crops which would certainly be grinding and is not particularly interesting. Nor is waiting for a timer to tick away.

Justin Nearing
profile image
@Robert - The main problem with RR3 is that the timed gates came too quickly and too hard, blocking content from users who had not yet emotionally invested in the game. However these are just data values that should be easily tuned. The problem is the tuning, not necessarily the monetization step of the core loop. Some may say the core loop is clumsy, but if it makes money and is retaining users, it doesn't matter what they say.

Robert Green
profile image
@Justin - I disagree. Fewer people might be complaining if the 'timed gates' were tuned better, but for fans of the series, their very existence is an unnecessary evil. The only decisions they add to the game are "should I pay, wait or quit?", and is either of those options more enjoyable than not having to be faced with that decision? I think the simplest explanation for why RR3 has a 24 point lower metacritic score than RR2 (despite graphical improvements and no big complaints about the driving model) is also the most likely.

Your last comment is right though - in 2013 no one is likely to be arguing against the idea that the F2P model works, and you could probably go as far as to say it's proving to be the best way to deliver gaming to the most people possible. But you could also say the same thing about McDonald's and Coke when it comes to food and beverages, but no one feels the need to argue they represent the ideals of cuisine.

jason bailey
profile image
Not saying games are without friction or that friction doesn't monetize. Time based game are still a series of interesting choices. Which, when, in what places, procured from where. Having luck be a factor is also a grind and friction. What I am saying is that by using metrics to maximize the friction and walls to juice whales does make for a game that people will love. Of course there needs to build difficult to obtain items that require a lot of 'work'. But they should be obtainable via that, not just by flipping open your wallet. Not every single choice/click is a eye ball exploding bonanza of unicorn bacon. They just need to be leading towards such.

Waiting 46 turns for your Sid's pyramids to be built was also a grind/friction. But you choose where and when. You had to have them first or the rest of the game was really difficult. civ was the same principle but before the days of F2P. If it was built today, I would definitely use my Zynga Bucks to speed that pyramid up so that I could move on to the next, would be good value in helping me win a game that I put 40 ours in to at a go.

Wessam Fathi
profile image
Great article, I totally agree with you that players can pay for many reasons other than just to keep the game dev's off their a** while having a good time. Keep up the good work and best of luck in your life.

Neil Wrischnik
profile image
Yes, amazing article. Thank you for taking the time to put these thoughts and experiences together.

Off topic or not, these uninterpreted metrics data systems seem to be a similar model for the majority of economic (or medical as you described) decisions made these days. Short term gains that neglect long term loses, that rely on impossible exponential growth. I wish this could only be compared to the world of games, and not general world decisions as a whole.

I wish you the best Laralyn. Thank you.

Andrew McKay
profile image
You're using "friction" in the Clausewitzian sense right?

PS The Answer is yes.


none
 
Comment: