Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 15, 2019
arrowPress Releases

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

GDC Europe: Playfish's Valadares on Intuition Versus Metrics: 'Make Your Own Decisions'

GDC Europe: Playfish's Valadares on Intuition Versus Metrics: 'Make Your Own Decisions'

August 16, 2010 | By Brandon Sheffield

August 16, 2010 | By Brandon Sheffield
More: Console/PC

Brains are horrible for predictability, said London-based Playfish studio director Jeferson Valadares in his packed session on intuition versus metrics in social games on the first day of GDC Europe in Cologne.

Most games in the traditional space are based much more on intuition and experience than they are on user data, but in the social game space, metrics have an importance that makes many developers feel uncomfortably tied to the whims of their users.

Citing psychological research, Valadares points out that if you throw a coin 9 times and it turns up heads each time, people will generally presume that the chance it will be heads is 10 percent -- when in fact its at least 50 percent.

Metrics can help take care of these sorts of decisions that shouldnt be based on intuition, though there is some element of marketing to it. Restaurant menus, for instance, generally have an anchor price, he points out. Its the most expensive thing on the menu, and it may be in a prominent place. But by comparison, it makes the second and third most expensive items seem much more reasonable.

These metrics are good for optimizing, says Valadares, but that wont take you to the next level. With metrics, if you look at the space of the solution for the problem, youll reach the top, he says.

What you might not know is that there may be a better top that you didnt even consider. Thats where intuition comes in. If you want to do things that really innovate and differentiate from whats happening, you really need guts, he says.

Valadares cited a quote by marketing guru Seth Godin, who wrote: The analysis, based on past events, certainly seems sound. But your instincts are the only way youre going to do something unsound.

By way of example, Valadares discussed engagement in Playfishs Pet Society. When they introduced fishing, there was a big jump in users, but there were no metrics to support the idea that this would be successful they had to trust the designer. But on the other hand, some new features, like gardening, brought the game down, he admitted. Its not going to get you good results every time, but its the only way to ever get great results.

Metrics For Metrics

Valadares believes in intuition, but that it should be used carefully. His own rules for use of intuition are:

Experience is important. You need a solid basis of prior work in order to base your decisions on intuition.
Lean toward it for complex problems, not for details. If youre deciding whether the border of your HUD should be one color or another, you test that with metrics. Dont spend your time thinking about it, because there are more valuable things you can think about.
Lean toward it for long-term planning. Even if you fail once but overall theres better improvementif you want to take a long-term view, data can be a bit deceiving at times, he says. Just because something happened in the past doesnt mean its going to happen in the future.
Back up your intuitive decisions with data whenever possible. Then try new things.

Metrics are good at things that our brains suck at, he reminds us, like numeric choices. Theyre great at optimizing details. Here are some good things about metrics, according to Valadares.

What gets measured gets done. Developers will actually act on things that have metrics, because its trackable and quantifiable.
If you have good metrics, it drives good behavior in the company. All discussion revolves around how to improve a part of the game that everyone can see is bad.
Good metrics help you make good decisions. Well-chosen statistics allow you to make very well-informed decisions about your userbase.
Metrics are good for highly-measurable environments, which lead to results very quickly (web environments, for example). Playfish, for instance, does postmortems for new features every week. If you take three years to figure out if your game is good or not, he prompts, maybe you shouldve used metrics.

There are bad things about metrics too though, which closely mirror the first list:

What gets measured gets done -- if you dont have metrics for something thats really important, people may not do it.
Bad metrics drive bad behavior in the company, if youre looking at the wrong things. A lot of companies have incentives tied to certain metrics, which is fine if you do it right, but it can lead employees to cheat the system in order to increase their numbers, he says.
Bad metrics help you make bad decisions. If youre not tracking the right stuff, you can be making decisions in the completely wrong direction.
Metrics are bad for noisy environments with long feedback loops, he says.
Metrics can age quickly. Three months go by and suddenly [what youre tracking is] not the thing to watch anymore, he cautions. Its time to start working on some other thing.

Best Practices

How should you use metrics then? Valadares says you should align them with your goals, business or otherwise. Dont just track whatever you can. If its not going to help you get to the next level, theres no point, he says.

Keep it simple -- chart a few good metrics, and dont bloat your stats with unnecessary data. Its better to keep three to five very important ones that are simple to understand, he says. At Playfish they had one particular variable, a measure of retention -- every time I asked people what it was, it took three minutes to explain, he said. Now we have another variable which people get easily. If people cant readily explain what the metrics actually chart, they cant really be used efficiently.

And you shouldnt worry about perfection right off the bat. Its better to start with something, then improve, he says. And then you should continue to think about those metrics and change them if theyre not right.

You really need both, says Valadares. Theres a time and a place for everything. And if you get good at using both at the same time, youre going to have a big advantage.

By way of example, he discussed a new web designer that joined the studio and thought they should get rid of their banners, because in his experience, nobody clicks on those. But Playfish had metrics to prove that they were working. Intuition certainly would indicate that banners arent that useful, depending on the market you come from, but metrics can prove this quantifiably.

People tell you one thing and do another, he reminds us, speaking specifically of the whims of the user. Say theres a fire in the forums, and everyone seems to hate a new feature, but metrics prove that it added users and people like it. Sometimes listening to themits always important, but it might not be the best thing for a business, he says.

Youve got to use both intuition and metrics at the right times, but dont hide behind either one. Many times I see people running away from making a decision because of what was done in the past, he cautions. Just because you did something in the past and it worked, doesnt mean itll work in the future. It might work, but dont use that as an excuse to take responsibility away.

As a final word, Valadares had this to say: Make your own decisions, and dont let metrics decide what you do.

Related Jobs

Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States

Project Manager
Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States

Outsourcing Manager
Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States

Environment Artist
Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States


Loading Comments

loader image