Gamasutra: The Art & Business of Making Gamesspacer
Hot Failure: Tuning Gameplay With Simple Player Metrics
View All     RSS
October 21, 2014
arrowPress Releases
October 21, 2014
PR Newswire
View All

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

Hot Failure: Tuning Gameplay With Simple Player Metrics

December 16, 2010 Article Start Previous Page 2 of 4 Next

The Basic System

The event logging system that I wrote has three parts: a thread in the game runtime that collects player events and sends them to a server; the server itself; and finally a tool to parse the data recorded by the server.

"Server" is a strong word in that second component. My server is actually a PHP script that, in about 30 lines of code, validates the HTTP Get query it is sent and writes the results to a MySQL database. The query itself is dead-simple: it's just an event name, level name, xy location, version code, session id, and time stamp. These fields are recorded to the database verbatim. The actual processing of the data is also done in PHP (a poor choice, in the long run; more on that later), though only on demand when a special dashboard page is loaded.

I started with just two events: player death and level completion. Each time a player dies or completes a level, the game reports that event to the server. From this data, I was able to construct a pretty detailed overview of the game flow. I could see which levels took the longest, which had the most deaths, and which were unusually short.

By dividing my values by the number of unique players, I could also see what percentage of players died on certain levels, and the average number of deaths for each player.

By looking at the spatial location of the event, I could tell the difference between a death from an enemy and a death from a pit. As a first-pass implementation, my simple metrics system proved to be pretty detailed.

Highlighting Failure in Bright Red

Once I had the basic reporting system up and running, I released an update to my testers and watched the data flow in. Very quickly, patterns emerged; there were some levels where almost 100 percent of players died at least once, and other levels in which players were getting stuck for hours (indicating a pretty major failure for a level designed to take five minutes). Just by looking at the numbers, I had a clear picture of which levels needed the most work.

But identifying problematic levels wasn't enough. Sometimes I couldn't tell why a particular level was a problem.

So I went a step further. Using the same data, I wrote a tool to plot the death positions on top of the level art so that I could see exactly where users were dying (and where they were not). The first pass of this system just drew a little dot on the level art when a player died, but once the number of players grew to be large, I switched to rendering heat maps of death locations over the levels, which was much easier to read (see "How to Make Heat Maps," at the end of this feature).

Heat map generated from player death statistics in Replica Island (click for full size).

Game Design Failures as Object Lessons

The combination of high-level play statistics and plotted death locations was illuminating. I learned, for example, that a huge number of players were dying at the very first enemy. This was not because the enemy was particularly hard; after considering the problem, I realized it was because the enemy appeared in a spot where the main attack -- a crushing butt stomp, performed from the air -- was difficult to accomplish due to a low ceiling.

I also learned that my simple dynamic difficulty adjustment system needed adjusting itself. This system secretly increases the player's life and flight power after a certain number of consecutive deaths, and by looking at the data, I could see that it needed to kick in a lot earlier.

I also made sweeping changes to my level geometry. I had a few levels with very high completion times but very few deaths, and I realized that players were simply getting lost. I reworked these levels to make the paths through them clearer; in one or two cases, I scrapped an entire level and made a new one from scratch.

But the biggest problem that I identified was with pits. Replica Island is a platformer, and as you can guess, it involves a lot of jumping over pits. But unlike certain spinning marsupials and pipe-dwelling plumbers, my character's main mode of transport is flight.

I needed a control system that did not require a D-pad, so the protagonist in Replica Island, the green Android robot, flies using rocket thrusters on his feet. The basic movement model involves getting momentum up while on the ground before jumping into the air and using that momentum, along with the thrusters, to fly around. The thrusters run out of juice quickly but refill when you land, so the idea is that a player will jump into the air and then carefully expend his fuel to reach distant ledges or line up a precision butt stomp.

All that is well and good, but when I looked at the death data coming back from my playtesters I found that they were dying in bottomless pits en masse. Droves of players were falling down even the smallest of holes. And of even greater concern, the death-by-pits numbers did not decrease over the course of the game; players were not getting better at making jumps as time went on.

With this information in hand, I reviewed my core game and level design and came up with a number of theories. The basic problem, I decided, was that players could not see the pits they were jumping over. First of all, there was no visual indication that a pit of death is a pit of death; since my levels are often very tall, it's hard to tell which pits lead to some underground level segment and which lead to a grisly demise.

Second, and most important, my camera was not doing a good enough job of keeping the floor visible when the player jumped into the air. Almost as soon as the player leaps into the air the ground would scroll off the bottom of the screen, making it hard to judge where to land.

Master platformers like Super Mario Bros. almost never scroll vertically; Mario has a whole set of complicated rules dictating which specific circumstances allow the camera to move up and down. In Replica Island, however, the flight mechanic meant that I had to allow vertical scrolling in the general case. After a bunch of tweaking, I came up with a smarter camera that does not begin to scroll vertically unless the player is close to leaving the visible space themselves.

After making these changes, I shipped another update to my beta testers and compared the results to the previous version. The deltas were very reassuring; deaths were down overall, level completion times were, for the most part, back into normal ranges, and pit deaths dropped by a pretty huge margin. I iterated several more times with these testers before I was ready for release, but with the metrics reporting system in place, it was easy to see whether my changes were having an influence on how my testers were playing.

Article Start Previous Page 2 of 4 Next

Related Jobs

Treyarch / Activision
Treyarch / Activision — Santa Monica, California, United States

Senior UI Artist (temporary) Treyarch
Treyarch / Activision
Treyarch / Activision — Santa Monica, California, United States

Lead UI Artist
Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States

Senior AI Engineer - Infinity Ward
Treyarch / Activision
Treyarch / Activision — Santa Monica, California, United States

Senior Software Engineer - Treyarch


Carl Chavez
profile image
Good article! Thanks a lot.

Simon T
profile image
Excellent article, very informative.

Just wondering, how many testers were in your test group?

Robert Boyd
profile image
Really interesting article.

Chris's blog on horror games & design is one of my favorite sites as well. Very informative.

Paopao Saul
profile image
A very informative read! I love articles that go in-depth with their technical details. Thanks Chris P.

Maurício Gomes
profile image
Whoa, cool article!

I have the same problem in my game, the testers play but never report back...

I only wonder what data I need to collect :/

Mark Venturelli
profile image
Very good insight on low-budget data gathering, but as your example proves, *nothing* replaces good old direct observation. If you just analyze data without actually seeing how people are playing the game, you are bound to make some misguided design decisions.

And when I say "people playing the game" it's not only the gameplay footage, but the actual person, their facial expressions, button presses, and so on. Just gather some friends who never played it before and you're golden (as long as you keep your mouth shut during the session).

Caylo Gypsyblood
profile image

Eric Schwarz
profile image
Excellent and detailed article. Using heat maps sounds a lot like what Epic did when developing Gears of War multiplayer levels - but rather than try to reduce deaths, they used them to make sure players were using the majority of the space and taking advantage of certain weapons and tactical options effectively.

I remember when I was designing a single-player mod campaign for Crysis, I got my friend to have a look at it. There was one specific place where the player was given a sniper rifle overlooking a village, with the intention that he or she use it to take out the guards below. The problem, of course, was that neither the cliff overlook or the sniper rifle were very visible, so I spent a huge amount of time just toying with the vegetation placement, lighting, and the approach to make the overlook that much more apparent, and more visible than the side path down. It's the kind of thing I simply wouldn't have known to fix without just one extra pair of eyes on it.

Manuel Mestre-Valdes
profile image
I did play "Replica Island" on my phone a few months ago. I was oblivious that all this data gathering was taking place, but I think it's a brilliant idea to improve the game.

I liked the level design, the clever use of the HTC Hero trackball, the open source origins and the overall graphical and design quality of the game.

The problem for me is that I don't think it's a genre suited for a smartphone. The use of trackball was clever, but it can't compare to the feel of buttons.

Jordan Lynn
profile image
Back end data is extremely useful and inexpensive (when implemented like you did it), but it can only tell you who did what, when, and where.

Direct observation is more time and resource heavy, but is the only way to answer any question that starts with "Why...?"

The best approach? Do both, using the former to identify problem areas, then observing new players playing those sections to get detailed player information.