Gamasutra: The Art & Business of Making Gamesspacer
Predicting Churn: Data-Mining Your Game
View All     RSS
October 2, 2014
arrowPress Releases
October 2, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Predicting Churn: Data-Mining Your Game

May 17, 2012 Article Start Page 1 of 3 Next
 

The sad truth about all online services and games? The most significant churn occurs right the first minutes and hours of gameplay. The issue has been already explored in a numerous ways, with many profound hypothesizes related to usability and simplicity of interface, availability of a free trial, learning curve, and tutorial quality. All of these factors are considered to be very important.

We set a goal to investigate why new players depart so early and to try to predict which players are about to churn out. For our case study, we used the MMORPG Aion, but surprisingly, the results appear to be applicable to a wide array of services and games. Although Aion, at the time this study was undertaken, was a purely subscription-based game with a seven-day free trial capped at level 20, the vast majority of churners left long before they had to pay to keep playing. Our research was about in-game triggers for churn.

Behavioral studies show that casual players have a limited attention span. They might leave the game today, and tomorrow won't even recall that it was ever installed and played. If a player left the game, we have to act immediately to get her back.

But how can we differentiate players who churned out of the game from casual players who just have plans for an evening and won't log in for a while? The ideal way would be predicting the churn probability when the player is still in the game -- even before she actually thinks about quitting the game.

Our goal was more realistic: to predict new players' churn the day they logged in for the last time. We define churn as inactivity for 7 days, and the goal was not to wait for a whole week to be sure the player has left the game and won't return, but instead to predict the churn right on their last day of play. We'd like to predict the future!

The Tech Side

We had tons of data. Fortunately, Aion has the best logging system I've ever seen in a Korean game: it traces literally every step and every action of the player. Data was queried for the first 10 levels, or about 10 gameplay hours, capturing more than 50 percent of all early churners.

It took two Dual Xeon E5630 blades with 32GB RAM, 10TB cold and 3TB hot storage RAID10 SAS units. Both blades were running MS SQL 2008R2 -- one as a data warehouse and the other for MS Analysis Services. Only the standard Microsoft BI software stack was used.

Phase 1. I Know Everything!

Having vast experience as a game designer, with over 100 playtests under my belt, I was confident that my expertise would yield all the answers about churn. A player fails to learn how to teleport around the world -- he quits. The first mob encountered delivers the fatal blow -- she quits. Missed the "Missions" tab and wondered what to do next -- a possible quit, also. Aion is visually stunning and has superb technology, but it's not the friendliest game for new players.

So I put on my "average player" hat and played Aion's trial period for both races with several classes, meticulously noting gameplay issues, forming a preliminary hypothesis list explaining the roots of churn:

  • Race and Class. I assumed it would be the main factor, as the gameplay for the support-oriented priest radically differs from the powerful mage, influencing player enjoyment.
  • Has the player tried any other Innova games? (We have a single account)
  • How many characters of what races and classes have been tried?
  • Deaths, both per level and total, during the trial
  • Grouping with other players (high level and low level)
  • Mail received and guilds joined (signs of a "twink" account run by a seasoned player)
  • Quests completed, per level and total
  • Variety of skills used in combat

The list was impressive and detailed, describing numerous many ways to divert the player from the game.

So let's start the party. The first hypothesis went to data mining models. The idea is very simple: we predict the Boolean flag is leaver, which tells whether the player will leave today or keep enjoying the game at least for a while:

Lift Chart 101: The bottom straight line is a simple random guess. The upper skyrocketing line is The Transcendent One, who definitely knows the future. Between them is the pulsing thin line, representing our data-mining model. The closer our line is to The One, the better the prediction power is. This particular chart is for level 7 players, but the picture was the same for levels 2 to 9.

Fatality! Our first model barely beasts a coin toss as a method of predicting the future. Now it's time to pump other hypotheses into the mining structure, process them, and cross our fingers:

Well, it looks better, but still, precision is just a bit over 50 percent, and the false positives rate is enormous, at 28 percent.

Precision and Recall 101: The higher the precision, the more true leavers the model detects. False positives are those players predicted as churners, when in reality they aren't.

Phase 1 Result: All my initial ideas failed. Total disaster!


Article Start Page 1 of 3 Next

Related Jobs

Pocket Gems
Pocket Gems — San Francisco, California, United States
[10.01.14]

Associate Product Manager
Trendy Entertainment
Trendy Entertainment — Gainesville, Florida, United States
[10.01.14]

Technical Director
Trendy Entertainment
Trendy Entertainment — Gainesville, Florida, United States
[10.01.14]

Technical Director
Trendy Entertainment
Trendy Entertainment — Gainesville, Florida, United States
[10.01.14]

Technical Director






Comments


Kostas Yiatilis
profile image
Even though you haven't determined why the players are leaving yet, the results are impressive.

Good job!

I think you'll need to send a questionaire to at least get some input from the players, which might give you some hints at what to look at.

http://www.gamasutra.com/view/feature/170332/finding_out_what_the
y_think_a_.phpK.

Aleksander Kline
profile image
Interesting read. I like the idea of using human understandable models, (presumably) with the hope of working back from the metrics to directly improve the game.

I did have trouble understanding the 'Recall' row in your last table, showing extremely low percentages. I'm thinking it's a typo, considering your high true positives and low false negatives would imply a very high recall percentage. Any comment on what's up with it?

Regarding the inability to determine much from the metrics - is it possible your metrics are simply hard to interpret? Playing for about 3 hours to get to level 7 is strong indicator, but getting more players to do that isn't much of a design / development guideline unfortunately. From what I gathered, your intent was to use more actionable metrics, but they in turn weren't predictive enough?

Jason Carter
profile image
One thing I think a lot of games miss is the aesthetics. Most games are simply overwhelming with GUIs and everything and too detailed graphics. Honestly? The Cartoony style graphics of WOW were much nicer to look at with all their bright flashy colors than SWToR and RIFT. Not that I don't like Rift and ToR, but as far as graphics goes, WoW draws you in MUCH quicker because it's pretty. Bright Colors, simple environments, interesting monsters.

I think we try too much to make games look like Real Life. Especially in MMOs.

Simon Ludgate
profile image
Although you don't yet know WHY people leave, I think metrics can start you out on at least a general path. For example, you state "Experienced players use all skills available and their auto-attack percentage will be lower" so your metric of auto-attacks can give you a rough guess about whether it's an experienced player who might be leaving because this particular experience doesn't meet their expectations, versus a new player who might be leaving because they can't figure out what this game experience is supposed to be like.

Consider collecting metrics about client customization. Does the player remap keys or reconfigure the User Interface? If the first thing a player does when they play a game is move elements around, add more action bars, and remap tons of keys, it's a good bet they're trying to replicate a game experience they had in another MMO. If they then quit, there's a good chance they felt they couldn't successfully replicate that experience. What incentives you could you offer them to come back? Not much, since you might have to redesign the core game experience to win them over.

Or, consider collecting metrics about inventory usage. If a player spends a lot of time with their inventory window open, rearranges items in their inventory a lot, and tends not to sell any item that might possibly have any use at all, that player might be a pack-rat. If they spend most of their game time with very few inventory slots clear and frequently fill their inventory and get "no spaces available" error messages, they might be quitting because they want to pack more stuff into their pockets. In this case, inventory expansion offers might be a successful incentive to retaining that player.

You could even try to dig into social aspects by mining chat logs. See if the player types into local or global chat. See if they type the word "help" or how many of their chats contain a question mark. See how often they receive tells after typing in chat and how often they reply. The player might be having a difficult time learning the game and might be turning to other players for assistance, and might not be receiving any help from them. By taking an active role in social networking and trying to connect these new players with helpful mentors, you might be more successful in retaining them as players.

Metrics are valuable, but they only work so far as the person who is collecting the set also understands the dynamics of the game. I don't think you can use metrics to FIND why people are leaving, but you can use metrics to test hypotheses and develop response options.

Dmitry Nozhnin
profile image
Simon, excellent ideas, unfortunately to test them I'd have to relocate to Seoul and work for NCSoft :) Client-side metrics are not supported in Aion, and in any MMO I have worked with. If it would be possible to track the UI - oh my, that would yield tons of useful data. The first thing I'll be willing to track are wasted clicks - on objects and UI elements that are non-interactive.

As for the chat activities, we actually take that into account. Probably in the next article on segmentation it will be covered more in details.

Dmitry Nozhnin
profile image
We've actually held playtests before and after the research, and still the silver bullet remains elusive. There are alot of complexities and nuances on the player's path but none of them seems to be the show stopper. My current guess, even after digging the issue for almost a year, is that churn reasons are deeply intertwined with psychology, aesthetics and product features. We won't be able to find out an "formula of churn". However what we can do and actually doing at Innova is to detect that player is about to churn out and to incentivize him to stay for a bit longer by offering guides, advice and gifts.

Aleksander, thanks for pointing out the recall thing. The original article was too much infested with statistics jargon, I had to clean it up and to insert those 101's. My bad with the last table, it should read "false positives" instead of "recall".

We've used over 60 metrics, most of those were actually game-related, like:
- How many deaths per level, per hour
- Skill books bought/used
- Skills usage in combat
- Teleport usage and miles walked
- Time spend in low-level locations
- Resurrections and bindings to resurrection point
- Consumables usage

While surprisingly those contribute not so much to churn predictions, they give me tons of data about individual player's performance and allow to make customized and personalized email guide.

Dean Gebert
profile image
Great read and thanks for sharing, Dmitry.

Do you know if NCSoft tried sending an email, asking (reminding) users to come back, within the first 7 days? We're about to start testing it at Face The Fans, but we're only 4 weeks into beta, so it's still a small data set (users and time).

Dmitry Nozhnin
profile image
Thanks Dean,

I'm not aware of NC's activities, but here at Innova we send personalized email with hints to our new players to help them start playing.

Jorge Diaz
profile image
I really enjoyed how you walked us through the process and told an interesting story at the same time. I have not worked in MMO's but have always been interested in the subject of data mining the stats for trends and information. Great job!

Suhel M
profile image
Great article, I really enjoyed it.

You mentioned that you read two books, but only mentioned Microsoft SQL Server book. What was the other book that helped you on this journey?

Dmitry Nozhnin
profile image
Thanks, here's the other one: http://www.amazon.com/Programming-Collective-Intelligence-Buildin
g-Applications/dp/0596529325

Kenneth Posey
profile image
This is a very interesting article, and it coincides with my reading Harrington's Machine Learning in Action, but I feel like you missed a vital data point.

Knowing *where* customers come from is probably just as important as knowing what they do in the first few minutes of the game. It's quite possible that a player from WoW or Eve Online would feel perfectly at home in Aion and last for several weeks, but a failed advertising campaign to farmville players would have a near 100% churn rate.

Dmitry Nozhnin
profile image
That data is nearly impossible to collect regarding the WoW/Eve background. As for the traffic sources - yes, in the latest version of the prediction system we've separated modelling for different channels.

Bart Stewart
profile image
Great description of learning the data-mining process, and a fun look at gamer behavior.

It's trivial compared to the latter levels, but there's a simple predictor for the first level or phase of any free-to-play game: assume they'll quit. In the absence of intrinsic (to the game) data that can be mined, you have to look at gamers themselves. And the simple observation is that most people don't stick with new entertainment, period. Once they do, there are things you can do to persuade them to continue. But you can't expect that your product is capable of grabbing a large majority of people who try it out with no sunk cost perception (e.g., shelling out the equivalent of $60 for a single-player game).

The second point concerns "why." What is the silver bullet metric that explains why so many new players don't get hooked enough to keep playing? My suggestion: there is no such factor.

As noted above, there are intrinsic (game features) and extrinsic (player features) elements that go into how people react to entertainment. And by far the more important of these where first impressions are concerned are extrinsic to the product. I believe the #1 calculation, made within minutes or even seconds, is "does this look like the kind of thing I know I like?" If the answer is not at least "probably," that person will leave, and no incentive will bring them back.

Note that, for a game, you can't data mine this from player behavior. The problem is not that the player tries some feature -- a recordable event -- and doesn't like it. The problem is that he does not see something he does enjoy, something he'd pay to keep doing but can't *because it's not in the game at all*. A gamer who comes to a game enjoying cooperative pet training and trading, but whose initial play experience in that game is melee combat, is likely to quit without leaving any kind of measurable footprint that explains why.

This doesn't mean there's anything wrong with that game, nor does it mean there's anything wrong with that player. It only means there's not a match between what kind of product that is (based on its immediately observable features) and what kind of entertainment that individual prefers. Short of trying to design a game that is all things to all people (which Joss Whedon would have to fund with his Avengers money), the "loss" of the majority of consumers is an extrinsics-dependent effect that can't be changed with game-intrinsic incentives. It's not really a loss; they were never going to play that game. So the logical conclusion would be to accept this and focus instead on making the greatest possible product for the people who are already naturally disposed to like that kind of thing.

Incidentally, a lot of this (and the mining techniques) also apply to TV shows. If you're predisposed to like that kind of show, and something about it uniquely grabs you in the first few minutes of trying an episode, you're likely to keep watching. But if either of those conditions is not true, then you're gone.

TV people know this, and have gotten really good about fanservice that retains the people who made the show successful. There's a useful lesson there for game developers, I think...

Nick Lim
profile image
Dmitry, great article and introduction to the topic of advanced predictive analytics for games.

These techniques have been used in telecoms and finance greatly to reduce churn with proactive marketing. They usually hire a few statisticians and IT administrators, and pay for expensive licenses from SAS and SPSS (now IBM). What's interesting is that telecoms and finance can support these types of advanced analytics because the ARPPU / month is very high. As such saving a few paying customers justifies the cost of the effort.

I'm curious to learn if there has been some ROI calculations into whether these churn predictions were worth doing. One example would be to take the revenue of "saved" players divided by the cost (people time + systems). Can you comment on that?

Again, great intro.


none
 
Comment: