Predicting Churn: Data-Mining Your Game
May 17, 2012 Page 1 of 3
The sad truth about all online services and games? The most significant churn occurs right the first minutes and hours of gameplay. The issue has been already explored in a numerous ways, with many profound hypothesizes related to usability and simplicity of interface, availability of a free trial, learning curve, and tutorial quality. All of these factors are considered to be very important.
We set a goal to investigate why new players depart so early and to try to predict which players are about to churn out. For our case study, we used the MMORPG Aion, but surprisingly, the results appear to be applicable to a wide array of services and games. Although Aion, at the time this study was undertaken, was a purely subscription-based game with a seven-day free trial capped at level 20, the vast majority of churners left long before they had to pay to keep playing. Our research was about in-game triggers for churn.
Behavioral studies show that casual players have a limited attention span. They might leave the game today, and tomorrow won't even recall that it was ever installed and played. If a player left the game, we have to act immediately to get her back.
But how can we differentiate players who churned out of the game from casual players who just have plans for an evening and won't log in for a while? The ideal way would be predicting the churn probability when the player is still in the game -- even before she actually thinks about quitting the game.
Our goal was more realistic: to predict new players' churn the day they logged in for the last time. We define churn as inactivity for 7 days, and the goal was not to wait for a whole week to be sure the player has left the game and won't return, but instead to predict the churn right on their last day of play. We'd like to predict the future!
The Tech Side
We had tons of data. Fortunately, Aion has the best logging system I've ever seen in a Korean game: it traces literally every step and every action of the player. Data was queried for the first 10 levels, or about 10 gameplay hours, capturing more than 50 percent of all early churners.
It took two Dual Xeon E5630 blades with 32GB RAM, 10TB cold and 3TB hot storage RAID10 SAS units. Both blades were running MS SQL 2008R2 -- one as a data warehouse and the other for MS Analysis Services. Only the standard Microsoft BI software stack was used.
Phase 1. I Know Everything!
Having vast experience as a game designer, with over 100 playtests under my belt, I was confident that my expertise would yield all the answers about churn. A player fails to learn how to teleport around the world -- he quits. The first mob encountered delivers the fatal blow -- she quits. Missed the "Missions" tab and wondered what to do next -- a possible quit, also. Aion is visually stunning and has superb technology, but it's not the friendliest game for new players.
So I put on my "average player" hat and played Aion's trial period for both races with several classes, meticulously noting gameplay issues, forming a preliminary hypothesis list explaining the roots of churn:
- Race and Class. I assumed it would be the main factor, as the gameplay for the support-oriented priest radically differs from the powerful mage, influencing player enjoyment.
- Has the player tried any other Innova games? (We have a single account)
- How many characters of what races and classes have been tried?
- Deaths, both per level and total, during the trial
- Grouping with other players (high level and low level)
- Mail received and guilds joined (signs of a "twink" account run by a seasoned player)
- Quests completed, per level and total
- Variety of skills used in combat
The list was impressive and detailed, describing numerous many ways to divert the player from the game.
So let's start the party. The first hypothesis went to data mining models. The idea is very simple: we predict the Boolean flag is leaver, which tells whether the player will leave today or keep enjoying the game at least for a while:
Lift Chart 101: The bottom straight line is a simple random guess. The upper skyrocketing line is The Transcendent One, who definitely knows the future. Between them is the pulsing thin line, representing our data-mining model. The closer our line is to The One, the better the prediction power is. This particular chart is for level 7 players, but the picture was the same for levels 2 to 9.
Fatality! Our first model barely beasts a coin toss as a method of predicting the future. Now it's time to pump other hypotheses into the mining structure, process them, and cross our fingers:
Well, it looks better, but still, precision is just a bit over 50 percent, and the false positives rate is enormous, at 28 percent.
Precision and Recall 101: The higher the precision, the more true leavers the model detects. False positives are those players predicted as churners, when in reality they aren't.
Phase 1 Result: All my initial ideas failed. Total disaster!
Page 1 of 3