Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
February 28, 2020
arrowPress Releases







If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

League of Legends - The Potential of A/B Testing

by Eduardo Cabrera on 10/20/14 02:12:00 pm

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Who am I?

My name is Eduardo Cabrera and I’m a jack of all trades, focused on programming and design at the moment, I recently got into the game industry. I graduated from Full Sail University’s Game Development program in 2012. My first job was as an associate programmer in a mobile game development company based in Dallas. I’m currently pursuing my Master’s degree in Game Design.

I’ve been in eSports for almost 2 years now and this paper is the combination of my experience and my knowledge applied towards what I’m passionate about.

Contents

  1. Introduction and Definition
  2. How it’s used in Mobile Games
  3. Examples using League of Legends
  4. How it can help eSports
  5. Pros and Cons

Introduction and Definition

The purpose of this entry is to focus on the properties of A/B testing and how it can impact the game League of Legends by Riot Games and the eSports scene. First we must define A/B Testing; it’s a methodology used to test two or more things at the same time in a software or game environment. An A feature is presented to a sample of users while B is presented to another sample, metrics are then gathered on both samples and a conclusion is drawn on which one is a better choice.

How it’s used in Mobile Games.

Most mobile games are live services, meaning they receive constant updates or maintenance to their content. They get updates as often as every 2 weeks; this timeframe varies depending on the game and the content being released. This requires very short sprints to deliver updates polished on time. Although the primary focus of most mobile games is revenue, user happiness and acquisition is always a high priority. Most mobile game companies don’t have the fan base League of Legends has so they rely a lot on A/B Testing and metrics to determine what the optimal choice or feature implementation is. A good example would be player rewards. Imagine you reward the player over time for spending time in your game, but you’re not sure how much to reward the payer with. So you set up an A/B test with both cases and when users update their game they get one of the cases. From this point on you track the user and how much time they spent on the game as well as how much they spend or even if they ever come back to play the game. From the data gathered you can (in most cases) reach a validx conclusion backed up by statistical data on which case is the most beneficial.

Examples using League of Legends

Here we have Sivir’s visual update. Naturally some new skins will be shipped with the upgrade just because Riot loves their community. The only problem is Riot made such awesome skins they can’t decide which to ship. Riot then decides to head over to their PBE (Public Beta Environment) and release both skins! Now, the only catch as a player is that you can only get one skin out of both, and this is either random or a choice. Through metrics, riot tracks that 90% of the players picked the B skin. Riot then releases the skin on the live servers with Sivir’s visual upgrade and all summoners are happy.

Now the example above might be a very simplified version of the potential of this methodology. In actuality this could be used to improve game balancing and even speed up champion reworks and new champion testing as well.

The core steps of a/b testing are:

  1. Develop N versions of a page. (N being the number of cases you have)
  2. Randomly give one of the cases to the users out of all possible cases.
  3. Track the user’s response.
  4. Evaluate.
  5. Use the optimal case.

How it can help eSports

ESports is rapidly growing, and while this is very good for League of Legends it’s also unprecedented. There has been no game so far that has risen to the heights League of Legends is at right now. However, handling growth is one of Riot specialties as they have managed to cater to all the game’s balance needs as well as content needs.

Now, in an ever changing environment speed and precision are both huge factors in iterating through features and balancing gameplay. The faster the game is balanced the healthier it is for competitive play. A/B testing allows for these changes to happen quickly. Riot also happens to have the perfect platform for testing champions and features, and that is the PBE (Public Beta Environment). In the PBE players can be split up into N groups for focus testing for certain features. Players should always stay in the groups they were assigned to originally during the time duration of the test. This is to get clear feedback on the player’s feedback (data or verbal) on the tested feature.

Pros and Cons

Pros:

  • Results of the tests provide clear and conclusive evidence
  • Allows the developer to test new and innovative ideas and compare them between each other
  • The  developer will be able to optimize the game based on results provided by actual large scale feature testing
  • Results can provide answers to intricate design questions with statistical validity provided by test results

Cons:

  • Testing can take up a lot of time and resources. More so during set up and planning phases
  • Testing might not be the best approach to test certain kind of features. Proper planning is required to determine if this is the best approach for feature tests
  • A/B Testing is designed to test working features; They won’t have any impact on testing features that are inherently flawed
  • Potential recurring testing. Features testing might develop into more features testing and so on. Perfection is always an inch beyond grasp; Embrace the perfect imperfections that make the game unique

PS. This is my first blog post entry so feedback is greatly appreciated!


Related Jobs

Remedy Entertainment
Remedy Entertainment — Espoo, Finland
[02.28.20]

Art Outsourcing Manager
Remedy Entertainment
Remedy Entertainment — Espoo, Finland
[02.28.20]

Lead Producer
Square Enix Co., Ltd.
Square Enix Co., Ltd. — Tokyo, Japan
[02.25.20]

Experienced Game Developer
Insomniac Games
Insomniac Games — Burbank, California, United States
[02.25.20]

Lead Project Manager





Loading Comments

loader image