Machine Learning Agents in Gamedevelopment

Introduction of machine learning:

The application of machine learning in daily life covers a broad area. For example, we can use voice assistants on our phones, smart home devices, e-mail inbox, self-driving cars, customer-facing chatbots, predictive web searches, product-, music recommendations, and more. Through these interactions, machine learning has become increasingly more prominent in our daily lives.

This article introduces the use of machine learning in the entertainment industry, specifically video game development. A central theme in this article is that machine learning and AI games development tools should not be viewed as replacing staff and creative workers but instead as tools that facilitate staff and creative workers in performing their activities.

Video game development is a complex creative process with various intermediate steps to assure the desired quality. Therefore, a video game development project needs many interdisciplinary experts and specialists to reach the desired quality. An area of interest is using machine learning to create better tooling for these professionals.

The increasing computational power available allowed for further research into machine learning for use in grand scale games to be self-played. OpenAI's project based on Go used modern techniques such as distributed computing (spanning thousands of CPUs) and the power of graphics processor accelerated algorithms over several months [1]. The performance of their 'OpenAI Five' agent observes approximately 16,000 values each time step, each timestep their model chooses among 8,000 to 80,000 actions. They compare this to Chess, which requires around 1,000 values per observation. Further, compared with Go, which is approximately 6,000 values per observation. These value observations eventually became comparatively large enough to enable their agent, OpenAI Five, to defeat the world champions at an esports game, Defense of the Ancients 2. OpenAI has another agent that is called AlphaStar, designed to play StarCraft II. This agent has been tested over a series of games and managed to place itself on the Grand Master level across all three races and above 99.8% of officially ranked human players. Both games are examples of imperfect information games, which are still challenging to solve with Artificial General Intelligence.

Each of these advances constituted a significant breakthrough in technological prowess. Yet, at each milestone, humans have had to re-evaluate their assumptions about the upper boundaries of computation. There are also efforts to use machine learning to improve the player experience with various approaches, for example, automated-game balancing or -playtesting.

For example, Deep Q-Learning (DQN) has been successfully used by Electronic Arts to make the gameplay experience more enjoyable for players [2], and their research experimented with other algorithms as well. These experiments were motivated by the DeepMind research team [3]. These researchers show how important it is to pick a machine learning strategy depending on the game design.

Starting small

Furthermore, other researchers state it is "important to start small" [4]. Human intelligence shares this trait of time-based development of mastery of a given skill. Using a different approach to create a less computationally expensive workflow to create machine learning agents using self-play is already suggested. They call this approach 'end-game-first', which seeks to minimise the creation of expectedly poor-quality experiences [5].

Designing a proper 'shaping' (reward) function takes considerable human effort and can create difficulties if the reward function is not properly designed [6]. Starting small is key in other areas of the game development process, e.g. using paper prototypes or more robust prototypes that include a functional user interface with a computational implementation [7]. In addition, these prototypes allow for early game balancing and playtesting, which provide valuable insight on product feasibility based on its financial or artistic potential.

Game balancing and playtesting are inextricably tied together, with their definitions having some overlaps. As professionals, we have to ensure that the game we are making is engaging and enjoyable for our end-users (the players). Using machine learning for playtesting and game balancing, automated or not, can be beneficial for our industry.

Tool Maturity

The tools available to machine learning researchers and video game developers alike are becoming more mature, from complete integration of Jupyter notebooks within Visual Studio Code to integration of PyTorch in a game engine. Unity has a Machine Learning Toolkit package available for a while now, and it is a promising package for interested parties to experiment with.

The maturity of these tools will eventually lower the entry-level skill floor and raise the skill ceiling for machine learning professionals and game developers.

I am Thomas R. Schoonman, currently graduating from the Creative Media and Technology masters programme at the Breda University of Applied Science. My thesis on using machine learning agents in game development can be found here.

[1]      OpenAI et al., “Dota 2 with Large Scale Deep Reinforcement Learning,” 2019, [Online]. Available: http://arxiv.org/abs/1912.06680

[2]      Y. Zhao et al., “Winning Is Not Everything: Enhancing Game Development with Intelligent Agents,” IEEE Transactions on Games, vol. 12, no. 2, pp. 199–212, 2020, doi: 10.1109/TG.2020.2990865.

[3]      M. Hessel et al., “Rainbow: Combining improvements in deep reinforcement learning,” 32nd AAAI Conference on Artificial Intelligence, AAAI 2018, pp. 3215–3222, 2018.

[4]      S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, and P. Stone, “Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey,” vol. 21, pp. 1–50, 2020, [Online]. Available: http://arxiv.org/abs/2003.04960

[5]      J. West, F. Maire, C. Browne, and S. Denman, “Improved reinforcement learning with curriculum,” Expert Systems with Applications, vol. 158, no. X, pp. 1–11, 2020, doi: 10.1016/j.eswa.2020.113515.

[6]      C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel, “Reverse Curriculum Generation for Reinforcement Learning,” Jul. 2017, [Online]. Available: http://arxiv.org/abs/1707.05300

[7]      M. Trankov, E. Hadzhikolev, and S. Hadzhikoleva, “Testing of games through software agents managed by artificial neural networks,” in IOP Conference Series: Materials Science and Engineering, Jul. 2020, vol. 878, no. 1. doi: 10.1088/1757-899X/878/1/012031.