Gamasutra - Feature - "Better Games Through Usability Evaluation and Testing"


				\|\|\|\|

By Sauli Laitinen
[Author's Bio]
Gamasutra
June 23, 2005

Introduction

Usability Testing

	Printer Friendly Version

	Change Login/Pwd Post A Job Post A Project Post Resume Post An Event Post A Contractor Post A Product Write An Article Get In Art Gallery Submit News

	Latest Letters to the Editor: Perpetual Layoffs by Alexander Brandon [09.21.2007] Casual friendliness in MMO's by Colby Poulson [09.20.2007] Scrum deals and 'What is Scrum?' by Tom Plunket [08.29.2007] [Submit Letter] [View All...]

	Upcoming Events: SPARK Animation Festival Vancouver, Canada 09.10.08 Women In Games Conference Coventry, United Kingdom 09.10.08 3rd ACM International Conference on Digital Interactive Media in Entertainment and Arts - DIMEA 2008 Athens, Greece 09.10.08 GDC Austin Austin, United States 09.15.08 Game Career Seminar Austin, United States 09.17.08 [Submit Event] [View All...]

[Enter Forums...]

Note: Discussion forums for Gamasutra are hosted by the IGDA, which is free to join.

Features

Better Games Through Usability Evaluation and Testing

Usability Testing

Usability testing is the most fundamental usability method. In a usability test, the game is tested with players that represent the target group of the game. Just like in an expert evaluation, the goal is to pinpoint the challenges in the game that were not intended by the game developers.

The reason why testing is important is that it provides direct and objective information about how real players play the game and what are the exact usability problems that players face when playing the game. This data is irreplaceable when developing the game and making it easy to use.

In a typical usability test 3-6 players from each target group of the game are recruited to come and play the game in a usability laboratory (see Figure 3). The players come to the laboratory one at a time and play the selected parts of the game for 1-2 hours.



	Figure 3. A usability laboratory consists of two rooms with a one-way mirror and a sound proof wall in-between. The user and the test instructor are in the laboratory room. The test is recorded in the observing room and the developers can follow test in the nearby meeting room.

In the laboratory, there is a test instructor together with the player. The role of the expert is to give tasks to the player and observe playing the game. The player is instructed to think aloud while playing. In practice this means that the player tells what s/he is doing while playing the game. Sometimes the usability expert may interrupt the player and ask questions about what the player was doing and why. The purpose of these questions is to get in-depth information about the problems and the reasons behind them. Game developers are also invited to come and observe the tests. The onscreen action and an overall view of the laboratory are displayed in a meeting room where the developers can follow the tests and discuss the findings with a usability expert.

After the tests, each recording is carefully analyzed to find all the usability problems and to understand the reasons behind them. Then the final report is created. In the final report, all the problems found are prioritized and solutions for the problems are provided.

Typically usability testing takes 2-3 weeks from the start to delivering the final report. Initial findings can be provided even earlier, and it is also possible to change the version tested between the players. This makes it possible to fix the problems during the testing and thus to verify the changes later on in the same test.

In the case of Shadowgrounds the usability test was conducted two weeks after the expert evaluation. The focus of the testing was broad, and all the aspects but the main menu of the game were tested. The testing was done with a version similar to the version used in the expert evaluation. Like mentioned earlier, some usability problems were fixed between the expert evaluation and the test. Next, examples of the problems found will be presented.

Results of the Usability Test

In the usability tests 97 usability problems were found. Finding less problems in usability testing than in expert evaluation is typical: players do not necessarily use every feature of the game, and some of the minor problems found in the evaluation do not necessarily show in the tests. One of the problems found was classified catastrophic, 26 severe, 28 intermediate, and 42 minor or cosmetic. This is illustrated in Figure 4.



	Figure 4. Classification of the usability problems found in the expert evaluation.

Next, five examples of problems classified as severe are presented. In addition to presenting the problems and their suggested solutions there is also game developer's comments after each problem.

Problem Picking up a new weapon went unnoticed.	Rating Severe
Description It happened twice in the usability tests that a user picked up a laser rifle but did not notice it before entering the weapon upgrade menu later on. As acquiring new weapons is an important reward in the game this should not go unnoticed.
Solution Give clear feedback when the user finds or acquires a new weapon.
Developers' comment There was feedback for picking up a new weapon - the information bar in bottom of the screen displayed a text message. In this case the feedback was not noticeable nor rewarding enough, and will be improved in the final game.

Problem The character moves too slowly.	Rating Severe
Description Two out of six users commented spontaneously that the character moves too slowly. The movement speed was considered especially problematic in open spaces. Inside the buildings the speed was considered to be good. See also the survey results.
Solution Consider implementing a running option.
Developers' comment This could be one of the biggest issues in games - the movement speed of the units/characters. In our case, the problem was mostly outdoors, which would suggest that the outdoor areas were not interesting by themselves, and the player wanted to get to the next interesting area (building). It will be interesting to see if this is the case, when the final game is tested

Problem The users did not understand how the radar works.	Rating Severe
Description The radar displays only the enemies that are moving. None of the users understood this. Most of the users thought that the enemies are shown in the radar when they are close enough. Many users also thought that the radar cannot see through walls. In reality, the radar does not care about obstacles.
Solution Inform the users about the radar and how it works in the tutorial of the game. Consider labelling the radar as motion sensor.
Developers' comment: The radar will be renamed to motion sensor (which it is). Apparently the aliens theme wasn't enough to make the players understand that the circular element in the top-right corner worked like a motion sensor, even if it had the familiar beeps.

Problem Finding doors is difficult.	Rating Severe
Description The users had difficulties in finding doors, and it happened often that the user thought that either a window or some other structure was a door.
Solution If the door is not hidden on purpose mark it clearly. Symbols and/or light effects should be considered. Change the shape of the windows and the buildings' support structures so that they are not mistaken as doors.
Developers' comment This is a tough problem, and hard to predict in the design phase. Additional lights could be added, and the buildings could be remodeled, but this takes a lot of effort. In the end, this problem seemed big enough to warrant changes, and each door was scrutinized individually to make sure it worked alright.

Problem Changing of the mission objective was not always noticed	Rating Severe
Description During the tests it happened a few times that the users did not notice that the mission objective had changed. This was because the users did not always pay close attention to the dialogue between the characters.
Solution Provide the users a clear notification about a new mission objective. It is not enough to inform the users about the new objective in the in-game dialogue.
Developers' comment Part of this problem was caused by unpolished dialogue (sometimes the texts went by too fast). English wasn't the users' mother tongue, which could have affected the outcome as well. The new mission objectives (and the target point) were presented in the map screen, but not all users had understood this very well (due to the lack of tutorial before playing the game). " Mission objectives updated" was displayed in the information bar at the bottom of the screen, and also as a icon on the right side of the screen, but the icon was not understood clearly. In the final game, all of these problems will be fixed.

Benefits of Usability Expert Evaluation and Testing

Usability expert evaluation and testing are systematic methodologies that provide information to support game development. Together they provide both the experts' view on usability and experimental data of usability problems in the game. The usefulness of the methods is illustrated by the following quotes from Joel Kinnunen, the development director of Frozenbyte:

On expert evaluation:

"Expert evaluation is a fast and effective way to check the usability of a game. In our case, the results arrived in a couple of weeks, and they helped us solve some major design issues. We were also able to fix numerous smaller usability problems, and avoid a couple of potential pitfalls in designing and implementing new features."

On usability testing:

"Usability testing provided us with a new perspective on the game. It is difficult to know how the game is played without testing it with the real users - gamers are not predictable, especially as it comes to navigating a given level. In hindsight, I wish more of the development team could have been present at the tests, so the endless amount of choices the player can make would be more clear to everyone on the team. Level designers would do well to study the various player behaviors."

On usability engineering:

"As a whole the usability expert evaluation and testing was a very positive endeavour. Frozenbyte will continue to use usability testing for its future games - starting right from the design document and lasting throughout the entire production phase. The impact of a smooth gaming experience cannot be overestimated."

One of the fundamental strengths of the expert evaluation and the testing is that they provide comprehensive and prioritized lists of problems in the game. This list, together with the suggested solutions, is a handy tool in focusing the development work especially when time and/or resources are limited.

Usability expert evaluation and testing also have other benefits, especially if the usability work is done by usability specialists external to the development team. They will save both time and effort of the game developers by organizing, conducting and reporting the usability work. External usability professionals are also likely to have fresh views on the design issues.

Acknowledgements

I would like to thank Frozenbyte for commenting the results and giving the permission to publish them. Many views on this article are based on the pioneering work done by the usability engineers at Microsoft. See http://www.mgsuserresearch.com/publications/ for more articles on the topic.

_____________________________________________________

[back to] Introduction