[Blitz Games Studios studio development director Chris Viggers explains how user testing revealed flaws in player interaction with two of the studio's Kinect titles -- and how the flaws challenged the assumptions of the development team.]
It's popular, it's a buzzword, it's a growth industry and many developers are afraid of it, but what exactly is usability testing and how can it be used to create ever more compelling entertainment?
From a developer's perspective, at its core, usability testing is something that tells designers, artists, coders, animators and managers what they are doing wrong. It is human nature to try to avoid any sort of pain (mental or physical) and many of us struggle to accept criticism, even when that feedback could lead to a better end result.
In addition to the problems of essential human nature, the games industry is changing even more rapidly than before and our creations are being experienced by more and more people, and ever more diverse demographics, each one with their own expectations of how they want to be engaged while spending their precious time on our games.
As any producers reading this will testify, during development you are always acutely aware that you never have enough time, enough money, or enough resources to achieve what you really want to achieve with that game.
Adding yet another process into your production pipeline is going to be a difficult sell both to the team and to upper management, especially when that process is highly likely to result in rework to fix the problems that the players are flagging up.
However, the reality is that your game is going to be tested at the end whether you like it or not, and that testing will come in the form of a review or Metacritic score. If you ignore end user feedback and usability results during development, you run the very real risk of getting poor reviews and feedback that will publicly damage your game and reputation. Facing up to this fact and integrating this with your development pipeline is key to success.
Our studio design director, John Nash, has already talked here about how input devices for gaming have changed massively over the years and are continuing to do so. This, combined with the huge variety of new target audiences, means that we need to not only embrace player input at a much earlier stage in development, but also learn to adapt our work flow to accommodate this input.
In this article I will outline some practical examples of how we did this at Blitz Games Studios, in this case focusing more on boxed product development rather than direct-to-consumer development as the challenges are very different.
Usability testing is firmly rooted in scientific methodologies and relies on generating data from the user experience of the game that is independent of what the player actually thinks about the game (or rather, what they think they think about the game). It is about hard metrics, unarguable data and demonstrating how the player is actually playing the game.
It is not only the collection of data that is important; understanding its value is clearly critical to being able to maximize its effectiveness in development. The first thing to look at is how does the data you have collected stack up against your game's experience and the core pillars of your vision?
For example: was the user frustrated in a certain area, only to be elated when they finally solved the puzzle, and is that something that you actually want to maximize? Or were they scared during certain areas and moving slowly through the environment due to their anxiety, rather than just because they're bored or lost? I'm looking at you, Limbo and Dead Space!
To provide a concrete example of how specific user testing helped on one of our projects (Yoostar 2) last year, over six sessions of testing we found that more than 40 individual items required further attention to ensure that we maximized accessibility and experience for the player.
40 individual items identified and resolved for Yoostar 2
Even with all this data collected, the team did not just jump on each issue and attempt to fix it. Each area was assessed against the core pillars of the vision and what we wanted the player to feel during the experience.
Would this change result in the game being more accessible or more complicated, would any change in terminology be understood worldwide, or do any of these changes affect core aspects such as the timing of the game loop or the user's ability to get right to the action? These are all questions you need to ask yourself before amending any parts of the game, even with the most 'expert' feedback in front of you.
It is also worth remembering that although the data you get back is important, some of the results you get will point to the need for more generic solutions; in other words, some of the problems are caused by developing innovative features with which people are not already familiar. Assessing peoples' expectations is great, but surprising them with unique and innovative ideas is also something we should be striving to achieve.
Accord the usability results all due respect, but do not treat them as gospel: trust your instincts and your vision as well.