Understanding User Research: It's Not QA or Marketing!
April 6, 2012 Page 2 of 3
The Methods, Tools and Techniques of Games User Research
According to Sorensen, a user test is, broadly speaking, a user (e.g., an actual representative from the potential target audience) interacting with product content while this interaction is tracked, analyzed, synthesized, and communicated in a "systematic way". In this context, "systematic way" refers to the method, tools and techniques used.
A "method" is a disciplined and structured approach to solve a problem or a challenge within a given context, for instance "usability testing to detect errors in how a game character moves in the world". A method has a certain "perspective", or in other words a theoretical foundation for understanding a phenomenon.
Finally, a method has practical guidelines for its application: "techniques", "tools", and "principles of organization". The techniques and tools are the practical instruments (surveys, observation techniques, interview techniques, data mining, eye tracking, etc) and how to apply them, and the principles of organization concern the division of labor, the project management model, the allocation of resources, and so on.
Note: different tools and techniques can be used in more than one method, and the difference between what constitutes a method or tool can be up for academic debate.
Even though all games user researchers do user testing (which is the recommended term for this activity, since neither developers nor user researchers are clear about what "playtesting" means exactly) there can be a huge difference in how the activity of user testing is carried out from studio to studio and publisher to publisher. This is caused by differences in the method, tools, and techniques used.
In turn, which methods, tools, and techniques are used depends on the time and budget available, as well as the background of the researcher, and the goals of the current user testing.
One group of researchers may prefer observations, video recordings, and questionnaires, using a small group of users (eight to 10) to get data on the key gameplay and UI issues very fast. Others may prefer group sessions for party games or for the multiplayer aspects of a game. It all depends on what problem the user researcher is trying to solve, but it is all games user research.
But What About Those Focus Groups and Focus Testing?
As a game developer you might think, "Ah, but I don't really care what you call it and how you do it. Just get me the results!" and that would be a great approach to take. The problems instead arise when developers ask, "Can we do some focus group testing on this part of our single-player game?" or "I've read some stuff about biometrics; can you do that for our open world RPG? By the way, the player can have up to 20 story and side-quests at any given time." Unfortunately such requests more often than not create opportunities for misunderstanding and wasted time rather than productive results.
The terms "focus group" or "focus tests" in particular seem to be often mentioned when user research is brought up, and it's not hard to understand why. They are terms with a long history and are methods of product evaluation that have been used in market research for decades. Therefore, since games are still products, why wouldn't you use such a focus group?
The problem with this is that in user research, a focus group is just one among a wealth of methods available, and not one that is often used, if ever, by a lot of games user research teams. To the outsider, the meaning of the term itself is as muddled as it can be.
Sure, it involves a group, but what is the focus? Is the focus on measuring qualitative attitudes towards the IP, on how much the users like the story or gameplay, if the game is experienced as fun or frustrating, or on something else? For all these aspects, there are other methods and tools that can offer better and more precise insights into the minds of users.
Page 2 of 3