Understanding User Research: It's Not QA or Marketing!
April 6, 2012 Page 3 of 3
The article Beyond Focus Groups: Getting More Useful Feedback from Consumers by Bill Fulton, an important figure in games user research who started the user research group at Microsoft Game Studios (MGS, now Microsoft Studios), and Michael Medlock, Senior User Researcher at MGS, offers an excellent perspective on the use of focus groups.
Essentially, what focus groups are good for is generating ideas at a very early stage, and gauging a general response to a concept, but they require a highly trained moderator to yield good and usable subjective data that isn't overly tainted by group dynamics.
However, if what you are after is both subjective and objective data on player behavior using a playable build of your game, a usability test with, usually, a small number of participants, is more appropriate.
This allows the user researcher to observe and evaluate the player's actions and compare them to the player's statements; a trained observer can combine the two to pinpoint the reason for a player's unexpected behavior.
Focus groups aren't necessarily useless or evil -- they're simply one method used for a specific goal. This goal differs from the generally evaluative goals that user researchers try to attain.
On the subject of the confusion between user testing and focus groups that exists today, Fulton points out that "the term 'focus testing' is often used in the same way as 'user research' -- not to denote a specific method, but as an umbrella term for 'bringing in outsiders'."
As such, Fulton suggests that "most people are not limiting themselves to what we'd call 'focus groups' when they say 'focus testing' (although they do sometimes mean 'focus group')," which creates opportunities for confusion.
The experience of working with developers who don't know what to call user testing is shared by Rich Ridlen, senior games user researcher at EA: "Many of the dev teams that I work with at EA use the term 'focus group' or 'focus test' to mean any method wherein consumers are brought in to test," Ridlen notes.
"They understand, have observed, and see value in different methodologies (or tools), like one-on-ones, small group, high N survey-based tests, etc. They have even correctly ascribed methods to tests from working with us. However, it's all 'focus groups' and it doesn't matter how many times I correct them or that I never use that word unless that's specifically what I'm describing."
Okay, Just Tell Me How to Get the Solution to My Problem
Here is what we suggest: If you want something to be user tested, perhaps it is best to simply tell your user researcher that you want them to perform user testing on a certain aspect of the game.
When you ask for something specific like "do a focus group", then that team's response might be that there's no point in doing so, even though perhaps you just wanted something user tested that has nothing to do with running focus groups.
Your user research team will likely propose another method for testing what you want, and perhaps that counter proposal will make you go, "Yes, well, that's what I asked for, isn't it?" and eyes will be rolled on both sides, with ten minutes having been wasted on just communicating the goals.
So, whatever your role is (development, management, etc.), save yourself some trouble and remember that user testing is what user researchers do, and it is their job to explain to you how they will do it and for what reason given the problem area that you lay out for them. As such, it is best for you to describe what your concerns are, and the games user researcher can then design a test that can examine the validity of those concerns (just like the report from the user researcher only describes the problem areas and does not prescribe specific solutions).
As a final point, it should be noted that user researchers are ultimately here to help you realize the designer's vision, and as such the data they collect is not intended to dominate or limit you. Rather user research aims to simply provide additional information to help you in building a great game.
As an aside, if you are a user researcher (or just interested in the discipline) and haven't joined the GUR-SIG LinkedIn group yet, you can do so here. Joining allows you the opportunity to participate in discussions such as the one that led to this article, connect with others in the area, and share your insights into methodologies and novel approaches.
Page 3 of 3