Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
Finding Out What They Think: A Rough Primer To User Research, Part 2
View All     RSS
May 26, 2019
arrowPress Releases
May 26, 2019
Games Press
View All     RSS








If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

Finding Out What They Think: A Rough Primer To User Research, Part 2


May 15, 2012 Article Start Previous Page 2 of 4 Next
 

Observational Studies

Observational studies (also sometimes called Behavioral Observation or Ethnographic Observation) are where you get some people to play your game and either record them doing so to watch later, or observe them directly. This technique is probably the most valuable of the traditional user research testing methods because nothing beats actually watching someone play your game. You can see how they progress, how they deal with challenges, and where they appear to become stuck or unmotivated.

You can also attempt to pay attention to their body posture and expression in order to try and gauge their emotions. Be careful, however, and only write down what you actually see -- don't try and infer any inner thought processes. Also the observer or facilitator should be as neutral and hands-off as possible. You want to see where people get stuck or make mistakes, so don't help them unless you really, really have to.

Observation can be done in large groups if you have the correct lab space. But it is probably better done on a one-on-one basis so that players do not distract each other and the observer is also not distracted.

Again, there are several things to remember when setting up an observational study, and I will try and go through some of the steps below:

Step 1 - Designing the Observation Session

Work out where you will be testing, if you will be recording the session or not, and what part or parts of the game you will be testing. It would be ideal if you could test exhaustively, however with modern non-linear and emergent games this is obviously difficult. Still, you should plan to test at the very least large representative chunks of your game, and certainly make sure that the core mechanics are fun.

One useful thing to do when running an observational type test is to create and use test scripts. Test scripts, like a movie script, or the recipe for a delicious cake, lay out how the test will go and typically include the order of events, what the events are, and even scripted dialog for the tester to use.

Be specific with this. What happens when the user comes into the room? Do you offer them coffee? Where do they sit? Do you tell them where the bathroom and emergency exits are? Do you talk to them at any time during the test, or only at set points? What build do you load up? In what order to you present the game portions you are looking at?

This may sound overly scientific, and in fact some argue that it is a bad way to run research, as it scares the player and gets them in a mindset that makes them feel tested. This is a valid point. At the same time, however, if you want the best data, it is good to have at least some consistency, and certainly you need to have some plan for the session. Good researchers can do this and still not come across as a robot.

Ultimately, it is no good if at the start of the day you welcome a user happily, chat about their ride over to the lab as you offer them a coffee, and then later in the day you just rush the next user into the room and try and get things done as fast as possible. Then you may find that it is not your game they are rating, but you.

A good test script should include welcoming the participants, administrative stuff, and some introduction to the procedure (what will happen, how long it will take, etc.) Remember, again, at this point it is good to be casual and relaxed to make it clear that the game is being evaluated, not the player.

The script should also clearly describe the tasks they are going to be doing during the observation session, this can be as simple as "start the level and then get to the next level" but it is important that they have clear defined start and end points. This is because you want to know exactly when in this process certain issues or behaviors occurred.

You should also think about what kind of behaviors (in the game, and perhaps outside of it, depending on the game) that you expect to see, just so you know what to look out for. It can be helpful if many people are observing to agree on a clear sentence (or two) that outlines the behaviors you are observing -- so what exactly does a double jump look like? What about a failed jump? A list of such agreed-upon descriptions of behaviors is called an ethogram, and it helps to cut down on confusion when discussing what was observed with others.

There are quite a few software packages out there aimed at helping out at this point (for example Morae or Observer, although there are many others) which can handle the recording of the screen and the participant, the timing of the test and also allow you to carry out quick predefined coding as you observe (or when you go back later to watch recorded video). This said, if you can't afford to record and use fancy software then having a nice well-defined ethogram and somewhere to write down observations (sit somewhere the user can't see you -- even if that is sitting behind them) can still produce useful results.

Once you have everything ready to go, you should test out the setup you have to make sure everything runs smoothly and the instructions are clear. This is also important as the only way to become a good observer is get experience observing.

Finally before you can start you need to look into recruiting people to test. I covered this in part 1, but just to be clear, the most important thing at this point is that you want representative users. It's no good testing it with the dudebro in the dorm next-door if your target market is young girls five to eight years of age. And if you are going to be testing with children you are opening up whole other box of issues (parental permission, testing in groups vs. alone, communication, and so on).


Article Start Previous Page 2 of 4 Next

Related Jobs

Digital Extremes Ltd.
Digital Extremes Ltd. — London, Ontario, Canada
[05.26.19]

Senior Lighting Artist
Digital Extremes Ltd.
Digital Extremes Ltd. — London, Ontario, Canada
[05.26.19]

Senior UI Artist
Digital Extremes Ltd.
Digital Extremes Ltd. — London, Ontario, Canada
[05.26.19]

Environment Artist
Gear Inc.
Gear Inc. — Hanoi, Vietnam
[05.25.19]

Technical Director





Loading Comments

loader image