This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.
Step 2 - Running the Observation Sessions
What do you do once the session is running? Well, observe, of course. Watch the participants and take notes on what they do in the game, in real life, and what they say. As mentioned above, there is software that can help you do this, and certainly recording the play session will help you later if you can go back and review it. Plus the recordings can be great material to show other folks who might need convincing that something needs to change.
Be careful to note not only what the player is doing, but also the time at which they do it, and the situation in which it occurs. This context is important for interpreting the data.
Also when observing, write down only what you see and hear. So, write: "the player is laughing at the cutscene and smiling" not "the player is happy" as this latter description makes assumptions about the user's internal states.
Again this may seems overly scientific, but in the long run the fewer assumptions you make, the better, and it helps when communicating with others what you found if you stick to clear descriptions of behavior.
In addition to the behaviors you see, you should also record down performance measures from in the game. In other words, you should note things like the time taken to clear a particular stage, the in-game score, the amount of damage taken and ammo used, and so on. This is especially important if you are not tracking these variables through a gameplay metrics system.
Finally, I should note that during this time it is good if the player is isolated from others (including you) and can concentrate on the game. This can be as advanced as an observation room at Microsoft or just you simply sitting in the same room out of the player's line of sight.
The important thing is that you let the player play without feedback from you. This can be frustrating when you see someone getting stuck in a place where the solution is "obvious", but this is exactly the type of thing you want to see. This means that you shouldn't really talk to the player once the session has begun, and you should politely ask them not to talk to you either. Having them wear headphones can help as it adds to the sense of a self-contained environment, and can help put the player in the right frame of mind.
This doesn't mean you can't step in and ask a question, or help if things are going really wrong. However, generally speaking, it is better to note down periods where you are not really sure what was going on and then ask the player about them later, after the test is done.
Step 3 - End of the Session
This part is pretty straight forward. Thank the player for taking part, and perhaps take the opportunity to give them a final questionnaire or even conduct a quick interview with them. This interview/debrief gives you the chance to ask about any interesting observations you made and try and get some feedback on what was going on. Again, be careful to do this in a non-confrontational and neutral manner; don't jump down their throat about how they should have just used the jetpack to get over that hole they kept falling in during level 3.
Finally, if you are running multiple sessions, it is usually advisable to give yourself about 15 to 30 minutes at this point to reset the room, and gather your thoughts before the next player/group of players. Observing for long periods can be very draining, so these little breaks can help recharge your batteries a little, and also allow you to think about and perhaps review the observations you have made so far.
Another type of observation study is a contextual inquiry. This is essentially fieldwork where you play David Attenborough and go out and observe your players in their natural habitat. The idea being that you can observe them in their normal play setting going about their normal routines when using your game, and therefore get insights that may not be available in your own test situation.
In this case the test scripts I mentioned above are somewhat less useful, as the point here is to see how players use your game in their normal routine. However being prepared in terms of what you will say when you show up and how you will carry out the observation is still important.
Contextual inquiries also tend to be more useful for improving existing games, or observing similar games to yours to learn about how people play them in the real world. As such, testing prototypes or games in development in this fashion can be less useful, as players haven't had an opportunity to develop a routine with them. Still, it is certainly nice to see how your game will be used (and abused) in the wild.
That is all I will cover as far as observations go (with the exception of think out loud, which I will discuss in a bit). The biggest advantage of observation studies is that you really get to see what new players do when playing your game. This is invaluable and makes observation studies the best methodology, in my opinion. This method can also give you unexpected insight, and reveal issues that you may have never seen without it.
However, observations are obviously quite time-consuming, especially if you are recording everything. Also, it can take quite some work to be a good observer. As I already said, a good observer has to be as neutral and non-intrusive as possible. But also they have to know what to look for, and when to ask follow-up questions.
What typically helps is first practicing, and making a list of typical behaviors you would expect to see. Then you know what to look for before you go in to a real session. Remember you want to primarily be looking for their behavior in the game, so think about the types of things they will be doing. So for example do you notice them jumping repeatedly in one place? Or running into walls?
Also, avoid coloring what you see with your own expectations. Don't try to infer internal states; just write down what you observe, and then if you want to know what was going on in their head, ask them later.