Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
Finding Out What They Think: A Rough Primer To User Research, Part 2
View All     RSS
October 14, 2019
arrowPress Releases
October 14, 2019
Games Press
View All     RSS







If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

Finding Out What They Think: A Rough Primer To User Research, Part 2


May 15, 2012 Article Start Previous Page 4 of 4
 

Think Out Loud Protocol

An expansion of the observation method, this is where you carry out observations as detailed above, but ask the players to talk about what they are thinking as they go along. You then record this while observing what they are doing. This adds information about the internal states of the player, and gives you an idea of what assumptions they are making based on the game. Again, you may hear things that are wrong, or that you don't like, but resist the urge to correct the player.

This method can lead to unexpected insights that wouldn't be gained otherwise, or at the very least gives you a little more insight into how people are mentally interacting with your game. However, it does have one big downside, and that is that it is quite unnatural to talk about what you are doing and feeling as you do it. This might change how people will actually play the game and affect how much or little fun they are having.

Related to this, you may find that some people talk non-stop, whereas others hardly say a word, a finding that may reflect more the differences in people than in your game.

Pros

  • Allows for the collection of objective (behavioral) data while also getting access what players are thinking and feeling
  • Can result in unexpected insights

Cons

  • Time-consuming
  • Unnatural
  • What people say is still subjective

Gameplay Metrics

Similar to, and a great addition to, observational methods are gameplay metrics. This is basically is statistics generated by the game itself. which tell you about what the players did. For example think of the statistics systems that Bungie and 343 Studios have for Halo, or the Autolog system in Need for Speed.

Such gameplay metrics give you hard, objective, statistical facts about how the game plays. The collection of this data has to be built into the game, but once it is, you have a very large data source at your fingertips. Want to know what weapons take down players most on a map? What power-ups they use? Where they tend to die? Gameplay metrics make that possible. Good metric triggers and hooks can also really compliment observations as they take some of the load off you in terms of recording the in-game details about the player's performance.


The power of metrics: on the left is where people are standing when they make kills with a weapon and on the right is deaths by this weapon in Halo Reach. With just a basic knowledge of FPS games, you can still probably work exactly what kind of weapon this is and where the elevated and the open spaces are in the level.

However, metrics can be time-consuming to collect and you need a good number of players to really take advantage of them. Plus metrics lack subjective information about emotion and what is going on with the players -- it is just a mass of data.

Also, you can end up with data overload if you don't analyze what you collect carefully. For example, load up any heat map in Halo Reach and turn on deaths by all weapons, and compare it to kills by all weapons. The result will generally give you a little information in that the kills and deaths seem pretty well spread out over the player area, but is not particularly useful for working out finer details of the player experience.

Pros

  • Objective data that can be collected remotely and discretely
  • Can be used to see trends and provide data that supports other methods
  • Allows for continuous data recording without interrupting the player

Cons

  • Time-consuming
  • No subjective feedback
  • Needs large sample sizes to get meaningful data
  • However, at the same time, you can get too much data

Biometrics

I will finish up by covering biometrics. I have already done a primer on this topic, so I am not going to go into too much detail here. Biometrics are basically when you record signals from the body, such as brain waves via EEG, or where people are looking via eye tracking, to try and give us a clue as to your player's internal body states. In this way they are again like think out loud is, and like metrics can be: an add-on to observation-based testing.

Much like observation and gameplay metrics, biometric measures can be useful because they give you objective data on what is going on. They also allow for data on emotions and excitement to be collected continuously during play without having to stop and ask questions about how people are feeling.

However, the ways you can collect biometrics are currently quite invasive in that you have to wire people up, and they cost quite a bit of money and time to use, not to mention you have to be trained in how to use them -- as well as how to analyze the data that you get.

Also they have problems in that you can get artifacts or fake signals in your data, and that there is no real solid agreement on what you are measuring actually means. For example, heart rate may go up in a certain part of your game, which could mean it is exciting, but it could also mean it is scary and unpleasant, or that your player is so bored they are now thinking about last night in bed with their partner, or even just that the person you are recording took a deep breath (which also increases your heart rate).

The above issues mean that you have to combine biometric data with other sources of data (observations, interviews, questionnaires) in order to work out what is going on with the physiological signals you are collecting. This has led some people to doubt its usefulness, arguing that it doesn't add anything that you couldn't already get but does add a bunch of complications and extra work (see an excellent discussion on this point, and on games user research in general, by Bungie's John Hopson and Valve's Mike Ambinder here).

Proponents of biometrics, on the other hand, suggest that the strength of biometric data is that it gives you access to emotions and reactions that the players may not know they are having, and therefore helps you look at other data sources in a way you haven't before. Ultimately, however, it is just another tool and you will have to judge if it is appropriate for you or not.

Pros

  • Gives objective quantifiable data
  • Allows for continuous data recording without interrupting the player

Cons

  • Invasive
  • Costs a lot of time and money
  • Problems with specificity, artifacts, inference and validity can make it difficult to interpret

Final Words

As I stated in the introduction these two articles are intended as rough primers, and are in no way comprehensive. I am sure that others will disagree with what I say, and perhaps others will be happy to add more or provide material of their own. However, I do hope though that this document can be of some use.

If you are interested in further reading on this topic then I can recommend checking out this excellent document from Mike Ambinder at Valve and watching this discussion between Mike and John Hopson from Bungie. As mentioned in the last article, you may also consider checking out the IGDA Special Interest Group for Games User Research on LinkedIn.

Furthermore a wiki-based games user research primer is in the works. This primer will arise out of a workshop at the CHI 2012 conference and is planned to be an evolving wiki based site that can provide the community with up to date information on game research methodologies. So watch that space.

Finally, please remember that the above methods are here you help you develop your games. The data they give can be rich and interesting, but it should not be domineering. In the end, it is your creative vision that guides the game; the data that can be collected via games user research is there to help you, NOT limit you.


Article Start Previous Page 4 of 4

Related Jobs

Take Two Interactive Software Inc.
Take Two Interactive Software Inc. — Westwood , Massachusetts, United States
[10.11.19]

Producer
Take Two Interactive Software Inc.
Take Two Interactive Software Inc. — Westwood , Massachusetts, United States
[10.11.19]

Cinematic Narrative Scripter
Insomniac Games
Insomniac Games — Burbank, California, United States
[10.11.19]

Lead Level Designer
Insomniac Games
Insomniac Games — Burbank, California, United States
[10.11.19]

Lead Environment Artist





Loading Comments

loader image