Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Gamasutra: The Art & Business of Making Gamesspacer
Finding Out What They Think: A Rough Primer To User Research, Part 1
View All     RSS
July 14, 2020
arrowPress Releases
July 14, 2020
Games Press
View All     RSS

If you enjoy reading this site, you might also want to check out these UBM Tech sites:


Finding Out What They Think: A Rough Primer To User Research, Part 1

April 24, 2012 Article Start Previous Page 4 of 4

Closed or open? Questions can be either open, as in the player can say whatever they like, or closed, where there are only a few limited options to select.

Open questions allow for richer data to be collected, as they let players give as much feedback as they want. However they can also give as little as they want, and often without direction, the answers may be vague.

One way to get around this if you do use open questions is to make sure you have time to read them before the end of the session, in case you want to ask for clarification on any points people bring up. However this is difficult if the questionnaire is being given remotely, such as when it is being used via email or online.

Closed questions on the other hand let you have much tighter control on the answers your participants give, and come in several flavors (scales):

  • Dichotomous: This is for simple yes/no questions. So usually it used to collect stuff like yes/no, true/false. It is nice, direct, and precise. However, it is also not very data rich. So generally speaking such questions are used just to collect demographics, or when you really want to force a choice between two options.

A dichotomous choice in Diablo III

  • Continuous: This is where you ask people to give ratings along a continuum or sliding scale (like when you are doing face customization in games). The big advantage of a continuous scale is that it gives a lot of resolution. However that much resolution is hardly ever needed (is there really a difference between a rating of 96.43 and a 93.21?).

A (semi) continuous scale in Saints Row: The Third

  • Interval: This is probably the type of scale we are most familiar with. Here, you give a rating along some sort of scale with each step being separate/discrete from the other (rather than continuous).

This does lower the resolution of the scale when compared to a continuous scale, but it is much clearer in terms of allowing for comparison of results. In my opinion, a 1-7 scale is probably the best way to go (over say a 1-5, or something larger) as it allows for some good resolution, while not being too broad. Others may disagree

Several interval scales in Arkham Horror

If an interval scale is used, then it can also be broken down further into several types: numeric/categorical, Likert, or semantic.

  • A numeric scale is simple; it just asks for a number, and is used often to rank things.
  • Likert scales you have probably seen many, many times and are usually for when you want to see how much someone agrees or disagrees with a certain statement, e.g. 1-7 where 1 is strongly disagree and 7 is strongly agree
  • Semantic scales are when you want to simply ask for a rating related to something or a value judgment, e.g. 1-7 where 1 is poor and 7 is good.

Each of these types of scales can be used where necessary, although generally speaking, it is best not to mix scale types too much. Otherwise people filling in the questionnaire might get confused and provide a rating that they think is using an agree to disagree dimension but is actually asking for poor to good.

Finally, with interval scales, you can either make them unipolar, where you ask for varying levels of one variable e.g. 1-7 where 1 is not very exciting and 7 is very exciting, or they can be bipolar where you contrast two different variables, e.g. 1-7 where 1 is very boring and 7 is very exciting. Unipolar scales zoom in on one area a bit better, but bipolar scales give people filling in the questionnaire more room to express their opinions. As with before, try to not mix these styles up within the same questionnaire too often, or at all.

Step 3. Put it together

You have your questions, so now's the time to put it together. First, work out what medium you are going to use. Will it be collected via a computer (say a through a web interface), or via paper? Collecting via a computer is preferable if possible as it means that there will be less data entry later on, however sometimes the portability and ease of paper can win out.

Give yourself plenty of time to do this, and if you are planning on using a computer-based survey, there are many companies online that offer this kind of service. In particular I hear that SurveyMonkey is a popular choice, although I have not used it myself.

You should next consider what order to put your questions in. Generally speaking I advise putting the easy questions at the start. This gets people rolling, and as long as your questionnaire isn't too long, they should be more prepared to tackle the bigger questions you want to ask later.

Then try and cluster the remaining questions in a sensible fashion, such as by subject, or what they refer to within the game. Don't ask a bunch of questions about the weapons, then move on to the boss, and then go back to the weapons.

Also check to see if answering some questions excludes other questions or could cause new questions to be asked later on. If so, make sure this occurs. In other words, if you ask "Do you own an Xbox" yes/no, make sure later on the people who answered "no" aren't asked to list the top ten Xbox games that they own.

Step 4. Test it

No plan survives its first contact with the players, so first test your questionnaire yourself. If you are using a computer-based questionnaire, make sure the data that comes out the end looks okay. Give it to a few other people (preferably people who are not too familiar with the game, or are not designers) and ask them to fill it in (with you outside of the room so you can't help them), and then ask for their comments on the clarity of the questions. Essentially, user test it, and then change anything that might be wrong (and then test again...) This is a great deal of work, I know, but once you have a good questionnaire, you can possibly use it again in the future (or at least cannibalize it for juicy parts).

Pros and Cons

There is much more that could be said, but perhaps this is already too much for a "primer" -- so I will move on. The best thing about a questionnaire is that you are asking the same questions to everyone, so you get consistent quantifiable data that you can compare between people.

However they lack follow-up, in that you can't ask people why they selected a certain rating, they are not objective, and you do need quite a few people if you want to draw completely solid conclusions from them. That said, even when testing with just a few people, providing a questionnaire can give you some ideas about what those individuals thought.


  • Consistent
  • Quantifiable
  • Relatively quick to administer
  • Can be used on a large scale


  • Can lack follow-up
  • Not objective
  • Really at their best with large sample sizes
  • It can take a while to put together a good questionnaire

Hopefully that is more than enough for this first article. While again this article is in no way comprehensive, I hope it has proven to be useful. In the upcoming part of this rough primer, I will be covering interviews, observational methods, game metrics and biometrics.

Finally, if you are interested in games user research or work in the area, then please consider checking out the IGDA Special Interest Group for Games User Research (GUR-SIG) on LinkedIn (just search for the group). It is a great place to get together and discuss GUR with others working in or around the industry.

Article Start Previous Page 4 of 4

Related Jobs

Remedy Entertainment
Remedy Entertainment — Espoo, Finland

Senior Gameplay Designer
Remedy Entertainment
Remedy Entertainment — Espoo, Finland

Gameplay Designer
Mountaintop Studios
Mountaintop Studios — Los Angeles, California, United States

Engine/Systems Engineer (remote)
Mountaintop Studios
Mountaintop Studios — Los Angeles, California, United States

Graphics Engineer (remote)

Loading Comments

loader image