Closed or open? Questions can be either open, as in the player can say whatever they like, or closed, where there are only a few limited options to select.
Open questions allow for richer data to be collected, as they let players give as much feedback as they want. However they can also give as little as they want, and often without direction, the answers may be vague.
One way to get around this if you do use open questions is to make sure you have time to read them before the end of the session, in case you want to ask for clarification on any points people bring up. However this is difficult if the questionnaire is being given remotely, such as when it is being used via email or online.
Closed questions on the other hand let you have much tighter control on the answers your participants give, and come in several flavors (scales):
A dichotomous choice in Diablo III
A (semi) continuous scale in Saints Row: The Third
This does lower the resolution of the scale when compared to a continuous scale, but it is much clearer in terms of allowing for comparison of results. In my opinion, a 1-7 scale is probably the best way to go (over say a 1-5, or something larger) as it allows for some good resolution, while not being too broad. Others may disagree
Several interval scales in Arkham Horror
If an interval scale is used, then it can also be broken down further into several types: numeric/categorical, Likert, or semantic.
Each of these types of scales can be used where necessary, although generally speaking, it is best not to mix scale types too much. Otherwise people filling in the questionnaire might get confused and provide a rating that they think is using an agree to disagree dimension but is actually asking for poor to good.
Finally, with interval scales, you can either make them unipolar, where you ask for varying levels of one variable e.g. 1-7 where 1 is not very exciting and 7 is very exciting, or they can be bipolar where you contrast two different variables, e.g. 1-7 where 1 is very boring and 7 is very exciting. Unipolar scales zoom in on one area a bit better, but bipolar scales give people filling in the questionnaire more room to express their opinions. As with before, try to not mix these styles up within the same questionnaire too often, or at all.
Step 3. Put it together
You have your questions, so now's the time to put it together. First, work out what medium you are going to use. Will it be collected via a computer (say a through a web interface), or via paper? Collecting via a computer is preferable if possible as it means that there will be less data entry later on, however sometimes the portability and ease of paper can win out.
Give yourself plenty of time to do this, and if you are planning on using a computer-based survey, there are many companies online that offer this kind of service. In particular I hear that SurveyMonkey is a popular choice, although I have not used it myself.
You should next consider what order to put your questions in. Generally speaking I advise putting the easy questions at the start. This gets people rolling, and as long as your questionnaire isn't too long, they should be more prepared to tackle the bigger questions you want to ask later.
Then try and cluster the remaining questions in a sensible fashion, such as by subject, or what they refer to within the game. Don't ask a bunch of questions about the weapons, then move on to the boss, and then go back to the weapons.
Also check to see if answering some questions excludes other questions or could cause new questions to be asked later on. If so, make sure this occurs. In other words, if you ask "Do you own an Xbox" yes/no, make sure later on the people who answered "no" aren't asked to list the top ten Xbox games that they own.
Step 4. Test it
No plan survives its first contact with the players, so first test your questionnaire yourself. If you are using a computer-based questionnaire, make sure the data that comes out the end looks okay. Give it to a few other people (preferably people who are not too familiar with the game, or are not designers) and ask them to fill it in (with you outside of the room so you can't help them), and then ask for their comments on the clarity of the questions. Essentially, user test it, and then change anything that might be wrong (and then test again...) This is a great deal of work, I know, but once you have a good questionnaire, you can possibly use it again in the future (or at least cannibalize it for juicy parts).
Pros and Cons
There is much more that could be said, but perhaps this is already too much for a "primer" -- so I will move on. The best thing about a questionnaire is that you are asking the same questions to everyone, so you get consistent quantifiable data that you can compare between people.
However they lack follow-up, in that you can't ask people why they selected a certain rating, they are not objective, and you do need quite a few people if you want to draw completely solid conclusions from them. That said, even when testing with just a few people, providing a questionnaire can give you some ideas about what those individuals thought.
Hopefully that is more than enough for this first article. While again this article is in no way comprehensive, I hope it has proven to be useful. In the upcoming part of this rough primer, I will be covering interviews, observational methods, game metrics and biometrics.
Finally, if you are interested in games user research or work in the area, then please consider checking out the IGDA Special Interest Group for Games User Research (GUR-SIG) on LinkedIn (just search for the group). It is a great place to get together and discuss GUR with others working in or around the industry.