I am sure you know what a questionnaire is, but do you know how to design and use one correctly? Mountains of books have been written on this, but hopefully I can make at least some important points clear.
First of all, when do you use questionnaires? Well, they are usually used to evaluate subjective views about your game, particularly value judgements. This may vary from specifically asking players about their favorite weapon to open-ended questions asking for general comments on the experience.
Questionnaires can be given to players during the game, which means the experience is fresh, but this risks interrupting the flow of the game (if possible, find natural down times to ask). They can also be used after a gameplay session is over. The big advantage of questionnaires is that they can be given to many people, and as such you can end up with lots of nice data to examine (in theory).
Before I go into the detail of constructing your own questionnaire, there are some pre-existing questionnaires out there aimed at evaluating the fun of gameplay experiences.
Examples of such questionnaire are the Game Experience Questionnaire (GEQ) for examining gameplay experiences and the affect grid [PDF link] or the manikin system [PDF link] for rating emotions. These pre-existing questionnaires can be great, because they are usually well-written and reliable. However, they also have a tendency to be more academic in nature, and care should also be taken that they fit your game, so modify them if necessary.
So how do you go about designing your own questionnaire? Here are four steps that I hope can help you.
Step 1. Work out what you want to know
As stated already, all of these methods require that you know what information you are after. But this is extra important when it comes to questionnaire design. You usually don't get any chance to follow up on people's answers, so you want them to be as clear as possible, and for the information you gain to be what you are after (and neither too little nor too much!)
So, brainstorm, make lists, do whatever is best for you in terms of getting down what you want to know. Then cut it down to only what you really, really need to know. Be focused!
A mistake that novice researchers often make is to ask for everything simply because the opportunity is there. However what this results in is a mess of data that will take forever to be analyzed, and may not produce a meaningful result. You don't really want your questionnaire taking more than 15 to 20 minutes to answer (this is not a target, by the way, but a maximum).
Step 2. Design the content
The design part is the meat of the process and can be further broken down into a few things to consider.
Questions or statements? Do you want people to answer questions, or rate statements? This is pretty straightforward but still important. Basically, questions are good for gaining information, for example:
How challenging was the Horrible Bog Beast?
1 2 3 4 5 6 7
Very easy Very Hard
Questions can also be worded in the form of an instruction. For example, "Rank the levels you just played in order from 1-6, where 1 is the one you enjoyed the most, and 6 is the one you enjoyed the least." It's effectively just like asking a question about the ratings for each level individually.
On the other hand, getting people to rate statements is more typically used to assess value judgments or agreement with ideas, so an example would be something like:
The Horrible Bog Beast is an interesting enemy to encounter.
1 2 3 4 5 6 7
Strongly Agree Strongly Disagree
Either questions or statements are fine; however, use them where they are best, and don't switch between the two types too often.
Language use. It is incredibly important that you use clear, everyday language. Avoid jargon. This is vital because you want to be sure that the people you are testing understand what you are asking. Many people will just answer anyway, even if they don't understand a question (be honest -- I am sure you have done it) and the data you get may not be useful at all. So be blunt, be direct, and ask for exactly what you want to know.
When you offer alternative answers to your questions (for example if they have to select from a list of consoles they own), these should be relatively exhaustive. So in other words there shouldn't be any alternatives that you haven't thought of. Adding an "other" where additional options can be filled in can help here, but it is best if this "other" option isn't used too often.
You should also be careful not to ask what is essentially the same question, but in a different way. Remember the goal is to keep the number of questions down. Also avoid asking questions that are phrased negatively. So "I like the jumping mechanic" and a gradient from agree to disagree rather than "I don't like the jumping mechanic". In the case of negatively-phrased questions, people filling your questionnaire have to select "agree" to disagree, and "disagree" to agree. It seems silly, but this can confuse people.
Also avoid leading, double barreled, and loaded questions. A leading question is one where the person filling it in is lead or biased to give a certain answer, e.g. "This game was fun. How fun was it?" This example is, of course, over the top, but you would be surprised how often leading questions can slip into questionnaires, so just try and keep your wording direct, and neutral. Don't assume. Ask!
Double (or multiple) barreled questions ask about more than one thing at a time. Here's an example from outside the world of games -- from a national referendum held in New Zealand:
"Should there be a reform of our justice system placing greater emphasis on the needs of victims, providing restitution and compensation for them and imposing minimum sentences and hard labour for all serious violent offences?" YES/NO
As you can see, there are at least six questions here; should there be a reform, should it place greater emphasis on the needs of victims, should it provide restitution and compensation, should it impose minimum sentences, should it impose hard labour, and should it be for all serious violent offences? But you only get to say yes or no once. As I say, this question was put to the whole of New Zealand, and really ruined my first ever experience of voting.
Loaded questions are ones that make moral judgments or make assumptions that are unfounded; an example of this type of question also comes from a New Zealand referendum where it was asked:
"Should a smack as part of good parental correction be a criminal offence in New Zealand?" YES/NO
This question is loaded in that it uses a moral term like "good". It is also ill-defined, as it uses the term "good parental correction" (whatever that is), and finally it is misleading in that if you are anti-smacking you have to say yes (agree) and if you are pro-smacking you have to say no (disagree).