What questions would you like answered about the product? Many developers want to know how customers are using their products and what they like or don’t like about them. The passive nature of these tests is the reduced bias since there is very little interpretation required (it’s pure).
This is a good start point to help define the requirements. Draw up an example report and then worry about the implementation. It is important to develop a credible process – one that is objective, simple and understandable.
This paper focuses on tracking the player experience but logging systems like this can be used to track test coverage (a favorite of mine), performance and a variety of other elements.
In the above picture you’ll find the process de-mystified. The phases are self-explanatory. Test passes are often done in a controlled environment with a large set of testers on a specific build of the product with a clear goal. The hard part is actually defining what to measure.
In this example we wanted to know what activities players engaged in and how long they spent in each activity in a full playthrough. We had some internal documentation that contained valuable information about building RPG experiences however the percentages of time players spent in those were quite off. This allowed us to add accurate information about timing to our knowledge pool (and in the document).
The Y value is total time and the X value is the activity buckets.
In this example we wanted to know what special powers were being used by players and for how long. This can be used to balance powers and find out what appeals (and what doesn’t) to players. Identifying root cause and coming up with appropriate solutions is up to those responsible for crafting the experience.
The Y value is the number of times the power was used and the X value is the character’s level.
These are just a few examples of what a system like this can be used for. From a QA perspective I really enjoy Test Coverage reports which get us accurate and timely information on what has been tested and what hasn’t.
It’s important to note that this is just one perspective and isn’t meant to replace other tried and true methods such as user surveys, expert review and first hand observation. When the information from all these processes is correlated it should provide a more complete picture of the experience and possible issues.
Playing games may be seen as trivial but making great commercially successful games isn’t. To summarize this in a few bullet points:
“21st Century Game Design: Designing for the Market” by Chris Bateman (Demographic Game Design)
“Just Who Are Our Customers?” by Chris Bateman
“Hearts, Clubs, Diamonds and Spades” by R.A. Bartle
“Using the RITE method to improve products; a definition and a case study” by Michael C. Medlock, Dennis Wixon, Mark Terrano, Ramon L. Romero, Bill Fulton
“Metrics in MMP Development and Operations” by Larry Mellon