The Test Processes And Reports

What questions would you like answered about the product? Many developers want to know how customers are using their products and what they like or don’t like about them. The passive nature of these tests is the reduced bias since there is very little interpretation required (it’s pure).

This is a good start point to help define the requirements. Draw up an example report and then worry about the implementation. It is important to develop a credible process – one that is objective, simple and understandable.

This paper focuses on tracking the player experience but logging systems like this can be used to track test coverage (a favorite of mine), performance and a variety of other elements.

The Process

In the above picture you’ll find the process de-mystified. The phases are self-explanatory. Test passes are often done in a controlled environment with a large set of testers on a specific build of the product with a clear goal. The hard part is actually defining what to measure.

An Example Report 1 - “Player Time Spent”

In this example we wanted to know what activities players engaged in and how long they spent in each activity in a full playthrough. We had some internal documentation that contained valuable information about building RPG experiences however the percentages of time players spent in those were quite off. This allowed us to add accurate information about timing to our knowledge pool (and in the document).

The Y value is total time and the X value is the activity buckets.

An Example Report 2 - “Power Usage”

In this example we wanted to know what special powers were being used by players and for how long. This can be used to balance powers and find out what appeals (and what doesn’t) to players. Identifying root cause and coming up with appropriate solutions is up to those responsible for crafting the experience.

The Y value is the number of times the power was used and the X value is the character’s level.

These are just a few examples of what a system like this can be used for. From a QA perspective I really enjoy Test Coverage reports which get us accurate and timely information on what has been tested and what hasn’t.

It’s important to note that this is just one perspective and isn’t meant to replace other tried and true methods such as user surveys, expert review and first hand observation. When the information from all these processes is correlated it should provide a more complete picture of the experience and possible issues.

Conclusion

Playing games may be seen as trivial but making great commercially successful games isn’t. To summarize this in a few bullet points:

• Involve the customer early and throughout the development process to gather player experience feedback both passively and through surveys to get valuable insights on product quality.
• Develop a process and set of tools that allow for effective and efficient gathering and analysis.
• Need good ideas and good people.
• It’s cheaper and easier to identify and fix issues earlier (an ounce of prevention is worth a pound of cure). I will continue to repeat this mantra in all my articles.
• Use a variety of tools, techniques and experts in addition to a system like this.
• As Thomas Edison said, “There is a way to do it better, find it!” Be on the lookout for best practices inside and outside the industry.
• Invest time and money in building tools that improve you capability and capacity to measure product quality.
• Do see your QA team as a valuable asset by supporting them, including them, being open minded and you’ll be surprised at the results you’ll get. Now go and give them a hug.

Additional Reading

“21st Century Game Design: Designing for the Market” by Chris Bateman (Demographic Game Design)

“Just Who Are Our Customers?” by Chris Bateman

“Hearts, Clubs, Diamonds and Spades” by R.A. Bartle

“Using the RITE method to improve products; a definition and a case study” by Michael C. Medlock, Dennis Wixon, Mark Terrano, Ramon L. Romero, Bill Fulton

“Metrics in MMP Development and Operations” by Larry Mellon