This article is about creating a window into player activities by using passive tracking systems to measure and improve the player experience.

We’ll start by looking at simple example from the television show to illustrate how using target audiences (potential customers) along with an iterative process of experimentation and observation can help improve product quality. We will then go over the people, tools and techniques that went into building this capability at BioWare.

Sesame Street

When I originally read this story in the “Tipping Point” by Malcolm Gladwell I thought it was a good simple real-world example to illustrate using target audiences to test product during the development process without getting into complex implementation details which I cover in the second part of this article. I’ve kept this part brief and would recommend reading the book.

“In the late 1960s a television producer named Joan Cooney set out to start an epidemic. Her target was three-, four-, and five-year-olds. Her agent of infection was television and the ‘virus’ she wanted to spread was literacy.” ¹ Sesame Street was the show.

There were a couple of must have’s in this equation - a good idea and a team of passionate and talented people to make it a reality. The team did a lot of research and wanted to bring in an expert to help measure and improve the show’s quality. They hired Ed Palmer, a psychologist and pioneer in the research of television as an educational tool. He developed a simple test process to measure the show’s “stickiness” (engagement) that is its ability to hold the audience’s attention and aptly named it “The Distracter”.

In one example, the test audience was setup with an episode of Sesame Street which was setup next to a slide show intended to steal away their attention. The audience was observed and time-stamped notes were taken whenever a portion of the test audience lost interest. The information was then passed back to the development team for review and resulted in many improvements.

The process was simple – hire an expert, define a test process, test the product with the target audience, objectively observe and recording findings, review, improve and repeat as needed. The show’s longevity is a testament to its success after some 37 seasons and 4134 shows.

Now applying this to interactive entertainment software is a little more complex – since each experience is different (non-linear) we need to look at a large number of individual samples and track a considerable amount of data (events and status information), aggregate the information, analyze it and generate valuable information out of it since the experiences can vary widely.

^{1 Malcolm Gladwell, The Tipping Point: How Little Things Can Make A Big Difference ( Little, Brown and Company ), p.89}