Feedback Loops

Typically game development at the beginning of pre-production sets its sights on a final product by way of a variety of documents that focus on game design, technology design, sound design and art design. This type of planning is basically linear in approach illustrated in Figure 1.  However, what really happens is anything but linear as demonstrated by the Figure 2.

The problem with the postmortem process in game development is that it does not recognize nor permit a feedback loop as the process of development continues. It comes too late in the process to be of much benefit to the current game. Even if previous postmortems were consulted at the beginning of preproduction, the current game is, and will be, different. Developers will face a different set of issues than the previous postmortem offered solutions for.

Compounding the problem is that game teams are generally getting larger, and will no doubt continue to do so. It is now commonplace to have team comprising 50 or more people, and large teams make it difficult for management to continuously receive and process feedback as production progresses.

The currently used postmortem process is fraught with inefficiency and process shortcomings that are no longer applicable to current game development. It is time for a change.

	Figure 1

	Figure 2

Some Positives About Postmortems

Certainly there must be some positive aspects of postmortem process since it has been utilized for so long. I'm amazed that developers actually take the time at the end of a project to capture at least something about what went right on the game and what went wrong. The fact that there is at least some documentation is in itself a positive, especially in light of the fact that developers work incredibly hard to come up with games that hopefully stretch all aspects of the game industry.

Positives to the postmortem process do exist, as I heard from the audience in my session at GDC. At least some of the developers are asked their opinions, both positives and challenges are requested of respondents, and documentation exists for those that want to read it. For those of us who didn't work on the game, it's fun to read what went wrong on a big-budget project. Compilations have appeared in book form on the topic. I'm sure you can think of a few more positive aspects. Yet the overwhelming message is that the negatives of postmortems far outweigh the positives and it's time to come up with something better.

Introducing Critical Stage Analysis

Let me introduce Critical Stage Analysis (CSA). CSA is a tool that effectively improves your current game by examining its progress at critical stages throughout its development cycle. It replaces the postmortem process.

The beauty of the CSA process is its simplicity. I've found that the most powerful tools are the most simple and straightforward ones, about which people typically say "Hey, I could have thought of that!" The CSA process does not disappoint in this area.

The process is easy. At each milestone (or "critical stage" - usually monthly) ask everyone on your team to respond in writing to three simple questions.

1. Five things that went right during the period
2. Five things that went wrong during the period
3. Five things that could be improved for the future

Incorporate a rating scale of 1 to 5 (1 being the most important, 5 the least) for respondents to rank their five answers to each question, in order of importance. It doesn't matter at this stage whether the team leads, project manager or producer feel any of these items are more or less important than the respondents. What's important at this stage is to get an accurate snapshot of what everyone on your team feels are the important items for each question and their order of importance. Of course, if someone submits more than five answers, this should we welcomed. But requesting five issues reinforces that you want important information.

Communicate to the team that you want their honest and accurate feedback, but also let them know that they should be respectful of any and everyone on the team. What you don't want is people complaining for complaining's sake, nor do you want to start a mud-slinging session. You are primarily looking for solutions to issues in their early stages before they become major problems. I find it good to lead with the positive aspects of the project, since celebrating successes leads to many good things, including raising individual and team morale.

Project managers are the best equipped to gather this information from a psychological perspective, because often they do not manage the personnel (teams) directly and are therefore considered a neutral party. It also fits in well with the other aspects of their job description.

This questionnaire (a sample of which is included in Figure 3, below) should be made available to the team no more than three days after a milestone. I've found that if you make this a regular part of your post-milestone art, code, design, sound and lead meetings, the process of gathering the information will be quick and painless. It can usually be done within 10-15 minutes in such meetings. I also recommend having every option available to respondents -- let them submit their responses via paper form, email, or online, and give them the option to submit their answers anonymously.

Once the project manager compiles all of the information into one document, it should be emailed out to the rest of the team leads and producer, and discussed in a team-lead meeting no later than two days after the information has been gathered. The most important points should be discussed, solutions determined, ownership assigned (to either team leads and/or team members) and then presented to the entire team for discussion. All of this should be completed within a week of the milestone that has just past.

During the team meeting, the producer or project manager should go over all of the feedback in order of importance, solutions discussed and ownership assigned to solve the issues within a specific timeline. Follow up is important, and team leads should be assigned responsibility to make sure that the issues are addressed.

At the next general team meeting, the status of previous issues that were dealt with should be brought up first. If there is a problem that can't be solved, then it should be honestly brought forward during the general team meetings and an explanation given. It is important not to hide any issues -- honesty among the team leads when dealing with the toughest issues will go a long way towards increasing team morale, even if the problem couldn't be solved for whatever reason. It is important to publicly acknowledge the issues and to demonstrate that you are at least trying to solve them even if you can't. Issues that are buried will just come back to bite you at a time when it's far too late to do anything about them. (no solutions are available)

During the general team meeting, lead with the positives first by celebrating those things that went well! This is also an opportunity to hail those on your team who went the extra mile or came up with innovative solutions during the past milestone.

Once discussed in the general team meeting, the compiled and unedited CSA should be emailed to all team members and made available in a central repository along with past CSAs. This could be located in the team's Perforce branch or on an internal team web site. You could also have a company CSA website where all teams deposit their monthly CSAs so everyone can learn from each other.

From a management perspective, what's most important is to create a positive atmosphere that will reinforce to everyone on the team that you take their feedback seriously. It may take two or three CSA sessions to do this initially, but if the management team does this correctly you will see magical things start to happen: everyone will fill out the CSA forms happily; everyone will begin to feel part of the process, and most important, part of the project. Their emotional involvement will increase, creating a positive feedback loop that will enhance the potential of the game.

The whole process, including having the team respond, compiling the responses, conducting the team-lead meeting and the general team meeting only takes about two to four hours, total. The most affected person from a time perspective is the project manager, but this is a small price to pay for the wealth of timely information and the benefits it bestows.

So, how do you measure success utilizing the CSA process? You should see fewer issues raised repeatedly, and fewer CSA responses of high importance over the course of your game's development.

The CSA process is simple in design, lightweight, very powerful with many positive attributes that contribute to making better games in a highly positive team atmosphere.

CSA - A Perfect System?

One respondent during my CSA presentation at the GDC commented that I had presented the process as being perfect. Well, I'm flattered but I'm sure there are ways we can improve on it. Already many of my GDC roundtable participants have emailed and said they are using the CSA process as a regular part of their game development with positive effects.

Please email me and let me know how we can improve on the process and I'll share this in a follow-up article or another GDC presentation.