"What's wrong with the games that are out there today?"

I was asked this question one day by a student of mine at AI Centre for Digital Imaging and Sound, where I have been teaching a course called "Business Management for Games" for a number of years.

"Well, where can I start?" I responded. A good place might be is to investigate the way we develop our games. On that note I dug into all of the postmortems I could find online, in various magazines and books, and from discussions with people who worked in the industry.

After many hours, days, weeks and months of reading articles and talking to many game developers about how they produced games, I realized that the game industry as a whole was continuously repeating the same mistakes over and over again. And not only were the same mistakes being repeated over and over again, everyone was using the same process to analyze those same mistakes as if it was some kind of "blind religion". I determined that the process that was being used to seemingly fix these issues was actually sabotaging the efforts to repair them.

A "Blind Religion"

What's this you say -- "blind religion"? C'mon, we've got a lot of really smart and creative people in the industry working with multi-million dollar budgets and with games that now take years to develop. We have loads of people with BAs, Masters Degrees, and even some MBAs and more than a few PhDs. All of these smart people are working away to come up with the next major commercially viable title. And some are even successful at it. Please note the word "some", since the ratio of success to failure is pretty small -- I've heard estimates ranging from 1 in 7 to 1 in 25. What I do know for sure is that the top 20% of games generate about 80% of the revenue in sales.

On one game I worked on, one of the team members complained to the producer that months of working 12-hours-per-day, seven-days-a-week was getting to him. He said he'd like to have at least one weekend off. The producer replied, "This is the game industry -- crunch time is a fact of life." I have a question for all of you game company owners, producers and leads out there: Ever wonder why you have such a high turnover rate at your company? Studies have shown that most game developers will put up with a severe crunch mode for one, or at most two, projects and then they start looking for another company. Believing that crunch time is a fact of life is a result of following the "blind religion" that's so pervasive in our industry.

Why are these same really intelligent souls continuously making the same blunders over and over again? We all work like slaves under constant crunch time, sweating each design idea into a proof of concept. That design idea requires more art for the modelers and animators to create and rework, more code to crunch, more sound effects and music, further iterations of the game, and so on. Then a GTA: Vice City hits the street, everyone pops up their periscopes and does a feature comparison analysis, takes a deep breath, and tries to cram a few more goodies in to make it more fun and commercially successful. All this, and the game still has to ship on the pre-agreed upon date.

Once the game's done, those still standing take a bleary-eyed look in the rear view mirror to try and recall what went wrong. Typically, someone actually has enough energy to do something called a "postmortem," the purpose of which is to presumably never relive this same craziness again and come up with a much better game the next time around under much better working conditions. "We'll not make the same mistakes" someone writes in the postmortem, only for the postmortem to never be seen or heard from again. Then the next production repeats many of the same mistakes.

Law of Diminishing Returns

While by no means is severe crunch time the only problem with game development, I'd like to speak out about the relationship between complaints about the lack of innovation in games and the expectation that working overtime is normal. While I'm sure there are many successful titles that have required some crunch time, my feeling is that the "crunch" is at least partly responsible for the lack of innovative titles we see today. A team may be able to "crank" for a considerable portion of the first iteration of a title, but eventually it will get burned out.

There are numerous studies that show how the Law of Diminishing Returns affects productivity. Other long-term studies have shown that health problems result from a diet of night-shift work, even if the schedule is periodic. Other studies show that creativity is one of the first things to disappear when the Law of Diminishing Returns sets in.

A "Postmortem" on Postmortems

So what is a postmortem exactly? Typically, it's a document written near the completion of the project (or shortly thereafter) by one or more people on the team, who gather information on what happened over the past eighteen months or so during the game's development. The postmortem is designed to document what went right and wrong during the game. But a more cynical definition might be the following:

"A common artifact of the game development process whereby the game industry documents the fact that everyone seems to continuously make the same mistakes"

During my presentations at the Game Developers Conference (GDC) this year, I asked the participants to list both the positive and negative things about the postmortem process as they have used it on their games. I'll start with the negative aspects, since these far outweigh the positive.

The postmortem only occurs at the end of a project. I don't know about you, but with the fast pace of game development, I have a difficult time recalling with a high degree of accuracy what transpired last week, let alone last month or last year. In addition, the phenomenon of selective memory kicks in. We like to remember what is most important to us and not necessarily what is most important to the project. Also, once the game's done, many people are already off the project on holidays, working other games or at another company. If not, the developers are heading in this direction and their motivation to fill out postmortem forms is pretty low.

A postmortem offers no opportunity to solve problems for the current game since the game's development is over. I thought we were trying to make better games while we were making them? What is the point of a system that can't provide any impact to your current project? Not much.

Some team members don't want to "rock the boat" by critiquing senior team members, team leads, the producer or management. This is a normal human tendency. New team members are just getting their feet wet and getting the lay of the land. However, they may have excellent, fresh perspectives on how to improve the process. Whether new team members are experienced developers or new to the industry, I've found that feedback from recent team additions is very valuable.

We all need feedback to improve our skills, both professionally and personally, and it doesn't matter if you're at the top of the totem pole or the bottom. Without a mechanism for constructive critiquing, and an atmosphere that supports it, your team is missing a golden opportunity.

Some long-standing team members feel left out of the process, or feel that their views are just archived and forgotten. We have a term for this: "jaded". It's an unfortunate fact of life that highly creative people who don't feel they are getting heard stop contributing and deliver the bare minimum to get their job done. It's not just a question of working the minimum number of hours each week - you need that all-important emotional commitment that helps create a high-caliber game.

Some veteran developers who have repeatedly filled out postmortem forms feel that their views were quickly forgotten for the subsequent project. Often this isn't management's fault; it's because of the postmortem process itself: it's performed once at the end of the project and then archived, never to be seen again. When the next project repeats the same mistakes, those who contributed to the previous postmortem quickly say "I told you so" and ask the all-important question - "why are we doing this again"?

Postmortem questions/responses are too technical or not their area of expertise. Artists aren't coders, coders are rarely designers, and sound designers don't program. I've seen many postmortem questionnaires littered with questions that only apply to one of these disciplines. Yet someone from another discipline could easily have contributed valuable information, if only the right questions were asked and presented in a different format. Most artists who are responsible for rendering cinematics could probably not answer the following question: "What in your opinion contributed to the failure of the surround pre-encoded LT/RT 48K 16-bit FMV audio files on the PS2 build?" However, the artist might have provided very valuable information, reporting that the audio files he received were of a different sample rate than the video compositing software allowed -- had a different feedback process been used or the question phrased differently. Postmortem questionnaires often ask detailed questions long after the details have been forgotten.

No scoring values were assigned to feedback - there was no way of communicating the importance of the feedback. How can one tell which issues had the most or least impact? Postmortems ask questions, but what about the importance of these questions with regard to their impact on the game itself? We all know that breaking the build is important, but how important was that to the animator trying to get his art visible on the target platform during integration week? Was it more or less important than the poorly designed tool he was trying to use in the same situation? Only the animator knows for sure. Postmortems don't ask these kinds of highly important questions, and even if they did, it would be way too late in the process. The game was late and the art not as stunning as it could have been, all because of a poorly designed tool or unstable build at critical times in the project's development.

Once development starts on the next project, everyone is too busy to go back and check previous postmortems. There is an old quote by Santayana that goes, "Those who ignore history are condemned to repeat it". This is an industry truth. During my GDC presentations, when I asked how many developers actually read postmortems from the previous game they worked on, less than 2% raised their hands. Developers are excited (rightly so) about the next project, and fresh from some needed time off, but who wants to dig up all of the nasty issues from the last project? We know we can do better, right?

Nobody is responsible for fixing or dealing with the issues raised in postmortems. This is a very important point, not just from a personal and professional growth perspective -- also from a game, team and company growth perspective. As I said earlier, postmortems are too late in the process for anyone to fix the problems, much less nip them in the bud before they become larger issues. Because it's too late, many problems just get buried, only to rear their ugly heads as bugs during the release stage of development -- if the project has gotten that far.

The same mistakes keep getting repeated. I talked about this in the introduction. The same problems crop up in subsequent games developed by the same team and across teams in the same company. Apparently very little learning is taking place. The lack of knowledge sharing is a huge tragedy for everyone and stunts team and company growth. For teams and companies to make better games, the team and company must grow together. Postmortems don't facilitate this kind of growth since the information gathered is neither high quality, timely nor utilized in any real positive fashion.

Here's a brief summary of postmortems from Game Developer magazine. See if you can recognize how many of these you've experienced first hand on the projects you've worked on. Note that each one of these issues occurred multiple times in different games:

• Scheduling issues - deadlines and deliverables weren't clearly defined or adhered to
• Lack of project plan or milestones
• Unrealistic schedule
• Poor communication within and across teams
• Major risks not front loaded
• Feature and content timing misaligned or not planned
• Insufficient staff
• Team casting problems
• The team structure didn't work
• Unrealistic goals
• Scoping issues - too much content, overambitious features
• Crunch time that seemed to last forever

All of these issues could have been reduced or avoided altogether!

Postmortems generally deal with high-level issues. Many of the seemingly "small" day-to-day items that end up causing larger problems are long forgotten or are fuzzy at best by the time the postmortem is tackled. Yet the micro details are the ones that often had the most impact.

Generally games are late or don't realize their true potential because the day-to-day issues were not properly dealt with early in production. A cumulative effect sets in; many of the small issues that don't initially seem large snowball and became monsters at the end. For instance, a world that's way too big halfway through production suddenly requires lots of sound assets near the end of the project, but by that time the budget is already blown. Later, when the postmortem is being done, people criticize the game audio for being late or sounding poorly. People tend to focus on the high-level results, not necessarily the seed that germinated into the problem.

Feelings and perceptions are reported, and facts are forgotten. When I joined a previous project late in its development, I heard that the game team hated one of the game's designers. At the end of the project, having arrived too late to fully apply the CSA process, I did my own postmortem to determine the reasons why this was the case. I interviewed most people on the team and heard complaint after complaint. I interviewed most people on the team and heard complaint after complaint. The complaints were generally laced with considerable emotion. Perfectly normal, reasonable people began spewing negatives as soon as this person's name was mentioned. It seemed as if a switch was turned on and the negative energy began flowing as soon as the discussion turned to the question of design.

After listening to considerable negative comments for extended periods of time, I finally asked the all-important question. Why? Why did you feel this way about this person? Interestingly enough, they couldn't give specifics. It was all generalities and lots of negative feelings. They would say things like "He does good work", or "It's not a question of his technical ability", and then they'd say something like "He doesn't fit in with the team", or "I don't know, I can't put my finger on it". It was as if the team had developed some type of mass dislike for this person but no one could really put their finger on the reasons why. And it wasn't because these were bad people. They were good people, generally friendly but something had happened somewhere during the game's development that no one could really recall. And yet, I'd say 70% of the team had a real emotional reaction to this one person.

At the start of the next project, a large number of the team members emailed the producer and said that if this same designer was going to be on the team again, they would all quit.

The story, however, does have a better ending. After considerable thought, a recommendation was made to bring in a trained mediator from outside. Even though it was one person against many during the process, the designer joined the same team on the next project and never a negative word was heard again. In fact, things turned around so well that at the end of the project nothing but positive things were being said.

I guess one could say that this is a situation where the postmortem process worked. We found a problem and initiated a solution to solve that problem. But how much time and energy was wasted during the previous game's development because of misunderstandings that grew into hardened, negative emotions over time? What if this kind of thing had been discovered early and solved early?

The previous game's postmortem issues do not apply to the next game. How many true sequels do we all work on? Unless we are lucky enough to work on a franchise product, I'd guess more often than not that the next project we work on will be quite different. Much of the information we gathered during the postmortem process is pretty much useless and non-applicable to what is coming down the proverbial pipe.

Even when we work on a sequel, things change. The AI or sound engine gets rewritten due to unforeseen limitations, pipelines get reworked, proprietary tools are scrapped due to architectural dead ends, processes get revised, critical team members leave and new ones are hired, art gets changed, and so on.