Postmortem: Sierra Studios' Gabriel Knight 3

1. Team casting problems. When someone is placed in a role in which they don't belong, I call this being "badly cast." Many of the problems with GK3 resulted from developers being badly cast in their roles, usually because the project requirements were so severely underestimated. To give you an idea of the casting problems we had, consider this: we went through a total of two producers, three art directors (we spent the last year of the project without one), and three project leads (the producer was forced to take over as project lead towards the end).

This was an ambitious, massive project that required experienced engineers and the original team was simply not up to this task. GK3 was initially built from members of the Shivers 2 team (one of the last games built with SCI) and they had practically no 3D experience. Engineers under the venerable SCI engine were basically scripters and putting them in charge of building a game engine from scratch was like feeding them into a furnace. To make things worse, the developers that were in over their heads didn't ask for help, which gave management a false sense of progress.

2. Severe morale problems. Hundreds of books and articles have been written about this and here we have yet another Postmortem listing it under "what went wrong." It's time for me to get on my soapbox. To managers everywhere: morale is one of those icky personal political things that many of you avoid dealing with, but you need to understand that your development team is not a factory churning out content and code. To paraphrase Peter Sellers in Being There, "The team is a garden of creativity that requires regular watering and sunshine in order to build strong roots." Loyalty is not something that comes easily. The job market is very competitive -- your best developers will simply leave and work for somebody else if they aren't treated well and maintained properly. On GK3, there was a serious lack of love and appreciation throughout the project. Recognition of work (other than relief upon its completion) was very rare, lacked sincerity, and was always too little, too late.

Just about all of Gabriel's inventory fits in his pants.

Internally, a lot of the team believed that the game was of poor quality. And of course, the many web sites and magazines that proclaimed "adventure games are dead" only made things worse. Tim Schafer's Grim Fandango, although a fabulous game and critically acclaimed, was supposedly (we heard) performing poorly in the marketplace. Rumors circulated among our team that GK3 was going to lose money, due largely to our high burn rate.

The low morale resulted in a lot of send-off lunches for developers seeking greener pastures. GK3 had a ridiculous amount of turnover that never would have been necessary had these people been properly cast or well treated in the first place. More than 45 developers worked on GK3 (the average standing team size was 15 to 20), and now, just a few months after it shipped, only seven remain at Sierra. Strangely, the opposite also happened -- several of our developers were included in Sierra's mid-1999 housecleaning layoffs but these individuals were allowed to stay on for a couple of months, postponing their last day until we shipped GK3. I believe this was done in good faith out of respect for the developers' hard work up to that point but it ended up being a prolonged drain on morale. Having a small group of people who are (understandably) upset with your company for laying them off and actively looking for a job while still trying to be productive and contribute to a project is a tough situation that should be avoided.

After a certain amount of time on a project like this, morale can sink so low that the team develops an incredible amount of passive resistance to any kind of change. Developers can get so tired of the project and build up such hatred for it that they avoid doing anything that could possibly make it ship later. This was a terrible problem during the last half of the GK3 development cycle and as a result there are many aspects of the game that we aren't proud of. These were problems that should have been fixed but nobody wanted to take the time to correct them because we were so focused on trying to get the game out. I don't think anyone on the team is directly at fault for this and I don't know what we could have done to correct this problem.

3. Schedule problems. Our engineers never had an accurate development schedule -- the schedules we had were so obviously wrong that everybody on the team knew there was no way to meet them. Our leads often lied to management about progress, tasks, and estimates, and I believe this was because they were in over their heads and weren't responding well to the stress. Consequently, upper management thought the project was going to be stable and ready to ship long before it actually was, and we faced prolonged crunch times to deliver promised functionality. More frequent and honest communication within the team would have avoided a lot of this.

GK3 had few real milestones, which undermined our ability to track progress. There were some milestones very early on in development, but the focus on shoveling visible features into the game turned them into worthless smoke-and-mirrors demos. The concept of milestones eventually was discarded and was replaced with two simple and unofficial goals -- beta and release to manufacturing. In the push to ship the game, we simply forgot about milestones (because "we're almost there!"), put the blinders on, and worked like mad.

Crunch mode is a reality on most projects, but you should not gear up for one unless the light at the end of the tunnel is really in sight. The GK3 team was pushed into crunch mode three separate times, each time thinking that we were almost ready to ship. Most of the last year of the project we spent in this mode, which meant that even small breaks for vacations, attending conferences, and often even taking off nights and weekends were looked down upon. It was time that the team "could not afford to lose." The irony is that this overtime didn't help anyway -- the project didn't move any faster or go out any sooner. The lack of respect for our personal lives and attention to our well-being caused our morale to sink.

Because of the high turnover, GK3 always had a high percentage of developers new to the team. Faced with mandatory overtime, these new people understandably felt it was unfair to be "punished" by paying for problems caused by the original team or things that they felt management had brought upon itself. GK3 became a black hole that sucked in many developers from other projects, often at the expense of those projects. Artists were shifted off the team to cut the burn rate, and then pulled back on later because there was so much work left to do. Our art team finally got on track in the last year of the project (thanks to Ray), but engineering never got a solid schedule together and as a result we were nearly always late in our feature delivery.

2D interfaces are overlaid on the 3D scene to keep the player immersed.

4. Engineering problems. When we rebuilt the game engine, we tried to retain as much of the original code as we could to get the game up and running again as soon as possible. With the exception of the G-Engine, this was a big mistake. Nearly every one of the systems that we kept caused us problems -- they were badly designed, buggy, inflexible, and should have been redesigned. Some of these systems never worked correctly throughout the lifetime of the project and had to be hacked around by the content developers to get the game to ship.

Specifically, we had serious problems with the "fidget" system (used by character models when idle or when involved in dialogue), the character model's walker, the vertex animation system, and the conversation and dialogue systems. All of these failed regularly and were regularly "fixed," but each bug fix introduced new bugs, usually in the form of hidden time bombs. The engineers responsible for these systems became very defensive about the problems with them, and usually ended up blaming artists and scripters or even other engineers for the cause. Management, thinking that it would save time, often encouraged content developers to hack and work around the problems rather than fix them properly. We should have ripped these parts of the game out and rebuilt them, rather than continually attempting to work around a flawed legacy design. It would have saved a lot of time and hard feelings.

We also faced a lot of problems that were out of our control. Most notable were the technical difficulties with the DirectX drivers provided by hardware vendors for their 3D graphics cards and sound cards, but this probably isn't news to any 3D game developers. These problems were generally features that were implemented improperly or inconsistently, or just outright bugs that caused system crashes or hangs.

We also wasted a few weeks trying to add copy protection. During the final push to ship, we repeatedly attempted to make Macrovision's SafeDisc product work with GK3. SafeDisc has a set of special (and we felt completely unnecessary) antihacking measures that got in the way of the game's execution. It heavily affected performance, dropping the frame rate to a third of its original speed and adding strange intermittent freezes of several seconds while the camera was moving. After getting nowhere with Macrovision's engineering department, we decided to ditch SafeDisc and roll our own (which took less than a day to do). This entire process wasted several weeks of our time and frustrated us all the more because, apart from this one remaining task, we were ready to ship the game. Lesson learned: If you are required to use copy protection, don't put it off until the last month, especially if it's SafeDisc. We weren't the first game to have severe problems working with SafeDisc and probably won't be the last, so if you're using this product be sure to do your homework and try it out well in advance of your ship date.

Gabriel Knight, from concept art to dashing real-time 3D model.

5. Art and (more) engineering problems. One of the most expensive mistakes a team can make is ramping up art before the engine is ready. This often happens at large game companies because developers need a place to go after they've shipped their most recent game. Unfortunately, this was a serious problem with GK3 -- artists were brought onto the game while the original engine was in development, and long before a stable engine was available. They created content for an animation engine that was untested and in doing so built up enough inertia that we ended up having to keep the design. Later we discovered that the engine's design was seriously flawed.

GK3's animation system is vertex-based, meaning that a model's individual vertices are animated. Even using some creative compression methods, this is very expensive in terms of memory usage. Contrast this method with a typical skinned skeletal animation system, which only requires that the bones be animated. The worst thing about this system was not the memory usage, however. It was the impact on content creation, and the repercussions of this requirement were not fully realized until the art team was ramped up and churning out models and animations.

Rendering lasers required a little bit of custom code.

GK3 animations are completely coupled to the meshes of the models that they affect. Once an animation is exported from 3D Studio Max, the models it involves cannot be changed in any way, otherwise existing animations created from the old versions of those models would break. Vertices can't be added or removed, and texture maps can only be tweaked, not remapped. Changing a model required re-exporting every single animation that affected the model, which was very time consuming, tedious, and repetitive for animators, and was often impossible to boot. The source assets (the original Max files) had a way of getting lost and usually ended up stored on backup disks as artists left the project. The end result was that once a model was created, it could never be changed. Consequently, GK3 shipped with a lot of bad art that the team was dissatisfied with but had to use. An example of this was the Mosely character (sometimes not-so-fondly called "T-Rex man" internally), whose arms were way too short. Just seeing this guy in the game hurt our morale, and made a lot of us feel poorly about the game. The art was bad and there was nothing we could do about it. Nearly every attempt to change a model was met with the response, "That will require re-exporting all of its animations, which will take too long."

What we should have done was just stop the presses and fix the art. When we first recognized the severe problems with the vertex-based animation system, we should have rebuilt that part of the G-Engine and replaced it with a simple skeletal animation system (as the SWAT 3 team later did on their project). We should have also thrown out all of our existing art and recreated it. In retrospect, that would have saved us a lot of time, given us better performance, and made development proceed more smoothly.

Out The Door

Gabriel Knight 3 was an extremely ambitious project combined with an extremely inexperienced team. With all that went wrong, you would have every reason to believe it would never have been finished. Despite all that was messed up with our development process however, we somehow managed to get GK3 out the door and ship it in time for Christmas 1999. Best of all, the game works and it works well, against all odds. This is a tribute to the testing skills of our QA lead, Matt Julich, and his team. Though severely understaffed, they did a great job ensuring that the game we shipped was of high quality. There are no crashes and no hangs in GK3 -- Sierra will not be issuing a patch.

Concept and final environment for the end game.

This isn't to say that GK3 has no problems, just no problems that we had any real control over as developers. To the best of our knowledge, every single critical problem with the game is caused by either bad hardware drivers (sound or 3D card) which can be fixed by upgrading them, or CD-ROM problems due to our oversized CD copy protection. Also, the first disk that shipped in some retail boxes was difficult to read on some CD-ROMs, apparently due to duplication problems. This can also be worked around, as the same file exists on the second disk and can be copied to the hard drive.

GK3 is certainly not the game it could have been had we been able to knock down a couple of those "what went wrong" issues early on, but it is a high-quality, entertaining game that has been well received by adventure gamers. And that's something we can all be very proud of.