1.
Team casting problems.
When someone is placed in a role in which they don't belong, I call
this being "badly cast." Many of the problems with GK3
resulted from developers being badly cast in their roles, usually because
the project requirements were so severely underestimated. To give you
an idea of the casting problems we had, consider this: we went through
a total of two producers, three art directors (we spent the last year
of the project without one), and three project leads (the producer was
forced to take over as project lead towards the end).
This was
an ambitious, massive project that required experienced engineers and
the original team was simply not up to this task. GK3 was initially
built from members of the Shivers 2 team (one of the last games
built with SCI) and they had practically no 3D experience. Engineers
under the venerable SCI engine were basically scripters and putting
them in charge of building a game engine from scratch was like feeding
them into a furnace. To make things worse, the developers that were
in over their heads didn't ask for help, which gave management a false
sense of progress.
2.
Severe morale problems.
Hundreds of books and articles have been written about this and here
we have yet another Postmortem listing it under "what went wrong."
It's time for me to get on my soapbox. To managers everywhere: morale
is one of those icky personal political things that many of you avoid
dealing with, but you need to understand that your development team
is not a factory churning out content and code. To paraphrase Peter
Sellers in Being There, "The team is a garden of creativity
that requires regular watering and sunshine in order to build strong
roots." Loyalty is not something that comes easily. The job market
is very competitive -- your best developers will simply leave and work
for somebody else if they aren't treated well and maintained properly.
On GK3, there was a serious lack of love and appreciation throughout
the project. Recognition of work (other than relief upon its completion)
was very rare, lacked sincerity, and was always too little, too late.
Internally,
a lot of the team believed that the game was of poor quality. And of
course, the many web sites and magazines that proclaimed "adventure
games are dead" only made things worse. Tim Schafer's Grim Fandango,
although a fabulous game and critically acclaimed, was supposedly (we
heard) performing poorly in the marketplace. Rumors circulated among
our team that GK3 was going to lose money, due largely to our
high burn rate.
The low
morale resulted in a lot of send-off lunches for developers seeking
greener pastures. GK3 had a ridiculous amount of turnover that
never would have been necessary had these people been properly cast
or well treated in the first place. More than 45 developers worked on
GK3 (the average standing team size was 15 to 20), and now, just
a few months after it shipped, only seven remain at Sierra. Strangely,
the opposite also happened -- several of our developers were included
in Sierra's mid-1999 housecleaning layoffs but these individuals were
allowed to stay on for a couple of months, postponing their last day
until we shipped GK3. I believe this was done in good faith out
of respect for the developers' hard work up to that point but it ended
up being a prolonged drain on morale. Having a small group of people
who are (understandably) upset with your company for laying them off
and actively looking for a job while still trying to be productive and
contribute to a project is a tough situation that should be avoided.
After
a certain amount of time on a project like this, morale can sink so
low that the team develops an incredible amount of passive resistance
to any kind of change. Developers can get so tired of the project and
build up such hatred for it that they avoid doing anything that could
possibly make it ship later. This was a terrible problem during the
last half of the GK3 development cycle and as a result there are many
aspects of the game that we aren't proud of. These were problems that
should have been fixed but nobody wanted to take the time to correct
them because we were so focused on trying to get the game out. I don't
think anyone on the team is directly at fault for this and I don't know
what we could have done to correct this problem.
3.
Schedule problems.
Our engineers never had an accurate development schedule -- the schedules
we had were so obviously wrong that everybody on the team knew there
was no way to meet them. Our leads often lied to management about progress,
tasks, and estimates, and I believe this was because they were in over
their heads and weren't responding well to the stress. Consequently,
upper management thought the project was going to be stable and ready
to ship long before it actually was, and we faced prolonged crunch times
to deliver promised functionality. More frequent and honest communication
within the team would have avoided a lot of this.
GK3
had few real milestones, which undermined our ability to track progress.
There were some milestones very early on in development, but the focus
on shoveling visible features into the game turned them into worthless
smoke-and-mirrors demos. The concept of milestones eventually was discarded
and was replaced with two simple and unofficial goals -- beta and release
to manufacturing. In the push to ship the game, we simply forgot about
milestones (because "we're almost there!"), put the blinders
on, and worked like mad.
Crunch
mode is a reality on most projects, but you should not gear up for one
unless the light at the end of the tunnel is really in sight. The GK3
team was pushed into crunch mode three separate times, each time thinking
that we were almost ready to ship. Most of the last year of the project
we spent in this mode, which meant that even small breaks for vacations,
attending conferences, and often even taking off nights and weekends
were looked down upon. It was time that the team "could not afford
to lose." The irony is that this overtime didn't help anyway --
the project didn't move any faster or go out any sooner. The lack of
respect for our personal lives and attention to our well-being caused
our morale to sink.
Because
of the high turnover, GK3 always had a high percentage of developers
new to the team. Faced with mandatory overtime, these new people understandably
felt it was unfair to be "punished" by paying for problems
caused by the original team or things that they felt management had
brought upon itself. GK3 became a black hole that sucked in many
developers from other projects, often at the expense of those projects.
Artists were shifted off the team to cut the burn rate, and then pulled
back on later because there was so much work left to do. Our art team
finally got on track in the last year of the project (thanks to Ray),
but engineering never got a solid schedule together and as a result
we were nearly always late in our feature delivery.
 |
2D
interfaces are overlaid on the 3D scene to keep the player immersed.
|
4.
Engineering problems.
When we rebuilt the game engine, we tried to retain as much of the original
code as we could to get the game up and running again as soon as possible.
With the exception of the G-Engine, this was a big mistake. Nearly every
one of the systems that we kept caused us problems -- they were badly
designed, buggy, inflexible, and should have been redesigned. Some of
these systems never worked correctly throughout the lifetime of the
project and had to be hacked around by the content developers to get
the game to ship.
Specifically,
we had serious problems with the "fidget" system (used by
character models when idle or when involved in dialogue), the character
model's walker, the vertex animation system, and the conversation and
dialogue systems. All of these failed regularly and were regularly "fixed,"
but each bug fix introduced new bugs, usually in the form of hidden
time bombs. The engineers responsible for these systems became very
defensive about the problems with them, and usually ended up blaming
artists and scripters or even other engineers for the cause. Management,
thinking that it would save time, often encouraged content developers
to hack and work around the problems rather than fix them properly.
We should have ripped these parts of the game out and rebuilt them,
rather than continually attempting to work around a flawed legacy design.
It would have saved a lot of time and hard feelings.
We also
faced a lot of problems that were out of our control. Most notable were
the technical difficulties with the DirectX drivers provided by hardware
vendors for their 3D graphics cards and sound cards, but this probably
isn't news to any 3D game developers. These problems were generally
features that were implemented improperly or inconsistently, or just
outright bugs that caused system crashes or hangs.
We also
wasted a few weeks trying to add copy protection. During the final push
to ship, we repeatedly attempted to make Macrovision's SafeDisc product
work with GK3. SafeDisc has a set of special (and we felt completely
unnecessary) antihacking measures that got in the way of the game's
execution. It heavily affected performance, dropping the frame rate
to a third of its original speed and adding strange intermittent freezes
of several seconds while the camera was moving. After getting nowhere
with Macrovision's engineering department, we decided to ditch SafeDisc
and roll our own (which took less than a day to do). This entire process
wasted several weeks of our time and frustrated us all the more because,
apart from this one remaining task, we were ready to ship the game.
Lesson learned: If you are required to use copy protection, don't put
it off until the last month, especially if it's SafeDisc. We weren't
the first game to have severe problems working with SafeDisc and probably
won't be the last, so if you're using this product be sure to do your
homework and try it out well in advance of your ship date.
 |
Gabriel
Knight, from concept art to dashing real-time 3D model.
|
5.
Art and (more) engineering problems.
One of the most expensive mistakes a team can make is ramping up art
before the engine is ready. This often happens at large game companies
because developers need a place to go after they've shipped their most
recent game. Unfortunately, this was a serious problem with GK3
-- artists were brought onto the game while the original engine was
in development, and long before a stable engine was available. They
created content for an animation engine that was untested and in doing
so built up enough inertia that we ended up having to keep the design.
Later we discovered that the engine's design was seriously flawed.
GK3's
animation system is vertex-based, meaning that a model's individual
vertices are animated. Even using some creative compression methods,
this is very expensive in terms of memory usage. Contrast this method
with a typical skinned skeletal animation system, which only requires
that the bones be animated. The worst thing about this system was not
the memory usage, however. It was the impact on content creation, and
the repercussions of this requirement were not fully realized until
the art team was ramped up and churning out models and animations.
 |
Rendering
lasers required a little bit of custom code.
|
GK3
animations are completely coupled to the meshes of the models that they
affect. Once an animation is exported from 3D Studio Max, the models
it involves cannot be changed in any way, otherwise existing animations
created from the old versions of those models would break. Vertices
can't be added or removed, and texture maps can only be tweaked, not
remapped. Changing a model required re-exporting every single animation
that affected the model, which was very time consuming, tedious, and
repetitive for animators, and was often impossible to boot. The source
assets (the original Max files) had a way of getting lost and usually
ended up stored on backup disks as artists left the project. The end
result was that once a model was created, it could never be changed.
Consequently, GK3 shipped with a lot of bad
art that the team was dissatisfied with but had to use. An example of
this was the Mosely character (sometimes not-so-fondly called "T-Rex
man" internally), whose arms were way too short. Just seeing this
guy in the game hurt our morale, and made a lot of us feel poorly about
the game. The art was bad and there was nothing we could do about it.
Nearly every attempt to change a model was met with the response, "That
will require re-exporting all of its animations, which will take too
long."
What
we should have done was just stop the presses and fix the art. When
we first recognized the severe problems with the vertex-based animation
system, we should have rebuilt that part of the G-Engine and replaced
it with a simple skeletal animation system (as the SWAT 3 team later
did on their project). We should have also thrown out all of our existing
art and recreated it. In retrospect, that would have saved us a lot
of time, given us better performance, and made development proceed more
smoothly.
Out
The Door
Gabriel
Knight 3 was an extremely ambitious project combined with an extremely
inexperienced team. With all that went wrong, you would have every reason
to believe it would never have been finished. Despite all that was messed
up with our development process however, we somehow managed to get GK3
out the door and ship it in time for Christmas 1999. Best of all, the
game works and it works well, against all odds. This is a tribute to
the testing skills of our QA lead, Matt Julich, and his team. Though
severely understaffed, they did a great job ensuring that the game
we shipped was of high quality. There are no crashes and no hangs in
GK3 -- Sierra will not be issuing a patch.

|
Concept
and final environment for the end game.
|
This isn't
to say that GK3 has no problems, just no problems that we had
any real control over as developers. To the best of our knowledge, every
single critical problem with the game is caused by either bad hardware
drivers (sound or 3D card) which can be fixed by upgrading them, or
CD-ROM problems due to our oversized CD copy protection. Also, the first
disk that shipped in some retail boxes was difficult to read on some
CD-ROMs, apparently due to duplication problems. This can also be worked
around, as the same file exists on the second disk and can be copied
to the hard drive.
GK3
is certainly not the game it could have been had we been able to knock
down a couple of those "what went wrong" issues early on,
but it is a high-quality, entertaining game that has been well received
by adventure gamers. And that's something we can all be very proud of.
Game
Data
|
Gabriel
Knight 3
Sierra Studios
Bellevue, Washington
http://www.sierrastudios.com
Publisher:
Activision
Full-Time
Developers: More than 45 total, averaged 20 at a time.
Contractors:
3
Budget:
Originally well below $2 million, final budget $4.2 million.
Length
of Development: Almost 3 years.
Release
Date: November 1999 (over a year late).
Platforms:
Windows 95, 98, NT, 2000.
Hardware
Used: Lots of expensive stuff that quickly became obsolete
-- Dual Pentium Pro 200s, Pentium II 266 up to Pentium III 500.
3D cards by Matrox, Nvidia, ATI, and 3dfx, and whatever else we
could get our hands on.
Software
Used: 3D Studio Max, Character Studio, Visual C++, SourceSafe,
CodeWright, VTune, MKS Lex & Yacc, Photoshop, special "sound
guy magic."
Notable
Technologies: DirectX, Bink Video, Standard Template Library,
LZO and zlib (free, open-source compression libraries).
Project
Size: 350,000 lines of C++ (not including original engine),
40,000 lines of script and logic, 8,000 lines of (spoken) dialogue,
36,000 unique resource files, 1,500 average triangles per actor,
185,000 cups of coffee consumed
|