     |
 |
 |
 | | | | |
 |
|||
 |
||||||
          |
 |
 |
||||
|
Postmortem: The Collective's What Went Wrong 1. Lack of resources and burnout. With Buffy so far past its ship date and consuming all the company's available resources, we were unable to obtain enough people to develop The Emperor's Tomb. We were able to temporarily scrounge developers from the Buffy team, but getting dedicated staff was initially impossible. By the time we completed the prototype, we were significantly behind schedule. And although we were entering full production, we didn't receive additional full-time staff for a number of months. This meant that the team leads had to assume a staggering amount of production responsibility. We were working "crunch mode" hours at the beginning of the project, yet the schedule was still slipping.
To complicate matters further, the Buffy team occasionally borrowed some of our limited resources when a critical milestone was approaching. The Indy team saw where this was heading and we knew there would be trouble. Months passed while all of us suffered under the massive workload, and it became difficult to remain objective about the company's handling of the situation. We simply needed more people, and the company could not provide them while Buffy was still in development. And nobody knew for sure when Buffy would be finished. The problem precipitated a secondary effect: employee burnout. At the beginning of the project the leads decided that we would do our best to avoid making people work extra hours. This was mainly because we worked with the Buffy team and knew we would be building our team from a pool of very tired people. But the veterans who came onto our project were transferred from one game in crunch mode right into another one. A number of them ended up splitting their time between the two projects. We managed to balance things out so the people almost always got at least one full day off each week. But even at this rate, it was difficult to avoid burnout. And we were still slipping. I believe it is impossible to do anything for 12-14 hours a day, six to seven days a week, for months on end, and not get sick of it. It is true that making games is a passion. But one thing many employers in this industry do not seem to realize is that it is also a job. A job has set hours. A job has set responsibilities, schedules, and expectations. And one's job cannot be the entirety of one's life. Desperation soon led to a third problem. When we did get permission to interview people for the vacant positions, finding the time to do a proper interview was almost impossible. So most interviews weren't thorough enough. It was not uncommon for one or more of the interviewers to be reading the applicant's resume for the first time during the interview. Many of the people we brought into the company ended up being very good hires, but a couple disastrous hires set us back further than if we hadn't brought anyone in at all.
2. Reach exceeding grasp. One of the drawbacks to working with the popular Indiana Jones license was that we simply had too many ideas for the game. Our original plan called for over sixty levels with almost one hundred hours of game play. Ideas included giving Indy two whips instead of one (one would be a magical grappling hook), a variety of game levels using various types of transportation (on foot, rail-mounted vehicles, boats, and a whole mystical netherworld), countless varieties of thugs and enemies, and an arsenal that rivaled that of the United States Army - including bazookas, flamethrowers and numerous pistols and machine guns. The game was thoroughly designed and documented, more so than any game I have worked on before or since. When we were writing these documents, everything seemed feasible. It was only as we began attempting to get items into the game and began facing problems that we realized it wasn't all possible. There are a few reasons for this. When writing up the schedules we weren't allowed to schedule time for management, which quickly became a problem. Time spent dealing with personnel issues, meetings, and anything not directly production-related became lost time that had to be made up. And as I explained earlier, we were short-staffed from day one. Not only did this cause problems in completing assigned tasks, it also delayed other tasks when someone had to assume an extra burden. It caused problems with deliverables because everything was rushed and stressful. Another problem was that the senior staff for the project were
all new hires. None of us knew enough about the existing technology
to draw educated conclusions, so we had to rely upon the word of
people who had been there longer than us. It is human nature to
overstate one's ability, and we probably accepted a few things at
face value that we shouldn't have. This problem was compounded by
the fact that nobody in the company was intimately familiar with
the PS2, which resulted in some incorrect assumptions about the
capabilities and limitations of that system. Perhaps the most aggravating aspect of the code was the amount of systems which were implemented directly, as opposed to data-driven solutions. Very little of our game behavior could be tweaked on-the-fly, which made even subtle changes a lengthy process. Direct implementation is frequently used when people are under pressure to deliver functionality quickly. It generally solves the immediate problem, but causes trouble when attempts are made to extend or alter functionality. Unfortunately my team wasn't able to do much better in this area, because we were under the same kind of pressure and time restrictions. In some cases we actually exacerbated the problem by extending systems which were alrady cluttered. This is one of the areas of software development that it is easy to soapbox about. We all know that well-designed code is better than the alternative. But in reality situations arise in which doing a complete design is not always possible, or design assumptions made at the beginning of a project later prove to be problematic. For example, one much-griped-about aspect of the technology we used as our base was the lack of a dynamic save system. The Slayer engine was written without this on purpose, under the assumption that the system would be too difficult and error-prone to effectively maintain. There is no doubt that such a system is complex. But the cost of not integrating it from the outset was very high, as it is virtually impossible to graft into the engine late in development. This forced both Buffy and Indy to become checkpoint-based games. If you die at the end of a level, you have to replay that level from the beginning. This lowered the review scores of both games and upset a number of players. For some team members, being forced to work around existing problems
was just too much. We lost one very talented artist (and friend)
partially because he wasn't generating art -- he spent all his time
wrangling with our convoluted attachment files, trying to properly
align slotted items to the character's body. Everyone knew this
system was broken, but nobody could spare the time to fix it. Other problems, and key staff departures, would arise during Indy's development. These problems all had to be worked around, which usually meant senior staff members had to attempt to learn a system quickly and take responsibility for it. Many of these systems were frustrating and time-consuming to work with. Even the simple task of writing a GUI was a grating process of placing an object, seeing if it looked right, then editing the code to move it a pixel and running the process over again. I actually had members of my team threaten to quit if I gave them this task, and I ended up doing the code for the GUI myself. It is important to state that this section is not meant to be a knock against any of the people at the company because it is certainly understandable how everything came to be in this state. There was a definite desire from the top of the company to the bottom to make things better, but with the workload we simply never had the opportunity. Nevertheless, these were problems we had to deal with on a daily basis, and they cost us an inestimable amount of time.
4. Problems with the PS2 port. Indiana Jones and the Emperor's Tomb was originally planned as a four-SKU game. The GameCube version was cancelled. The Xbox and PC SKUs were comparatively trivial, since the Xbox is so similar to a PC and the development tools are easy to use. But because we were doing a PS2 version too, we had to limit the number of cutting-edge features we hoped to include. For example, we originally planned to do real-time shadowing on almost every item in the game, including self-shadowing for the characters. This had to be somewhat limited due to the lack of a stencil buffer on the PS2 and the general slowness of the stencil shadowing algorithm. Lighting had to be greatly simplified for the PS2 version of the game, and even when the lighting system was scaled back, it was still very slow. These types of compromises are common in multiplatform development. They resulted in a game that, while still very attractive due to excellent work by our artists, level designers, and sound engineer, did not exactly push the envelope in the visual or audio sense. The PS2 SKU was quite an experience. As late as eight months prior to the shipping of the Xbox version, we still had no functioning PS2 technology. There were no programmers in the company with solid PS2 development skills, and attempts to get such programmers were not going well. At one point we managed to hire an experienced PS2 programmer who started working on the port of our massive code base. But after six months of effort, with little progress, he left to pursue other opportunities. We were under tremendous pressure for the PS2 version because, with the largest installed base of all consoles, that is the money-making SKU. So in addition to attempting to recruit permanent staff members with experience in this field, we began seeking out hired guns -- often at great expense to the company. Some of us started to learn PS2 programming, but the PS2's Vector Unit code can be very confusing, even after a full night's sleep. Thankfully the PS2 engineers we hired were very good, and they managed to get a close approximation of the Xbox version of the game onto this much less powerful console. Unfortunately, the PS2 port came together so late that some major problems couldn't be fixed. Some textures which looked fine on the Xbox and PC looked awful on the PS2, and there was little we could do about it due to time constraints. The levels were frequently too large to fit in memory, and the resources in general took a considerable amount of time to load, resulting in lengthy pauses. Our animations used a staggering number of bones. There were too many unique character variations and too many assets in general for the limited memory the system provides. All we could do was cut, mangle, and squeeze as much as possible. These were all difficult hurdles to clear and all contributed to the PS2 SKU shipping last and with the most problems. 5. Communication and morale. Early in the project, the tone was set between the leads and the owners of the company. Each week we would have a meeting where we would examine the schedule and assess how far we had fallen behind. The leads would point out that we were lacking the development staff that had been expected to join the team, which was the primary cause of the slip. The owners would counter this argument by telling us that there was nothing they could do about the situation, and that it was up to us to "make up the difference". After a few months of this, and seemingly endless crunch time early in the project, it became difficult for the leads and owners to converse rationally. They just wanted the work done; we just wanted the proper manpower to do it. Conversations became tense, there were some arguments, and after a while we began to subconsciously avoid one another. Unfortunately, this adversarial situation combined with the intense workload to cause a complete communication breakdown between the upper tiers of management. Although the project leads conversed on a daily basis, we rarely spoke to, or heard from, the owners. Morale at the company was less than stellar. The general attitude had become very critical over the long course of Buffy's development. After all, it is difficult to maintain excitement and enthusiasm indefinitely. People had become quick to notice and point out flaws, and there was very little positive feedback to offset the constant criticism. In particular, many people were fond of what I call "drive-by riffing": someone would walk past someone else's desk, see what they were working on, and point out the first visible flaw. Unfortunately this behavior was extremely contagious, because everyone likes to be the center of attention and to make other people laugh. But almost nobody likes to be made fun of. We desperately tried to fight this destructive attitude, but it was a constant battle. For the first part of the project we tried to build team unity by having progressive build demonstrations for the team. We would gather everyone in a large meeting room and play the game, showing off new features and levels. But, like open-mic night at the local comedy club, inevitably the mean-spirited jokes would start and feelings would be hurt. Eventually we stopped having the meetings because it just seemed people would only laugh at the flaws, rather than enjoy the good parts. A Frustrating, Yet Rewarding, Experience Looking back, making Indiana Jones and the Emperor's Tomb was both the most rewarding and the most frustrating experience in my career as a game developer. Prior to coming to The Collective I had never had a project canceled. I had never had a project struggle or go through such difficult times. Obviously this can happen at any studio, and I was unquestionably lucky to not have gone through it before.
In many ways I feel this experience made me a better lead and a better programmer. Having worked through such adversity I now know some of the problems to look out for, and I have experience working through difficult situations. There are some salient lessons to be imparted from this experience. First, it is never a good idea to make ambitious plans based on resources you do not currently have. Plan based up the people you have at the moment, and if necessary, expand the plan when more help becomes available. It is irresponsible management to expect an understaffed team to pick up the slack forever. This only hastens a team's disenchantment with their work. Second, one cannot overestimate the importance of the team being completely involved in the development of their game. An unmotivated workforce will not do quality work. Despite what seems to be a common misconception, it is the responsibility of a company to ensure that its workers are happy in their roles on a project. Ignoring this fact is only harmful, never helpful. The amazing part is that it usually doesn't take much to make an employee happy. Listen to their grievances. Solve their problems when you can, and explain things when you can't. Pat them on the back when they do a good job. It pays off greatly in the end. Third, and what I consider to be the most important, is that planning is essential. Good management is so often overlooked in this industry. It is the part of our job we see as a grim necessity. We need to mature beyond this misconception. Spending one hour on management often means saving ten hours of debugging poorly-designed code, or days reworking botched assets. It is critical to lay out your plan entirely before beginning, and even more critical to be able to alter that plan as development progresses. What I find the most interesting about this project is the fact that there were so many problems, so much that didn't go right; and yet we still managed to ship a solid game very close to our original schedule. This is entirely the result of having a great project concept and a team who cared about what they were doing. Even when times were at their roughest, everybody knew this would be a good enough game to justify pushing through to the end. I worked on many games prior to taking the Lead Programmer position on Indiana Jones. Even with the problems this one was a great experience due to the talent and passion of the people with whom I was able to work. I can truthfully say that I was never more proud to see one of my games on a store shelf. My appreciation goes out to all the people on the Buffy and Indy team whose hard work, patience, and dedication made this game possible.
 [back
to] Introduction |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Features
|
|
|