Postmortem: Lucas Arts' Star Wars Starfighter
August 1, 2001 Page 2 of 3
What Went Right
1. Good team communication. I've read many Game Developer Postmortems that blamed failures on a lack of communication, so I'm particularly proud that we got this one right. From the beginning of the project, Daron worked hard to impress on the programmers that it was the level designers and artists that would ultimately secure the success of Europa. As I became fond of saying, programmers build the picture frame, but it's up to the rest of the team to provide the most important part, the picture.
To bring this to fruition, the programming team needed to understand as best we could the way the rest of the team worked. While Andrew worked with lead artist Jim Rice and our world-builders to understand their workflows, Brett Douville, our AI and mission programmer, filled a similar role with the level designers. Brett scheduled regular "LD Days" with each individual member of the level design staff. This gave each designer the opportunity to meet with Brett on a regular basis and show him the specific challenges and problems that they were tackling in their missions.
This image depicts the design schematics for one of the Lok missions. Level designers made elaborate plans such as these for every level that appears in the game.
Europa periodically had full-blown team meetings where we could get together and kibitz about the overall state of the project. However, the most valuable meetings were at the subteam level. Both the programming team and the art teams would meet weekly to discuss the issues of the day, and each of these meetings would have an attendee from the other camp -- a role we referred to as the "exchange student." This meant that if questions came up in the art meeting, for example, that required answers or input from a programmer, there would always be someone present that could give an informed opinion. Likewise, as programmers would discuss issues or new features in their weekly meeting, the art or level design representative would be able to disseminate this information among the other team members.
Finally, we relied on an internally maintained web site as a pivotal communications tool. We tried to make the site as comprehensive as possible, organizing areas along the lines of programming, art, level design, project management, and so on. When artists had questions about how to implement a particular effect, or a level designer needed a refresher on our class-file script syntax, there was usually a web page that they could be directed to that would answer many of their questions.
2. Project discipline. Star Wars Starfighter was a well-organized project. In the heat of battle it's all too easy to let requirement lists and schedules get lost in the shuffle of the moment. We were determined not to let this happen. As soon as our technology began to take shape we started to follow an iterative process of milestone planning and execution. These milestones were typically four to six weeks in duration, with no milestone extending longer than eight weeks. Milestones were also required to demonstrate some visual or gameplay aspect of the game. As a consequence, we had very few milestone tasks that looked like "complete the Foobar class"; instead we would have a milestone task that might read "Explosion smoke trails," and the assigned programmer would know that completing the Foobar class was an implied requirement. By keeping our attention focused on a discrete and relatively small body of work, we were able to avoid the cumulative errors that invariably creep into longer schedules, while still allowing for demonstrable progress.
Most of the milestones were driven by the progress of the technical team. Programmers were solely responsible for estimating the duration of their tasks. We would occasionally adjust these estimates outward but would never change an estimate to be shorter. Tasks were structured so that the shortest scheduled task was never shorter than a half-day. Even if a programmer was certain that a task could be completed in less than half a day, experience clearly showed that the time would be lost elsewhere. Using these simple rules of thumb, we were consistently able to build schedules that were fair and accurate. Out of eight scheduled milestones, we never missed one by more than a handful of days. Best of all, most team members completely avoided extended periods of crunch time. Like most game teams as they approach their ship date, everyone was working hard and often into the evenings; however, this period of time was short, and we never had to resort to all-nighters.
We also closely managed the process we used to distribute new binaries to the team at large. Since most of our development occurred on the PC even after making the decision to ship on the PS2, it was important that team members have timely access to stable builds of the game. We accomplished this through weekly public builds. Once a week we would package and distribute the current code as a full-blown Install Shield-compiled install. This provided team members with debug and production versions of the game, along with level design tool and art exporter updates. Predicting that public builds would become critically important, we tried to be as ruthless as possible about maintaining the build schedule. As we got closer to our ship date, the frequency of these public builds increased until we were performing new builds as often as three times a week. By this point we had a full-time staff member dedicated to managing the public build process and ensuring that the distributed code met quality and functionality expectations.
3. A well-executed PC-to-PS2 transition. Making the decision to move the project to the PS2 could have been a complete disaster. Yet, despite paying little attention to portability during the earliest stages of the project, the Europa code base was well positioned to make the jump to the PS2 platform. With the aid of strong and generally stable development tools provided by Metrowerks, the core port went off without a hitch. The biggest trick was on the graphics side, because this was clearly where we were most vulnerable. None of our programmers had any console experience, and none of us was up to the task of tackling the PS2's infamous low-level vector units. Enter LEC gl.
LEC gl was the brainchild of Eric Johnston and Mark Blattel, two of Lucas Arts' most senior console programmers. They had recently shipped Star Wars: Episode I Racer for the N64, and they welcomed the opportunity to tackle a problem temporarily that was one step removed from the day-to-day pressures of a project team. Although Europa was the most immediate recipient of their efforts, Eric and Mark were never officially on the project. Instead they worked in a support role, providing us with regular LEC gl library drops and immediate "on-call" PS2 graphics support.
There was another, more subtle problem that we had to conquer when we made the decision to adopt the PS2 as our primary platform. Most of the team members were big PC game players, but very few of us played console games. Intellectually, we knew that there were huge philosophical differences in game design between consoles and the PC. Much of our original game design had used the X-Wing games as a conceptual leaping-off point, wandering into the arcade action of i only when it suited us. Now that we were on the PS2 we recognized that our design priorities needed to be completely flipped. Instead we would use Rogue as our primary point of reference and work from there, layering on gameplay elements borrowed from X-Wing as needed. As such, I think the final game demonstrates our successful indoctrination into the console mindset. We were having so much fun blowing things up that we had little desire to start adding sim-like features to the gameplay experience.
4. Macromedia Flash. As we approached the end of summer in 2000, we realized that we had a serious problem on our hands. Despite our best efforts, we still had not addressed the issue of our out-of-game user interface. We had a 2D virtual-page system that we were using for our HUD (heads-up display) symbology, and we had always planned to evolve that into something that could be used for what we called the "administrative interface." However, in August, with the quality assurance department nipping at our heels and our ship date looming ominously in the distance, things were not looking good.
We had heard that a small San Francisco-based company named Secret Level was adapting Macromedia's Flash technology for use in PS2 games. After meeting with company representatives, we were excited by the prospect. The Macromedia content-authoring tools were far more elaborate than anything we could come up with in the same time frame. We also suspected that there was a wealth of Flash authoring expertise available from out-of-house contractors, which would help us smooth out the work load. Most importantly, we were very impressed by the intelligence and games savvy of the Secret Level staff. When we realized that building our user interface in Flash would significantly ease our localization efforts, we decided to take the plunge.
Early concept art for the mercenary Vana.
Soon afterward, we hired a design firm named Orange Design to help us implement our administrative interface in Flash. Orange not only had a ton of experience with Flash, but they also brought a technical perspective to the table. We knew that this technical emphasis would be critical for working with our programming team on integration issues.
Integrating Flash into the Europa engine was not a completely smooth process, however. Performance in the first-generation Flash Player was poor (current generations of the Player are now much faster), and we had to spend a lot of time integrating the user interface Flash movie with the core game systems. That said, the five months that a single half-time programmer spent on this task ended up yielding a user interface that was far beyond what we would have been able to custom-code in the same period of time.
5. Good debugging systems. Our programmers built several tools that greatly helped our pursuit of high-quality code. One of the most instrumental was a Windows-only library that provided detailed stack-tracing information. This library was largely based on the code and concepts covered by John Robbins' "Bugslayer" column published in the Microsoft Systems Journal.
As is standard practice on many games, we built a custom memory manager that could detect when the application was leaking memory. However, unlike most implementations, when our memory manager detected a leak it could provide a comprehensive stack trace of arbitrary depth, leading directly to the leaking code statement. This capability represented a significant advantage over other implementations that could only provide the immediate location of the allocation request. If the memory allocation was being made by the Standard Template Library (STL) or one of our widely used utility classes, it was usually not enough to know what part of the STL or which one of our utility classes was the culprit. What we really needed to know was what class called the STL method that caused the leak. In fact, the leak was usually several steps up the call chain. Our stack-tracing library made finding these cases almost trivial.
We also incorporated stack tracing into our exception- and assert-handling systems. When the game encountered a hard crash, we trapped the exception and generated a complete stack trace; a similar process occurred when our code asserted. This information was initially reported back to the user in dialog form. However, we also packaged up this same data and had the game send the programmers an
e-mail detailing exactly where the problem occurred. This ended up being an invaluable tool for us. As a matter of practice, the Europa programmers got into the habit of checking the assert mailbox regularly. In addition to appraising the current stability of the code, we could also use this data to spot trends and note when people weren't being diligent about installing new builds.
In the end, we had an exceptionally smooth QA process because the bugs we did have were generally easy to track down and fix. There were no last-minute "heart attack" bugs that required us to set up camp and track a single problem for hours or days at a time. This made life easier on the programmers, but it also made things easier for the testing team and improved morale across the entire project.
Page 2 of 3