Contents

Staffing Up What Went Right What Went Wrong

What Went Right

1. We maintained a conservative game plan.
Most projects are too ambitious in this area, leading inevitably to painful feature cutting as ship dates loom. However, because we scheduled so pessimistically, we were able to survive even with fewer programmers than planned and actually get "wish list" items into the game. Our original plans were for a single directional light on the skater and keeping the straight-down opaque black shadow from the Playstation version. However, our conservative scheduling provided us with some free time to add features such as dynamic lighting and realistic shadows. I had something to prove with this project, which I learned from reading Steve McConnell's book Rapid Development (Microsoft, 1996): it's better to schedule pessimistically than optimistically. As our producer Greg John puts it, "Underpromise, overproduce."

We scheduled the project out in weeklong chunks and even though some programming tasks seemed like they should only take one or two days, we nevertheless gave them a whole week. When Don Likeness, Treyarch's founder and no slouch of a programmer himself, saw the schedule he said, "Why is this going to take a whole week? I could code this up myself in a day!" But we stuck to our guns.

This system turned out to work great, because even though it would often take a programmer only one day to implement a feature we had assigned him a week to do, it then usually took a couple more days to get each feature to the state where we could put it in a box and ship it -- we didn't settle for features that were 99 percent implemented. And then it took a couple more days to fix any new bugs that had cropped up.

For scheduling my own programming tasks I was even more pessimistic. This is the first project I've been lead programmer on where I wasn't also the only programmer. After seeing how this company tends to give its lead programmers ulcers, I decided to do a couple of things: For any task assigned to me we doubled how long we thought it would take, running under the assumption that as lead programmer I would be so distracted I'd have a hard time getting anything done. We also didn't assign me anything tricky: just because I was the most experienced game programmer on the team didn't make me the best. And even though I had written a Dreamcast renderer before I didn't take on any mission-critical rendering tasks. I basically did crap work, the front-end stuff and memory card stuff, and left all the graphics and special effects and sounds and optimizations to the other guys, which not only made my life easier but also helped to keep them challenged and motivated. Still, even with my extra-pessimistic schedule, I barely managed to stay on top of my programming tasks.

How did we schedule optimization? The goal was to be at 30 frames per second, always, even in two-player split-screen mode when both skaters are being rendered in both windows. To track the scheduling progress we took one of the worst case scenarios, found the frame duration (it was around 60 milliseconds and we had to get down to 33) and made a rough estimate of how much progress Srini and Wade would need to make each week to hit 30FPS by alpha. Knowing that optimization gets tougher as you go along, we assumed that each week we'd make about half the progress of the previous week: trim off 16ms the first week, then eight, then four, and so on. Srini and Wade managed to stay well ahead of this wild guess, so we didn't have to worry. Hitting 30FPS was easy but hitting 60 would have been very difficult. (That would have meant cutting the worst-case frame time from 33ms to 16.) We estimated how much we'd have to do to get past limitations in the code that prevented the Dreamcast from hitting its full stride and it ran out into several weeks. Since we weren't doing a fighting game, all that 60FPS would have gotten us would have been a nice marketing bullet point.

We were often wrong in our scheduling estimates, but the mistakes balanced out. Sean would see things on his schedule that he was pretty sure he could knock out in a day and to prove it he would just go ahead and do it. So we found ourselves readjusting the coding schedule every couple of weeks as we learned that something we had scheduled a week for could be banged out in an afternoon (getting the sky rendering correctly, for example). But we also learned that things that we thought would take two weeks actually took four, such as the game's new shadows. We ended up with fewer programmers than we had originally planned for, but our pessimism weathered that storm as well.

Greg didn't use any funky "Project"-oriented software to do our scheduling, just Excel: programmers across the top, weeks across the side. One box represented one week and if he needed to reschedule, he would just cut and paste boxes -- we could have just as easily used three-by-five cards and corkboard.

In the end, we finished one week early and we could have finished two weeks early if we had better QA. We could have saved two more weeks if we didn't have to wait for Sega's R10 library, skipped implementing dynamic lighting, and left the shadows the way they were in the demo. (The demo version of Tony Hawk for Dreamcast, which ships with new Dreamcasts now, has shadows that have some artifacts). I ask myself; if we hadn't had to wait for R10, would it have been better to schedule a little more optimistically, lose those features, and shave those last two weeks off the project? I don't know. So it's possible we were too pessimistic. Still, having now been on projects that were both underscheduled and overscheduled, I'll take an overscheduled one, hands down. We had to put in some overtime, work a weekend here and there, but compared with other projects I've been on, it was a vacation.

2. We had a Playstation dev kit on hand and kept the Playstation build going and stable well into the middle of the project.
This one seems like common sense. Start with a project that builds, runs, and work from there. It's tempting to skip this step since you don't technically need a Playstation development board to do a port to a different platform and because they're so pricey you might rather go without. There are obvious ways in which having the Playstation build on hand helps: First, you can step through the two versions and see why the behavior on the Dreamcast doesn't work when the behavior on the Playstation does. Also, if someone discovers a bug on the Dreamcast version, you can check to see if it's a Dreamcast-specific bug or if it's something that you also introduced on the Playstation as well. For example, we'd implement a new feature on the Dreamcast side and discover something wrong with it, and then we'd recompile on the Playstation and see if it was a bug we introduced or a bug due to just not completely implementing or understanding the feature. It's very unpleasant having a code base that just doesn't work and you don't know why and you don't know how much work you have to do to get something that does work. Having a working build on hand gave me the confidence to know that if something did go wrong, we could figure out why.

Eventually we got to the point where the time taken to make sure it would build and run on the Playstation wasn't worth the effort. This was about halfway into the project. After we let it go, there were a couple of times that I regretted not having the Playstation build anymore to check things with, but we survived.

3. Daily builds and smoke tests, and an internal bug list.
This is discussed in Frederick Brooks's, The Mythical Man-Month as well as Rapid Development. Every morning (well, almost every morning), I would do a get of our programmers' latest changes and artists' latest data, make sure it built, and make sure all the levels loaded and ran. This process would take an hour or two out of my day, but we caught severe bugs almost immediately and dealt with them. So we almost always had a solid code base, and were never hit by the problem (all too common on other projects I've worked on) where a milestone was due and we would get ready to submit it and discover that half of the levels didn't even load because they hadn't been tested in a couple of weeks. Things broke in our code base quite often, more than once a week I would say, but any bug that was accidentally introduced would get caught by this process. I'd also do soak testing overnight as well, to make sure memory leaks weren't a problem.

Everything near the skater is rendered in two passes: one for the geometry, and another to project the shadow texture on to the geometry.

We had an internal bug list from the start. Those of us with development systems were encouraged to add bugs to the list by submitting them to Greg when we discovered them. We also would occasionally have people come in and do a little in-house QA, about one tester-day per week. A lot of the bugs on the list were nonfatal bugs that were due to us not having implemented a certain feature yet (or ones that we weren't planning on fixing at all). These were the bugs that we left until later to fix but everything else we got on as soon as possible. The "A" bugs were the kinds of bug that we'd stop what we were doing to fix, but as for lower priority bugs, we'd got to them as soon as we were at a good stopping point.

When we got to alpha, (feature-complete except for the movies streaming into textures), we shared our internal bug list with Crave, creating a master list which their QA and our QA could both add bugs to. This may not have been such a good idea because it meant they had to regress our bugs as well as theirs, and they could have spent more time looking for new bugs if they weren't so busy regressing bugs that we had already regressed internally.

It took six weeks from when we were feature-complete to when we made our final submission to Sega, and the bug count reached nearly 800, which surprised Crave since most of the ports they had seen had far fewer bugs. Part of the reason for this was because we were reporting bugs to the master list rather than fixing them internally and more than 200 of the reported bugs were discovered internally.

4. Continual communication between all team members.
We kept the Tony Hawk Dreamcast team fairly localized in the building. Sean and Wade shared an office and the artists were in adjacent cubicles. I spent the first few weeks in Srini's office while we were getting the game up and running for the first time and I think having Sean and Wade in the same office helped make shadows as good as they are; they got to throw ideas back and forth about how to make them look the best. We all had our own phones. I was also reachable by cell phone and Greg had a pager.

On top of that, there was "pair programming." Pair programming is discussed in Larry Constantine's On Peopleware (Prentice Hall, 1995) and also on the Extreme Programming site (http://c2.com/cgi/wiki?ExtremeProgramming). I'm a big fan of pair programming because I think it's almost as beneficial as doing code reviews. The advantages of pair programming are that it keeps you focused on the current problem because you have somebody looking over your shoulder, the programmer looking over your shoulder can spot bugs and bad practices in your code, and, since most programming is actually problem solving, having two sets of eyes working on a problem means the problem might get solved that much sooner. The disadvantage, of course, is that those two programmers could be working on two separate features in parallel instead of being tied up on one feature together.

We did a lot of pair programming in the beginning of the project. We pair-programmed the first draft of the renderer to get flat-shaded levels up and running, and the construction of the Playstation emulation math functions. I would team up with someone when the feature they were working on started to slip on the schedule and we would also pair-program some of the more involved bug fixes at the end of the project to make sure we didn't screw up.

5. Marketing requirements were usually requested with sufficient lead time.
This is something that's so easy for developers and publishers to forget. Suddenly you're two weeks out from having to get a demo going for a magazine but you have no time scheduled for it and the game isn't ready for prime time. For us, it was understood from the beginning of the project that we'd need to have a demo running halfway through, and thus we scheduled ample time for it. Because we had the demo running so early, we were able to get it on the disk that shipped with Dreamcasts, without having the demo impact our schedule.

 

________________________________________________________

What Went Wrong