Resource Guide

Production Testing and Bug Tracking

Should We Even Have An Alpha Phase?

Some have suggested we completely discard alpha phase, on the grounds that alpha encourages sloppiness earlier in the project. Why can't we slowly and steadily implement high-quality features all the way up to ship? I think that's a bad call, for a number of reasons:

• A game is a holistic experience. You need to get that entire holistic experience finished early enough that there's some time left to tweak, balance, and polish.
  
• Alpha is usually when the publisher starts to pitch in with testing. The earlier you can have them help test, the better.
  
• Alpha provides a safety net for the bugs you don't catch in house, or the bugs that you don't see because you've become too close to the project.
  
• Alpha gives you an optimistic goal for when you'll be done by. Your goal as a developer should be to have the game ready to put in a box by alpha; the alpha phase then becomes the safety buffer by which you can miss that goal. (God forbid you finish the game early.)

So, given that we still have an alpha phase, how do we manage it? That brings me to the following rule: After feature freeze, use your bug-find and bug-fix rates to estimate your ship date.

Once in alpha, you may want to have some idea if you're going to be at "zero bugs" on time. (As I hinted before, "zero bugs" is a nebulous concept, as your criteria for what you consider a stop-shipment bug is going to gradually get more and more stringent as you get closer and closer to your ship date. It's almost as if, to quote Jim McCarthy's Software For Your Head, bug count is a constant.)

You might think you could take the entire bug list, ask everyone how long it's going to take to fix their bugs, and be done. You shouldn't do this, because:

• The time they spend estimating is time they spend not fixing bugs.
  
• The aforementioned evidence that programmers work most productively when there are no estimates.
  
• According to Steve Maguire in Debugging the Development Process, bug fixing is notoriously hard to estimate. I can vouch for this one: when Die By The Sword was in alpha, I was convinced that I didn't have enough time to finish all the bugs on my plate, and had a lot of them assigned to others. Then I finished my list in record time, and asked to have those bugs reassigned back to me. I might as well have kept my mouth shut.
  
• At any given moment, your open bug list is a small fraction of the long list of bugs that remain to be found and fixed; your estimate is only going to represent how long it takes to fix the current set, not the total.

Rather than attempting to schedule bug-fixing, Greg John uses this process on our projects. The way it works is each day you count how many bugs you have in your database, and make a chart. It should, ideally, be a curve that shoots up rapidly after the product goes into testing (and by testing I mean publisher-side testing), hits a peak, and then trails off towards an asymptote of zero. While the curve is still shooting up, the way to estimate your ship date is to make up a guess as to how many bugs you are going to have, total. You can do this by looking at your previous projects and extrapolating. If you don't have previous projects, now's a good time to start gathering this kind of data, but until you have the data, use Greg's rule of thumb: take the number of people-months that have gone into the project and multiply by ten. (In other words, every one of us introduces an uncaught bug every three days.) At your shop, your number will quite likely be different, depending on your process and your testing team. It could vary from under a thousand (LucasArts), to three thousand (Lionhead Studios), to eighteen thousand (us.)

Early Alpha: Bug Count Is Rising

	NHL 2K3

At first, you're in what Chris Busse, producer of NHL 2K3, calls the "Bugs are like fruit on the ground" stage. In this stage, you can't play the game for more than a couple of minutes without hitting a stop-shipment bug. When the game is in this state, the testers aren't going to try to do tricky things to break the game, like force their avatars into tight crevices where they might drop out of the world, or find some way to throw the thug who has the key to the waterfall onto the other side of the waterfall. (This exact bug was revealed in our last project, after we shipped, by the guys doing the localization for Japanese. They have some good testers. They sent us a videotape. Thanks guys. Why don't you just give us paper cuts and rub lemon juice in them?) When you're in this stage, you are still at least a month from being done, and probably two months. A lot of developers start blaming the testers for doing a poor job at this point. I have been guilty of this sin. "Why aren't you guys finding the tough bugs? Why didn't you find this bug sooner?" The answer is because they were so busy writing down things like, "Game crashes when you try to punch thug," they didn't exactly have time.

During this phase you are ascending the bug-count curve: testing is finding bugs faster than you're fixing them. In this stage your resources -- the developer resources -- are the bottleneck. Some overtime should probably be mandatory during this period, as it's one of the only ways to bring the project in sooner. You may even scrounge up people from other teams at your company. And you can mark as many bugs "as designed" or "will not fix" as possible. If you're lucky enough to have the kind of guys on your team that care so much about the project that they implement their own features when nobody's looking, it's definitely time to stop that if you haven't already.

Late Alpha: Bug Count Is Falling

Once you're over the hump, and fixing bugs faster than you find them, it means two things. First, you need more testing, as now testers are the bottleneck. This is the time (okay, one of the many times) you yell and scream at your publisher, because you're doing all you can to bring the project in on time, and they are the ones holding you back. (Evil publishers may even have a completion-on-time bonus they don't want to give you if they don't have to, and will therefore give you just the right amount of testing to ensure that you complete just a week or two late. Or they can use the "this lame bug must be fixed" trick.)

Second, you can get an idea of when you're going to hit zero bugs by looking at the trajectory of the graph. You can see how closely this number relates to your previous estimate of how many bugs there were going to be. It's also a good idea to devote your own people in-house to testing, although it may take some work to train your idle artists and coders how to be good testers.

This is the "finding the hard bugs" stage. You are officially within striking distance of being done. You can start sending presubmissions to the console manufacturers. (And fix the slew of bugs that they report.) You are a few weeks from being done.

Some Call It Beta: Zero Bugs

Finally, you hit zero stop-shipment bugs. You're not done; you've only hit zero for the first time. This is the point where some publishers change their criteria as to what they consider a "stop-shipment bug". From here on, the publisher becomes much more stringent, letting cosmetic and gameplay bugs slide, as fixing a cosmetic bug always runs the risk of introducing a stop shipment bug.

This brings up story I like to tell about the first game I ever worked on, Magic Candle 2. Right before we shipped it, we discovered that the player could walk on a kind of foothill terrain they weren't supposed to be able to. When fixing the bug, we introduced a real bug: the player could walk on water but not sail on it, and the game shipped that way.

Greg John calls this the "firemen" stage. Each morning you come to work and the testing team has found half a dozen new bugs overnight. Most of these you will not fix ("WNF"), the rest you get fixed by mid-afternoon, and then you sit around and browse web sites and pray. Maybe you're getting two sets of bug reports a day. It's a good idea to give people time off, with the understanding that they are on call in case a bug crops up that only they can deal with. At this point, you are almost done, and as soon as you've gone some number of days without a bug report, you fire off submissions to console manufacturers. The number of days is up to you. It could be one -- which amounts to having the console manufacturer do your last round of testing for you -- or it could be as long as the console manufacturer is going to spend testing your project, in which case you can be pretty sure (but never one hundred percent positive) they won't find any bugs you haven't dealt with.

If push comes to shove, you can actually ship to the console manufacturer at any time during the beta stage. We did this on our last project so we could meet our commitment to retail, and we got lucky. I don't recommend it.

The Cold Hard Reality

I'm making it sound like once you're armed with these tools you never have to feel stress during those last few months again. Unfortunately, you can always be surprised. On our last project, we blew through our rule-of-thumb estimates for how many bugs there were going to be. After we thought we had gotten over the first hump, we asked for more testing, and we got it. The bug find rates climbed right back up again. Our open-bug graph ended up looking like the stock market. Times like these make you feel like you're a first-year game developer.

When Shouldn't You Fix The Bugs First?

I like to get on my high horse and spout general principles and rules of thumb and then frequently find myself violating those rules in the heat of battle. The "fix your bugs first" rule has its exceptions.

• Frequently you'll find yourself with a whole subsystem of legacy code that's buggy as hell. The guy who wrote it originally has been promoted to some management position so he can't fix it, or he's left the company, so you decide the best move is probably to scrap it and start over. But you have a demo coming up, or the programmer who's going to replace the module is on another project still, or it's going to take him a while to get the new system in. At this point, it's better to just limp along with the current buggy system then to try to fix bugs in code that's going to be completely replaced anyway.
  
• Somebody has an idea for a feature that's new and risky. Everybody agrees it would be good to try, but you know that once it's been implemented it's going to need a lot of tweaking and experimentation, and it's going to create a lot of issues. You should probably get that feature in and see what issues it creates and deal with them early.
  
• You might have idle people on the team who are waiting for a tool or feature or system to become available so they can get to it. Here, it's often better to make that system available to them, bugs and all, and let them start working with it, then to fix bugs (in the system or unrelated ones). Don't be the guy who kept working on the sextant until it was perfect when there were sailors dying who needed it yesterday.
  
• You're in the middle of something. DeMarco and Lister discuss how important it is for developers to achieve flow; it's like our brains and dev. stations have a certain amount of setup time, and if we switch tasks, we lose that. Switching tasks to fix a bug can fall into this category. To mitigate this, we break our bugs down into priorities: priority 1 means stop what you're doing (crash bugs); priority 2 means get it done at your earliest convenience (most bugs); priority 3 means we might leave it in when we ship (suggestions, polish).

Be careful with these exceptions. They can quickly become excuses which allow you to let the product get in a sorry state. When you can, refactor legacy systems instead of replacing them. Cancel those risky features and fix bugs instead. And, sometimes, let people be idle; it's better to have a few idle employees than to compromise the project.

Where Do We Go From Here?

I'm feeling a little shame as I write this article because my brother has just been playing the last game I worked on and has already found three bugs that I would have considered stop-shipment. So you might be tempted to disregard this article. But imagine how buggy our product would have been if we hadn't taken these measures.

Still, obviously, these measures are not enough. Use them to fill in gaps in your own testing regimen; use them as a jumping off point. But they are not a cure-all.

The introduction of Sun Tzu's Art of War goes like this: "According to an old story, a lord of ancient China once asked his physician, a member of a family of healers, which of them was the most skilled in the art. The physician, whose reputation was such that his name became synonymous with medical science in China, replied, 'My eldest brother sees the spirit of sickness and removes it before it takes shape, so his name does not get out of the house. My elder brother cures sickness when it is still extremely minute, so his name does not get out of the neighborhood. As for me, I puncture veins, prescribe potions, and massage skin, so from time to time my name gets out and is heard among the lords.'"

These days, what people really care about is the healthy patient, not whose name is heard where, so for us, there's something even better than fixing bugs before moving on to new features: create your game in such a way that bugs are not introduced in the first place. That's where my advice comes to an end, because while I am good at dealing with the problems I create, I do not yet seem to have the knack or knowledge to prevent these problems in the first place.

Still, even if you are the sort of game developer who removes the spirit of sickness before it takes shape, the techniques I have listed here can still be a valid part of your toolkit: a safety net for when the other measures fail.

References

Debugging The Development Process and Writing Solid Code by Steve McConnell - discusses the importance of getting bugs fixed first.

Rapid Development by Steve Maguire - discusses the daily build and smoke test.
Peopleware by DeMarco and Lister - discusses scheduling and flow.

The Goal by Eliyahu Goldratt and Slack by Tom DeMarco - why an idle employee is not the sin that upper management may think it is

Critical Chain by Eliyahu Goldratt and Waltzing With Bears by DeMarco and Lister. Discuss the safety buffer, the project management equivalent of what we call alpha.

Game Development and Production by Eric Bethke. Discusses a unique way to estimate project status during alpha that goes beyond what I've described here.

________________________________________________________

[back to] Introduction

join | contact us | advertise | write | my profile
news | features | companies | jobs | resumes | education | product guide | projects | store



Copyright © 2003 CMP Media LLC

privacy policy | terms of service