Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
Production Testing and Bug Tracking
View All     RSS
December 16, 2019
arrowPress Releases
December 16, 2019
Games Press
View All     RSS







If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

Production Testing and Bug Tracking


July 14, 2003 Article Start Page 1 of 2 Next
 

My thesis is simple, common-sense, and has been said many times before: finish one part of your project before moving on to the next part. Finishing a part -- be it a feature, character, mission, level, whatever -- means getting it ready to go into the box. It doesn't mean you can never touch or polish that part again, it just means that if you had to ship it the way it was, you'd be willing to. Which means you have to fix bugs now, rather than put them off until later. Somehow we keep forgetting this. For instance, just a week ago at our leads meeting, we nearly forgot it again when we said, "Let's get each mission to alpha and move on to the next mission."

One reason we keep forgetting this is because of that phase of the project we call "alpha" or "feature complete." The fact that we have this phase implies that we're not supposed to fix our bugs now; we're supposed to get the features in, and then we'll fix all the bugs later in the alpha phase. This becomes exacerbated, because with each project we do when we're done we say, "The game wasn't as polished as we would have liked last time around. We need to devote more time to alpha." One side effect shrinking the full production portion of the development cycle has is that there's additional pressure to put off bug fixing until later.

We need to resist this pressure. We need to fix our bugs before moving on. Why?

Some people find this surprising. They say things like, "What? We can't fix the bugs now, we've got a milestone to hit!" or "People will put up with crash bugs for a fun game but will not put up with a game that isn't fun no matter how solid it is." or "If we start fixing bugs now, we'll never finish." That's our disease talking. For the most part we'd rather stick pins on our eyes than track down bugs, but we must.

There are a few reasons why it's important to fix bugs now rather than putting them off until after alpha:

  • Bugs are cheaper to deal with the sooner they're dealt with.
  • It's easier to estimate how long it will take to implement a feature than it is to estimate how long it will take to fix a bug.
  • Crash bugs make it harder to implement features because you sometimes can't even test those new features reliably.
  • When your game is buggy, when a new bug is introduced, you don't know if it was a bug that was already in there or not.
  • It's harder to implement features because you don't know if the reason your new feature isn't working is because you screwed up or because of somebody else's bug.

If you try to keep the number of defects down you speed up your development cycle, and end up with more time to make more game. Even with a throwaway prototype you should fix bugs, for the same reason: you'll end up with a better prototype. It only takes a few days of ignoring bugs to turn your development process into a living hell. In short, an attitude of "fix the bugs now" is one way you can increase Bang for developer Buck.

Sometimes I think of making a videogame as an endless process of fixing thousands of bugs. Some of the bugs are stop shipment bugs but most of them are suggestions for how the game can be improved. We prioritize as best we can, fix as many as we can, and at some point we run out of time, make sure all the stop-shipment bugs are fixed (hopefully), mark the last of our bugs "Will Not Fix", and ship it.

Given a choice between scheduling software and bug-tracking software I'd take the bug tracking software. According to Peopleware by DeMarco and Lister, some (admittedly sketchy) studies have shown that programmers work more productively when they aren't scheduled. This may explain the feeling you get when you're in alpha and dozens of bugs get fixed every day. Everyone's trying to fix each bug as quickly as possible, instead of trying to do the best job they can do on a feature within the time allowed. And I think you can benefit from that sort of "ASAP" mentality, even at the beginning of a project, by getting the bug list up and running, and putting the small tasks and high-priority suggestions ("orcs are way too easy to kill") on there.

Techniques For Fixing Bugs Early

Now that I've gone over why it's a good idea to fix bugs first, let me go over some techniques for making that happen.


Kelly Slater's Pro Surfer

Commit to tracking bugs. The very first thing Greg John, my producer, and I did when we started our last four projects was get the bug database up and running. (To quote David Cook, the lead coder on Kelly Slater's Pro Surfer: "The first bug is - there is no game.")

Use an automated bug tracking system. If your team is larger than half a dozen people, you'll find that unless you have an automated way for people to submit bugs, you will be spending a ton of someone's time recording e-mails about bugs into the bug database.

We use an access database with a web-based front end that we created with Seven Simple Steps a few years ago and which we've been modifying and changing steadily. If you don't want to take the trouble to write your own, FogBugz from Fog Creek Software is a good product that didn't quite meet our needs three years ago but has since matured a great deal, and they provide source so you can tailor it to your own needs.

Don't rely on your publisher to do all of your testing. Do testing in-house. Either have a testing department, or, if you don't have the budget for one, you'll have to have people on the team do the testing. That's what I mean by production testing: the producers do it. Make a disciplined, exhaustive effort to find bugs and get them in the database instead of letting your bug list just be a dump for people to throw their occasional gripes in.

Generally, any asset or feature is going to need a lot of testing. The person who creates the asset or feature should test it before submitting it. Their lead should test it before signing off on it. In house testing should test before each milestone is submitted. The publisher should test it before it is sent to the console manufacturer. The console manufacturer tests it before releasing it.

Gameplay testing should be done as well; don't just answer the question "does it work" but also "does the player like it?" (Find out if you've done the right things, along with finding out if you've done things right.)

It's quite possible you work at a place where upper management isn't going to spring for in-house testing. All this means is that you have to do more testing yourself. Because we had no testing department handy, I, the lead programmer, spent about a third of my time every day playing Tony Hawk when we ported it to the Dreamcast. Rough job, eh? But if I hadn't done that, the game would have taken longer to ship. And even if we had in-house testing back then, I still would have had to spend some time testing, to make sure that features were being implemented correctly.

Autobuild. Rapid Development stresses the importance of a daily build and smoke test to make sure, each day, nobody's checked in anything so horrible that it breaks the game completely. On our current project, Toby Lael, one of our programmers, has put together something even more effective: a program that watches our source control, and as soon as a change is checked in, gets that change and does an incremental build of the data and/or source that's changed, making sure that it still works. If it doesn't work, an e-mail is sent out to the whole team, and whoever checked in the bad stuff will quickly act to make things right. This way, for most of the day, people know that they can safely get the latest stuff out of source control, and when it's broken, everyone knows it.

Zero-Defect Pushes. Unlike Steve Maguire, author of Writing Solid Code, I don't believe it's possible to always keep the master sources free of bugs. I have yet to meet anyone who succeeds at this noble task. One tenet of zero-defect development is that everyone stops working on their own stuff when there's a bug in someone else's stuff; they all pitch in to help, or at the very least they don't break anything more while that one person is muddling through. I think that on a large team someone is always wrestling with a bug. If everyone pitched in to help, it would kill our ability to work on multiple fronts in parallel, slowing our development to a crawl.

What I do believe is that it's possible to make a push for zero defects a few times in the life of the project. We don't have to wait until alpha.

Sometimes I think that if it wasn't for E3, most of the projects I've worked on would have failed. Although people complain about the time wasted on E3 making demo gameplay that's going to be scrapped anyhow, one thing E3 does is it forces us to try and get our bugs fixed; we make a zero defect push.

When we're in this mode, we do a number of things:

1) We lock access to source control. People aren't allowed to check in at will.
2) We have a check-in queue. We go through the queue one by one, allow people to check-in, test their work for several minutes, and go on to the next person. Although this means people are frequently idle while they wait for their turn to check in, the amount of time individuals spend idle tends to be about the same or less than the amount of time the whole team spends idle when the build gets broken.
3) When a new bug gets discovered, we halt the queue completely, not allowing anyone to check in until the bug has been found and fixed.
4) We don't allow low-priority fixes, new features, or cosmetic improvements. We try to only fix what truly embarrasses us, although there is some give and take between risk and importance. A texture change, or a game balance tweak (thug's kick is too effective) might get in, simply because it is deemed safe.

Note that what makes this process work is really the last step. On Chris Busse's team at Treyarch, steps one through three are a way of life, no matter what phase the product is in. My suggestion: try living with it for a while. I think although your team will be chafing at the bit to check in their work, they will have new confidence that the build is in a good state.

So why wait for E3 to do a zero-defect push? On our last project we did pushes a few times, when we needed to show the game to upper management. We never actually did hit zero defects with these pushes, but the efforts made the game that much more stable each time.


Article Start Page 1 of 2 Next

Related Jobs

Square Enix Co., Ltd.
Square Enix Co., Ltd. — Tokyo, Japan
[12.15.19]

Experienced Game Developer
Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States
[12.12.19]

Principal Writer
Sucker Punch Productions
Sucker Punch Productions — Bellevue, Washington, United States
[12.11.19]

Producer
Sanzaru Games Inc.
Sanzaru Games Inc. — Dublin, California, United States
[12.11.19]

Producer





Loading Comments

loader image