Features

How Funcom Squashed Bugs With Bugzilla

When Funcom launched in 1994, like many small companies, it lacked sophisticated bug-tracking and project-management tools. With small projects and few people, the company didn't really need such tools in the beginning. As the company grew, however, it began sporadically using tracking tools and instituting production procedures. Today, the company has good tools and a highly efficient production process - and the company spent very little on software to get to this point.

	Bugzilla was first adopted by Funcom around the time that Anarchy Online went into beta.

This is the story of how we developed our core production management tools, based a modified version of the Open Source bug-tracking system, Bugzilla, and Microsoft Project 2002. While you're probably familiar with Microsoft Project, let me briefly explain what Bugzilla is. Bugzilla is an Open Source, defect-tracking system that allows individuals (or groups of developers) to keep track of bugs in their product. Originally developed by Terry Weissman, it is now the de-facto standard defect-tracking system against which all others are measured. (See http://www.bugzilla.org for more information.)

At our company, these two products have come together to form an inexpensive yet powerful tracking and planning solution - one that other game development teams may want to examine. If you're currently using Bugzilla, this article will give you some concrete ideas for extending the tool. If you're considering investing in a project management or bug-tracking tool, you should consider Bugzilla. But even if you're not using or interested in Bugzilla, this article might still inspire you to extend your existing project management tools to support some of the functionality we've created.

Bugzilla Versus Commercial Applications

While much of its power is its extensibility, even without any custom modifications, Bugzilla is a good product. It tracks tasks and bugs (really just tasks -- bugs can be considered a special kind of task), and I believe a big reason we reached our current level of efficiency is that we have a single application to store for all tasks. Bugzilla can work like a knowledge database, too, due to its flexible searching capabilities. Once you come to grips with this ability, you can utilize the real power of the application.

Over the years, we have looked at various commercial solutions from companies like Microsoft, Rational, and NXN. There are various reasons we didn't choose any of these tools, but in general each failed meet one or more of our team's requirements:

1. Platform independence
2. Not too expensive to purchase
3. A simple bug lifecycle
4. Not too expensive to implement
5. Fast deployment

When Funcom's MMORPG, Anarchy Online (AO), reached the beta stage in 1999, we started using Bugzilla. At first we just used Bugzilla as intended, for bugs. Today the AO team consists of around 40 people, and everything in our development process revolves around a modified version of Bugzilla. If somebody wants a task done by another person, and it takes more than 10 minutes to do, a Bugzilla report is created. (If the task isn't worth spending at least 30 seconds to fill out a report, the task isn't worth adding to the production schedule anyway.)

Our goal for using Bugzilla was to improve our production process. For management, that translated into more control over the project: the ability to measure statistics related to the production, and a better view of the project's status at any given time. For rank-and-file developers, it meant tools to track individual contributions to the project. Each developer is assigned between 10 and 200 tasks at any given moment, and everyone felt that storing a personal daily task list in the same location as the whole project's task list would be a huge benefit.

Problems With Early Bugzilla Versions

When we began using Bugzilla in 1999, we started modifying it. The modification process was fairly organic: somebody got a bright idea to customize the application, and it was implemented. For example, we started to flag bugs or tasks to be "Research", "Implementation", and so on. We also added fields for people to estimate the minimum estimated hours to fix a bug, a maximum-hours estimate, and actual hours spent on each bug. But shortly after introducing these fields on reports, we began to wonder whether we could trust the data, since we weren't sure whether people were entering the correct values into the different fields. That turned out to be a valid concern. For instance, if we discovered that people had different opinions of what "ASSIGNED" really meant, and we found out that many tasks' time estimates were being entered as "0-0-0" (corresponding to the minimum hours estimate, the maximum hours estimate, and the actual hours spent). So we learned we couldn't trust our bug reports. We needed a way to ensure that the information about a bug could be trusted. This meant that the meaning of each field had to be unambiguous, and the definitions had to be quickly communicated to everybody.

To solve the problem, we agreed upon the following Bugzilla changes:

1. Use Bugzilla for all tasks
2. Add time estimates to every task (minimum hours estimate, maximum hours estimate, and actual hours spent)
3. Make Bugzilla easier to use. Numerous small, simple visual improvements.
4. Make Bugzilla faster to use
5. Add fields for better user navigation and comprehension.
6. Remove unused fields to get better overview. For example, the AO team doesn't use Bugzilla's built in fields like "Operating system", "Platform", "QA Contact", "Votes" or "Keyword"
7. Make the task completion process stricter. A task needs to follow a defined life cycle.
8. Assign users into groups or professions.
9. Add a group, "super user", to override all the new strict checks in the system
10. Create a "master task" -- a container task that holds subtasks, like a node in a tree.
11. Create a "task owner". Not to be confused with Bugzilla's normal assignee. The task owner is responsible for carrying the task through the system. Time estimates are connected to the task owner
12. Create external tools to analyze the data in the Bugzilla database.

With these changes in place, we saw the following improvements:

• Strict adherence to our bug-tracking processes and project management processes. Bugzilla prompted you for more information (and wouldn't proceed) if a developer assigned a task without inserting a time estimate, tried to set a task to fixed without spending any time on it (setting 0 used hours), tried to set "used hours" higher than the "max estimate", tried to close a task without a test case (and this feature alone has saved us a lot of problems), or tried to close a task without a "changes description" message.
  
• Better time estimates. With this information it was easy to generate reports about how much time has been spent on a given task, generate reports about how much time is left for a group of tasks or a person, and import project data into Microsoft Project for detailed analysis and planning.

The changes we made to Bugzilla involved expanding the tables in the Bugzilla database, which in turn required knowledge of MySQL (the database Bugzilla 2.11 uses) and the SQL language itself. We also needed to know some Perl. The "main" table, bugs, was modified like this:

	Field Type bug_id mediumint(9) groupset bigint(20) assigned_to mediumint(9) bug_file_loc text bug_severity enum('blocker',cr... bug_status enum(UNCONFIRMED)'... creation_ts datetime delta_ts timestamp(14) short_desc mediumtext op_sys enum('All','Window... priority enum('P1','P2','P3... product varchar(64) rep_platform enum('All','DEC','... reporter mediumint(9) version varchar(16) component varchar(50) resolution enum('','FIXED','I... target_milestone varchar(20) qa_contact mediumint(9) status_whiteboard mediumtext votes mediumint(9) keywords mediumtext lastdiffed datetime everconfirmed tinyint(4)
	The main table, before modification.

	Field Type bug_id mediumint(9) groupset bigint(20) assigned_to mediumint(9) bug_file_loc text bug_severity enum('blocker',cr... bug_status enum(UNCONFIRMED)'... creation_ts datetime delta_ts timestamp(14) short_desc mediumtext op_sys enum('All','Window... priority enum('P1','P2','P3... product varchar(64) rep_platform enum('All','DEC','... reporter mediumint(9) version varchar(16) component varchar(50) resolution enum('','FIXED','I... target_milestone varchar(20) qa_contact mediumint(9) status_whiteboard mediumtext votes mediumint(9) keywords mediumtext lastdiffed datetime everconfirmed tinyint(4) pipeline_status enum('Unknown','Id... changes mediumtext min_hours float(4,1) max_hours float(4,1) used_hours float(4,1) testcase mediumtext
	The main table, after modification. (Bold fields indicate additions.)

Since people don't always embrace change, it was a struggle to convince our team to modify their work processes. Initially, the leads were accused of forcing the team to use an overly strict control system. We were met with arguments like "everything is becoming so much more cumbersome and time consuming", "the overhead per task is going to waste the company's time", "making games should be fun", and "this requires too much administration". These comments usually came from people who weren't thinking about the team as a whole. Fortunately, demonstrating the new tools helped everyone on our team understand the big picture.

After we began to use the planning tools (described in a moment), we saw improvements. After a while everybody found out that the current output and control was better than ever before.

We found couldn't abandon our old rules and routines completely, though. One routine that we kept was the one that lets team leads set task milestones. In our system, each developer has so many tasks that they have to be ordered in some fashion, and in our system, the next milestone determines the order of tasks. (Usually we don't work on a task that doesn't have a milestone.)

Funcom's Bugzilla Modifications

One of the key features in Bugzilla we used was the ability to track activity related to tasks or bugs. It lets us see a history of all the events associated with a given bug. You'll see how we try to exploit this data in a moment.

Things that differ from the regular Bugzilla:

• Added help on almost all the fields
  
• Added tooltips
  
• Added the ability to spawn a new subtask by pressing "[NEW]"/"Enter new bug/task based…". New subtask gets initial values from parent task.
  
• Added these fields: Task Owner, Testcase, Changes description, Min-, Max-, Used-hours, See also, Production pipeline, "Change task owner to assignee"-checkbox
  
• Added these fields for mastertasks only:
  
  • Sum of Min-, Max- and Used- hours (not editable)
    
  • Number of unestimated subtasks
    
• Easily see when the task was last updated in upper right corner.
  
• Easily see history on these fields:
  
  • Assigned to
    
  • Task owner
    
  • Target milestone
    
• The system complains at specific fields when you try to submit some reports that are inconsistent or faulty.
  
• A link to something called "Time estimates: (details)" (see Figure 1)
  
• For text in comments: Any references to other tasks are automatically replaced and hyperlinked with the full title of a task. The text "task nnnn" automatically becomes "task nnnn (<Title of task>) ".

The net effect of these modifications is that people can more easily see the data they want. If they want to find a test case, they look in that field. They don't have to read through 40 comments to find it. Also, several operations can be combined which lets you save time and counteract annoyance with the application.

	Figure 1: The estimate details page. One can easy get the overview of the work done and the work remaining, pr profession in the company. In this case all tasks are done, so estimated time left is 0.

 

	Figure 2: Simple query result page. Get the sum of hours on the tasks listed. Vertical headers squeeze more info into the page. With a good search you can get the overall status right away.

join | contact us | advertise | write | my profile
news | features | companies | jobs | resumes | education | product guide | projects | store



Copyright © 2003 CMP Media LLC

privacy policy | terms of service