The tools programming team at BioWare is responsible for the content creation, asset management, build construction, end-user and localization tools on each project. This has typically amounted to anywhere from 20 - 50 individual pieces of software per project ranging in size from just a few lines for some conversion utilities to hundreds of thousands of lines for the designer toolset. While some of the tools can be reused with varying degrees of modification, there is still considerable new development required for each project.
At the beginning of 1999, the tools programming team at BioWare consisted of two programmers working on Baldur's Gate Tales of the Sword Coast and Baldur's Gate 2 Shadows of Amn. Currently, the team has 16 programmers working on five projects and by the end of 2004 it is expected to increase to 22 or about 30% of the entire programming department. The average turn-over rate is approximately one programmer per year; with the majority of programmers being transferred to the game and graphics teams.
paper will discuss the issues surrounding this growth. It will explain
why BioWare places such emphasis on tools development and how it has been
able to expand to such a degree. In addition, this paper will describe
the roles and responsibilities of the team, how the team is best used
on the projects, the consequences of success and the price of failure.
Finally, the paper will describe the practices used for operations, change
management and development.
Why a Dedicated Tools Programming Team?
BioWare devotes a lot of time and energy to the continual evolution and improvement of its tools. This is money that could be spent directly on the game. As such, it is fair to ask "Does BioWare benefit from having a team of dedicated tools programmers as opposed to using whichever game programmers happen to be available?"
The Tools Programmer Advantage
Everyone relies on tools. In software terms (game development, specifically), tools are necessary to help people perform data entry and automate repetitive or error-prone tasks, improving reliability and efficiency.
If, for example, it takes one tools programmer one week (40 man-hours) to write a tool that simplifies and reduces build times by one hour and builds are performed twice each week, it will take 20 weeks before the development time has paid for itself. On a two-year project, that will save 160 man-hours in build time (about one man-month), but that is not the total benefit. Faster build times, means more QA can be done, which hopefully results in fewer problems in the final product and overall less time spent on support. Assuming that there are four QA analysts on the team, the above one hour reduction in build time results in four extra hours of QA per build. The payback time for the development effort is then reduced to four weeks and results in 800 extra hours of QA on the project.
In the roughest of terms, each tools programmer should be responsible for saving or returning to the project at least 2000 man-hours per year. If it is no more than this, then nothing has been gained. Depending upon the tool, it may be possible to amortize its development over the project's post-ship support cycle as well. That is, a tool that reduces build time will continue to be of value if and when it comes time to release an update. Again, looking at the improved build time example, with one week of work, this tools programmer returned 960 hours (160 + 800) to the project or almost half of the annual "requirement".
Note that it will not always be possible to assign values such as these to the work that is done by a tools programmer. This is because new tools may enable the content creators to take advantage of key features that were simply impractical on previous projects. In such cases, the value of the tools is essentially the value of the finished project.
Specialization and Dedication
Tools programmers should be actively finding and exploiting as many opportunities to increase the overall effectiveness of the development team as possible. The best way to accomplish this is through experience working on other projects. This implies the creation of a team of programmers hired for and dedicated to that task alone.
Originally, BioWare focused on developing a small, but highly cross-trained tools group. The idea was that any tools programmer should be able to easily migrate from content creation to asset management tools, for example. While this worked when BioWare was smaller, it became clear that this is a bad idea for the same reason that dedicated graphics programmers should not be expected to easily switch to AI or networking. That is, the strategy started to fail when the complexity of the tools was such that it required significant training time to be proficient in each area.
Eventually, BioWare realized that creating a dedicated team encourages the development of particular skill-sets which, while expensive, can be applied across multiple projects. BioWare has already identified several key areas where its tools programmers are beginning to specialize such as asset management, content creation and localization. This has started to allow the solutions to problems in these areas on one project to be more easily carried forward to the next.
Staffing the Team
This is the most fundamental part of building a tools programming team. If the team is not staffed properly, nothing else will matter. The key is to hire programmers with the right skills, who want to write tools and then keep them there.
Finding and Hiring the Right People
Hiring programmers who want to write game code, AI or graphics onto a tools team is a waste of effort. Eventually, these programmers will become unhappy and unproductive at which point they either must be re-assigned or will likely quit altogether. In the end it does not matter what happens to these programmers after they leave the team because all the training time spent on them and experience they have gained will be lost.
During BioWare's first major round of hiring for the tools team, in the spring of 2000, this distinction was not made. In fact, it was just the opposite. The approach was to deliberately hire programmers onto the tools team as a probationary exercise. After six months or a year, these programmers would be 'promoted' off the tools team onto the game or graphics teams if they were working out. This was a terrible idea. Out of the five programmers hired in that round only two remained by the end of the first year. The other three had been transferred to game programming teams either at their request or the request of the lead programmer on that game.
Tools programming is already perceived as being somewhat less glamorous than the graphics and AI disciplines. Perpetuating the notion that tools programming is a role out of which people are 'promoted' will not help attract the talent necessary to create first-class software. Tools programming must be thought of as a valuable long-term career option into which only the best candidates will be accepted.
Candidates during subsequent hiring rounds were hired explicitly because of their desire and passion for writing tools. This attitude has cost BioWare some potential new employees but all that really means is that it eliminated the subsequent round of hiring that would otherwise have been necessary. Consequently, the average turn-over on the team is now very low (approximately one programmer per year).
The kinds of skills a tools programmer needs depend upon the kinds of tools required to build the next project. Without experience on previous projects, guidance in this area must come from the project's leads and producer. As experience is gained, the tools programmers themselves should be able to help drive the list of skills required to increase the team's value to the project and company.
BioWare's games are typically, extremely design-heavy (i.e. contain a large volume of non-art content), contain tens of thousands of resources and have hundreds of thousands of words. This requires programmers specializing in, among other things, Windows-based application development, user interface design and databases. Excellent communication skills are also critical at BioWare.
In spite of this focus on Windows and database programming, there is still a need to have tools programmers who can build art tools. Until recently, BioWare's tools team has lacked specific expertise in that area. This gap developed during Neverwinter Nights as the company shifted from making 2D to 3D games and widened on Star Wars: Knights of the Old Republic. It is only now, on Jade Empire, that this gap has started to close as the tools programming team began to actively recruit tools programmers for that purpose.
Keeping the Team Together
Getting the right programmers on to the team is not enough. Even the most committed programmers may lose interest if they are not satisfied with their role on the team. Satisfaction typically arises from some combination of challenge, contribution, growth and compensation. The exact proportions depend upon the particular programmer in question and will have to be worked out in detail with each one either on an ad hoc basis or during more formal reviews.
The need for a challenging work environment where people feel as though they are contributing to the success of the project applies to almost all game developers and programmers, tools programmers specifically, are no different. Compensation comes in many forms: a comfortable salary, a good benefits package, bonuses and a fun place to work are all part of it. Again, by providing a solid overall system of compensation for the entire company it will, by definition, apply to the tools programmers.
Growth is a unique issue when it comes to tools programming. In order to really combat the perception that these positions are where entry-level game programmers start, a career path must be set up that offers real long-term opportunities. At BioWare, tools programmers have the option to specialize in a particular area such as asset management, content creation or one of the other areas identified earlier. Starting out on a project helping a more senior tools programmer in a particular area, the tools programmer may take on full responsibility for this area on a subsequent project. Once a level of expertise in that area has been acquired, that tools programmer may choose to further specialize in that area, select another area of specialization or take on more leadership and management tasks. In reality, it usually involves a combination of all three.
When BioWare (and the tools programming team) was smaller, all the tools programmers were expected to be able to work on tasks from any project. This and their relative inexperience made it impractical to divide the group up amongst the different projects. Instead, the tools programmers were located in a generally centralized area in the office space. One benefit to this was the development of a strong group bond. This bond has proved invaluable for the communication and adoption of the group's roles and responsibilities (mentioned below). This sense of group identity helped to foster collective ownership of all the tools across all the projects.
As the team has grown and the number of senior tools programmers has increased, it is now possible to assign groups of tools programmers to projects for extended periods and locate them in the project area. This has resulted in improved communication between the tools programmers and their customers (artists, designers, programmers, producers and QA analysts). The benefit to this is better and more useful tools that meet the needs of the specific project which, in turn, should produce higher quality and more successful products.
Which is a better organization? Assignment and close proximity to their assigned project should produce a bigger short-term benefit for the project and company. On the other hand, group association means higher quality tools across all the projects and should produce a bigger long-term benefit for the company. They both have their place depending upon the size of the company, the size and experience of the tools programmers and the number of projects currently under way. No matter which model is followed, be careful to still take the time to build that group bond.
Roles and Responsibilities
How does a tools programming team fit into the rest of the project or company? This has been an evolutionary process at BioWare; the tools programming team growing and adapting as the company continues to change. What follows is a description of the key points around which the tools team at BioWare has currently structured itself.
Tools programmers are no different than anybody else on the project team; their ultimate focus should be on the end product, the game itself. This should not be a surprise.
Unlike the rest of the team however, the primary customers of the tools programmers are in the next office or just down the hall. The level of satisfaction the team has with the quality of the tools and responsiveness of the tools programmers will have a direct impact on the final game. Good communication is the key to that satisfaction and that is why it is such a vital skill for tools programmers (actually, everyone needs this skill).
There are lots of ways to maintain or improve communication. Put those who need to communicate in closer proximity, conduct regular meetings and support informal or "extra-curricular" gatherings are just a few possibilities. What will work depends largely on the corporate culture, work environment and the current state of the project. Watch for signs of communication breakdowns and deal with them immediately.
To facilitate better communication, the tools programmers at BioWare have a primary contact among the artists, designers and other customers for each major system. The tools programmers and their contacts meet fairly frequently to discuss what changes (if any) are coming up in the corresponding system. The contact's responsibilities are to help prioritize the feature requests that come in from the other users, communicate when the tools programmers feel the features will be ready back to the users and validate the new features (primarily the UI) before they are released. Getting feedback on that user interface early is critical. Do not let the content creators put this evaluation off "until they have time" because that will not be until they actually need the tool (i.e. when it is too late).
One thing to note: the most effective tools become a natural extension of whatever process you are following and do not impede it. The users should be able to forget about using the tool completely and be free to concentrate on their work. One consequence of this is that unless there is a problem, no one will really care about the tools that are used on the project, no one except the tools programmers. This means that tools programmers are likely to encounter only two kinds of comments about their work from the users: it is buggy or it is late. Close proximity and reliable communication can go a long way to increase the compliment to complaint ratio.
Reduce Development Time
The role of any tool (in either software or physical terms) is to make the job easier or the output better. The result of either is a corresponding reduction in the time necessary to complete development on the project. This means that there is more QA that can be applied to the product and / or that the product will be delivered sooner. Overall, reducing development time must be a consideration in all areas of tools development, from the performance of an algorithm to the number of clicks in a user interface required to get the job done.
Reducing development time involves three fundamental principles that must be kept in mind when writing tools: First, usability is king (or queen) with the caveat that personal productivity should not come at the expense of the team's productivity. Second, tools software should try to catch errors at the earliest possible moment. Finally, a planned and scheduled delay is less expensive than an unexpected outage.
One example would be a request to add validation checks before the content creators can post their changes to the production environment (the checks would be optional for local testing). The production environment is the set of data and software that comprises the current game and its corresponding construction tools. This is where everyone goes to get the material required to do their work and advance the development of the game. This request arises from the fact that it takes more time and people to track down and fix content problems after they have been submitted than if they were caught earlier. In some cases problem content can actually block other content creators.
For the purposes of this example, assume that these validation checks add between 2 and 20 minutes to the submission time depending on the size of the content. Also assume that the checks cannot be made any faster. Now, adding these checks will slow down the introduction of content which reduces individual productivity but should stabilize the entire build and increase the team's overall productivity. By not enforcing these checks on the local environment, the content creators still have a very usable system.
The example above, demonstrates the first two principles. The last principle suggests taking the time to do things right even if it results in a bit of a delay because in the end it will actually save time. This is really just an extension of the first two principles, considering the team as a whole and catching errors before they have an opportunity to cause a problem. Think of this as a productivity enhancement for the tools programmers; typically, they can use all the help they can get.
Protect the Data
Obviously, if a software tool does not manage to reliably create, edit or otherwise manipulate its data without damaging it, it is not going to reduce development time. In addition to the time needed to re-implement or regenerate what was lost, tools failures will cost the project the time spent waiting for the program to be fixed. This actually costs the project on several levels as the users can not proceed with their work and the tools programmers are not working on something of more value.
Protecting the data starts with using solid engineering and change management practices during the development of the tools themselves. It also involves preventing the uncontrolled introduction of new and untested tools software into the production environment. The challenge is in convincing the rest of the project that this is necessary and workable. Fortunately, the easiest way to get buy-in from the rest of the team is to do the exact opposite for a project or two, although that, too, offers a certain amount of pain.
Between 1999 and 2002, BioWare used rather ad hoc methods for tools development. New software builds would be made and released on demand, usually without any more testing than was required to see if the requested feature worked in a single case (and that was often insufficient). Application crashes were frequent and generally accompanied by some degree of data loss. Unfortunately, because there were so few tools programmers and their workload was so high, it was thought that proper design, documentation and testing were simply luxuries that could not be afforded. This led to significant tools quality problems up to the end of Neverwinter Nights and part way into Star Wars: Knights of the Old Republic.
This chaotic approach to tools development meant that by the end of Neverwinter Nights, new toolset builds had to go through a two day verification cycle in QA before being released to the design team. Often, the QA cycle would be incomplete when another new build became available. As getting the latest build out as quickly as possible was of paramount importance, QA on the current build would have to stop and the entire cycle would start over again with the new build. As a result it could take anywhere from three to five days for a particular change to get into production use.
It was clear that something had to change. Starting in early 2002, changes to tools software were planned and coordinated with the team, then rolled out into the production environment on a regular, weekly, schedule. This appeared to slow things down at first, but the critical bug rate started to fall and more time could be spent adding the required features. The builds were not coming out much faster, but they were coming out much more stable and reliable.
Tools, Not Systems
Focus tools development effort on creating the custom software necessary to complete the game construction pipeline. Do not waste time writing software that can be otherwise purchased from another company. Unless there are extenuating circumstances such as the software available is too expensive or does not have the complete feature set required, take advantage of the expertise these other companies possess. Remember that any time spent developing an alternative solution to a commercial product is time taken directly from improving the project.
During Neverwinter Nights, the tools programming team at BioWare was asked to write a replacement for the current bug-tracking system. The bug-tracking system that was in use at that time had been purchased during the development of the original Baldur's Gate and was showing its age. This Project Manager System would have a data driven work-flow engine that tracks game features and their state (including any bugs associated with them). The additional features on top of standard bug-tracking that were being requested simply were unavailable in "reasonably" priced third-party packages. Development started in the fall of 2000 and took five man-months of design and implementation. Unfortunately, those additional features that were unavailable except in the more expensive packages were not very well received and went largely unused. Since then, seven additional man-months of effort have been spent on support, maintenance and general improvements but for all intents and purposes, it remains only a bug-tracker.
Considering the scope of the program and how few of its features were ultimately used to their full potential, the Project Manager System was not as successful as it could have been. In hindsight, a third-party piece of bug-tracking software should have been purchased to allow the programming resources that were spent on it to be spent directly on Neverwinter Nights. That being said, the system has been and continues to be used on several other projects although it has finally been scheduled for replacement around mid-2004 (by a third-party product).