     |
 |
 |
 | | | | |
 |
|||
 |
||||||
          |
 |
 |
||||
|
Postmortem: The Tools Development of Turbine's Asheron's Call 2 What Went Right1. Management and corporate buy-in. From the beginning, Turbine's vision for our internal tools was far-reaching. Many changes were being made to the Turbine Engine for AC2, so the idea of having a core engineer develop tools only part-time or wait until later in the content cycle was foolhardy and dangerous. We decided to create a dedicated tools position in the core engineering staff, recognizing that having a full-time engineer available for tools development would address many of the artist and content designer concerns. Instead of leaving tools development until the majority of the assets had already been created, the tools would be authored at the same time as the engine. But what good is a full-time position without a plan? Turbine had several clear goals for the tools development process for AC2 and a roadmap to get there. We planned sufficient time, schedules, and resources for the process: we did not want to be developing tools at the alpha stage or spending nights and weekends cobbling together code to get the job done. We drafted milestones and goals for the tools that aligned to concurrently developing features in the engine. Through tool prototyping, we often discovered engine bugs early in the process. We also learned in the process that while schedules are nice, flexibility in tools development is critical; we often were required to rearrange our milestones and due dates based on functionality that the users needed "now." Additionally, employees outside the core engineering group provided plenty of support. Artists, game systems engineers, and designers had helpful opinions and input into the tools development process. We brainstormed tool features and prioritized our lists. As the tools lead, I was able to schedule artists and designers to produce content used to prototype tools and work out bugs prior to their release. The company's support made my job that much easier.
2. Leveraging the engine code wherever possible. Turbine built the original set of world-building tools for Asheron's Call using the actual Turbine Engine game user interface. This presented a few problems: long compilations and tweaking screen sizes and positions made tools work unpleasant. Moving forward, Turbine decided that the tools should provide a consistent interface around an engine-rendered viewport. We developed the game on and for the Windows platform, so it seemed natural that the tools should be as well. Several benefits were immediate: With the user interface and rendering engine separated, code could more easily be shared between the tools and the engine. WYSIWYG - what you saw in the tool-rendering viewport was what you got in the client. Many lighting and visual effects could be previewed and adjusted before building client data, which saved a great deal of time. The flexibility of Visual Studio allowed us to lay out dialog boxes quickly and gave us the ability to use window controls, such as listboxes and tree views, that weren't currently available in the Turbine Engine game UI. We also extended existing window controls to provide object information in new and logically organized ways. Using the English text representations of enumerated type data in listboxes aided the designers with easily understandable names, while at the same time prevented errors that would occur if those enumerated types were entered by hand into a text field. As a result, we dramatically reduced broken builds due to bad data. A common API of virtual functions was provided for every object in the database. This made each instantiatable object "tools-compliant." Our world-building and art asset tools could query, modify, and update any object with the correct interface calls, which spawned one of the favorite expressions at Turbine: "Every engineer is a tools engineer." Engineers could use the common API to create new database objects with tools functionality built right in. Failure to fill in the API hooks resulted in debug assertions when an object of that type was loaded into the tool. Distributing the workload of tools development at the lowest level forced other engineers to provide previously omitted functionality. I would often find that providing functionality for a new object type became trivial; the majority of code had already been written. 3. Frequent tools design discussions. Turbine's internal tools cover a wide range of functionality. Art and content tools allow the user to preview and tweak all art assets before they are converted into the game data. World-building tools provide fast modification of terrain and the ability to place art assets into the game world easily. Engineering tools automatically generate code and data files and provide an interface to the source control system. At the beginning, there were hundreds of tools tasks that needed to be organized and prioritized. Early in the tools development process we created two committees: one to address art and asset concerns and one to address world-building concerns. These committees met weekly, consisting of representatives from each functional area at Turbine: art, design, engineering, and game systems. The status meetings permitted opportunities to brainstorm new functionality, discuss problem areas and possible solutions, and reprioritize task lists and schedules. In this way, everyone felt they had input to the tools process, making it "our job" instead of "my job." Keeping people up-to-date was a high priority. Daily status reports to a centralized mailing list let any interested artist or designer see what was happening to the tools. When new versions of the tools were checked in, another mailing list identified the new features, the bugs fixed, and the next required tasks. Each month, we prioritized the list of outstanding tasks and reshuffled the schedule, if necessary. We probably kept people too informed, but the adage "forewarned is forearmed" was definitely true. 4. Feedback, feedback, feedback. Unlike Turbine's monthly episode cycle where the client program is released to thousands of subscribed users, all internal tools were built daily and provided only to those who wanted them. The customer wasn't hidden behind a bulletin board or an anonymous IP address. And, in some cases, my customer was the person sitting right next to me. When they were unhappy, boy did I hear about it. Continual feedback made the tools better, but we worked hard to create productive feedback loops.
One single point of contact existed for all tools-related questions, problems, and enhancements: me. While this provided me with the opportunity to extend my organization skills, it also gave the users comfort to know that someone was always looking at a problem. ICQ and MSN Messenger gave the artists and content designers a way to contact me at any time. However, walking over to a desk and checking out a problem is often the only way to catch a crash that has no good reproduction case. I made a point to provide feedback as soon as possible, even if I was just letting a user know I couldn't get to their problem immediately or that their enhancement would be added to the list of items to discuss at the next committee meeting. A program crash is always unwelcome, especially when artists and content designers get in the groove of creating content. When a crash occurs, they will sigh, curse the program, and reload to continue what they were working on. I had no idea how often our tools crashed, and the users were more interested in getting work done than in reporting problems. We overcame mystery crashes by using a common "assert" dialog box in the debug versions of all Turbine's software: the tools, the client, and the server. This dialog box contained a stack trace, an optional message, and a Send E-mail button. Sending the e-mail directed the contents of the dialog box to an internal mailing list. Program crashes still put out the artists and designers, but now there was a clear record of where - and sometimes how - the assert occurred. The entire core development team monitored the assertion mailing lists, which proved to be very useful. For critical problems, we set up the development environment on every machine in the office. If a problem needed resolution immediately, or there was no good reproduction case, we could break out of the assertion dialog box directly into the code and analyze the stack at that point. This was useful for debugging not only the tools, but debug versions of the client as well. In some cases, the assertion mailing lists were insufficient to track the progress of a problem. When a problem severely affected the performance of the tools, we entered bugs into Bugzilla, the open source defect-tracking system. Bugzilla provided us with several important benefits: an HTML interface for reporting, modifying, and querying bug status; an e-mail interface to notify the appropriate engineers; and, because it's open source, a very attractive price tag. Once the baseline tools functionality was implemented, the ability to get a listing of bugs, priorities, and due dates was indispensable for organizing a daily or weekly task list. 5. We met our goals. There was a sense of accomplishment when Asheron's Call 2: Fallen Kings moved from production to Live Team support, not only from the client and server teams, but with respect to the tools as well. "Meeting our goals" may sound trite, but the Live Team would be using the tools we produced going forward as they created content for the monthly episode cycles. A defined list of objectives let us systematically fulfill the needs of the artists and content designers. Our first major goal was to speed up content iteration, which we did in several ways. To remove the possibility of entering bad data into the tools, menus and lists were pre-populated only with valid options. Allowing the database objects to reload in place from disk, artists could change models and textures behind the scenes, which eliminated the "modify, shutdown, reload" shuffle. Finally, overnight generation of data became possible by enabling batch processing of various assets. Hiding the engine complexity from the user was our second major goal. By using an engine-rendered viewport in the tools, we guaranteed that content visualized in the tools would be identical to the client. We also created APIs into external tools; enabling us to add revision control functionality and remove some checkout/merge/check-in problems. Next, we presented coherently organized data; we imposed a visual structure on the data hierarchy and clarified our database inheritance scheme. Lastly, by modifying our build system, we were able to automatically get the latest code from source control, build any number of executables, and e-mail the user with a summary status at the completion of the job. Overall, our internal tools development was worth the effort we put into it.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Features
|
|
|