     |
 |
 |
 | | | | |
 |
|||
 |
||||||
          |
 |
 |
||||
|
Postmortem: The Tools Development of Turbine's Asheron's Call 2 What Went Wrong 1. Coding before design. All the old lessons drummed into my head during school still apply: design in any complex software system is crucial and cannot be skipped. In the early phases of tools development, I tended to jump right into the code pile and start hacking out a solution to the problem. This caused no small amount of headaches when a seemingly small task blossomed into a days- or weeklong struggle. Our art asset tools were originally limited in that we could edit and load only a single entity at a time. To speed content placement, it soon became apparent that we would need to create groups of entities. Doing so meant loading multiple objects into the tool concurrently and providing grouping/ungrouping functionality. I jumped into the task and finished in less than two days. When demonstrating the process to the users, it turned out that several important features had been skipped, prompting questions such as: What world-space position were the entities moved and rotated relative to? If one object in the group was selected, was the whole group selected? Did the tool remember if several different groups were created? My understanding of the problem wasn't complete and it showed. It was a frustrating lesson to be learned all around. Obviously, more up-front design would have saved lots of time in this case. I now create a one-page, high-level summary of new features, which I have affected parties sign off on. A detailed design, usually down to the class and interface level, takes just less than a day. Applying this procedure cut tools development time by about 30 percent and drastically reduced the number of migraines I take home. 2. Feedback, feedback, feedback. As mentioned previously, creating our feedback loops was a lot of work. In the beginning, there wasn't much feedback from the users. Content designers did not realize that tools programmers didn't use the feedback functionality in the same manner; when the program crashed, they figured, "That's the way it is." Users found creative workarounds for various problems: "Don't press the 'V' key when your object is selected or it will crash." Later in the design process, I got a lot of requests for desired features which ended up being used rarely, if at all. During content crunch periods, feedback was nonexistent.
The dungeon creation process in AC2 highlighted a good example of feedback problems. Originally, the tech specified 400 objects in a dungeon, including all lights, decorations, creatures, and structures. However, late in the AC2 process, designers hated working on dungeons, especially when the number of objects ballooned to more than 1,000 objects: teleporting into and loading a dungeon location could take upwards of 20 minutes. Once loaded, rendering the dungeon yielded one to two frames a second; every object was drawn regardless of its distance from the camera. The content team shouldered these issues stoically, so I didn't find out about it until I had the chance to build a dungeon myself. It turned out that the process of loading a dungeon recursively built an internal graph of data connections every time a new object was loaded. This system worked fine for 400 objects, but got worse the more objects were added. A few lines of code to pause the data graph construction during loading dungeons and functionality to cull objects at a user configurable distance dropped load times to 45 seconds and increased frame rate to a manageable 15 fps. I have since learned to schedule some time for myself where I can use the tools in several different circumstances. There's nothing quite like getting a taste of your own medicine to spur improvements. 3. No good testing regimen. Insufficient testing plagues any software designed for internal use only. Many different trade-offs must be considered: How much time and effort is going to be spent testing a product that isn't going to generate revenue? Who is responsible for the testing of the internal tools, the developer or the quality assurance team? And how much of the QA budget can be utilized testing tools instead of the client? The developer making the change in a specific area would usually test our tools and administer a functional overview test. Such limited and random testing didn't find many problems, so usually the users would end up being the alpha, beta, and production testers. Every so often we ran across problems big enough that they required several new versions of the tool to fully isolate and fix them. Having users on different versions of the code base led to several head-scratching dead-ends before the version inconsistency was recognized. We started solving this problem with a growing internal QA team and the diligent use of bug-tracking systems for our tools. 4. Engine limitations. Wrapping a rendering viewport with a Windows system provided many benefits, but ended up causing other problems. The world-building tools allowed users to switch between three different workspaces: an entity workspace to preview and tweak art assets, a world-building workspace that allowed content placement and terrain modification, and a dungeon-building workspace for assembling dungeons and placing creatures. Since the tool never knew what the user would be working on, massive amounts of memory were required to make sure all assets were preloaded, adding to the amount of time the tool needed to start up - going from double-click activation to "ready to work" took up to five minutes. As I mentioned earlier, having the tools and engine closely bound was beneficial, allowing the core engineering team to find bugs in the Turbine Engine earlier. On the other hand, every memory leak or problem in the engine was exacerbated in the tools. One of the goals of the tools called for handling data errors cleanly and permitting the loading process to continue; users would be warned and their object would fail to load, but the program wouldn't crash. Such was not the case: text files that failed parsing by the low-level loader often caused a nonrecoverable crash, which required restarting the tools. It also turned out that not all engineers were tools engineers. We found a few cases where one of the tools API functions returned "true" from a virtual function but provided no actual code. Developing a Windows application also has its own quirks and can be difficult for engineers unfamiliar with the framework. Finally, we wrote the tools generically to handle core functionality that could be shared across games using the same version of the Turbine Engine. It turned out to be difficult to integrate game-specific functionality on top of the existing tool set. Going forward, we have resolved to educate our engineers, create smarter objects, handle assets smartly and completely, and use profiling tools whenever possible. We're still working on the load times.
5. Lack of tools documentation. Implementing new features and fixing bugs requires the majority of tool development time. There doesn't seem to be enough time in the day to write documents and get the work done. "How-to" documentation takes a good amount of time to do well, and it's a task not everyone enjoys. The AC2 content process suffered from a lack of tools documentation, and since the tools knowledge had to be passed by word-of-mouth from content designer to new hire, we lost valuable time. There were a few short documents describing complex features, but roughly 30 to 40 percent of the functionality went unused because no one knew it was available. Moving forward, we created an internal "tools news" mailing list. When new tools were compiled, the list was updated with the new version's features, bugs resolved, and short "how-to" information. However, not everyone read the mailing list, and I would regularly get questions for how processes worked and feature requests for tools that already existed. In the future, we hope to have more documentation and as a team be more committed to both updating and referencing it. Tools of the Trade As games become larger and more complex, the potential for build-breaking errors becomes more and more likely. Software tools - both internal and external - can provide many features: error checking, error prevention, content generation, and productivity enhancements. If someone must do a repetitive, complex, or boring task, a software solution is ideal. During Asheron's Call 2, Turbine vowed to be smarter about tools development and planned accordingly. At Turbine, we devoted roughly 20 percent of our core engineering budget to tools work. With a dedicated engineer and a plan for tools functionality, we knew exactly what we needed to accomplish and how we hoped to do it. The team found that sharing code between the Turbine Engine and the tools saved a lot of time previewing and tweaking art assets. We learned that design is critical for any software project, not just our client software. Feedback and discussions gave everyone a vested interest in the tools. We've got some work to do for testing and documentation, but we're moving in the right direction. Tools
developers know theirs isn't the most glorified position in a game
company, but it is one of the most critical. People at fan gatherings
don't want to talk to you, and half the people in your company don't
know exactly what you do. Your job can be the bottleneck that strangles
a game, and you may ask, "Is it worth it?" A content designer
once told me that in the original Asheron's Call, he destroyed
the town of Arwic using the first-generation terraforming and content
placement tools. Time spent? One week. Modifying a similar set of
terrain for Asheron's Call 2 with the new tools took an hour
and a half. I'd call that improvement.
[back to] Introduction |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Features
|
|
|