|
|
 |
 |  | | | | |  | |||
 | ||||||
          |  |  | ||||
|
A
Software Process for Tools development has historically been very bad about software testing. There's no time, no budget, and no personnel. We end up pushing out untested tools on our users, who in turn are frustrated with buggy code. This approach ultimately costs us more time in fixing those bugs. Testing software tools takes two forms: white box testing and black box testing. White box testing is done by the developer, stepping through the code pathways and making sure things are coded correctly. It's called "white box" because the developer can see inside the box (his code). Black box testing is performed by the Quality Assurance Team; given a set of inputs they expect a set of matching output. They don't have any idea what's inside the box. Your software process should include both sets of testing. Your developers should at least be testing their own code. White box testing is pretty specific to the functionality being tested. At a minimum, the developer should provide complete code coverage. This means that the developer should test every line of code they write: all of their if/switch statements, successful code path, and error cases. This can often be time-consuming, testing all of the code pathways, but it is the only way to guarantee that the module or function is going to execute correctly under all circumstances. Prior to checking in code, completing a peer-review of the changes (plus a demonstration where applicable) with a fellow developer will make sure that all requirements are addressed, reduce errors, and increase adherence to the coding standards. Your software process will directly affect how easy it is for a QA team to complete black box testing of your tools. The requirements specification lets the QA team know exactly what changed. The "use cases" in the high level design indicate how the new functionality can be tested. Using those two documents, the QA team can write a test plan for the functionality. Using the test plan and the software tool, bugs or unexpected results are added to a bug databases for fixing and tracking. Additional daily testing can be performed by the QA team, using a functional overview test that exercises the basics of the tool. Turbine now has a dedicated QA department to perform the black box testing. It is true that they are focused on testing the game clients and filing bugs in our internal database, but we are moving forward with plans to have them begin testing the tools as well. One idea I want to implement is self-testing software tools. By using a command line parameter or having a "Test" menu item, we could allow each software tool to complete a self-diagnostic. Perhaps the tool could create an object in the world, move, rotate, and scale it, delete it, and undo the deletion. Such a simple test case would easily exercise fifty percent of the most common tasks inside our world-building tool. Extending the test cases for each particular tool would give the QA department a single entry point in the tool for daily functional testing, and simplify the burden of testing our tools. Polishing Great! The code has been written and you managed to find some time to do thorough testing; the tools have been distributed. All the users are happy, right? The polish phase is the time to make sure that everything is easy to use, runs fast, and performs well. One of the best ways to do this is to have an internal post-mortem. This may be as formal as gathering everyone in a conference room and airing grievances about the tools; it may be as informal as dropping by the user's desks and asking them what their biggest impediment to productivity is. Turbine conducted a full internal tools post-mortem towards the end of the AC2: Fallen Kings product development cycle. By doing the post-mortem prior to the game going live, we got a glimpse at the problems the designers would be facing during the monthly props. As a result, we created a brand new tool that simplified the generation and editing of our string tables (localizable versions of the game text), as well as pointed out the need to rewrite portions of the basic AC2 text controls. Using your own tools to complete a software task is like taking a dose of your own medicine. You can receive good feedback by performing usability testing; pairing users and the developer after the tool or added functionality is complete. Tacking on the pressure of working with real game assets alongside a designer, the developer quickly sees what improvements can be made. During polishing, we add features for the "expert user." The two most common additions are keyboard shortcuts and toolbar icons for frequently used commands. Turbine has also implemented a timing viewer that allows us to take a snapshot of the time taken inside various subroutines and displays the results in a tree view. This tool has helped identify times where excessive loading and processing takes place, and allowed us to make enhancements. Iterate Finally, are we done? We've covered all five steps of the software process, but we aren't quite done yet. Now that you've completed the process, it's time for the sixth, hidden step: iteration. Iteration is especially vital for MMO tool development. Fifty percent of your Live team will probably be content designers, who will be using your software tools every day for years to come. In our experience, the Live team budget and developer staff does not take into account tools development. This leaves the Live team with whatever was completed by the time the game goes public.
At the end of AC2's development, the major "tools to-do list" contained more than 75 pending feature requests. Fortunately, we were able to spend another three months on some of those issues following the game's release. However, there are still problems that haunt us to this day. Our engine changes significantly between projects. Consequently, the tools for Middle-Earth Online and Dungeons & Dragons Online are considerably different from those in AC2. We attempt to back-port changes from the latest engine development tree into other game trees whenever possible, but sometimes the underlying engine changes (and budgets) preclude the ability to do so. A small example of our software process may help illustrate the various phases discussed above. This example was taken from a recently requested feature for our world building tool. The content designers wanted functionality similar to Maya or Photoshop, such that any selected entity could be assigned to a visibility layer and the visibility of the layer could be toggled. Entities and layers made invisible in the world building tool would not be rendered by the engine. A small section of the requirements document follows:
Note that each requirement above consists of a single, clear, testable statement. This focuses the reader on one item at a time as well as making testing the requirements easier. At this point, the requirements are distributed, signed off on, and scheduled. This particular task involved creating a new menu and dialog box, as well as a few supporting classes to track the visibility states. The task was further complicated by another requirement to save the visibility state in our data files and restore it when the entities were later reloaded. The next step was to create "use cases" for these requirements in a high level design document:
The use cases contain more information than the requirements. It is still possible to map the requirements back to the design: the first use case contains the first four requirements, while the second use case contains the fifth requirement. Numbering the requirements is not required, but it can help identify if everything has been addressed. The use cases are broken down in the detailed design document to the class/function/module level:
From the detailed design, we begin to see the classes, functions, and member variables that will ultimately be created in code. One small section of code generated from Use Case 1 illustrates some of the coding standards we use at Turbine.
White box testing of the above code is completed in the debugger, setting a breakpoint at the first line of the function and adjusting the input parameters, as well as the return values from called functions. Some code, such as assertions, has been removed for brevity. Black box testing involved several test cases: right clicking with entities selected and unselected and visibility layers created and absent. The resulting menu was visually inspected to verify that the menu items were only displayed when there were entities selected, and that visibility layer menu items only appeared when there were existing layers. During the polish phase, we added a few features to the LayerManager: dragging and dropping was enabled for entities between VisibilityLayers and doubleclicking a VisibilityLayer name in the LayerDialog caused the layer and all entities assigned to the layer to toggle their visibility status. Future requests and iteration will likely reveal more improvements for the users. While the previous example is understandably small due to space considerations, it reasonably displays the level of effort and detail used in Turbine's software development process for our tools. Massively
multiplayer online game creation is more than getting a box onto a store
shelf. You must think about the future. Similarly, your software development
effort is not solely about the game, it encompasses the tools that the
Live team will use to support the game in the future. Ensure the future
operation (and income) of your MMO with well designed tools that make
content creation easy for your designers. As the saying goes, "An
ounce of prevention is worth a pound of cure." Catching a problem
early in your tool's design phase will cost significantly less if fixed
at that time, rather than waiting and discovering a bug after the tool
has been released to the users. A sound software methodology that is iterated
on from requirements through polish is crucial not only to game development
but to software tool development as well. Is your software process working
for you?
 [back
to] Introduction |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
Resource
Guide 
