Tools development has historically been very bad about software testing. There's no time, no budget, and no personnel. We end up pushing out untested tools on our users, who in turn are frustrated with buggy code. This approach ultimately costs us more time in fixing those bugs. Testing software tools takes two forms: white box testing and black box testing.
White box testing is done by the developer, stepping through the code pathways and making sure things are coded correctly. It's called "white box" because the developer can see inside the box (his code). Black box testing is performed by the Quality Assurance Team; given a set of inputs they expect a set of matching output. They don't have any idea what's inside the box. Your software process should include both sets of testing.
Your developers should at least be testing their own code. White box testing is pretty specific to the functionality being tested. At a minimum, the developer should provide complete code coverage. This means that the developer should test every line of code they write: all of their if/switch statements, successful code path, and error cases. This can often be time-consuming, testing all of the code pathways, but it is the only way to guarantee that the module or function is going to execute correctly under all circumstances. Prior to checking in code, completing a peer-review of the changes (plus a demonstration where applicable) with a fellow developer will make sure that all requirements are addressed, reduce errors, and increase adherence to the coding standards.
Your software process will directly affect how easy it is for a QA team to complete black box testing of your tools. The requirements specification lets the QA team know exactly what changed. The "use cases" in the high level design indicate how the new functionality can be tested. Using those two documents, the QA team can write a test plan for the functionality. Using the test plan and the software tool, bugs or unexpected results are added to a bug databases for fixing and tracking. Additional daily testing can be performed by the QA team, using a functional overview test that exercises the basics of the tool. Turbine now has a dedicated QA department to perform the black box testing. It is true that they are focused on testing the game clients and filing bugs in our internal database, but we are moving forward with plans to have them begin testing the tools as well.
One idea I want to implement is self-testing software tools. By using a command line parameter or having a "Test" menu item, we could allow each software tool to complete a self-diagnostic. Perhaps the tool could create an object in the world, move, rotate, and scale it, delete it, and undo the deletion. Such a simple test case would easily exercise fifty percent of the most common tasks inside our world-building tool. Extending the test cases for each particular tool would give the QA department a single entry point in the tool for daily functional testing, and simplify the burden of testing our tools.
Great! The code has been written and you managed to find some time to do thorough testing; the tools have been distributed. All the users are happy, right? The polish phase is the time to make sure that everything is easy to use, runs fast, and performs well. One of the best ways to do this is to have an internal post-mortem. This may be as formal as gathering everyone in a conference room and airing grievances about the tools; it may be as informal as dropping by the user's desks and asking them what their biggest impediment to productivity is.
Turbine conducted a full internal tools post-mortem towards the end of the AC2: Fallen Kings product development cycle. By doing the post-mortem prior to the game going live, we got a glimpse at the problems the designers would be facing during the monthly props. As a result, we created a brand new tool that simplified the generation and editing of our string tables (localizable versions of the game text), as well as pointed out the need to rewrite portions of the basic AC2 text controls.
Using your own tools to complete a software task is like taking a dose of your own medicine. You can receive good feedback by performing usability testing; pairing users and the developer after the tool or added functionality is complete. Tacking on the pressure of working with real game assets alongside a designer, the developer quickly sees what improvements can be made. During polishing, we add features for the "expert user." The two most common additions are keyboard shortcuts and toolbar icons for frequently used commands. Turbine has also implemented a timing viewer that allows us to take a snapshot of the time taken inside various subroutines and displays the results in a tree view. This tool has helped identify times where excessive loading and processing takes place, and allowed us to make enhancements.
Finally, are we done? We've covered all five steps of the software process, but we aren't quite done yet. Now that you've completed the process, it's time for the sixth, hidden step: iteration. Iteration is especially vital for MMO tool development. Fifty percent of your Live team will probably be content designers, who will be using your software tools every day for years to come. In our experience, the Live team budget and developer staff does not take into account tools development. This leaves the Live team with whatever was completed by the time the game goes public.
Dungeons and Dragons Online
At the end of AC2's development, the major "tools to-do list" contained more than 75 pending feature requests. Fortunately, we were able to spend another three months on some of those issues following the game's release. However, there are still problems that haunt us to this day. Our engine changes significantly between projects. Consequently, the tools for Middle-Earth Online and Dungeons & Dragons Online are considerably different from those in AC2. We attempt to back-port changes from the latest engine development tree into other game trees whenever possible, but sometimes the underlying engine changes (and budgets) preclude the ability to do so.
A small example of our software process may help illustrate the various phases discussed above. This example was taken from a recently requested feature for our world building tool. The content designers wanted functionality similar to Maya or Photoshop, such that any selected entity could be assigned to a visibility layer and the visibility of the layer could be toggled. Entities and layers made invisible in the world building tool would not be rendered by the engine. A small section of the requirements document follows:
Requirement 1: A new menu option "Add to Layer ->" will be added to the right-click menu, which will provide a method for adding entities that are selected in the various workspaces to a layer. Entities added to a layer in this manner will assume the visibility status of the destination layer.
Requirement 2: The right click menu will contain an option for "New Layer..." where the user will be queried for a new layer name.
Requirement 3: The right click menu will contain a programmatically generated list of layers.
Requirement 4: A "Remove from Layer" menu option will remove the entity from a layer, if it is currently assigned to one.
Requirement 5: When one or more entities are selected and the right click menu option "Add to Layer" is selected, the entities will assume the visibility status of the target layer.
Note that each requirement above consists of a single, clear, testable statement. This focuses the reader on one item at a time as well as making testing the requirements easier. At this point, the requirements are distributed, signed off on, and scheduled. This particular task involved creating a new menu and dialog box, as well as a few supporting classes to track the visibility states. The task was further complicated by another requirement to save the visibility state in our data files and restore it when the entities were later reloaded.
The next step was to create "use cases" for these requirements in a high level design document:
Use Case 1: The user has selected some entities in the 3D workspace and clicked the right mouse button. The right click menu will contain the "Add New Layer…" menu option at the top of the menu. A list of menu options will show for "Add to LayerName" where "LayerName" is each layer from the list of previously created layer names. Finally, the "Remove from Layer" menu option will be displayed. (Reqs. 1, 2, 3, 4)
Use Case 2: The user has selected some entities in the 3D workspace, clicked the right mouse button, and selected the "Add to LayerA" menu item. If LayerA is visible, each of the selected entities will toggle their visibility to "visible." If LayerA is invisible, each of the selected entities will toggle their visibility to "invisible." LayerA will now own the visibility state of each selected entity. The VisibilityLayer dialog box will be updated to show that "LayerA" contains each entity name as a child node in the tree view. (Req. 5)
The use cases contain more information than the requirements. It is still possible to map the requirements back to the design: the first use case contains the first four requirements, while the second use case contains the fifth requirement. Numbering the requirements is not required, but it can help identify if everything has been addressed.
The use cases are broken down in the detailed design document to the class/function/module level:
Use Case 1: The user has selected some entities in the 3D workspace and clicked the right mouse button.
- All functionality will take place in Wb3DWorkspace::BuildMenu.
- Determine if any entities are selected. If not, skip generating the menu.
- Add a menu option "Add New Layer…".
- Get the list of VisibilityLayers (new class) from the LayerManager (new class).
- For each VisibilityLayer, retrieve the LayerName and add a menu option "Add to LayerName"
- Add a menu option "Remove from Layer"
From the detailed design, we begin to see the classes, functions, and member variables that will ultimately be created in code. One small section of code generated from Use Case 1 illustrates some of the coding standards we use at Turbine.
Wb3DWorkspace::BuildMenu( const bool& i_bSelection,
Menu* io_pMenu )
// if the LayerManager hasn't initialized, we cannot // build a menu
if( !m_pLayerManager )
// if the menu is invalid, we can't add to it
if( !io_pMenu )
// if nothing is selected, we don't need to add items to // the menu.
// NOTE: this case is not an error!
if( !i_bSelection )
// add the "Add to New Layer" menu item
// add all of the existing layers as menu items
const SmartArray< VisibilityLayer* >& arrayLayers =
if( arrayLayers.count() > 0 )
for( il = 0; il < arrayLayers.count(); ++il )
VisibilityLayer* pLayer = arrayLayers[ il ];
if( pLayer )
io_pMenu->AddMenuItem( TEXT( "Add To %s" ),
ID_ADD_TO_LAYER + il );
// add the "Remove from Layer" menu item
White box testing of the above code is completed in the debugger, setting a breakpoint at the first line of the function and adjusting the input parameters, as well as the return values from called functions. Some code, such as assertions, has been removed for brevity. Black box testing involved several test cases: right clicking with entities selected and unselected and visibility layers created and absent. The resulting menu was visually inspected to verify that the menu items were only displayed when there were entities selected, and that visibility layer menu items only appeared when there were existing layers.
During the polish phase, we added a few features to the LayerManager: dragging and dropping was enabled for entities between VisibilityLayers and doubleclicking a VisibilityLayer name in the LayerDialog caused the layer and all entities assigned to the layer to toggle their visibility status. Future requests and iteration will likely reveal more improvements for the users. While the previous example is understandably small due to space considerations, it reasonably displays the level of effort and detail used in Turbine's software development process for our tools.
Massively multiplayer online game creation is more than getting a box onto a store shelf. You must think about the future. Similarly, your software development effort is not solely about the game, it encompasses the tools that the Live team will use to support the game in the future. Ensure the future operation (and income) of your MMO with well designed tools that make content creation easy for your designers. As the saying goes, "An ounce of prevention is worth a pound of cure." Catching a problem early in your tool's design phase will cost significantly less if fixed at that time, rather than waiting and discovering a bug after the tool has been released to the users. A sound software methodology that is iterated on from requirements through polish is crucial not only to game development but to software tool development as well. Is your software process working for you?