|
 Features
Automated Build and Test Systems for Games
 Executing Build Requests
Once a build request percolates all the way down to a server that is actually going to do the work, we come to the second component of the build system - the scripting framework which executes the builds. We are currently using Apache Foundation’s open source ANT language. It is an XML-based scripting framework developed specifically for build systems. It excels at file system management and allows for conditional logic.
Additionally, it has the ability to create your own plugins if the built in functionality does not cover all of your requirements, though that hasn’t been necessary at Nihilistic. These scripts are responsible for doing the actual grunt work. They execute compilers (versioning executables on the way), run our own tools to parse and pack assets, create ISO images of game discs that are ready to burn, and launch our test system (more on that later). The scripting code is easy to maintain due to its straightforward method names and it’s even possible to data drive - ANT allows for external parameter files - so that the same script code can work with multiple projects.
 Snippet of an ANT XML build script, the syntax is largely straightforward.
Storing Build Results
That brings us to the third component of the build system, a
place to store the results. For the majority of our build types,
this means storing binary data in our source control management
(SCM) application, Perforce. Since the team is already well versed
in using Perforce on a daily basis for assets and code, it’s
a logical place to have them easily retrieve the latest official
builds. Unfortunately, it’s not always practical to store
huge binary data in an SCM database primarily due to speed concerns.
Large compiled asset bundles - which typically don’t benefit
much from version history that an SCM provides - are stored on
a networked server to be retrieved by clients via an automated
batch file which is run on their local system. It’s important
to note that although Perforce is a great application, it’s
definitely not the only option - you should be able to employ any
SCM application for this purpose. A popular open source option
is the Concurrent Versions System (CVS).
Reporting Build Results
Finally, we needed a way to report the results of builds. Once
a build has been requested, fulfilled, and checked into SCM or
copied to a file depot, how is a client to know that occurred?
For us, this was simple - if a build was successful, our build
system sends an e-mail to the person who requested the build and
a mailing list of perpetually interested parties (the build system
maintainer, producers, key QA staff). If a build fails, for example
because code could not compile, the build system sends an e-mail
to the requester and a mailing list of the people who would likely
know how to fix the problem along with an attached log that gives
details on why a failure took place. In our example, the system
sends an e-mail to all of the programmers with a compile log that
directly identifies errors. In addition, the build server does
not submit any data to the SCM application on failure to ensure
no
bad data pollutes the repository. For our medium sized team (for
a next-generation console project) of approximately 50 people,
this method of reporting has been straightforward and has worked
well. On very large teams, especially ones with lots of people
working remotely, it might make more sense to report results in
a web page that the build system updates automatically.
Automated Test System
Automated test systems for games have some intriguing problems
that don’t apply to many other types of software. What does
it mean for a game's overall behavior to be correct, especially
when design can fluctuate frequently? How can you verify if a rendering
system is outputting results that are correct? Is there even a
definition of “correct” behavior when the typical
rating systems for games are fun factor and whether or not something
is subjectively visually pleasing? At a macro level, these questions
are difficult to answer, however there are plenty of metrics you
can measure fairly easily which can start to formulate an answer
to some of these questions. What follows is an overview of the
types of automated testing we have been successful with, specifically
regression testing for crashes as well as benchmarking memory and
performance in our game’s levels. This type of reporting
is useful for all disciplines, but project managers take note -
how nice would it be to receive a formatted spreadsheet of the
state of which parts of the game are loading successfully and are
operating within set budgets, automatically on all platforms, every
morning? Read on to find out how.
A portion of a memory test spreadsheet from a mid-development build of our last project.
|
|
|
|
|
Previous