Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Automating the Build Process
May 23, 2019
May 23, 2019
Press Releases
May 23, 2019
Games Press

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

# Automating the Build Process

December 8, 2000 Page 2 of 4

Implementing It Yourself

Creatures 3's build process uses Unix tools running under Windows. These are free tools, licensed under the GPL, and ported to Windows by Cygnus.

The main tool is the Bash (Born Again SHell), which is a popular Unix command line shell. It has a powerful scripting language, which provides all the logic conditions and loops that you need and much more. It calls the standard Unix commands for file copying and moving.

The second important tool is Perl, the famous "glue" language. You can easily do complex text file manipulation with Perl.

We used Unix tools for three key reasons. They've got a very rich feature set, like DOS on steroids. Once you know how to, you can do anything with them. Secondly, the skill is portable. We've used the same skills to administer Unix-based servers running game web sites, and they may help us to make best use of the PS2 development kits. The third reason? They're free and open. We could play with them straight away without a purchase order.

There aren't any very strong alternatives to the Unix tools for scripting. The Windows command line is impotent, and commercial alternatives like 4DOS/TakeCommand/MKS gain you nothing for their price.

More recently, the Windows Scripting Host has got a lot better, and you might want to try it out. Find out more at http://msdn.microsoft.com/scripting/. You can do file copying using the FileSystemObject.

The machine that you run the build on is quite important. The Creatures 3 build process ran on a dedicated computer, which was of high specification at the time. This stops it interfering with the developer's main machine, and is also useful for soak tests, and other tests running in the debugger. We use Windows NT or 2000 for build machines. They are much more stable than Windows 95/98, particularly for running command line programs.

Accessing Version Control

Most teams store at least some assets in a version control system, so it is essential for the build script to be able to retrieve data from them. At Creature Labs we used Microsoft's SourceSafe for development of Creatures 3. Since then we have moved on to the excellent CVS. This was because we've found it to be more reliable, it's portable (I've transfered game web site code between Windows running IIS and Linux running Apache) and it's free.

If you know how to use CVS from the command line then you will find it very easy to call from your build process. SourceSafe is a little trickier. You have to set up some environment variables which aren't very well documented.

The following code is the set of environment variables that I use.

echo -n Setting environment...
export VSSDir="C:\\PROGRA~1\\Microsoft Visual Studio\\COMMON\\vss\\win32"
export PATH="$VSSDir":"$PATH"
export SSUSER=cbuilder
export SSDIR="$SOURCE_SAFE_C2E/Database/" echo done SSUSER is the user name in SourceSafe. We have a special user called "cbuilder" which is the build script. There's also a Windows network user with the same name. This way anyone can be taught to log into the build machine as "cbuilder", and network drive permissions won't cause trouble for the script. With your environment set up, it is relatively easy to call SourceSafe. Have a look at the documentation, and these example commands. ss Undocheckout$/Code/engine/build.h -I-Y
ss Checkout $/Code/engine/build.h "-cAutobuild checkout" || error ss Checkout ss Label \$/Code "-L$VERSION_LABEL" "-CMade by automatic engine build" || error ss Label ss Get$/Code/engine "-GLengine/" -O- -I-Y -R -GF- -GWS "-VL$VERSION_LABEL" || error ss Get latest version ss Checkin$/Code/engine/build.h "-cAutobuild checkin" || error ss Checkin

The exclusive checkout nature of SourceSafe does causes a few problems. You need to make sure the script undoes any checkouts if there is an error. Alternatively the script can try to undo checkouts as the first thing it does. That way it recovers from previous aborted runs.

You should make your script automatically tag your code and assets with the build or version number. When you do this make sure you are tagging the actual version that the script is using, even if somebody else changes files while the script is running.

For example, with SourceSafe you need totag a version first. Then do "ss Get" with that label, as in the example line of code above. This guarantees you are compiling exactly the labelled version.

To use CVS from the command line, you don't need to do anything special. It comes included with the latest version of Cygwin. Partly as a code example, here is a function which ensures the user has logged in (using pserver protocol) on the build machine.

# Set HOME and CVSROOT before calling
function cvs_loggedin()
{

if [ ! -f "$HOME/.cvspass" ]; then echo Please log in the first time cvs login fi cat "$HOME/.cvspass" | grep "$CVSROOT" >nul if [$? == 1 ]; then
elif [ $? == 2 ]; then error grep fi } Tagging a version in CVS is much easier. If you check out a particular version and tag it, then the tags apply to the version that you checked out. This is in contrast to SourceSafe where the tags apply to the latest version in the repository. Compiling the Executable It's very handy to be able to automatically make a new version of your game engine. It's often useful to do this between full builds, so level designers or artists can get working with new features. Before compiling you need to fetch the code from source control. It's useful at this point to automatically increase the version number. The Creatures 3 game engine has a hard-coded version number. The script updated this version by editing a header file, and checking it back into source control. According to your compiler, there will be some way of building your project from the command line. If you are using Visual C++, don't follow the obvious documentation and go messing about with exported make files and nmake. You can just call the IDE in command line mode. msdev "engine.dsp" /MAKE "engine - Win32 Release" /OUT "build.log" || error msdev Before you call it you need to set up some environment variables - mainly PATH, LIB and INCLUDE. You can find these in the file VCVARS32.BAT which is in the Visual C++ bin directory. Under Windows NT/2000 the Visual C++ installer can set them up for you in the global environment. When the build has completed the script can copy the file to the network, with an appropriate name to mark its version number. So that everyone knew what has changed in the new executable, Creatures 3 had a ChangeLog file in version control. Whenever someone made a change, they added a comment to this file saying what they have done. The engine build process checked this file out, and posted its contents to an internal newsgroup announcing the new engine. It then cleared out the file for future use, and checked it back in. At last, agent engineers and level designers know exactly what is new! File Copying To transfer assets from and to the network you need the full suite of file copying commands. They can seem quite slow under Cygwin, but they are fairly intuitive to use. Here are some random examples: mv engineSDL.exe engine.exe mv -f *.gen old/ mv My\ Agents/* "Resource Files Directory/" cp -R "CD Image" "//oracle/c3/Completed Builds/new build being copied" mkdir Source\ Data/Catalogue rm -fR Source\ Data/Bootstrap cd$BUILD_DIR

Notice the backslash to escape spaces, as in My\ Agents. Alternatively, you can use quotes to similar effect, as in "Resource Files Directory/". If you do that you need to put any wildcard * outside the quotes, or else it will be protected as well.

The -R stands for recursive, and confirms that you want to copy or delete a whole tree. The -f in the rm (remove) command is used to suppress errors if the file isn't there. This is useful in a build script. You can use network paths (UNC) directly, as in "//oracle/c3/".

Text Processing

The Unix command line is excellent at manipulating text files. They are its bread and butter. You can use text processing to update version numbers, to fiddle with InstallShield scripts and to check for errors in your game script files. It's a bit of a black art, but well worth learning.

Quite a lot can be done within Bash. Have a read of the manual, particularly the section on "Parameter expansion". It can be more fun, and clearer, to use standard Unix command line tools.

Grep (or egrep, which is the same with a slightly more standard syntax) can be used to search for lines within a file in a powerful way. The following example finds all the lines in a file which contain the text "sndc" or "sndl" or "sndq" or "snde".

egrep -h sndc\|sndl\|sndq\|snde "$COSFILE" >> /tmp/sounds Those are all sound commands in our scripting language. The egrep line is part of a function to verify that all sound files are present in the build. All the funny characters form regular expressions. This is a powerful way of pattern matching. It can do whatever you want, when you work out how to do it. Type "man grep" to find out the basics. Notice that there is some more escaping with \ to make sure the | (or) characters get through to grep, rather than being treated as a pipe by the shell. This example replaces the text "My Creatures" with the Dutch "Mijn Creatures" in an InstallShield script file. perl -pi.bak -e"s/My Creatures/Mijn Creatures/gi" InstallShield/Script\ Files/Setup.rul Using Perl has the advantage that it can edit a file in place. Sed (short for Stream EDitor) is similar to this restricted use of Perl, but can only be used with redirections. You have to make a new file. It's useful if you're in the middle of a long pipe of other commands though. sed <"$COSFILE" >tempcos "s/^[^*]*dbg: outv/\*&/i;s/^[^*]*dbg: outs/\*&/i"

That slightly crazy command comments out debugging lines in our internal scripting language. Any lines which begin "dbg: outs" or "dbg: outv" get replaced with "* dbg: outs" and "* dbg: outv". The asterisk is the comment character. Notice the complex regular expressions - they make more sense when you write them than when you read them.

Also worth looking at is "diff".

diff /tmp/soundsfail /tmp/soundsok
if [ $? == 1 ]; then error "Sound files missing" elif [$? == 2 ]; then
error diff
fi


These are very powerful tools that can be strung together to do lots of things. Since learning them I've find them very useful for analysing log files and writing one-off scripts for many purposes.

Page 2 of 4

### Related Jobs

Crystal Dynamics — Redwood City, California, United States
[05.22.19]

FX Artist
Hyper Hippo Games — Kelowna, British Columbia, Canada
[05.22.19]

Software Developer - Unity
Wizards of the Coast — Renton, Washington, United States
[05.22.19]

Sr Lead Software Engineer - Arena
Wizards of the Coast — Renton, Washington, United States
[05.22.19]