Implementing It Yourself
Creatures 3's build process uses Unix tools running under Windows. These are free tools, licensed under the GPL, and ported to Windows by Cygnus.
The main tool is the Bash (Born Again SHell), which is a popular Unix command line shell. It has a powerful scripting language, which provides all the logic conditions and loops that you need and much more. It calls the standard Unix commands for file copying and moving.
The second important tool is Perl, the famous "glue" language. You can easily do complex text file manipulation with Perl.
We used Unix tools for three key reasons. They've got a very rich feature set, like DOS on steroids. Once you know how to, you can do anything with them. Secondly, the skill is portable. We've used the same skills to administer Unix-based servers running game web sites, and they may help us to make best use of the PS2 development kits. The third reason? They're free and open. We could play with them straight away without a purchase order.
There aren't any very strong alternatives to the Unix tools for scripting. The Windows command line is impotent, and commercial alternatives like 4DOS/TakeCommand/MKS gain you nothing for their price.
More recently, the Windows Scripting Host has got a lot better, and you might want to try it out. Find out more at http://msdn.microsoft.com/scripting/. You can do file copying using the FileSystemObject.
The machine that you run the build on is quite important. The Creatures 3 build process ran on a dedicated computer, which was of high specification at the time. This stops it interfering with the developer's main machine, and is also useful for soak tests, and other tests running in the debugger. We use Windows NT or 2000 for build machines. They are much more stable than Windows 95/98, particularly for running command line programs.
Most teams store at least some assets in a version control system, so it is essential for the build script to be able to retrieve data from them. At Creature Labs we used Microsoft's SourceSafe for development of Creatures 3. Since then we have moved on to the excellent CVS. This was because we've found it to be more reliable, it's portable (I've transfered game web site code between Windows running IIS and Linux running Apache) and it's free.
If you know how to use CVS from the command line then you will find it very easy to call from your build process. SourceSafe is a little trickier. You have to set up some environment variables which aren't very well documented.
code is the set of environment variables that I use.
echo -n Setting environment...
export VSSDir="C:\\PROGRA~1\\Microsoft Visual Studio\\COMMON\\vss\\win32"
SSUSER is the user name in SourceSafe. We have a special user called "cbuilder" which is the build script. There's also a Windows network user with the same name. This way anyone can be taught to log into the build machine as "cbuilder", and network drive permissions won't cause trouble for the script.
environment set up, it is relatively easy to call SourceSafe. Have a look
at the documentation, and these example commands.
ss Undocheckout $/Code/engine/build.h -I-Y
ss Checkout $/Code/engine/build.h "-cAutobuild checkout" || error ss Checkout
ss Label \$/Code "-L$VERSION_LABEL" "-CMade by automatic engine build" || error ss Label
ss Get $/Code/engine "-GLengine/" -O- -I-Y -R -GF- -GWS "-VL$VERSION_LABEL" || error ss Get latest version
ss Checkin $/Code/engine/build.h "-cAutobuild checkin" || error ss Checkin
The exclusive checkout nature of SourceSafe does causes a few problems. You need to make sure the script undoes any checkouts if there is an error. Alternatively the script can try to undo checkouts as the first thing it does. That way it recovers from previous aborted runs.
You should make your script automatically tag your code and assets with the build or version number. When you do this make sure you are tagging the actual version that the script is using, even if somebody else changes files while the script is running.
For example, with SourceSafe you need totag a version first. Then do "ss Get" with that label, as in the example line of code above. This guarantees you are compiling exactly the labelled version.
To use CVS from the command line, you don't need to do anything special. It comes included with the latest version of Cygwin. Partly as a code example, here is a function which ensures the user has logged in (using pserver protocol) on the build machine.
# Set HOME and CVSROOT before calling
# get them to log into CVS, if necessary}
if [ ! -f "$HOME/.cvspass" ]; then
echo Please log in the first timefi
cat "$HOME/.cvspass" | grep "$CVSROOT" >nul
if [ $? == 1 ]; then
echo Please log in the first timeelif [ $? == 2 ]; then
Tagging a version in CVS is much easier. If you check out a particular version and tag it, then the tags apply to the version that you checked out. This is in contrast to SourceSafe where the tags apply to the latest version in the repository.
Compiling the Executable
It's very handy to be able to automatically make a new version of your game engine. It's often useful to do this between full builds, so level designers or artists can get working with new features.
Before compiling you need to fetch the code from source control. It's useful at this point to automatically increase the version number. The Creatures 3 game engine has a hard-coded version number. The script updated this version by editing a header file, and checking it back into source control.
to your compiler, there will be some way of building your project from
the command line. If you are using Visual C++, don't follow the obvious
documentation and go messing about with exported make files and nmake.
You can just call the IDE in command line mode.
msdev "engine.dsp" /MAKE "engine - Win32 Release" /OUT "build.log" || error msdev
Before you call it you need to set up some environment variables - mainly PATH, LIB and INCLUDE. You can find these in the file VCVARS32.BAT which is in the Visual C++ bin directory. Under Windows NT/2000 the Visual C++ installer can set them up for you in the global environment.
When the build has completed the script can copy the file to the network, with an appropriate name to mark its version number.
So that everyone knew what has changed in the new executable, Creatures 3 had a ChangeLog file in version control. Whenever someone made a change, they added a comment to this file saying what they have done.
The engine build process checked this file out, and posted its contents to an internal newsgroup announcing the new engine. It then cleared out the file for future use, and checked it back in. At last, agent engineers and level designers know exactly what is new!
To transfer assets from and to the network you need the full suite of file copying commands. They can seem quite slow under Cygwin, but they are fairly intuitive to use.
some random examples:
mv engineSDL.exe engine.exe
mv -f *.gen old/
mv My\ Agents/* "Resource Files Directory/"
cp -R "CD Image" "//oracle/c3/Completed Builds/new build being copied"
mkdir Source\ Data/Catalogue
rm -fR Source\ Data/Bootstrap
Notice the backslash to escape spaces, as in My\ Agents. Alternatively, you can use quotes to similar effect, as in "Resource Files Directory/". If you do that you need to put any wildcard * outside the quotes, or else it will be protected as well.
The -R stands for recursive, and confirms that you want to copy or delete a whole tree. The -f in the rm (remove) command is used to suppress errors if the file isn't there. This is useful in a build script. You can use network paths (UNC) directly, as in "//oracle/c3/".
The Unix command line is excellent at manipulating text files. They are its bread and butter. You can use text processing to update version numbers, to fiddle with InstallShield scripts and to check for errors in your game script files. It's a bit of a black art, but well worth learning.
Quite a lot can be done within Bash. Have a read of the manual, particularly the section on "Parameter expansion". It can be more fun, and clearer, to use standard Unix command line tools.
egrep, which is the same with a slightly more standard syntax) can be
used to search for lines within a file in a powerful way. The following
example finds all the lines in a file which contain the text "sndc"
or "sndl" or "sndq" or "snde".
egrep -h sndc\|sndl\|sndq\|snde "$COSFILE" >> /tmp/sounds
Those are all sound commands in our scripting language. The egrep line is part of a function to verify that all sound files are present in the build. All the funny characters form regular expressions. This is a powerful way of pattern matching. It can do whatever you want, when you work out how to do it. Type "man grep" to find out the basics.
Notice that there is some more escaping with \ to make sure the | (or) characters get through to grep, rather than being treated as a pipe by the shell.
replaces the text "My Creatures" with the Dutch "Mijn Creatures"
in an InstallShield script file.
perl -pi.bak -e"s/My Creatures/Mijn Creatures/gi" InstallShield/Script\ Files/Setup.rul
has the advantage that it can edit a file in place. Sed (short for Stream
EDitor) is similar to this restricted use of Perl, but can only be used
with redirections. You have to make a new file. It's useful if you're
in the middle of a long pipe of other commands though.
sed <"$COSFILE" >tempcos "s/^[^*]*dbg: outv/\*&/i;s/^[^*]*dbg: outs/\*&/i"
That slightly crazy command comments out debugging lines in our internal scripting language. Any lines which begin "dbg: outs" or "dbg: outv" get replaced with "* dbg: outs" and "* dbg: outv". The asterisk is the comment character. Notice the complex regular expressions - they make more sense when you write them than when you read them.
looking at is "diff".
diff /tmp/soundsfail /tmp/soundsok if [ $? == 1 ]; then error "Sound files missing" elif [ $? == 2 ]; then error diff fi
These are very powerful tools that can be strung together to do lots of things. Since learning them I've find them very useful for analysing log files and writing one-off scripts for many purposes.