Gamasutra: The Art & Business of Making Gamesspacer
Product Review: Xoreax Software's Incredibuild

Printer-Friendly VersionPrinter-Friendly Version
View All     RSS
April 24, 2014
arrowPress Releases
April 24, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM TechWeb sites:


 
Product Review: Xoreax Software's Incredibuild

February 15, 2003 Article Start Page 1 of 2 Next
 

Overview

Large render farms of high-end workstations have long been the secret weapon of the artist and as a programmer I've looked on with jealousy at the software provider's support, however minimal it has appeared, of providing distributed rendering systems for packages such as Maya, Kinetix's 3D Studio Max and Pixar's Renderman. This has been exacerbated for me even more by Weta Digital's gushing enthusiasm for their several hundred processor great wall o'renderers that created the large-scale battle scenes in The Two Towers.

Discounting large multi-processor mainframes the ability for a non-mainframe owning developer to compile a single large project across multiple machines has, for a number of years, been the sole domain of Unix & Linux platforms using distributed makefiles. Though difficult to configure and make robust for team use, once the system is up and running it can drastically reduce even the longest compile times. The major downside to it has always been that all systems involved in the build process must have the same compiler versions installed, along with header files, libraries, tools, etc. Hundreds of files, across possibly dozens if not hundreds of machines, that must be kept in lock-step synchronization with each other.

Until recently there has been no viable Microsoft Windows or Microsoft Visual C++ solution so programmers were stuck with their single platform compilation capability, spending, on very large projects, upwards of two hours waiting for a full rebuild of the tool chain or application executable.

But no longer…

Xoreax software released IncrediBuild and changed the way the game is played. IncrediBuild, currently on version 1.133 release and 1.14 beta is their answer to the programmer lusting after the equivalent of distributed makefiles spread over a compilation "render" farm.

What It Does & How It Works

What Xoreax IncrediBuild does is perform a distributed compilation of your C & C++ source files across multiple computers connected via a TCP/IP network - even 802.11b WiFi, even a 56Kbps modem - running Microsoft Windows NT, 2000 or XP - essentially, any Microsoft OS that has the ability to run services in the background that's connected to a network. The incredible part of this - absolutely no puns intended anywhere within this review - is that it can do all of this on machines that do not have Microsoft Developer Studio installed.

For all of this to work Xoreax recommends a minimum of a 200MHz Pentium with 64MB of RAM but I'm sure most game development studios these days would be hard pressed to find a machine that underpowered actually doing any useful office work. A single machine with Microsoft Developer Studio 6.0 along with the Xoreax IncrediBuild DevStudio Add-in installed is all that's required for a development machine, Agents on remote machines and the Coordinator do not require DevStudio to be installed forgoing both the expense of an un-used software package and the requirements of a capable machine. This removes any maintenance headaches that would be caused by ensuring that the same version of compiler, libraries, header files and Service Packs are installed on a diverse network. This also means that the IncrediBuild Agent can be placed on to office machines that are not directly involved with development.

IncrediBuild performs its magic by sharing out the compilation of source files to individual machines running the IncrediBuild Agent. A source file is considered atomic, no further subdivision of labour in compiling it is possible, so the more source files in a project, the better the performance gains witnessed.

Each IncrediBuild Agent, and there can be up to 100 Agents in a build farm though 40 is the recommended maximum, creates a small cache on the HD, and, utilising spare CPU cycles - it runs at the "Idle" priority by default - compiles C & C++ source files just as though they were being compiled on the machine that's running Microsoft Developer Studio. Should an Agent detect that the host processor usage has spiked from user interaction it stops processing and informs the IncrediBuild Coordinator to find another Agent to perform the work. Considering that powerful desktop machines sit idle most of the time, waiting for the user to press another key or move their mouse, there are an awful lot of wasted CPU cycles hovering around, completely under utilised.

The Network Configuration

Setting up an IncrediBuild network is straightforward, install the IncrediBuild Coordinator on a designated machine, and then proceed to load the IncrediBuild Agent -- you must be part of the Administrator or Local Administrator group to be able to use IncrediBuild -- on to as many spare computers as you can find, referring back to the Coordinator either by machine name or IP address, which should preferably be static. The Coordinator & Agents simply run as services in the background, remaining mostly transparent to the regular user. Each machine in the IncrediBuild build farm places a small icon in the system tray to indicate status, and thankfully, Xoreax provided the ability to hide the icon to lessen taskbar clutter.


Figure 1. IncrediBuild DevStudio Build Output window.

In DevStudio, IncrediBuild integrates as an "Add-in", providing another Build menu and toolbar. After correct installation, open the largest DevStudio project you can find and click the "Build Current Project" button on the IncrediBuild toolbar. Remapping the F7 (Build Project) key to IncrediBuild makes it completely transparent. The IncrediBuild DevStudio Add-in supplants the standard Build Output window with it's own, providing extra tabs (see Figure 1) showing how many Agents are available, the progress of the build across each Agent, and how long your build is taking. The Agent progress window is also available from the icon in the system tray. Configuration of available machines (see Figure 2) is trivial via the Build Coordinator, giving the administrator the ability to add, what Xoreax refers to as "subscribe", and remove machines from the build farm through an intuitive interface To quote Ron Popeil on late night TV, the software is very much "set it & forget it."


Figure 2. Build Coordinator

As the build proceeds, the IncrediBuild DevStudio add-in aggregates all of the output, e.g. errors and warnings, (see Figure 3) from individual Agents into a single list that can be navigated normally by double-clicking or pressing the F4 & Shift-F4 keys for "Go to Next Error Tag"/"Go to Previous Error Tag" feature.


Figure 3. The IncrediBuild DevStudio add-in aggregates errors and warnings.

I first tested IncrediBuild on a small network of four machines in various configurations and later, on a sunny Saturday morning in January at a local Art College in Los Angeles on a network of newly installed workstations. The IncrediBuild DevStudio Add-in reports all sufficiently idle Agents as a single number in Megahertz, quintessentially an aggregate of all available CPU power, not the most scientific total, but it gives some idea of the network resources at your disposal. It's quite something to see your overall CPU speed reported as approximately 64,000 MHz, or 66 GHz in modern parlance when you have more than thirty 2.1+ GHz CPUs all ready to do your bidding.

The test network makeup was diverse enough to reflect that found in most companies, a speedy, 2.8GHz Pentium 4 Northwood equipped high-end laptop, a low-end, 1.1GHz AMD, a low-end, dual processor 950MHz Pentium 3 - any multi-processor machine will remain underutilised as IncrediBuild only makes use of a single processor though Xoreax has promised to rectify this in the future - and finally, a large collection of mid-range, 2.1GHz Pentium 4 machines.. The entire network utilised standard, 100MBit Ethernet. An 802.11b WiFi test was performed merely for curiosity, and worked flawlessly, but the timing results are not included.

The home network was configured to run through two consumer level LinkSys routers, a later reconfiguration of the routers allowed me to exercise IncrediBuild through a hardware firewall solution with no detrimental effects so it is possible to compile between remote office locations utilising standard TCP/IP. IncrediBuild has an encryption facility to safeguard your source files as they are passed around the network, all of the test runs were performed with it off, the default setting. With IncrediBuild encryption switched on I saw no performance difference.

Other than the College machines, all computers were running ZoneLabs ZoneAlarm and it worked almost flawlessly, ZoneAlarm popped up the usual dialogues asking for confirmation to access the Internet or act as a server. Only on one machine the IncrediBuild Agent refused to start up correctly until after a reboot, this may have had something to do with ZoneAlarm being installed but not running at the time of installation as ZoneAlarm never truly shuts down when you stop the service and that may have confused the Xoreax IncrediBuild installer. There was no problem with IncrediBuild when ZoneAlarm was switched off after installation.

On one machine the IncrediBuild Add-in installation proved troublesome, it was running a trial copy of Visual Assist and IncrediBuild failed to create the toolbar correctly, though the IncrediBuild menu showed up fine and the IncrediBuild system worked correctly. Thoughtfully Xoreax provides an option to re-create the menu and toolbar from the "Settings…" dialogue.

During testing, though not during the timing runs, I went around unplugging various machines from the network to simulate service outage, even so far as disconnecting the submitting machine from the Build Coordinator during a long complex build, and without a hiccough the whole operation kept processing files, albeit at a much reduced rate, amply showing the graceful degradation of service that the IncrediBuild software provides.

Computer Name
CPU
RAM
OS
Usage
Comments
Miyuke
2x950 MHz P3
512MB
2000 Pro
Agent
The dual 950 MHz machine had a low usage web server with no clients, and an FTP server with 1 user transferring files.
Skuld
2.8 GHz P4
512MB
XP Pro
Agent
See Below
Buffy
1.1 GHz AMD
1GB DDR
2000 Pro
Coordinator
Running Microsoft Developer Studio, the Eudora e-mail client, two web browser windows and ICQ. The user did not compile anything during the test runs.
Giles
400MHz AMD
256MB
2000 Adv Server
Agent
This machine did nothing except run the IncrediBuild Coordinator software.
College
2.1 GHz P4
512MB
XP Pro
Agent
None of the workstations had Microsoft Developer Studio installed.

The load on Skuld, the 2.8 GHz laptop, was what could be considered average for a development machine - running Adobe Photoshop, two copies of Microsoft Developer Studio 6.0, one performing the actual project compile, WinAmp playing MP3s, two windows of Microsoft Office, one to write this review, one to read over a design document that had a lot of graphics in it, and Outlook checking e-mail in the background - and I can't say I noticed any slow-down when the Agent was compiling a project for a user on another machine. Skuld generally chews through a C++ source file with a full complement of nested "#include" headers at about 1 per second so it's a good base machine to be comparing against cost savings when weighing the speed gains given by a faster CPU and more RAM versus 3rd party software to perform a distributed task across idle development machines. Skuld was configured as both Agent and Coordinator when networked with the thirty College machines with no apparent detrimental effects other than a spike in network utilisation.

Buffy, the 1.1GHz AMD used SETI@Home configured to run constantly, the IncrediBuild Coordinator detected this high CPU utilisation and promptly removed the Agent from the list of those available. Certainly something to be aware of if you are considering installing IncrediBuild in a team environment, team members need to be aware that running processor intensive screen savers, the SETI@Home project or those that show off the capabilities of a GeForce card, is undesirable.

Configuration A
Configuration B
Configuration C
Configuration D
Configuration E
Skuld (2.8 GHz)
Skuld (2.8 GHz)
Skuld (2.8 GHz)
Skuld (2.8 GHz)
Skuld (2.8 GHz)

Miyuke (2x950 MHz)
Miyuke (2x950 MHz)
College (2.1 GHz) x9
College (2.1 GHz) x30


Buffy (1.1 GHz)


      All nine workstations were idle and unused for all of the test runs. All workstations were idle except during Project 5. See the notes for Project 5 for a further explanation.
Table: Network Topology

As the build progresses the DevStudio Add-in reports a running total of source files compiled, normally this ticks over at one or two per second on the laptop, on the College network it was ticking over at between thirty and forty source files per second. Every time I watched it I was mesmerised by the rapidly increasing source file counter. I'd suggest to Xoreax that they also add a source line running total too.


Figure 3. A horizonal progress-bar represents each file as it compiles on various agents.

Interestingly the IncrediBuild DevStudio Add-in shows the progress of the build graphically, (see Figure 4) a horizontal progress-bar or coloured candlestick represents each file as it compiles on the various agents, the candlestick changing colour depending on the results of the compile, light blue for dependency updates and local tool execution, blue for compiling, green for successful compile, yellow for issued warnings, and red for fatal errors.

IncrediBuild obviously supports simultaneous builds submitted from multiple clients, constantly dividing up the available pool of CPU's until the scheduled jobs are completed.


Article Start Page 1 of 2 Next

Related Jobs

Turbine Inc.
Turbine Inc. — Needham, Massachusetts, United States
[04.23.14]

Director, Analytics Platform Development
Gameloft Toronto
Gameloft Toronto — Toronto, Ontario, Canada
[04.23.14]

Technical Artist
Muti Labs
Muti Labs — Santa Monica, California, United States
[04.23.14]

Senior Game Engineer
Digital Extremes
Digital Extremes — London, Ontario, Canada
[04.23.14]

Programmers






Comments



none
 
Comment: