Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
July 30, 2014
arrowPress Releases
July 30, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Long build times? You're doing it wrong.
by Bram Stolk on 07/10/14 12:39:00 am   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

When I hear developers complain about long build times, it immediately makes me suspicious about the quality of their code base. They have simply built a big ball of mud, a.k.a. spaghetti code. What has happened is that the compiler gets bogged down by an avalanche of include files. Don't fight the symptom by using precompiled headers! Instead, fix the root cause: the crappy architecture of your game. What follows in this article, is a recipe for a clean code base that builds in a couple of seconds, not minutes.

Here's the TL;DR version of this article: do not include header files in header files. The dependencies that foo.cpp uses to implement its functionality should never leak into the foo.h interface. I'll state up front that a C-like programming style lends itself better for this than a C++ programming style. But even when using C++ classes, it is often possible to avoid include directives.

If the interface of class Foo only contains uses of class Bar that are references or pointers, then the compiler does not need to know what Bar looks like when parsing the header for Foo. All that is required is that Bar is forward declared for the Foo interface. You tell the compiler that there is a class named Bar, and leave it at that. Typically, there will be some classes in your game that need to include large third party frameworks. It is paramount that these frameworks do not leak into the users of class Bar, like class Foo.

If you constantly ask yourself: "How can I keep the implementation details out of the interface?" then you can cut down on the include directives. Also preferring primitive types char/int/float/bool and pointers over custom object types is a great strategy for pruning include hierarchies.

So where can this get you? To illustrate this, I will examine the code base of the game I am currently creating. The only uses of middleware are OpenAL and OpenDE. It is a fully featured game with biplanes, AI, formation flying, dogfighting, explosions, menus, tutorials, whathaveyou, totaling roughly 50K lines of code. But building from scratch (including linking) takes less than three seconds.

real    0m2.687s
user    0m5.076s
sys     0m0.542s

This is because the include hierarchies are extremely flat. For instance, here is what gets pulled in to implement the logical representation of my game world.

And here is what the code that uses the game world sees when it includes wld.h:

In my code, dependencies are hidden, and almost never propagated throughout the system. The notable exceptions are the leaking of my vector math header and my rendering context data into the interface. The latter includes no further files, and the former only three system headers.

The world contains terrain, planes, bullets, hangars, explosions and much more. But the code that uses (creates, destroys, renders) this world should not be bothered by these details.

Personally I think that distributed build systems, build caches, precompiled headers are all curses disguised as blessings. These mechanisms that make your build go faster, allow you to ignore the root problem for longer. The mud ball grows and grows, and when you are finally fed up with long build times, the tangled mess may be beyond repair. Top tip: disable precompiled headers!

Let me conclude with a tip on how to inspect the sanitation conditions in your code. Snowballing include directives are brutally exposed with the 'Doxygen' tool if you tell it to generate include graphs with graphviz's dot program. Use the following directives in your Doxygen configuration file:

DIRECTORY_GRAPH = YES
INCLUDE_GRAPH = YES
HAVE_DOT = YES

If the Doxygen output includes huge dependency graphs, you know that you are on the wrong path. Also, cycles in the graph with lower level files pointing back to higher level files are a huge red flag.

I'm curious to know how my peers view this issue. Also, what is the longest build time that you have encountered? Drop your remarks in the comments, they are highly appreciated!


Related Jobs

Disney Consumer Products
Disney Consumer Products — Glendale, California, United States
[07.30.14]

Contract Game Programmer
Zindagi Games
Zindagi Games — Camarillo, California, United States
[07.30.14]

Software Engineer
Telltale Games
Telltale Games — San Rafael, California, United States
[07.30.14]

Core Technology – Client Network Engineer
Gearbox Software
Gearbox Software — Plano, Texas, United States
[07.30.14]

Release Engineer






Comments


Trent Reed
profile image
The big issue I've run into with Precompiled headers is people using them wrong. I think there is a fundamental misunderstanding that Precompiled Headers are just "magic settings with files that make things compile faster." Many people use PCH without knowing exactly what it does or why it's used. And furthermore, they enforce me to use their PCH by hardcoding the #include at the top of the file. DON'T DO THIS!

You should be generating your solution anyways, so just make a step of the solution generation force include of the PCH on all compiled files if you want PCH on (MSVC, Clang, and gcc all have this ability!) However, well-formed code should be able to compile without PCH or the PCH .h include and compile fine.

tuan kuranes
profile image
On that exact same spot, Another nice tool is https://code.google.com/p/include-what-you-use/

"The main goal of include-what-you-use is to remove superfluous #includes. It does this both by figuring out what #includes are not actually needed for this file (for both .cc and .h files), and replacing #includes with forward-declares when possible."

Javier Degirolmo
profile image
This seems to be mostly an issue for C++ (where headers contain functional information), since in C it's pretty much pure declarations what you find in headers.

One thing I noticed in C once while messing with some broken toolchain setup (that required me to rebuild the entire project even if a single file changed, since it couldn't detect which files had changed) is that spawning the compiler process seems to be the biggest bottleneck. It had gotten to the point that building the entire thing took like about a minute. Then I decide to merge everything into a single file with #includes (I know, not the best idea, but the toolchain was broken =P). It went down to about five seconds. Huh.

Then again a minute is nothing compared to some stuff I hear. I've read about one project taking up 45 minutes to build... With build times like that it's no wonder that programmers end up commiting code that they didn't test if it even works (resulting in broken builds more often than not).

sean lindskog
profile image
Good advice.
I try hard to minimize the number of headers included within another header.

One place where things get complicated is when you need to include one class as part of another class.
Not this:
Class Foo
{
Bar* bar;
};

but this:
Class Foo
{
Bar bar;
};

You can get away with forward declaring a a pointer, but not a full class data member.

So why not always just use pointers for class data members?

For me, it's primarily an efficiency issue. If you're iterating over a large number of say game entities every frame, it's much more cache-friendly if all those entities are aligned in a big array. If you use an array of pointers, the extra indirection has you jumping all over the place, wreaking havoc with the cache.

Dan Lynch
profile image
The way I've gotten around multiple indirections in the manner you describe is to declare an inner struct intended to contain the array (but don't define it in the header), add a member pointer to that inner struct, then in the implementation define the struct in a cache coherent manner, e.g.

foo.h

class foo
{
struct bar;

bar* m_barData;
}

foo.cpp

struct foo::bar
{
entity entities[256];
}

This way, clients of the public interface to foo do not need to know about the details of bar (i.e. entity), encapsulation and cache coherency are preserved.

Matt Heinzen
profile image
Not only is placing class instances directly in the container class more cache friendly, it also results in less boilerplate code for allocation, construction, and destruction, it also is less bug prone because there are no explicit allocation, construction, or destruction to worry about.

Out code uses this style, and I spend very little time tracking down memory leaks because of it. Fortunately, our codebases are never huge, so build times are not a big worry. Otherwise, it becomes a matter of prioritizing code cleanliness, build times, and cache performance based on the needs of the project.

Robert Flesch
profile image
For complex projects, you really have to try to take this approach early on, otherwise the Gordian knot gets out of control.
One thing I have notice is when we started using STL templates extensively, our compile time when WAY up.
Be nice to have a tool that identified slowness in the compile tool chain.

John Maurer
profile image
Excellent tidbit!

Jimmy Alamparambil
profile image
Just Say No to headers : Use C# or Java or umpteen other language choices out there

Christopher Myburgh
profile image
End-user apps in C/C++? You're doing it wrong.

Andrew Haining
profile image
80% of your C# application is compiled unmanaged C++, this article is for the people who write those parts, not gameplay programmers. You can't run performance sensitive code in a managed language because on the spectrum of optimisations vs readability you sometimes need to move towards unreadable optimised code that a managed language does not allow to hit your performance requirements.

Thanks for your valuable contribution. Please consider the possibility in future that people with a deep understanding of a computer system might've chose a technology for a good reason. Implying all jobs should be solved with a screwdriver because you are a screwdriver expert will not help you become a better engineer. At learn what a hammer is for even if you refuse to use one.

Christopher Myburgh
profile image
"this article is for the people who write those parts, not gameplay programmers."

streamers.h, cones.h, bullets.h, smoketrails.h, planes.h, hangars.h, airports.h etc. all sound like gameplay elements to me. Please read the article before commenting.

Jason El-Massih
profile image
The article is clearly written for people who are using a C++ code base. It provides some useful ideas that hopefully makes the reader re-evaluate their own code base.

As Andrew said, people are *usually* not stupid, and have good reasons for choosing the technology they use. Everybody who has ever written C++ has heard the obligatory "C++ is dum, your doing it wrong" from a Java/C#/etc.. developer. Therefore, telling people "They are doing it wrong" because you are a C# programmer accomplishes nothing, wastes everybody's time, and reflects poorly on your own understanding of Computer Systems.

All I ask is that you respect the technology decisions of others, and dont blindly mock them for using something you don't understand/disagree with.

Russell Sullivan
profile image
Please don't put down other languages because you don't like using it. There is nothing wrong with C++ and many incredible games are still made with it.

Christopher Myburgh
profile image
Yes, many incredible games are still made with C++, but only because game developers on average are still stuck in the 20th century regarding software development tools and technologies.

Those same games could still be incredible but made with a fraction of the resources using a modern programming language. For starters, they wouldn't have to deal with header files and their associated bullshit, the likes of which Bram Stolk kindly attempts to address in this article.

Alanna Kelly
profile image
Sound advice there.

For all the people flying the C# or Java flag. Java also has problems with ridiculous compile times on badly set up projects. The Java project (~200,000 LOC) my employer has me working on right now takes a ridiculous 18 mins to build. I don't know enough about large C# projects to comment, but I'm guessing if they are stupidly set up then it is the same problem.

Right tool for the job as always and don't be scared of getting closer to the metal if you need to. Learning how the hardware actually works will make you a better software engineer even in Java or C#.

ignace saenen
profile image
Totally, predeclare as much as possible. That said, this advice does tend to push you away from:


- inlined data membership (has a). If this is a problem for your caching, take a long had look at how you are iterating your data, and wether the semantic splits make any sense at all.

- inheritance. Useful sometimes, but of course you can mimic this with types and pre tables all by yourself too. I tend to prefer the language construct: single inheritance, with shallow depths. Use multiple inheritance only when everyone else is already dead and the last chopper just left.

- templates. If you must... Just one level is my credo, and sometimes an allocator and typetrait can be of help. your mileage may vary.

- typewrappers, debug tools, metadata hacks, etc.

Timely builds are important, and the flat header tree is a great demonstration on how to achieve that for large code bases. The context is also important (mockup/prototype, vertical slice,middleware,..),

If you are in really big projects, a bit of sane project management can also get your link times down. .

Ian Hatch
profile image
Longest build times I've had was 20 minutes for an incremental build.
Shortest for a project of similar size was 2. The difference was partly the problem that you describe here, and partly the same thing for DLL interfaces - there were a lot of projects in the solution that included and called classes from other libraries directly, producing dependency hell and horrible spaghetti linkage.

Kris G
profile image
STL will kill your C++ compile times because it needs to recompile a copy of the STL headers for each .o it generates, and then the link stage becomes huge and bloated because the linker must remove dozens (hundreds?) of redundant instances of the STL code. This is where PCH comes in handy. A different strategy I've used in the past is to compile multiple .cpp files together at once in one auto-generated 'composite' .cpp file. And yet another strategy is to use distributed build tech like Incredibuild or SN-DBS.


none
 
Comment: