[Black Rock (Split/Second, Pure) core technology group member Julian Adams explores the practical ins and outs of code branches -- including how his team has implemented them, productivity gains, and compromises inherent in the approach.]

Motivation

These days, everyone uses a Version Control System (VCS). Like many game developers, the first VCS I used was Visual Source Safe. Back in the 1990s, it was incredible. All five of us in the team shared our work, merged our work, and tagged our releases. Basically, we all knew what was going on, and we all sat together. There were rarely problems, and all was generally good.

Even as teams grew to be about 20 people everything was fine... mostly. Now, though, if something goes wrong, and the build breaks, you might not know the code; you might not even know whose code it is. You can still collaborate pretty freely with anyone on the team -- but sometimes you need to share half-finished code with the guys working directly with you on a feature, and it's a pain. You try and make sure that the interim commits won't break anyone's work, but everyone makes mistakes.

These days, game development has changed. Split/Second had 40 programmers at its peak. Pure had 25 with another 15 developing core technology -- all working on the same code base. There were also 65 people working on game assets in a separate per-project Perforce depot.

Let's define a "broken" build as one that stops someone from working. With the smartest 55 people in the world pushing into the mainline, the build will always be broken for some of them.

As a development team scales up, one way to mitigate build breakages is to start adding automated tests to test whether the build is good, and using continuous integration to run those tests as code is checked in, giving continuous feedback as to the state of the build.

If the tests are sufficiently fast they can be run by coders before checking in any code. If the tests had full coverage of everything that anyone was interested in, and took zero time, that would be it: you'd never check in broken code, as you'd run the tests against the latest, and the latest would always be good!

Tests never have 100 percent coverage, of course, so right there is the chance that code which passes the tests is broken for someone.

On the other hand, test run time (latency) means the test results you see for a branch are out of date. It makes it likely you won't run all the tests yourself pre-commit, or that you ran all the tests against the code you updated to pre-merge, but not against the latest code, post-merge.

The combination of non-zero test time and incomplete coverage combine to mean that the build will get broken at times. As more people check into a single branch, the chance the branch will be broken at any given time increases greatly.

With a large code base the chance that you'll know how to fix a build breakage, or even who to ask, decreases. Branching addresses these issues.

Terminology

One issue with talking about VCSes is that the terminology isn't totally standard. Most of what I'm going to talk about will be pretty obvious to anyone who's used a VCS. I've used terms which are mostly standard and defined in the Wikipedia entry on revision control.

The terminology for branching is less clear. I'm going to talk about "committing", which refers to storing local changes in the repository, without making them visible to other people, and about "pushing", which refers to publishing changes from one branch to another.

What's a Branch?

A branch is a set of files in a VCS which are changed compared to the mainline, and evolve independently. Unlike a working directory (for which this is also true) many people can work from and push to a branch.

Figure 1. Branches diverging from and merging with the mainline of development.

What Does a Branch Get You?

Branches let you go back to developing in smaller teams. You work directly with five to 10 people. Now you only see changes relevant to what you're working on, related test breakages, and have a small team to keep track of. You've got the opportunity to share code with just the people you're working directly with, and have a staging ground to develop that cool new feature. When it's stable you merge against the mainline, and push your work to the rest of the team.

What's the Advantage for the Mainline?

Conversely, when teams stabilize their branches and then push up to the mainline, the mainline becomes more stable. Again, this is simply a question of numbers. Instead of 70 people pushing into the mainline there are maybe seven teams pushing code in there. Immediately there's a lower frequency to the changes going into the mainline. Also this is code which has already been tested and stabilized on a branch, so it's less likely to cause problems.

It's Not the Branching, It's the Merging

So far I've talked a lot about branching, but branching has always been easy. Visual Source Safe could branch, CVS could branch. On the other hand, merging was incredibly hard, and nothing like the normal workflow, as discussed here.

These days, many VCSes make branching and merging as easy as local changes. Look at it like this: every local change you make is effectively a branch. Merging your working directory is merging a branch, committing the changes is reconciling your local working directory (branch) with the mainline. Branching and merging should be this easy, and in a lot of VCSes these days they are.

Distributed VCS (DVCS) has become the big trend in VCSes over the last couple of years. You've probably heard of Git and Mercurial. In short, the idea of these systems is that every branch is a clone of the full depot, so that all revision control functionality, except updating from non-local branches, is available without a network connection. In a situation like that a DVCS had better be good at merging or it won't get many users.

There's an excellent online tutorial for Mercurial. If you've not used a DVCS system, it's well worth checking this out. It's command line-driven, but the simplicity is there, and there are GUIs available if that's what you want. If your VCS doesn't handle branching and merging this well, maybe it's time look at the options, or talk to the vendor!

At this stage you're probably thinking that for game assets, having a copy of the repository with every branch isn't ideal. That's true! I'm not suggesting you should jump on any of the systems and start using them, but they're freely available and show the level of branch and merge support that is the state of the art. Although the examples I've given are distributed, a centralized system can offer equally good branch support. We use Accurev.