Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
Branching For The Win: How Black Rock Uses Code Branches
View All     RSS
May 24, 2019
arrowPress Releases
May 24, 2019
Games Press
View All     RSS








If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

Branching For The Win: How Black Rock Uses Code Branches


July 20, 2010 Article Start Previous Page 3 of 3
 

There Must Be a Catch

There are a few trade-offs. It's important to think about these and consider the options in advance if possible. The first is having to merge the mainline with your branch. Without a branch, each developer merges his or her own changes with the mainline. When there are conflicts each developer knows what the intent of their changes was. If the social side of the studio culture is healthy, they most likely know why there are conflicts also.

A branch needs merging with the mainline in addition to normal merges. Somebody has to do this and that somebody might not be the person who made either of the conflicting changes. At any given time one person is responsible for merging a branch. In general we change this weekly, in a round-robin way. For merges, we encourage people to grab the authors of the original changes if any merge isn't totally obvious. It's not perfect, and there's some overhead, but it works well enough in practice.

The key is responsibility: somebody needs to be responsible for each branch, for testing it, smoothing out any problems, and pushing changes to the parent. Without this, a branch stops working as a means to propagate changes. Although we almost always share ownership of our branches it's equally valid for one person to have responsibility.

In general having fewer, larger changes entering the mainline makes it easier to push into a branch. However there's one point in time when this isn't true: at a milestone. Here everyone's changes need to be in the milestone build. Time is typically short. If everyone is waiting for slow tests to pass in the merge branch, it can take much longer than desired to push up changes.

An alternate strategy is to push up changes from lots of branches almost simultaneously. These changes are all individually tested in their branches. The merges between them are not fully tested before they are pushed into the mainline. It's quicker to get changes to the mainline, although the changes are more likely to break the game initially. In this case that's not a problem: the time taken to merge the milestone's features is reduced, and breaking the delivery build in the short term does not hold up the development team working on the branches.

Binary files represent a significant issue for any workflows which rely on parallel edits and textual merges between branches. These cannot be merged, so in case of conflict you have to pick one change or the other. Traditionally, when developing on a single branch, these files are set up so you have to get exclusive lock on the file before editing it. No merges are required.

With multiple branches can you have a lock which applies to multiple branches? Yes, but what happens if a file is edited in one branch and you want an exclusive lock in another? What version would you edit?

Editing the version in your branch is likely what you want. This gets you a conflict though, even though you have an exclusive lock. Bringing the changes across from the other branch without the rest of the files from that branch may not be correct, as typically changes on any given branch are interdependent. I don't know of a VCS which implements this solution. The commonly implemented alternative is for locking to be exclusive to a single branch only. In case of conflict during the merging of two branches, one version of the file is discarded.

For test files we simply live with this. These files support the tests we run and lack of merging isn't a pain point. We store the assets for the games in separate Perforce depots. This setup has evolved over time. We don't branch these assets and use exclusive locking.

Initially branching the code in Accurev, and having a single branch for assets in Perforce, made it hard for the artists to freely collaborate with each other, and with the coders. Development assets in Perforce simply weren't adequately compatible with the code in all branches of Accurev. The build was often broken.

To coordinate revisions of assets in Perforce to the revision in Accurev we now map each top-level directory in Perforce to a single branch in Accurev. In Accurev we track known-good revisions of the Perforce directories in text files. Since these are text files, they can be easily merged as they are pushed between branches.


Figure 7. Relating Directories of Assets to Branches of Code.

When you want new assets from Perforce, our tools get the latest version of assets for the Accurev branch you're currently in, as well as the known-good versions of everything else. When code is pushed from a branch in Accurev we use a script to automatically update the known-good Perforce revision based on the tests. This setup enables us to work with multiple branches in Accurev and to collaborate freely with artists. It's dramatically improved the workflows across the teams.

A weakness of this system is that assets must be owned by a single branch and not be edited from multiple branches. Since this system evolved after we set up the asset structure, we had cases where multiple teams wanted to edit a file. For example, both Gameplay and Vehicle teams wanted to alter portions of the car setup. In the end we reorganized the data to remove this contention.

In The Future

Having a VCS system which enables flexible, comfortable branching and merging is a big win for Black Rock Studio. There are issues which we'd like to address going forward which will improve our workflows further. I'd like to investigate having everything in one VCS. Tracking revisions in two VCSes complicates matters. At Black Rock this is an arrangement which has evolved, perhaps we can consolidate to a single system.

Our games are reasonably tolerant of missing or out of date assets. Incompatible data and a lack of fault-tolerant systems in the games still break builds due to the propagation of the required code through branches. We're going to make absolutely everything the game loads have forward and backward compatibility, with all systems handle loading failures gracefully. This will significantly improve the experience for everybody working on the games.

Although branching increased stability of the mainline it also increased the time taken for changes to reach the mainline. It was overly hard to get a build from the mainline with all the latest features; at times the latency could be one to two weeks. We'd like to make it more routine and easy to push features to the mainline as soon as they are complete and tested to bring the latency down.

We are increasing the flexibility of our build and test systems to make these more easily handle ad-hoc branching and most efficiently use the build machines. This will reduce further still the current barriers to ad-hoc branches. I'm sure that in a couple of years' time, when we finish another game, there will be a lot more to tell.

Acknowledgements

Thanks to Tim Swan, Jez Moore, Jim Callin, Neil Hutchinson, Rory Driscoll, Tom Williams, Steve Uphill and Shawn Hargreaves who all gave valuable comments and feedback on this article.

References

[1] Branching and Merging in CVS: http://kb.wisc.edu/middleware/page.php?id=4087
[2] Mercurial Tutorial: http://hginit.com/01.html
[3] Continuous Integration: http://martinfowler.com/articles/continuousIntegration.html


Article Start Previous Page 3 of 3

Related Jobs

Petroglyph Games
Petroglyph Games — Las Vegas, Nevada, United States
[05.23.19]

Senior Game Engineer (C++)
Sanzaru Games Inc.
Sanzaru Games Inc. — Foster City , California, United States
[05.23.19]

Junior Gameplay Engineer
Brace Yourself Games
Brace Yourself Games — Vancouver, British Columbia, Canada
[05.23.19]

VFX ARTIST
Disbelief
Disbelief — Cambridge, Massachusetts, United States
[05.23.19]

Senior Programmer, Cambridge, MA





Loading Comments

loader image