Many of the agile practices discussed help distributed teams overcome these challenges. They give teams the opportunity to maintain a shared vision, increase collaboration, and avoid going down separate paths.
Scrum Teams Align with Location. Usually each distributed team is a separate Scrum team. There are occasions when it's beneficial to have members of a Scrum team distributed to share knowledge and create bonds between locations (Cohn 2009), but for the most part having each Scrum team colocated is best.
When each Scrum team is colocated, they can more effectively collaborate on a shared sprint goal. Such teams need a local product owner but should find ways to hold sprint review and planning meetings with other teams and the lead product owner either in person or through a video-conferencing system.
Shared Scrum of Scrums. A video- or phone-conferenced Scrum of Scrums meeting is essential for distributed teams. These don't have to occur every day, but they should be held at least once a week.
If teams are spread across many time zones, the time of the conference call should not be fixed so as to impose a constant burden on one team more than the others. The meeting time is changed on a regular basis so that attendees from different locations have to come in early or stay late.
Shared Release Planning. It's critical for a release plan to be developed and shared to the greatest degree possible among the teams. Often this will include flying one or more members from each location to a central place where the meeting is held. It's often a necessary cost that cannot be avoided with distributed teams. You either spend money on airfares and hotels or spend much more on the costs incurred by divergent goals.
There are many ways to organize depending on the how the teams are distributed. The sidebar "A Global Game Development Story" tells the story of how one company organizes their releases across three continents and sixteen time zones.
Improved Sharing of Builds, Assets, and Code. A problem with large projects is how to share the large number of changes that occur. Frequent small changes can perpetually break the build, and bulk changes committed weeks apart can bring teams to a halt for days at a time.
With colocated teams, this problem is bad enough. With distributed teams, defects passed across in shared builds, assets, and code can be disastrous. When a single question can take a day for an answer, tracking down a problem and the necessary expertise to solve it consumes days rather than hours or minutes. Extra care must be taken to protect distributed teams from external defects. This requires a focus on improved commit practices and testing described in Chapter 9, "Faster Iterations."
Processes and Tools. Agile methods value "individuals and interactions over processes and tools." A wider team distribution can impede interactions and place more weight on processes and tools. Distributed teams often use more tools to track and display sprint progress and product backlogs and share knowledge. Wikis and other documentation tools are beneficial and are used more with colocated teams.
CCP games, the developer and publisher of EVE Online, was founded in 1997 with development split between studios in Reykjavik, Iceland; Atlanta, United States; and Shanghai, China. It has grown to have more than 400 employees and to host more than 300,000 active subscribers in a single online world.
In the fall of 2008, CCP undertook the development of its tenth expansion pack called Apocrypha. Apocrypha was the most ambitious expansion of the EVE universe. It added major technical features and significantly extended the size of the EVE world. The goal of the company was to release the expansion pack within six months following a four-month development cycle (see Figure 8.7).
Figure 8.7: The Apocrypha timeline
This ambitious goal required CCP's worldwide development studios to work in parallel. Features and content developed simultaneously across three continents had to seamlessly come together to achieve their goal. Normally this would be the introduction to a disaster story, but CCP pulled it off. CCP is a longtime adopter of agile methods, specifically Scrum.
In the case of Apocrypha, with more than 120 developers in 13 Scrum teams spread across three continents, 9 product owners were required. These product owners took their direction for the game from a project management group in Iceland.
The stakeholders identified two release cycles of development to release the expansion pack in March 2009. Ideally, release planning should gather every developer, but it was nearly impossible to collect 120 people from around the world in one location, so CCP had to create some innovative practices to perform release planning.
Apocrypha release planning meetings took place over a single 12-hour period. A high-definition video conferencing network facilitated this meeting and allowed every developer to take part.
Since most of the development staff, including the project management group, was in Reykjavík, the meeting was focused there. It started at 9 a.m. Because of the time difference, it was far too early for the developers in Atlanta to attend. It was the end of the day for the developers in Shanghai, so they and the Reykjavík group met first. Hours later the Shanghai team would leave, and the Atlanta team would join to discuss the release (see Figure 8.8).
Figure 8.8: Overlapping meeting times
You can see a time-lapse video of this amazing meeting in Reykjavik on YouTube.
It's important to note that the time zone limitations and the limitations of video conferencing allowed only two teams to meet at one time. In this case, Atlanta and Shanghai could not attend together. To avoid potential problems, two developers from Shanghai flew to Reykjavík to participate at the core meeting and to represent the Shanghai teams.
Each pair of locations would discuss the release goals and begin breaking them down into smaller goals that were reasonably sized to fit into a sprint. The conferencing network enabled multiple simultaneous meetings to occur. This was necessary since many questions raised during this process required a great deal of live conversation among the entire staff or individual teams.
The goal of the release planning meeting was to generate a set of potential sprint goals for each of the 13 teams that would fulfill the release BHAGS. This wouldn't be the last time that the teams would communicate. Daily issues were communicated, and a build of the game was demonstrated every sprint, which were two weeks long. Changes to the release plan were made every sprint to adjust for the realities of development.
After two releases, Apocrypha was ready to do final testing and polishing, and on March 10, 2009, it shipped on schedule.
A globally distributed development team can be successful in creating a high-quality product within budget and within the schedule limits. The raw ingredients for this success can be summarized in several points:
Local vision and ownership: Scrum enables individual cross-disciplined teams to take ownership of their sprint goal and achieve it independently. Having a product owner on-site who is responsible for the vision of what the team is creating is essential.
Iterative development methodology: Creating an integrated, potentially shippable version of the game every two weeks from work done around the world forces problems to the surface and demonstrates real progress. Without this, critical problems can remain submerged until late in the project, when it is too late to avoid delays and cost overruns.
High-bandwidth communication: Communication on large colocated teams is difficult enough. On large distributed teams, it can be the main challenge. Teams must have the tools to communicate effectively, such as a networked conferencing system, that are reliable and ubiquitous, as well as a methodology that creates transparency, such as Scrum.
Distributed teams will never be as effective as colocated teams, however. What is lost from daily face-to-face communication cannot be made up through conference calls. But it can come close enough to ensure that the teams are productive.
There is no single formula for creating the best teams on a game project. The teams and practices described here result from adapting the existing Scrum roles and team to improve how people work together and how a vision is unified and shared. Exploring roles and team structures is an ongoing process.
Teams must use the retrospective practice to find ways to improve how they work together and with the stakeholders of the game. By allowing teams to take ownership and authority over some of the daily aspects of their life, they are more likely to take responsibility for their work. This doesn't happen overnight. It takes years in many cases and will create head-on collisions with studio leadership culture. The results are worth the effort.
Scrum scales up to projects of any size but drives changes in team structure, the project owner role, and practices such as the Scrum of Scrums.
Scrum also drives changes across disciplines, focusing the team on changing how they work day to day to improve communication and the pace of iteration. It also drives changes in how each discipline works. The remainder of this part of the book will address these changes.
The goal is to create teams that can take more ownership of development and have greater purpose in what they do.
DeMarco, T., and T. Lister. 1999. Peopleware: Productive Projects and Teams, Second Edition. New York: Dorset House Publishing.
Katzenbach, J. R., and D. K. Smith. 2003. The Wisdom of Teams: Creating the High-Performance Organization. Cambridge, MA: Harvard Business School Press.
Larman, C., and B. Vodde. 2009. Scaling Lean and Agile Development: Thinking and Organizational Tools for Large-Scale Scrum. Boston: Addison-Wesley.
Leffingwell, D. 2007. Scaling Software Agility: Best Practices for Large Enterprises. -Boston: Addison-Wesley.
Schwaber, K. 2007. The Enterprise and Scrum. Redmond, WA: Microsoft Press.
This chapter is an excerpt from the book, 'Agile Game Development with Scrum' authored by Clinton Keith, published by Pearson/Addison-Wesley Professional, May 2010, ISBN 0321618521, Copyright 2010 Clinton Keith; to see a full Table of Contents please visit this link.