3. Difficulty creating stable work builds.

During RCF, our greatest periods of progress occurred when all of the pieces clicked, and the project seemed to take on a life of its own. The end result would be a stable working build of our game that could be play tested and further refined.

Unfortunately, stability remained elusive throughout development and often required Herculean efforts from our tools, tech, and gameplay programming teams to achieve. There were times when we could not produce a clean build of our game for up to three weeks.

Because of our compressed development cycle, we needed to continue creating assets and wiring the game but we had to do it blindly as our stabilization efforts took place.

Once we fixed the problem(s), the resulting change lists submitted to the project were so massive that they invariably broke the game, which again required us to re-stabilize. This frustrating downtime hampered our ability to tune the game iteratively. This represents an ongoing struggle at Insomniac.

4. Developing two games simultaneously.

In addition to working on new hardware, it was a new challenge to have two games in simultaneous development, and we felt our share of growing pains. Previously, we focused the collective effort of our studio on one title at a time. Any important issue could receive the attention of our entire company if necessary.

But now, we had two games to think of, and needed to work with people who were considered shared resources and not always available.

What proved more difficult than sharing people was hiring people. When RCF went into production we were understaffed, especially on our programming team. Yet, we did not adjust our plans and remained hopeful that we'd find the right person soon.

We did not complete our hiring until three months before our release, and this meant that our understaffed departments had to take on extra work, which is not a sustainable strategy.

5. Expecting the unexpected.

RCF also experienced its share of curveballs. We were given the opportunity to show our game as part of the TV show Extreme Makeover: Home Edition, provided we create a unique character to tie in with the episode (see Figure 5). This resulted in some good publicity and one happy boy in Oklahoma, but it cost us a week.

Figure 5: Captain James, who appeared in RCF during an episode of Extreme Makeover: Home Edition and lives on as an unlockable secret in the game.

We were running builds on PS3 devkits but Sony Q/A only worked on test kits, a miscommunication that forced us to scramble to fit our builds on disc earlier than planned.

We prepared a downloadable demo of our game to coincide with E3 but it was an in-progress demo that only ran at 30fps. So we scrapped it and made the demo again as we were closer to finishing the game.

Future Tools

I doubt any of these production problems are unique, and many of them have been solved by other studios. Our biggest challenge was to deal with them in the context of creating a game, which certainly raised the urgency and stressfulness of the situation.

A lot of things did not work out for us, but a lot more of them did. We were able to stabilize many of our development techniques but know there will always be more ways to improve. And as we continue our mission to make great games, we've moved a step closer to a smoother production model.

Game Data

Developer: Insomniac Games
Publisher: Sony Computer Entertainment
Release Date: October 23rd, 2007
Platform: PlayStation 3

Number of Developers: 70 full-time, 30 shared, 25 contractors (music, voice acting, testing, localization).

Length of Development: 23 months: 11 months preproduction, 12 months production.

Hardware Used: Artist workstation: Dell Xeon 3.2G with Quadro FX3400. Programmer workstation: Quadcore PC with as much memory crammed into it as possible.

Software Used: Proprietary engine and development tools. Artist software: Maya, Photoshop, ZBrush, Turtle. Asset management and revision control: Perforce. Design/gameplay scripting: Lua. Compiler: custom GCC compiler provided by Sony. Character lip synching: Annisoft.

Number of Times Art Director Spoke Ill of Dragons: Zero -- for he is crunchy and good with ketchup.

Technologies Licensed: SpeedTree, Bink, Microsoft Visual Studio, Anark (UI menus), DevTrack (bug database).

Project Size

Game size: 22.5gb
Project size: 1000gb
Bugs: 16,000
Source Files: 4429
Lines of code: 980,184