5. Testing the Networking Code. Even though we nearly shipped a game with a decent networking code a year ago and had an understanding of what it takes to get it done -- we still underestimated the complexity of the task. We set a high quality bar, which was to create a robust multiplayer engine rivaling those in the best products -- and we did not want to settle on anything less than that.

Our multiplayer mechanics were very strong and fun to play, so we had to support them with proper tech to allow 16 players to play online. In order to reach the mass-market we established pretty low bandwidth requirements, and to achieve this goal we had to do a pretty major re-architecturing of the networking engine.

Once the commitment was made and the work started -- there was no going back. Our old networking code was no longer compatible with the rest of the engine, but the new system was insanely complex and needed to be properly debugged.

Normally, you want to stay away from "insanely complex" systems because they are hard to develop, debug and maintain. But it looks like it's the nature of the business when it comes to multiplayer. All the issues we had in single player were now multiplied by a factor of 16 (the maximum number of players TimeShift supports).

You want to test stability? Performance? Bandwidth handling? Great! All you need to do is to find 16 people that are available at request, ideally sitting in different offices (St. Petersburg, NJ and LA), make sure they all have the same build, and do the test. Then repeat as needed.

The amount of testing we had to go through was absolutely insane. We came up with a concept of mini-builds, which allowed us to distribute builds faster. We ran dozens of tests on a daily basis assembling bandwidth handling stats, trying to provide the multiplayer team with enough information to do the analysis.

A dedicated multiplayer QA team in LA was running daily tests with their counterparts in India, running multiple 16-player matches with different settings on various bandwidths. But a week before release we were still unhappy with the results -- even though our lead multiplayer engineer, Stas, was leaving the office at 5 AM nearly every day of the week. It still remains a mystery how he pulled it off -- but he did.

All the crashes were addressed, bandwidth handling streamlined, and we shipped with very solid code which allowed people from various parts of the world to play on a server with minimal lag and use the voice over IP feature.

Even though we finally succeeded, this entire experience was scary, and worked totally against our normal planning. We are always trying to implement "big systems" first to minimize the risks and bring the game to a shippable quality long before the finishing time, which needs to be reserved for minor tweaking and polish.

That was certainly not the case with multiplayer. A lesson we learned was that we underestimated the massive scope of testing required to bring the multiplayer component to the quality level we wanted. Also, multiplayer coming together so late in the process did not allow us to have any beta testing of this component of the game.

Surprisingly, though, based on the user feedback, multiplayer implementation is very solid in TimeShift. In fact many consider it to be better than the more established shooters that released this season.

Conclusion

Working exceptionally long hours, dealing with unexpected technical hurdles, managing multiple SKUs, and adapting to ever-changing business requirements, working on TimeShift was a very trying experience for everyone involved, both at Saber and at Sierra.

We could have never anticipated what it would take to put together a AAA product under such tight deadlines, but we were determined to make it to the finish line and to create a great game. We emerged from this experience as a much more mature team ready to take on even bigger challenges.

TimeShift is Saber's first console game, our first AAA title and probably the highest profile game ever to come out of the former Soviet Union this side of Tetris. We are very proud of what we accomplished and look forward to a normal development cycle where we can really show off our talents as a team

Game Data

Developer: Saber Interactive

Publisher: Sierra

Platforms: PC, Xbox 360, PlayStation 3

Release date: Fall 2007

Development Time: 4 years

Number of full-time developers at peak: 92

Number of contractors at peak: over 50

Hardware: over 120 hi-end PCs with latest graphics hardware, including various "farm racks"; about 30 Xbox 360 dev and test kits; about 25 PS3 dev and test kits

Software: in addition to standard dev tools, we used SN DBS (making distributed builds on PS3), gcc 4.1.1 (PS3 compiler), SN Tuner (PS3 performance analysis), ProDG, NvPerfHUD, NvShaderPerf, ATIGpuPerfStudio (graphics performance analysis on PC), VTune Performance Analyzer and Thread Profiler, PIX & XbPerf (performance analysis on Xbox360)

Middleware technology: GameSpy, Fmod, Bink, Havok