Manager In A Strange Land: Benchmarking, Part 1
By Jamie Fristrom
Gamasutra
November 7, 2003

URL: http://www.gamasutra.com/features/20031107/fristrom_01.shtml

I mentioned in my last column that learning from experience sucks. Most of learning from experience is learning from mistakes; learning from mistakes means you have to make the mistake at least once. It would be nice if we didn't have to make the mistake at all. But there's another problem with learning from mistakes: there are way too many mistakes you can make when developing games. I used to work at a game company where, every several months, the president would give us a speech that went something like this: "Well, we haven't been doing too good lately. We made a lot of mistakes, but I think we learned a lot from them, and we won't make those same mistakes again." No, we made all new mistakes.

What would be really nice is if we could prevent some of these mistakes from happening in the first place.

That's where benchmarking comes in. Benchmarking is looking at exemplary companies and adopting their best practices.

Sounds good so far. Here's where this article gets sketchy. I'm about to break every rule of good statistical analysis and still try to draw some conclusions. I've gone through almost every postmortem in Gamasutra, and listed the ones that got higher than 70 (Why 70? Because it's the mean.) on gamerankings.com, along with how long they took to develop, how many people were on the team, and their gamerankings score.

Some of the obvious problems with this approach are: score on gamerankings is not a valid measure of game quality. (But it's a better measure than sales.) Also, the numbers different people report for their team sizes use different methods: some report averages, some report total number of people who ever worked on the project, some report peaks. Also, the sample is not representative.

The problem is, we're starved for data, so I'm willing to take what I can get, in the hopes that some information that might be right is better than no information at all. So, like with all the articles I write, use it to stimulate your thinking, but don't take it as gospel.

Studio	Title	Months	Team	Buck	Bang
Epic	Unreal Tournament	18	16	288	94
Factor 5	Rogue Squadron II	9	32	288	89
Ritual	Heavy Metal: FAKK 2	18	18	324	80
Ritual	Sin	20	17	340	71
Raven	Heretic II	11	32	352	85
Sierra Studios	SWAT 3: Close Quarters Battle Spec	18	20	360	84
Zombie	SpecOps: Rangers Lead The Way	20	18	360	77
Activision	Heavy Gear 2	19	20	380	82
Raven	Soldier of Fortune	23	20	460	84
Red Storm	Rainbow Six	21	22	462	86
Multitude	Fireteam	30	16	480	80
Nihilistic	Vampire: The Masquerade	24	20	480	77
Treyarch	Draconus	24	20	480	73
Micro Forte	Fallout Tactics	18	27	486	81
Bohemia Interactive	Operation Flashpoint	49	10	490	86
Muckyfoot Productions	Startopia	24	21	504	86
Monolith	No One Lives Forever	24	22	528	90
Irrational	System Shock II	18	30	540	92
Mythic	Dark Age of Camelot	18	30	540	89
Outrage	Descent 3	31	19	589	87
DreamForge	Sanitarium	16	37	592	84
Looking Glass	Thief: The Dark Project	30	21	630	92
Ion Storm	Deus Ex	34	20	680	91
Surreal	Draken	28	25	700	82
Treyarch	Spider Man	18	40	720	77
Raven	Voyager: Elite Force	24	33	792	85
Ensemble	Age of Kings	24	40	960	92
Lionhead Studios	Black & White	37	28	1036	89
Lucas Arts	Star Wars Starfighter	30	40	1200	85
Naughty Dog	Jak & Daxter	36	35	1260	89
Gas Power Games	Dungeon Siege	44	32	1408	89
Blizzard North	Diablo II	36	40	1440	88
Westwood	C&C: Tiberian Sun	36	40	1440	78
Sierra	Gabriel Knight 3	36	48	1728	79
Bioware	Neverwinter Nights	60	75	4500	90

For a while I succumbed to the temptation to divide the "buck" score by the "bang" score, but "buck" is nonlinear. Any formula I came up with to normalize "buck" could be adjusted to make almost any game on this list the highest bang-for-buck of them all. It's better to just stare at the data and try to get an intuitive feel for it.

So whose practices should we follow? That depends on what our situation is. For example, if I was a new developer with a small budget, I'd want to look at what people managed to accomplish with small budgets.

Suppose we arbitrarily choose 85 as the score we'd like to hit. The games that got 85 or higher, and took the fewest developer-months to make, are Unreal Tournament, Rogue Squadron II, Heretic II, and Rainbow Six.

And what do we learn from them?

• Don't Write An Engine From Scratch. If you want bang-for-buck, code reuse is a damn good idea: Unreal Tournament used the Unreal Engine, Rogue Squadron II used Rogue Squadron I's level editor and other technology, Heretic II used Quake II, and Rainbow Six used the Virtus renderer and 3ds max for their level editor. (Side Note 1: they had major problems with Virtus and ended up having to overhaul it. Brian Upton believes they should have written their own renderer. I believe using the crappy renderer allowed them to get started making content earlier than they could have if they had to wait for their own systems to be built, and they would have shipped later if they wrote their own. But who are you going to believe? I wasn't even there. Still, these days the point is moot: you can license a good engine.) (Side note 2: they wrote a brand-new engine for SWAT 3, and still managed to ship a great game in eighteen months. The trick they used to allow them to start developing content before the engine was online was this: they used the WorldCraft editor, and kept their engine compatible with it.)
  
  
• Crunch. All these guys did their crunch time.
  
  
• Spend as little time as possible in preproduction. Neither Rainbow Six nor Unreal Tournament had a design document. None of these games had a prototype phase. It looks like that if you want to get your game done quickly, you can't prototype it first. This was fine for most of these games because they were sequels; the previous iteration was the prototype. The exception is Rainbow Six. Here, the design that was in their minds turned out to be just as fun as they imagined once it was made flesh.

The key thing is to look at advice from people in a similar situation to you, because the techniques of a small developer won't necessarily scale to a large one. Management techniques that work for a small team (let's get everyone in a room and hash it out) don't scale to a large one. (We don't even have a meeting room big enough for our team.) A developer who has four years might benefit greatly from writing their own engine from scratch. A developer who has two years might benefit greatly from a prototyping phase.

So suppose we wanted to make the best original game possible, and not spend a fortune? Here the winners are a totally different crowd: Thief, Deus Ex, and No One Lives Forever. What can we learn from these guys?

• Let a smaller team (approx 20) work for more time. (2-3 years)
  
• Have a preproduction phase. (Although Looking Glass may not have realized that they were in preproduction when they were.)
  
• Have, or arrive at, a focus or mission statement for the game.
  
• Strive to make your game design systemic.
  
• Make tools to empower art and design to add to game without programmer assistance.
  
• Playtest all through the project and incorporate the feedback from playtesters.
  
• Make a first-person PC title with a stealth element and a sniper mode.
  

All of these suggestions sounded good up until the last one, huh? Like I said, use this technique to stimulate your thinking rather than override it.

Copyright © 2003 CMP Media Inc. All rights reserved.