Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
June 23, 2017
arrowPress Releases
June 23, 2017
Games Press
View All     RSS






If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Manager In A Strange Land: Benchmarking, Part 1

November 7, 2003
 

I mentioned in my last column that learning from experience sucks. Most of learning from experience is learning from mistakes; learning from mistakes means you have to make the mistake at least once. It would be nice if we didn't have to make the mistake at all. But there's another problem with learning from mistakes: there are way too many mistakes you can make when developing games. I used to work at a game company where, every several months, the president would give us a speech that went something like this: "Well, we haven't been doing too good lately. We made a lot of mistakes, but I think we learned a lot from them, and we won't make those same mistakes again." No, we made all new mistakes.

What would be really nice is if we could prevent some of these mistakes from happening in the first place.

That's where benchmarking comes in. Benchmarking is looking at exemplary companies and adopting their best practices.

Sounds good so far. Here's where this article gets sketchy. I'm about to break every rule of good statistical analysis and still try to draw some conclusions. I've gone through almost every postmortem in Gamasutra, and listed the ones that got a score higher than 70 (Why 70? Because it's the mean.) on gamerankings.com, along with how long they took to develop, how many people were on the team, and their gamerankings score.

Some of the obvious problems with this approach are: scores on gamerankings are not a valid measure of game quality. (But these scores are a better measure than game sales.) Also, the numbers different people report for their team sizes use different methods: some report averages, some report total number of people who ever worked on the project, some report peaks. Also, the sample is not representative.

The problem is, we're starved for data, so I'm willing to take what I can get, in the hopes that some information that might be right is better than no information at all. So, like with all the articles I write, use it to stimulate your thinking, but don't take it as gospel.

Studio

Title

Months

Team

Buck

Bang

Epic

Unreal Tournament

18

16

288

94

Factor 5

Rogue Squadron II

9

32

288

89

Ritual

Heavy Metal: FAKK 2

18

18

324

80

Ritual

Sin

20

17

340

71

Raven

Heretic II

11

32

352

85

Sierra Studios

SWAT 3: Close Quarters Battle Spec

18

20

360

84

Zombie

SpecOps: Rangers Lead The Way

20

18

360

77

Activision

Heavy Gear 2

19

20

380

82

Raven

Soldier of Fortune

23

20

460

84

Red Storm

Rainbow Six

21

22

462

86

Multitude

Fireteam

30

16

480

80

Nihilistic

Vampire: The Masquerade

24

20

480

77

Treyarch

Draconus

24

20

480

73

Micro Forte

Fallout Tactics

18

27

486

81

Bohemia Interactive

Operation Flashpoint

49

10

490

86

Muckyfoot Productions

Startopia

24

21

504

86

Monolith

No One Lives Forever

24

22

528

90

Irrational

System Shock II

18

30

540

92

Mythic

Dark Age of Camelot

18

30

540

89

Outrage

Descent 3

31

19

589

87

DreamForge

Sanitarium

16

37

592

84

Looking Glass

Thief: The Dark Project

30

21

630

92

Ion Storm

Deus Ex

34

20

680

91

Surreal

Draken

28

25

700

82

Treyarch

Spider Man

18

40

720

77

Raven

Voyager: Elite Force

24

33

792

85

Ensemble

Age of Kings

24

40

960

92

Lionhead Studios

Black & White

37

28

1036

89

Lucas Arts

Star Wars Starfighter

30

40

1200

85

Naughty Dog

Jak & Daxter

36

35

1260

89

Gas Power Games

Dungeon Siege

44

32

1408

89

Blizzard North

Diablo II

36

40

1440

88

Westwood

C&C: Tiberian Sun

36

40

1440

78

Sierra

Gabriel Knight 3

36

48

1728

79

Bioware

Neverwinter Nights

60

75

4500

90

For a while I succumbed to the temptation to divide the "buck" score by the "bang" score, but "buck" is nonlinear. Any formula I came up with to normalize "buck" could be adjusted to make almost any game on this list the highest bang-for-buck of them all. It's better to just stare at the data and try to get an intuitive feel for it.

So whose practices should we follow? That depends on what our situation is. For example, if I was a new developer with a small budget, I'd want to look at what people managed to accomplish with small budgets.

Suppose we arbitrarily choose 85 as the score we'd like to hit. The games that got 85 or higher, and took the fewest developer-months to make, are Unreal Tournament, Rogue Squadron II, Heretic II, and Rainbow Six.

And what do we learn from them?

  • Don't Write An Engine From Scratch. If you want bang-for-buck, code reuse is a damn good idea: Unreal Tournament used the Unreal Engine, Rogue Squadron II used Rogue Squadron I's level editor and other technology, Heretic II used Quake II, and Rainbow Six used the Virtus renderer and 3ds max for their level editor. (Side Note 1: they had major problems with Virtus and ended up having to overhaul it. Brian Upton believes they should have written their own renderer. I believe using the crappy renderer allowed them to get started making content earlier than they could have if they had to wait for their own systems to be built, and they would have shipped later if they wrote their own. But who are you going to believe? I wasn't even there. Still, these days the point is moot: you can license a good engine.) (Side note 2: they wrote a brand-new engine for SWAT 3, and still managed to ship a great game in eighteen months. The trick they used to allow them to start developing content before the engine was online was this: they used the WorldCraft editor, and kept their engine compatible with it.)
  • Crunch. All these guys did their crunch time.
  • Spend as little time as possible in preproduction. Neither Rainbow Six nor Unreal Tournament had a design document. None of these games had a prototype phase. It looks like that if you want to get your game done quickly, you can't prototype it first. This was fine for most of these games because they were sequels; the previous iteration was the prototype. The exception is Rainbow Six. Here, the design that was in their minds turned out to be just as fun as they imagined once it was made flesh.

The key thing is to look at advice from people in a similar situation to you, because the techniques of a small developer won't necessarily scale to a large one. Management techniques that work for a small team (let's get everyone in a room and hash it out) don't scale to a large one. (We don't even have a meeting room big enough for our team.) A developer who has four years might benefit greatly from writing their own engine from scratch. A developer who has two years might benefit greatly from a prototyping phase.

So suppose we wanted to make the best original game possible, and not spend a fortune? Here the winners are a totally different crowd: Thief, Deus Ex, and No One Lives Forever. What can we learn from these guys?

  • Let a smaller team (approx 20) work for more time. (2-3 years)
  • Have a preproduction phase. (Although Looking Glass may not have realized that they were in preproduction when they were.)
  • Have, or arrive at, a focus or mission statement for the game.
  • Strive to make your game design systemic.
  • Make tools to empower art and design to add to game without programmer assistance.
  • Playtest all through the project and incorporate the feedback from playtesters.
  • Make a first-person PC title with a stealth element and a sniper mode.

All of these suggestions sounded good up until the last one, huh? Like I said, use this technique to stimulate your thinking rather than override it.

 

______________________________________________________


Related Jobs

Wargaming.net
Wargaming.net — Chicago, Illinois, United States
[06.23.17]

Software Engineer (Audio)
Wargaming.net
Wargaming.net — Chicago , Illinois, United States
[06.23.17]

Lead Tools Engineer
YAGER Development GmbH
YAGER Development GmbH — Berlin, Germany
[06.23.17]

Gameplay Programmer (f/m) for Dreadnought
Tangentlemen
Tangentlemen — Playa Vista, California, United States
[06.22.17]

Technical Art Lead





Loading Comments

loader image