[In this Game Developer magazine article, designer Ara Shirinian (The Red Star, uDraw) takes a look at two years' worth of magazine postmortems to mine them for common data points -- examining both what went right and what went wrong how often, and why.]
After having participated in numerous game projects throughout my career as a designer, including many failures and successes, I noticed that certain types of development mistakes appeared to recur with surprising frequency. Was this just another Twilight Zone-inspired idiosyncrasy of my career? Or is there something more going on here?
I imagined a development hell where throngs of teams all ran into the same pitfalls over and over, without knowledge of what they were doing wrong, and without the realization that anyone else might be making similar mistakes.
As developers, we have our own career histories to depend on for knowledge and experience about the rights and wrongs in game development. Beyond that, we can only rely on other developers' willingness to relate their own experiences, mistakes, and solutions to us.
To that end, things like postmortems, conferences, and just plain open discussion amongst developers are great tools.
But these are only vignettes in a sense, and often highly contextually dependent. For example a team's troubles with Lua integration are not helpful if you never use Lua.
Beyond that, I was curious whether there was a bigger picture, what it looked like, and if the results could help us learn something new about game development in general. That, in a nutshell, is the motivation behind this analysis.
Collecting Data about Game Development
Without any expectations about whether I would find any interesting trends or commonalities across game development projects (or whether I'd even find anything coherent in the first place), I decided that Game Developer's extensive history of postmortems was the best and most consistent source of data.
Conveniently, postmortems written for Game Developer all share a similar structure: the developer writing the postmortem is required to select and illustrate five things that went right during the project, and five things that went wrong. Despite the enormous range of project types and sizes, this made it relatively straightforward to collect and organize data about the good and bad things that were reported about projects.
The data set for this analysis consists of 24 successive postmortems published in Game Developer, covering a period of two years, from articles that were published from February 2008 to January 2010.
Collecting Data about Common Issues
Before data collection began, I defined a set of various issues or situations that I thought would be interesting to track across all postmortems. For example, whether the postmortem mentioned using scrum or some agile process, whether a successful experience with outsourcing was reported, and so on.
For each of these items, each postmortem was scoured for mention of that issue or situation. I didn't just search for specific words; collecting the data required a lot of extensive re-reading to ensure that if the postmortem was counted in a certain category, there would be no question about its inclusion.
Here are some metrics that characterize the projects whose postmortems were included in this analysis.
Team Sizes Reported
The largest project was Far Cry 2, boasting a team of 265 members. The smallest projects were Age of Booty and n+, both reporting just five team members. Four postmortems reported a range of team size; of those, the maximum was recorded.
Two postmortems did not report a team size and were excluded from this section. Team sizes tended to fall into four clumps, with the smallest teams consisting of five to 10 members. The next five largest teams consisted of 30 to 40 members. There was another clump representing 68 to 90 members, and six projects reported over 100 team members.
Multiplatform vs. Single-platform
Slightly more than one third of the projects were released on multiple platforms.
Multiplatform projects are counted once for each platform they were released on (there were no PSN games represented in this selection of postmortems).
The average development time for projects was 2.4 years. Two projects did not report a development time and were excluded from this section. Interestingly, development time also seems to clump at yearly or half-year marks, although this is probably an artifact of how durations were reported. There is also a large jump between the two-year mark and the three and a half-year mark.
Use of Contractors or Outsourcing
Eleven of 24 postmortems reported utilizing contractors in the project. Only 7, or about 29 percent, reported outsourcing work to separate companies.