|
Is it the company or the people on the team that determine the success of video games? And, to the extent that it is the people that determine success, is it the designer or producer role that has the largest impact? A recent research paper of mine attempts to answer these questions, by studying 854 computer games, 537 individual producers, 739 individual designers, and 395 companies between 1994 and 2006 (representing $4 billion in revenue), using a sophisticated econometric technique called Multiple Membership Cross-Classified Multilevel Modeling. Controlling for a wide variety of factors, I found that the difference among individual producers accounted for over 22% of the financial success of the game, the differences among designers around 7%, and the rest of the game developer’s people and organization, about 21%. These results, while surprising, turned out to be extremely robust over many types of analysis – the choice of producer is key to determining a game’s revenue, while the choice of designer matters very little. Further, these skills are portable, even when producers move from one developer to another
For this analysis, I used the MobyGames database, combined with the work of a team of RAs, to get data on game credits and company success. The list of games included everything from the top selling games (The Sims, Roller Coaster Tycoon, Starcraft) to smaller and more specialized projects (Immortal Cities, Two Thrones, Battlecruiser 3000AD). I used NPD Funworld’s revenue figures to track success. (It may be unsurprising to people in the industry that I tried to use GameRankings and MetaCritic ratings as well, but they turned out to be bad statistical measures of game success, though they showed similar results – that the role of producers were at least twice that of designers). I statistically controlled for the size of the team, the genre of the game, year of release, whether titles were based on licensed property, whether they were sequels, and the publisher of the game. After accounting for these many differences, I looked at how much of the revenue differences between games could be explained by the designers and producers on the project, and how much by the company.
Compared to the impact of individual designers and producers, the importance of the game developer as a whole was surprisingly small, accounting for 21% of the difference between games. Especially given that I do not include other important team members in my analysis (such as programmers and artists), it is unclear how significant top managers at game developers actually are in explaining performance, but they are, at most, on the same scale as the role played by just one individual within the product team. Far from being interchangeable, the individuals involved uniquely contribute to the success or failure of a game.
Additionally, the relative importance of producers over designers in explaining a game's success is surprising. Even in an industry that rewards creative and innovative products, the difference between top and bottom designers seems to have much less impact than the difference between top and bottom producers. Rather than acting as cogs in the machine, dwarfed by the importance of top managers, the effect of producers on performance was actually larger than that of organizational factors, implying that differences between producers play an outsized rolee, even over the $4 billion in revenue generated by games in the sample. Though this finding might seem surprising, it is supported by research on the role of project middle managers in the innovation process.
Recent research on the role of individuals and groups in industries as diverse as consulting and comic books supports a longer literature on project management that has demonstrated the complex interaction between individuals and teams in successful innovation. The finding that producers have significantly more impact on firm performance than individual designers aligns with this tradition. It suggests that high-performing designers alone are not enough to generate performance variation; rather, it is the role of individual producers to integrate and coordinate the innovative work of others.
It is worth noting that even though producers seem vitally important to the success of a game, they are not compensated in a way that acknowledges the vast difference between the best and worst producers. Based on a survey conducted with the attendees of the Game Developers Conference in 2008, I found that individuals reported relatively weak connections between performance and reward in the game industry. Producers, for example, reported bonuses that accounted for less than 9% of their total salary on average, with only 10% of producers reporting either a share of royalties or profits. It may be the case that many producers are not truly aware their own significance, or at least cannot prove it, without the kind of detailed longitudinal data I used in my analysis. Successes are claimed by many, while the blame for failures is often blurred, or, in the words of the old proverb, “victory has many fathers, defeat is an orphan.”
It is worth returning to the data one last time and noting that the difference in individual ability among producers, and, to a much lesser extent, designers, is one of the most important drivers of a game’s revenue. A game that would normally generate $30M in revenue could see their sales change by as much as $7M, depending on the quality of the producers involved. The difference between selecting either the top and bottom designers, however, accounts for a less significant $2M. And, keeping the same producer and designers, but changing the developer, could result in a difference in revenue of about $6.5M. You can read more detail in the full paper, People and Process, Suits and Innovators: Individuals and Firm Performance.
This does not mean that designers are not important, or that they do not play a vital role, but it suggests that getting the "best" versus the "worst" design talent has far less impact on the game than getting the best possible producer.
Some additional information (updated, again): Based on the comments I have received, a few additional points. First off, this is basically the result of a statistical, economic analysis - the results surprised me as well, I was certainly expecting to find a bigger impact on revenue from designers than I did. The results are not intended to show a lack of respect to designers, nor was the title of the post, and I have attempted to clarify the language in hopes that I do not cause offense. It is merely the results that I found, and that I am reporting here. The analysis shows that having the best versus worst designers on a team only changes project performance slightly, while having the best versus worst producers on a team has a tremendous impact on the performance of a game. I can only offer theories about why this is the case, based on other academic work. It might be the case that producers have a wider range of talent than designers (the best producers are really good, and the worst producers are really bad, while most designers fall somewhere in the middle), for example. Or, as is the case with many project managers in creative industries, the producer's role in bridging the gap between all of the stakeholders during development might be far more important than is realized. Any ideas you have to explain the gap would also be appreciated.
As to the source of credits, I used Moby Games, with some spot-checking of the data to ensure accuracy (which seemed high). I realize credits do not always reflect contributions, but unless there is a reason to suspect that there is severe bias that would cause either designer or producer credits to be particularly inaccurate over 12 years and over 850 games, this shouldn't have a large effect. There is much more discussion of Moby Games and data collection in the comments below.
|
While I would dearly love to take your conclusion at face value and run with it (since I'm a producer), there are many questions that leap to mind in reading your post. The first of which is, "What description do you apply to the role of producer?" Throughout the industry there are many different types of producer position, some of which are almost entirely creative while others are better described as project managers with little creative input.
Those people talented enough to consistently reach the higher-level (of production or development) will undoubtedly perform better by being better at their job, being more capable of finding a higher level company, and being more capable choosing a higher level project.
Also, your model would need to account for many levels of fluidity. Some companies/projects consistently do a better job. But the creativity of designers may influence their decisions to switch to less economically viable but more creatively enticing positions away from a successful series of titles, where as a producer seems more likely to stick with a series of games that will consistently perform at a high level because of the established game design, engine, etc.
I attempt to deal with some of the fluidity you identify by having the model also take into account the performance of the game company itself, as well as controlling for sequels.
By the way, an additional complicating factor is that EA has often used the term 'producer' to mean _designer_, unlike the rest of the industry -- though this is gradually changing.
Anyways, the Mobygames database really is pretty bad. On a brief skim, many credits for (each years' release of) NBA Live and Madden are missing - and what few credits there are, are often of auxilliary projects outsourced to other developers.
One last concern - the paper says that you filtered out designers and producers who had repeatedly been part of the same team at the same company. If this was sports, wouldn't this be like eliminating out the top performers (who are more likely to stick with a successful situation)?
That is certainly true of my games - The ones I did for Sierra On-Line sold much better than the games I made for Legend Entertainment, Legacy Software, or JetSetPoker. This had nothing to do with the producers - For example, Bob Bates of Legend is one of the finest executives in the industry. But Sierra had a more established product line, marketing, and sales power. So an equally good game at Sierra of course sold better than our Legend game.
Further, sales within a company vary by the prominence of the project internally as well as by the success of previous games in a series.
I hope that it is okay, I've posted a link to this article on my website.
For example, I'm down as "special thanks", rather than "mission scripter" on the credits of the first two titles I worked on (NARC and State of Emergency 2), because those games were completed by another company when they reached the beta stage. The same goes for the rest of the development team on those games.
And that's not even getting into people at other companies that just don't get credited at all, for one reason or another
I'm sure similar research in the 1800s would have found that the slave master was more "critical" to the output of the plantation than any individual slave.
Full disclosure: programmer who worked with horrible producers at a past company, biased toward dismissing this as utter garbage but waiting until I read the full paper out of fairness.
Since I wrote that statement yesterday, I have read the paper and had a back and forth in these comments, and I feel what is happening is that Ethan is presenting objective data (from questionable sources) but also that he is interpreting it in a way that is incredibly subjective. He went from "change in producer capabilities affects profits more than change in designer capabilities" (a dubious stance to begin with; I'll get to that) to "producers matter more than designers", a very insulting manner of wording things.
Now, how is "change in producer capabilities affects profits more than change in designer capabilities" dubious? It seems like a neutral and non-insulting way of interpreting the data, but it is still somewhat off. I don't think it is that producers add more to the project, or that there is somehow a larger gap between the good producers and the bad producers. I think it simply comes down to power. I will admit this is subjective and anecdotal, but I have seen good and bad decisions from designers get snuffed while good and bad decisions from producers go through unchallenged. So another way of interpreting the data is "people given power affect profits more than people who have no power", and as is often the case in our industry the people given power are producers. It's a self-fulfilling phenomenon.
Now, do I think "this" is utter garbage? It depends on what "this" is, there are many facets of this discussion that I have become aware of since yesterday. I guess, to put it simply, I feel that Ethan's article is utter garbage for the way it was written, and that his paper is not necessarily utter garbage and is the result of some impressive efforts. I also fear it can be used as a datapoint weapon to further stifle innovation and creativity in this industry for the purpose of "Safe" development, which ends up being "Development that fills the pockets of the executives while developers get laid off". Our industry needs no help in furthering these agendas I'm afraid :(.
Having worked on both sides (Game Designer and Producer) I must say that I don't agree with your evaluation that lead to the bombastic title that this post has.
Of course that having good producers running the show will lead to better results. As it was said before, they are in charge that the game get's done on time and on budget.
To say that producers matter much more than game designers is extremely aggressive and quite a lack of respect for the enormous work that rests on the shoulders of a designer. It is the designer's role to create the world, story and gameplay mechanics. It is the designer's role to give a piece of software one thing called... Fun! And voila... you have a game! Of course the designer needs a team to accomplish all that, but he is the person with the vision. He is the soul of the projects.
To take the kind of conclusions above would be the same than to say that Joel Silver was much more important for Matrix than the Wachowskis. Doesn't makes sense, right?
And do not forget that you are talking about an industry that was forged by people who did programming/design/graphics,etc by themselves and that released products that sold hundreds of thousands (with no producers).
Of course that as the industry has grown, the role of the producer has become more and more important. I specially like to work with creative producers and not just project managers. In my opinion, the producer is in charge of providing the designer with the resources that he needs to do his job (taking in consideration publisher relations, etc...).
The question of which employees "matter" more in a company is so subjective and certainly cannot be determined "statistically". You'd have to take into account pay and hours worked and many other factors.
I think what you might be trying to say is that certain producers are more highly correlated with being involved in successful games than certain designers. That is to say you can have the best design team in the world and producers can still screw it up. Vice versa you can have the worst design team in the world and a bunch of really good producers can get the assets together to make a good game.
Then again I am not sure what the heck you are trying to say. Your title certainly sucks.
I liked your nonsensical graphs too.
I appreciate you taking the time to not only post the brief of your paper, but also respond to comments and even edit your post title (not sure I would have given in were I you, though...if it's your work, feel free to stick up for it, title included). As Bruno pointed out, good discussions are a must.
Don't be discouraged by some of the responses you're likely to elicit by posting on this topic. Producers are widely regarded as evil incarnate throughout game development -- and not without reason. Nonetheless, I encourage you to endeavor to persevere. Welcome to the war.
What "war" do producers need to fight? (genuine question). I've worked with producers that deserved their appreciation, and I've worked with producers that were significantly overappreciated, but I've never worked with one that I felt was underappreciated. Contrast that to testers, or, lo and behold, designers. Testers have it the worst in this industry because they are seen as easily replaceable (a dehumanizing metric, coming down harder on someone because they can "be replaced", similar to the one that fuels this article), but designers have it pretty bad too considering they founded the damn industry and that their craft is unique to this industry (unlike any other skillset we use to make games) yet they are seen as almost as disposable. So while it is obvious to me (once again, I am a programmer, so I feel disposable too; bias alert) the need for low level workers such as programmers or testers or designers to constantly fight to improve a faltering position in this industry (against bullshit such as this paper), I really don't understand your stance that producers need to fight a war. Are they not getting paid enough more than designers ($88,544 for producers, $70,223 for designers: http://www.gamasutra.com/view/news/34304/Game_Developer_Reveals_2010_Game _Indust
ry_Salary_Survey_Results.php)? On top of the pay, is the power that they hold over the project not enough? The way they come into projects we have been working on for months and immediately get to tell us what to do? On top of that, the fact that they are more likely to get face time with game magazines, thus promoting their own worth in the public eye over us peons? What do you need to fight for?
What is the intended outcome of the rhetoric in your comment and this article, if not to widen the gap between those who have power in this industry and those who don't? Maybe it's just defensive, to wash out the "evil incarnate" stereotype (I'm all for that, like I said I've worked with good producers and they shouldn't be lumped in with the bad ones), but I don't see this article doing anything more than aggressively positioning to give producers even more money and bargaining power. The ones with power in this industry should be those who directly create, with those in support roles such as producers and CEOs answering to the creators. Unfortunately, the industry has become completely backwards in those regards due to the creatives that were once in power suffering from the blindness of short-term greed and selling our artform to suits, and yes I will fight a war, a million wars, to set things straight.
But if war can be avoided, better for both sides. I am curious, Keith - are your statements offensive or defensive? Because the article you defend is most clearly offensive.
To clarify my stance with a little background, I'm now an independent production consultant. After 12 years in AAA dev, most recently as a producer, I left studio life to start a business based on improving project management for any and all studios. In my own experience as a developer, I saw creatives (whom you and most others term "designers") being heralded with greater authority while those in the role of project manager (at my studio these people were termed "producers") were left the more unseemly task of finding a way to manage the resources necessary to execute the designers' vision. The conflict I've seen and that I referenced above is really the one between the creative direction and project management.
My personal stance is that those responsible for successful project management go largely unheralded in game development while those with creative vision tend to be elevated when they meet with success. One need only go down a list of industry luminaries to see the distinction. I can easily name several creative geniuses well deserving of praise, but I don't know of anyone in the project management realm who is known by name for having shipped well-made games on time.
Monetary value isn't a factor in the discussion to me. Rather, the focus is on recognition of excellence regardless of whether one's role is creative or managerial. Both are difficult, both should be lauded when they succeed.
Regarding Ethan's brief, it strikes me as being noteworthy from the standpoint of an academic effort but it doesn't use reliable data (most in the industry find Moby to be a deficient source of data) and it unwisely tries to make sweeping categorizations of industry positions. I'm not defending the content of the brief, but I encourage Ethan's research into the industry.
Note that Keith is a consultant(apparently). The only way he makes money is if producers pay for his services. That MIGHT influence his views a little bit.
I won't get into the designer vs. producer war, because I agree with you that the "low" level people are the ones who should be getting more credit.
My problem is not with producers or designers. It is with people who get into positions of power simply because their fathers or mothers or uncles or cousins are in positions of power. Or because they are just really good at sucking up. This is a disturbing problem in North American business culture.
I detest the more mundane production task of assembling teams, scheduling, etc, etc, etc. I'd rather be creative.
Unless you speak to the individuals to clarify their roles on their respective projects, and then categorize them based on mostly creative (design) or mostly management (producer) your findings are completely invalid.
I've found LinkedIn to be pretty good... for those developers who use LinkedIn, anyway.
However, without the designers -- or any other discipline -- you won't have a product. More importantly, without designers, you won't have a _game_.
Reading your paper, I wonder about some of your assumptions. For example, on pg. 4 you mention that, "(W)e would expect managers charged with creative or innovative tasks to matter more than the 'suits,'" which seems off. Consider other creative jobs, such as graphic design or interior design. The creative manager typically has to bow to the whims of the client or "suit," regardless of the creative's reputation or internal authority. Because we are an industry filled with for-hire creatives, rather than _artists_ (who, in ideal, answer only to their own inner voice), we are bound to satisfy the needs of the client for the work at hand.
[Aside: I find it funny that you call the producers "suits," as -- in my experience -- that was always a term reserved for _external_ production/management personnel, not _internal_ production/management.]
You also mention that "Designers often start their careers as programmers" (pg. 11), yet in my 14+ years in the industry across 5 different companies, I've only known 4... maybe 5... designers who were programmers originally. A few programmers that I've known have wished they were designers but were also reluctant to relinquish the pay of a programmer for the apparent creative "freedom" of a designer. I know designers who can code a little -- mostly hobbyists, although I know of a couple who majored in CS -- but that's not the same as starting in the industry as a programmer. Where did you get the info for this point?
Also, I don't see an examination of the different classes of games made. For example, the development of a licensed game by a third-party developer working for an external publisher will have different creative, organizational and budgetary restrictions than a first-party proven original IP (e.g. Mario games by Nintendo). Continuing, I wonder what the commonalities are between the approaches of the most successful producers of the most successful teams. Are there any?
Why do you lump together beta testers and voice actors into the game design team (pg. 12)? One better fits QA and the other better fits audio production.
Was the decision to focus on PC games (pg. 13) because of the greater perceived homogeneity of development, even though PC games have not dominated the marketplace in the same way as consoles, handhelds, or phone-based games? I don't see any clear references to social game development in your focus on the PC market, and I can say confidently that social game development is unlike other "more traditional" game development. (Ah, never mind. I see that PC games were selected, in part, because of the _scope_ of possible development, from low-res to high-res experiences.)
I do agree that the success of a game is dependent on the producer's management acumen, organizational skills, and ability to see the "big picture" of the process at work. The producer is also, frequently, the only person with the bandwidth or perspective to actively and effectively engage any external interests. However, when we say "who matters more," it seems we're really discussing who has more power within the context of a project. The producer, as the one who manages the leads, is obviously the one driving the car (or should be, if that producer is effective), with the designer riding shotgun as navigator. But how is this surprising? I don't mean to sound callous, but it seems fairly apparent to someone who has worked in the industry. I suppose I can see how your paper would certainly dispel the illusions of someone who _believes_ they know how a team is guided in corporate game development.
Just some thoughts. Thanks for being brave enough to make your paper so readily available!
http://www.mobygames.com/game/windows/arcanum-of-steamworks-magick-obscura
During my interview with Troika Games cofounder Tim Cain for my book, I asked him about the credits listed at MobyGames. He replied that only 12 people actually worked on Arcanum and none of those 12 were credited with titles. An additional eight were credited for contract work, and then Sierra added credits for producers, marketing, QA, etc.
The titles listed in the credits for at least this game at MobyGames were fabricated. So, not only are there are many more people credited with working on Arcanum than who actually did at MobyGames, but their titles are suspect, too.
Your source of data is your research's greatest stumbling block.
As many of you point out, Moby Games is not perfect. My results are not really affected by omission - games missing credits in Moby Games - but could be affected by errors - games where credit information is incorrect. In my incomplete spot checks of the data I used (which again, covered 12 years) I didn't find any errors, but Morgan's story (and others like it) are always worrying. To the extent that credits are manipulated, it could bias my results, but it would generally only bias them if errors were widespread AND disproportionately affected either producers or designers, but not both. I don't think that this is occurring in my data, but I can't completely rule it out.
Duncan, you also raised lots of good points. To address a couple - I exclude QA and voice actors from the design team, for the reasons you suggest. The programmer-designer track was much more common earlier in the industry (my data starts in the early 1990s). Also, some of the oddness of language of the paper is because it addresses a more general case, and is for an academic audience. I tried to make this blog post more directly aimed at the game industry, which I have studied extensively.
To return again to what the results mean: given a number of factors (genre, year of game launch, publisher, team size, etc.) the choice of who will fill the producer role will have three times the affect on the game's revenue (and to some extent, ratings), than the choice of who fills the designer role. It could be, that in the words of Armando, the "more mundane production task of assembling teams, scheduling, etc, etc, etc.." are more important than we give them credit for. It could be that all designers are excellent, and the quality of producers varies widely, but the effect itself is remarkably robust and persistent across a variety of measures and statistical techniques. It doesn't mean that I can't be wrong, but I do think that the data cannot be dismissed out of hand. Hopefully, it helps add usefully to the discussion, which is what I try to do as an academic who studies this industry. And, if I am not scared away entirely, my next post is going to be on entrepreneurship in the game industry, which I hope will be less controversial.
Anyhow, I am taking all of your comments into account, and feel free to ask any other questions....
Thanks for the reply.
I think I may be misreading the section on page 12:
"The average game design team in the sample has 56 people, and often several dozen more temporary workers, such as voice actors and beta testers."
If you're excluding VO actors and beta testers, then the language _might_ be confusing to your target audience. Just a minor note.
Also, 56 people seems like a rather high mean for the sample presented. Once, when I was interviewing at a company for a design position, I was asked how many direct reports I had in my department. When I said "nine," I raised a few eyebrows, as that was considered a large number of designers to manage. The largest design team I was on had 14 designers at its peak. Does current 1st-party AAA development skew the mean that drastically? In your survey, were there more designers per project on average at any specific point in time (i.e. the beginning of the current gen of game development on AAA titles)? Are you taking distributed teams for specific SKUs and incorporating them into the grand design total? (That last question seems like a plausible explanation, but could also lead to flawed data regarding the roles and responsibilities of design and production.)
Thanks again. Don't shy away from provocative material!
EDIT: Added the words "current 1st party" to my question regarding AAA development and design team size.
It could even be that producers are simply given more power by the higher ups to do what they feel is right and thus are engineered to have a larger sway on the outcome of the product (hint: it's that one. The one I just typed.)
The title of your article was "Why Producers Matter Much More than Designers" on an industry site visited by professional developers of video games, including producers and designers. It is an unstable industry too, whose members constantly have to fear layoffs and naive college grads salivating for their jobs. You, an outsider to a very unique industry, are effectively waltzing in out of the blue telling a large swath of this industry's veterans (who already don't get the respect that they deserve) that they are insignificant. With inaccurate and dehumanizing statistics the main source of which (mobygames) is a common laughing stock due to its inaccuracies. How on earth could you not expect backlash?!?!?
Ok, so assuming you meant no ill, there are obviously better ways of presenting this data (changing the title and general tone of your article would help). I do have one question though. Why present this at all? Sure it's data, but humans have limited time on this earth and only put that time into mining data that serves their agenda, even if their agenda is a mild curiosity (rare). Why designers and producers? Why did you feel the need to write this paper in the first place?
When you were scrounging through credits, did you make sure you separated producers from the actual development team and the publisher? I've worked on titles where the publisher felt they needed to include all their producers even tho they had absolutely nothing to do with the project..
Duncan - Thanks for the feedback, the distributions of the sample are all in the paper, I am not sure how to show graphs and tables in a blog response.
Jefferey - Sorry you are annoyed that I have decided to share these results. I should clarify that I don't mean that designers are insignificant, I do mean that the choice between a "top" and "bottom" designer doesn't have much affect on the revenue from a game. That should be comforting to at least 50% of designers who might not be considered the top of the field. Also, I went into academia to try to uncover some of the mechanisms that underlie innovative and entrepreneurial work. There are huge amounts of academic work on the industries that produce music and movies, very little on games. Understanding how the game industry works is important and interesting, and, in this case, it yields a result that the "non-creatives" may play a bigger role than many suspect - even if you don't like the result. You are welcome to disagree with me, but I am not sure why I shouldn't share findings. If you want to understand more about "why" I study this topic, you can read the academic paper itself, which I think lays it out. That being said, I am taking the criticism you are leveling to heart - I changed the title of the post to be less provocative, and I'll do the same with the text, now. If there are other things that you think are provocative, I'll do what I can to make sure the post adds light, not heat.
I should also say that this represents statistical averages. You, or your company, or your game, may be an exception, but this describes the overall workings of the industry with reasonable robustness.
I don't think this statement from your article is true: "it yields a result that the "non-creatives" may play a bigger role than many suspect". What I mean is, it's true that non-creatives play a large role in game development, it's just that _everyone_ _knows_ this. I know how much of a role producers play, because they are allowed to. I've seen a producer make boneheaded decisions that forced a game to ship with flaws that every designer and programmer wanted fixed. I did some stat crunching of my own with a coworker to comb through reviews after the fact and show just how many reviewers dinged us for these shipped flaws. Similarly I've worked with great producers that provide good feedback for what should go in the game and use schedules to help, not to control. The point isn't the producer themselves affecting the end results more as if they come from some hidden school of excellent game development. The point isn't even that the tasks that they do (scheduling, PR with publishers, holding meetings) is more important than the tasks designers do (mechanics balancing, level layout, stat tweaking); it's that they are in a position of authority and _can_ affect the game more, by the design of the workplace hierarchy. I've been fired for disobeying a command from a producer as innocent as "stop airing our dirty laundry in front of upper management" with no prior marks against me (part of a larger layoff, but it was made clear that my "insubordination" against his evil and selfish command was part of the reason); I'm sure the same would happen if I told a producer that his idea is terrible and I'm not going to implement it. But if a low-level designer has a good idea (let's just say), then the producer can easily shoot it down. He doesn't even have to get permission from higher ups to do so. I've worked with talented designers that would blow your mind away if they could creatively lead a project, but they weren't given the chance because creatives take a back seat to schedulers in this industry. I'm not saying we don't need schedulers or scheduling, just that if the scheduling becomes more important than the integrity of the product then you have a crisis of purpose. And in the game industry, we have a crisis of purpose.
I doubt I am alone with these feelings, and (perhaps unfairly) there is a bit of tension between production/publishing and design/development in this industry (a reason I was asking "why" you chose producer vs designer). A lot of businessmen have invested a lot of money in this industry and want to protect their investment, but a lot of creatives work a lot of crunch to make this industry succeed and then get laid off at the end perhaps _because_ they aren't "as important" as producers or upper management. Against this backdrop, your paper and this article need to tread carefully if you want to avoid picking at scabs.
And then apologizing to Jeffrey for "presenting" the data. I put quotes on "presenting" because his presentation of the data is not the problem(for me the data itself is not even the problem).
It is his interpretation of the data that is inherently flawed.
Determining "statistically" which employees "matter" more in a company. What a load of crap.
Learn 2 communicate.
The research is flawed in many ways that have already been pointed out but a glaring one that hasn't really been mentioned is that higher level producers such as "executive producers" are often credited in sources like Moby Games for every project that fell under their executive umbrella while a designer is almost always only involved in a single project at a time.
Here's an analogy:
A single 911 dispatcher at a local department fields on average 100 calls per 8 hour shift, which must be evaluated and distributed between all the on-duty police officers, the fire department, EMTs, and emergency public works response teams like Gas and Water workers.
A single police officer or fireman at the local departments responds to an average of 5 calls per shift, must come to a resolution for the issues gather information and file a report.
Dispatchers are at some level involved in a greater number of crime and fire prevention activities than an individual law enforcement officer.
Therefore: "Dispatchers matter statistically more than police officers or firemen"
You want to go and tell that to a group of veteran police officers or firemen?
How about the New York Fire Department?
You want to tell them: "Thanks for your contribution but it's the dispatchers that really matter the most"?
That is why you are pissing people off. You are disrespecting our craft.
Furthermore it appears that you are telling the industry with the content of this article that we designers, as individuals, do not have significant value and don't "statistically" have a significant positive impact on a projects commercial success. That our skill sets and talents are interchangeable and that we can be treated like expendable unskilled laborers. It is salting the wounds of an industry that already is notoriously accused of lack of crediting for work and inhumane work schedules. You had to know what can of worms you were opening if you've ever read a site like this in your life.
A nice parallel study would be to do the same research with critical success (e.g. metascore) as the dependent variable. Comparing those two studies, we could see who is really contributing to the industry. After all, if you feed the public a well promoted bad title, it might sell well, but it will only devalue the industry in the long run.
I'm wondering though if there isn't many outside correlations that would also lead to that. I've thought up some.
The first that comes to mind is that sales reflects public awareness as much as the quality of the product. High selling products could be because of the high profile name of a good producer, or more likely, the games with good producers also have good marketing.
Big designers like Sid Meier and Will Wright are often labelled as producers rather than designers, because they take on both roles. This overlap could be skewing the results.
My conclusion is that Designers might have a greater effect on the style or quality of a game, but not in how much money it makes. Sales being driven by publicity and hype rather than quality. In this case a good Producer makes sense. In restrospect, it's kind of depressing.
Be a game designer but have a strong understanding of producing to boot.
You should change the title, or at least the article description, to "The Choice of Which Producer(s) to Hire Matters More Than The Choice of Which Designer(s) to Hire: A Study of Game Revenues".
Second: I'm one of those crazy people that thinks of game design as more art than anything else. However, the scope issues of this art are unlike any other creative endeavor in the world. The role of a producer can make such a difference in that they help bridge the gap between the creative vision and the practical reality. Too many games overstretch themselves and its all-too-easy for a product to suffer from too much vision! However, this vision is the reason the product exists at all, and the best producers are aware of this. I think the fairest thing to say is that games that are well-produced tend to be more polished, and that raises the value of the product. However, that value is limited to that game, and is not tied to the value of the experience that the game is meant to evoke. Intrinsic value is not taken into account here.
In conclusion, although its great too see metrics of the industry as a whole and get more of an idea of what's really going on beyond our own experiences, I don't think this paper will do anything to progress the industry. This seems like a paper that will only convince producers to lobby for higher pay and serve to create an industry that is even more focused on the short term than it already is. Leaving the fact that it compares apples to oranges, it assumes that games are ''good" if they make money and "bad" if they don't. In a world where all creative forms are being consistently devalued, do the suits of the world really need any more reason to feel big and important?
a) The fact that the designer matters very little is only akin to the fact that creativity , ideas and innovation are not critical to the "success" of a game.
b) I am not really sure that Economical Success of a game ( an objective , measurable variable ) can be put at the same level with the fact if the player finds a game enjoyable or not ( a subjective , non-measurable variable ). Since a game can be an economic success due to a good marketing campaing and not for how creative/innovative it is.
c) i dont really understand why are you discriminating Total Revenues and Average Ratings , i mean , wholdnt be a more significant Effective Revenue Measure to Convolute Total Revenues and Average ratings by Multimpling those two Variables ? . That would show if , actually , a Sucessfully Marketed Product had a similiar reception by the gaming community.
Anyways, my take away from this entire post and thread is this: given a team over a certain size, without a good producer at the helm, you are in trouble.
A great game designer is one pillar of a project. Without coordinated support from the other pillars -- facilitated by the producer -- the project will likely fail. Or, perhaps, be a soul-destroying clusterf*ck of epic proportions (see: Brendan McNamara of Team Bondi).
Producers account for 22% of the financial failure of a game, and designers, for 7%.
A good large game cannot be made by haphazardly throwing together designers, artists, and programmers. It is just a train wreck unless somebody can get them all working together. This does not mean that any single job is more important than any other.
There seems to be some sort of ego rockstar complex that people associate with designers. Unless a designer is making pen and paper roleplaying games he needs artists and programmers to get anything of value done, but I don't see artists and programmers whining about being left out of this battle of who is most important in video games. Playing like somebody is extra important because if we left them out of the equation there would not be a game proves nothing. Everybody is essential.
Some of the negative comments on here are just silly.
Designers arguably have more at stake in this debate than any other single camp, because ours is the least portable craft. Programmers, artists, and producers have a much better chance at applying their experience to another field with success. That's not to say that designers' skills aren't useful outside of games, but they're definitely less understood by other industries at this point.
So, respectfully, what you see as a "rockstar ego complex" I interpret as a survival response. Combine that with many designers' ability to identify key assumptions in systematic arguments, and it's easy to understand why so many people have showed up to poke holes in this paper's premise. And rightfully so, I think. Many of these commenters have brought up legitimate concerns with the paper's conclusion.
In fact, I just read all of the comments again to make sure I'm not forgetting anything. None of the negative comments are being silly. None of the positive comments are being silly, although it was naive (if not silly) for Ethan to say "I am not sure how I insulted you by presenting the research", as if it is the presentation of the research that is seen as rude here, completely unaware of the manner in which his title comes off (maybe English is his second language?).
Now, feel free to tell me which negative comments you thought were silly. I won't get offended if they were mine, though I will likely disagree ;).
Let's compare producers to head coach and lead designers to star players. There are generally between 1-2 star players on any given team where as there is only one head coach. But there are a limited number of titles that can be won each year resulting in a limited amount of financial success.
It's easy spot out a few head coaches whose teams consistently outperform other head coaches. Those very successful coaches' teams will bring in more financial success. Base on this model then, the top coaches should be receiving more money financially because their teams are that much more successful over time.
Where as even the best star players peak for only a few years and they never continuously win championships. Based on this model then, the top players should be receiving less money financially because their teams are that much less successful over time.
Its arguable that Phil Jackson represents the best basketball coach ever. He coached Michael Jordan, Scottie Pippen, Kobe Bryant and Shaq. These hall of fame players will never achieve the same number of championships that Jackson has as a coach. Based on this model, Phil Jackson should have been the highest paid person in basketball... problem is that no one goes to a basketball game to see the coach coach. And what makes up the success of the "game" (of basketball) is in either playing the game or seeing players (school to pros) play the game - not in seeing someone manage the game.
What makes games interesting is the design - it is the hardest aspect of game creation to get right. And (like nearly all artistic professionals) most designers fail at achieving something truly amazing and just end up with something passable or good or even great. Those that succeed at a high level generally have a limited time to peak (buyers' choice in game design are often cyclical - RTS, FPS, Sports,... 10 year later, RTS again).
So would I want the best designer with a mediocre producer or the best producer with a mediocre designer? For me as both developer and a buyer, I would much prefer the game created by the best designer. As a player, I'm just paying for my next high... always hoping that next hit will expand my imagination and consciousness. And on those rare occassions, do you think that am I thinking the producer of the game is the greatest of all time... or the designer.
My impression is that the study is not primarily about the importance of producers vs. the importance of designers, it's about good producers vs bad producers. It seems that some producers are allowed to keep producing games despite being so bad at it that it demonstrably hurts the bottom line. The question is why this is the case.
Perhaps, as producers are a more relationship-based profession, their recruitment is more based on relationships as well. Producers with lots of friends in the industry manage to get hired without being very good at what they do.
Either way this is a troll blog
To me, the most interesting questions that many of you have raised are about why this is the case. The numbers don't give us a definitive answer. Outside of the simple interpretation - that the choice of producer, and therefore competence at a the tasks a producer does, is more consequential for games than the choice of individual designer, you suggested alternatives that are worth considering. It might be, as Jack and Karl suggested, that average bad producers are so much worse than average bad designers that this explains the difference. Alternately, it could be that producers have all of the power, and therefore the role of designers is minimized because they are being "kept down" by the production side. I am less inclined to believe the latter, only because of the persistence of the pattern over so many years and types of companies, but it is possible. It was also clear that there is not a real consensus in the industry over this issue, so it is hard to walk away with a definitive view.
I am happy to answer other questions, and, if people are interested, I intend to post again, eventually, about some of my other work on the industry on an issue closer to what I teach and study - the factors associated with successful startups in the game industry.
Dissecting The Postmortem: Lessons Learned From Two Years Of Game Development Self-Reportage
http://www.gamasutra.com/view/feature/6309/dissecting_the_postmortem_lessons_.ph
p
"A new analysis of postmortems published by Gamasutra sister publication Game Developer magazine finds that production issues are overwhelmingly responsible for the success or failure of a project."
I just refused a project where the producer wanted to find any random designer (no experience necessary, not interested in portfolios) to write a design document which they would spend up to 6 months implementing with no play testing or iteration. And he was extremely cold to the idea of the designer interacting with the artists in any way.
It's the second such project I've turned down; some reductionist individual who can't judge the people who actually get brought in to make the darn thing.
No matter what kind of designer winds up with that producer, how is that going to turn out?
Portal is a winner. Design, play-testing, iteration were key in Portal's development and that is the producer's influence.
Zynga is a winner. Their famous test-it-on-five-million-people strategy is the producer's influence, and a design issue, even if their rule stick is more monetary.
If winning producers and losing producers were analyzed, I would not be surprised to see that good producers enable, among other things, good design and effect everyone's performance while losing producers hamper it.
On a final note, Duke Nuk'em Forever is a famous case of horrible production gone wrong at every level... And how was the game design? Folks took a pretty heavy dump on it.
Ask yourself; if you move DNF's producers to a new project and DNF's designers to a different one, which project will suffer more?
Given that most for-profit developers are now iterating on known "formula" games, the primary measure of success becomes "how well can we execute on the formula?" This environment minimizes designer contribution ("don't change what works!") and maximizes producer contribution ("coordinate teams, maximize pipeline efficiency, ensure bug fixes").
This isn't a bad thing -- the process of making games is as important as the art of making games -- but it belies the idea that game designers get to realize "their vision." More often, we put a small spin on an existing design and hope that it's 10% more fun than the spin our competitors come up with.
It would be interesting to repeat this study on 850 indie games, where design usually trumps production values -- but then, how many indie developers have dedicated producers? Similarly, it would be telling to repeat the study over "AAA" games that have defined new genres (Katamari Damacy, Portal, etc) -- but there probably aren't enough of those to provide a stastically meaningful sample.
The best designers do not account for the success of Game as Product, but the success of the Game as Art. This is far harder to measure than sales figures, but a good statistician could probably make a go of it by tracking the games which have the longest lasting fan communities, fan Wikis, and which are mentioned the most often in retrospectives of the Best Games of All Time, or in lists of games which were most influential on other developers and gamers who grew up to become developers. You might be surprised to find that the correlation between the greatest art and the highest sales is NOT one to one.
Further some game designer seem to do not understand their role within a production. Yes, they have ideas, but others do as well. You don't need to be a designer to have ideas. Your job is it to make a game mechanic out of it, and that's the challenge that only a designer is capable of. To write explanations, formulas and schematics so that every other person involved with it will understand and be able to create what was intended by the idea.
One could have the greatest game design of all times, but if you can't manage a team in order to ship it it's no use. And that's what producers are for, should you determine success of a game by its profit. In the end we're not doing this for the sake of universal peace, but to pay our rents and bills along the way.