Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
August 23, 2014
arrowPress Releases
August 23, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


Metacritic Removes Individual Developer Scores, Still 'A Work In Progress'
Metacritic Removes Individual Developer Scores, Still 'A Work In Progress'
March 29, 2011 | By Mike Rose

March 29, 2011 | By Mike Rose
Comments
    23 comments
More: Console/PC, Business/Marketing



Following an outcry from the online gaming community, the gaming press and featured developers, review score aggregator Metacritic has removed career scores from the pages of individual developers.

Last week, news emerged that Metacritic had begun offering listings for individual game developers, providing a profile page with a career score for all the work each developer had been credited on.

In the latest post on the Metacritic site, Metacritic games editor Marc Doyle explained that "users’ feedback has indicated" that the latest addition to the scoring site is "a work in progress." As such, individual scores have been removed, although the developer pages are still available.

Doyle said, "As part of our relaunch of Metacritic in August of last year, one of our goals was to make the site much more dynamic and to allow our users to discover new products by exploring other titles by the creative teams behind the movies, games, TV shows, and albums our users enjoy."

"In our games section, we encourage our users to click on the publisher or developer (development company) of a game they enjoy to learn more about other games those companies have produced, the Metascores of those games, and the companies’ overall career scores," he said.

Regarding the individual developer pages, Doyle noted, "In addition to creating dedicated pages for corporate publishers and developers, on a given game’s 'Details & Credits' page, Metacritic displays those individual people who contributed to the games in our database, including designers, programmers, producers, voice actors, and artists."

"In turn, we have produced dedicated pages for those individuals featuring their games and associated Metascores, and, until today, their individual career scores."

Doyle admitted, "Although our credits database (which is powered by our sister site GameFAQs) is growing, as our users’ feedback has indicated, it is a work in progress and is not nearly as comprehensive as it needs to be to accurately provide a career score for these individuals."

He noted that, while career scores have now been removed, Metacritic "are still very much committed to building a credits database, and welcome your participation in that process."


Related Jobs

Raven Software / Activision
Raven Software / Activision — Madison, Wisconsin, United States
[08.23.14]

Sr. Software Engineer (Gameplay)
AtomJack
AtomJack — Seattle, Washington, United States
[08.22.14]

Level Designer
FarSight Studios
FarSight Studios — Big Bear Lake, California, United States
[08.22.14]

Lead Android Engineer
Churchill Navigation
Churchill Navigation — Boulder, Colorado, United States
[08.22.14]

3D Application Programmer










Comments


Rob Wright
profile image
I'm kinda bummed. I was honestly waiting to see what Sid Meier's score was going to be after MetaCritic gave him credit for all the Civilization games after the first one.

Timothy Barton
profile image
I actually think this kind of scoring is a great system. The developers complaining about it are likely the same ones who feel they got worse scores than they deserve (which pretty much everybody will always feel like). The problem with this system is that they indicate some individual power over the project. They can track athlete's statistics because they are easily measurable as an individual contribution. However, many great developers work on crappy games because of other factors such as the team, schedule, etc. An outstanding level designer may get a low score because the combat system was screwed up, and this would unfairly reflect in the developer's score. I fully support development company or other metrics though.

[User Banned]
profile image
This user violated Gamasutra’s Comment Guidelines and has been banned.

Timothy Barton
profile image
I just mean that it is great to associate track scores, almost like statistics in sports. They give a more meaningful tracking over time, so that we can determine who the true hidden gems are. Without this, a company makes one great game and everyone forgets about the million crappy ones they made. The problem is in the definition of the unit. A developer is too specific, I would say it should go more along the lines of the development team, or something like that.

Mathieu MarquisBolduc
profile image
*Assuming that critic scores even make sense in an industry where most reviewers dont have a press card or any credentials at all*:



I believe such a system only works (barely) for people who are accountable for the quality of the game: i.e. the Producer and the Creative Director. The hundreds of other credited people have an equally important, sometimes more important contribution, but the scope of their work is focused, and the score of the game is often unrelated to the quality of their work. Personally I think we're already too obsessed with review scores as it is.

Jacque Choi
profile image
It's a completely silly and inaccurate guage.



As an artist, my work should be judged by my art portfolio.

Producers should be judged on their ability to ship games on time and on budget.

Programmers should be scored on the quality of their code etc.



I could almost see Designers being graded on something like this, but Miyamoto has an 80.



:/

Craig Timpany
profile image
Nope, wouldn't work for designers either. GDDs inevitably get cut to pieces when they hit the difficulties of production. Not all projects are well-resourced enough that the designer's full and unaltered intent makes it into the finished product.

Dan Nikolaides
profile image
The problem with this score is it's too easy to misinterpret it as a developer's "value" or "quality" versus what it is - an average of the games they worked on. There are many developers who are fantastic at what they do, but have worked on many games with poor ratings. It takes a fair amount of luck to be a part of a team that consistently gets 90s. Even if it was 100% accurate, it doesn't adequately represent your impact on the games in question, nor does it truly represent the extent of your ability. I fail to see the value in this.

J D
profile image
Said it better than i could. Its is metacritic distilled to its core and suprise suprise, its flaws are even more glaring in this form.

Dave Dundy
profile image
The system seems to be born of boredom from the site. It doesn't even make sense to "rate a developer" based off of games they've worked on. I've worked on plenty of games where the games were mediocre cause the leadership blew nuts and game did ok. Is that a reflection of my work ethic ? No? So why the hell would they put up a rating in public giving my work ethic a "score". Also, who gives them the right to do that sort of shit? I didn't. Glad they took it down though. It was a pretty stupid concept.

Todd Boyd
profile image
It isn't a "right", or even a privilege. This is the internet, guy. If your information is out there, people can do stuff with it. Period.

Bart Stewart
profile image
An aggregate rating score makes some sense for a development studio or a publisher. Those scores measure perceptions of the whole group that makes the games.



But it doesn't make any sense for individuals unless all their games have been developed, marketed and distributed entirely by that individual. Other than Notch, that probably doesn't apply to many people working on games scored by Metacritic. Team efforts need to be scored as a team.

Paolo Pace
profile image
Anyone who believes this as a fair and accurate method of rating developers deserves a 10/100. I'm being generous here.

Tim Carter
profile image
Also, there is the whole issue of "when am I working on a project I really believe in", and "when am I just taking a gig to get paid". Even Spielberg did Jurassic Park 2 (or whatever) so he could then get a deal to allow him to make Schindler's List.

Benjamin Leggett
profile image
Given that Metacritic has a very limited dataset, a lot of long-time devs had woefully incomplete listings.

Wylie Garvin
profile image
Wow, what a dumb idea by Metacritic. I think its rather offensive to the developers of all these games, to have their entire career reduced to a single meaningless number. I give Metacritic a 13% for this one.



Its bad enough that they summarize the *games* as a single number (games which are often the work product of hundreds of individuals and multiple years of their full-time effort). Doing it to people is just disgusting.



Mind you, Metacritic is not the only site that does stupid things like this. Consider this credits list for Far Cry 2 at MobyGames: http://www.mobygames.com/game/windows/far-cry-2/credits

At the bottom, it has sections for "Other Games" and "Collaborations" where they are apparently attempting to list the most influential developers based on the number of other games they have worked on, or something...



But their results are meaningless, because the top names are all executives of the publisher who get their name in the credits of every game published by that company. They've been credited in 100 or more titles, but that doesn't mean the contribution of a developer who worked full-time on contributions to a handful of great titles over a 10 year period is any less significant.

Noah Falstein
profile image
Whether you believe in the theory of this being a good thing, in practice their information, even on very high profile people, was both incredibly incomplete and often terribly misleading. Given what I saw, if they took the same approach to rating movie people, Steven Spielberg would be rated only on his job as executive producer of the Flintstones movie and directing 1941 (but for the video release only). Maybe he would also get credit for co-writing Big, if they didn't check his sister's first name.

Kim Pittman
profile image
I personally think it is a terrible idea and I am glad they have backed off it. Metacritic in it's self is barely a usable tool to get a possible accurate representation of the overall response to a game. But by taking a single person and averaging the scores of all of their games to say this person is x quality is just absurd in the extreme. For one, many people take jobs where they can. We have all worked on a game that could have been better. In addition, it's not always a person's responsibility as to how a game ends up. A script sample from a shipped game, regardless of the quality of the game can show I am an excellent scripter, regardless of the final game score. Artists have no control over how the game plays, which is generally what determines the score. Designers often don't have control over the high level design, but rather are just implementing. Programmers can't control the art style, etc.



Also anyone who is hiring should also be aware that working on a terrible train wreck of a game can be an invaluable learning experience. I worked on Iron Man 2, a game blasted by critics, but I can point to concrete examples of things I did during development that made that a better game. Does that mean I should get a low score because of a design that was set before I even started working at the company? No. Should I get a low score because I accepted a job, to end my unemployment, that I knew was going to broaden my experience despite not being a 90+ critical game?



I did note that Metacritic also listed ports separately, some of which had terrible scores as compared to the originals, and credited them to the original team, even if that team had nothing to do with the port. And what about companies like Gearbox, who list their entire company in the credits, even if specific people didn't work on that specific game? It's just a bad idea all around and shouldn't be done. Maybe for studios and publishers, but not individuals.

J D
profile image
I like the idea but it seems since its people now instead of companies peoples feelings are getting hurt.



Its no less flawed than standard fare metacritc, which is already deeply flawed, so i say why not go with it.



Metacritic is too pretty/novel of a system to let go of regardless of its faults, even if it is/will be grossly misused.

Ben Droste
profile image
The majority of people who work on a title are not directly responsible for the game. That's the problem. I'm an environment artist, how can you score the quality of the work I do in a very specific roll from the arbitrary average score of all the games I've worked on?

Or what about UI artists, or tools programmers, and so on. Game scores are generally a reflection of the overall quality, influenced heavily by quality of gameplay. It's ridiculous to think you can assign a score to someone when that score is based on any number of factors outside of their influence - I have nothing to do with gameplay design, or AI, or narrative, or any other factor that might influence the average score (for better or for worse) far more than my own contribution.



I was once contracted to work on a title for three months and build two small environments out of a very large game. I'm credited on that title and so its score is now reflective of my work? Can anyone honestly say that makes sense? (And I haven't even said if it was a high or low scoring game, but what difference would that make?)



Studio scores I could see having some merit, since that's more representative of company direction and the talent of the team as a whole. But even then you need to be careful to acknowledge that studios can be made up of multiple teams, and the teams (and studios!) themselves change over time.

Peter Kjaer
profile image
I think another thing that this score doesn't reflect is how you grow as a professional. Chances are you are getting better and better at what you do over the course of your career, and this score just gives you an average of all and in no way reflect where you are currently at.

I think this is a destructive idea for the industry, but it seems like we all agree on that :)

Judy Tyrer
profile image
The next article I want to read is "Metacritic has figured out how to give accurate credits for game developers". Until then, the website has zero credibility.

Christian Allen
profile image
Metacritic can't even figure out that a B- doesn't equal a 65. Look at the inaccuracies on MobyGames (I am somehow listed on a FEAR2 product, but half my games aren't listed), and that is ALL THEY DO. Same with IMDB.



Also, game credits aren't all equal. Some products you contribute to more than others. I'm pretty sure I'm credited in the Xbox and PS2 versions of GRAW, but didn't actually work on them other than to give feedback, and I sure as hell wouldn't want those scores computed in my overall score.



If they moved forward with this, pretty soon employers would be basing pay rates at hire on it.



Bad idea in general.


none
 
Comment: