Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 31, 2014
arrowPress Releases
October 31, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


Opinion: 'Video Game Review Scores: Pointless or Pertinent?'
Opinion: 'Video Game Review Scores: Pointless or Pertinent?' Exclusive
June 6, 2008 | By Simon Parkin




[In this pointed opinion piece, British games journalist and producer Simon Parkin looks critically at how the media rates and ranks games, suggesting that review scoring acts as more of an impediment than an advantage.]

Last month a British games journalist reviewed Xbox Live Arcade’s Penny Arcade Adventures for two different publications. In one of the magazines the game scored 4/10 while, in the other it was awarded 68%. While it’s a discrepancy that caused some to raise their eyebrows, most commentators acknowledge that the difference simply reflects each publication’s own particular use of the numerical review scale.

Two weeks later Microsoft announced their plans to remove games with an average Metacritic score of 65% or lower from their XBLA service. If the decision on whether to keep Penny Arcade Adventures on the service were to be based solely on the judgement of this reviewer, its fate would swing on which review was looked at.

While a game’s Metacritic or Gameranking average score has often been used to dictate the size of a development staff’s bonuses, EA’s decision to use numerical scores as the criterion for has elevated the numbers issue a whole new level of consequence.

Some argue that scores represent different things to different publications, one title’s 4/10 being another’s 68%. Others question why, when scores rarely tally with a game’s commercial success, we should use them to make commercial decisions? Always, the question behind the question is: do review scores actually matter and, if so what do they even mean?

At a glance, review scores seem to be the most harmless of things. While good critics will bemoan having to reduce a 1000-word piece of incisive criticism to a number on a 10 point scale (or, um, 19 point scale if you’re GameSpot), to the average consumer they offer a useful shorthand reference point with which to compare different titles and inform buying decisions.

But to fully understand the confusing tangle review scores have landed both reviewers, consumers and the wider industry in, it’s important to understand their origins. Review scores are a system imported from those publications that review and rate consumer products like televisions and toasters. For example, look at this review of the Canon EOS400D camera. It’s 25 pages long and is the most objective dissection of this model of camera as it is possible to create.

Every aspect to the product is pulled apart, rated and weighed with statistical graphs and comparative data. By the end of the review you know every single detail about the camera and how it empirically compares to its rivals.

It’s a huge exercise in absolute objectivity and, at the end of the gigantic review the author sums up the good points and the bad points and there is no shadow of a doubt that everything said is ‘factually correct’.

Additionally, there is a place on a defined scale of quality upon which the product sits at that moment in time. It compares to other cameras on the market in defined ways, despite being a complex product. Using the review data it would be possible to arrange all of the digital cameras into a ‘truth’ line of quality, with the ‘best’ camera sitting at 100 and the ‘worst’ at 1 and to place this camera somewhere along that line, thus communicating to a consumer its relative and inherent qualities in a single representative digit.

It seems sensible then to believe that such an exercise could be applied to video games to construct a similar scale of quality. Indeed, this is exactly what many video game consumers want from their reviews.

The average reader (even if they don’t know it) is after a complete objective, scientific comparison between game x and game y with data and statistics and, finally, a numerical point on a linear scale by which they can compare, for example, Mass Effect with Rock Band and see which one is empirically better.

Except, of course, video games don’t work in the same way as toasters or digital cameras. Sure, they have mathematical elements and measurable mechanics and it’s possible to compare the number of polygons between this one and that and spin out ten thousand graphs detailing how two specimens compare. But, unlike with the Canon EOS400D, I would have no idea at the end of those 25 pages which game was better or where they would sit on the ‘true’ scale of quality.

Games are experiential and it is impossible to be wholly empirical or objective about them. Game reviewers instead present their experience of the game with, hopefully, lots of reference points and their weight of knowledge behind them. They might make empirical comparisons between game x and game y’s framerates but they will also argue whether they think this in any way effects the experience for better (in the case of bullet hell shooters such as DoDonPachi) or for worse. They have to argue their points because there isn’t data on the overall, indefinable quality of a game.

In the early days of magazine publishing, video game reviewers would often break a game down into all of its constituent parts (graphics, sound, ‘lastability’ etc), score each on a comparative line of quality and then present the average of those scores as the game’s overall measure of quality.

However, this approach presumes that it’s possible to put each of a game’s constituent parts on a definable scale of quality. The truth is that gauging a game’s graphical appeal is a subjective pursuit in the same way that trying to comparatively score a Monet against a Picasso would be. Call of Duty 4’s competent stab at sunset-drenched realism has a certain appeal, but then so does the 8-bit elegance of a Chuckie Egg or Geometry Wars.

Secondly, games are more than the sum of their parts. You could have a visually astounding videogame with a gut-wrenching soundtrack and astute, nuanced voice acting and it could still be terrible to play and vice versa. Aggregating scores from extrapolated game elements tells you nothing anyone would actually want to know about the game.

At this point, defendants of the review score will offer: ‘Why not just review the game on how fun it is, then?'

The problem with wanting a purely objective ‘review’ of a video game is made doubly complicated by the fact that a video game’s purpose is never so narrow nor so easily defined. Consumer goods have a very clearly defined job to do. A digital camera is there to take the best possible photographs, a toaster is there to make toast to whatever specification the consumer requires in the shortest and most efficient timescale. And because their purpose is tight and the measure of the product’s success easily calculable, they lend themselves to ‘review’ and ’score’ testing.

In contrast, the purpose of a video game is much less narrowly defined. Most game ‘reviewers’ would say that the purpose of a game is to be fun and to entertain. But actually pinning down such abstract concepts is tricky as there are as many criteria and understandings of what is entertaining and fun as there are humans. Thus, reviewing a video game in the same way as you’d review a digital camera or other similar consumer product is inappropriate or, at very least, misleading.

All this is not to say that review scores are entirely meaningless or misleading. In fact, they do have a very clearly defined purpose; it’s just that it’s a different purpose to the one that’s widely understood.

Scores have come to represent whether a game over achieves or underachieves on the preview hype that was generated by the publication ahead of its release. As previews in the average video game magazine are so heavily influenced by advertisers (after all, a preview is offering no judgment on the quality of a game, so a magazine/website can print riotously positive spin in it and maintain clear conscience) this weighting of preview coverage sets imbalanced expectations in readers.

Rather than focusing on the most interesting, promising or innovative games coming out, readers are made to get excited about those whose publishers pay the most for, be it directly through advertising or indirectly through the general marketing promotion of a title.

This is why when a game like Koei’s Bladestorm gets 8/10 in some publications, readerships become incredulous. Their expectations for the game haven’t been set that high because they were being fed hype of a different flavour.

Then, conversely, when Metal Gear Solid 4 scores an 8/10 on Eurogamer last week, the readership revolts the other way - because that’s far below their expectations. Remember: in both cases nobody but the reviewer had played the game at the point the reviews came out - why then were people so quick to damn each respective score (for opposing reasons) if they’ve no hands-on experience?

Scores then become a reference to a game’s preceding hype. An 8/10 for a game that was hugely hyped to hobbyist gamers is a punch in the stomach for excited fans (see the anguish exhibited in the MGS4 comments thread). Conversely, an 8/10 for a game nobody cares about is viewed a gross over-generosity.

And that, is why video game review scores are pointless: they often answer a pertinent question that nobody realised they were asking.


Related Jobs

Activision Publishing
Activision Publishing — Santa Monica, California, United States
[10.31.14]

Tools Programmer-Central Team
Vicarious Visions / Activision
Vicarious Visions / Activision — Albany, New York, United States
[10.31.14]

VFX Artist-Vicarious Visions
Magic Leap, Inc.
Magic Leap, Inc. — Wellington, New Zealand
[10.30.14]

Level Designer
Magic Leap, Inc.
Magic Leap, Inc. — Wellington, New Zealand
[10.30.14]

Lead Game Designer










Comments


Anonymous
profile image
Totally agree that there's something funky in the water when it comes to reviews. While I think a score can give a user a pretty good idea of how well a game can be enjoyed, there are biases in reviewers that affect scores but aren't necessarily shared by users.



Examples of these were touched on in the article, but there are more, such as whether the game has right amount of puzzles, or the idea that a game has to have some new feature that sets it apart in the genre.



It's ok to have those biases, but there should be some way to identify them, so that the user can make the best-informed decision.

Anonymous
profile image
Grassroots, do you even READ the articles before you post? This is about how reviewing is a tough job and assigning a number to a complex opinion on a subject like this is hard. And gaming if you hadn't noticed IS an extreme commercial success. If you're not going to bother reading the article and making an on topic post please just don't post at all alright?



As for the article, I have to agree with this. As much as I like looking over reviews I often find myself tossing aside the offical ones and looking for things written by gamers for free on places like gamefaqs to find out what the game itself is like. The worst case with these reviewers is when they try and review a game but they review it like it's supposed to be something it's not, decrying 'bad points' which really were design choices towards an end. For an example, imagine if you reviewed devil may cry as if it was supposed to be a "flight sim.". Granted an extreme example but you can see where it would cause problems.



Part of the problem I think is that too often people arn't willing to sit and read a review so they can get a real feel for a game, they just want a number so they can run off and decide between game A and game B by that. Sad but true.

Anonymous
profile image
Classic case of Gamasutra running an impassioned opinion piece about an ultimately trivial concern. Look, I wouldn't say review scores are pointless at all. Game journalists are paid to play tons of games and tell me about them so I don't have to. Everyone has different ways by which they approach a review and its score (just as every reviewer approaches the reviewing process differently). For me, when I go to say, Gamespot, I keep tabs on games I've heard a lot of hype about and become interested in, and I also look out for games that float under the radar but look intriguing.



Then I wait for a review to come out, if it receives under 7.0, then chances are it's not worth my time, and I move on. If it gets a 7.0 or above, then I read the review and decide whether I'll get it or not. That's how I use reviews, and I'm sure others use them differently, but regardless, they help me make informed decisions.



It always amazes me how close various review sites' scores are to one another. Maybe a point or two off, but I think people have an inherent sense of where a given game stands (in terms of quality) with similar titles. Why do we need reviewers then? Because someone needs to play all the games out there, and it's not going to be me. Reviewers are also paid, not because of their abilities to assign arbitrary numerical values, but because of their abilities to articulate information about a game to someone who has never played it.

Anonymous
profile image
'Classic case of Gamasutra running an impassioned opinion piece about an ultimately trivial concern.'



Hardly a trivial concern for those developers who are to see their games removed from XBLA on the basis of a mostly meaningless digit, devoid of any of that supplementary articulation...

Anonymous
profile image
I have to agree with both 1:54 and 2:37. The first anon because he's right about it shouldn't have much weight and that reading a review is what should make or break a game for you if you're not planning to give it a try first.



And I agree with the second because if these scores are so innacurate they should NOT be being used to judge what should and shouldn't be on XBox live.

Stephen Chin
profile image
I think the piece is both right and wrong. What do I mean?



He's right in that a lot of how games are reviewed now make game scores fairly silly - it's for many reasons more a high score to post on the arcade wall than a particular measure of quality (I in fact, highly enjoyed Bladestorm and enjoy some of the smaller things that other, bigger, titles have frankly not been able to accomplish). More than that, as he also said, as games are about fun but fun is many things to many people (a FPS person is going to have a different opinion of a Sudoku game than they are of the latest FPS), scores can be biased unintentionally because it's merely the wrong audience. How many casual games for instance appear in a gaming review sites or magazines?



However, I think he's wrong in that review scores are pointless. I don't think it's the answer to a question no one was asking. I think it's more that the question itself is wrong to begin with. A game isn't fun/not fun - a game is fun for certain reasons and to certain people. Perhaps instead of asking "Am I having fun?" (or what have you), the better question that reviewers need to asking is "What is this game about, what does it accomplish, and who would I recommend it to?". Get rid of scores and values; really, what is the distinction between a 70 and a 75? 20 and 80, sure, but 70 and 75? Formulai about something subjective? No; instead, perhaps reviewers should simply end a review with a bullet list of Good stuff (it's a Sudoku game with lots of puzzles and a easy to view UI), a list of bad stuff (lack of ability to get new puzzles, UI is pretty but controls poorly), and recommendations (Good fun for the average Sudoku player, probably quickly flown through for a master). And then let the player/reader decide from that if the game is appropriate for them.

Arthur Protasio
profile image
It's hard enough to write objectively about something subjective in so many ways. Scores tend to compare games in a way they shouldn't because many aren't based on the same elements, not to mention quantifying that on a scale.



Still, although I personally am against scores per se, I am not against a game review/recommendation. Gamers and consumers (or gamers as consumers) often resort to review scores in order to make a decision and whether to buy a game or not. They want to make sure they're paying for something worthwhile and there's nothing wrong with that. Therefore a review should be through and tread carefully when analyzing a game, but to say a first person shooter is 2% better than a racing game tends to just complicate things.



In that sense I agree with Chin. Recommending would probably be easier if pros and cons were taken into account and the fact that different gamers tend to like different genres. There's no point in recommending an excellent FPS to a gamer who will say it's horrible because he's into strategy games.



Not to mention, many can't distinguish personal preference from factual elements.

Marc Standley
profile image
"Games are experiential and it is impossible to be wholly empirical or objective about them."



This is the only sentence that needs to be in this article. It cuts right to the heart of why review scores, when viewed objectively, are so absolutely irrelevant.



But the entire article is so well-written and concise. Thank you for saying what needs to be said.

Kirk Battle
profile image
The concern with scores, as the article notes, is that they're all arbitrary and undefined (or wildly varying in terms of gauge).



An example would be the Times Online review for 'Shadow of the Colossus' (the lowest score), and its complaint that the game has nothing to do in the vast landscape except interact with giant monsters. Now call me crazy, but the point of that game was to create a very lonely and sad atmosphere to tell a fairly dark story. Am I supposed to knock a game that's designed to create an emotion in me just because I don't like that emotion? What if a gamer would've perked up at someone saying a game was emotionally complex and not just like Zelda?



With metacritic becoming a basis for developer bonus pay-outs and XBLA status, these huge scoring variations need some kind of reconciliation.

Richard Strother
profile image
We are, by and large intelligent people... we can generally assess and even give a "score" to any number of things. Though it is true that games are experiential they can also be qualified rather than quantified. I don't think any game should be given a single number, nor should movies, or any other media that is highly subjective. I do however support the idea of rating a game or movie by rating the various parts and certainly with pros and cons to qualify any numerical data. Also any number rating should be about one thing... Quality! How good are the graphics? Whether they're 8bit or photo-realistic or even how they compare to other games released that week/year do they work for THAT game? The graphic style in MySims is cute and certainly adds greatly to that game but such a graphic style in say HALO (while funny to ponder) just wouldn't add to a good experience. Does the sound convey the mood or is the background music just droning on and making you crave an aspirin? To attempt to be objective about the quality of the elements of a single game would make a numerical review a little more valid; but then it's a standard and I'm sure we all know how easy it is to get an established standard in place!

Tawna Evans
profile image
*Yawn* All that reading hurts my eyes! Couldn't the author find a way to say more with fewer words?



Mm... in my opinion, the review score by itself doesn't influence my purchase decisions. I base my purchase decisions according to the genre of the game, the publisher, and if the game is part of one of my favorite series. The written review generally explains all of that. So, if a game that's part of my favorite series gets a mediocre review score, I would buy it anyways. I generally don't consider purchasing games that don't seem appealing, based on their genre, even when they get perfect review scores. Gamers all have their own tastes, and I look for games that I know will meet my desires.


none
 
Comment: