 |
 |
|
If you enjoy reading this site, you might also want to check out these Think Services sites:
 Game Career Guide (for student game developers.)
 Indie Games (for independent game players/developers.)
 Finger Gaming (news, reviews, and analysis on iPhone and iPod Touch games.)
 GamerBytes (for the latest console digital download news.)
 Worlds In Motion (discussing the business of online worlds.)
 Game Set Watch (the Group's alt.game weblog.) |
|
|
|
| |
Opinion: 'Video Game Review Scores: Pointless or Pertinent?'
by Simon Parkin
|
|
| |
|
June 6, 2008
|
| |
[In this pointed opinion piece, British games journalist and producer Simon Parkin looks critically at how the media rates and ranks games, suggesting that review scoring acts as more of an impediment than an advantage.]
Last month a British games journalist reviewed Xbox Live Arcade’s Penny Arcade Adventures for two different publications. In one of the magazines the game scored 4/10 while, in the other it was awarded 68%. While it’s a discrepancy that caused some to raise their eyebrows, most commentators acknowledge that the difference simply reflects each publication’s own particular use of the numerical review scale.
Two weeks later Microsoft announced their plans to remove games with an average Metacritic score of 65% or lower from their XBLA service. If the decision on whether to keep Penny Arcade Adventures on the service were to be based solely on the judgement of this reviewer, its fate would swing on which review was looked at.
While a game’s Metacritic or Gameranking average score has often been used to dictate the size of a development staff’s bonuses, EA’s decision to use numerical scores as the criterion for has elevated the numbers issue a whole new level of consequence.
Some argue that scores represent different things to different publications, one title’s 4/10 being another’s 68%. Others question why, when scores rarely tally with a game’s commercial success, we should use them to make commercial decisions? Always, the question behind the question is: do review scores actually matter and, if so what do they even mean?
At a glance, review scores seem to be the most harmless of things. While good critics will bemoan having to reduce a 1000-word piece of incisive criticism to a number on a 10 point scale (or, um, 19 point scale if you’re GameSpot), to the average consumer they offer a useful shorthand reference point with which to compare different titles and inform buying decisions.
But to fully understand the confusing tangle review scores have landed both reviewers, consumers and the wider industry in, it’s important to understand their origins. Review scores are a system imported from those publications that review and rate consumer products like televisions and toasters. For example, look at this review of the Canon EOS400D camera. It’s 25 pages long and is the most objective dissection of this model of camera as it is possible to create.
Every aspect to the product is pulled apart, rated and weighed with statistical graphs and comparative data. By the end of the review you know every single detail about the camera and how it empirically compares to its rivals.
It’s a huge exercise in absolute objectivity and, at the end of the gigantic review the author sums up the good points and the bad points and there is no shadow of a doubt that everything said is ‘factually correct’.
Additionally, there is a place on a defined scale of quality upon which the product sits at that moment in time. It compares to other cameras on the market in defined ways, despite being a complex product. Using the review data it would be possible to arrange all of the digital cameras into a ‘truth’ line of quality, with the ‘best’ camera sitting at 100 and the ‘worst’ at 1 and to place this camera somewhere along that line, thus communicating to a consumer its relative and inherent qualities in a single representative digit.
It seems sensible then to believe that such an exercise could be applied to video games to construct a similar scale of quality. Indeed, this is exactly what many video game consumers want from their reviews.
The average reader (even if they don’t know it) is after a complete objective, scientific comparison between game x and game y with data and statistics and, finally, a numerical point on a linear scale by which they can compare, for example, Mass Effect with Rock Band and see which one is empirically better.
Except, of course, video games don’t work in the same way as toasters or digital cameras. Sure, they have mathematical elements and measurable mechanics and it’s possible to compare the number of polygons between this one and that and spin out ten thousand graphs detailing how two specimens compare. But, unlike with the Canon EOS400D, I would have no idea at the end of those 25 pages which game was better or where they would sit on the ‘true’ scale of quality.
Games are experiential and it is impossible to be wholly empirical or objective about them. Game reviewers instead present their experience of the game with, hopefully, lots of reference points and their weight of knowledge behind them. They might make empirical comparisons between game x and game y’s framerates but they will also argue whether they think this in any way effects the experience for better (in the case of bullet hell shooters such as DoDonPachi) or for worse. They have to argue their points because there isn’t data on the overall, indefinable quality of a game.
In the early days of magazine publishing, video game reviewers would often break a game down into all of its constituent parts (graphics, sound, ‘lastability’ etc), score each on a comparative line of quality and then present the average of those scores as the game’s overall measure of quality.
However, this approach presumes that it’s possible to put each of a game’s constituent parts on a definable scale of quality. The truth is that gauging a game’s graphical appeal is a subjective pursuit in the same way that trying to comparatively score a Monet against a Picasso would be. Call of Duty 4’s competent stab at sunset-drenched realism has a certain appeal, but then so does the 8-bit elegance of a Chuckie Egg or Geometry Wars.
Secondly, games are more than the sum of their parts. You could have a visually astounding videogame with a gut-wrenching soundtrack and astute, nuanced voice acting and it could still be terrible to play and vice versa. Aggregating scores from extrapolated game elements tells you nothing anyone would actually want to know about the game.
At this point, defendants of the review score will offer: ‘Why not just review the game on how fun it is, then?'
The problem with wanting a purely objective ‘review’ of a video game is made doubly complicated by the fact that a video game’s purpose is never so narrow nor so easily defined. Consumer goods have a very clearly defined job to do. A digital camera is there to take the best possible photographs, a toaster is there to make toast to whatever specification the consumer requires in the shortest and most efficient timescale. And because their purpose is tight and the measure of the product’s success easily calculable, they lend themselves to ‘review’ and ’score’ testing.
In contrast, the purpose of a video game is much less narrowly defined. Most game ‘reviewers’ would say that the purpose of a game is to be fun and to entertain. But actually pinning down such abstract concepts is tricky as there are as many criteria and understandings of what is entertaining and fun as there are humans. Thus, reviewing a video game in the same way as you’d review a digital camera or other similar consumer product is inappropriate or, at very least, misleading.
All this is not to say that review scores are entirely meaningless or misleading. In fact, they do have a very clearly defined purpose; it’s just that it’s a different purpose to the one that’s widely understood.
Scores have come to represent whether a game over achieves or underachieves on the preview hype that was generated by the publication ahead of its release. As previews in the average video game magazine are so heavily influenced by advertisers (after all, a preview is offering no judgment on the quality of a game, so a magazine/website can print riotously positive spin in it and maintain clear conscience) this weighting of preview coverage sets imbalanced expectations in readers.
Rather than focusing on the most interesting, promising or innovative games coming out, readers are made to get excited about those whose publishers pay the most for, be it directly through advertising or indirectly through the general marketing promotion of a title.
This is why when a game like Koei’s Bladestorm gets 8/10 in some publications, readerships become incredulous. Their expectations for the game haven’t been set that high because they were being fed hype of a different flavour.
Then, conversely, when Metal Gear Solid 4 scores an 8/10 on Eurogamer last week, the readership revolts the other way - because that’s far below their expectations. Remember: in both cases nobody but the reviewer had played the game at the point the reviews came out - why then were people so quick to damn each respective score (for opposing reasons) if they’ve no hands-on experience?
Scores then become a reference to a game’s preceding hype. An 8/10 for a game that was hugely hyped to hobbyist gamers is a punch in the stomach for excited fans (see the anguish exhibited in the MGS4 comments thread). Conversely, an 8/10 for a game nobody cares about is viewed a gross over-generosity.
And that, is why video game review scores are pointless: they often answer a pertinent question that nobody realised they were asking.
|
| |
|
|
Once again, I have to say we need to pry the reins of creative-decision-making in game development out of the hands of the short-sighted beancounters.
Examples of these were touched on in the article, but there are more, such as whether the game has right amount of puzzles, or the idea that a game has to have some new feature that sets it apart in the genre.
It's ok to have those biases, but there should be some way to identify them, so that the user can make the best-informed decision.
As for the article, I have to agree with this. As much as I like looking over reviews I often find myself tossing aside the offical ones and looking for things written by gamers for free on places like gamefaqs to find out what the game itself is like. The worst case with these reviewers is when they try and review a game but they review it like it's supposed to be something it's not, decrying 'bad points' which really were design choices towards an end. For an example, imagine if you reviewed devil may cry as if it was supposed to be a "flight sim.". Granted an extreme example but you can see where it would cause problems.
Part of the problem I think is that too often people arn't willing to sit and read a review so they can get a real feel for a game, they just want a number so they can run off and decide between game A and game B by that. Sad but true.
Then I wait for a review to come out, if it receives under 7.0, then chances are it's not worth my time, and I move on. If it gets a 7.0 or above, then I read the review and decide whether I'll get it or not. That's how I use reviews, and I'm sure others use them differently, but regardless, they help me make informed decisions.
It always amazes me how close various review sites' scores are to one another. Maybe a point or two off, but I think people have an inherent sense of where a given game stands (in terms of quality) with similar titles. Why do we need reviewers then? Because someone needs to play all the games out there, and it's not going to be me. Reviewers are also paid, not because of their abilities to assign arbitrary numerical values, but because of their abilities to articulate information about a game to someone who has never played it.
Hardly a trivial concern for those developers who are to see their games removed from XBLA on the basis of a mostly meaningless digit, devoid of any of that supplementary articulation...
And I agree with the second because if these scores are so innacurate they should NOT be being used to judge what should and shouldn't be on XBox live.
He's right in that a lot of how games are reviewed now make game scores fairly silly - it's for many reasons more a high score to post on the arcade wall than a particular measure of quality (I in fact, highly enjoyed Bladestorm and enjoy some of the smaller things that other, bigger, titles have frankly not been able to accomplish). More than that, as he also said, as games are about fun but fun is many things to many people (a FPS person is going to have a different opinion of a Sudoku game than they are of the latest FPS), scores can be biased unintentionally because it's merely the wrong audience. How many casual games for instance appear in a gaming review sites or magazines?
However, I think he's wrong in that review scores are pointless. I don't think it's the answer to a question no one was asking. I think it's more that the question itself is wrong to begin with. A game isn't fun/not fun - a game is fun for certain reasons and to certain people. Perhaps instead of asking "Am I having fun?" (or what have you), the better question that reviewers need to asking is "What is this game about, what does it accomplish, and who would I recommend it to?". Get rid of scores and values; really, what is the distinction between a 70 and a 75? 20 and 80, sure, but 70 and 75? Formulai about something subjective? No; instead, perhaps reviewers should simply end a review with a bullet list of Good stuff (it's a Sudoku game with lots of puzzles and a easy to view UI), a list of bad stuff (lack of ability to get new puzzles, UI is pretty but controls poorly), and recommendations (Good fun for the average Sudoku player, probably quickly flown through for a master). And then let the player/reader decide from that if the game is appropriate for them.
Still, although I personally am against scores per se, I am not against a game review/recommendation. Gamers and consumers (or gamers as consumers) often resort to review scores in order to make a decision and whether to buy a game or not. They want to make sure they're paying for something worthwhile and there's nothing wrong with that. Therefore a review should be through and tread carefully when analyzing a game, but to say a first person shooter is 2% better than a racing game tends to just complicate things.
In that sense I agree with Chin. Recommending would probably be easier if pros and cons were taken into account and the fact that different gamers tend to like different genres. There's no point in recommending an excellent FPS to a gamer who will say it's horrible because he's into strategy games.
Not to mention, many can't distinguish personal preference from factual elements.
This is the only sentence that needs to be in this article. It cuts right to the heart of why review scores, when viewed objectively, are so absolutely irrelevant.
But the entire article is so well-written and concise. Thank you for saying what needs to be said.
This single point really drives home the issue with game reviews; scores in particular. Mr. Standley, although I whole-heartedly agree with the statement you have focused in on, as it is precisely why games cannot be reviewed in the same manner as other consumer products, I feel that in the end the quote that I have attached above really pinpoints what reviews achieve (or fail to achieve) in relation to their audience.
I feel that it is important to keep in mind who the audience is for game reviews. These are most likely the same people that are aware of a game's impending release, have been on top of the news and media related to a game's development, have been directly on the receiving end of the 'PR machine,' and ultimately have a level of anticipation and expectations for a title well before any review ever goes to print.
A review, when it comes to a creative, artistic medium such as video games (re-emphasizing the point by Mr. Standley), is essentially an opinion piece, however a well informed one. I for one am glad that there are people out there being paid to devote their time to a vast array of gaming content and writing about it. I enjoy reviews because I enjoy video games and as a result enjoy hearing the perspectives of others. If I'm trying to decide upon a film to watch, a book to read, or a new album to pick up, I get in touch with friends who are actively seeking out and have an appreciation for a variety of content. Games are no different. I look forward to reading the opinions of others who are as passionate about an artistic/entertainment medium as I am, especially if those people have a great deal of comparative experience to draw from.
The issue with ratings affecting salaries of development teams, corporate decision making, etc., . . . that is a whole other can of worms. I have a feeling that in time those issues will work themselves out appropriately, and as in other creative industries, dollars and sales figures will by far be the loudest voice. Sadly, these numbers don't often accurately reflect creativity or originality either, but I don't see the laws of economics and business changing overnight.
Nice article. More awareness brings change. Thanks Simon!
An example would be the Times Online review for 'Shadow of the Colossus' (the lowest score), and its complaint that the game has nothing to do in the vast landscape except interact with giant monsters. Now call me crazy, but the point of that game was to create a very lonely and sad atmosphere to tell a fairly dark story. Am I supposed to knock a game that's designed to create an emotion in me just because I don't like that emotion? What if a gamer would've perked up at someone saying a game was emotionally complex and not just like Zelda?
With metacritic becoming a basis for developer bonus pay-outs and XBLA status, these huge scoring variations need some kind of reconciliation.
"And gaming if you hadn't noticed IS an extreme commercial success."
Game companies don't release "gaming". When you go to the game store you don't buy a little bit of "gaming". What I'm talking about are specific games. Say Film X tanked when it came out, but years later made money. I wouldn't say "films" tanked... Same with games... um... game titles. You'd think if the game industry more diligently built up libraries - instead of just dumping them after the initial release - they would make more money in the long run. There's no reason why games should go to the bargain bin so fast. It's just an assumption they should. Certainly movie companies aren't so stupid they dump their films into a bargain bin 2 years after they're released. If you measure them as pieces of software, maybe they become obsolete. But as a piece of entertainment, they still have value.
Mm... in my opinion, the review score by itself doesn't influence my purchase decisions. I base my purchase decisions according to the genre of the game, the publisher, and if the game is part of one of my favorite series. The written review generally explains all of that. So, if a game that's part of my favorite series gets a mediocre review score, I would buy it anyways. I generally don't consider purchasing games that don't seem appealing, based on their genre, even when they get perfect review scores. Gamers all have their own tastes, and I look for games that I know will meet my desires.