Gamasutra: The Art & Business of Making Gamesspacer
The Problem with Game Reviews (And Why Games Are Like Restaurants)
Printer-Friendly VersionPrinter-Friendly Version
View All     RSS
April 16, 2014
arrowPress Releases
April 16, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM TechWeb sites:


 
The Problem with Game Reviews (And Why Games Are Like Restaurants)
by Ben Serviss on 05/02/13 09:15:00 am   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

This article originally appeared on dashjump.com.

Restaurant Empire 2
Screenshot from Restaurant Empire 2.

What’s the one thing missing from every book review that’s ever been written? The number at the end. Isn’t it interesting how books, an artistic medium thousands of years old, has resisted the kind of numerical classification that has invaded film, food, games, and even prospective romantic partners within the last century?

While the widely used star system may have been popularized in the late 1920s to offer a quick way to summarize lengthy film reviews, its ubiquity in film has long since been adopted by professional game reviewers, most often in the form of numerical scores. Yet the more you look at the usage of these grading systems for the unique medium of interactive games, the stranger the fit appears.Yet the more you look at the usage of these grading systems for the unique medium of interactive games, the stranger the fit appears.

Start with film. Since films are linear and take approximately the same amount of time to complete, the criteria seems reasonable, with the question coming down to “Is this film that is roughly 90-120 minutes as good as this other film, which is also roughly 90-120 minutes?” Books arguably escape this rationale due to the wide range in pages and time needed to finish reading, especially since people read at different speeds. Now you have a variable in how the media is consumed that isn’t present for film, which complicates the simple comparison afforded by film reviews. Variables like this complicate assigning a subjective score based on a uniform experience.

It gets more confusing when you look at food reviews, specifically restaurants. Now, instead of looking purely at the experience of eating, you’re rating the entire experience of all of the things you could possibly eat, as well as the restaurant’s ambience, furnishings, layout, cleanliness, friendliness of the waitstaff, skill of the kitchen staff that night, etc.

Restaurant reviews still often boil down to a blunt number or star ranking, but the shift from the direct linear experience of film and books to the ‘big picture’ composition of the atmosphere and the food quality is a significant difference.

Then you have games.

Game and Restaurant Menu Comparison
Left: The menu for Porsena in New York. Right: The menu for Halo 4.

When you are first seated at a restaurant, what do you see? When you first start up a game, what do you see? That’s right, the menu. Far from linguistic serendipity, this is the first indication that games have more in common with restaurants than they do with traditional linear media.

In a restaurant, you study the menu in preparation for your few key choices – the meal decisions that will drive your experience. In a game, the menu screen is a launchpad for whatever type of experience you’re looking for at the moment (depending on the game, of course) – single-player story, multiplayer, challenge mode, coop, etc.

While this choice is much less permanent in games, both require the same initial input used to begin. It goes deeper – once you’re in the game, every action you take is essentially another, smaller, menu choice of all of the in-game options available to you. With every decision, a unique experience unfolds that reflects the aggregate choices of the designers and the mindset of the player at that given moment in time.

In the case of Halo 4, as you decide to rush, snipe, or grenade your foes, the game will 'order up' the experience you've decided on (melee battle/long-range shootout/evasive maneuvers) and serve you the corresponding simulation. Because you make so many of these choices as you sculpt your particular experience, their significance is diluted on average, leading to a sustained sense of control and lengthened immersion.

Compare this to a restaurant, where your choices are few, but have dramatic effects on your enjoyment. Better yet, look at buffets, where even more agency is given over to the participant to guide the experience (sandbox games) in exchange for giving up the authored experience of a professionally prepared meal (story-based linear games).

Looking at these linear and interactive mediums, the difference can be summed up easily:

In film, literature, and linear recorded media, you will have the same experience every time.

In games, dining, and interactive experiences where the participant has agency, while it’s possible to have the same experience every time, not only are the odds against it but the nature of the media discourages it.

And that’s where the legacy review systems show their limitations. If games are experiences that so depend on agency for their end results, relying on a reviewer to quantify their particular experience in a hard number is bound to be problematic. Remind me, what exactly is the difference between a game that gets a 6.8 and one that earns a 7.2?

Instead, looking at the reviewer’s particular experience as an indicator of all possible experiences that could be had with the game leads to a much more reasonable method. Building off of experiments to find a better way to phrase this, like defunct magazine Game Buyer’s Rent or Buy rankings, Kotaku’s rating system has come the closest to putting this into practice with a simple “Should you play it? Yes/No” question, with follow-up reasons to support the verdict.

Is it a perfect system? Probably not. Yet in the face of the complexity afforded by interactive experiences, sometimes the simpler ways are the best.

Ben Serviss is a freelance game designer working in commercial, social, educational and indie games. Follow him on Twitter at @benserviss.


Related Jobs

Giant Sparrow
Giant Sparrow — Santa Monica, California, United States
[04.15.14]

Game Designer
Nintendo of America Inc.
Nintendo of America Inc. — Redmond, Washington, United States
[04.15.14]

Associate Software Support Engineer
Nintendo of America Inc.
Nintendo of America Inc. — Redmond, Washington, United States
[04.15.14]

Software Support Engineer
Nintendo of America Inc.
Nintendo of America Inc. — Redmond, Washington, United States
[04.15.14]

Software Support Engineer






Comments


Jeremy Reaban
profile image
I think you are overstating the effects of player agency. Just how different is an FPS based on how you play it? Not a lot. But even conceding it is true, how many sandbox games are there out there? Not a whole lot. Games are often good at giving the illusion of choice, but in reality, there is very little.

Beyond that, I think the numerical rating makes more sense when you put it into context. In the US, it was pretty common to use a letter based grading system in school. A, B, C, D, and F. A is outstanding, B is good, C average, D is barely acceptable, and F is failure. With +s and-s added to each.

Of course the catch is, what does that map to in terms of numbers? Traditionally a 72 would be a C-, while a 68 would be a D+.

The problem with Kotaku's system is that in the real world, it's not a question of playing it, so much as a matter of buying it. Ultimately the reviewer is looking out for the reader's pocketbook.

In my experience, there are very few games (or books or movies or even songs) that aren't worth playing at least once. But are they worth buying? That's where a more granular scale is useful.

Evan Aoki
profile image
Games are only truly linear in terms of the stories they are telling, whether it be a Mass Effect or Skyrim - everyone is working towards ultimate victory. (No game have ever concluded with your failure.)

But before we reach the end, we have the experience of the game. Interacting with it: navigating it, understanding it, messing with it's systems. For example, as a player in an FPS I could locate a great corner or perch in a difficult level with tons of ammunition. A place where I could mow down enemies because I took the time to explore a level during some downtime. Or alternatively, I could've taken the most treaded path and had a more difficult time. Now, granted that isn't a ton of player agency but it can change the game for some people like myself.

In an FPS, sure the controls are the same, the story is the same, but how did (emphasis) I (emphasis) feel playing that game. The answer to that is going to change person to person. Sure you could probably get a glowing consensus on a popular FPS but why does that mean that everyone, everywhere will like it? I know for me, with a game like Halo 4, I couldn't bring myself to finish it. My interest had dropped significantly for a game in that setting ever since Halo Reach. But the reviews for Halo 4 were fairly positive. So how would these reviews even come close to reflecting my interest for that style of game? They can't unless they came up with something less arbitrary.

And if we move away from the FPS as an example - there are just so many types of games out there and equally so many types of gamers. How do we mitigate the decimal point review score? How do we make sure the point of the review is actually understood for (emphasis) all (emphasis) gamers?

Also, I don't feel as though your point about buying games versus playing games is all that accurate. Don't get me wrong, I do believe that it is in a way helpful to the reader for now. But I feel as though you're missing the point. If their reviews tell someone that Game X is good then they know that when they can buy it, they will buy it. Giving a game an arbitrary score from 10 - 100 doesn't help a person make up their mind but provides a basis of inaccurate comparison against another game.

Kotaku's system is less arbitrary like Ben says but it's still cloudy.

Now, this is just a thought, but maybe reviews based on genre would help things.

Andreas Ahlborn
profile image
"Far from linguistic serendipity, this is the first indication that games have more in common with restaurants than they do with traditional linear media."

You make a rather "forced" analogy with games and restaurants and simply ignore that restaurants are getting "numerically" rated all the time, ( http://en.wikipedia.org/wiki/Michelin_Guide), so no I´m not convinced a simple Yes/No , guity/innocent will do.

Eric Schwarz
profile image
Good thing I write for a site that doesn't do review scores.

I personally think giving scores for games is broken, anachronistic and completely unnecessary. It's not so much a legacy as it is for an easy way for publishers to bribe and coerce publications into doing free marketing for them, and a way that they can push developers harder to add features to their games that will get high scores rather than improve the game itself.

The experience of playing a game can not be so easily summed up in a score like 4/5. Every single game has positives and negatives, and it's very common to come across games that have exceptional presentation, polish and tons of work put into them yet are boring in the extreme (Call of Duty, Battlefield, etc.), as well as games that are broken messes that don't look very good, yet have strong core mechanics that make them enjoyable in spite of problems (STALKER). Where do you draw the line? Do you say "This game gets a 10/10 for gameplay but a 5/10 for graphics, so its final score is 7/10"? That's how it seems to work in most of the reviews industry (and I do use that term intentionally), and it's every bit as stupid, pointless and arbitrary as it sounds.

It astounds me how so many games with such terrible ideas can consistently get high scores. Skyrim has literally the worst user interface in any game I have ever played, and it represents a major mechanics regression from past Elder Scrolls games, yet it gets near-perfect 10s across the board because it has slightly better graphics? Epic keeps churning out the same Gears of War every year with only minor changes to gameplay and they still top the charts, even though the cover shooting mechanic has long grown stale? If you went to a McDonald's and you found that it was the best McDonald's you'd ever been to... does that mean it deserves a five-star rating? It's still a McDonald's!

Review scores don't help the games industry. They reward big developers for playing it safe and they punish smaller developers for trying new ideas.

Evan Aoki
profile image
Agreed but as damning as a review score can be it's the perfect marketing fodder. Just look at those Bioshock Infinite ads.

Emppu Nurminen
profile image
"In film, literature, and linear recorded media, you will have the same experience every time."

Yet most of the appreciated pieces of stagnant medium (literature, comics, films, art even) are raved simply because they bring different experience each time you consume it. They rely the consumer's gained life experience to make it different. Surely, they aren't designed that way, but that's what makes or brakes the good novels from rubbish ones. How many games can do that? How many games do tap the consumer's gained life experience to make the game experience different?

Ben Serviss
profile image
I was waiting for someone to make this comment :)

You're totally right, of course - but while the authored experiences in these linear media don't change, the reader/user's *own life experience* may have. In the case of games, the default experience is mutable by nature, so there's more inherent variety. But just because games have yet to even grasp the potential of this fact doesn't mean they might discover how to use it in the future.

Alan Youngblood
profile image
A suggestion for improving reviews: Since not all players are in it for the same reason, perhaps have a simple yes/no/definitely buy/play it but also add a category for the game's particular appeal. Base the appeal on the Aesthetics of Play (See Marc LeBlanc's theories on this: http://8kindsoffun.com/).

For instance If we were back in 2005 pitching the first Guitar Hero, we might say it's review bottom line is "For those who dream about the 'fantasy' of being a rock star and enjoy listening to rock music this is a must-buy."

What this does is tell someone that wants to enact the fantasy of being a soldier on the front lines of WWII, wait a year we've got Call of Duty 3 coming and you'll enjoy that.

In a very different example, take Solitaire, the card game. This falls more into the 'submission' or 'abnegation' category. But if we compared it to say, Skyrim, on basis of the primary aesthetic appeals of Skyrim (narrative, Tolkienesque fantasy, discovery, expression) we find Solitaire to be one of the worst games ever made. But that is not nearly the case, look up the Angry video game nerd if you don't believe me and see worse games. No, Solitaire is a game that is enjoyed by many for being simple and predictable.

Games could have more than one major aesthetic like the Skyrim example, and players may desire more than one type at different times (over a lifetime, or day-to-day change). Most people will be able to suss out which is a good choice for them at the moment. This is similar to the restaurant analogy. I may really like steak dinners and bacon for breakfast, but I also want to get sushi, pizza, salads, thai, chinese, sandwiches, or other dishes for other meals.

Ron Dippold
profile image
I get your point, but review scores serve the real customers - the publishers. Suits like metrics, even if they're mostly worthless. They have a couple uses:
- Something to base bonuses on.
- Measure of quality (even if wrong, they just want a measure).
- Give and take for 'exclusive' reviews at gaming magazines and big gaming sites. And the magazines/sites then like them because they think they can use them as pitiful leverage.
- If you hit the jackpot of 90+, you can market the hell out of that and do a Game of the Year edition.
- If you don't hit 90+, you can then disingenuously argue the system is flawed and/or the reviewers are to blame.

There's one more group who love review scores. Rabid fans. They'll gloat about a favorable 10 on a platform exclusive or have total meltdowns about 'only' an 8/10 (and enjoy doing it). These are the people most likely to be your regular readers and angry posters (more impressions!).

Again, to use your restaurant example, who do the scores serve? The restaurants have to make crazy efforts to try to keep a 5 star rating (good luck), and the star spread is so wide that they're useless for the customer - a detailed customer review without the score is much more useful. In fact, the most useful thing about review scores is that you can toss the 1 and 5 star reviews out and get a much better picture. So who are they serving? Yelp. It's a calculated way to play the restaurants and the customers against each other for more hits, and then offer their services to the businesses to get their scores up.

I believe your premise, but people have been making the argument against numeric game review scores for at least 20 years, so it might help to think about why it persists so strongly anyhow.

Craudimir Ascorno
profile image
What I see as a problem with game reviews (and restaurant reviews too, for the record), is that while eating and gaming experiences are almost purely subjective, the effectiveness of a review is its objectiveness. However, because it is a subjective experience, it is very hard for the reviewer to leave the subjectiveness out of the review.

Let's take the restaurant example. I like pasta, but what makes a good pasta? Are there universal values of what is a good pasta or is it subjective? What makes someone who prefers a 3-star Michelin Guide restaurant pasta right and someone who prefers the average instant noodles wrong? What makes the Italian Restaurant's next door pasta better or worse than my grandma's pasta? So how can a reviewer prove to me it is worthy to go to a certain restaurant instead of grandma's home when I feel like eating pasta?

The truth is he can't, and the best he can do when evaluating a restaurant is naming the dishes available and maybe talking about the pasta consistence or if the tomato saucy is sweet or spicy, nothing more. However, if the reviewer hates spicy dressings, he will not evaluate well a restaurant where the dressings are spicy, and his review will be misleading for me if I am a spice-addict.

The same goes to games. What makes Call of Duty a better FPS experience than Killzone? There are surely some things that need to be pointed in a review, like movement issues some shooters can have, bad collision detection, etc, but apart from that, it is difficult to say.

Take turn-based RPGs (games that have mostly the same mechanic) and how a reviewer can prove one game is better than the other? A gamer who wants a good RPG doesn't benefit from reviews that state that the reason for buying Ni No Kuni is that it is "well-done, has the traditional RPG mechanics, and provides a charming whismical experience", while another RPG gets "everything works fine, but nothing stands out in this RPG" so it should be avoided. What if the player isn't in love with Studio Ghibli's works?

The same goes for a Super Mario platformer and any other platformer. What if the gamer doesn't have a deep fondness for Super Mario games from the NES and SNES era and doesn't agree with the mantra that "if a Mario game isn't broken, it is perfect" that populates the mentality of many reviewers that were raised with Mario games and always rate average Mario games above the average?

Then again we have reviewers making choices for the gamer based on their own tastes and experiences. It doesn't matter how many "experience" the reviewer has, it is still their taste against the gamer taste. It may happen that a certain reviewer often prefers a game that the majority of the audience for that genre prefers too, but most of the time the reviewer will be doing a marketing job, maybe killing the chances of an excellent platformer, an outstanding FPS, a revolutionary RPG by reassuring the readers that they are better sticking with the obvious Mario, Sonic, Call of Duty, Halo, Final Fantasy and Elder Scrolls games based only on their tastes, preferences, love for the series, etc.

Then we end with a big problem: what makes a worthy review? New Super Mario Bros is a platformer with n different stages, power-ups xxxx, boss fights yyyy, enemies zzzz. Final Fantasy XIII has many cut-scenes, an ATB turn-based battle system with xxxx options, character development through yyyyy and zzzzz, battles wwww, bosses vvvvv, etc. And let the buyer choosse what it thinks better for him, like the guy who loves spicy pasta sauces. The problem with those reviews? They are boring, like deciding if you want to buy a game by a wikipedia article. It won't sell magazines and it won't generate clicks for the sites. It will not create discussions, it will not stir controversies. What is better to the gamer interest is not good for the industry. Not for the publications that will lose part of their appeal, not for the big developers that want the status-quo of big names like GTA, CoD, Halo, Mario, Pokemon, God of War, Assassin's Creed, FIFA, Final Fantasy, etc to sell bazillions without having to provide radically different experiences and getting lots of hype and free-marketing through positive reviews from fan-reviewers (reviewers are humans and gamers). It is part of the business trying to dictate the customers what they should buy instead of letting them decide by their own, and reviews are on the business side of things.

Joe McGinn
profile image
I'd be happy if game reviewers stopped writing 900 words about the frackin' story and 100 about the gameplay. I want to know if a game has a story, and if it's good, and that's IT. Any game review actually starts *describing* what happens in the story I stop reading.


none
 
Comment: