At the Triangle Game Conference last Wednesday, Julianne Greer, editor of Escapist magazine, moderated a panel discussion titled â€śTeaching to the Test: The Impact of Reviews on Game Developmentâ€ť.Â To explain the title, K12 teachers tend to teach what is on end-of-grade tests to the exclusion of almost everything else.Â The panel considered how much game development studios and publishers create games to meet the â€śtestâ€ť of reviews.
Panelists included Juan Benito (Creative Director, Atomic Games), Eric Peterson (CEO of Vicious Cycle), Dana Cowley (PR Manager, Epic Games), and Shaun McCabe (Production Director, Insomniac Games east coast).
Their answer to the main question was â€śdefinitely not,â€ť though they do pay attention to what individual game fans say on forums and email.Â The only exceptions would be a sequel, especially if another studio did the previous game, or a licensed property, where reviews of past games for that license can give clues to what needs to be changed or added.
Benito saw fan opinion as more "pure from the heart" than the reviews, which led into a discussion of whether reviews are influenced by manufacturers.Â This can be overt, through junkets or other â€śbribesâ€ť, or through the influence of advertising money.Â (Consumer Reports magazine refuses to accept advertising to avoid any appearance of influence by manufacturers.)Â While the panelists had heard of this kind of shady dealing, only one knew of it happening (from his days at Microprose); however, Greer stated that Escapist magazine had received such influence offers (which they rejected).
Yet thereâ€™s a danger of shutting out the non-hardcore audience if you base your decisions solely on feedback from the minority who express opinions online.Â McCabe said listening to players is so important that some companies have changed their marketing department name to something like â€ścommunity relationsâ€ť.
How much do reviews affect sales?Â I was surprised that no one cited any survey, as surely someone has investigated this question; panelists speculated that reviews have a strong influence on hard core players, but virtually no influence on casual (e.g., Wii) games, as those are impulse buys.Â Greer showed slides from research company EEDAR showing that certain categories of games (RPGs, Music & Rhythm, Sports) received consistently higher aggregate review scores than others, with some lagging far behind (Arcade, Skill & Chance, General Ent (such as Wii Fit), and Narrative).Â We have no way to know whether this is a bias from reviewers or an actual difference in game quality, though Iâ€™d suspect itâ€™s because most reviewers are hardcore players.
Another interesting slide compared scores for 360 and Wii versions of the same title; the 360 scores were much higher than Wii for â€śhard coreâ€ť game categories, much lower for â€ścasualâ€ť titles.
Benito looks at the Wii as a "critique-proof platform".Â Another panelist joked that if you put the word â€śPartyâ€ť in a Wii title, it will automatically sell at least 200,000 copies as parents want their children to be playing â€śpartyâ€ť games.
Peterson cited the Wii-exclusive game Madworld as a game that has very good reviews, but poor sales (180,000 according to VG Chartz, only 66,000 in the month of release according to Wikipedia).Â And his company has a childrenâ€™s game with two and a half million sales but reviews in the 40s.
This was part of a discussion of the quality of reviews.Â Panelists clearly did not care for reviews in general, probably because they felt so many were poorly-written and often contained mistakes.Â One panelist specifically referred to the reviews on IGN and Gamespot as â€śwhite noiseâ€ť, and all panelists clearly felt that reviews are often â€śsubjectiveâ€ť rather than â€śobjectiveâ€ť.Â Of course, â€śsubjectiveâ€ť can be just as accurate (in fact, more accurate) than objective, depending on the situation, the problem is that reviewers donâ€™t explain their biases and why they feel as they do, so readers have no basis to judge the opinions.Â
Moreover, with a single numeric rating, reviewers are going to go with their personal preferences, so, for example, a shooter fan reviewing a childrenâ€™s game isnâ€™t likely to give it a high rating (which is likely what happened with the childrenâ€™s game Peterson mentioned).
The discussion was not intended to be a critique of reviews, but Iâ€™d make a number of observations.Â I used to be paid to review board- and RPG-related materials for TSRâ€™sDragon and other magazines 25-30 years ago, but Iâ€™d not write assigned video game reviews, as a proper review requires playing through the game, a much larger time commitment for video games.Â Someone pointed out that film reviewers commit only two or three hours to watching a film, quite a contrast.Â Panelists had seen reviews where they knew the reviewer had barely played the game.Â In fact, this has influenced some companies to make sure the first few minutes of a game are exceptionally engaging, a good idea in general but especially good for the â€śsnap reviewersâ€ť.
Magazine and Web site â€śexclusivesâ€ť tend to be more favorable than reviews, as the writer knows the studio-publisher has done his company a favor by granting the exclusive.Â "Maybe that's why previews are so different from reviews.â€ťÂ This comes back to the nature of the fans, who go to the sites with the â€śnewest newsâ€ť, who want to see the latest artwork for the Zerg (in a recent PC Gamer Magazine) or see the latest screenshots.Â In my opinion this is a major reason why video game magazines are having a difficult time surviving: they canâ€™t be as up-to-the-minute as the Web sites.
The â€ścult of the newâ€ť also tends to drive reviewers to snap decisions and sloppy behavior; if the review comes out too late, itâ€™s no longer â€śnewsâ€ť and is ignored by many.
Some reviewers clearly donâ€™t understand how reviews, of any medium, work.Â They should answer three questions:
Â·Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â what were the creator(s) trying to do
Â·Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â how well did they do it
Â·Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â was it worth doing
To answer these questions they must explain â€śwhyâ€ť, not merely say â€śthis is a piece of junkâ€ť or â€śI donâ€™t like the graphicsâ€ť or â€śwhat a dumb ideaâ€ť.Â But this makes reviewing more difficult, more work.
If the reviewer separates these questions sufficiently, a reader can see that the childrenâ€™s game was well-done, even though the reviewer thought it wasnâ€™t worth doing because he isnâ€™t interested in childrenâ€™s games.Â
One panelist suggested reviewers ought to â€śtake a step backâ€ť and watch others play the game, in order to acquire more than one point of view.Â They also need to put themselves in the shoes of a person whoâ€™s saved his pocket-money to buy a game, as opposed to reviewers who have piles of freebies to try out.
Reviewers who assign an actual numeric evaluation should provide several scores for different types of players, e.g., hard core, casual, RPG fans, shooter fans, whatever is appropriate to the audience.
Aggregation of reviews, and use of the aggregates in contracts, touched off a lively discussion.Â The personal preferences of the reviewer means a lot when he or she assigns only one number, not numbers selected for different kinds of players.Â Yet the numbers are now used in many contracts to govern the royalty received by a studioâ€“the higher aggregate score leads to higher royalties.Â Panelists felt this was a poor way to do business, and that it was being used to take advantage of inexperienced developers to trick them out of some of the profit.Â
Possibly it was also an excuse for a publisher to save marketing money, yet â€śyou can't sell games without marketingâ€ť (Cowley). The panel did not officially include audience participation, but one beside-himself audience member finally asserted that the panelists were dead wrong (I canâ€™t say his actual words!), that if the developer has confidence he should take the opportunity to make more with a high aggregate.Â Â Clearly this is a contentious issue, and with that we ran out of time.