It's good to see this understanding starting to percolate down into the video games industry. The issue of research denial/rejection crops up whenever you have designers/developers involved in the process. They've spent months/years working on a game and are just too close to the product to be completely objective.

It is natural for them to defend their choices, but the severity of rejection can be managed by adopting a testing-friendly culture. This usually means a lead developer or producer setting expectations for the role of external testing throughout development iterations. If the input from external testers is properly framed as being “premium research” (i.e., it is more valid but also more expensive) then knee-jerk reactions can be avoided.

And even if testers are stupid, so what? The chances are that testers *will* have lower IQs than professional software engineers. So this means that developers are smart enough to understand that the target segment for their mass-market game will *not* be “software engineers in the 95th IQ percentile,” right? In my experience, if all else fails, spelling out the connections between [segment – satisfaction – sales – paycheck] seems to work on stubborn client employees.

Ah, so good to see this rule spreading into video games. As Hemingway blandly noted, in good writing you must "kill your darlings".

I think there's a fine line between making changes to improve the quality of the game and sacrificing your creative vision in favor of more commercial appeal. The latter is, in my opinion, what makes games a commodity rather than art.

Good point, Neil Sorens. In other media - books, film, music, etc - it took years for something new and "weird" to get recognition. Take punk rock for instance. The first punk artists - like the Ramones, the Sex Pistols, Iggy and the Stooges - were looked at as complete lunatics when they initially came out. Now, years later, they are legends - recognized as geniuses for having revitalized rock and roll.

Neil - Unfortunately, games are neither a commodity nor art...they're a business. I am not advocating "design-by-focus group" at all. That is not what they are for. Two elements that external testing *are* fantastic at identifying are confusion and frustration. And I wouldn't count either of those as artistic choices.

I agree with Neil. Playtesting is definitely a valuable asset for catching bugs in not only your programming but also your gameplay, pacing, direction, etc.
However, you shouldn't let that deter you from trying something new or different; players confused about what they are supposed to be doing as the point of the game is different than players not finding a ladder or finding the enemy boss too difficult, and in this case you need to explain more clearly what the point of the game or gameplay is.
For example, back when Splinter Cell came out, they made it clear that stealth was the preferred approach to the game, and anyone trying to run through shooting everything in sight either died or ran out of bullets and then died. Quickly, you learned that you should sneak past if you can, and being caught is bad. (I'm not saying Splinter Cell is the first game to include stealth, I'm just using it as a for instance.)

Taylor Strait -
Quote - "And even if testers are stupid, so what? The chances are that testers *will* have lower IQs than professional software engineers. So this means that developers are smart enough to understand that the target segment for their mass-market game will *not* be “software engineers in the 95th IQ percentile,” right?" Unquote.

Wow. Arrogant and wrong. I'm stunned you said that.

Steve, I think this may be the case of the internet removing context from a statement. My point there was that it doesn't matter whether testers are "smart" (by programmer standards) but that they represent the product consumers and therefore should be listened to.

You are both correct...

If you are trying to make a game that will be easy for someone to pick up and play then playtesting will focus your game play design to something more easily accessible to play. This is not a bad thing as it generally results in more commercial success. And why not, not every game has to be innovative to be fun. In fact most games that are enjoyable are games we have already played with different styles put on... again this should not offend anyone as we are industry that provides entertainment. And most people like challenges within the bounds that they are comfortable with... otherwise they get frustrated and don't enjoy the game.

Coming up with truly innovative game play is an excellent endeavor to strive for, but the odds of it being commercially successful (in the first game of the new form) are rare. This isn't to say that it should not be attempted as whole new game forms lift the entire industry... its just that it is rare and even more rare when based on a current genre. So if you think you are going to make a FPS with some new innovative twist then good luck... the odds are stacked against you and better people than you have tried and failed. And if you find people think your game odd and not enjoyable don't blame them... it probably isn't, but then neither is most things until people get used to them.

So in the end you often simply have to determine whether you want an enjoyable game with commercial success (an ok chance) or an weird game with legendary success (very little chance).

Hehehehe.. good article.

It's amazing to me that so many groups out there do not put much faith in outside playtesting...they are your end-users afterall and if they can't 'get it' chances are the 'market' won't either.

I'm afraid to touch the enigneer IQ discussion but in my experience, [most] software engineers make the worst gamers!

Those of us from a HCI/Ergonomics/Usability background see this as an obvious essential activity, especially considering the overall investment required to develop a big title these days. Who wouldn't you want to check that explicit and implicit design assumptions are providing the desired effects? In my experience, quite a few people. It seems that in companies where external testing is not part of the culture there are a number of prejudices and assumptions that prevent many from seeing these activities as valuable. Some of these prejudices are similar to those encountered in other industries, but some are more apparent in the games industry. Whenever I have suggested such an approach, to people I have met, responses have included: "So you want to be a games designer? Everyone wants to be a games designer."; "Pfft Focus Testing... we already do that."; "We already have testers, and what they tell us about gameplay is just a load of differing and conflicting opinions most of the time."; "Why would we invest in yet another production process? If we could just get the ones we already have running properly and let the designers do their jobs...".
Hopefully with Valve, Microsoft, Sony and a few others leading the way in integrating timely research and feedback into the design process, such 'external' testing will become the norm rather than the exception, just like it is in some other domains.
Hopefully though, with the big budgets involved, there won't be a shift to 'player expert' specialists who are employed to pontificate on the ideal player, tweaking designs based on another set of guestimates, before rubber stamping the product as Playable, as 'user experts' are often employed in other software development industries. Without real players such 'playability' testing would be divisive.
Yes I am angling for a job ;)

I see this article got blown up with responses. For everyone out there, its all about getting used to Agile game development. It is the most heavy testing development process and obviously shown here, it works. I now perfer it over scrum.

It's got little to do with which software development process is used, even old fashioned waterfall should include relevant user input at each stage. Employing a resource to collect, collate, interpret and disseminate this information at the right time to the right people is the best way of doing this, and this process will be agnostic about the process it is supporting.

I'm not sure how the example of the broken texture is meant to highlight the need to listen to testers: it was an obvious and reproducible technical issue. The second example is missing some key information: why did the testers reject this functionality?

A better line to have taken would have been "familiarity breeds contempt". Developers are often too close to the system and hence overlook gameplay issues: when you know how the AI will respond to player actions and you know the map and/or enemy weakpoints, it's unlikely that you'll be able to conceptualise the issues that a new player without this experience will have. Unfortunately, the same issue also applies to playtesters: even if they've never played your game before, they're likely to have far more extensive gaming experience than the average gamer and it takes a concious effort to step back from this experience and try to find bugs or issues that a new player would encounter.

This then leads to another issue: keeping the balance between "new" and "jaundiced" without impacting timescales or spending money bringing people up to speed...

Taylor,
I'll take you at your word. My intent isn't a flame war. :)

Other posters -
As for Outside playtesting - I think it's extremely important. In my experiences, it's far tougher for the design team/coders to dismiss gameplay/design element criticisms from people they don't work with daily. Flip side - it's also good to get fresh praise from non-familiar viewpoints as well.

Still, I think it's important for the QA lead(s) to be honestly open to the entire process. I've worked in a few outside Betas where very valid points were dismissed out of hand. What's the point if you have pre-set results in mind going into the test?