|
[Starting a new series, former Ubisoft designer Luban looks at why regular and detailed playtests are vital to center a game's development around the player.]
There is nothing new about asking testers for their feedback on a game in development. However, the practice of managing playtests by following near-scientific protocols, and of integrating them very early in the development cycle, is a more recent trend.
The spread of real playtests in the game development cycle is probably part of this silent revolution; a revolution profoundly affecting the development environment.
How? Playtests force game development to center around the players instead of the hopes of the development team. Let's look at the effects of this shifted focus:
- Playtests allow the identification of gameplay or level design flaws that could elude the grasp of normal testers.
After all, testers are always seasoned gamers who are not necessarily representative of the target audience. Who better than a casual gamer to pinpoint issues related to the difficulty curve or the overall understanding of the game?
- Playtests fulfill a moderator role in situations of disagreement or controversy within the design team.
A series of playtests can quickly settle a contested issue by resolving almost any counter-argument or dispute, thereby preventing the disagreement from spiralling into an impasse. Playtesting is also a management tool.
- The partnership between playtesting and design can be very constructive. For example, it can be quite instructive for game and level designers to observe gameplay during playtesting, allowing them to immediately determine whether or not particular aspects of their design work as planned.
- Playtests executed on pre-prod mock-ups allow the anticipation of problems very early on, as well as timely corrections of said problems (the faster a problem is corrected in the development cycle, the less expensive it is). Game development can therefore become truly "player-centric".
- According to the playtest protocol and the selection of playtesters (hardcore, casual, etc.), playtests allow the examination of a specific aspect of the game with heightened acuity: game balance, navigation, understanding of the game objectives, etc.
We all have the opportunity to play games that display high production values but nonetheless suffer from obvious flaws: erratic difficulty curve early in the game, navigation issues, overly complex interface, and so on.
Such flaws could often have been easily avoided if they had been identified early enough.
|
Printer-Friendly Version
Your stress on genuine diversity in playtesting is something I think is generally overlooked, perhaps even deliberately avoided. There's no point in any company complaining about sales of a product only appealing to a limited market when they don't bother to incorporate a wide, diverse market in the development team as well as the playtesting groups.
I think one additional point is worth mentioning: you get what you pay for. If playtesting is not reimbursed or is reimbursed poorly, the results will only be as good as what any individual is able to afford to offer out of their own pockets. Obviously, this is going to undermine any other effort at a rigorous scientific approach. This is an area where companies need to devote a decent amount of a development budget just as any professional, well-respected research organization does. Of course, there's plenty of research that does not do this, but the results are highly questionable.
And yet I wonder what this increased focus on playtesting means for video games as an art form and a medium of communication. When people talk about focus testing a movie or a pop group, the reaction is generally revulsion. But when we talk about playtesting video games, the reaction is, "It will improve the final product." Why does that difference occur? Can anyone here imagine writing a song, then bringing in a group and asking them the following questions:
1. Should we throw in an extra chorus?
2. Is the intro too long?
3. Are the lyrics too cryptic?
Most people, I think quite rightly, would consider that to be a horrible way to go about writing music. It would lead to everything sounding indistinguishable. I realise that making a record and developing a game are different processes in a number of ways, but I still think it lessens the idea of a game as a work of art. It seems like games are being produced primarily as consumer commodities, and I'm worried that the focus is on "shifting units" rather than producing the best work possible. I guess the main thing I'm getting at is this - at what point does playtesting make a game *worse*?
I tend to use playtesting in combination with Rapid Iterative Usability Testing and Evaluation. We run players (within a given profile) one at a time while the core members of the team watch and discuss issues that we find. We then try to integrate fixes on the fly so that we can validate them with future users.
The result is a vastly improved game after a few days of testing -- instead of just a written report of issues at the end of the sessions.
We then validate things like "is it fun enough? or "is it well paced?" through playtesting methods like you describe.
I hope you will be at GDC this year: A bunch of games user-research folks will be attending the inaugral meeting of the IGDA Games User Research SIG (http://tinyurl.com/df5xds).
Cheers!
J
@Adam
The difference between games and the media you mention is that games are fundamentally interactive. The only interaction with music is listening to it; there's no control scheme to get wrong, there's no challenge curve to balance. Games are software as much as they are creative expression and it's the software bits that need to be user testing. Focus testing to maximize cynical pandering and shift more units is, of course, problematic. But usability testing of the interactive parts of the game will only improve its final state. No amount of artistic symbolism (which I wish we saw more of in games, don't get me wrong) is going to save a game that's frustratingly difficult to control.
Plus, there's a lot of games that aren't about being art. User testing Puzzle Quest or Mario Kart isn't going to damage its artistic message, because there isn't one. They're just meant to be fun and it would behoove developers to evaluate that goal early and often instead of just assuming they got it right.
Would presenting the publisher with playtesting data to support your position or convince them be worthwhile or even work? Seems like something to look into.
Not only were we able to convince our publisher (albeit it was Microsoft, who is hugely invested in games user-testing) to keep fixing the game, we were able to fix the game much later during the project lifecycle because we had established a pattern of dramatic improvements each time we did a combined iterative usability/playability study.
Usability and playtesting -- when done right -- can be a great way to try out risky new ideas and validate them with actual user experience data.
I hadn't heard that Aeron Chair story before, but I don't doubt it. Like all skills, there's good user-research, and there's not-as-good user research--just like there's good art and code, and not-as-good art and code.
One of the goals of the upcoming Games User-Research SIG at GDC that Jason mentioned (http://tinyurl.com/df5xds) is to help share 'best practices' for games user-research to avoid misleading research outcomes.
@ Adam:
You're quite right--asking listeners how to make a song better is a horrible way to write songs. But that's not what (good) user-research does--it is about learning what is problematic from users, not asking the user to tell you how to fix the problems.
To put it back into game terms:
* Players know when they don't like something in a game--ignore them at your own risk
* Players don't know how to fix that something--take their suggestions with a handful of salt
User-research's job is to help find problems by testing the game with actual gamers; it is Design's job to find feasible solutions to those problems.
>Give the mockup of the near-final game to a group. Watch them play.
Not really the same thing, as your use of the phrase "near-final" indicates. Imagine instead if the makers of a board game playtested specific elements of the boardgame design throughout the development cycle. Roll dice or draw cards to decide how many spaces to move? If this is a point of contention in the design, playtest it.
Of course, board games have done this kind of testing too, though I doubt it's the norm. The point of the article isn't that playtesting itself is brand new. It's the degree that it's been adopted throughout development, and the increasingly scientific approach to organizing the sessions and collecting results. It seems more like a scientific drug trial less like a traditional focus group.
I am reminded of a sneak preview of a movie I went to many years ago. As we walked out a studio staffer asked us a couple desultory questions, and that was it. Seemed to me the goal of the preview wasn't to fix anything, or find out what was wrong; it was to see if they had a hit or disaster and to decide how much money to spend on marketing accordingly. Exactly the opposite of what this article describes.
Looking forward to part 2.
* Players don't know how to fix that something--take their suggestions with a handful of salt
User-research's job is to help find problems by testing the game with actual gamers; it is Design's job to find feasible solutions to those problems."
Excellent point, Bill. It's important to know what the users are thinking, but there's a reason that you're the developer and they're just the player.
Now to the article, I'd like to add a couple more drawbacks to asking testers for feedback that weren't previously mentioned.
The first kind of ties into Bill's point that I quoted above, in that users (including testers) can't always judge why they don't like a game. They just know that they aren't having fun and thus label it as a bad game.
When I worked in testing, I found the general trend to be that testers hated the game they were testing, even if it turned out to be wildly successful after release. The reason is because they're playing a game for 40 hours a week and not having fun, so while you've got a few here and there that can separate their experience from the game itself and analyze it objectively, most consider it to be a bad game because they don't have fun playing it.
The second danger of getting tester feedback is the culture of game testing (at least, the low-end black box testing). Regardless of what the truth actually is, you learn in the testing lab that developers are gods who are quick to anger and easily offended, and criticizing them in any way is a REALLY BAD thing. Whether or not this is a good system for testing, it's a bad one for getting honest feedback on a game.
- The article is not so much talking about professional or full-time in-house testers, it is talking about limited engagement user testing. IE - inviting people of varying gaming backgrounds to come and play an in-progress version of the game for a few hours and give feedback. This is useful to identify user-interaction problems, while full time QA personnel is better at identifying technical issues and pin pointing where software breaks down (in addition to their own user-interaction feedback).
The new challenge is not to get people bought into doing playtests but to make the process better and easier for all involved especially for the ever in crunch mode developer.
Unfortunately I wont be able to attend the new SIG at GDC...
I enoyed the article (and follow-up conversations) and think it highlights many important points:
- Need for identifying test goals and creating test plans up-front (looking forward to Pt. 2)
- Importance or recruiting approriate playtesters
- Differences between playtesting and focus testing
@ Ray
I totally agree. At THQ (where I manage Usability and Playtesting), the issue isn't so much do we test a game, but when and how (and how much) do we test in order to get the most effective (and efficient) results.
In the process of game development, many features require time, attention and tweaking before they show to be useful and good additions to the game - however, these features in their unfinished state, to the "average Joe" who picks up the game, can seem to be the most terrible, game-ruining element.
Also, often when the game is in an unfinished state especially early in development, even hardcore gamers rarely get past commenting on the place-holder art and roughly implemented features in order to actually give useful insight into the game. Again, this is different at big developers because they're often able to get something polished up and running a lot earlier in the dev cycle, but for any project where it comes together later in the cycle, you really need someone in QA looking at it who is experienced at overlooking placeholder stuff in order to give decent feedback, even if they can sometimes overlook too much.
Finally, as mentioned on the first page, these playtests can be used as ammunition by whomever is running them (publishers, external producers, etc) against the development team, being treated as a be-all-and-end-all resolution to any disagreement. It's very easy to say "it doesn't playtest well, get rid of it", even if that isn't justified because the feature simply hasn't had all the work it requires.
I don't disagree that early and mid development playtests are compatible with huge budget game development - with tens of millions of dollars at stake, it's a required element of risk reduction. However, I think that the process does in general hinder the natural trial-and-error process of innovation, and can keep developers from creating a more original and unique game.
If those are the experiences you've had with playtesting, I'd have to say that you may not have experienced user-research done well. :)
To your specific points:
1. "...playtests that happen any time before the game is in a complete and decently polished state can hinder as much as help a game's development cycle."
I agree to a point--BADLY-RUN playtests can definitely hinder a game's development cycle (regardless of the state of completion and polish, IMO). The role of user-research is:
* To know when tests are worth doing, and when they aren't.
* To help game development teams know what they need (and don't need) to build in order to test a particular feature and system
In my experience doing user-research, the most valuable tests are when a team develops a very focused slice of crucial game experience and tests it well, to make sure that their fundamentals are solid before building a game around a problematic core.
2. "...many features require time, attention and tweaking before they show to be useful and good additions to the game..."
Agreed. I think testing a feature before its essential component parts are prototyped is not a good use of time. But my experience is that most game developers are not a good judge of what participants can and can't look past; and so they err on the side of not wanting to test until features are more 'done'. This often puts them in the situation of having serious problems in the game but not having the time to fix them, or do the 'right' fix.
3. "I don't disagree that early and mid development playtests are compatible with huge budget game development"
I disagree on this one. User-research is just as valuable with smaller budget games because it is always necessary to provide a more compelling experience than your competitors. Because the casual space is so crowded and try-before-you-buy is the norm (meaning that if the game experience isn't compelling within a few minutes you've probably lost that customer forever), I think it well-done and cost-effective user-research is probably more important than in the bigger budget games. Plus, the user-research is cheaper, as there's usually less game to test.
4. "I think that the process does in general hinder the natural trial-and-error process of innovation, and can keep developers from creating a more original and unique game."
I couldn't disagree with you more on this one. By definition, unique & novel gameplay isn't common; because players wont' have seen that novel gameplay before, and thus will have to learn it without prior experience to help them along--in fact, they may have to unlearn behaviors in order to do the 'new' thing. In these situations, user-research shines.
But regardless of my disagreement and analysis, the following successful game have given talks about how crucial user-research has been to their success:
* Halo 1 (they developed their controls in the usability lab--and those controls are now the 'stardard' for console FPSs)
* Portal
* Call of Duty 4
* Resistance 2
In fact, both Bungie and Valve are giving talks at GDC this year about user-research.
Anyway, if you're interested in learning more about how user-research can be done well (and avoid some of the problems you cite above), contact me (billfu@roninux.com) or attend the Games User-Research SIG at GDC mentioned above.
For those of you who don't know me, I also do Game Usability Research (Playtesting, Game usability testing, etc), as a consultant for my company Behavioristics-- we are ever evolving the techniques of research to help make games more fun.
In my varied experiences working for many game publishers and studios, I find that some companies are very open to the "new" methods of game research, and others that are quite archaic!! I'm still finding some publishers, even one yesterday, HESITANTLY moving to newer game research methods-- many are still "married" to Focus Groups... but on the other hand, many as you say above, have come along way. I'm looking forward to GDC's SIG this year--- there is certainly a call to action to share methods, and move them forward, to raise the bar. To make all games more fun---