Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 31, 2014
arrowPress Releases
October 31, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
UX & QA Testing: What's the Difference?
by Lindsay Lauters Miller on 06/02/14 01:57:00 pm   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Hi there! My name is Lindsay. I'm the Director of The Research Centaur, the UX+QA divison of The Behemoth. We made Castle Crashers, BattleBlock Theater, and your favorite capsule machines at San Diego Comic-con.  

I get into situations like this a lot: I'm walking the show floor at E3 or PAX, or wherever, and I meet somebody awesome (woo!). We get to talking about what we do -- they make games, I make games and then help test them. I mention I have a background in both user experience (UX) and quality assurance (QA) testing, and they think for a minute.

Then they ask, "What's the difference?"

Well... I'm glad you asked.

TESTING IN VIDEO GAMES

There are so many things developers can do to make a great game more amazing, and testing is a critical piece of that process.

The goal of testing is two-fold, in my mind:

  • Ensure players experience what the developers want them to experience (which usually means lots of fun!).
  • Ensure that the final game is the one its developers have dreamed about -- or is as close to it as possible.

And I'd argue that you need both user experience and QA testing to help you get there.

USER EXPERIENCE TESTING

So what is user experience testing, exactly? If you ask a dozen user researchers that question, you'll get a dozen answers (and hear two dozen names for various types of tests).

Ultimately, though, UX testing is about examining real player reactions to and experiences with an in-development game.

At The Research Centaur, we do this in a variety of ways. Most commonly, we'll bring a group of gamers into our lab and ask them to play a specific part of a prototype game they've never seen before.

Then, we watch what they do and ask the right questions to help figure out whether the gamers are understanding the mechanics and game flow (or, failing that, at least enjoying themselves and reacting in a way the developers like).  If the developers are unpleasantly surprised by the player experience, the researchers running the test will also provide insight into and suggestions on how to change the game for the better.

This type of testing is great at finding out how real players will experience the game once it's released, and what they'll do. It's not so great at finding crashes, graphical problems, or other bugs -- which is where the quality assurance (QA) piece of the puzzle comes in.

QUALITY ASSURANCE TESTING

Quality Assurance is what most developers think about when they hear the word, "testing."

It's the process of systematically walking the game software through as many features and behaviors as possible, and seeing what it does. This helps catch critical bugs before the players ever see them.

The trick to good QA testing is doing it in a smart, risk-based way that doesn't cost too much -- but maybe I'll talk about that more in a future blog post.

HOW THEY WORK TOGETHER

So, to summarize! User experience testing:

  • Is about examining the player's perception of and experience with an in-development game
  • Outputs areas of misunderstanding, unexpected player reactions, difficulty spikes, or shelf moments (times when the player gets so frustrated they would give up on the game)
  • Results in better understanding of how players experience the game, and researcher recommendations on how to bring the player experience into alignment with the developer's vision

And QA testing:

  • Is about examining software behavior
  • Outputs bugs
  • Results in cleaner, better software that behaves in the way the developer intends

Together, these two test disciplines create a better game. Because both bugs and misunderstandings can keep a player from having the best possible experience with a game, I'd argue that they are both equally critical to a fantastic launch.

Do you have any questions about user experience or quality assurance testing? Please let me know in the comments!


Related Jobs

Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States
[10.30.14]

Senior Sound Designer - Infinity Ward
Treyarch / Activision
Treyarch / Activision — Santa Monica, California, United States
[10.30.14]

Multiplayer Level Designer - Treyarch
Petroglyph Games
Petroglyph Games — Las Vegas, Nevada, United States
[10.29.14]

Producer
InnoGames GmbH
InnoGames GmbH — Hamburg, Germany
[10.29.14]

Community Manager The West and Tribal Wars (m/f) on -site






Comments


Koen Deetman
profile image
Hey Lindsay,

Nice article!
Its great to distinguish the software behavior & user experience testing.
I have just watched a GDC session with Alexander Bruce, talking about how he used a lot of iteration on his game Anti-Chamber.

http://www.gdcvault.com/play/1020586/Antichamber-Three-Years-of-H
ardcore

It seems Alexander used a lot of User Experience Testing at large conferences and showcases, and sometimes mixed in some QA based on some user experiences.

Afterwards, he got a question if he used any 'metrics'.
His response was that he didn't find any advantages, metrics would give him in a game like Anti-Chamber.

Now I am curious, what are your views on metrics used for testing?
I know you have to lock metrics on the right content to get the right information.
So in a single player game aimed at PC, where would metrics come-in handy? or don't you recommend metrics at all? If you do, what information would be crucial metrics could 'print' for your game?

Lindsay Lauters Miller
profile image
Hi Koen! Thanks! I'm all about gathering data, actually -- whether you are manually gathering metrics via encoding your sessions in some way, gathering it automatically via some telemetry system, or gathering via external tools (particularly biometrics like eye tracking and GSR, when you can get it to work for you)!

We gather all kinds of data at The Research Centaur/The Behemoth -- generally we target our data very specifically toward the game that we're testing and our research goals for a specific UX test. For example, for BattleBlock Theater, we did a lot of collection of qualitative/subjective "difficulty" ratings, as well as more objective measures like timing level completion and death counts/clustering.

We are fortunate in that we work with a lot of smaller developers as well, who tend to be very open about adding hooks into their game to auto-collect metrics while we are running UX or playtesting sessions. You may also wish to consider rate of usage of different weapons, enemies skipped/engaged with, and order of completion (when working with a non-linear game).

I would definitely say that contextualization of any data you collect is key -- it's easy to use metrics for evil (or assumptions!).

I highly recommend Mike Ambinder's talk on data-driven decision making from this year's Steam Dev Days. It is one of my favorite talks and very accessible. Valve is doing some really exciting things with metrics collection:
https://www.youtube.com/watch?v=HQwL6zh7AgA

Of course, there are different set of metrics you can collect to examine effectiveness of QA testing and likely areas of defect clustering -- but that's a whole other topic.

Hope that answers your question!

Ian Custer
profile image
Hi Lindsay,

What kind of specialized skills / tools should a person become familiar with if they want to start pursuing a UX-based career? You can read countless papers on user experience design / interaction design / data-driven design and so on, but the theory only gets you so far. Where does one start gaining practical experience in the field? Or does academic work tend to be enough to get you started?

Thanks!

Lindsay Lauters Miller
profile image
Hi Ian -- most user researchers I've met have an MS or PhD, although that's not the case for everyone. Tableau is a pretty popular visualization tool, and I would recommend getting familiar with A/V set-ups and recording tools (which is something I don't see talked about a lot -- not every studio can afford a separate IT person to handle these things). Experimental design and statistics is also important but under-represented. I got a lot of my practical experience from working as a tester who also did breakdown/usability analysis, which I had gotten practical knowledge working on projects as a student and as a student assistant in a distributed cognition lab.

One of our clients actually told me that the UX testing for their first small game was done entirely by a budding volunteer researcher looking to get their foot in the door in the industry. They landed a job at a major publisher because of their work, so it seemed to work out for them! The IGDA GUR SIG and LinkedIn group is also a great way to get more exposure to researchers. Good luck!

Mario Wynands
profile image
User Experience Testing doesn't get anywhere near enough love. Great to see an article making the distinction and flying the flag for it.

User Experience (or usability) testing definitely makes a huge positive impact on the quality of our games. For those interested, I did a talk at Casual Connect a couple of years ago about our experiences, and how we were able to initially integrate it fairly cheaply and easily into our process https://www.youtube.com/watch?v=o0MMbrYkfU4

ganesh kotian
profile image
Thank you so much for sharing such a wonderful article on UX and QA


none
 
Comment: