Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 1, 2014
arrowPress Releases
October 1, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


Opinion: Let's talk about why QA sucks
Opinion: Let's talk about why QA sucks Exclusive
April 12, 2012 | By Brandon Sheffield




["Quality assurance is an incredibly important pillar of game development, and the industry as a whole does not treat it accordingly," argues Game Developer magazine EIC Brandon Sheffield in this op-ed originally printed in the April 2012 issue.]

Maybe you're an artist. Maybe you're a coder. Maybe you're a designer, or a producer. Maybe you're all of these. But chances are, if you fit into one of those categories, you have a career path of some kind set before you. You can become an effects lead--you can move laterally and become a tech artist. Learn some behaviors and become an AI-oriented designer.

Your QA team doesn't have a career path though. Quality assurance is one of the most important aspects of game development, and all we encourage our QA staff to do is move "up and out" of the QA slog, if we inspire them to move up at all. Many QA professionals talk about "getting out" and moving to the production or design path. Don't you want good QA folks to keep doing QA? Shouldn't they enjoy and want to start in their jobs? Isn't there something wrong with this picture?

Get Bonus

Your QA staff is your front line of defense against bad reviews. Obsidian creative director Chris Avellone recently mentioned that the company didn't get a bonus for Fallout: New Vegas because the game missed its Metacritic target of 85 percent by one percentage point. They had to reduce staff as a result.

Game reviewers play a lot like testers sometimes. They push against boundaries, and find those things you're pretty sure nobody will ever bother to do. They complete quests in the wrong order. They jump over a wall and trigger a cutscene that wasn't meant to happen for hours. In short, they break your game, then call it buggy. What does that do to your Metacritic score?

Not all bugs can be prevented, but with creative testing and enough time (that's the kicker!) you can get most of the showstoppers. Everyone knows QA is important, but let's really think about that. Is your QA team inspired to think creatively, and go that extra mile? Are they treated like important members of the team?

Most QA is hired from a pool of fresh-faced kids who just want to get into the industry any way they can. They are passionate about games, but aren't sure how to break in, so they take the QA route. Maybe they've heard about game designers who started as testers. What they don't want to do is stay in QA forever. And why would they? The hours are long, and the pay is the lowest in the game industry, at under $48,000 average across all years of experience, almost $30k under the next-lowest discipline. Many are hired on contract, at low wages, then get let go when a project is complete.

Does this sound like a job you'd want to stay in? Or a job where you'd be incentivized to think creatively? Is this job with a career path? If the biggest incentive you're given is to become a lead and then move to another department, how much can you really care about working in QA?

How many times have I heard "the dev team" and "QA" spoken of as though they're different things? In many companies there's a physical wall between "the developers" and QA, if not an entire building. QA is part of the dev team. Why is there this mental space between the two?

It's a self-fulfilling prophecy. If you think the QA kids are scrubs, stop hiring scrubs. If you want people other than scrubs to apply, there needs to be a fundamentally different way of thinking about the entire department. If QA is thought of as a viable career path, and a truly important part of game development, it won't be considered lower-tier, and your games will get better, because creative people will be thinking about how to improve your games and processes.

At Valve, for instance, everyone has specialties, but everyone is a developer. Everyone plays the game all the time, and thus everyone is QA. That's not so bad, is it? How can we reach this level of integration in our own companies?

Quality time

The first thing to do is to change the company culture and mind space surrounding QA. Invite QA leads to important meetings. Creative meetings! Make sure you, or your leads, speak of QA with the same respect you'd have for any other discipline. Make sure your entire team is speaking to everyone else on the team, and regularly.

Next, offer an appealing career path within QA. Certainly there will be generalists that check everything, and some general leads. At the same time, QA professionals that are interested in music should essentially be part of the sound team, working to develop an audio map to test against, with the power to implement changes. Those with a tech bent should be speaking regularly with the leads in that area to monitor frame rates, and suggest areas for reducing load. And so on and so forth.

Finally, don't lay them off when a project completes! If you want loyal employees, your company should be hiring for the long term. You should be recruiting QA professionals with a variety of skills that can be applied across the project, like you would in any other discipline. Unless your team is gigantic, you likely don't hire an artist who is only good at rigging. It's a specialty, sure, and you rely on them for that. But when it's time to do a bit of modeling, or mocap cleanup, they can be counted on. So too should it be for your QA staff.

If you really need to ramp up and down our QA, use external QA groups managed by your permanent internal QA team. If you think of any members of your team as an expendable resource, they will not do their best work for you.

This is only the start of the discussion. QA is an incredibly important pillar of game development, and the industry as a whole does not treat it accordingly. If you want to see your next bonus, maybe it's time to change all that.


Related Jobs

Red 5 Studios
Red 5 Studios — Laguna Hills, California, United States
[10.01.14]

Web Developer/Web Architect
Halo
Halo — Kirkland, Washington, United States
[10.01.14]

Senior AI Engineer
Telltale Games
Telltale Games — San Rafael, California, United States
[10.01.14]

Tools Engineer (Qt)
Telltale Games
Telltale Games — San Rafael, California, United States
[10.01.14]

Narrative Technology Engineer










Comments


Ron Dippold
profile image
If you're not reading In the Trenches you really should. I'm enjoying the comic, but the real gold for anyone in the game business is in the stories from QA people:
http://trenchescomic.com/tales/post/drive-fast-and-crash-a-lot

When you say 'How did QA not catch this?' or 'Didn't they have any QA on this?' the truth is they probably had QA and QA probably caught it, but they decided not to fix it.

I respect QA. I cringe every time something show up in my Jira folder, but then I suck it up. I also treat QA guys with respect and strangely my bug reports are pretty useful.

So yes, I totally agree with Brandon's article here. It's like getting a prostate exam or tooth cleaning - not pleasant, but you're better off getting it than not.

MaurŪcio Gomes
profile image
This particular story is very more reflective of the article:

http://trenchescomic.com/tales/post/the-politics-of-meal-time

Poor Audio guy :(

Nathan Verbois
profile image
When you say 'How did QA not catch this?' or 'Didn't they have any QA on this?' the truth is they probably had QA and QA probably caught it, but they decided not to fix it.

Yep, we caught these bugs and they got KS'ed or AD'ed. :)

Ronildson Palermo
profile image
I think you hit the spot there... a LOT of companies think it's easier changing the design doc than the actual game.

Michael Rooney
profile image
@Ronildson Palermo: Just a thought experiment for you. Some background, almost all QA happens after vertical slice. Most of it happens post-alpha (1 month-ish before launch).

Given the following situation, what would you do:

1. You have 1 month before you submit to the manufacturer. You are required to have 0 TRC bugs to pass this.
2. You have 1 developer (1 is arbitrary because it is simple. Just know that the bugs in this situation scale with the number of developers). Assume they are already working overtime.
3. You have 150 bugs.
3.1 25 are TRC violations. Estimated time to fix is 1 week, These must be finished or your game will not ship.
3.2 80 are low priority bugs. Your developer can fix 10 of these per day.
3.2 35 are mid priority bugs. Your developer can fix 5 of these per day.
3.3 10 are high priority bugs. Your developer can fix 1 of these per day.

4. Prioritize your bug fixes before reading further and imagine everything goes fine with those priorities for a week.

5. A QA guy just found a 100% reproducible crash that affects a major, but not core, feature of your game. It will take your developer 5 days to fix, because it was an edge case the system isn't designed to handle.

What do you do?

Ronildson Palermo
profile image
The surefire for the developer is to alter the feature, and I agree it does seem like a great idea in order to fool the consumer into believing you did everything right during production. But the consumer will suffer!

First because the he will not be playing with a, as you put it, major feature that was originally designed to be in the game and, most certainly, was heavily advertised by your ever enthusiastic marketing team. And god forbids that being the reason you remove entire levels from the game! Like that's not wrong at all, taking those $59,99 from your consumer that was originally meant to pay for 14 levels but instead you're delivering 8 because said major bug breaks them all.

That's ripping off the consumer without him even knowing it...

If I'm the producer? I patch the game later. Skyrim did it, Mass Effect 3 did it and that's what's happening with pretty much every other big release in one platform or the other these days.

I'd never go around ripping off chunks of content for the game like I've seen, and heard from other people, being done.

Daniel Martinez
profile image
This article is really just the tip of the iceberg as Brandon mentioned. There is a stereotype that "QA does nothing but play games" and while that is technically true, it's still a job none the less and does not warrant compensation that puts sweatshops to shame. I don't ever sit in front of the same game for 8 hours feeling my retinas burn away for leisure, much less if the game is buggy and requires the constant repetition of arduous, mundane tasks (searching for mapholes anyone?). After the first 40 hours or so of being on a project the novelty wears off and then the game of sleep vs work begins. With that said, if QA is regarded as a cancer separate from the dev team and the company as a whole, what makes developers and publishers think that QA will put forth their best efforts? A happy employee is a productive employee. In the worst of cases, all QA employees really want is some respect, recognition, appreciation, and maybe some munchies every now and then. :)

Jr Hawkins
profile image
I'll start off saying there's a huge difference between internal QA, and publisher QA. Most internal teams I've been involved w/ have tried to treat QA like developers (at least the internal QA team). Publisher QA is a mess. This is where you hear all the horror stories from QA, and I'm afraid to say it's well deserving. Honestly I think it's a lot worse than most developers realize, and that's pretty scary.

So why don't developers just keep internal teams?
QA tends to be the largest team on any project, and for an internal studio it's hard for them to justify keeping 20 guys on who have no work to do since most of QA is only needed for the 2nd half. It's much easier for a publisher to organize QA teams since they'll have 5-6 projects a year instead of one project every 2 years. I'm not really sure why publisher QA has so many problems though. Maybe it's the pay is too low to get quality employees, or maybe it's just bad management, but it seems every publisher QA facility in the world is crap. The worse thing about it I see publishers throw giant teams onto projects when only 20-30 testers are needed (and that's being liberal). Then the team is 1/2 as productive, and cost 3X as much.

Daniel Martinez
profile image
I'd say it's a combination of both a sub-par pay rate and lack of good management and possibly other factors. Even with the terrible pay though, that isn't to say that many of the employees that work are not good quality, they're just paid like they aren't good quality.

Joe E
profile image
Agreed - Valve's case is actually pretty typical for 3rd party and independent studios. It's when you have external publisher QA that the separation occurs and things start to get messy. And I'd argue, in those cases, this separation (sometimes even confrontation) is not isolated to the QA process.

John Theodore
profile image
I've worked on many internal QA teams where the treatment was very similar to what you're referencing with an external QA team.

Here are some highlights:

I was yelled at and cussed out by a developer for reopening a bug that was still open because it had missed the build.

On my first job in the industry, I noticed that a popular feature from a previous version of the game I was working on had been removed from the version I was working on. When I mentioned this to my manager, he told me I was just a tester and that I should not be thinking about that.

On that same job, QA was literally locked out of the space where the developers worked. They actually locked the door to keep us from interacting with the developers.

After the project was completed, half the team was let go. The game set records for sales. No one in QA got a bonus.

On the next major game I worked on, I was full-time, not contract. That game also set records for sales. Reviews of the game specifically mentioned how much cleaner and polished the game was compared to its previous version. While developers were buying expensive sports cars that were made in Europe with their residual sales-based bonuses, QA got nothing. Months later, we got a $1,000 bonus. After the DLC packs were released, the team was downsized, and I was laid off.

In summary, as a *former* tester, I felt the way I was treated to be degrading and insulting. I did professional work that was greatly appreciated by many of the developers I worked with. Many developers have told me how much they appreciated having someone who paid extra attention and took the job seriously. I had many great experiences, but at the end of the day, the juice simply wasn't worth the squeeze.

Joshua Sterns
profile image
There is a huge difference between internal and publisher QA. This doesn't mean QA is treated with respect at either.

QA's treatment varies from studio to studio, and within, from project to project. Working at one studio with the right producers made a huge difference in my experience. Other QA teams were jealous of the involvement with the dev team. I was treated with some respect, and that went a long way towards my tolerance level for the other issues with the job.

Then there was that other developer. The ones who denied us access to the main building. Separate kitchens. Don't take breaks in front of producers or else you are a slacker. Not allowed to attend company sponsored events unless we forced the issue. Denied silly things like Inn N Out when the rest of the studio gets it for free. 2 to 1 PC ratio, and daily threats of being fired (we kept a tally and the most was three times in one day). Daily roll call in the morning and after OT dinner. (I'll stop here but trust me I can keep going.)

Not everyone will have the same experience I did. That said, I'm confident when I type that QA sucks ass across the board in the videogame industry.

Joe McGinn
profile image
Good article, couldn't agree more. Always try to involve QA in the design and other processes, and they are as much a part of the team as anybody. Also agree that if they don't "want to get out" they need a viable career path. A great QA lead is incredibly valuable.

Paul Boyle
profile image
I would love to support this article. QA guys are people, and they deserve fair treatment. However, tbh, Brandon's suggestions would lead to making worse games, not better ones.

Unfortunately, there are problems with both his suggestions:
1st, "Get Bonus" Hiring people who are not 'scrubs' and giving them a better career path, not laying them off, essentially amounts to paying them more money. The fact of the matter is that most projects don't solve all their bugs, pre-ship, NOT because they aren't found, but because they dev team ran out of time. The only way to get more dev time, usually, is to have more money. Less money = less dev time. Hence, 'higher quality' QA would, in general, lead to a bigger bug backlog, and a fewer bugs fixed.

The key misunderstanding here is that QA is NOT your first line of defense against bad reviews. It's your LAST line of defense. Its there to catch all the things that made it through the cracks, the little things. If your game has huge issues that are going to significantly affect its metacritic rating, these issues problem stem from failures that are present before QA can and should get involved. Yes, QA is important, but its job is not to help you build a game that will score higher - it's there to help you prevent a game that will otherwise do well from failing due to uncaught bugs.


2nd "Quality Time"

Most development teams already have too many people contributing ideas and suggestions without having development responsibility. Sure, QA has a perspective, and I've never failed to hear it voiced, there's a certain percentage of bugs that are 'buggestions' in every project. But if you really lack for ideas and perspective on your team, that's a failure of design and production, which should be addressed, not the lack of a QA person pointing those things out.

Michael Rooney
profile image
I think a third issue is that not all QA people are good. Being a low barrier to entry/high demand position with a hard to prove skillset (no qualified QA certifications/degrees and even lots of experience doesn't guarantee quality), a lot of QA people are not that great. The ones that are good at their jobs are indispensable, but unfortunately all the crappy ones make it harder to correctly appreciate the good ones.

Crappy ones can also be extremely detrimental to workflow. I've spent loads of time in the past just clearing out bugs that were clearly as designed or called out as in progress to QA before they were marked as bugs.

It's one of the most difficult positions to find good people for, and one of the most important once you find them.

Evan Newton
profile image
Mmm... Not to disagree with you, but I strongly disagree with about 90% of what you said here. The attitude that "QA is just here to catch bugs" is a self fulfilling prophesy--you get single minded QA whom, for the most part, you would never hire into any other position. Then, as developers, we complain about QA people being inflexible, single-minded, and a little too enthusiastic about issues they find.

This is an area where some companies are starting to change, slowly, but it's an important evolution:

QA is not just the "bug-catching" team. QA should be viewed as the primary authority on customer experience. With that kind of thinking, what kind of people would you hire for QA?

In a "primary authority on customer experience" capacity, you have QA who provide meaningful and important feedback on how features relate to one another, as well a drive to actively understand what works and what doesn't in terms of gameplay. From an experience standpoint, that means you preferably want QA who have broad design, production, and possible engineering backgrounds. Jack-of-all-trade types are awesome for this.

Also, QA is an active stakeholder in the game, just like any other department in the company. It's my belief that QA should ABSOLUTELY have a seat at the creative table from the very beginning, not as a "Cook in the Kitchen", but as a stakeholder in customer experience. They can help head off design headaches at the pass, and very effectively help prevent wasted work.

Finally, everyone should have an opportunity to be the first line of defense for the game, including QA. QA just happens to be in the unique position of being one of the last lines as well.

This is an industry-wide attitude problem about QA. People with bad experiences will tell me how wrong I am, but I will tell you right now, it works, and it's working well at one of top companies in the industry.

Dave Malmberg
profile image
Sorry Paul, you're wrong on a few points but I'll focus on this one: "The key misunderstanding here is that QA is NOT your first line of defense against bad reviews. It's your LAST line of defense." this is a common misconception in the game industry. In short you suggest "trailer hitching" QA into the process. QA can (and should) be implemented as early as possible even bringing in QA into Pre-production to help highlight potential design flaws. There would be far less things to get through the cracks if you implemented QA in this manner. Many other industries place a much higher emphasis on the QA process, in the game industry it's become acceptable to release "buggy" software much to the detriment of Reviews and the User Experience.

Michael Rooney
profile image
@Dave: I think you misunderstand what he meant by "last line of defense." Your first line of defense is good design, good implementation, and good market data to base all of those off of.

QA is the last chance for a bug to get picked up by somebody to put it better. The problem is always there. The design solves that problem (first line of defense). If it doesn't, then somebody involved with implementation will hopefully catch it (second line of defense). If the problem makes it through implementation, then QA will hopefully find it (last line of defense). If they don't then your defenses are breached and the public finds the problem. I don't think it's right to make a correlation between being a 'last line of defense' and being bad.

@Evan: The problem he illustrates is just having too many cooks in the kitchen. It is right for everyone to point out problems. It is not a good idea to have everybody coming up with possible solutions to every problem.

Jeremie Sinic
profile image
It's a vicious circle I have witnessed too often already in my short career: internal QA people are often hired without any specific qualification, then they are not considered worthy of being listened to, then they quit and are replaced by new faces without experience...

I think the QAs should work directly under the Producers, so that their voice really counts. Currently it feels like QA's power is only in informing developers about issues and bugs. They unfortunately don't have their say when it comes to taking decisions like fix or not fix.

QAs are the closest people to end users: ideally they should be the ones with the last word.

Ryan Marshall
profile image
It's not entirely true that QA has no say. QA knows that other members of development don't have a lot of time, and that what time they do have is fairly valuable, so they utilize significant keywords to make an issue seem worthwhile or not.

As a lowly QA grunt, this always seemed like the most important duty of the QA team lead: he was the one to make the call on whether to play a bug up or down, taking into account time deadlines and all that other stuff.

Bob Stevens
profile image
If QA is gonna have final say over whether or not a bug gets fixed, they should be the ones fixing it.

Brandon Sheffield
profile image
I definitely don't have all the answers here - and due to its origins in the magazine, this had to be under 1,000 words, so there's a lot I couldn't cover. Really I just hope to open up the dialog about QA, and the situation can improve - I'm sure better ideas will come forth in the comments here than the ones I presented above.

Scott Williams
profile image
This is a really interesting read and tackles quite a few key points. For me as a QA Manager I found that the career path in QA was hard to obtain. It took years working on temporary contracts until I got my first permanent lead role and it went from there.
When recruiting for my last project I only hired experienced testers. This was the strongest team I ever created and the reception from the development team was great.
The problem doesn't lie here, it's convincing top management that the team is worth keeping around for future projects and that they should be paid much better than todays rate.
Unfortunately when the end of project comes, QA is one of the first on the chopping block regardless of talent and feedback from the production team

Glenn Storm
profile image
QA is an essential specialist position; a critical part of development able to provide invaluable information to each team member, particularly the designers and producers.

While the artists paint, model, and animate; while the audio crew mix, tune and tweak; while the engineers organize, implement and optimize; while the designers research, explore and refine; while producers gather, communicate and cooperate, _someone_ needs to help be the eyes and thumbs of the player.

Everyone on the development team is ideally an advocate for the player, especially the designer, but the only position dedicated to being an objective observer, the only specialist with the bandwidth capable of giving the kind of feedback your reviewers will give post release, is the tester. Bugs are a development team's golden eggs; each one is an opportunity to make the reviewers' experiences better, to make the game better for every player. And that opportunity is directly proportional to the severity of the bug.

Welcome the bug from QA with gratitude, not just respect.

John Flush
profile image
Unfortunately too many business owners see QA as the department that tells you that your other departments (Development) can't do their job right. As long as that is the view of QA and the solution is viewed that "Development should just do better" QA will always be at the short end of the stick.

That and the revolving door of QA. Having gone to a few testing conferences over my career it is amazing to see the churn of ideas. It seems like QA gets reinvented every 5-10 years with new nouns describing the same workflows and processes because the people that figured it out 10 years ago got tired of never being listened to and moved on.

Simon Ludgate
profile image
As someone who's worked as both a QA and a reviewer, I have to say: these are not in any way, shape, or form similar jobs. While working in QA I've often seen things I'd absolutely smash a game for as a reviewer, even things that could be fixed quite easily, but if I enter them in as "bugs" all I get are angry managers telling me "that's not what your job is."

One of the big problems is that Quality Assurance is poorly named. Most of the time, it's "technical requirements checking." For example, if you were to publish a game on a PS2, no UI element could enter the 15% outer edge of the screen, because it could get clipped off by an old CRT. So "QA" people "play" the whole game with tape on their screen, making sure nothing goes out of bounds. There really is no glory to this kind of task.

Jeremie Sinic
profile image
"Most of the time, it's "technical requirements checking.""

I agree, and I think there's a big lost potential here.
That's why the role of QAs should evolve to include "giving opinion on the overall fun of the game".
That would value their role and lower job turnover.

Ariel Gross
profile image
We treat our embedded audio QA tester as a member of Team Audio. I believe he's invited to all of our meetings, and we take his input seriously. He is truly part of our team, and his contribution is totally valued. Also, he's just an amazing dude. Brendon Ellis, WE LOVE YOU. /salute

I agree with the sentiment of this article, too. There is a sort of stigma about QA with some people, and it's a shame, because they really do make our games better in a huge way.

Nathan Mates
profile image
It's not that QA is overloaded. It's that mathematically, they're a very thin line of defense. Let's do the math. Numbers are approximate, based on my experience in AAA games. Let's have a 25-person QA team working 100 weeks on a game for 45 hours a week. (Yes, QA generally starts smaller and ramps up, but let's smooth that out.) That's 112,500 hours of eyeballs on the build, but most of those hours aren't on the final (or even near-final) build. Now, for a AAA game, selling 100K copies in the first week or two is pretty normal. Assume each of those buyers plays (on avg) 1 hour each. Guess what -- consumers have now have a pretty comparable number of hours playing the game as QA, and way more hours playing the final build than QA ever had. And if the customers play 2-10 hours you're way behind.

When it all comes down to the bottom line, QA is not a monolithic force. QA is comprised of people, who may be good, bad, or indifferent. I've worked with some QA who's worth their weight in gold, and some QA who aren't. (Worst case: a two-week fight where QA insisted that unscrewing and disconnecting the network adapter from the Fat PS2 was comparable to pulling the network cable or controller. It only ended when I bought one of those adapters for my PS2 at home and quoted chapter and verse from its manual saying to power down the PS2 before [dis]connecting the adapter.)

QA managers who can really enforce best practices on QA -- like writing up a bug report with somewhat passable English spelling, grammar and sentence structure, or discouraging them from re-opening a bug 5 minutes after I mark it closed, saying that my checkin somehow didn't make it into last night's overnight build are great. Work with developers, and we'll work well with you.

J Z
profile image
As QA that works outside of the game industry (telecom) I can tell you that no serious QA professional would take a second look at the average salaries of QA in the game industry. Why submit myself to tortuous repetitive testing for minimum wage when I can be pushing the boundaries of technology for other companies and making twice the money with half the time spent?

I love games and game development, I would love to get into that industry. But the path would never be through QA... and the only way I could see anyone half-interested in those lay paying QA jobs is if their endgame is to be a developer... thus resulting in what the article said, good QA don't stick around.

As for general QA comments, there's always going to be scrubs, and those worth their weight in gold, even outside of the game industry in higher paying QA jobs. I think QA management is the real dark horse here and should not be discounted. I was very disappointed to see even the QA leads having subpar pay. I would look for the best of the best of QA management if QA salaries on a whole will never really approach those of development.

Joshua Sterns
profile image
I left QA in the videogame industry, and eventually found an entry level finance position for an ad agency. That one move from entry level to entry level doubled my annual salary. All of a sudden my college efforts seemed worthwhile, and I can actually enjoy life. What a concept?!?

Dave Malmberg
profile image
So true J Z. You could get a job as a QA Manager, which might compensate you properly. However, you might not like how QA is dismissed or constrained by the demands of Production. Best Practices are rarely followed here. Don't get me started on Documentation!!!

Christopher Marker
profile image
This 1000 times over.

The pay and treat QA like they are trash, and reflects in in the caliber of people who show up to the job, mostly on the entry level.

I am not exaggerating when I'm talking kids who this is their first ever job, and freaks who don't wash and store food in their pants. Not their pockets, but food...in their pants.

When you offer minimum wage, but McDonalds has superior job security, it shouldn't be surprising you don't get and keep good people.

David Phan
profile image
Tighten up the graphics on Level 3.

http://www.1up.com/features/top-5-game-design-college

Mitchell Cronin
profile image
There's a lot of good points in this thread.

While I certainly can't support keeping a huge 20-30 man internal QA team after a project finishes, I think there is a lot of merit to keeping a group of people full time depending on the size of the studio and how the work is spread out. While a lot of QA initially starts as 'random kid off the street', if you keep them around through several project cycles they'll develop skills that will eclipse anything you can get out of that random kid on the street situation.

In addition you can take the time to train them on other skillsets so they can do more than just test the game. Where we are at QA is taking a more active role in testing the internal development tools and outsourcing pipelines. This kind of work can bridge the gap between projects and gives more incentive to keep a portion of internal QA on after a project.

Still, the major issue is career path advancements and incentives. There are lot of good reasons to keep on QA and there's probably still many other creative ways to utilize QA that haven't been thought up yet. However there is little to no incentive for someone in QA to want to stay in QA. Ask anyone in QA and they will want the shortest path out. (These days that is getting much longer. With many universities offering game development degrees now QA isn't the awesome foothold in the industry it once was.) It isn't until they reach supervisor or manager levels that the salaries can even be considered an 'adult wage'. I am not arguing that freshly hired QA testers need a massive pay increase, but there needs to be greater incentives to stay in QA. For the testers that stick it out and build and improve their skills there needs to be compensation for it, and incentive to actually just...stay in QA. This will help filter out the scrubs from the standouts. You can essentially use someone's temp contract as an extended interview to really see their work habits instead of just taking a gamble after a two hour interview. Once into a full time role they need a respectable wage with opportunity to advance beyond just becoming a manager. As mentioned above there could be people with programming backgrounds who help test internal tools and pipelines, and are expected to do more than just hit the crash and post it. There can be focused compliance testers who know all the ins and outs of Microsoft, Sony and Nintendo's rulebooks. You can have tech art focused testers who do more than just point out the holes in the world but know the ins and outs of how a streaming system behaves. You can even have QA design and write testing tools to help automated some of this process. The best person to write QA tools is someone who's had hands on experience testing the products they work on.

There are lots of possibilities for QA. No, it doesn't make sense to keep all of them around, but there is a lot of merit to keeping small teams on to build skillsets and knowledge. As a lot of you have mentioned one trained and awesome QA tester is worth way more then piles of kids on the street. The position does have a low barrier of entry, but once someone has proven themselves why not offer a legit career path with adult wages beyond just becoming a manager?

Joshua Sterns
profile image
Great article. Glad to see attempts at generating awareness. Personally, only my time in retail was worse than my experiences in QA.

I don't have the answers, and my experiences are limited to a two year period. I do have some general suggestions that may be fun to discuss.

In regards to the difficulty of hiring QA. Change the interview/application process. Have potential employees write a number of paragraphs/pages. Do more then ask if they like videogames. Find out if they have a college degree, or if they have a writing hobby. The application processes I experienced were a joke, and did little to indicate what my abilities were. Attracting top talent is also difficult when the pay is so low.

Keeping QA around full time is too expensive. Then diversify the job description. Have QA do more then test. Is there really nothing to do for this department before the home stretch of development? Also having QA in at an early stage may help prevent some of the buggestions. Many of the buggestions I saw were a result of a lack of communication between the devs and QA.

I also have a hard time understanding why QA is too expensive, but paid time off, bonuses, benefits, etc. for other departments are not.

Overall I believe many of the issues with QA stem from the hire/fire cycle, low wages, and lack of respect from both sides. Address these problems and discover what a good QA squad can do.

Evan Newton
profile image
Sup Josh :)

There are newer companies starting to see the light. Oddly enough, much of the software industry outside of games has had a pretty good grasp on how to make QA the best possible for a number of years. Games, for the most part, are behind.

I will say, however, I currently work at a company where QA is absolutely valued and granted a seat at the table from the beginning. Our games are rocking that much better for it. (We also pay closer to software industry standards as opposed to game industry).

Joshua Sterns
profile image
Hi Evan ^_^

Nice to hear you found one of the good ones. If I had more positive experiences, like working with Michael Saladino at Pandemic, then I'd probably would have stayed in the industry. Still glad I got out, but also happy to hear old colleagues are doing well.

Dave Malmberg
profile image
@ Evan: Where do you work? Got any openings for a QA Manager/lead? :)
@ Josh: Totally agree with you and when I'm hiring employees (QA) I'm looking for one's where there's a passion for the work. I agree with Brandon as well, low pay and no career path is a detriment to some degree. I have to say, regardless of where I end up in the Games Industry, my time spent in QA will only strengthen my skills as a developer. It should be a 3 month requirement for all employees in the industry.

Evan Newton
profile image
@Josh

Oddly enough, I kinda feel Michael Saladino doesn't take it far enough in many areas. He definitely has some amazing viewpoints, and QA at Riot Games has been that much better for it. But Riot still has many issues that other industry leaders have (i.e. lower pay and longer term contracting, for starters.)

Evan Newton
profile image
@Dave

Backflip Studios. Find me on linkedin and message me, I'd rather not put my contact info on here.

Dave Malmberg
profile image
Awesome Evan, I was sort of kidding but it never hurts to learn where people (&QA) are treated well. BTW I totally loved Army of Darkness Defense, my kids and I have fun playing Ragdoll Blaster, Strike Knight and Paper Toss! Great work!

Jeremy Tate
profile image
So, I've been working as a Tester for 13 years now. (I'm not a fan of the term QA, since we don't assure quality, that's the job of the entire team). I find a common thread is that, even with teams where you have a stated respect for Test, few developers know what to do with a tester early in the process.

Personally I find I have to constantly be vigilant about progress the team is making, and if I'm not a part of conversations early in the process, I try to insert myself and provide as much value as I can. However, it sure is nice to work on a team that has committed to having that customer advocate early in the process.

I was lucky that I learned my craft at a publisher which was part of a large software company, where Test is expected to be its own career path. That's not saying that you can't move on to other areas from Test, but if you really want to focus on having more and more impact as a tester, a clear avenue is there to follow. I've since left that company and found many places that don't have a good attitude towards Testers, if I set the expectation early on that I'm not there to move somewhere else in the company, I get a lot more respect and it helps change the culture.

Heinz Schuller
profile image
As a developer, if you're not making QA a core part of your development team and treating them as such, then you're doing it wrong. A well-run QA department is an asset at nearly every stage of the development process. The key to this is working with QA directly to define & set metrics cooperatively. Then they can help you do your job better when they know what to look for.

QA can run build-verification tests on your tools to help keep your team at peak productivity. They can do competitive research on similar products and provide you with a wealth of data. Given the deadlines we all work under, it's fair to say these folks have saved me countless hours and saved my butt on numerous occasions.

Adam Dials
profile image
I was a Test Director in the military simulations community on a sim called OneSAF. In the military, QA is treated with a significant degree of respect (partly because some of the testers are retired operators who can kill you in a variety of ways :) ).

Iím a long-time gamer and modder and hope to one day work in the industry. No insult is intended here and I genuinely respect the work the industry does. But the impression I get, which may not be correct, is AAA game testing is haphazard. It seems throwing large numbers of enthusiasts at the game is the norm. Is this the case?

In military simulations, test engineers examine the sim at a number of points. First, we evaluate designs. Before a given design is implemented, we conduct a design review to focused on several things, like requirements traceability, realism, and usability. If we say a given design is not a good idea and present a cogent argument supporting our position, the design may be modified. This heads off problems before significant effort is invested.

Next, our test engineers write test cases. The lead test engineer does a traceability analysis on the requirements, comes up with a test plan to ensure every requirement is examined, then assigns the construction of those test cases to test engineers. Test writing is divided between areas of expertise. We have technical test engineers, functional test engineers, usability folks, etc. The technical folks write load tests, system and platform tests. The functional folks look at combat systems and mechanics. And so on. Before the tests are executed, all of them are scrubbed - the development team gets a look at them as does management to help in getting the tests focused correctly.

Finally, our test engineers execute the tests. All the engineers step through their tests, and the test lead compiles the results. A management board convenes and categorizes the bug according to severity with developers and testers in the room. Funds/effort/time is then allocated according to severity and ranked importance of the requirement.

Is this level of rigor employed? Some of these testing techniques have to be used, but maybe you guys do it on the developer side of the aisle. Any perspective would be interesting to hear.

Jeremy Tate
profile image
It depends on where you are working at. This is not unique to the games industry, I've seen plenty of enterprise applications that don't provide the same rigorous testing methodology that you describe.

However, in general, because games are bought and sold on their fun factor, over their stability, you will see testing methodology that is much more flexible. I didn't see a true spec that I could test off of until about 3 years ago.

Michael DeFazio
profile image
Does anyone think Obsidian and specifically the QA department who tested FONV did not raise flags because of the games bugginess before it was shipped? Playing the game normally (not trying to break anything like a QA engineer would do) had me experience terrible freezing, corrupted savegames and and serious companion AI issues (to the point of breaking the game twice)... and I only played the game for 40 or so hours. Everyone I know who bought the game day 1 had almost exactly the same issues or worse. (How could they not have known?)

Smart money says they released the game KNOWING it was broken. (And I have heard that Bethesda basically moved up (rather than back) the release date to push the game out the door, even though the devs knew it was a mess).

I know people might think that I am putting the blame on Bethesda here, but to be frank, I waited and waited for patches to fix these issues and they took WAY to long... I never finished the game because even after some of the first patches, I still got freezing issues and corrupted save games... (Loosing 40 hours of progress was a huge turn off so I eventually gave up).

The idea that Obsidian's QA department didn't recognize the problems with the game's bugginess which contributed to the low metacritic score seems like a farce.

Sean Conrad
profile image
Does anyone actually believe the average QA tester is making $45k a year!?

Jeremy Tate
profile image
Average? yes. This salary range does bunch up both the "Thumbs tied to a controller" tester with those who bring a technical skill-set. Personally, I make much more than that, but I come with technical knowledge and a lot of experience.

John Caminiti
profile image
Maybe senior management. Most testers start around 10-12 and hour. Whenever I see that number I laugh.

Joshua Sterns
profile image
I wonder if that includes temp workers.

Brandon Sheffield
profile image
Sean, that average is for salaried QA, not contract workers.

matt landi
profile image
As someone previously mentioned this is deceiving since it's the average and not the median. Another thing, is that salaried QA are usually in the upper levels(mostly management) of QA. When I was a Senior QA Lead I was hourly.

Christopher Marker
profile image
I cannot agree more. I worked for a while as a QA tester at a major publisher and I was never made to feel like a valuable part of the development process. Instead I felt like worthless badge trash, and was seeking another job frantically. The pay wasn't even enough to pay my bills living with 2 roommates in a dump of an apartment. Quite literally the cheapest in the area. The minimum-required-by-law benefits were a joke, and cost 20% of my weekly paycheck that was already too little to pay my bills. I don't know of a single person who actually took any of them. Then add in the fact that I had to worry about being let go every couple weeks, and knowing I will be let go no matter how well I do, it was just a terrible experience. Honestly you have the same pay, and better job security working a cash register or flipping burgers. To move up out of associates tester was something like a 5 year process, and only happened if you displaced one of the full career supervisors.

We didn't come within a 100 miles of the Devs. Hell we would be fired, possibly sued, if we did speak to them personally, since we didn't have clearance to enter that part of the campus. It as standard operation assumption that QA testers would steal IP, cause leaks, or even try and steal equipment.

It was honestly only the job some kid who lived at home, who didn't know any better, could even afford to do, and unsurprisingly, most of the saff was 18-20, lived at home, and this was the first job. And the supervisors, the only true "full time" guys didn't test, just yelled as us to not talk all the time. And of course working with "summer job" types who didn't have a clue, some of whom didn't wash regularly too boot, was just as poor as being treated like trash by the company.

Of course they wouldn't have to have treated the staff like children if they had compensation and working conditions that could allow actual professionals to even survive at the company. I feel like at that company, they might as well have tired prison inmates to the do the job, since that is how much they valued and trusted their temp QA testers.

Matthew Cooper
profile image
Sean, it refers to average in the mathematical sense. You're probably reacting to what you perceive as the median or mode.

Michael Liaw
profile image
I've worked in QA for 6 years now and I've been treated quite well at my company. I find that treating QA as QA and not as QC (Quality Control) has made all of the difference. We aren't constantly in the game playing it for our job. We are also analyzing the design docs, working with developers on specific issues they have, and generally supporting the other developers in making the best game possible in whatever way we can.

One thing that makes it hard for QA to be treated well and looked upon as a profession, not just some high school dropouts, is that most companies do not bother trying to improve upon their QA processes it seems. The game industry as a whole tends to think getting enough people tossed at a game playing it will guarantee that all bugs will be found and ensure that the quality of the game is good. This is definitely not the case, regardless of how much time you have to fix the bugs found from doing this. If people are able to find bugs before code is generated or content has been developed, the savings are immense and this is where I find QA is the most useful, specifically in the early stages of a game where most people would rather not have QA.

Hopefully the videogame industry as a whole can grow to better understand how to effectively use QA within a game development cycle.

John Caminiti
profile image
I worked QA at a major publisher for three years and it was a terrible experience. We were treated like poorly by the company, the pay was unlivable (I think the most I ever made with overtime was around 25k a year) and employees in other departments called us the monsters in the basement (the QA department was kept in the basement). At this company, no matter what your background and education was, once youíre QA, you are only looked at as QA. People were desperate to move out of QA but, the only people that moved out were the ones that had the right friends, or it was just pure luck. And if you moved up in QA or became too good at your job, you wouldnít be allowed to move up because you had became too valuable. Most testers are kept as temp employees through a temp agency, so they can easily be fired after the project ends (next time you finish a game, watch the credits and watch the large number of names under Volt).

QA is definitely not the correct term for the job. There is no assurance of quality that is performed, even though a test team is the greatest focus group you can have. The test team (QA group) is an incredible resource for feedback and focus testing but is looked down upon so much that they are completely dismissed. Apparently itís a much better idea to get a few teenagers to come in and pay them each $50 for two hours to focus test the game, then it is to get feedback from a group of people that know your game inside and out (and at $50 for two hours, focus group kids make more than double then a testers hourly wage).
You are paid to test the game, if you try to make a suggestion to better the quality then you are shot down because you are only a tester and itís up to Production or the developers to worry about the quality. All the testers really do is to make sure you donít fall through the map, make sure the menus and gameplay work, the game can be saved and the game can be completed. Oh and we also have to correct the mistakes written by testers from outsourced companies in foreign countries (they may be cheaper to hire but the quality of these outsourced foreign groups are terrible, so if you get a really buggy game, the testing may have been outsourced).
Our QA group worked with Production, which is a terrible idea, we didnít normally have direct contact with the developers, Production was always the middleman. Every bug we entered had to be approved with Production before it was sent to the developers and there are a lot of bugs that would not be sent out to the developers, usually due to trying to hit a ship date. I came to loathe the term ďWill Not FixĒ (WNF).
QA is a terrible place to work for the most part, and itís not at all a good starting point to break into the industry. Companies will hire absolutely anyone to test and pay someone a terrible wage and let them go when a project ends (although if publishers would learn to not release all of their titles in the last 4 months of the year, and spread them out throughout the year, you can have better groups of testers working on your games constantly). QA is looked down upon and treated with disrespect even though itís such an important part of the game development process. I have seen companies/publishers hire kids right out of college for Production positions and would not even consider people in QA, even though people in QA are far more qualified. From my personal experience, the only people that have respect for the testers are the ones that have been testers themselves.

Joshua Sterns
profile image
Sounds like a company that rhymes with vision. ha ha sigh

Mathieu MarquisBolduc
profile image
It may sound obvious but you need to have several levels and types of QA, just like there are several levels and types of programmers.

Game Tester, Development tester, Support Specialist, are just a few types.

Your entry-level game testing job will never be great, just like entry-level programming sucks. The important thing is that people have opportunities to grow and move up within their skill set. In all my gaming jobs testers (QA, whatever) HAD a career path, should they have the skills and will to climb it. Maybe its more of a Canadian thing? I dont know. I know at least one studio lead that started as QA...

Michael Rooney
profile image
I think the president of Bungie was a QA guy at one point.

Part of me gets the feeling that a lot of the people complaining about bad QA positions think all QA positions have the same job; many are only loosely related. There's no reason to pay someone twice minimum wage to make sure nothing is within an inch of the edge of the screen on all your menus. There's plenty reason to pay someone much more than that who knows when 20 bugs are related to a single cause with the minimum amount of steps required to repro noted as common to all 20 cases.

I totally agree the former is a shitty job, but we've all (I hope) had shitty jobs at one point or another. Many of us I'm sure were totally overqualified for those jobs. There is grunt work to be done everywhere, game industry not excepted.

Jason Carter
profile image
"Your entry-level game testing job will never be great, just like entry-level programming sucks."

-What entry level programming jobs? X-D I have yet to see an entry level programming position in my searches, but i suppose they must exist >_>. Doesn't help that most of my programming knowledge is self taught D: Stupid International Business Degree.

Fiore Iantosca
profile image
I've been in software QA since 1996 in many different software shops. It's the same everywhere.

QA does not get respect and when projects need the dates moved, QA ALWAYS ALWAYS ALWAYS SUFFERS.

Kym Brainard
profile image
I will admit that QA is generally disrespected by members of the Dev team, and even worse by producers. You'd not believe what I hear beyond my cube wall. They also do need to be paid better than what they are. However, many aspects of this article, I cannot agree with, and think are over-exaggerated. In the end game development is a business like any other. It's all about supply and demand. There are less spots on a team for QA, and far more unemployed floating around, and it's also a job you do not need to have a college degree to do.

Most QA is brought on towards the middle of a project because that's when there is actually something TO test. They're let go at the end because, there's nothing for them to do. You can't seriously believe that small to midsized studios, that only work on 1-2 projects at a time, can afford to keep them on the payroll, while they sit around catching dust. For that matter, at most studios, many artists, and programmers are let go towards the end of a project as well, because they're not needed in pre-production for the next project.

As far as 'nowhere to grow into,' that statement also not fully sound. The rest of the development team generally has a college degree, or some kind of official training. Albeit, there's more to QA than 'just playing games.' FAR more! But in the end, it's a job that you are capable of hiring someone out of high school to do, and with several weeks of training, they'll at least be suitable to hold their own.

If you, as QA, have a desire to go beyond that job title, then you need to do as the rest of us have, and either train yourself, or go to school. Heck, half of my studio is filled with Designers, and Producers, that started as QA... It's not impossible. If you feel the desire to stay QA, but get paid more, then you should still get extra training into say, programming, scripting, etc. Make yourself someone that the Leads feel the need to keep around, even during downtime, because you're the only one who knows how to set up great test units, or can look through code to give actual analysis over something that's broken, etc.

Tony Forte
profile image
QA and devs need to function hand and hand. Period. If it's production QA vs devs, or internal QA vs devs, whatever... there is no progress made on either front. Mistakes are made on both ends, and if there is no professional respect for each other, including QA respecting devs, negative progress is made.

Devs need to accept QA is here to point out their mistakes. Suck it up you need to re-design something or whatever. You are paid to develop over a period of time, not to make a set of features once then dance around QA feedback.

QA needs to deflate their giant ego and focus on technical integration and improving their own skillsets. Become something where "kids off the street" are only used for focus tests and full time/long term contractors can design, run and execute game-breaking test scrips and ad-hoc strategies. Pay increases can not be justified for un-skilled labor.

BUT publishers or devs also need to see the value in someone who gets paid to play games, broken or not. Qualitative feedback should be taken at face value, but look at the face who is giving it to you. Literally a random person or a video game specialist.

Aaron Haaf
profile image
An interesting takeaway note from all the Game Production Workshops (6 months for a game in Unity with several classmates) in my school has been that QA immediately beginning development has made a large difference. All of the "successful" games have made a huge push for internal and "recruited" user/playtesters.

One of the biggest things has been the large amounts of feedback for the design of the core mechanics. It might have been the inexperience showing of the teams, but it seems that people of any discipline have a hard time of seeing what's wrong right in front of them. A platformer that was pushed out last quarter that I had the pleasure of testing, had it's controls VERY clunky, and this wasn't addressed at all despite many complaints from these QA'ers. It really did hurt the rest of the game.

Abel Bascunana Pons
profile image
I think QA is closely related with game design. QA detects issues after they have been implemented. A game designer should see potential issues before they are implemented. That's why a good QA guy can become a good game designer if he has some capacity of abstraction and a great deal of experience playing games from all genres.

About all the QA stories we've heard of and lived through personally, i can only say... a QA guy must inevitably end learning the meaning of the phrase "estoic calm" if he wants to be happier with his job. Sometimes we may think our observations are too right not to be taken into account, but making a game is not as easy as saying "change this or that", and involves the coordination of many busy people in different departments. To top this, deadlines are really tight and many QA efforts end up down the drain. We must live with this.

Mathieu MarquisBolduc
profile image
I wrote some first, then I deleted it, because its so easy to read these things the wrong way. I love the QA on my team, and their role is important, so please people, read this with an open mind, because I think its important.

What you seem to describe is not QA. Its playtesting. QA find execution issues, and playtesting find design issues. Now the whole team, including QA, necessarely playtest the game to varying degrees, and all constructive feedback *should* be said and heard. But QA's role is not playtesting, because while they overlap a bit, playtesting is VERY different than QA, and should be done properly in its own right.

*Again* Im not undermining the role of QA here, which is very important, just that producers shouldnt rely on QA for playtests and that QA shouldnt expect to fulfill that role either; its not the same thing. There are paying jobs in playtesting too, but usually they observe and manage the play tests, rather than play it themselves.

Daniel Martinez
profile image
There was an article on Gamasutra a couple years ago that said if QA ever unionizes, it will not be because of pay, working conditions, or the like, but because of credits. I worked on a couple of projects where no one in QA was credited, in one project even the development team and admin was not credited, only the licenses that had been acquired were given credit and that was to avoid a lawsuit.

Jeff Lindsey
profile image
On the related topic of "why games release with bugs", in my experience, it starts with the publisher or client. They want more features and fast progress, and force teams to accrue massive technical debt in order to meet their expectations. They're not held accountable during the payoff of that technical debt (team crunch, KS'd bugs etc), so they have no real interest in addressing the bigger issue (until buggy releases can be quantified in revenue lost). In fact, nearly every contract I've worked with explicitly assigns technical debt to the team - "Alpha: all features complete, placeholder art and bugs exist".

I'm a big fan of constant visibility on technical debt and constantly paying it off. I would rather be stuck having to justify doing less (but doing it better) to stakeholders than setting my team and customers up for the inevitable mess.

Roberto Estrella Froment
profile image
I heard amazing things from Nintendo's QA working in Pre-production and along with the design team from beginning to end, does anybody have an article about it to support it?

Chris Gallant
profile image
Having been on both sides (internal vs. external QA, as a partner in development vs. a walled-off tester/manager) I can relate to the ideology of the OP.

QA is a thankless job at times. You have to deliver the worst news at the worst times, and your well-timed advice can go unheard. Testers can often go unrecognized when a solid product is released. All things mentioned above in the comments.

In my experience, the clients we work for who have the best ROI:

- Engage QA early and often (doesn't need to be fulltime)
- Employ many methods of communication (IM, phone, etc, all very important when QA is remote)
- Respond to the needs of testers (can't tell you how many times I've watched a QA lead scramble without build notes, debug, or bits of guidance/info- a lot of wasted time spent testing "everything" instead of what matters here-and-now)
- Recognize the importance of exploratory testing (just because it isn't scripted doesn't mean it was random, unstructured, and useless testing - the tester is following a schema in their head).
-


none
 
Comment: