Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 20, 2014
arrowPress Releases
October 20, 2014
PR Newswire
View All
View All     Submit Event

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

How To Not Lose Money On External QA - And Have The Best Productivity
by Rad Smyk on 04/09/14 09:50:00 am   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.


This blog post is a bit of an addition to a post by Anna Jenelius, and I suggest that you really should read it first. The post is also a bit of musing on topics that disturbed me when I was working as a QA tester. Obviously, no connections should be made between my personal opinions and my former employer's, as they are completely separate.

I have been working for almost 3 years in an environment which is hardly comparable to the one described by Anna. It was an external QA, located in Poland, which was a place for developers to outsource purely "bug-hunting" tasks. We had different sorts of clients, ranging from the biggest players to the small indie studios. All of them made common mistakes - some of which are easily avoided. So, if you want to outsource your QA, get the job done, and not leave some people with an impression that you are a terrible company, follow my advice. 

1. Do not be late with your builds.

I cannot stress this one enough. If you say that the build will be ready for Monday - it should be ready for Monday, 8AM in the morning. Because your testers will not test the build, when the next one is due in a few hours or a day. Nobody wants to reproduce all freshly found bugs a few hours later, to make sure that they will not go to trash immediately. Instead of that, people lose focus - they explore your game, wander around, but do nothing productive. This is losing time and money.

2. If you're in alpha - add things!

This is especially important when you are in the early phases of the development. Your game is boring and ugly, and people who test it are staring at the grey screens filled with geometry for 8 hours daily. And there are no features added properly. And there is barely anything to test, because the system is still so simple, that all interactions can be explored really fast. And what happens then is that testers are losing focus - and you lose productivity and money. The solution is adding new content to the game, bit by bit. Testers love new content. If you are in alpha - add anything, but add something. They will test it thoroughly and creatively. Leave them for a month with builds that are exactly the same, and you will lose them.

3. Fix the bugs. Really, just fix them.

I am not sure if this should not be the first. Imagine a situation, when you work for one week, producing reports. Then the second one. And the third. And you reported a hundred of bugs, including some of big importance. But they are stuck as "in progress", somewhere in the great, deep void of "that damn dev team". This usually happens when you use lab's bugtracker and your own - somebody just pastes the bugs into your tracker and they get processed there. But the testing team does not see that. And at some point, suddenly, nobody cares. Because if you, as a developer, do not care about testers' work - they won't care about yours. 

4. Maintain your bugtracker.

That’s another “obvious” one. If you have a properly configured bugtracker, it does everything by itself, right? But apparently that does not happen so easily. If you work with an external QA, please have some things prepared. Summary standard, what exactly should go into the description, proper version control present in the bugtracker. Also – react as fast as you can to the reports. They will have flaws, and if you do not get rid of them quickly, they will be reproduced in another reports. That’s because for testers every project is different, and sometimes they switch projects 4 times a week. Things can get confusing, and if you do not remember that, you will have a huge mess. And lots of duplicates.

5. Keep testers informed.

This is probably the most overlooked one. To have typical bugs found, you do not have to do this. Game design doc will not change anything in terms on finding flickering textures. And your internal QA will handle the weapon’s balance just fine. But external QA always has the drive to ask questions and post suggestions. And there is nothing raising morale in a team better than an e-mail addressing even the weirdest of concerns, or a thoughtful comment on a suggestion, even if it is rejected. These are the easiest ways to keep in touch with your remote team, and to connect with them in a way that will make them involved and caring. And if they care – you get a superb QA.


All in all - external QAs are the trenches, and we all know about it. But if you, as a developer, change some things, you can change your relationship with your testers dramatically. And believe me that they will work hard to test your game, given that you show that you want them to. Because testers are usually intelligent and passionate people - most of them could be somewhere else, doing other jobs, earning more cash in better conditions. But they love playing and testing, and exploring the boundaries of your systems. Just let them do that more easily.


Related Jobs

Yoh — Vancouver, British Columbia, Canada

Build & Test Engineer
Nix Hydra
Nix Hydra — Los Angeles, California, United States

Nix Hydra
Nix Hydra — Los Angeles, California, United States

Game Designer
Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States

Senior Sound Designer - Infinity Ward


Chris Watkins
profile image
The main thing that I would add to this is that the external QA team needs a good lead. That lead can help both sides a lot by working between the two teams to raise issues, highlight the bugs that are affecting test the most and getting information to distribute to the team.

Pretty much any external team can only be as good as the 'face' of it. If that lead is not working hard for both groups, there's not a lot of hope for the relationship.

Stephen Etheridge
profile image
I agree with all of your points except #3, which is quite naive. Just because a bug isn't immediately fixed, it doesn't mean that the developer doesn't appreciate the work of the tester. This is quite a selfish and uninformed reaction to take.

Severity does not dictate when a bug should be fixed; the Priority does. If a crash only occurs on the lowest 5% of system specs, it's not going to be fixed early in the development process. The internal Priorities are not always logged and if they are they aren't always updated quickly; in my experience that's an area of communication that could be improved, certainly.

Furthermore, just because a bug was found recently, it doesn't mean it will be fixed soon. Although, I do agree that the testing schedule of the external test team is not always set to be in line with the areas of development the developers are working on. I find it works best to try to map one to the other, so if you know that the developers are going to begin a polish pass on a level in the second week of November, you schedule the external test team to do a full collision, surface types and art fidelity pass in the first week of November.

So, perhaps instead of demanding that all bugs get fixed immediately (which misunderstands the very fundamentals of what QA testing is about), Point #3 could instead read "Communicate the bug fix priorities effectively to the external QA team".