Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 24, 2014
arrowPress Releases
October 24, 2014
PR Newswire
View All
View All     Submit Event

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

Part 4: “Adopt, Adapt and Improve” – Agile QA BioWare Style
by Tulay Tetiker McNally on 04/30/13 06:18:00 pm   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.


“Adopt, adapt and improve” was the motto of the Round Table, a social networking and charitable organization for men in their 20s, 30s and early 40s, founded in Norwich, England in 1927.

Its motto was to bring professionals together “around the table, adopt methods that have proved sound in the past, adapt them to the changing needs of the times and wherever possible improve them!” Sounds familiar? 

If you work in agile development you know what I’m talking about. This basic principle of ‘Plan-Do-Check’ is part of our core BioWare QA Philosophy. Lisa Crispin and Janet Gregory wrote a book in 2009 called “Agile Testing” (published by Addison-Wesley) which provides some guidance in agile testing. I was surprised to read that the core QA principles we had been following at BioWare were actually recommended standard in the software QA community. It was surprising, but also encouraging in a way.

In this context: Our 10 “Agile Testing Commandments” are:

1. Deliver value to the customer  
QA in our studio focuses on defect prevention, enabling/unblocking developers, and providing valuable business intelligence to leads and decision makers in the studio. QA will also communicate important metrics and data about test coverage, game and system stage and quality to applicable development customers and stakeholders. A strong QA department will give a studio confidence that the product will meet the standards that are expected by the developer and most importantly, by the end-user.

2. Provide continuous feedback to the team
Continuous feedback should be provided to the producers, developers and your own QA team. Good feedback is about risk management paired with root cause analysis; it is about keeping the communication/feedback channels open with all relevant teams. The feedback provided, and its visibility, is critical for the success of all teams involved.

3. Enable better communication
In many ways QA acts as mini scrum masters and as facilitators, because production and QA are probably the only two departments that talk to every other development discipline; they need to connect the dots and always keep the big picture in mind. In this role QA helps to ensure better communication between developers and ensures that no miscommunication exists regarding standards and quality expectations within the team. 

4. Have courage
to raise risks to quality. QA is neither a gatekeeper nor should they be a bottleneck. We provide information to development stakeholders who truly own quality. Feedback helps development to reach their goals and to react more effectively. QA also needs to advocate for the end-user as much as possible, even if that means we ruffle feathers, we often help to keep developers on their feet :) 

5. Keep it simple! 

6. Practice continuous improvement 
Inspect, adapt and maintain practices and processes - the motto of the round table!  

7. Respond to change
There is no "one solution fits all" model. We will do what is best for the project and adapt current strategies and come up with innovative ideas wherever there is an opportunity or a challenge to be resolved. There is one thing that is certain in game development: change. As QA we aim to always stay adaptable in order to react to change quickly and without (m)any casualties.    

8. Be self-organized
At BioWare our QA is encouraged to develop new tools, processes and skills to improve work and workflow for QA and for development. Development of new tools and automation is not limited to QA Programmers - we train our Analysts to be able to detect these opportunities.

9. Focus on people
QA often ends up having to define and figure out what a particular feature is intended to do - our wealth of experience across a variety of titles helps us to achieve this. 

10. Have fun!
Evaluating the fun-factor of a game can be a real challenge, but at the same time it can be a lot of fun! QA is a department that is always in the product, constantly using it and seeing it change. QA also often sees the worst sides of a product, so keeping a sense of humour is often very useful and healthy, even if only to keep the developers spirits up and to keep them focused. If work becomes too repetitive without real challenges, we can run the risk of becoming complacent and will eventually make mistakes. 

My secret 11th commandment and our QA Department Mantra is: Keep calm and continue testing. Many of our developers genuinely appreciate the fact that our QA can stay calm and doesn’t panic despite the fact that QA typically sees the worst side of everything; this can give developers the confidence to focus on making awesome games!

To quote Clinton Keith, who introduced scrum to game development 10 years ago, “scrum is about people over processes - just like a good leader of creative people is more like a gardener than a process mechanic”.

QA teams at BioWare stay agile to the point that the organisational structure depends very much on what the project teams need and how it is structured. Yet our main goal remains the same: to ensure that quality is built into everything developers are working on. 

During prototyping and pre-production of our games, our analysts are mostly involved in the verification of systems since there are not yet many assets to test. These analysts are typically testers that we refer to as “Tech QA” who have an understanding of the systems that are used to implement the content. 

They identify issues in the tools, systems and workflows and mostly support the programming teams. We also have analysts that have expertise in what we call Content QA, focusing mostly on Level Art, Level Design, Cinematic Design, Gameplay and Narrative.

QA Analysts are often embedded with the scrum teams, with one analyst sometimes supporting multiple scrums. As the project starts to progress, more content comes in to test and more platforms start coming online. The growth and complexity of development often requires more testing of added features and developer tools; more regression testing is required to ensure newly added features have not broken anything.

We made the decision that during production it is more efficient to move QA into a pool system to even out the workflow. In the current development of our products we are actually experimenting with different versions of embed and pool models or different team set-ups and techniques such as Rapid Software Testing.

We want to converge our QA processes between our franchises wherever we can and the benefit of running different franchise game-teams in parallel is that we can learn from successes and failures of the other game teams, quickly apply these lessons learned and continuously help our organisation and people evolve.

There is just too much to talk on the subject of the different types of support and specializations we have in our studio (in the context of RPG-QA) that I will not be able to cover in a single article. So, stay tuned and watch this space!

I like the analogy of comparing testers to Military Intelligence, leading people away from seeing testers as gatekeepers (QC vs. QA). In many ways the objective of a tester and an intelligence officer are very similar as it is implied in our mission statement (read Part 2 of this series). 

Testers are tasked with providing information to developers and production to do their work better and to make the correct decisions. Just as the Intelligence Officer is not tasked with fighting the enemy in close combat, testers are not tasked with fixing a bug in the code as an example. 

We help the developers on the front lines and not being a developer doesn’t mean that QA does not have an intricate and highly technical job!

QA often has to think about and simulate strange but realistic scenarios where customers will be using our product in unusual ways. We often have to work with incomplete information, making assumptions and explaining the risks of situations that may or may not happen in the wild.

Personally, I know my team prefers the ninja analogy to that of Military Intelligence, and they like to consider themselves as a "network of ninjas" who work in support of developers and producers of the product. My QA Ninjas have mastered knowledge of a product’s systems, its content and tech goals and report on both the functionality and the subjective quality of the product, enabling the rest of the team to make decisions with the best information possible.

Let's take the pillars of game design for an RPG as an example. Imagine, that exploration, combat, progression and story are all gears in a machine - if one of these gears malfunctions, they tend to grind (for example when a story element is missing, the player can't progress). In this case, Quality Ninjas act as the lubricant that makes those gears turn more smoothly. QA support exists at every level in our studio.

Predictability is important to guarantee success, especially during production. In theory you should be able to ship a quality game earlier the more knowledge and experience you have, therefore knowledge is the greatest asset your studio can create! 

 Keep Calm

My blog posts so far have been from the QA perspective. In the spirit of partnership that we are talking about here, I asked some developers at BioWare Edmonton to share their thoughts about our embedded QA and I would like to conclude this entry with some of their responses:

“Embed QA is a good fit for the agile approach that we try to take at BioWare, where an interdisciplinary group commits to delivering a feature. The verification of that feature should be considered part of that work, and not as an afterthought. I think that embed QA is a key strength in how we approach game development. If anything, we should be pushing farther with it.”  [Technical Director]

“Having QA team members who are kept throughout development over multiple projects has created a field of QA “experts” in many fields, including Audio. These people know what they are doing and our review scores reflect that. They don’t waste our time, and genuinely care about the art of game development.”  [Lead Sound Designer]

“QA offers reporting on our bug lists, immediate support for things like build failures, and can provide suggestions for improving the game that go beyond just fixing the actual bugs.”  [Technical Director]

“I believe that QA provides an objective viewpoint over the project as a whole. As our projects tend to be so large that we often will only have visibility over certain areas, usually the ones that you are working on and the transitions into and out of them, QA provide the missing link. As well, I appreciate that they are not “married” to the content, as they had no hand in producing it. This better sets the QA team up to appreciate and critique the project as a consumer, rather than a content creator."  [Audio Artist]

PS: And here’s a video for those of you who didn’t get the Monty Python reference in the title :)

Related Jobs

Red 5 Studios
Red 5 Studios — Orange County, California, United States

Graphics Programmer
Red 5 Studios
Red 5 Studios — Orange County, California, United States

Gameplay Programmer
Gearbox Software
Gearbox Software — Plano, Texas, United States

Server Programmer
Forio — San Francisco, California, United States

Web Application Developer Team Lead


Kirtan Bhatt
profile image
QA is a luxury that pays great dividends when implemented in the way you state. As an aspiring designer, I wish I will have access to such a team or similar minded people in future teams: the smaller your team the more hats people wear. As I gain more and more design experience, I will appreciate such team members even more because in my limited capacity I often find myself asking questions that your QA team regularly addresses.

Clinton Keith
profile image

These are fantastic articles. I was looking forward to this one and you didn't disappoint. There are a lot of key points in here (QA vs. QC, embedded QA, etc).

Bioware was the first studio I visited as an independent trainer years ago and it was an eye-opener about what a great development culture can look like.

Thanks for sharing this with the community!

Eric McVinney
profile image
[side topic] Truth be told, there needs to be a "QA" title (or those related to it in hierarchy terms) set in stone in this industry. As in, no more "Beta Tester" or "Gome Monitor," just QA. And from there it would be Assistant QA Lead, QA Lead and so on and so forth. Some sort of standard that the industry keeps on messing up :/ [/side topic]

Going back on topic here... Your article is fantastic and has some well-detailed topics, as well. I'd really like to see some vids on such processes.

John Trauger
profile image
I'd like to see how this works in practice as this article is all high-level stuff that could easily disintegrate when the pressure is on. When the process wants to break down is when it's needed the most.

I'd love to see how Bioware handles it when the marketing's timetable for the game says it needs to ship and and QA says "not ready". I've been a lead QA and sat in on meetings where "ship or not" was discussed and "ship it anyway" was thick in the air. I'm just not sure if that's *my* experience or a common industry experience.

Eric McVinney
profile image
Nah, that has happened to me, as well. I always got that itch to say "It's never gonna be as good until we fix it!" during the meets, but kept my mouth shut... usually >:)

Jason Cuffley
profile image
Do you think that outsourcing QA is as good as internal QA? What is the ratio of Devs with internal teams or Devs with outsourced teams? I am a QA tester for a outsourced business and I was just curious.

Keith Meyer
profile image
Outsource vs internal QA, quality match up. That could be a blog post unto itself, but ultimately it depends on the engineer and on the management. Good people are always good and should be leveraged, no matter where they are. Outsourcing just adds a layer of communication that has to be overcome and time synch up that has to be factored it.

The ratio question is a good one...I have never seen a good number or one that made sense. To me it always comes down to budget, get as many quality QA as you can afford. At the end of the day most companies are working on a budget.

Lastly...great article. Honestly the 10 commandments could be simplified, but then less catchy:)

Mitchell Fujino
profile image
When I worked in Bioware QA, we tossed around numbers like "One internal QA is worth 20 publisher/outsourced QA".
While that number is hyperbole, there is a huge difference in: access to the developers, having a working relationship with them, experience in the how the pipelines work, knowing the processes the developers use, and even just having a stake in how the game turns out.

An experienced and well trained internal QA would often have a 4x higher fixed bug rate, with higher severity bugs found, far less duplicates and "will not fixes", and much faster turnaround time.
That doesn't even take into account differences like the agility of an internal team to deal with surprises.

Working with publisher/external QA also has a social difficulty that doesn't get talked about much. Overcoming the "us vs. them" mentality can sap productivity and lower the quality of life for everyone.

The reason there is no one "true ratio", though, is that different types of tests will get different results.
The best way to do it (if you have the option) is to split up the work by types of testing. Some types of tests are better suited to having more people thrown at it, where a knowledge of the inner workings is not as important. Outsourcing is a good fit for those situations, and will give you the same output for a lower cost.
There's a lot of other testing that's simply inefficient when done externally, however. An internal team is worth its weight in gold for those.

jaime kuroiwa
profile image
I'll echo Mitchell's reply that outsourcing has its advantages in some cases, but from what I remember about outsourced QA, there were problems with repetition -- same bugs described differently -- unclear steps, mislabeling, and long waits for replies; problems that would not exist with internal QA. I'm not saying the quality of the bugs were not "as good," just that there were clear differences in communication.

Thomas Petersen
profile image
I wrote a few blog posts on the subject of dev/test ratios:

With regards to outsourcing, I only consider it for tasks I see no career opportunity in. Mundane and boring tasks I would not want to waste my precious employees time with. So effectively I use as little outsourcing as possible.

Curtis Stuehrenberg
profile image
I can't speak for BioWare, but the trick to doing "agile testing" like this is to insert yourself into the process concurrent with the development team. As long as you're accepting "done" code as part of a hand off you will always find yourself playing this game.