Gamasutra: The Art & Business of Making Gamesspacer
arrowPress Releases






If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
How much people, time and money should QA take? Part1
by Mathieu Lachance on 01/13/16 01:11:00 pm   Expert Blogs   Featured Blogs

2 comments Share on Twitter Share on Facebook    RSS

The following blog post, unless otherwise noted, was written by a member of Gamasutraís community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Project planning is one of the most crucial and difficult steps of a videogame development project. It sets the scope of the game and its features and, through those, the amount of effort, time and money that is planned for it. With this entry I’ll share a couple tips and techniques my team and myself use to calculate how much people, time and money QA should take.†

The most basic external factor rules we rely on are:

  • The 10% rule &
  • The QA/Debug Dev rule

The main internal factor technique we use is the “Puzzle technique” which considers:

  • Game content
  • Feature overlaps &
  • “Multipliers” (Positive and negative QA project/ratio influencers)

I’ll discuss the Puzzle technique in a future entry; this one will cover the two external factor rules mentioned above. And before I start and get the question: “Matt! Does this apply to Dev QA or Publisher QA? Or outsourcers?”, this applies to all videogame QA!

The 10% rule

The 10% rule is based on the budget value that you assign to QA. It’s called as such because for most mid-sized projects, approximately 10% of the development budget would be assigned to QA testing. Bear in mind that this calculation should only be based on developer and tester expenses. It doesn’t take in consideration Art, Music & SFX, Marketing, Legal, etc., but solely what can be considered development or “engineering” expenses. Below is an example of what it would look like typically in terms of expenses; 10% of costs to testing, 20% to Debug Devs (essentially your developers fixing/debugging, or the amount of time spent fixing/debugging, the issues found by the testers) and the rest, 70%, to your core development team creating and implementing technical assets in your game.

As a general rule, the bigger the project, the less QA should cost in terms of total proportion. This can be explained by a few things:

  • QA team productivity is increased when they can see multiple assets and features simultaneously which is normally the case for bigger projects.
  • For longer projects, QA Leads will often assign testers to specific features, making them experts and which will gain productivity over time.
  • For bigger projects, teams and roles are usually more well-defined – testers test, and devs develop, and developers are usually a bit more expensive than testers.

Below is a guideline for that cost. As you can see, the concepts explained above can be clearly identified. I can already hear some of you complain that your QA cost experiences differed. Remember, this is a guideline, and if you see a drastic difference, there might be a few points you didn’t take in consideration. One may be that your dev team, or yourself if you are an indie, have been doing that testing instead of an assigned QA team. Another big influencer might be what I call “multipliers”, factors that can impact QA efforts and costs drastically (we’ll have a look at multipliers in my next entry).

Here’s an example from a developer that was kind enough to share information. His game cost him 571K$. So, based on the above cost you’d expect QA to be close to 20% (between Indie/B game and A type).

If we solely take the development budget of 278K$ (Dev and QA costs), we get 22.3% for testing, which is indeed very close to what we were expecting.

The QA/Debug Dev rule

While the 10% rule is based on expenditures, the QA/Debug Dev rule is based on average staff volumes while testing occurs. Essentially, on average, the amount of testers you assign to your project should not exceed the amount of developers assigned to fixing/debugging bugs on that same project. The biggest exceptions to this rule can sometimes be right after the beginning of testing, when testers are starting to understand your game, get productive and find all of the most obvious issues, and when nearing the end of your project, when quality is already quite high and the amount of bugs found per tester drops drastically. Both of these usually even out as an average and are partly why test teams are quite small at a beginning of a project and a lot bigger near the end.

The reason behind this rule is, on average across a whole project, the average amount of time it takes for a tester to find and regress an issue is about the same as the amount of time it takes a developer to debug that same issue.

In the example above, I’ve taken the same 3 core roles and the same “10% cost QA team” example (mid-sized AA project). The reason why the above is showing an 18% staff ratio rather than 10% as before is because a Dev usually costs about twice what a tester would.

As you can see, the effort or volume value between testers and the developers fixing/debugging the issues found is balanced. You will want to maintain a relative balance for most of the project. If you don’t, one of two things could occur:

  1. If you have more testers finding bugs than developers fixing them:
    1. The QA team will find all/most findable issues.
    2. The Debug Dev team will be unable to fix all/most reported issues between QA rounds, and the backlog of issues will go up drastically, more than what the Debug Dev team can handle.
    3. The QA team’s efficiency will gradually go down, and even potentially go to a stop, if bugs are not fixed rapidly enough, thus wasting time and money.

A good example of the above is something that happened to me a few years back. A big publisher approached us with a project; 2 guys in their basement built a great prototype and the publisher decided to back it up. After reviewing their timeline of 6 months and the game’s high amount of features, we evaluated that the project would need about 5 testers to get everything checked and fixed. 5 testers, 2 developers… After 3 weeks of starting QA, the testers had nothing to do and we suggested to stop QA until the development team could fix most of the issues reported and add in additional content. I won’t go into all of the details but, unsurprisingly, the game project was completed after more than 2 years. A lot further away than the initial requested 6 months!

  1. If you have more developers fixing/debugging bugs than testers finding them:
    1. The QA team will likely work close to 100% of their potential, because most of the found issues could be fixed between rounds.
    2. The Debug Dev team might have no bugs left to fix very rapidly, but can then move back to implementing and creating technical assets for the game.

Obviously, the second situation is a lot better than the first IF your development team is flexible regarding roles and IF you’ve actually assigned enough testers to your project. It also proves the proposed balance by forcing it on you naturally, when no bugs are left to fix, by adapting the role of your development team.

For the curious types, here’s what the volume ratio graph would look like:

So, that’s it for the basic rules to calculate how many resources QA should take. In my next entry, I’ll show you how to use the Puzzle technique, more specifically how to prepare the high-level QA strategy part of your project plan, step by step, and calculate how long your QA rounds should be based on:

  • Game content
  • Feature overlaps &
  • “Multipliers” (Positive and negative QA project/ratio influencers)

It’s the main internal concrete analysis of your game’s content and features which provides a concrete QA cost calculation to your project. Be there!


Related Jobs

Age of Learning, Inc.
Age of Learning, Inc. — Glendale, California, United States
[12.09.16]

Software Engineer - PHP Web Services
Visual Concepts
Visual Concepts — Novato, California, United States
[12.09.16]

Senior Software Engineer, Graphics
Visual Concepts
Visual Concepts — Novato, California, United States
[12.09.16]

Principal Software Engineer - WWE 2K
Visual Concepts
Visual Concepts — Novato, California, United States
[12.09.16]

Senior Software Engineer, Maya Tools





Loading Comments

loader image