Gamasutra: The Art & Business of Making Gamesspacer
Quality Quality Assurance: A Methodology for Wide-Spectrum Game Testing
View All     RSS
October 1, 2014
arrowPress Releases
October 1, 2014
PR Newswire
View All

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

Quality Quality Assurance: A Methodology for Wide-Spectrum Game Testing

April 28, 2009 Article Start Page 1 of 3 Next

[In this Gamasutra article, Nintendo and Microsoft Game Studios veteran Wilson talks about the value of diverse video game testing, suggesting a formula to make sure that your game debuts with the fewest bugs possible.]

The need for wide-spectrum testing

In the process of software development, there are constant pitfalls and perils to avoid, both to application developers and game developers alike. Software testing, one of the most resource-consuming stages of the development cycle, possesses more than its fair share of these problems, as well as tending to have an unfortunate stigma attached to it.

Some developers consider it a short process to find glaring errors, a necessary evil or a secondary concern. Consumers consider it to be the stage in which any problem they have with the product should have been found (often rightly so). Testers themselves often consider the testing stage to be rushed or insufficient once it ends.

The biggest misconception about software testing is that any one method of testing is better than another. There is both an art and a science to software testing, and neither of them should be ignored.

Testing a strict set of conditions or performing seemingly random tests just aren't enough by themselves, no matter how extensive the process becomes; both the art and the science are needed to find as many of the bugs as possible, leaving the software as functional and polished as possible.

The differences between ad-hoc testing and test cases

Ad-Hoc Testing

Ad-hoc testing, also known as free testing, is a style of testing that definitely falls into the artistic side of the spectrum. This form of testing is most often used in game testing, but it can also be found in consumer-use panels and focus groups, where individuals are brought in to try new software with very broad goals that they are directed to complete.

As a style, it is very fluid and often seems random; a game tester may progress along half of a level as the developer intended only to attempt to jump through a crack in the environment, causing their character to leave the bounds of the environment and become stuck, unable to return to the normal flow of play.

When testing a piece of presentation software, a tester may attempt to loop through the entire presentation rapidly multiple times, not allowing time for the images or videos to load, which may stress the available memory to the point that the software stops responding and locks up.

The Good

While these seem like random things to try during the testing process, they are things that may occur in the real world. This is where ad-hoc testing becomes an art: finding things that the end-user may attempt that the developers haven't planned for.

This may seem simple enough, but the amount of creativity necessary for this form of testing can sometimes seem staggering.

Entering a dungeon in a game, letting it populate halfway, then deciding to go back to town and save the game quickly may lock up the game -- but if a player forgets to save their game before entering a dungeon, it could definitely happen.

Plans can certainly be made to test these sorts of situations, but there's no effective way to plan for them all. This is where ad-hoc testing is the most useful: testing situations that may otherwise occur after release that weren't planned for during development. The number of unusual actions good testers will try when left to their own devices can be surprising, and they will often find a fair number of problems that can then be corrected before release.

The Bad

While this sounds great, and it often finds some major issues that would otherwise have been unnoticed, there are a few problems with this method. It's almost impossible to cover all of a piece of software's functionality in this way; there's often too much space to cover to allow a testing team to perform free testing without any focus.

Also, it is likely that more testers will focus on areas that they have a preference for over others, which will leave some areas of the software with less coverage. To diminish this problem, many advocates for this style of testing temper their test plans by assigning ad-hoc testers to specific portions of the software, but this still isn't enough to compensate for the lack of disciplined testing.

Test Cases

This is the scientific side of testing. Developers and test leads will produce a list of tests to be performed based on the functions in the product, which functions interact with other functions, what different parameters there are for each function, etc

Test cases are the counterpoint to ad-hoc testing; where ad-hoc testing seems random, test cases are strict and disciplined. They are used to go over a function, which can be as simple as moving between cells in a spreadsheet or as complex as casting an intricate spell at a group of enemies, from every point of view and with each command style that the writer of the test cases can think of.

The Good

Test cases perform where ad-hoc testing can often lack: they ensure that the most common actions that will be performed are tested in a large variety of ways in every area of the software.

This alone is a boon to the testing process, as the color palette that an ad-hoc tester may take for granted in the software they're testing may have an incorrect variable call in just one format style, resulting in the desired blue becoming green. A test case to check the color implementation for each color in each format would easily catch such problems.

The Bad

The amount of coverage a title receives through test cases is dependent upon the people writing the cases. This coverage can be very extensive, especially when the test cases are written by people with years of experience and an in-depth knowledge of the functions that need to be tested, but nobody can account for everything that an end-user may attempt.

There are just too many random variables to be considered for test cases to cover every possible occurrence. It's also important to note that some testers may be easily bored by such strict testing protocols, which in rare cases could result in the test cases not being completed properly.

Article Start Page 1 of 3 Next

Related Jobs

Crystal Dynamics
Crystal Dynamics — Redwood City, California, United States

Senior Environment Artist
Harmonix Music Systems
Harmonix Music Systems — Cambridge, Massachusetts, United States

Senior Product Manager
Harmonix Music Systems
Harmonix Music Systems — Cambridge, Massachusetts, United States

Web Developer
DoubleDown Interactive
DoubleDown Interactive — Seattle, Washington, United States

Principal Game Designer


jaime kuroiwa
profile image
"Documentation for various mechanics and functions can always be helpful. The more information that a testing team receives, the better they'll be able to test the software."

I would like to emphasize this statement, because documentation enhances communication in so many ways. Documentation establishes nomenclature, which helps testers better describe issues to the developer. Documentation tells testers which user styles to emulate -- this is especially helpful in ad-hoc testing. Documentation shows the software's progress and direction to the test team, which helps in focus. Documentation may even allow the test team to catch bugs before the code is implemented (e.g. grammar, continuity, etc.).

Most of all, documentation gets the test team involved in the overall process. Believe it or not, testers are not only there to test; they want to learn and share.

Victor Gont
profile image
This article sums up perfectly the process of testing a game, be it an AAA or an iPhone casual title. Despite that, i feel that the essential part of successfully applying this process has not been sufficiently emphasized. Documentation and communication with the dev team both help a great deal, but the real issues arise when the management of the teams responsible for parts of the work (especially in multi-platform environments) is done in an irresponsible manner. Adaptability and quick response are of the essence, and applying methods known from previous projects can restrict and seriously harm current test activities, despite the experience and success encountered earlier.

The team management must also focus on assigning relevant tasks to its members, discovering everybody's strong points and preferences early in the work flow. Trying to turn a gameplay tester into a generalist and assign him to look for level design bugs is a sound and rather often employed method for crashing the team's morale and efficiency. There is also a problem when the tests are becoming some sort of daily 'grind', as there is not enough attention focused towards rotation of the tasks and creating new testcases between different versions of the project.

When these issues appear at the same time, and the project is delayed or completely redone (leading to a title being in testing for extremely long periods of time), the morale of the team is the most important thing to be considered. Having 90 bored or on the point of breakdown professional testers trying to find flaws in a title they may have otherwise enjoyed counts for little. It's a small thing, but it can be easily overlooked when taking note of the test team just through the number and severity of the bugs they submit in a database.

Johnathan Murrill
profile image
I will glady work as a game tester if anyone could assist me to find this job or hire me in my area it would be a big help because this is what ive been longing to do this is what I want to do in my life.

Nicholas Muise
profile image
Thanks for a great informative article David

Mitul Theiya
profile image
As of now experience with various domain, I'm still fresher in game testing after testing IPhone/IPad/Android game based on words.

Most I found that ad-hoc testing can be very tricky arms that improves testing skills in shot time. As also, Test case may approach near to complete testing, it's still extensive and ensure about coverage, not each single issue mimic in real-word i.e. i.e. low bandwidth-short response time, multiplaying task.

I absolutely preferred ad-hoc testing, but with combination of Test cases. Here test cases will help to quick execution and identification about external behavior to ensure about positive result of playing game. Ad-hoc will help to identify few of exceptional cases that can may improved regularly in frequent duration and also hint to be careful about same issues in next version of game.