Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
October 24, 2014
arrowPress Releases
October 24, 2014
PR Newswire
View All
View All     Submit Event





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Evolving Test in the Video Game Industry
by Mike Burghart on 06/26/14 08:50:00 pm   Expert Blogs   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

In every studio I've worked in and, for the most part, everyone in the video game development community agrees that it's a good idea to have time and energy dedicated to the quality of a game. But QA departments incur massive costs in personnel and overtime, and are sometimes vilified in P&L discussions. QA management is often tasked with finding leaner ways to do business; increasing test coverage without impacting development team practices.  So far, the answer has been to have more people testing more things, longer. Black box or functional testing has been the usual and customary focus of the gaming industry. This approach can be loaded with unidentified and risky pitfalls.

The quality solution for the video game industry does not lie with the tester or the QA manager. I've spent a decade in the video game industry shipping AAA titles, and before that several years in trans-media development houses. I've had countless discussions with development, QA, and operational oracles in the game industry. Neither I, nor anyone I've spoken with, has ever seen quality tested into a game they were working on.

Our quality solution resides in the game development team processes, and should be addressed at that level. If quality solutions were built into the beginning of the development pipeline, independent QA departments could be substantially reduced in size. Instead of functional testing to find errors, these streamlined departments could be tasked with establishing best practices across the organization and managing the ebb and flow of contract personnel for usability testing. QA Managers could facilitate the internal transition and external hiring of qualified, passionate Engineers in Test and Testers development team integration.

Software development houses have been integrating test earlier in the pipeline for years, of course. But game studios lag behind software development best practices, favoring perceived "tried and true" methods. When I first entered the industry, I sat with a game studio executive and asked if the company had a project management tome for best practices that all teams could share. The executive replied with an emphatic "No!" followed by "... and this is why we make the best games."

When agile development gained traction many game studios liked the idea but floundered on implementation, with many development teams adopting differing approaches or making up practices to fill gaps. The dynamic was game-like; each team tried to "win" by coming up with the best agile process, but failed to share successes and lessons with other teams.

Shifting from traditional practices to a new paradigm can be difficult, and often inspires resistance. But, when the benefits to collaboration between development and test are weighed against endless hours of overtime and diminishing returns, change becomes more attractive than continuing the pain. Using integrated testing solutions, teams can identify and correct issues as soon as an hour after creation. Traditional processes will usually surface those issues, but normally over a year later, during functional testing. More rapid identification and correction translates into greater agility for both developers and testers.

Perceptions

The image below is making the rounds on professional media networks:

dev-qa-environments.png

Ask anyone in QA, and they'll give you some variation on we are all working together to make a better game experience. But Development often seems to feel that QA is standing by to "rip apart" any and all game content they encounter. It is not supposed to be an adversarial relationship, but can be contentious, especially during crunch times.

Changing the Paradigm

QA, in most game studios, lives behind a wall at the end of the production pipeline. Development throws completed features over that wall and QA conducts functional testing on those features – usually with little to no requirements documentation for traceability. Not the best approach if you are looking to avoid bottlenecks and ship the best game possible. In response, QA tends to dog pile on each new feature, conducting extensive manual testing without increasing staff. The result can be considerable overtime.

Imagine the improvements from an integrated testing organization. Risk could be identified, planned for and managed throughout the development lifecycle. Working closely with developers and programmers, engineers in test would be part of the feature team, sharing their sprints, goals and expectations. The need for a separate QA department, aside from a center of excellence and user experience team, could be objectively evaluated. Communication is the usual culprit in any disastrous release. The job of a "tester" should be to identify risks and communicate those risks to all stakeholders for collaborative prioritization. Test coverage is how we mitigate risk. Integrated test is how we get there.

Conclusion

We can realize an application feature lifecycle where testing does not exist solely at the end of the pipeline, but rather permeates the entire development lifecycle. Over the next few installments, I'll offer an approach to make this type of change possible. Topics will include:

  • Test in an agile development environment
  • Evolution of QA into a game quality Center of Excellence
  • Building a successful Engineering in Test Department
  • Moving to a risk-based test approach
  • Key Performance Indicators (KPI) validity in measuring test coverage and tester performance
  • Mitigating risks from inflated QA management and leadership groups

This is the first installment of a series on professionalizing Test in Video Game Development. Several topics will be discussed. Most positions were developed in a team and individual contributor context. A number of studios were evaluated for these articles providing a broad view of process and performance. 


Related Jobs

Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States
[10.23.14]

Senior Sound Designer - Infinity Ward
Treyarch / Activision
Treyarch / Activision — Santa Monica, California, United States
[10.23.14]

Multiplayer Level Designer - Treyarch
Petroglyph Games
Petroglyph Games — Las Vegas, Nevada, United States
[10.22.14]

Producer
Nix Hydra
Nix Hydra — Los Angeles, California, United States
[10.22.14]

Art Director






Comments


Terry Matthes
profile image
All the suggestions in this article seem great, but it's hard to attract good QA staff. Wonky hours, weeks without work, and very low starting pay make it hard for employers and employees.

For a lot of studios I think a slick integrated QA department is the goal, but it's a goal far out of site when retaining quality employees is a bigger issue.

David Lejeune
profile image
"Wonky hours, weeks without work, and very low starting pay make it hard for employers and employees." You say that like those factors are unavoidable and universal constants for QA and the video game industry.

Stop treating QA like it's a job for high school kids who only want to play video games for a living and offer decent wages and benefits, and quality employee retention will stop being a problem. Decent QA then makes development easier and more efficient, which will in turn reduce crunch, which will make everybody's lives better.

Mike Burghart
profile image
Your point is well-taken Terry as is David's. One of the biggest obstacles we face in evolving test is the perception of QA as an unskilled entry-level position. Game testers work long hours with little real impact on the playability of a game. Checklists for regression testing and often haphazard functional testing take precedent over robust exploratory testing due to time constraints. This is what the industry has come to expect; both on the Development and QA side.

I'll be writing more about this in an upcoming blog post, but, for now, let me distill this down. Testing is a profession and many people in our industry and on the fridges of it are passionate about test. Since we have relegated game testers to the bottom of the food chain, we have made it more difficult to find, hire and retain talented Engineers in Test. As we move forward toward integrated testing throughout the pipeline, these qualified and passionate professionals will become the cornerstone of a new testing paradigm. In short, we need to change the way we hire and raise the bar for candidates.

More on this later.

Terry Matthes
profile image
Well said!

Teressa Wright
profile image
A lot of larger companies are going this route, with automation tools, integrated QA (even from the concept stage) and higher skillset requirements from their testers. It's a slow process, and it can take up to 2 years before you see dividends, and the large upfront cost of making the change can be quite prohibitive. It's worth it in the end, and the QA departments that lead the way are seeing a huge return on investment. But as I said, it's not cheap and it's not easy.

One major problem we're all facing currently is that there is a serious shortage of SDETs available who want to work in games. At a recent conference I attended purely about Game QA, game SDETS were literally described as 'unicorns', and most companies that have actually found one have had to go outside of the game industry to find them. What we've got here is a chicken and egg situation; game QA needs to be a respected career choice so that people choose to become SDETs in our industry, but we're not going to get that respect without SDETs.

Mike Burghart
profile image
I agree with you Teressa that automation is part of the solution. A robust test harness is a product of an Engineer in Test working with Software Engineers; providing them with requirements for the harness based on the test plans crafted from identified risks. Consequently, a Test team should include Engineers in Test and Software Engineers.

A talent shortage is a definitely a risk in adopting Engineers in Test. Whether called SDETs, EiTs or SQEs, we should be looking for someone who can understand the code they are working with and are passionate about test. If you have an experienced QA department, you should need no more than a couple of Engineers in Test for test strategy, planning and execution. Mentorship of testers coming out of college or already with the company is key to building a team for the next decade. Additional testers can focus on user experience. We often don't have time for that and it is critical for customer satisfaction.

One final thought, if pursuing recent graduates for assistant or associate work, try looking for candidates with a computer science, engineering, or mathematics degree. Great discussion so far. Thank you everyone!


none
 
Comment: