Co-authored with Steven Gaston
In part 2 of this post (in case you missed, Part 1 here), as promised we will cover how to set up external usability tests, preparation done at developers' (client) and research agencies end, see how internal and external UX designers work in sync combining their final observation & analysis to produce unbiased results.
In Part 1, we saw how most studios typically conduct usability test in ways that allow biases to creep in which could drastically alter the results.
We are happy to know there are studios that have a more sophisticated UX pipeline & UX designers (some of your comments hinted at that) in place who conduct sessions in a clinical setting. Our process will outline the ideal practices that should be a part of the DNA of any usability session.
STEP 1: Pre-Test Prepping
We conducted an external usability tests in a neutral setting to get unbiased results. Here we have two parties involved:
The Developers (Clients): Comprises of the studios in-house UX designers game designers, producers, UI artists and other stakeholders.
The Agency: A moderator like Steve and an assistant who might help him set the test and take notes during the play sessions.
Client Side (Developer) Setup: From the developer’s end, the team identifies (apart from general usability & interaction) the features we want to test; e.g. core loop actions, social actions, crucial aspects like on-boarding etc. Others include things like budget for the research facility and number of participants involved, case building for usability test for directors, defining scope, negotiation with the agency.
Agency Side (Steve) Setup: First we identify the objectives of the research. For example, is there a particular focus on the on-boarding experience? Are there social elements that need to be tested?
We then take these objectives and combine them with the Ipsos UU philosophy of conducting research with real people in real life. We develop a research design that is as close to reality as possible.
Given the constraints of actually conducting the research itself, we still conduct the research in a research facility so that we can have the cameras set up for the designers/developers to view in a separate room.
Once the research design is developed, we put it into place. Typically, the design looks like this:
Insight 1 : Reason for segregating observers, players and moderator This is done to make sure unbiased viewpoints and observations are recorded with no way one party affecting/influencing the play-test and observation of other party.
We deliberately do not allow Developers to come in contact with the play testers so as to avoid any of the biases creeping in (play testers can sometimes feel obliged to say nice things out of the need to be courteous or not hurting developers feeling or getting intimidated by being observed by lot of people), hence they can't play as naturally as they normally do.
An insight of how even presence of moderator can skew the players test behavior
Insight: Removing the moderator from the equation even for a small period of time allowed players natural behaviour/true opinion to surface. We made sure we did this deliberately. As part of the test from next cycle onwards.
Insight 2 : Importance of video recordings of play session
Why do video recordings? Isn’t observing people
first hand and taking their feedback good enough?
People are oblivious and sometimes say things contradictory to their actions!
For a particular control scheme in one section of the game, players were asked by the moderator after the players had play tested what their experience was and their understanding of that particular feature. Players said it was very easy to understand and taking the action/feature in question was very straight forward and smooth.
Cut to the video recording that we analysed later of players tackling that feature and we could see from their action (tablet device and player hand movement close up camera):
So there you have it. In this two part series we have taken you through some of the issues witch a lot of the UX testing organisations are doing and presented you with an approach that we have developed that we feel effectively leads to more insightful testing. Our approach is beneficial in the sense that it:
We have had a lot of success taking this approach and we hope that what we have shared can help you in your next UX testing session!