Swords & Soldiers is a critically acclaimed WiiWare side-scrolling RTS -- and it's about to be released for PlayStation Network. The new controls required a user test. For this, the game's development studio, Ronimo, collaborated with Valsplat, a usability research company. Here is how we at Valsplat set up the test, what Ronimo did with the results, and what we learned from the experience.

The Game, Its Developers, and the Test Company

The goal of Swords & Soldiers is to destroy the opponent's tower and defend your own. You build soldiers with gold you've dug. Your soldiers attack enemy soldiers and their tower. Magic automatically builds up, and is used to cast spells on the enemy or to heal your own people. You can also sacrifice your own people for more magic. As you progress, soldiers and spells can be upgraded to more powerful ones. There are three tribes to play with: Vikings, Chinese, and Aztecs.

The game was released last year on WiiWare and met with positive reviews, resulting in a Metacritic score of 84.

The use of the Wii Remote plays a big part in the chaotic fun. The point-and-click controller enables you to quickly and easily build soldiers, cast spells, and scroll back and forth across the battlefield -- or as IGN put it, "Swords & Soldiers was built for Wii, and as such it plays very well on the system."

Now, Ronimo is releasing the game for PlayStation Network. One of the main challenges is to make the game as easy to control as the Wii version.

To test if players understood the controls, Ronimo collaborated with us, Valsplat. We're a Dutch company with a wide experience in website usability research, but relatively new to game testing. In this playtest we applied our experience from the web, but also tried some new things.

Test Goals and Research Questions

Together with Ronimo, we established the test goals; these formed the base of our tests: These goals let us focus on the main objectives and largely determined the setup of the tests. ­

The main goal was to test the usability of the in-game controls and HUD. Since the quick and semi-conscious point-and-click is arguably more difficult with the traditional Sony controller than with Wii Remote, Ronimo had to rely less on aiming and more on button combinations. The new controls also required a new HUD. Therefore, we wanted to know if the button mapping and screen display info felt intuitive enough to leave precious mental resources for snap strategic decisions in a chaotic battle.

Left: With the Wii, you point-and click the desired spell or soldier. Right: With the PlayStation, you press and hold L1 for building soldiers, R1 for casting spells. This opens a radial menu in which you select the spell/soldier with the right analog stick and submit by pressing X.

Our research questions were:

• Can players quickly and easily build soldiers and cast spells?
• Can they scroll the battlefield?
• Do they understand the upgrade menu?
• How do players figure out controls, through trial and error, do they read tutorial instructions or access the pause menu?
• What is the learning curve -- how long does it take to get the basics?

A secondary goal was to determine the usability of the game menu. Point-and-click was replaced by tabbing navigation using the right analog stick. Would it work? Can players adjust settings, and easily start a game? Do they read and understand the popup text explaining a menu item?

Things like bugs, level balancing story development weren't the scope of this test, but of course we stayed alert for unexpected findings. You never know what a player might do or try.

Test Setup and Fieldwork

The right players. With the test goals in mind, we designed the test setup. A website is usually tested with five to 10 users. Six users, as a rule of thumb, identify about 90 percent of usability issues. To our knowledge, there isn't such a rule with playtesting, but we estimated that with eight players we'd answer most research questions.

With only eight participants, it is essential to test the right people. Together with Ronimo, we determined characteristics of the target audience. This included: familiar with PlayStation controls, aged between 14-30, RTS players.

Valsplat has a database with about 5,500 people. We e-mailed a sample of 140 people, aged between 14 and 30 years, saying there was a 45-minute test for a new game. We also named a test date and the incentive. When interested, people could click a link leading to a questionnaire which asked them to name the consoles and RTS games they played, their gaming frequency, preferred game genre, and job or line of study.

To minimize bias, we excluded players if they had a job or study related to gaming such as game design or game art. Out of a response of 35, we selected eight players: seven guys and one girl. After Ronimo approved the gamer profiles, these players were invited for the test.