Each subproject focuses on a specific aspect of the software, while attempting to not narrow down activities too far. A good example of this would be the addition of V8 engines to the Engine Designer in Automation. Within eight weeks' time, this subproject encompasses checking 2D/3D artwork, playtesting and balancing scenarios, gathering real-life engine data, polishing up entries in the bug-tracking tool, checking all new strings in the game, and last but not least: good old bug reporting.

At the start of each subproject, a subproject-specific scorecard is defined. Every tester has a single post with this scorecard in the beta forum's "Hall of Heroes" thread, which each individual continually updates as work progresses. The following is an example of such a scorecard in the V8 subproject mentioned above.

Tester ID: PaLaDiN1337 (4*)
Tester Status: ACTIVE
Current Score: 17 (+0)
Scenario Testing: (V0.88) S1, S2, S4, S5, S7, S8, S11
Tracker Entries: ID28804129, ID28796925, ID28786633
Engine Data: "Ford 4.6L Modular V8 3V"
Engines Tested: "Ford 5.4L Modular 'Triton' V8 DOHC"
Forum Excellence: "Proof-reading helps, 2p"
Achievements: 3x "Bug-tracker clean-up, 1p"

Keeping it simple is the name of the game. Tested a scenario? Get a point. Researched engine data? Get a point. Posted a bug report? Get a point. The system needs to be painless to administer, simple to use, easy to understand, and progress readily visible.

At the end of each subproject, the Hall of Heroes thread is locked and scores are transferred to a spreadsheet that lists all testers, their current status, and their power levels, along with carry-over scores -- all displayed as a timeline throughout different subprojects.

Click for larger version.

Not all activities can be planned for, or set up well in advance, and some tasks may pop up as development chugs along. For exactly this purpose there is the Quest Board -- a subproject-specific, continually updated section attached to the subproject announcement post in the Hall of Heroes thread. It allows for quick shifts in testing focus if required, and makes the testing system dynamic if need be. Here, testers will find non-standard tasks that yield achievements and points when completed. These tasks could be things like "clean up this spreadsheet for us," or "find some information on this topic," or maybe even "find additional testers that don't suck."

Testers do have a life -- or at least sometimes pretend to -- so more often than not, they may need to either take a break from testing or even retire. To accommodate this, there are passivity rules that allow for taking breaks, but quickly get rid of people that either never get started working or decided to not care anymore without notifying the development leads.

The "Rule of Thirds" does not only do well in photography but also here: being new to the team, coming from a passive status, or a failed active status, testers need to achieve at least one-third of the subproject's minimum score within the first third of the subproject to not be expelled. The administration of this system is rather trivial, too; changing a single flag in a spreadsheet, and checking the corresponding score cards of all arguable candidates one-third into the subproject.

The key benefits of POMMS to the developers:

Guiding: Developers can gauge and plan how much testing will be done on average.
Rewarding: The system creates a positive working environment for both devs and testers.
Flexible: The communication and the creation of new tasks is simple and efficient.
Manageable: The system is easy to administer and user participation is self-regulating.

The key benefits of POMMS to the testers:

Guiding: There is no guessing what things to do next, only options.
Rewarding: There are clear measures of progress and the tester's value to the team.
Flexible: There are always tasks that fit all kinds of tester preferences.
Manageable: The system is easy to understand and the scorecard simple to maintain.