Using Rackspace we simulated the load of roughly 100,000 concurrent players for one hour to see what our costs looked like for server maintenance. Using mobile metrics we could safely extrapolate 100,000 concurrent players to roughly one million players per day. This took roughly 12 instances of Rackspace and it spun up an approximate 50 instances of our own servers to support it for datastore and bandwidth usage. We knew that the hit on datastore was highest when the player first started to play the game, so adding 100,000 fresh players would give us a worst case scenario.

After running the simulations we were pretty happy with the numbers. Our servers would set us back, but we could handle it.

To further protect ourselves, we made the caching frequency of a players' social data (one of our biggest server hits) remotely patchable. Players were updating their friends on how they were doing and what heroes they were using roughly every hour. Having this variable remotely changeable meant that we could scale this out to six hours, or even 24. This slowed down social updates, but if this proved to be a big tax on the servers it gave us a throttle that we could adjust without shutting down the game. 

Finally we put in a Big Red Button to shut all online services down and made sure that the core loops of the game would still work. If the lending of heroes proved to be too taxing on servers, we could flip the switch and shut down things gracefully on an app level. Though this was more work, it was far more elegant than pulling the server and creating tons of error messages -- or pulling the app entirely, which meant missing out on potentially millions of downloads.

3. Mitigation of additional risky components  

At the beginning of the project, we scoped out our feature set within a roughly 80 percent accuracy and then began triaging what we believed the riskiest technical and gameplay components were. Emerging at the forefront was how our heroes and gear would work. 

The design of Animal Legends was that players were collecting heroes, sending them on adventures and outfitting them with gear. Gear gets you stats and makes your hero better but also changes the visuals -- allowing players to customize them. We thought this was fun, compelling and the level of customization seemed endless... Until it became a little bit too endless.

The gear system allowed for more than 200,000 permutations with just our launch content.  Heroes could hit a certain critical mass where they were wearing different suits of gear, hitting different texture pages and finally overrunning memory in the app. We believed that if we shipped with three heroes, each carrying five sets of gear, we would be at the edge of expansion in the system. This meant we couldn't add more heroes or gear. This wasn't an acceptable solution for a game about collecting heroes and outfitting them with gear!

We spent a good 10 hours brainstorming solutions. There were lots of possibilities but most of them proved to require a very large effort in overhauling our internal engine.

Finally we conceived a simple alternative which was to create a dynamic texture page system. This would mean that each hero in play would get their own page and, regardless of what they were wearing or how many suits of gear we had in the game, the player would only have as many pages as heroes. 

However, this also meant that each hero had to fit in a single page, so now we not only had to scope out these heroes but we also had to scope out everything that they could ever wear and the maximum PNG size for each item. If, months down the road, we wanted to add a "necklace slot" it just wouldn't be possible unless we put it on the character right there and then.

We began by sketching out all of the various body pieces which would compose the heroes and all the various pieces which they would wear. This would give us a complete footprint for our texture pages, to ensure a hero could fit in a single page. It would also give us the footprints for creating the gear and pieces themselves. 

Click for larger image.

Once we understood the boundary conditions of the system, we could begin blocking out the size of the gear and how it would work with character animation.