Gamasutra: The Art & Business of Making Gamesspacer
Postmortem: Intelligence Engine Design Systems' City Conquest
View All     RSS
October 30, 2014
arrowPress Releases
October 30, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Postmortem: Intelligence Engine Design Systems' City Conquest

February 6, 2013 Article Start Previous Page 3 of 4 Next
 

3. Discovery-Driven Planning

While I was earning my Penn/Wharton MSE degree, I had the good fortune to attend a few lectures by Professor Ian MacMillan, coauthor of Unlocking Opportunities for Growth, and learn about discovery-driven planning. The lectures were a huge eye-opener.

The discovery-driven planning (DDP) process is explicitly tailored toward innovation-oriented projects and focuses on reducing uncertainty. For all the changes we know are inevitable in our designs, developers don't typically factor that learning into our planning or use planning methodologies directed toward identifying and reducing uncertainty.

We adopt Scrum and other "agile" methods, and we adjust our plans as circumstances change, but we generally make no effort to quantify our uncertainty from the outset or to plan in a way that will ensure that we focus on the most important risks.

On City Conquest, the DDP system forced us to explicitly identify and quantify all of our major assumptions, build a reverse income statement, and perform a sensitivity analysis of the effect of each assumption on profits. This methodology ensured that we directed our development efforts on the areas with the greatest potential for uncertainty to affect the project's costs and revenues.

But even if you don't use a DDP plan explicitly, the general principle of focusing your efforts at any given moment on those features or areas with the highest value of ((uncertainty) x (financial impact)) can go a long way toward ensuring growth while proactively managing risk.

4. Technology Choices

Technology choices are fraught with risk. There are many recent examples, including Glitch, whose developers made clear that the choice of Flash made it impossible to achieve their project goals, and another high-profile case where a developer sued their own engine vendor for technology delays and inadequacies.

Our early tech choices were critical to the success of the game. My own experiences have made me extremely paranoid about the risks associated with licensed technologies and the performance and productivity impacts of scripting languages.

City Conquest places huge performance demands on mobile hardware, including some relatively modest devices such as iPod Touch 4. It required extensive optimization and careful attention to performance at every level.

We chose to build the game with Marmalade precisely because it was not an engine, but only a thin, high-performance cross-platform API layer. It allowed all of the programmers on the team to do 95 percent of our coding and debugging on Windows in Visual Studio using C++, test the game using Marmalade's PC emulator, and deploy identical code to iOS and Android (and possibly Mac and Windows in the future now that Marmalade supports them). Outside of a bit of platform-specific multiplayer code, the initial "first-playable" Android port literally took under an hour.

Marmalade is not without its downsides. Like any tool, it comes with its own trade-offs. The lack of on-device debugging was problematic at times (though this was very rarely actually necessary, as 99 percent of our bugs were our own and were reproducible in the Visual Studio debugger using the Marmalade emulator on Windows). We also needed to extensively customize the resource management.

The use of Marmalade and C++ ensured that we could achieve the critical optimizations necessary to make such an enormously graphically intensive game run at reasonable frame rates on a wide range of mobile devices.

5. Careful Cost Management and Outsourcing

We have seen far too many studios grow too quickly and implode under the weight of their own aggressive expansion. Not only does this hiring dramatically inflate a company's cost structure and the breakeven level required for profitability, but it also usually changes the company's character for the worse. A company's teamwork usually grows much more slowly than its team size, and studios often fail to spread the culture and principles that made them successful to all of the new hires.

Our tiny studio size and careful cost management ensured that we minimized our overhead and kept our breakeven as low as possible. We are far more interested in growing slowly, hiring the right people, managing our risks, and building City Conquest and other games into stable, ever-evolving, long-term franchises. We found that our art, audio, and multiplayer engineering contractors produced a surprisingly high level of quality at reasonable costs, and we could not be happier to have been able to work with all of them.

What Went Wrong

1. Mission Design

We knew long before we even began developing City Conquest that it's best to teach players gameplay concepts in small, easily digestible pieces, and wash each one down with a good helping of fun. This is Game Design 101, and as a veteran Nintendo developer, this had almost become second nature, so obvious that it hardly merited a second thought.

Yet somehow, inexplicably, we started by building a tutorial mission that tried to teach every major gameplay concept -- building, upgrading, territory control, dropship pad swapping, game effects, cloaking fields, resource management of gold and crystals -- in a single 15-minute mission with a nonstop barrage of text popup boxes.

After all that work to develop what I felt was a very solid tutorial mission, I received some brutally honest feedback from an external tester. No one wanted to sit through 15 minutes of text popups before having any fun or actually playing the game. And it was just too much information all at once: the gameplay concepts that seemed so obvious to me were of course much less so for the playtesters.

And when they finally did get through the tutorial, they were suddenly overwhelmed with options in the very next mission: the player could immediately build all 20 buildings and use all four Mothership effects before being taught to use them properly.

Ultimately, there was no choice but to take a deep breath and refactor the mission design completely. We separated the game into five different conceptual elements -- attacking, defending, expanding your territory with Skyscrapers, using game effects, and swapping dropship pads -- and built at least one specialized mission to teach each of those five core gameplay elements and use them in an entertaining way. We also overhauled to the user interface to allow us to limit the player's building choices to a limited subset defined in the mission scripting, helping us ensure that we would not overwhelm players with options too quickly.

2. Monetization

We released City Conquest on iOS as a free download, with a single $4.99 in-app purchase for the full game. The free version contains the full multiplayer experience plus the first five missions of the single-player campaign (out of 14) and the first of the six bonus "challenge" missions.

We had initially intended to release the game as separate "Lite" and "Full" apps, with the "Lite" version as a free limited download. However, discussions with the Marmalade team ultimately convinced us that it would be better to combine these into a single free download, with an in-game paywall separating the demo from the full game experience.

Our reasons for this were well-intentioned: this approach ensured the user only had to download the game once, minimized the potential for confusion between "lite" and "full" versions, made it easier to convert players from free players into full players, ensured that all players could play multiplayer for free, and ensured that achievements and campaign completion were shared across both the lite and full versions.

In retrospect, although we're happy with our conversion rate, it's clear that there are serious limitations to this approach and any others that aren't based on full-fledged monetization. The nature of the mobile market makes it extremely important to do proper price discrimination across the full spectrum of mobile users. There have been several articles on monetization on Gamasutra, and we refer the reader to these.

We are currently nearing completion of the Android version, which we expect to release as an advertising-supported free game.


Article Start Previous Page 3 of 4 Next

Related Jobs

Square Enix Co., Ltd.
Square Enix Co., Ltd. — Tokyo, Japan
[10.30.14]

Programmers
Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States
[10.30.14]

Senior Sound Designer - Infinity Ward
Treyarch / Activision
Treyarch / Activision — Santa Monica, California, United States
[10.30.14]

Multiplayer Level Designer - Treyarch
Nexon America, Inc.
Nexon America, Inc. — El Segundo, California, United States
[10.29.14]

Localization Coordinator






Comments


Paul Tozour
profile image
I just wanted to note that Gamasutra's tagline on this article, "how automated testing saves time, and how Kickstarter wastes it," is Gamasutra's characterization, not my own, and is not the way I would have characterized this article. I'm working to get this fixed, but in the meantime, Evolver is not exactly an automated testing system, and Kickstarter isn't a waste of time.

Paul Tozour
profile image
EDIT: The Gamasutra editors have tweaked the tagline to avoid confusion. Thanks, Gamasutra!

Don Hogan
profile image
Excellent write-up, Paul. It's always good to hear your take on game development, there's never a shortage of food for thought. Glad to hear the project went well!

Michael DeFazio
profile image
Paul,
Fantastic article--

Wish the Kickstarter video could have been attached it was also awesome (Seeing how you created algorithms to find optimal strategies and had them play against each other was fabulous.)

Love your philosophy about games (problems spaces) and AI... And the amount of times you mentioned "decisions" in the article put a smile on my face (I'm sorta a gameplay first kinda guy, and great games to me always find a way of presenting "interesting decisions").

You completely sold me on this game (one android copy Sold!)... and I will continue to watch for other revelations/advancements you and your company find about making compelling (and balanced) gameplay in the future.

Cheers

Paul Tozour
profile image
Thanks, Michael! For anyone who's interested, the Kickstarter video is here: http://kck.st/GzJ324

I'm also going to be giving a talk on Evolver and some other aspects of my approach at the GDC AI Summit in March (along with Damian Isla and Christian Baekkelund, who will be discussing the role of AI in the design of Moonshot Games' terrific new game Third Eye Crime).

Paul Tozour
profile image
.

GameViewPoint Developer
profile image
I think the AI approach to game testing is definitely interesting but would be a lot of work for a small indie team to implement, perhaps if there were 3rd party tools available it would be a useable solution.

Paul Tozour
profile image
To be clear, the point of Evolver was not as a game testing system. It was designed to help explore the design space and guide the game balancing. It was fundamentally a design tool, NOT a testing tool.

It did help find some bugs, of course, but that was just a nice side-effect. The real point was to help us optimize the balancing between all the different units and towers in the game.

The difficulty of implementing something like this really depends on the game. It's not a magic bullet and it's not an approach that will work for every game. And it's the type of system that has to be carefully designed for the particular game in question, so it's not the kind of thing where you can really create a tool that will work for any game.

In the case of City Conquest, it cost us about 2 weeks' worth of coding and other work, and easily saved us a good 3-4 weeks worth of design time -- while also giving us better results than we would likely have been able to get by hand, AND giving us a system that would give us instant visibility into the ramifications of any given design decision. So, it was clearly a net win just in terms of the time savings alone, even before you consider all of the other benefits.

In any event, as I mentioned in one of the previous comments, I'm going to be speaking about this in more detail at the GDC AI Summit in March. So be sure to stop by if you'll be at GDC.

Denis Timofeev
profile image
Hi! That's a great article, thanks. A lot of inspiration. I'm just wondering how many users you had on TestFlight, how did you get them and how helpful their was.

Paul Tozour
profile image
We had around 50 or so at first, but once we opened the testing up to *all* the Kickstarter backers, we ultimately got to the maximum number of users allowable based on Apple's developer restrictions for the maximum allowable number of devices per app (100). The actual number is below 100 since some of them had multiple devices (iPhone + iPad) so they took up multiple device ID slots.

We started with personal friends and industry contacts, then added backers at the appropriate backing level in Kickstarter, and then, a few months later, opened it up to absolutely all the backers who had iOS devices.

Their feedback was extremely helpful overall. We got a very wide range of feedback from a lot of people at a lot of different skill levels. We had a few industry veterans in there (see the game's Credits for the full list), who provided terrific and detailed feedback, along with a few non-gamers. We also did some diagnostics, such as adding buttons so the testers could send back valuable data on mission completion and achievements earned. So it was nice to see that about 90% of the ones who responded were able to finish the single-player campaign, and every mission was completed by at least one tester at every available difficulty level. Also, they were able to get me invaliable feedback on devices I didn't own / couldn't get my hands on at the time, such as the iPhone 5 and the "new" iPad aka iPad 3.

Louis Gascoigne
profile image
Great article Paul, worth the read just for the tech section.

Bram Stolk
profile image
Impressive stuff, and great article.
Amazing that you could get Computer Aided Game Balancing working in just 2 weeks.
Writing a system like Evolver sounds like more fun than tediously going through manual iterations of game balance.

Jeremy Tate
profile image
@Paul How did you approach actual code testing? It seems like it would be crucial under this model to have technical integrated testers on the team if you were going to maintain a running total of 10 bugs. Otherwise, you are kind of pushing those undiscovered bugs to later in the project.

Paul Tozour
profile image
I definitely agree in principle with having as many testers as possible doing testing from day one. This is a great idea whenever you can afford it, and it worked very nicely for Retro Studios when I worked with them on Metroid Prime 2 and 3 -- the internal elite ninja testing team was super effective.

I think we were able to get away with not having dedicated testers on the City Conquest team thanks to a combination of factors: it was the relatively limited scope of the project, the involvement of our external playtesting team (friends and Kickstarter backers on TestFlight), our defensive programming practices, and our habit of playing through the entire game on a regular basis. Also, the fact that the Evolver tool found a few of the most difficult/subtle bugs on its own just by virtue of the fact that we were running a million simulations overnight and could pretty quickly find anything in the game logic that caused a crash or a hang.

But again, I do agree with you that having dedicated testers throughout the process is the better way to go if at all possible. The earlier you can find bugs, the better.

Paul Tozour
profile image
And when I say "10 bugs," of course I mean 10 *known* bugs. Naturally there's no way to count bugs that you don't know about.

But, yeah -- the earlier you find them, the earlier you can fix them.

Jeremy Tate
profile image
In addition, since a game by its very nature is almost entirely focused on usability vs functionality, so if something has to go, it has to be the latter. It's not like its a piece of medical software where lives depend on the functionality. Which, by focusing on playtesting, was the choice you made.

I'm thinking that a person who would the upkeep of the Evolver scripts, managing the data, disseminating data out and making sure issues are tracked and are acted on would be logical next step though. You could easily scale the concept out and get even more detail.

Paul Tozour
profile image
Absolutely. Evolver is really only scratching the surface of what's possible.

There's also been some very interesting academic work related to this, as well as interesting procedural content generation work, being done by folks like

Alexander Jaffe http://homes.cs.washington.edu/~ajaffe/
Adam Smith http://users.soe.ucsc.edu/~amsmith/
and Gillian Smith http://sokath.com/main/publications/

... and several others whose names escape me at the moment.

James Yee
profile image
Good to see your game came out Paul.

As a writer in the Kickstarter community I recall seeing your project at the time and not being overly impressed by it. (Hence the reset you did) Do you think the Kickstarter would have gone better if it wasn't up to YOU to create it? Basically would it have been more cost effective for you to keep working on the game while letting someone else do the PR/Campaign management of Kickstarter?

Also for future Kickstarter creators how do you avoid the Apple Promo code problem you ran into? Just make sure your full game is a separate app if you plan on a free version?

Paul Tozour
profile image
Hi James -- I'd love to get your thoughts on the Kickstarter and what could have been done better there. Please feel free to send us a direct message via Twitter or e-mail us directly; I always appreciate honest, direct feedback.

Yes, the Kickstarter probably would have been more successful if I'd hired separate PR for it, but I'm not sure it would it have been cost-effective or that it ever would have actually brought in more funding than the cost of hiring the PR firm in the first place.

I didn't bring on a PR firm until the game was ready to launch on iOS; I did consider doing it for the Kickstarter campaign but it seemed like overkill.

As for avoiding the Apple Promo code problem: I don't have a good way to recommend that developers make their app available to backers for free outside of the limited promo codes Apple gives you.

Other developers I've spoken with have recommended briefly making your IAPs free for a very short time frame and telling your audience exactly when they need to download them, or putting hidden features / secret codes in your app (players need to click on a certain pattern on invisible hotspots). But the latter approach especially is risky since it risks incurring the wrath of Apple (and potentially getting banned from the App Store) or becoming open knowledge (and allowing people to pirate your game easily once they know the secret code).

Damian Connolly
profile image
Thanks for the excellent post-mortem, and congrats on the game!

Can you elaborate a bit on how you used discovery driven planning with your game? Did it change the game design, and if so, to what extent? Did you use it mainly in the prototyping stage, or throughout the entire process?

Also, your Evolver tech sounds pretty amazing. Is it specific to the game, or do you see yourself eventually spinning it off as middleware? It seems like an ideal candidate, especially for smaller studios.

Paul Tozour
profile image
Hi Damian -- Thanks for the kind words!

Although I'd love to be able to build middleware someday to help with this aspect of game design, I don't think it will be a case of extending Evolver to do that. The genetic algorithm component of Evolver isn't really anything unique or protectable, and the aspects relating to integration with City Conquest is too game-specific to be able to put it in middleware.

Regarding Discovery-Driven Planning: As luck would have it, I'm working on an article (or two) on DDP for Gamasutra right now. I hope to have them up by the end of the month or maybe early in March.

DDP is really a project planning methodology, so you want to use it throughout the project to make sure you have a good handle on the risks of the project, and ensure that you're prioritizing your efforts to learn as much as you can to reduce uncertainty as cheaply and as quickly as you can.

So on City Conquest, it drove every milestone. It didn't really change the game design directly but it did ensure that we worked on the riskiest aspects of the project first to ensure that we reduced our risks and ensured the smoothest possible path a completed, profitable game.

Paul Tozour
profile image
FYI, Gamasutra has now published my article on applying discovery-driven planning to games:

http://gamasutra.com/view/feature/191523/managing_risk_in_video_g
ame_.php

Josh D
profile image
Hi Paul,

Thanks so much for writing this postmortem, I was hoping you could elaborate a little bit on your monetization decision. You said that in retrospect, you didn't think making a free app with one large IAP was the best choice. I'm wondering why that is and what you think would be more effective? I'm currently involved with an app considering a similar pricing structure, so I'm very curious as to your thoughts and experiences with this. Any feedback would be greatly appreciated, thanks again!

Paul Tozour
profile image
Hi Josh,

What it comes down to is that to really maximize your revenues, you need to be able to do proper price discrimination. And by that, I mean that you need to be able to serve all the customers across the whole curve of different levels of willigness to pay -- the 5% of "whales" who will pay $20+ per month of your app without hesitation, the 10% of "dolphins" who might pay $5 per month, and the "minnows" who will pay maybe $1 a month on average ... in addition to all the non-paying users who just won't pay anything, which will likely be 3-10x your number of paying users.

So, I've come around to the realization that the full-unlock-through-a-single-IAP model is suboptimal because it can only do price discrimination at a single point: the decision of whether or not to pay for that one IAP. So you are getting no revenue at all from the many users who consider your price point too high, and you're getting a lot less revenue than you could from the minority of very wealthy "whales" who would be willing to pay much more to enhance their game experience.

Paul Tozour
profile image
I'd also recommend checking out GamesBrief -- http://www.gamesbrief.com/

They have some interesting papers on this.


none
 
Comment: