Improved sync system. Nobody on the Age of Empires development team would argue the need for the best sync tools possible. As with any project, when you look back on the development process during a postmortem, some areas always stand out as the ones you spent the most time on, but could have spent much less time on given more up-front work. Synchronization debugging was probably at the top of this list as we started development on RTS3.
The RTS3 synchronization tracking system is primarily geared towards rapid turn-around on sync bugs. Our other priorities in developing it were ease of use for the developers, the ability to handle an arbitrarily massive amount of sync data pouring through the system, the ability to totally compile out synchronization code in a release build, and finally the ability to completely change our test configuration by toggling some variables rather than requiring a recompile.
Sync checking in RTS3 is done through two sets of macros:
#define syncRandCode(userinfo) gSync->addCodeSync(cRandSync, userinfo, __FILE__, __LINE__)
#define syncRandData(userinfo, v) gSync->addDataSync(cRandSync, v, userinfo, __FILE__, __LINE__)
(There is a set of these macros per sync "tag," where a tag represents a given system to be synced -- in this example, the random number generator, cRandSync) These macros both take a userinfo string parameter, which is a name or indication of the specific item being synced. For example, a sync call might look like:
syncRandCode("syncing the random seed", seed);
Synchronous console commands and config variables. Console commands and configuration variables are of immense value to the development process, as any Quake mod creator will attest to. Console commands are simple function calls, done via a startup configuration file, the console within the game, or UI hooks, that call into any arbitrary game functionality. Config variables are named data types, exposed through simple get, set, define and toggle functions that we use for all sorts of testing and configuration parameters.
Paul derived multiplayer-enabled versions of our console command and config variable systems. With these, we are able to easily turn a normal config variable (such as enableCheating) into a multiplayer config variable by adding a flag to the config variable's definition. With this flag enabled, that config variable will then be passed around in a multiplayer game, and synchronous game decisions (such as whether to allow free resource tributing) can be based off of the value. Multiplayer console commands is a similar concept -- calls to a multiplayer-enabled console command are passed around and executed synchronously on all client machines.
Through the application of these two tools, the developers have a simple way to use the multiplayer system without writing any lines of code. They can quickly add new testing tools or configurations, and easily enable them in the network environment.
The synchronous simulation, peer to peer model was used successfully in the Age of Empires series of games. While it is critical to acknowledge the importance of investing time creating tools and technologies to combat the key challenges of this approach (such as synchronization and network metering), the viability of this architecture is proven when applied to the real-time strategy genre. The subsequent improvements we have implemented for RTS3 lead to an overall multiplayer experience that is virtually indistinguishable from single player in all but the most horrible network conditions.