The original Xbox version of MotoGP, developed by Climax and published by THQ, was released in May 2002. This included a system link mode supporting up to 16 players, which was hugely popular with both visiting journalists and people at Microsoft, but unfortunately not accessible to the majority of gamers due to the practical problems of getting enough people, Xboxes, and televisions into the same room at the same time. This was a great shame because racing games are always more fun in multiplayer mode, and ours was no exception.
Microsoft first approached Climax about the possibility of adding support for their upcoming online service in May of 2002, but it wasn't immediately obvious how this could be done. We all agreed it would be great if we could have something out in time for the Xbox Live launch, but this left us very little time to add the necessary support, and people who had bought the first game would be justifiably unhappy if we released a new version just a few months later.
After much discussion, we decided the best option would be to produce an online-only demo version of the game, which would be bundled with the Xbox Live starter kits. This would offer three tracks and a limited selection of bikes free to everyone subscribed to Xbox Live, but would also be able to read save files from the original game, so if the player had previously bought this and unlocked more features, they would be able to use the full set of 11 tracks and import their customized bikes into the Live version. We hoped this would encourage more people to go out and buy the full game, at the same time as making the Live support a free upgrade to anyone who already had the original title. Designing our new version as an online-only game hugely simplified the implementation, as it allowed the entire team to concentrate purely on Live support for the duration of the project.
The original Xbox version of MotoGP
Unfortunately, negotiations between Microsoft, THQ, and Climax took so long that we didn't get the go-ahead to start work until early July, by which point we only had seven weeks left to complete the project. As if that wasn't enough of a challenge, George Foot, the programmer who wrote our original networking code, chose that summer to go away and get married, so I had to take over responsibility for the network subsystem, starting from a position of total ignorance.
I will never forget the moment when, six weeks into the project, with our matchmaking system finally working well enough that you could create and join sessions, and a Microsoft programmer due to show up the next day to help us add voice support, I finally got around to opening up a UDP socket ready to start talking to other machines, only to realize that I had absolutely no idea how to do this! I was tempted to email Xbox developer support and admit I didn't have a clue what I was doing, but a quick Google search turned up a beginner's guide to Winsock, which was enough to see me through.
Amazingly enough, we did manage to get the game finished on time, and even better, it has been widely praised as having the best Live implementation of any of the launch titles. I'm immensely grateful to everyone at Microsoft for their help - and most importantly, trust - during this project, because we had so little to show right up to the very end. We didn't even have a proper schedule, because we couldn't spare the time to write down anything more formal than "I'm pretty sure that if we don't sleep much and drink enough coffee we might perhaps be able to maybe get it done." It was an immense relief when, just a week short of the deadline, George returned from his honeymoon and merged his position interpolation code in with my packet sending layer, so we finally had bikes moving around the track in roughly the right places. Thankfully this worked on the first attempt, because it would have been extremely embarrassing if we'd ended up without any working races at all!
Seven weeks' experience as a network programmer isn't really enough for me to be able to claim more than beginner status, but for what it is worth, here are some things I learned during the course of this project.
Supporting Xbox Live Is Mostly A User Interface Problem
Supporting Xbox Live is really not about networking at all - it's primarily a user interface (UI) problem. Microsoft has already wrapped many of the more awkward network protocols into simple API functions, so the biggest challenge left for an Xbox Live programmer is to call these routines in the right order, and to present information to the player in a convenient way.
Xbox Live requires some standard user interfaces, such as this friends list, to be available both from the main menu system and via the in-game pause menu
Xbox Live requires some standard user interfaces, such as this friends
list, to be available both from the main menu system and via the
in-game pause menu
Most game interfaces are simple, with the bulk of development time being spent on attractive presentation rather than a solid technical underpinning. This is not good enough for a fluid Xbox Live implementation however, which requires a large number of UI screens, some non-trivial flows of control from screen to screen, and far more options than would be normal for a single-player game. An Xbox Live UI is constrained not only by what works in the context of your game, but also by Technical Certification Requirements (TCRs) for standard Xbox Live features such as the "friends" list and the invitation system.
At the very least I think an Xbox Live-capable UI layer should provide:
-choose game mode
However, when entering an Xbox Live session, this sequence is different depending on whether you are creating a new session, doing an Optimatch search, or looking for a Quickmatch, and in the latter case, there are yet more options depending on whether the matchmaking subsystem found a suitable session for you to join or decided to create a new one. This can quickly get out of hand unless you have a nice way to bring up a single screen in many different contexts, without the screen itself having to know where it came from or where to go next.
For example, our scoreboards were initially accessed from the top level menu, but once we started playing the game, we realised it would be better if they could be reached from inside the Xbox Live lobby as well. Although our scoreboards UI is a complex chain of three different screens, it only needed a single line of code to call this up from a new menu, telling the scoreboards system where to return when the user backed out of it, or if there was a network error. If this change had not been so easy to make, we would have had to settle for an inferior scoreboards implementation.
Rewriting Code Is Better Than Working Around Flaws
In a code postmortem at the end of the first MotoGP project, we came up with the theory that rewriting flawed code always saves time in the long run, and is better than trying to work around existing design flaws. When we started on the Live version, our initial analysis of the work involved suggested the UI requirements discussed above, but at the time our UI system provided none of that, and had already proven awkward to work with even for the simpler situations encountered in a single player game.
Rewriting your entire codebase in two months might sound like a tall order, but that is exactly what we did. Starting with such basic elements as our text rendering system, we rewrote every single UI screen, every speedometer, lap counter, and position indicator overlay, and every bit of game mode logic.
A month into the project I was worried that we had bitten off far too much, and were never going to get it all working in time. But after six weeks it was working well enough that we were able to start moving screens around and experimenting with different layouts, taking advantage of a flexibility we never had before. After seven weeks, we shipped what one gamer called "the best example of what an Xbox Live game can be". Today, we have a solid codebase that was a pleasure to work with as we moved on to MotoGP version 2.
It made for a stressful couple of months, but I'm convinced we would never have managed such a good Live implementation if we had stuck with our old UI system, and might well have failed altogether. I can now confidently state that rewriting huge swathes of code takes a lot less time than you might expect, and the benefits can be enormous.
A State Machine Can Help Solve Asynchronous Networking Problems
When any operation may take an indefinite amount of time to complete, you have to either build a state machine to track the current status of every task, or fire up a separate thread for each potentially blocking operation.
I don't much like threads, mostly because they can be a nightmare to debug, but also because of the wasted resources of separate stack memory for each one. So I decided to build a state machine.
MotoGP uses a list of NetTask objects, each of which encapsulates a single online operation. These are pumped once per frame (60 times a second) to do whatever network processing they may require, and each task provides a status that can be checked by the UI system. If all the tasks have an OK status, everything is normal and the UI carries on as normal. If any tasks have a BUSY state, the UI displays an animating progress indicator, and ignores user input until the task has completed. This allows the network code to be written in a simple way, completely ignoring UI issues, for instance:
For its part, the UI code for entering the friends screen looks like:
In the normal case, the NetBusyScreen will display an animating "network busy" graphic as long as the gLiveFriends network task is in a BUSY state. When the network state changes to OK, the NetBusyScreen detects this and calls the provided hook function, which brings up the friends screen to display the information that has just been downloaded.
If the friends enumeration fails, the network task sets an ERROR_MESSAGE status, in which case the UI layer displays that message to the user and dismisses the NetBusyScreen, so the hook function will never be called and the friends screen will never be activated.
If a really bad error happens, one of the network tasks might set a status like SESSION_LOST, DASHBOARD_TROUBLESHOOT, or CABLE_UNPLUGGED, in which case the UI layer will take some more drastic action such as quitting right back out to the top level of the menu system, or rebooting to the Xbox Dashboard network troubleshooter.
This design worked extremely well on MotoGP, keeping the UI and network systems simple, separate, and as linear as possible. It got to the point that I sometimes forgot we weren't multithreading, and I started worrying about possible race conditions if the network layer changed state in the middle of a piece of UI code - only to feel extremely stupid when I remembered we were just pumping each of these tasks one after another. Lack of sleep was my only excuse.
I'm sure there are many other nice ways of structuring such things, but my main point is that you need to choose a good one at the start of your Live project. Asynchronicity isn't going to go away, but a good design can make it easy to live with.
Our system link code in the first MotoGP game blocked while establishing TCP connections, and the very last couple of bugs that held up our submission were caused by too many people trying to join a session at the same time, and crashing the network stack because it was unable to respond to them. Blocking is evil: don't ever do it.