Failure to anticipate platform and API design requirements. Many of the required features of the Xbox 360 platform were implemented hastily after-the-fact only after their absence was discovered, rather than being integrated into the original design of the code. Guest support on Xbox Live is a good example of this. When a player joins a split-screen multiplayer game, he must be able to bring players who do not have personal Live accounts with him.
By the time we realized this feature was not working correctly, a very strong relationship had been established between player accounting in the network game UI, player representation in the game world, and player profiles on the Xbox. Supporting players who did not have matching profiles required a significant overhaul of that accounting system. An extra wrinkle came from the fact that profiles have privileges associated with them that may require a player with a profile to be treated as a guest anyway under certain circumstances -- in other words, game developers must themselves implement the difference between Xbox Live Silver and Gold accounts. The system used by the final game relies on the persistence of data in partially re-initialized structures, an unfortunate compromise that should have been avoided.
The performance requirements of the platform also had a huge influence on our design. A smooth, stable framerate is important to a good presentation, and to the proper functioning of the Xbox Guide, so no blocking could occur when dealing with files or the network. As the Xbox 360 uses polling for most asynchronous operations, this necessitated the use of state machines in many places. The most egregious example of this is the game saving and loading screen, which is running two separate state machines at the same time -- one to update the UI, and one to perform the file I/O.
The number of synchronization bugs and mishandled edge cases resulting from this was frightening. Another tricky spot was the loading screen. Most of the time, our engine uses a dedicated thread to manipulate the Direct3D device object, but the bitmaps that needed to become texture objects were being loaded by a different thread. The device had to be switched over to the loading thread and the load screen rendering had to be interleaved with the actual loads, then the device returned to the render thread. At other places in the engine, we needed to be able to show the progress screen but were not able to tick it "by hand" during a lengthy process, so the load screen class also had to be able to update and render itself without being ticked by another object.
Sometimes the requirements for certification diverged from the functionality provided by the system libraries. The mute feature, for example, must enforce muting in both directions -- when a player mutes another, neither the muter nor the mutee may hear the other. The catch is that the list of muted players is only available for local players; it is not automatically propagated. The mutee doesn't know he has been muted until the game implements a method of notifying him.
Once this was done, a bug was found (in our game code) by which a user could force another user to un-mute him by using his own mute button. The was so late in the project that rather than redesign the mute system additions, a "countermand" message was created so that different consoles could order each other to re-mute players when necessary, wasting network traffic and complicating the related code. And it’s still not perfect -- while it passed cert, we've heard several reports of players being unable to communicate when appropriate.
Flaws in the team. The personnel we assembled for this project had all been Mac and Windows developers for years, but among them there was little porting experience, even less experience with console development, and for most of us this was the largest single team we had ever been a part of. Some of the items listed in this postmortem may seem obvious or mundane to more experienced readers -- well, it was new to us at the time. We were expanding our own envelopes. In theory, the presence of console veterans "on call” elsewhere in the company and our partner studios would keep everyone on track.
Printer-Friendly Version
