What Went Wrong
1. Out of Sync errors. This was actually something that went wrong with the Myth II 1.5 update, and we managed to remedy it with version 1.5.1. To understand what an "Out of Sync" error really is, it is important to understand how Myth handles networking and gameplay in general. When a game begins, multiplayer or not, there is a starting random seed that is generated by some arbitrary value. In a network game, this initial random seed is synchronized between all the players at the start of the game. As the player, or the many players issue commands (or orders) to units, these commands are recorded, and saved to a file and, in the case of a multiplayer game, sent over the network to other players.
Since Myth only sends the player's commands over the network to other players, and not actual updates of the locations of objects in the game, this approach to networking is very bandwidth efficient. All players receive the same commands for a given game tick (the base unit of time in Myth, equivalent to 1/30th of a second), and these are all processed individually on each client's machine. Since the code running on every player's computer is the same, the expected result is for the game to play in exactly the same way when it receives the same input commands from all the players.
Furthermore, since these commands are also stored in a file, which also stores the game's initial random seed, and the other game-specific information such as the map on which it was played, it is possible to playback the sequence of gameplay events in the exact order in which they had happened, to view basically a recording of a previous game. This is a very useful feature, and players often like to save their best recordings so they can view them at a later time. But more importantly, this feature also happens to be an excellent debugging tool. If something wrong happens in the game, its sequence of events may simply be replayed in order to examine what exactly went wrong and then fix it.
|The Myth II solo campaign level "Twice Born."|
However, while such a system may appear to be a superior design to other systems, such as those that update the state of the world over the network, it is not without its flaws, especially when used with multi-platform clients. The problem arises with the basic assumption that the exact same behavior will be happening on all machines playing together in a network game, or viewing the same recording. The problem is, due to various issues, this may not end up being the case, resulting in an "Out of Sync" error - that is, a situation where the game-state on one machine differs from the game-state of a different machine when they should be the same. I will discuss in detail the two major causes of such errors that we came upon after we released Myth II 1.5, and how we were able to debug and fix them in version 1.5.1.
The first, and often overlooked issue when designing cross-platform software is related to the specific characteristics of the target hardware on which it will be running. The particular problem we ran into with Myth II had to do with the precision of floating point variables on the different types of CPUs. In particular, we ran into this issue when we added a new unit propulsion feature in Myth II 1.5 that turned out to be the cause of numerous "Out of Sync" errors that players were experiencing. The programmer who had coded it had used floating point numbers for all physics-related calculations having to do with unit propulsion. However, it turned out that the floating point result of a division operation would contain a different number of significant digits after the decimal point on the x86 PC than on the PowerPC Mac. Of course, these resulting values which had different levels of precision were then used for further calculations, such as multiplication, and eventually the tiny amount of extra digits would cause a different logical path in Myth's decision making tree. This, of course, resulted in different gameplay happening between the Mac and Windows version of the game, and caused an "Out of Sync" error to be reported.
Bungie had already known about this issue when designing Myth II, however, and all existing logic that involved decimal numbers was done using fixed integer values. A fixed integer value is simply an unsigned short (2-byte) integer that represents a value from 0.0 to 1.0, where the maximum 65535 value represents a 1.0, and every value below this would be divided by 65535 to achieve its decimal equivalent. In effect, this made Myth not use any floating point arithmetic at all, disregarding the use of the FPU completely, and arguably wasting potential processing cycles on that execution unit. Perhaps those cycles could have been put to better use, for non game-play critical code that was local to each client and could not cause "Out of Sync" errors, but this was not done. In any case, after we had discovered this problem with the new propulsion feature, the feature had to be re-coded using fixed integer arithmetic.
The second cause of "Out of Sync" errors turned out to be a result of poor programming. The problematic pieces of code in question, some coming from the original Bungie codebase, while others from our past updates, were essentially doing the same thing. A logical gameplay decision was being made based off of an undefined value in memory. This was often caused by the use of an uninitialized variable, or trying to reference a value in an array using an out-of-bounds index. In both such scenarios, the memory location would contain a different value on the different client machines participating in a Myth II networked game, and thus creating the "Out of Sync" error when a gameplay decision was made based on that value. In order to fix these types of "Out of Sync" errors, the faulty code had to be rewritten or altered so that it would not access and use memory containing undefined values.
It turned out that both of these causes were platform dependent. A Mac user would not go "Out of Sync" if playing only with other Mac users, and the same was true for Windows users. It is obvious that the "Out of Sync" errors that were being caused by the differences in floating point precision would have been behaving in this way. But why were the other "Out of Sync" errors behaving in the same manner? The answer comes from the fact that a different compiler was used to create the executable for each of the two platforms. While theoretically the memory locations being accessed would hold undefined values, in practice these values were identical for all players on a given platform. This was because the memory locations that were being accessed happened to be part of static code blocks or values related to gameplay in memory, rather than dynamically allocated memory or values that are based on the local game state.
Because of this, these values were guaranteed to be the same for all players on the same platform, as the same build of Myth II was being used, created by the same compiler, and thus having memory ordered in the exact same manner. However, when the games were played with other players on a different platform, their version would not have memory structured in the same manner, since it was made by a different compiler, and thus would give different values when accessed - leading eventually to an "Out of Sync" error.
So now we know that the "Out of Sync" issues are platform dependent, but how do we actually find the faulty pieces of code that are causing these? The answer comes with the gameplay-recording feature that I've mentioned earlier. When an Out of Sync error occurs, we can simply save a perfect replay of everything that happened in that game. This replay can then be played back, and will play fine on the same platform as the one on which the game was hosted, while giving an "Out of Sync" error when played on a different platform.
Myth detects "Out of Sync" errors by periodically comparing the current random seeds of all players in the game to make sure they match. If the game is actually a replay being watched, the seed of local machine is also compared with those random seeds. If the seeds don't match, a red bar of text saying "Out of Sync" will appear on the screen, and the error is logged to the debug log file, along with the exact game-time when it was discovered. The problem is that the game-time appearing in the log does not correspond to the exact time when the gameplay code began to diverge - as the "Out of Sync" detection checks only happen at specific time intervals. So, we had to come up with a mechanism to get closer to the part of the code that is causing the problem.
What we did was make a toggle that, when activated, would replace the function myth_random, our random number generator, with a new function called debug_myth_random, having two extra parameters: a string and an integer. The string would correspond to the name of the source code file from which the function was being called, and the integer would be the exact line number in that file. Rather than changing every piece of code that used the myth_random function, we made a C-preprocessor macro that would replace all occurrences of myth_random() with debug_myth_random(__FILE__, __LINE__). These two parameters, __FILE__ and __LINE__, are built-in variables in C that hold the file name and line number for the code that is currently being compiled. The function debug_myth_random would work identically to myth_random, except for the fact that it would log some information to a debug text file every time it was called. Particularly, we made it log the file name and line number from where it was being called, the current game time (in ticks), and the random seed that it was returning. After this change was made, the PC and Mac version of Myth II were recompiled, and the game recordings that produced "Out of Sync" errors were viewed with both versions. By comparing the outputs produced by both applications, we were able to pin point the exact game tick at which the random seeds started diverging, and what parts of the code were making extra calls to the random number generator between platforms.
Using that technique, we went through all reported cases of "Out of Sync" from version 1.5, and were able to fix a good number of these issues for version 1.5.1 of Myth II. This end-result might appear fit to be in the "what went right" category, but the fact that these "Out of Sync" errors happened in the first place as well as the amount of headaches they caused us before we came up with a good strategy to debug and fix them, persuaded me to list it under "what went wrong".
|"Blue and Grey", a third-party plugin for Myth II, themed around the American civil war.|
2. Graphic issues. One area we had a lot of difficulty with was the problems we encountered with the different 3D rendering options that Myth uses, and making sure everyone got the best possible experience with Myth - that is, decent FPS (frames per second) when playing. Myth II, being a game that was released in 1998, supported some pretty early and obscure rendering technologies, in addition to some more common ones. First off, it had software rendering - which means it could (and still can) run on machines that provided no hardware 3D acceleration - it would just do it all in software on the CPU. This mode was very compatible, as it ran perfectly everywhere - but the resulting graphics did not look as good compared to when they were rendered using hardware renderers (ie: 3D graphics cards). In the hardware acceleration area, Myth II originally supported the following technologies: Rendition (Windows only), 3DFx (Windows and Mac), RAVE (Mac only), and Direct3D (Windows only). With version 1.4, we also added support for OpenGL for Mac OS X (since RAVE and 3DFx were unsupported on the platform).
Early in the 1.5 development cycle, we made a decision to drop Rendition support, because hardly anyone (not even us) was using such hardware to play Myth, and we couldn't test if the ancient code was still working properly. We have yet to receive a complaint from any players who miss that particular rendering option.
As we were working on 1.5, Apple released update 10.3.3 to its Mac OS X Panther operating system - breaking Myth II 1.4's ability to use OpenGL on that (and future) Mac OS X versions. The cause of the problem was a very particular (and in retrospect non-accurate) way that we were using to test if the video card had enough memory, and the fix was simply to disable that test. Additionally, we made several other changes to our OpenGL code in 1.5 (specifically changing the amount of memory that's allocated to Myth's texture cache) that boosted FPS for some users with lower end video cards.
While the OpenGL code changes produced satisfactory results for most users on Mac OS X, not everyone was happy with Myth II's rendering performance. It turned out that Myth II's Direct3D implementation was causing extensive "water lag" (that is, a huge drop in FPS when water areas were rendered) for certain users - specifically those with Geforce 4 and Geforce 5xxx graphics cards (which, one would think, should be able to run circles around a game released in 1998). When we tried to the debug the problem, it turned out that the slowdown was due to CPU-intensive code that the NVIDIA driver was executing - specifically, it was converting pixel formats on the CPU - and this was slowing everything down. What's interesting, is this was happening with the latest driver downloaded from NVIDIA's website - but the one downloaded from Windows Update (for Windows XP) did not have this problem at all, for the same video card. To get a sense of the performance drop, on a certain map where one test system would get approximately 80 FPS on a certain water area with the Windows Update drivers, the same machine would get 10 FPS on the water area with NVIDIA's latest drivers. This was unacceptable, and since not everyone was using Windows XP (thus we couldn't tell them to get those drivers), nor did we want to tell users to use old drivers (which most likely had other problems), we decided to port our previously Mac OS X-only OpenGL code to work on Windows for version 1.5.1 of Myth II.
The changes that were needed to be done to our OpenGL code for it to work on Windows were fairly minimal, since OpenGL is for-the-most part cross-platform. The differences between the Mac OS X OpenGL code and the Windows OpenGL code were how the OpenGL context was made to bind to the window, as well as cleanup and unbinding of that, and the code to switch resolutions. Other than those two areas, we had to make a couple minimal other changes to get it to work correctly. In the end, our OpenGL Windows implementation gave higher FPS than Direct3D on the Geforce 4 and Geforce 5xxx video cards (the ones suffering problems with rendering water), and the performance on these cards was better both under regular circumstances (no water) and when rendering water (which under OpenGL gave no lag). So essentially, we had solved the main complaint about 1.5 in regards to rendering - but still, all was not well.
It turned out that in the process of trying to eliminate the water lag, before deciding to make OpenGL run on Windows, we made a change to some Direct3D code. The change involved locking some data with a Direct3D call before manipulating it, and unlocking it afterwards, which increased FPS slightly on our Windows XP test machines. Unfortunately, and our internal testing team missed this, it severely hindered performance on computers running Windows 98, and we had to come back and correct that after finding that out. Another problem was that while porting OpenGL to Windows, we made several changes to the underlying code, including disabling an Apple OpenGL extension (which had no impact on performance on our Mac OS X 10.2 and 10.3 test systems). However, one of our internal testers was using Mac OS X 10.1, and for that individual that change gave a huge FPS drop (by about 6 fold), so we had to revert and put that extension back into use. Furthermore, while OpenGL ran perfectly and beautifully on NVIDIA graphics cards, this was not the case for other hardware. For example, on an ATI Radeon x800, a user reported getting 30 FPS in Myth II - a very low number for such a high end card. However, Direct3D worked at much higher FPS for that player, so it was not really a problem. Also, our OpenGL code was apparently not compatible with integrated graphics chipsets found on some laptops, as users who had those reported getting a black screen when trying to play using OpenGL.
In the end, between all the different graphic rendering options that Myth II 1.5.1 offers, most players can find a rendering option which runs well on their system. However, I was disappointed by the different behaviors that different graphics hardware and operating systems exhibit to the same code - which, theoretically, should work fine everywhere.
3. OpenPlay networking. Bungie Studios used a cross-platform networking library called Uber to handle all network code in Myth and Myth II. This library was created by Apple Computer and was made available exclusively to Bungie to be used in Myth. The library supported AppleTalk networking on the Mac, and TCP/IP networking on both Mac OS and Windows. At the time, most Mac game developers used the separate Mac-only NetSprockets library for networking, also made by Apple Computer, and because of this most Mac-versions of games were unable to network to their PC counterparts. Thanks to the cross-platform Uber library, this was not true for Myth and later Myth II.
However, while Uber proved to be very useful for Myth: The Fallen Lords, which was the first game that used the code, Apple did not release Uber as an official SDK to other game developers in that form. Eventually, in 1999, Uber was renamed to OpenPlay 1.0 and released under an Open Source license by Apple. With version 2.0 of OpenPlay, the old NetSprocket APIs were incorporated into OpenPlay to support developers that needed to port code to Mac OS X that relied on (unsupported) NetSprocket APIs.
Since OpenPlay was an updated version of Uber, we thought, why not replace Myth II's outdated Uber code with the more modern version of the same thing in OpenPlay. This was one of our goals in Myth II 1.5. We managed to modify the code and our build process to link to OpenPlay, and it was what we shipped with both of our Public Beta releases of Myth II 1.5.
However, all was not well. First off, OpenPlay, unlike the Uber code that Myth II used previously, separated the different protocols that it was able to use into their own module files that it loaded on startup. It turned out that when we shipped the latest version (at the time) of OpenPlay with a public beta, that the TCP/IP OpenPlay module for Windows would not run correctly on Windows 98 - while the same version worked fine on the Mac and on more modern Windows operating systems. So, we quickly replaced that module with one from a previous version of OpenPlay that had no such problems. Unfortunately, now we were using parts from different releases of OpenPlay together - and that was just asking for trouble.
Although the majority of the games that were played on these new OpenPlay-using Myth II builds were fine - a large number of them, significantly more than of previous versions of Myth II, went "Out of Sync." Though many of these "Out of Sync" issues were unrelated to OpenPlay, and I talk about these earlier in the article, others seemed to be directly caused by OpenPlay networking problems.
The difference was easily visible: bugs in the game engine code that caused "Out of Sync" would, when watching the recording of the game, give different behaviors across platforms, while these other "Out of Sync" errors would either go "Out of Sync" on both platforms or play normally on both. Basically, packets were not being delivered or received correctly by OpenPlay, which caused inconsistent behavior between different machines in multiplayer games.
We could not leave Myth II in such a state, and since we were near the release date of Myth II 1.5 - the easiest solution was to just pull OpenPlay. For the final 1.5 release (and subsequently, 1.5.1), we reverted back to the Uber code that Myth II used originally, with a few changes (mainly some additional support for Mac OS X) that we extracted from the OpenPlay source code and merged into our version of Uber. We've also sent our modified Uber code back to the OpenPlay developers, in the spirit of Open Source so that they can potentially make some use of it in making OpenPlay better.
4. Installers. The area of installers is often overlooked, but it is very important to choose the right options in this area to ensure a smooth end-user experience. With our updates to Myth II, as well as the new demo that we were releasing, we needed to make a number of different installers.
|The Mac Installer for Myth II 1.5.1, made with Apple Installer.|
On the Windows side of things, we tried two options, and ran them through our internal testing team. First, we tried using the default Windows Installer made by Microsoft to package and distribute our updates and the demo. The results were less than stellar. Windows Installer comes pre-installed on all recent Windows operating systems and is designed to read installer package files (with the extension .msi) that contain all the required information and data to install the software. The problem is that we could not just offer the .msi package file for download, as we could not guarantee that all users would have the latest version of Windows Installer available on their computer. Microsoft, of course, offers a solution for this problem, which is basically bundling a whole copy of Windows Installer (in fact, several copies for newer and older Windows operating systems) with the .msi file, to produce something that will work on all the target platforms. This works, but the result is the complexity of the installer being composed of a bunch of files in a folder, rather than a preferred single-file approach. Since we would be making our installers available for download, we would need to make it one file either way, thus zipping the folder - which then produces redundant compression (as the contents of the .msi file are already compressed). What's more, when we were experimenting with this approach, one of our beta testers reported that Norton AntiVirus gave a "malicious script detected" alert when running the installer - something that we obviously don't want our users to see and freak out from.
So we decided to look for an alternative installation method for Windows. In our search, we came upon NSIS - Nullsoft Script Install System. Originally created by the same people who made Winamp, it has been released as an Open Source project on http://sourceforge.net/ and could be used free of charge - so we tried it out. The way it works, is you write a text-file .nsi script (in a language/syntax specific to NSIS), in which you reference other files that would be installed, and then compile that, producing a single executable (.exe) file containing everything the installer needs in a compressed form. Since the result is a single .exe file, it is easy to download and use, and the interface that was produced was very elegant (especially when we customized it with Myth-specific graphics). For Myth II 1.5.1 final, we used this method of making an installer for both the Myth II 1.5.1 update, and the Myth II 1.5.1 Demo. The installer for the demo was more complex, as it installed it from scratch, and thus had to make start-menu shortcuts and come with a complete uninstaller.
On the Mac side, we had to support Mac OS X and Mac OS 8.6-9.x with our installers, as Myth II 1.5.1 could run on both of those platforms. Originally, we tried using the proprietary Installer VISE installer maker application by MindVision Software, which we had a license for. The advantage of Installer VISE was that it was able to create a single installer that would work on both Mac OS 8.6-9.x and Mac OS X. However, our experience with it was not so good. Some users of Mac OS 8.6 reported problems with getting the installer to work, despite Installer VISE supposedly supporting that platform. Installer VISE was also not as flexible as we wanted it to be, and thus some of the more complex things we wanted to do (such as setting permissions on files after they were installed) were not possible. The final straw for us was when an update to Mac OS X (version 10.3), broke Installer VISE such that we couldn't run it to make the installer. MindVision Software released an updated version of Installer VISE, but unfortunately in order to upgrade we would have had to buy another license of the product. So we decided to ditch it, and go with Apple's Installer instead, which supported Mac OS X only. The beauty of Apple's Installer was that we could do the whole installation using a UNIX shell script, which we programmed in Perl, allowing us to have a lot of control what the installer was doing. For the users running Mac OS 8.6-9.x, we had to make a manual installer archive with a ReadMe explaining where the files had to be placed to upgrade their version of Myth II. This was all for the Myth II 1.5 and 1.5.1 update installers, as for the demo we did something different.
The recommended way of doing installers for new software, according to Apple, is to use drag-and-drop installers whenever possible. This is something that is uncommon on other operating systems, but the idea is that you end up with a folder of the application (or for some programs, just the application itself), and then drag it to wherever you want it installed (such as the Applications directory). This is most often done on the Mac with a technology called disk images, that is - virtual disks that are mounted from a file, and then can be used just like any read-only disk (to copy files from it, for example), before being unmounted. This is a very common practice for installing software on Mac OS X, where the disk image files come in a file format with the extension .dmg.
Unfortunately, we could not use that - as it is not supported on Mac OS 8.6-9.x, and thus we used the Apple's older disk image format with the extension .img, that could be used on both platforms, which then housed a self-contained folder of Myth II Demo which could be dragged and dropped to anywhere on the user's hard drive and then played.
While in the end, we arrived at something that works (and indeed our final installers are pretty good), we had a lot of trouble along the way finding the options that best suited our needs, and there's still a couple of areas where our installer situation can be improved (such as having a real installer for the Myth II update on Mac OS 8.6-9.x).
Though we were definitely faced with numerous obstacles during the development cycles of Myth II 1.5 and Myth II 1.5.1 updates, I believe the team at Project Magma did a great job in overcoming them for the release of Myth II 1.5.1. Indeed, the Myth II community has been very receptive and happy with the update, which is really the ultimate test, in my opinion, of whether Myth II 1.5.1 was a successful update. I would like to congratulate the whole team at Project Magma, as well as all of our beta testers and players who have submitted bug reports and assisted us in making Myth II 1.5.1 possible!