Running Backend Services

Early on, we realized that there would need to be a number of services running on the backend to support Sandstone. Each of these services potentially needs to talk to other services, and some of them need to be accessible by web servers that allow people to play our games. We chose to have most of the API for these services be HTTP requests.

This makes it easy for any web framework to talk to the Sandstone service, no matter what language it is written in. As long as it can make HTTP requests (which is a fundamental ability of most web frameworks), it can interact with the services.

Initially, most data was passed back and forth as XML. This proved to be fairly unwieldy, as most data was simple data, the most complicated parts being name-value pairs and lists.

Eventually, we changed over to use JSON for this data, which was a lot smaller than the equivalent XML and was much easier to parse in the languages we used.

Since these service APIs are implemented as HTTP requests, we needed to use a language that makes it easy to implement HTTP requests.

We chose to use Python instead of PHP or Ruby, because it is a very self-consistent language, has a lot of support for web services, and is easy to bridge to C++ code (which is very useful for running game servers).

Content Distribution

When it comes to running a game in a web page, rendering 3D graphics in a browser is not the real problem. Rather, a large part of the problem is getting the 3D art assets to the client machine in the first place. The art assets for 3D games are generally much larger than the art assets for more traditional browser-based 2D games.

From the beginning, we recognized that it would not be acceptable to download assets as needed, stored in memory. It has gotten to a point where waiting for some of the more complex Flash games to load is becoming unwieldy, and most 3D games would likely be larger than this.

Rather than come up with a way of compressing art assets to the point where the download and extract times would be acceptable (an extremely hard, if not impossible problem), we instead decided to try to build the Sandstone Player so that it made it possible to create better user experiences while waiting for the download.

We chose to write our downloader in Java, since we were already using Java in the Sandstone Player to bridge the gap between the browser and our C++ game engine. This also turned out to be easier than C++, because Java has much better library support for making HTTP requests, which is how the content is downloaded.

This downloader can be run as a separate applet from the applet that runs the game, and it can communicate with the page around it, so the page can do whatever it wants with this information.

It can put up a progress bar, it can let you chat with other people also waiting for the game to download, and it could even give you a small 2D minigame to play while you're waiting for the download to complete. The bottom line, it becomes more of a user experience question than a technical question at that point.

We made two other important decisions aimed at improving content distribution. First, we chose to break up our content in small, reusable chunks. And second, we chose to cache these chunks on the local disk of the client machine.

This means that if two scenarios use 99% of the same content, if you've already downloaded all the content to play scenario A, then you will already have 99% of the content it needed for scenario B cached on the local disk. Therefore, the scenario B download will be very small.

Next, we gave our content system two versioning systems. The first system simply changes the identifier of a content package, and is used for major upgrades of a package, when a change is made that will "break" other existing content.

For example, when code is changed that won't work with other modules calling it, or a 3D model is changed and is a completely different size so that it won't fit where it used to. When this is done, any other package that uses this package can manually change to use the new version.

This is useful because the new and old versions are identified differently, and they can coexist on a client machine at the same time. So if Scenario A uses Version 1 and Scenario B uses Version 2, a client machine can have both Version 1 and Version 2 on disk at the same time to play both Scenario A and Scenario B.

The other versioning system changes the revision of a package in place. This is used for non-breaking changes of a package, things like bug fixes. For example, a memory leak in code is fixed, or a 3D model is changed to fix a mesh that sometimes renders incorrectly. Updates of this type happen automatically when getting content, so the game you play stays up-to-date all the time.

The Making History Gaming Headquarters (GHQ) facilitates the sharing of user created content.