Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
Infinite Space: An Argument for Single-Sharded Architecture in MMOs
View All     RSS
June 19, 2021
arrowPress Releases
June 19, 2021
Games Press
View All     RSS







If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

Infinite Space: An Argument for Single-Sharded Architecture in MMOs


August 9, 2010 Article Start Previous Page 3 of 5 Next
 

Game Design Challenges

As we noted earlier, player density can be a real challenge from a technical perspective as well as a game usability perspective. In EVE, we run into two general kinds of harmful player clustering which cause design headaches. The first is what used to be called the "Yulai problem."

A few smart players determined that the Yulai solar system was particularly well connected to the various areas of the star cluster, and started selling their wares there in bulk. Those few canny marketeers on their own weren't a problem, but it didn't stay that way for long -- as more buyers visited the system more sellers set up shop there.

As more sellers flocked in it became more desirable to buy there. Pretty soon it seemed like the entire playerbase was shopping in Yulai, and that system's population made up a noticeable percentage of the total online playerbase at any given time.

In an effort to curb the growth of this trade hub before it started causing serious server issues, we made some changes to the map by shifting jump routes around to lessen the appeal of Yulai. Within a few months the "Yulai problem" became the "Jita problem," as players figured out the new best location and moved all their business there instead.

The formation of such hubs seems to be an inherent human phenomenon, and while we still have regular design discussions about effective ways to distribute market activity more evenly without distorting the market itself, it's eventually something that we solved with hardware and software solutions.

The second clustering issue is almost the polar opposite to the problem of market hubs -- that of huge battles for strategic objectives. Whereas hubs are permanent and predictable fixtures, fleet-sized combat tends to be transitory and unpredictable. In our case at least, the emergent gameplay that delivers such value regularly compels huge political power blocs numbering thousands of players to make spirited attempts to beat the life out of their rivals.

Without warning a particular system's population will shoot up from maybe half a dozen players to over 1200, generating a huge spike in server load before rapidly dropping back down to its original level. The abstract design solution to this is to spread the combat out across multiple systems simultaneously, but this is directly opposed to the ageless military principle of "hit them with everything we've got." It is an ongoing challenge to figure out how to implement this in practice in such a way that it's actually beneficial for military commanders to split their forces under the majority of common circumstances.

Programming Challenges

A large-scale single-sharded environment is not without its unique challenges on the code side either. Here are several of the problems we faced in our implementation.

Running out of memory. Over the years, maximum memory usage on the nodes that run EVE's solar systems (Sol nodes) have been steadily increasing with the increased population and expansions to the game. In particular, solar systems such as Jita have been pushing the process memory usage up. When EVE launched, it ran on 32-bit Windows Server 2003, giving us a virtual memory limit of 2GB for each process. Later we upgraded to a 64-bit OS and this increased the limit to 3GB. But sometimes even this would not be sufficient, so we decided it was time to make the server processes 64-bit.

Initially we were slightly worried, as the physics simulation has to stay in sync on the client and the server. We were uncertain whether it would drift apart due to different code being generated for the complex mathematics involved. Eventually careful testing revealed that this would not be a problem and that the algorithms were numerically stable in such a mixed environment.

The release of the 64-bit binaries was without incident. In fact, performance increased quite a bit, due to the better optimization possible with a larger number of registers in 64-bit mode. Baseline memory consumption did rise, of course, because all pointers were now twice the size. However, we were finally free from the virtual memory constraint of 3GB. Now each process can allocate as much as it wants and never run out of address space.

As for physical memory, it turns out that the 4GB of physical memory on each machine -- which typically runs two Sol node processes -- is quite sufficient (as seen in Figure 1). Most of the allocated virtual memory lies dormant and there is little paging. We still haven't seen a node die because of page thrashing.


Article Start Previous Page 3 of 5 Next

Related Jobs

Disbelief
Disbelief — Cambridge, Massachusetts, United States
[06.18.21]

Programmer
Disbelief
Disbelief — Cambridge, Massachusetts, United States
[06.18.21]

Senior Programmer
Insomniac Games
Insomniac Games — Burbank, California, United States
[06.18.21]

Technical Artist - Pipeline
Insomniac Games
Insomniac Games — Burbank, California, United States
[06.18.21]

Technical Artist - Pipeline





Loading Comments

loader image