Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
Infinite Space: An Argument for Single-Sharded Architecture in MMOs
View All     RSS
June 14, 2021
arrowPress Releases
June 14, 2021
Games Press
View All     RSS







If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

Infinite Space: An Argument for Single-Sharded Architecture in MMOs


August 9, 2010 Article Start Previous Page 4 of 5 Next
 

Programming asynchronous systems and distributing execution. Running logic on top of a cluster of distributed nodes means that a typical function call might cross process boundaries and even go across the public internet before returning. This calls for a lot of asynchronous programming, which is notoriously cumbersome to implement. This is where Stackless Python comes to the rescue.

At CCP we firmly believe that we need to make programming as simple and intuitive for the game programmer as possible. We decided early on that in order to create a complex game such as EVE we would need some kind of multi-threaded programming. We also recognized that using scripting for game-level code was necessary simply to get things done. We were very fortunate to come across Stackless Python at a very early stage in development and ended up using it for all game logic.

Stackless Python is a variation of the Python programming language that introduces the concept of "tasklets" and "channels." A tasklet is a thread of execution that is independent of OS threads, and tasklets communicate and synchronize with the help of channels. They consume no more memory than their execution stack and don‘t require kernel mode switching, which makes them very fast.

With impunity, a programmer can create a new tasklet to fork off any processing he or she needs to. Tasklets in EVE use cooperative scheduling, meaning that we only switch to another tasklet at known switching points. This virtually eliminates the need for complex locking and synchronization, as is often the case with multithreaded programming.

A particular case where this programming model shines is with I/O. Efficient I/O makes use of a nonblocking interface to the OS. On Windows, this often takes the form of sockets and I/O completion ports: A thread starts an I/O operation and then polls an I/O completion port to see when it is done. But from the programmer‘s perspective this is extremely complicated and error prone. A programmer just wants to send() and recv() and not worry about event loops and such things.

To facilitate this, we present the Python programmer with a blocking I/O interface that is in fact only taskletblocking. When a tasklet executes a socket.recv( ) function we issue an asynchronous I/O request to WinSock. We then suspend the tasklet, allowing another tasklet to run. Later, when we notice that an I/O request has completed, we prepare the result for the blocked tasklet and make it runnable again.

This way we can program discrete pieces of game logic with their own straightforward execution paths that just behave as you would expect, while behind the scenes we suspend and reschedule their execution, making good use of the operating system‘s resources.

Designing and maintaining a real-time database. We chose a relational database to be at the center of the server architecture. The key focus with our database design is to keep things simple, as we strongly believe that simple is incredibly powerful, and easier to maintain. All important DB usage goes through stored procedures to make things as efficient as possible. There is not a single trigger or a cascading foreign key in the database, simply because we don't want complex magic happening behind the scenes. We want things to be visible from the source code, whether it is a child record delete or an update of related records.

We also have strict coding guidelines for DB code that we have generated over the years. We have inhouse experts reviewing DB code checkins and forcing developers to fix code not done correctly. Monitoring the database usage is extremely important, and we store all kinds of statistics for that purpose in the database as well. We track how often each stored procedure is called per day, how often an index is scanned, and so on.

Looking at such statistics usually tells us right away if a developer has committed code that has sent the database into a frenzy. We are running a real-time game on a huge scale, which means speed is top priority, and we can use tricks which would not be allowed in other situations -- similar to the operations of a bank (although recent events in Iceland might suggest otherwise).

One example of this would be when we use "read uncommitted" when loading a solar system configuration, knowing there won't be any inserts or updates to the data we are reading so we can allow ourselves to read with no locking. Another trick would be that we most often only need to allow the user to filter data within a day. For example, we do not need to allow a user to select all records from the player journal between 14:00 and 15:35; it is sufficient to allow filtering by only a date. In that case we simply need to keep track of the clustered key at 00:00 every day and use that in our queries. This means we don't need to index on date/time columns in every nook and cranny, making things faster and slimmer.

While having only one database creates performance challenges, it also makes some things easier. Having a distributed database system, where one database stores all characters and other databases store each shard, brings all kinds of complexities that we don't have to deal with. There is no need to replicate data between databases, nor to call multiple databases nor move characters and belongings between sharded databases and so on.


Article Start Previous Page 4 of 5 Next

Related Jobs

Disbelief
Disbelief — Cambridge, Massachusetts, United States
[06.11.21]

Programmer
Disbelief
Disbelief — Cambridge, Massachusetts, United States
[06.11.21]

Senior Programmer
Insomniac Games
Insomniac Games — Burbank, California, United States
[06.11.21]

Technical Artist - Pipeline
Insomniac Games
Insomniac Games — Burbank, California, United States
[06.11.21]

Technical Artist - Pipeline





Loading Comments

loader image