Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
September 23, 2014
arrowPress Releases
September 23, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
MMO servers 2013
by Juan Belon Perez on 08/06/13 06:00:00 am   Featured Blogs

The following blog post, unless otherwise noted, was written by a member of Gamasutra’s community.
The thoughts and opinions expressed are those of the writer and not Gamasutra or its parent company.

 

Game Server Cluster

The evolution of MMO game servers 

Intro

Create a multiplayer game that can be turned to a MMO server has always been and will continue to be rather expensive, considering the costs of maintaining the servers and the infrastructures involved. [See Wikipedia first]

If we take a look to the present, there are some examples of games that close their servers after some years working: FIFA's, NBA's, Madden NFL, The Sims, Everybody's Gold World Tour, Monster Hunter Tri, ... including World of Warcraft (in Korea), etc...You may ask why, perhaps you are already conscious of the expenses of having a game server always online, 24/7,...and you will know that even depending on the number of users ,it is hard to see clearly whether the benefits of a game will be enough to keep them up and running. 

We are here to discuss this, because one of the rules of gamification is to include other players in the game, a logical consequence is to know how to maintain the servers that connect our players to the worlds. And this is one of the things I love more about gaming.

mmo_subs

Let me put this in another way:

To avoid things like happened with WoW: the lost of more than two million players in one year (2011-2012) which is hard to recover, and it resulted in some of the servers closed, we need to find a balance between tech and costs. If the money income can't pay the servers bills, we have to move the users to another server/s or shut down the game, that is the worst scenario, so this is what we will discuss in this article: the evolution of online game servers for a search of the true balance.

The present game servers

In order to give a historical perspective of a server technology and its costs, first we need to know what are the associated maintenance tasks. And this depends on the operators that keep it running everyday of its life.

LoL: login queue
Let's check the common beta launch games such as GoW: Ascension, FUSE, Dota2, LoL, Final Fantasy XIV, (same as WoW -several years ago-) and so on, where developers must test the server efficiency at high load, they can observe the behavior of the players and the maximum values of lag / load balancing / ping / server connections - number of users - synch data, etc., with these information values you can make the necessary adjustments to the server and find a more smoothly flow as well as correct possible bugs. 

Maintaining an online server today 

Some of the administrators tasks are:

  • Patches implementation and deployment: 
    Software fixes may put the whole system at risk and need from days to weeks to be completed.
  • Prevention and response to attacks: 
    Out there , malicious hackers are trying to bring down your system for various reasons, sometimes for money, competition or other reasons, therefore, the staff responsible for maintaining the server has to have high system security knowledge, or at least , part of the team should be like this.
  • Scalability: 
    It is normal that the more game purchases are made, it reflects in using the server more, so, study bandwidth and using different nodes will cause the server complexity is defined at the beginning and not at the stage of production.
  • Load Balancing:
    This could define the balancing algorithm the point above? 
    There are different approaches or strategies to comprehend load balancing,  depending on the type of game the server can be:

    • For an online FPS game like Quake or Half-Life, where any user can become the host when the creator of the match leave the session (P2P networking). In this case, the mission of the server node is to distribute the information to other client...You can give a more actualized version of this server with cloud computing, where the main node search for a "closer to the user" IP, always trying to adjusting the connection based on latency of each client, reflecting their interactions with the game on the server part, this is the case of CoD: MW3.
    • For an online persistent game like FFXIV or Guild Wars 2, the big part of the logic of the game is inside the server, the systems are more complex because we also need a set of servers in our architecture: login, backups, core gameplay, tournaments and player-versus-player arenas, economics, web portals, etc. those are embedded and glued with connections into a "realm" and must be auto balanced all the time. Each player has a limit of effect in the realm and must not break the server border, the efficiency implication of a simple spell could be very deep:
      A simple spell affects only a small part of the server

      the objects in the world only affect to a small part of the server side.
    • For an online turn based game the servers need to store data but they dont need to struggle for online efficiency, but they need to balance the write/read operations as well.
    • We can expect that there are so many kind of online servers as games. It is not easy to find the better tech neither the costs balance.

  • Backup and data management: 
    To handle a large volume of data is something you can not do in a single machine, that's why we are establishing a set of them interconnected, and to keep data synchronized in these farms/clusters of machines the software is more powerful than normal software: Twitter is the better example of this.
    Backups should be done daily, in fact there are legal implications of this, networks like Twitter are using modified Apache (Hadoop) and NoSQL databases which are able to write more than 12TB of data from tweets per day,this is 4 PicoBytes aprox. , a year.
    Now took a game server in a massive online game with millions of users, it can multiply those sizes by two or three, because keeping game data structures are more complex, considering that we are storing: 
    - Positions of the characters of the user, routes, etc.. 
    - History of missions, achievements, play areas, information relevant to the game 
    - Objects, equipment, character settings, relationships, messages with other users, its public properties in the world, and so on...
    Each user is generating a huge volume of data while playing, and this must be stored and backed up every day, every small amount of time, see the daily connection rates example: 75 deaths per second in LoL.
  • Learning from data and social graphs: 
    The actions of the players provide a wealth of information about people behind them, and the information has different types,
     - computational: ip addresses, device, bandwidth, location, etc.. 
     - And human information: this is, what kind of attitudes can we find in a particular user or group of them, behavioral patterns and other human connections data. 
    Let's observe the behavior of users of Minecraft, while you are reading this , they are building wonders together, organizing, talking, etc., Would not be something smart to take this initiative to create tools that facilitate the task?. This will increase the value of the game itself and the community. 
    Or in the case of EVE online, users can use their social skills to adquire more power, money, etc. They even made up political stuff by themselves.
    To make the learning data an automate process to improve the whole system and community, first, we analyze the social graphs that the architecture information engineer can write using new platform tools, such as - Pig -, for more info about this I wrote an article of how Twitter works, here (in spanish).
    An example of designing automatic mechanisms is to bring together groups of users to perform tasks such as join raids, merchandise and exhibitions, competitions, etc.. 
    When a server is often overwhelmed and it is not able to maintain a proper average ping for users, the engineer must create a query using the learning data, which turns into a learning mechanism applied to any server node,and when it executes it should move users while maintaining the current state of the game, to a better average ping server, this is something you can seen in games like Call of Duty: Modern Warfare 3, but I doubt it uses such tech.

The old days' servers 

The first ones

We can find Quake I servers online and running even today, they are not dependent of nothing more than a small machine that does host a repository of names and IP's of the nodes that host online matches, and even that main node is not necesary because you can provide the IP of the server and connect to the match.

Quake 1 Online 

It's easy to understand why these are not closed, the cost of this kind of servers is practically zero, it can be mounted by yourself with an Apache and MySQL. 

I myself have mounted several servers for games such as for Ludum Dare Jams or games produced and that are currently working, using Glassfish, Apache or Google App Engine, which I maintain and I would like to keep them alive forever, ...well, they are not consuming so much so it can be done. 

ID software knew how to do the things right, a first glimpse of the perfect balance.

What are the differences with the present servers?

So, the big difference between an expensive present server and another old is not only determined by the gameplay logic included or not in it, but in the connection too. Look at today's QuakeLive servers (Quake 3), you need to be a Pro user (for a fee of about $40 or prepare to get banned) to create a dedicated server on your machine, here we have a hybrid strategy, now we will discuss this strategy as a solution to the problem of the closure of servers in this evolution of online game servers.

Pros and cons of including the game logic in the server

Technology:

Including a game logic in a server is already complex and expensive,but users must not see the implications of this in the gameplay. I have recently been playing several online multiplayer games on PS3 and PC, for example, in Lord of the Rings: Guardians of Middle-Earth, the ping is quite good and usually it have people to play with, however when it appeared the new LOTR: War in the North, users got bored of the old and moved to this new one. Funny thing is that people on this game forums calling for their char items and statuses (the game save itself) when stored online, because if you played at a friend's house with your account but with lower level, your data is not saved online...Well, this is a protective mechanism? or a mess for users? some of you probably recognize this problem and this is one more reason of my favorite enjoyment of talks and discussions...

So technology is a pro or contra depending on how you design the deployment strategy and the actual logic on the server, perhaps the best is to use a hybrid strategy, i.e., a minimal part of the game logic to check that the data sent by the users are correct, this implies having machines that are hosts,but at a relatively low cost and with low maintenance... which drives us to next pro-or-cons.

Hacking

Sometimes it is difficult to check such things as the use of special devices, such as CoD: MW3 hacked pads which can automatically target - duck - shoot once it "sees" an enemy, which is solved by placing a reporting service that other players can use when they detect the cheaters.


This is a contra, because, have a cutting edge technology does not relieve us of possible attacks, just take a look to the daily EVE online hack attemps ...
The use of hacks/cheats/bugs to be above the other players are breaking the balance of the game ,so there is a tendency to "spy" on the players, storing the messages, VoIP conversations about the game, etc.., to analyze the behavior of its participants and to ban, punish or warn those who do not follow the rules, for example by insulting, threatening, etc. 

Or maybe it is something that is on the agenda. The cost is high if we consider that we must review thousands of reported conversations or conversations that have detected the improper use of language, or a derogatory / threatening (cyber-bullying) against other players, cheaters, and so on. 

Another contra of hacking is that although the game logic is included in the server part, the messages can be analyzed and replicated, this is the case of rogue/private servers of World of Warcraft, Guild of Wars, etc.., which are still in operation right now a punch for the designers..., think of the investment , not only in technology but in qualified personnel to support the service (forums, game managers, beta testers, etc.) costs...

User Experience

Not everything could be bad, the experience provided by an online server with game logic can be better for management and gameplay improvement. 


Being able to change the game from the server side and turn it into something else, just adds to the social gaming a whole new experience, always from the point of view of the player. And since you can keep it alive longer thanks to the players, these will be attracting new players too, this allow to pay the server bills.

The future servers

Your own server

There are new server offers for enterprises and independent developers, now, is it time to find free servers for your game?, well, let me say that this is a magical moment,...Are you realising that such a thing like a connection with someone is something that increase the value of your game? Have you considered including game logic on your server? 

The future for companies and indies

It is difficult to predict the future, but with the data that we have now -from the evolution of game servers- and, seeing the growth in sales of mobile devices, this is supporting the resurgence of hiring servers that deliver content to these devices to consume (cannibalize?).

This new situation has been allowed to flourish a bestial number of new online games for mobile platforms, I believe that this will be an expansive boom due to the price reduction for multiplayer servers and technology.

Still, we must review the cost of the server based on regular income from the game to do  a good use it. Using a hybrid deployment with server online game logic part of the same and is easily replaceable / scalable is not so trivial.

Let's talk about prices ,we have services from 100 concurrent users on a cloud server (maintained by others) which is approximately $90 per year, a very little amount considering the laborious task that entails (it includes APIs to endorse so easily the game logic).

Although the game's success depends above all, in the original online gameplay more that in the server ,for non-indie companies with greater purchasing power, prices are different, $2k a year for a cloud game server 1k concurrent users, made the calculations for million (or just multiplicate for the number of realms).

I think that the choice made by large companies with no trust in third parties, will be that some of the software resides in their own machines as it is cheaper,and they would benefit from these cloud servers to accommodate updates and other online services that can safely rely on third parties enterprises, for example, the downloads of new versions to avoid having to wait for the review of a game by the markets revisions and that is easier to maintain, or connections for little games/servers...

In fact, big companies like Sony, Microsoft, Google, Amazon or Apple, have set up their own online games generic servers, which store information as achievements, scores tables, etc.., they also have several types of game logics:
 - online with turns: works especially well in mobile gaming
 - online real-time play: like Respawnables, it is complicated with 3G ,because you can have 2 seconds pings and constant connection losses,

...this is why that users tend to prefer games with turns and notices, to play whenever they wants, and also without depending on a live connection... There are games that have proliferated like Rizzle, Letterpress, Apalabrados (Angry Words), etc.,  and as it happens in other games, the hacks are inevitable to occur as the number of users grow, -human nature has not changed and we still want to be "better than others"-.

The present and future of video game companies is linked to the online servers: current game download services, scores, connection to social networks, and so on.

Something that players are always asking for is to buy once and play with any platform online, but, since each network (PSN, XBOX Live, Steam,  etc.) looks like not interested in sharing game connections, it would be nice to have this feature..., but it has another implications like gameplay and other thins that we are not going to discuss right now...well,maybe in the future you could play from your PS4 with players from PC.  

So, are we ready for the cloud?

In principle it seems that cloud computing is what will ultimately be implemented at large scale, because the costs are lower if we do not have the infrastructure to provide these services.

Some examples of massive cloud computing not only for games are the use of processing data sent from mobile, like: patterns recognition, voice recognition, processing of big volumes of social data, etc..

But we have seen that game users have been able to replicate the behavior of a server that needs a supercomputer on a basic web hosting, the future is uncertain, but knowing this, we also can say that this is the future and that things will not change unless there is a radical change today, right now!.

So we are back in the present , where we live with lag servers because of more income and that is not a trivial issue.

Lag warriors
Lag have turned some players in "lag warriors" where they use the lag of the opponent as a weakness.


Related Jobs

InnoGames GmbH
InnoGames GmbH — Hamburg, Germany
[09.22.14]

Senior Producer (m/f)
Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States
[09.20.14]

Producer - Infinity Ward
Infinity Ward / Activision
Infinity Ward / Activision — Woodland Hills, California, United States
[09.20.14]

Senior Tools Engineer - Infinity Ward
Blizzard Entertainment
Blizzard Entertainment — Irvine, California, United States
[09.19.14]

Senior Vice President, Cross Media






Comments


Eric Robertson
profile image
I built my own game server (C++) for my first MMO and my current MMO project, and found many of these issues the OP mentioned. For my own personal experience key things I had to implement were:

1. One database read or write per pulse per thread. Once I queued up players, it solved a lot of latency issues. DB optimization is a must. Consult an experienced DBA.
2. Optimize all game object lists. If you need to look for a nearby mobile, only search locally.
3. Players love to befriend others to build trust and rapport, then ultimately steal all of their assets. Piracy is faster itemization then questing, unless you change that as a developer. Human nature like OP mentioned.
4. DOS attacks. I had to modify my code to track every IP connection/frequency and deny access as early as possible on repeat connects.
5. Movement hacks. I found the most effective/tolerable punishment is just teleport/yank them back if they are moving faster then appropriate. My initial 'stunning' offenders was not well received.

Game server coding is fun but requires constant cat and mouse gameplay for devs and players.

Andrew Sum
profile image
Very interesting article, particularly on the backup and data management aspects. Twitter sure has a lot of data!


none
 
Comment: