Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Gamasutra: The Art & Business of Making Gamesspacer
Sponsored Feature: Restless Entities Never Sleep -- The Back End of Warhammer Online: Age of Reckoning
arrowPress Releases
July 24, 2021
Games Press
View All     RSS

If you enjoy reading this site, you might also want to check out these UBM Tech sites:


Sponsored Feature: Restless Entities Never Sleep -- The Back End of Warhammer Online: Age of Reckoning

December 30, 2009 Article Start Previous Page 3 of 3

The Challenge of Maintenance

A seamless game experience requires servers that don't crash, servers that aren't offline for long periods for patches and updates, and servers that can be managed easily and even remotely, if need be. Mythic employs a number of techniques to deal with the thorny challenge of maintenance.

"Among the challenges," Shaw said, "we have to distribute on a regular or emergency fix basis new data and executables to our users who run the Warhammer client. We distribute components all over the world, including to our partners, who then stage it out to their users."

"Then," Shaw continued, "we have to take care of the servers; whenever we do server updates, they do have to be managed. There are several different ways to manage this many servers. You can bring them all down for a maintenance operation once a week; we only bring down our servers as needed for major updates. Often we can make dynamic changes while the servers are running."

"We built a system to automatically deploy both major and dynamic updates reliably to servers as needed. We had to create this system for Warhammer. In our earlier projects, like Dark Age of Camelot, there weren't that many servers. Now there are so many, there's no way you could apply an update to every server by hand."

"We could design the systems to be 24/7 and attempt to patch them," Mann said, "but it would introduce a lot more complexity. For instance, a design that allowed us to patch the system in pieces would require us to coordinate shifting load away from each piece before we updated that piece. A single mistake during that process, and the entire system could come crashing down, and it would require a lot of unexpected, time-consuming work to put things back together."

"Instead," Mann said, "our approach is to schedule downtime for updates in advance-which is a little trickier than it seems. Our worldwide deployment means we have to schedule downtime in phases. In North America, we'll bring it down during our low time, which is usually early morning mid-week. And, then at that same time, our Australian players are 12 or 14 hours ahead of us-depending on what time of year it is and where they are on the continent and such."

"If you think about that, if we bring it down about 6:00 or 7:00 a.m. in the U.S. then that's 7:00 or 8:00 p.m. in Australia-prime time for players enjoying an evening of gaming. So, we wait several hours after we update the U.S. servers to pass the Australian prime time, and then we take the Australian servers down and roll the update out to those."

As the sun sets on the Necropolis of Zhandri, a Bone Giant continues to patrol the ruined kingdom

"One of the great things that's come along in the last five or six years is the remote access capabilities-it's one of the big things we like about the blade architecture," Mann said. "It gives us the ability to connect into the blades and manage the full set of hardware capabilities from anywhere in the world. Being able to power down blades, being able to get a virtual console, mount CD-ROMS and DVD-ROMS from remote, any type of media there-all of this has definitely assisted us with executing our worldwide roll-outs smoothly."

"With the hardware that we rolled out in 1999 with Dark Age of Camelot," Mann reflected, "we did not have the capability to effectively manage systems that were on another continent without having someone local being there. And, to touch on that, our data centers in Australia and Germany don't have a 24-hour staff. We don't even have any Mythic staff on the same continent."

"If we have a situation that requires a physical hardware swap, we'll arrange for the hosting facility to do it for us (such as replacing a failed hard drive), but with the increasing ability to remotely and pre-emptively detect hardware failures, physical maintenance is no longer a part of the time-critical emergency response. We've developed a lot of techniques to take care of most emergency situations from remote locations-24 hours a day anywhere in the world."

Threading Opportunities

For server-side operations, networking challenges represent a large part of the performance considerations, and it is also in this area that Mythic finds abundant opportunities for multi-threaded operations. On the server, the implementations tend to be processes with extensive network communication between them. WAR uses TCP connections effectively as queues between the processes, providing a degree of asynchronous separation between the individual processes.

"As we're moving more heavily towards the use of threads," Mann said, "we're following the same basic model. Instead of making very small processes that use TCP communication to talk to each other, we turn the processes into threads that use queues to communicate. This eliminates the overhead of network communication while keeping our game systems discrete."

The Clarity of Darkness

No matter where you are in the world reading this, somewhere in another quarter of the planet the inhabitants of Warhammer Online are tromping through vegetation, casting spells, charging a hill with a group of team members, poking through underground dungeons, or rallying together to confront an opposing Realm.

When the players rest and night unfurls across the globe, the undead continue to prowl the expanses of the WAR fantasy world, monsters continue their quest for humanoid flesh, and the intricately realized world vibrates from the dance of billions of electrons, quivering in anticipation of the next humanoid to enter the fold. Step cautiously gamers. Danger lurks in every direction.

When the action gets heated, the servers keep their cool, thanks to the energy-efficient performance of an infrastructure based on Intel Xeon processors, providing the kind of rock-solid stability that effortlessly supports 45 square miles of virtual real estate teeming with artificial life and epic gameplay on a grand scale. WAR exists in a realm that is part imagination, part silicon-a world where fantastical beings roam freely, fantasy plotlines twist and take unexpected turns daily, and the entities never sleep.

The International Reach of Warhammer Online

The Warhammer Online franchise encompasses countries around the world with participants engaging in battles in five distinct regions. There are North American servers, Oceania servers, European servers, Taiwan servers, and Russian servers-each supporting different variations of gameplay, as well as a number of test servers. Localization is an important element when dealing with a variety of regions and cultures. Warhammer Online has been localized to support 12 languages.

On local differences, Andrew Mann said, "There is a distinct character from different regions of the world. It's a little hard to say, from a technical perspective, what regions have each character. And, of course, there are different people within each region. It tends to be that each region has kind of a majority persona. There are all these people that are around throughout the entire spectrum, but, for example, the Russian player base plays during different peak hours. They play longer during the day and there's a lot higher percentage of them online than we saw during our North American launch."

"Now," Mann continued, "whether that's because there are different levels of competition over there, whether it's because we did an especially good job of localizing the game, whether it's because they just like the kind of combat system in our game, or the kind of play style in our game versus other games, it's kind of premature to actually tell. It's difficult to tell that just from the raw data that we see on the technical side."

While it's difficult to cull data on gamer activities without becoming intrusive, Mythic uses anecdotal information and large-scale data correlation to scale server operations to the gameplay patterns for each region and to ensure that servers have sufficient headroom to operate effectively during peak periods of activity. There's both an art and a science to keeping actions in all parts of the world load balanced, and the technical staff at Mythic confronts this challenge with all the tools at their disposal.

Article Start Previous Page 3 of 3

Related Jobs

Insomniac Games
Insomniac Games — Burbank, California, United States

Technical Artist - Pipeline
Insomniac Games
Insomniac Games — Burbank, California, United States

Engine Programmer
Legends of Learning
Legends of Learning — Baltimore, Maryland, United States

Senior Gameplay Engineer - $160k - Remote OK
Bytro Labs GmbH
Bytro Labs GmbH — Hamburg, Germany

Senior Product Owner / Live-Ops Owner (f/m/x)

Loading Comments

loader image