Gamasutra: The Art & Business of Making Gamesspacer
Sponsored Feature: Restless Entities Never Sleep -- The Back End of Warhammer Online: Age of Reckoning
View All     RSS
October 31, 2014
arrowPress Releases
October 31, 2014
PR Newswire
View All

If you enjoy reading this site, you might also want to check out these UBM Tech sites:

Sponsored Feature: Restless Entities Never Sleep -- The Back End of Warhammer Online: Age of Reckoning

December 30, 2009 Article Start Previous Page 2 of 3 Next

Pay No Attention to the Man Behind the Curtain

In the midst of the worldwide gaming enterprise, Andrew Mann bears the responsibility of overseeing the server infrastructure that keeps the world alive. He describes his role modestly: "I guess you could say the areas that I oversee are fairly broad. But, the way I like to look at it is from the bottom up. The software engineers work from the code side, and they essentially try to look at things at a high level and work downward."

"I work up from the hardware level. So I work with our system administration team, I work with our IT team, our networking team here, and then I work up into the code level at that point, as well. Along with all of the operating system details, all of our database software, and everything like that. That's basically my purview."

After pausing thoughtfully for a beat, Mann continued, "My day-to-day work is actually extremely varied. Some days it'll be just a lot of discussion with team members. I usually touch all of those divisions on a day-to-day basis. Some days, however, are heavier in the system administration side. Sometimes we've got to do a lot of work on hardware. Sometimes we've got to swap equipment around.

Sometimes we have a lot more work on code bugs that are coming up. Maybe we've got a big crash issue or we've got a big feature push. Sometimes it's networking-we've got new equipment coming in and we've got to expand the network- those kinds of things."

At any given time, approximately 2,000 servers are in operation, supporting the gameplay in WAR. Matt Shaw commented, "What we call a server to the user-that main server is actually a cluster of a number of machines."

"Our Server Farm in Virginia, for example," Mann said, "has about 60 Dell Blade chassis running Warhammer Online-each hosting up to 16 servers. All in all, we have about 700 servers in operation at this location."

"The servers that are in the UK are not managed by us," Mann continued. "They're IBMBlades managed by our partner, GOA, and they're organized a little bit differently."

Servers on the Back End of Warhammer Online

Running the vast server farms underlying the Warhammer Online world cost-effectively requires using each server's resources efficiently. The Mythic technical team relies on the blade server architectures and application design to maximize server resource usage.

"We use blade architecture heavily for Warhammer Online," Mann noted. "Almost every server that we deploy is a blade system. We don't use virtualization; our software is somewhat virtualized itself. We've always had the technology to run our game world across several pieces of hardware. It's application-layer clustering at a process level. Virtualization wouldn't gain us much because we already run very close to peak CPU usage on these systems."

"We're watching developments in the virtualization area with interest," Shaw said, "as we plan for achieving maximum server efficiency with minimal power use in the future." Currently, power budgeting is handled automatically by the Dell powering options within the blade server cabinets. Mythic doesn't attempt to control any of the power usage at the application level.

The normalized server configuration-in use across all of the Mythic-managed facilities-features dual Quad-Core Intel Xeon processors running at 3 GHz with 8 GB of RAM.

"The Intel CPUs we're using," Mann said, "are all the same model CPUs that we normalized on about a year and a half, two years ago when we first looked at our production hardware specs. We're in pretty tight control of what our hardware specs are when we roll out the game. As you can imagine, with anything this complex, if we roll out to a partner that's managing their own hardware and it ends up being slightly less powerful, or it ends up not having quite as much RAM, or it has some difference on paper that doesn't look like too big of a deal, it can cause pretty catastrophic results to the entire game system. And when you've got 700 systems deployed it can be a big deal to go back and update them all."

Energy-efficient performance is a key consideration when running hundreds of servers, an area where these second-generation quad-core technology machines excel, providing an effective solution for coping with cooling and density challenges. The Intel Xeon processor-based servers also deliver an exceptional degree of stability and proven reliability, important when you have thousands of dedicated gamers who expect the uninterrupted action to continue unabated around the clock.

A Slayer faces off against a Liche Priest and two giant scarabs

A giant bone construct crackles with energy as a Liche Priest channels life back into its body

Performance Scaling

As might be expected in a game scenario in which the levels of participation vary dramatically at different times of the day, in different regions, and with different types of activity, performance scaling is an essential component of successful server operation.

"One of our ongoing challenges," Mann commented, "is where to distribute people in the world. Our processes-that we distribute across the physical hardware-correspond to locations in the virtual world. One of the focuses of our game, the big focus, is to get a lot of people in one place and have them all fighting with each other. And that, unfortunately, works against us in the process distribution model."

"When you put a lot of people in one place, you're putting their entire server load onto one piece of hardware. We do have some technology to mitigate that. Our scenario system (which spawns up smaller arenas for smaller teams dynamically) allows us to split people off to different pieces of hardware if we need to, dynamically, in smaller chunks."

Using this approach, the application, instead of coping with 800 people in an area on one system, can take 400 of those people out of an area and engage them in smaller fights. Most of the parallelism for these kinds of operations, Shaw noted, is done by process, not by thread.

Taking It to the Extreme

On the client side, the processor-intensive activities likely to be generated by excursions into the Warhammer Online environment can be smoothed out substantially with the help of the platform capabilities of the highest performing desktop processor on the planet.1 Currently creating a buzz across the gaming world, the Intel Core i7 processor Extreme Edition features intelligent multi-core technology that accelerates performance in response to increasing workloads.

New features enhance the overall gaming experience, such as Intel Turbo Boost Technology (to maximize speed for demanding applications), Intel Hyper-Threading Technology (for advanced multi-tasking and support for up to eight threads), and Intel Smart Cache (to provide a higher performing, more efficient cache subsystem). Experience Warhammer Online in its best light with the processor that has become the gold standard in the gaming world, the Intel Core i7 processor Extreme Edition.

1 Performance based on select industry benchmarks, game titles, and multimedia creation applications.

Article Start Previous Page 2 of 3 Next

Related Jobs

Twisted Pixel Games
Twisted Pixel Games — Austin, Texas, United States

Senior Graphics and Systems Engineer
Twisted Pixel Games
Twisted Pixel Games — Austin, Texas, United States

Mid-level Tools and Systems Engineer
Sega Networks Inc.
Sega Networks Inc. — Madison, Wisconsin, United States

Mobile Game Engineer
Forio — San Francisco, California, United States

Web Application Developer Team Lead


Allen Danklefsen
profile image
Wow, a lot of this interview sounded like a big commercial for hardware.

Not sure how it is now, but war couldnt handle more than 300 in a zone ~2 months after launch; however, daoc could handle 500 all in a keep.

Most of the old daoc players I played with quit all about this time as there werent any epic battles. Just 80% deserted zones, with mini-zergs running around taking objects, if they happened to meet up it would just end up being a scatter match back to an objective. You couldnt CC them for long term fights like in daoc.

Douglas Rae
profile image
^^ Agreed, that really was tailored towards a massive advertisement around the power of intel. Hence why it was an intel sponsored article... oh yeah and all the intel adverts dotted around this site... and the massive copy & paste from intel's Xeon feature sheet... im going to go and cut intel into my forehead and have an intel cup of coffee to wash down my xeon muffin.

Tim Hesse
profile image
Allen, when I played it was the opposite, the guild I was in would flips zones like clockwork to enable fortress attacks, this was attempted or completed daily, to the point of other guilds on the same side getting huffy that they would have to coordinate with our guild leader; that guy was amazing.

Too bad the game died, I quite like it and the potential it had.

Andrew Grapsas
profile image
Wow... I expected a technical article... which this is not.

[User Banned]
profile image
This user violated Gamasutra’s Comment Guidelines and has been banned.

Mark Fowlis
profile image
"noiseless except for the soft whir of cooling fans".

Server fans only typically run quietly when the servers are under low or minimal load.

In our datacentres (running MMO games - on Intel mostly) you almost need ear defenders due to the noise of hundreds of servers' cooling fans, plus Datacentre Aircon, UPS, SAN, Network equipment etc. etc. It's certainly not noiseless !

I'm presuming some creative licence for effect. :-)

Richard Putney
profile image
I remember playing WAR early on and noticing that scenario's rankings polled the server several times per second, causing tremendous lag any time you tried to check the game's standings. If the query was being cached at all then they were not doing it right.

So, I can see why Intel chose WAR to promote blade servers. with net code this bad you're going to need all the hardware you can get.

And I agree with Julien. I would love to have seen a technical discussion on handling large scale battles in MMOs.