Pay No Attention to the Man Behind the Curtain
In the midst of the worldwide gaming enterprise, Andrew Mann bears the responsibility of overseeing the server infrastructure that keeps the world alive. He describes his role modestly: "I guess you could say the areas that I oversee are fairly broad. But, the way I like to look at it is from the bottom up. The software engineers work from the code side, and they essentially try to look at things at a high level and work downward."
"I work up from the hardware level. So I work with our system administration team, I work with our IT team, our networking team here, and then I work up into the code level at that point, as well. Along with all of the operating system details, all of our database software, and everything like that. That's basically my purview."
After pausing thoughtfully for a beat, Mann continued, "My day-to-day work is actually extremely varied. Some days it'll be just a lot of discussion with team members. I usually touch all of those divisions on a day-to-day basis. Some days, however, are heavier in the system administration side. Sometimes we've got to do a lot of work on hardware. Sometimes we've got to swap equipment around.
Sometimes we have a lot more work on code bugs that are coming up. Maybe we've got a big crash issue or we've got a big feature push. Sometimes it's networking-we've got new equipment coming in and we've got to expand the network- those kinds of things."
At any given time, approximately 2,000 servers are in operation, supporting the gameplay in WAR. Matt Shaw commented, "What we call a server to the user-that main server is actually a cluster of a number of machines."
"Our Server Farm in Virginia, for example," Mann said, "has about 60 Dell Blade chassis running Warhammer Online-each hosting up to 16 servers. All in all, we have about 700 servers in operation at this location."
"The servers that are in the UK are not managed by us," Mann continued. "They're IBMBlades managed by our partner, GOA, and they're organized a little bit differently."
Servers on the Back End of Warhammer Online
Running the vast server farms underlying the Warhammer Online world cost-effectively requires using each server's resources efficiently. The Mythic technical team relies on the blade server architectures and application design to maximize server resource usage.
"We use blade architecture heavily for Warhammer Online," Mann noted. "Almost every server that we deploy is a blade system. We don't use virtualization; our software is somewhat virtualized itself. We've always had the technology to run our game world across several pieces of hardware. It's application-layer clustering at a process level. Virtualization wouldn't gain us much because we already run very close to peak CPU usage on these systems."
"We're watching developments in the virtualization area with interest," Shaw said, "as we plan for achieving maximum server efficiency with minimal power use in the future." Currently, power budgeting is handled automatically by the Dell powering options within the blade server cabinets. Mythic doesn't attempt to control any of the power usage at the application level.
The normalized server configuration-in use across all of the Mythic-managed facilities-features dual Quad-Core Intel Xeon processors running at 3 GHz with 8 GB of RAM.
"The Intel CPUs we're using," Mann said, "are all the same model CPUs that we normalized on about a year and a half, two years ago when we first looked at our production hardware specs. We're in pretty tight control of what our hardware specs are when we roll out the game. As you can imagine, with anything this complex, if we roll out to a partner that's managing their own hardware and it ends up being slightly less powerful, or it ends up not having quite as much RAM, or it has some difference on paper that doesn't look like too big of a deal, it can cause pretty catastrophic results to the entire game system. And when you've got 700 systems deployed it can be a big deal to go back and update them all."
Energy-efficient performance is a key consideration when running hundreds of servers, an area where these second-generation quad-core technology machines excel, providing an effective solution for coping with cooling and density challenges. The Intel Xeon processor-based servers also deliver an exceptional degree of stability and proven reliability, important when you have thousands of dedicated gamers who expect the uninterrupted action to continue unabated around the clock.
A Slayer faces off against a Liche Priest and two giant scarabs
A giant bone construct crackles with energy as a Liche Priest channels life back into its body
As might be expected in a game scenario in which the levels of participation vary dramatically at different times of the day, in different regions, and with different types of activity, performance scaling is an essential component of successful server operation.
"One of our ongoing challenges," Mann commented, "is where to distribute people in the world. Our processes-that we distribute across the physical hardware-correspond to locations in the virtual world. One of the focuses of our game, the big focus, is to get a lot of people in one place and have them all fighting with each other. And that, unfortunately, works against us in the process distribution model."
"When you put a lot of people in one place, you're putting their entire server load onto one piece of hardware. We do have some technology to mitigate that. Our scenario system (which spawns up smaller arenas for smaller teams dynamically) allows us to split people off to different pieces of hardware if we need to, dynamically, in smaller chunks."
Using this approach, the application, instead of coping with 800 people in an area on one system, can take 400 of those people out of an area and engage them in smaller fights. Most of the parallelism for these kinds of operations, Shaw noted, is done by process, not by thread.
Taking It to the Extreme
On the client side, the processor-intensive activities likely to be generated by excursions into the Warhammer Online environment can be smoothed out substantially with the help of the platform capabilities of the highest performing desktop processor on the planet.1 Currently creating a buzz across the gaming world, the Intel Core i7 processor Extreme Edition features intelligent multi-core technology that accelerates performance in response to increasing workloads.
New features enhance the overall gaming experience, such as Intel Turbo Boost Technology (to maximize speed for demanding applications), Intel Hyper-Threading Technology (for advanced multi-tasking and support for up to eight threads), and Intel Smart Cache (to provide a higher performing, more efficient cache subsystem). Experience Warhammer Online in its best light with the processor that has become the gold standard in the gaming world, the Intel Core i7 processor Extreme Edition.
1 Performance based on select industry benchmarks, game titles, and multimedia creation applications.