Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

Gamasutra: The Art & Business of Making Gamesspacer
View All     RSS
June 28, 2022
arrowPress Releases
If you enjoy reading this site, you might also want to check out these UBM Tech sites:

That time  World of Warcraft's  servers buckled under thousands of players in a single zone

That time World of Warcraft's servers buckled under thousands of players in a single zone

July 21, 2021 | By Bryant Francis

In World of Warcraft’s 16-year history, the developers at Blizzard Entertainment have experienced all kinds of highs and lows that come with supporting one of the game industry’s longest-running massively multiplayer online games.

At GDC 2021, Blizzard’s Joseph Cochran and Kurtis McCathern—both long veterans of the network engineering team—gathered four of these stories to chart WoW’s course in this 16-year battle to survive on the internet, offering a rare look behind the scenes at Blizzard during some of its most technically tumultuous moments.

One story in particular stood out because well—I lived it. In a rare overlap of a technical postmortem and historical documentation, McCathern told the tale of how Blizzard discovered what happens when you invite 1,000 players to stand in the same spot just to watch someone ring a gong.  

Opening the Gates of Ahn’Qiraj

In 2006, before Blizzard launched World of Warcraft’s first expansion The Burning Crusade, the MMORPG advanced with a series of new 20 and 40-player raids. Ahn’Qiraj, a city buried beneath the sand in a region called Silithus, hosted one of each.

Its arrival was heralded with a massive server-wide event that culminated in a handful of players on each server earning the privilege to ring a massive gong and open the city’s gates to the entire world.

The event began by tasking players across every server with a mission to support their respective factions by completing quests, capturing towers, and gathering supplies. Once enough supplies were gathered, a countdown started, and players who had more deeply participated in the questline could ring that bell.

“This event was problematic for us,” McCathern explained. The core question for network engineers was “How many messages do I need to send to update connected clients about changes to unit state?”

This created a math problem, where if you have n players (where n = number of clients), you’ll have n^2 number of messages. In a four-person multiplayer game, that’s 16 messages per frame. In Overwatch’s 6v6 games, that’s 144 messages per frame.

If 1,000 players gathered in Silithus during this event, that’s 1,000,000 messages per frame. “I’m not aware of any service that can deliver a million messages per frame that would deliver a smooth experience for a thousand players,” he deadpanned.

In an age before streaming, the only way to see the event take place was to log in and visit Silithus, which made every viewer an active participant as well.

With different servers in different time zones with different completion rates, players would sometimes just create new characters on different servers and haul a level 1 dummy account halfway across the continent just to watch the gates open, adding to the total number of players in the zone.

If you’re already thinking “wait, why didn’t they just…” McCathern is ready to head you off at the pass; whatever technology could solve this problem today didn’t exist in 2006. Severs were overloaded, laggy, and crashed frequently.

There was only one viable solution: teleporting players out of the zone. Blizzard had to conscript the company’s customer service team (known to players as “GameMasters”) who had the power to teleport players to different locations—a last-ditch tool usually meant to catch players exploiting glitches or who had gotten caught in geometry.

And so, as server after sever lined up to ring the gong, a bunch of customer service wizards with godlike powers were forced to hang out on the fringe, teleporting any looky-loos away who hadn’t fairly reached the power level to exist in this high-level zone.

Players wound up doing their own emergent mitigation as well, manipulating the supply collection process to make sure the gong would be rung during off-hours and reduce busybodies.

(For my part, I think the Cenarius server rang its gong while I was still in school, so I didn’t get to see the crowd assembled for myself).

With no silver bullet for managing the “massive” in “massively multiplayer,” McCathern said Blizzard learned to design future content with levers designers could pull to manipulate when events were completed. The team also tried to design events to be more diffuse, so a massive event could be experienced across the game world, and not just one region.

Redemption in the desert

In a COVID-19-driven quirk of chance (the talk was originally going to be scheduled in 2020), McCarthen had a chance to update his talk to include details of how the team handled server load when the Gates of Ahn’Qiraj opened in World of Warcraft Classic.

Now, Blizzard can use automation tools to pre-test servers with several thousand headless clients. Modern profiler tools allow the team to identify hotspots. “Even these were not enough to get us to where we needed to be, so we decided to start ignoring some movement updates,” he said.

That means simple player movements—turning slightly just to catch a better view, or just fidgeting while waiting for something to happen—wouldn't be logged in the server. Even essential movement was halted if servers were underwater, just to make sure they stayed up.

This time around, the servers held, the War of the Shifting Sands broke out, and players spread out on their own, taking pressure off the servers and restoring harmony to the realm.

This was one of four tales McCathern and Cochran had from World of Warcraft’s history—you can catch the rest on GDC Vault, or on the archived video that will be live on the GDC Swapcard platform for the next week.

Related Jobs

Treyarch — Playa Vista, California, United States

Tools Engineer - Treyarch
Treyarch — Playa Vista, California, United States

Audio Engineer - Treyarch
Build a Rocket Boy Games
Build a Rocket Boy Games — Edinburgh, Scotland, United Kingdom

Lead Physics Programmer
Build a Rocket Boy Games
Build a Rocket Boy Games — Edinburgh, Scotland, United Kingdom

Lead UI Programmer

Loading Comments

loader image