|
|
 |
 |  | | | | |  | |||
 | ||||||
          |  |  | ||||
|
Enabling Player-Created Online Worlds with Grid Computing and Streaming "Within minutes, players have an uncanny ability to discover the size of the possibility space." - Will Wright, speaking at PCForum, 2003 The hours and intensity with which players engage in games makes for very fast consumption of content. Players rapidly feel out the design space of games, requiring creators to adopt strategies to provide tremendous amounts of content for the user to experience (Final Fantasy); make the content reusable enough to provide extended replay value (GTA3, Gran Turismo); or make the experience dependent upon direct interaction with other human players (Quake, Ultima Online). Persistent-world online games are similarly differentiated by their ability to put tremendous amounts of frequently updated interactive content in front of users. Games which offer 100 hours of content exploration can easily be exhausted by the player in the first month or even weeks of play - making it a real challenge to extract a monthly subscription fee to stay in the environment.
Player customization and modification of game features is a strategy that already extends the lifetime of many FPS games, but extending this approach to the MMOG space raises challenging issues. Whereas for single-player games, custom content can be as simple as documenting the various file formats for the players (or in some cases letting them figure it out for themselves!), in MMOG environments a number of weighty technical problems are added. The content (graphics, sound, geometry, animation, and behavior) associated with player-created content must be delivered to other players, potentially in real time. The various permissions associated with ownership and exchange of content must be handled. If player-created objects can exhibit behaviors and via scripting, the resources consumed by those objects must be appropriately limited and balanced. For our debut project, Second Life (http://secondlife.com/), Linden Lab implemented a distributed grid for computing and streaming the game world which supports a large, scaleable world with an unlimited amount of user-created and real-time editable content. In addition, we feel it exhibits a level of performance and content complexity that rivals the best existing 'static' environments. In this article we'll describe how our grid works, and how it benefited the design of the game. The Trouble with Shards Shards (an expression originally used by Ultima Online in its server deployment) split an online game into a number of parallel worlds through clustered servers which have identical sets of world content but different sets of users. The use of shards in a shard cluster has benefits, but shards aren't suitable for deeply user-driven or highly scaleable worlds. The advantage of shard clustering allows creators to multiply their content development efforts by reusing content across multiple sets of users. But the drawback is that it separates users into separate, non-interacting worlds. Friends are unable to collaborate unless both players are logged into the same server. Additionally, shards are not dynamic load-balancers -- everyone may choose to play on a certain shard, leaving the others empty and losing money. Since some user-developed content (bank accounts, inventory contents, and dropped objects) exists on even the simplest MMOG games, there is no straightforward way to balance or unify the load by moving players between shards. Less populated shards become useless in games which depend heavily on social interactions and are removed, resulting in displaced and unhappy players. The shard model relies on a fundamental separation between static and dynamic content at the level of the environment (mostly static) and the user (mostly dynamic). The static content can be replicated and the dynamic stuff moves around. So what if you want the whole world to be dynamic? Then either you have to have too many shards, or the whole world needs to somehow become one big one. The Scale of the Problem An online world that is comprised of one big contiguous space populated by players who are heavily engaged in creating, editing, and moving objects is potentially a major database problem. In just two months since launch, the number of in-world objects in our game has reached into the hundreds of thousands -- and that isn't counting the things that are in players' inventories. Each of these objects, potentially, has unique textures, physical properties, shapes, permissions, prices, and behaviors. Shipping the entirety of Second Life's current contents to users in a box would require at least 100 CD-ROMs, and attempting to "patch" users with new in-world content every day when they logged in would require downloads of tens of megabytes per day (and this would grow linearly with new users).
If all of this was handled by a single database, the transaction rate during peak hours would be tens of thousands of database reads/writes per second. This is the challenge faced by many upcoming games as well - the amount of content that needs to be manipulated by successful online games in the near future is vastly larger than what we see today. Enter the Distributed Grid Like many similar problems in scaling, the massive object explosion mentioned in the last section can only be handled by some sort of solution that distributes these objects across a large system in a way that doesn't create any critical bottlenecks that will fail as the user base and content set scales. Whether they use a shard cluster or a grid, all MMOGs use multiple machines to distribute the game processing load. To solve the problem of player-created content in Second Life, we built a topologically tiled grid of "nearest neighbor" connected machines, each of which simulates the physics, manages all the objects, runs scripts (behaviors) and manages the terrain for everything within a fixed square region of space (about 16 acres per machine). These "simulator" machines talk only to the four nearest neighbors, so there isn't a transactional scaling problem as the world becomes really large. Typical bandwidth between adjacent machines is in the hundreds of Kbps, so the servers can be at different co-location centers, and don't need to be connected via gigabit ethernet or in the same physical cluster. As
objects move around the world, their representation is transferred (along
with scripts, objects inside them, textures, and so on) from simulator
to simulator as they cross over the edges. Through the use of high-order
prediction of the client, these transitions are not visible to the users,
resulting in the world feeling seamless. The game's design required the
ability for players to move through the world smoothly without loading
delays, and while that problem required significant engineering time and
effort, solving it correctly proved to be a critical success factor in
building a physically simulated world across multiple machines. As you move around inside the game world, you maintain a streaming connection only to the simulator machines you are near. The simulators compute the objects and information that you can currently see, and transmit information appropriate to only those objects that are either new to you or that have changed. This approach offers the advantage of being implemented with inexpensive Linux-based server hardware, and allows a very thin client to be the only thing a player needs to download and install. As the world grows, the number of back-end server machines grows with it. Currently, each simulator machine is a commodity 1U Linux box that handles a square 256 meters on a side (16 acres), plus all the airspace above it. Linden Lab creates terrain for a rack of machines at a time (specifically, a short rack of 33U for power density reasons) and deploys new parts of the world in relatively large chunks. This allows our users to engage in land rushes with hundreds of users flying out into the unexplored parts of the world looking for the perfect new home location.
|
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
Introduction
Resource
Guide 
