|
|
 |
 |  | | | | |  | |||
 | ||||||
          |  |  | ||||
|
GDC 2003: Neverwinter Nights Client/Server Postmortem: How I Learned To Stop Worrying And Love The Magic Missile Replay back to the end of 1998. Y2K was going to eat all of our machines unless every programmer who knew what the acronym COBOL stood for was working on converting all databases to four-digit year formats. Most gamers who could not program were too busy to care because we were playing Starcraft and Half-Life to an obsessive degree. BioWare had just finished a game called Baldur's Gate, and we came back from our Christmas vacations in January 1999, slightly frazzled but assigned to another project: Neverwinter Nights. Neverwinter Nights (NWN) was supposed to be the best multiplayer Dungeons and Dragons (D&D) role-playing game (RPG) ever made. Not only were we going to tell a story of our own creation, but we were going to enable others to tell their stories, on their own servers. Dungeon Masters (DMs) could log in to a server and help (or hinder) players to their heart's content. Designers could translate their D&D modules from pen and paper via a toolset that we provided for them. It sounded pretty cool to us -- then we realized that we had the task of making it! It really started to sink in when other developers where telling us that they couldn't wait to play it and that they were glad that they weren't developing it. Although Baldur's Gate was a good single-player RPG, we did a number of things very poorly for the purpose of accomplishing the vast and ambitious goals of Neverwinter Nights. The major issues that we are going to address (both the good and the bad) are the design for user-created content, design limitations imposed by the multiplayer focus of the game, the staffing requirements required to fulfill this vision, developing the game for multiple platforms simultaneously, and reputation systems. It is our hope that you will be able to learn something from what we did right and the problems that we ran into. User-Created Content
At the core of every game there are a few core requirements that dictate the form of the entire design. Sometimes these are dictated by technology, sometimes by user interface design and game play. In the case of Neverwinter Nights, it was the vision. The vision was to build a game where the user-created content was just as important as the content provided by the game creators. Somehow we needed to get a significant percentage of our users creating custom content. We were sure that the desire was there, it seemed that every fan we talked to had some old pen and paper Dungeons & Dragons modules scribbled onto graph paper collecting dust in their bookshelves. We just needed a system that allowed the fans to dust off those old modules and translate them into a format readable by Neverwinter Nights. Neverwinter Nights certainly wasn't the first game to try this. Stuart Smith's Adventure Construction Set released by Electronic Arts in 1987 was the first commercial attempt at this. This was a complete RPG construction set. You had the ability to edit maps, created custom graphics, edit creatures and items and it even came with a pre-made mini adventure. The biggest problem with this product was that once you built an adventure you only had your limited circle of friends to share with. If you were lucky, you had a modem and had access to a bulletin board system where you could download a handful of files.
Neverwinter Nights wasn't even the first attempt at RPG construction set for D&D. In 1993, SSI released Unlimited Adventures, using the classic engine and content from their Gold Box series of AD&D games. Like Adventure Construction Set, this was a classic case of an idea that was ahead of its time. It wasn't until the wide spread use of the Internet that a single user could have access to the entire pool of user-created adventures. We knew that almost the entire user-created content for other games was created by a fraction of a percent of the user base. This presented a real problem. What we really wanted was a role-playing game creation toolset that "even your grandma could use." When we were first considering the creation of Neverwinter Nights, there was a thriving mod (user modification) community for games such as Quake and later Halflife. While many of these mods were of professional quality, the numbers that were being created didn't justify the effort that creating a full RPG construction tool would entail. It seemed that many of the users that would have liked to built mods for Quake and Halflife just didn't have the technical skills required. On the other hand, games like Starcraft had a thriving custom map community. Some how we needed to take the ability to create 3D levels like in Quake, add to it all the tools needed to create a RPG adventure and still keep the system within the reach of users capable of creating maps in Starcraft. After some experimentation with arbitrary mesh systems and height field manipulation, we decided to provide a system that used 3D tiles. The major advantage of the system was the simplicity of area creation. Users could simply select a terrain brush and paint rivers, bridges, corridors and roads. If you are expecting a large number of end users to use the tool, then they have to be able to see concrete results the first time that they try it or most of them will give up at that point. Another advantage of using tiles was that the areas could be represented with only a minimum of data. This was also extremely important. One of the key game play features was hosting an adventure for other players to connect to. What we didn't want was a long delay while the newly connected client downloaded a massive file containing all the art for the area. With a tile system we could send a full area description with just a few kilobytes of data. The major limitation of any tile system is that there are never enough tiles. There is always just one more special tile or combination of tiles that the end user wants. This is also true for creature and item models too. Since the creation of 3D art is beyond the skills of the majority of the toolset users, we did not include any direct ability for the users to modify the core art resources. We did realize that there would still be a few brave souls who would want to create custom 3D art and other types of game modifications. Sometimes, the end users can be too clever for their own good (and ours too). When we released Throne of Bhaal, the expansion pack for Baldur's Gate II, there was a popular rule tweak modification in circulation. Unfortunately the creator of this modification released a version during the weeks between our gold candidate and the time it arrived on the store shelves. The problem was that our install package for the expansion pack did not overwrite files with newer time stamps. This caused the expansion pack install program to fail on every user that had that particular modification installed. When we were designing Neverwinter Nights, we planned for a method of allowing end users to make significant modifications with out breaking the core game or causing the game to crash when non-modified clients tried to join a modified server. The result was the 'hak pak' file (remarkably named because the file extension is .hak). This file would contain all of the non-canonical files (whether introduced by the end-user or modified versions of our files) and would be associated with only specific adventure modules. If a client tried to connect to a server running a module that required a hak pak, the client would be informed and gracefully disconnected. On the plus side, the hak pak system did accomplish all of these goals. The major problem with the hak pak system was in the way in which art resources are added into Neverwinter. To reduce network bandwidth, art assets were generally referenced by index numbers. For example, the appearance of a creature was dictated by its appearance id. This referred to a row in a table that contained all appearance-related data. This included the name of the 3D model, sound information, movement speed, personal space etc. To create a new custom creature model, a new entry had to be added this table. Storing a custom creature in a hak pak was not a problem since the hak pak also stored a modified version of the appearance table. Unfortunately, as we released more creature appearances or as users tried to merge hak paks together, conflicts in the table row numbers were created. In the end we had to release a hak pak merging utility to rectify this situation. If we could do it all over again, we would consider building a system where all information was encapsulated in a single info file and creatures store the name of the file instead of an index. The server could broadcast the correspondence of new creatures to indices as the initial area is being loaded so that each client would know how to link the hak pak resources to the server's indices. Another consequence of officially supporting user created content was the amount of testing required. In a normal game you only have to test the content of the actual released game content. If you are officially supporting user created content, you now have the problem of an unlimited testing set. For example, since we needed the pathfinding information to be integrated into the tiles, we had to deal with the fact that not only are all of combinations of tiles too numerous to test, you also have to worry about the users placing objects over key pathfinding locations. In the case of Neverwinter Nights, there was also additional content and features that were not used in the official campaign that also needed significant testing. Multiplayer Issues We were acutely aware of the issues surrounding the multiplayer version of Baldur's Gate after its initial release. Integration of key features of Baldur's Gate was still going into the engine two weeks before the ship date. There was little time to go back and fix things to make the same campaign work as well in single and multiplayer. The first major problem that we wished to address was the issue of bandwidth. Multiplayer was a primary focus of NWN and we wanted to ensure that we did everything in our power to limit the bandwidth consumed by each client. In Baldur's Gate, the bandwidth used up by each client during regular game play usually ran at over 2 kilobytes per second, and we would routinely send around 100 KB save game files to each client while others were joining in. For NWN, we expected to be able to handle six or seven clients through a single 56K server. A rough calculation led us to approximately 500 bytes / second. The approaches taken in Baldur's Gate had to be reworked. If we wanted to send a single game resource of 100 kilobytes across to the other side (as we did in Baldur's Gate), it would have taken 3 minutes to do so through a 56K server, and most people would have considered this unacceptable. This required us to have all of the data already on the machine during regular gameplay. However, this seemed to conflict directly with user-created content. Thus, we initially shelved the idea of automatically downloading the hak pak file during regular gameplay from a server, since it would be significantly slower than downloading off of another web site. After launch, we discovered that the lack of automatic downloading of hak pak files was a serious barrier to entry for people who wanted to play multiplayer, since almost every module has a different hak pak. This has not been fixed as we write this, but should be fixed by the time you read this!
Since the world was subdivided into a number of different areas, we needed a way of sending down the area's description in the minimum amount of bandwidth. We discussed the tile system earlier, but it isn't just as simple as saying "tile 432 at location x = 20, y = 30". The lighting and shadows in each area had to work in a bandwidth-friendly manner. To make an area look attractive, you sometimes needed hundreds of lights. Since the video cards are unable to compute dynamic lights in these quantities at a reasonable speed, some form of static lighting system was needed. The standard methods, such as shadow maps, were fundamentally too slow to generate on the fly, and they generated data too large to download on the fly. The solution that we eventually come up with was to embed the light information into the tiles themselves. We then only needed to specify which lights for each tile were active and what color they were. When the client received the area description from the server, it then calculated the vertex lighting values for each vertex within the range of that light. Shadows volumes for the sun or moon were also pre-calculated at this time. We did discover that the higher end machines were able to generate shadow volumes on the fly for a limited number of lights. This resulted in the distinctive look of shadows in Neverwinter Nights. As the last step of bandwidth reduction, we needed to do a better job on reducing the bandwidth used during regular game play, since a lot of redundant data was sent to each client in Baldur's Gate to ensure that each connected machine could run its own AI. However, the server runs all of the AI in NWN. This allowed us to use a networking model similar to the Tribes networking model [Frohnmayer00]. In a multiplayer game with only a few attributes required to display creatures properly, this shouldn't be too taxing on the server, and it wasn't when it was first added. However, we not only had a location, orientation and animation to play, but we also needed to display any one of a number of visual effects on a creature. These visual effects were allowed to stack on a creature to indicate all of the various effects the creature had on it; some creatures such as liches may have up to seven visual effects on them. For the player's creature, they had a number of long lists including prepared, memorized spell lists, as well as prepared and useable feats. The server regularly examined these lists to see if there were any changes to the spell list that needed to be broadcasted. Thus, we had numerous expensive linked list to linked list comparisons for the creature that the client controlled. We ran out of time to optimize this before the game shipped, and we uncovered the beautiful fact that in our zeal to reduce bandwidth usage, the CPU actually became the major bottleneck on the server, preventing some high-end machines from hosting more than 40 player characters.
In spite of the restrictions of downloading files during regular game play, we were still required to send some data (in the form of a character file) from the client to the server. For the pen and paper fans amongst us, it was the equivalent of handing a character sheet to the Dungeon Master and asking "Can I play this character?" Of course, there were all sorts of security issues associated with this. Most servers would not have active DMs that could visually inspect and approve characters, so we were required to write code to validate that the rules were followed in the creation and playing of a character when it was brought into the server. The process of writing the validation was scheduled to take a couple of weeks. However, we severely underestimated the amount of time that this would take, due to the complexities of the D&D rule set. In the end, we actually had to remove some of the level-up validation checks to meet our deadline. Unfortunately, the missing checks were quickly discovered and one of our revisions was made to repair this issue. What were some of these complexities? In role-playing games, characters also include all of the items that the creature holds, or has in his or her backpack. The majority of the size of each creature wasn't the creature itself, but the items that the creature held! However, the standard items were not that much of a problem. Each item could be examined to see whether a creature had the appropriate abilities to use the item; furthermore, each item could be validated to see if it was appropriate for the creature! One does not expect a first-level character to be carrying around a +5 vorpal sword. Thus, we had to develop a system that could be left on a server to prevent that from happening. This system not only had to work through the import of a character on to the server, but also had to work through the trade of items. We implemented an item level restriction (ILR) system, which forced you to only have equipped items of a certain cost. If the item was too expensive and the server chose to turn on the ILR system, you couldn't equip it, you couldn't even bring it in your backpack to give to someone else! However, some of the items required for the plot had properties, so plot items were initially given a zero cost. This made all plot items usable by level 1 characters. However, we didn't actually have a different check to equip and to possess the item. Our enterprising QA staff discovered that they could make a "plot" +5 vorpal sword, hand that to a first-level character, and it would be fine to both equip and possess! So, we had to change the system again to impose the item level restriction when the item is equipped, but still maintain its cost at 0. We also had secret inventory slots for creature hides, claws and bites. This allowed us to implement some of the more bizarre rules and "on hit" properties for Dungeons & Dragons monsters. The end users discovered that, even though they had no in-game method of putting items in these slots, that they could hack the character file, and put the +5 vorpal sword in the creature's "claw" slot, allowing them to swing their fist and have a very good chance of killing characters instantly. The exploit may have taken significantly more time to discover if we did not provide a toolset that showed end users that all creatures had these "hidden" slots! We could not guarantee that the end-user could not write a script within the scripting language that could take down a server, insert it on their character, and destroy the game experience for others. Thus, clients were forbidden from sending scripts that could be run by the server; hence, there were no scripts on anything in the character file. However, this imposed all sorts of odd restrictions on the game design. In The Sims, each object contains all of the code for how the sim creatures deal with the object. In NWN, we were required to store a global script on each module that dictated how every object that could be brought into the world could be activated. Since end-users could create their own objects, they might accidentally create an object that had exactly the same visual properties (appearance and colors) as a special object in the module. Hence, we had to provide a password system (tags) that allowed module makers to reduce the likelihood of a collision between their objects; only if the passwords matched would the object behave in the way defined by the module. This makes it very difficult for end-users to integrate their own custom content, since they had to cut and paste the appropriate part of their Item Activation script into another module, as well as moving over the item file into the next module. However, security of the server is paramount in a multiplayer game. Now, you're probably wondering why we went to all of this detail to tell you all of this. The parable is this: Leaving a character in an insecure environment (the client's machine), and giving them the tools to modify their own characters requires a gigantic engineering push to make it work properly. However, it can be made to work. Our original plan for the NWN story called for approximately 30 2-hour missions that people could play together, enjoy a wonderful experience, and exit from the game. However, market research led us to believe that a vast single-player story was important to end-users; they did not want to see Mechwarrior-style missions where the party would be briefed, charge off like raging lunatics, slay and loot the villains, and then return to the tavern for a nice cup of mead. Thus, the story was reorganized into four chapters, which meant six times as many creatures and objects to deal with in the same amount of RAM! The mission to chapter conversion required refactoring in some systems to guarantee that the amount of RAM being used was kept under tight control and a number of CPU intensive algorithms had to replace the RAM intensive algorithms that we had used earlier in development! It also increased our save games to a size unbeknownst to mankind. It also forced us to dramatically optimize our file format a couple of months from the ship date to get the save times down to the small levels they are at now. (At the point where the end-users started seeing the beta toolset with the old generic file format in it, some save games were taking 7 minutes to save. Our producer mentioned that he saved his game, made a tuna fish salad, made a sandwich using the salad, ate it, sat back down at the machine and it was still not done saving!) Of greater concern was that the server wasn't required to have a 2-hour stability between complete reset of campaign data; it had the potential to live and survive for 10 hours in the same campaign world, and anything that could support 10 hours without crashing should be able to handle 24 hours times 7 days without crashing! Unfortunately, we never did get the time to refactor some of the other systems to ensure that the server could stay up for very long periods of time. We are still working on those issues today. ______________________________________________________
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
Introduction
Features
