Multiplayer Issues

We were acutely aware of the issues surrounding the multiplayer version of Baldur's Gate after its initial release. Integration of key features of Baldur's Gate was still going into the engine two weeks before the ship date. There was little time to go back and fix things to make the same campaign work as well in single and multiplayer.

The first major problem that we wished to address was the issue of bandwidth. Multiplayer was a primary focus of NWN and we wanted to ensure that we did everything in our power to limit the bandwidth consumed by each client. In Baldur's Gate, the bandwidth used up by each client during regular game play usually ran at over 2 kilobytes per second, and we would routinely send around 100 KB save game files to each client while others were joining in. For NWN, we expected to be able to handle six or seven clients through a single 56K server. A rough calculation led us to approximately 500 bytes / second. The approaches taken in Baldur's Gate had to be reworked.

If we wanted to send a single game resource of 100 kilobytes across to the other side (as we did in Baldur's Gate), it would have taken 3 minutes to do so through a 56K server, and most people would have considered this unacceptable. This required us to have all of the data already on the machine during regular gameplay. However, this seemed to conflict directly with user-created content. Thus, we initially shelved the idea of automatically downloading the hak pak file during regular gameplay from a server, since it would be significantly slower than downloading off of another web site. After launch, we discovered that the lack of automatic downloading of hak pak files was a serious barrier to entry for people who wanted to play multiplayer, since almost every module has a different hak pak. This has not been fixed as we write this, but should be fixed by the time you read this!

	Some of the fan-created NWN persistent worlds, each with their own hak paks.

Since the world was subdivided into a number of different areas, we needed a way of sending down the area's description in the minimum amount of bandwidth. We discussed the tile system earlier, but it isn't just as simple as saying "tile 432 at location x = 20, y = 30". The lighting and shadows in each area had to work in a bandwidth-friendly manner. To make an area look attractive, you sometimes needed hundreds of lights. Since the video cards are unable to compute dynamic lights in these quantities at a reasonable speed, some form of static lighting system was needed. The standard methods, such as shadow maps, were fundamentally too slow to generate on the fly, and they generated data too large to download on the fly. The solution that we eventually come up with was to embed the light information into the tiles themselves. We then only needed to specify which lights for each tile were active and what color they were. When the client received the area description from the server, it then calculated the vertex lighting values for each vertex within the range of that light. Shadows volumes for the sun or moon were also pre-calculated at this time. We did discover that the higher end machines were able to generate shadow volumes on the fly for a limited number of lights. This resulted in the distinctive look of shadows in Neverwinter Nights.

As the last step of bandwidth reduction, we needed to do a better job on reducing the bandwidth used during regular game play, since a lot of redundant data was sent to each client in Baldur's Gate to ensure that each connected machine could run its own AI. However, the server runs all of the AI in NWN. This allowed us to use a networking model similar to the Tribes networking model [Frohnmayer00].

In a multiplayer game with only a few attributes required to display creatures properly, this shouldn't be too taxing on the server, and it wasn't when it was first added. However, we not only had a location, orientation and animation to play, but we also needed to display any one of a number of visual effects on a creature. These visual effects were allowed to stack on a creature to indicate all of the various effects the creature had on it; some creatures such as liches may have up to seven visual effects on them. For the player's creature, they had a number of long lists including prepared, memorized spell lists, as well as prepared and useable feats. The server regularly examined these lists to see if there were any changes to the spell list that needed to be broadcasted. Thus, we had numerous expensive linked list to linked list comparisons for the creature that the client controlled. We ran out of time to optimize this before the game shipped, and we uncovered the beautiful fact that in our zeal to reduce bandwidth usage, the CPU actually became the major bottleneck on the server, preventing some high-end machines from hosting more than 40 player characters.

	A list of running servers.

In spite of the restrictions of downloading files during regular game play, we were still required to send some data (in the form of a character file) from the client to the server. For the pen and paper fans amongst us, it was the equivalent of handing a character sheet to the Dungeon Master and asking "Can I play this character?" Of course, there were all sorts of security issues associated with this. Most servers would not have active DMs that could visually inspect and approve characters, so we were required to write code to validate that the rules were followed in the creation and playing of a character when it was brought into the server. The process of writing the validation was scheduled to take a couple of weeks. However, we severely underestimated the amount of time that this would take, due to the complexities of the D&D rule set. In the end, we actually had to remove some of the level-up validation checks to meet our deadline. Unfortunately, the missing checks were quickly discovered and one of our revisions was made to repair this issue.

What were some of these complexities? In role-playing games, characters also include all of the items that the creature holds, or has in his or her backpack. The majority of the size of each creature wasn't the creature itself, but the items that the creature held! However, the standard items were not that much of a problem. Each item could be examined to see whether a creature had the appropriate abilities to use the item; furthermore, each item could be validated to see if it was appropriate for the creature! One does not expect a first-level character to be carrying around a +5 vorpal sword. Thus, we had to develop a system that could be left on a server to prevent that from happening. This system not only had to work through the import of a character on to the server, but also had to work through the trade of items.

We implemented an item level restriction (ILR) system, which forced you to only have equipped items of a certain cost. If the item was too expensive and the server chose to turn on the ILR system, you couldn't equip it, you couldn't even bring it in your backpack to give to someone else! However, some of the items required for the plot had properties, so plot items were initially given a zero cost. This made all plot items usable by level 1 characters. However, we didn't actually have a different check to equip and to possess the item. Our enterprising QA staff discovered that they could make a "plot" +5 vorpal sword, hand that to a first-level character, and it would be fine to both equip and possess! So, we had to change the system again to impose the item level restriction when the item is equipped, but still maintain its cost at 0.

We also had secret inventory slots for creature hides, claws and bites. This allowed us to implement some of the more bizarre rules and "on hit" properties for Dungeons & Dragons monsters. The end users discovered that, even though they had no in-game method of putting items in these slots, that they could hack the character file, and put the +5 vorpal sword in the creature's "claw" slot, allowing them to swing their fist and have a very good chance of killing characters instantly. The exploit may have taken significantly more time to discover if we did not provide a toolset that showed end users that all creatures had these "hidden" slots!

We could not guarantee that the end-user could not write a script within the scripting language that could take down a server, insert it on their character, and destroy the game experience for others. Thus, clients were forbidden from sending scripts that could be run by the server; hence, there were no scripts on anything in the character file. However, this imposed all sorts of odd restrictions on the game design. In The Sims, each object contains all of the code for how the sim creatures deal with the object. In NWN, we were required to store a global script on each module that dictated how every object that could be brought into the world could be activated. Since end-users could create their own objects, they might accidentally create an object that had exactly the same visual properties (appearance and colors) as a special object in the module. Hence, we had to provide a password system (tags) that allowed module makers to reduce the likelihood of a collision between their objects; only if the passwords matched would the object behave in the way defined by the module. This makes it very difficult for end-users to integrate their own custom content, since they had to cut and paste the appropriate part of their Item Activation script into another module, as well as moving over the item file into the next module. However, security of the server is paramount in a multiplayer game.

Now, you're probably wondering why we went to all of this detail to tell you all of this. The parable is this: Leaving a character in an insecure environment (the client's machine), and giving them the tools to modify their own characters requires a gigantic engineering push to make it work properly. However, it can be made to work.

Our original plan for the NWN story called for approximately 30 2-hour missions that people could play together, enjoy a wonderful experience, and exit from the game. However, market research led us to believe that a vast single-player story was important to end-users; they did not want to see Mechwarrior-style missions where the party would be briefed, charge off like raging lunatics, slay and loot the villains, and then return to the tavern for a nice cup of mead. Thus, the story was reorganized into four chapters, which meant six times as many creatures and objects to deal with in the same amount of RAM!

The mission to chapter conversion required refactoring in some systems to guarantee that the amount of RAM being used was kept under tight control and a number of CPU intensive algorithms had to replace the RAM intensive algorithms that we had used earlier in development! It also increased our save games to a size unbeknownst to mankind. It also forced us to dramatically optimize our file format a couple of months from the ship date to get the save times down to the small levels they are at now. (At the point where the end-users started seeing the beta toolset with the old generic file format in it, some save games were taking 7 minutes to save. Our producer mentioned that he saved his game, made a tuna fish salad, made a sandwich using the salad, ate it, sat back down at the machine and it was still not done saving!)

Of greater concern was that the server wasn't required to have a 2-hour stability between complete reset of campaign data; it had the potential to live and survive for 10 hours in the same campaign world, and anything that could support 10 hours without crashing should be able to handle 24 hours times 7 days without crashing! Unfortunately, we never did get the time to refactor some of the other systems to ensure that the server could stay up for very long periods of time. We are still working on those issues today.