| |
| | ||||
![]() | ||||||
| | | |||||
|
GDC 2003: Neverwinter Nights Client/Server Postmortem: How I Learned To Stop Worrying And Love The Magic Missile How Many Programmers Does It Take To Screw In Neverwinter Nights?Neverwinter Nights was by far the largest project that BioWare has ever attempted. By the time in was completed almost everyone in the company had made contributions to it. At its peak, there were 25 programmers working on the project. This was a radical departure from Baldur's Gate, our largest project previously attempted, which only had 7 programmers. The fact that we had so many programmers on Neverwinter Nights was a direct result of the dual need for an unbelievably large feature set, and a relatively short development time of three and a half years. This is comparable to Baldur's Gate, which took us almost three years to the day to program. Our best estimate is that Neverwinter Nights ended up with about double of both the number lines of code and identifiable features. This is a good indication of the loss of efficiency that is generated by increasing the complexity of project while trying to maintain the same time lines. Another interesting "truth" that we learned was the fact that if you have a senior programmer managing and providing technical design for more junior programmers, once the number of junior programmers exceeds 4, then the senior programmer is unable to do any actual programming of their own. Even with a team of experienced people, there still needs to be someone that is ultimately responsible for each major system to keep the system from self-destructing.
With a four-fold increase in programmers, we had to implement new ways of dividing up tasks and maintaining areas of responsibility. Separating out the tools and graphics teams worked very well because of the well-defined points of contact between their systems and the core game. Dividing up the game code itself was a completely different situation. Our experience with Baldur's Gate had taught us some lessons on the importance of integrating the multiplayer code in right from the beginning. We didn't start actually multiplayer programming in Baldur's Gate until over halfway through the project. We had a single programmer go through all of the systems and add the multiplayer support with the predictable significant number of multiplayer problems. In Neverwinter, we started with multiplayer support from day one. We created two teams, one for the client and one for the server with the mandate that only communications between the client and server modules would be through the network layer. In fact for the first year, we only had stand-alone client and sever versions of the program. This separation was continued even after the integrated client/server executable was built. By having separate programmers on the client and server, there was very little temptation to "peek over the wall" and share data directly. This created a game where there essentially wasn't a single player mode and as a result, we had many times less multiplayer specific issues. The downside of this decision was the increased complexity in both implementation and organization. Since every system had to be implemented on both the client and server before it could be tested, it caused several programming bottlenecks as the programmers were waiting for their counterpart to finish. In addition, there was also a significant lessening of ownership for each system. Since no one programmer was directly responsible for the system, it was harder to keep the programmers interested and motivated. The separation of the programmers that caused problems with the game code did work well between the tool and game programmers. The main difference between these two situations was in the manner that they interacted. First of all, the tool to game data path was essentially a one-way process. The tools wrote out data files and the game reads them. Since the tools were also required to read the data back in for editing purposes, they were able to validate the integrity of the data without the game code being written. Secondly, the data was mostly written into Generic File Format (GFF) files. This was a tagged field file format that allows for adding and deletion of fields without destroying the original data files. This allowed the tools to generate newer versions of the files and still have them readable by older versions of the game. Contrast this against the two-way client/server communication. In this case bandwidth was at a premium causing the message formats to be bit packed. Any changes to either the client or the server broke the system until the other side was also updated. Another issue was the passing of ownership of game systems. Due to demos and the need to keep ahead of both the design and art teams, game systems were not implemented in the most optimal order. Occasionally, the expertise of a particular programmer was needed on a more critical system and the ownership of the system that they were previously working on had to be reassigned to another programmer. Ideally when this happened, the other system was just delayed until the more critical system was finished. Unfortunately, there were times when both systems were needed for a particular demo. In the case of the inventory system, ownership was passed four times before it was completed. In our experience, the more times that the system was passed along, the more brittle that system became. We concluded from this experience was that the areas of responsibility should be divided along well defined sections preferably into sections that can be implemented and tested independently by a single programmer. In addition, passing of ownership of any given system should be avoided if at all possible.
This is not to say that the Neverwinter Nights toolset did not also have its own set of problems. It didn't take us very long to determine that producing a full role-playing toolset that would be accessible by a large segment of our audience would be a massive undertaking. While we only had a single tools programmer for Baldur's Gate, we had seven at the peak of Neverwinter and we still had to delay some of the optional features like the plot wizard until post release. The key to making any product user friendly lies in the design and implementation of the user interface. It's acceptable for internal tools to be of the "point click and crash" variety because you can spend time training the development staff not to press particular buttons in particular sequences. Please don't get the impression that we are saying that buggy tools are ok, it's just that doesn't take a lot of revisions of a user interface before you are spending more time developing a tool then you would spend time training people to use a slightly more confusing version. If you are planning on releasing a tool to the general public and expect them to actually use it, then you better assume that most of them are going to give up if they have to crack the manual before they can see any results. Because the large number of interface revisions that we expected, we decided that the toolset had to be written in a rapid development environment. Since we also wanted to share source code between the toolset and the game, this limited it to a development environment that supported C/C++. Our final choice was Borland's C++ Builder. Overall this was an excellent tool to use although we did run into several incompatibilities between it and the game's development environment, Microsoft's Visual C++. Another problem involving the toolset was caused by the tight design timeline. Ideally the entire game and toolset should have been completely finished before any designer started to make final content. In reality, the tight timeline forced the designers to create content before the various games systems were complete. This also meant that the toolset was also still in development while the designers were creating final content. Theoretically, the parts of the toolset that the designers were using should not have been affected by new features. Unfortunately, general instabilities were the cause of significant work loss. One of the biggest problems was the addition of last minute fixes. Since the design team had the tightest schedule, the development of the toolset was scheduled to allow the greatest number of designers to work without being blocked. If a bug or a missing feature blocked any designer, a fixed version of the toolset was immediately posted. Nothing could ever go wrong with this plan! Let's make up some time by steaming through this iceberg patch!
Inconceivably, this had the unfortunate effect of posting untested code and other newly introduced bugs. After a couple of errors, we only did scheduled releases of the toolset (just as we did scheduled builds of the game). Before the designers would have access to the new version of the toolset it had to pass through a complete cycle of testing by our QA team. While we would still have a few designers blocked for a day or so waiting for a new feature or bug fix, the amount of work lost due to crashes or data corruption was significantly reduced. My Player's Handbook says We have a fairly open communication with our fan community at BioWare Corp., and using a licensed rule set (such as Dungeons & Dragons) does get you a lot of fans who are excited about the game. However, this induces a much heavier load on implementation. For example, an obscure exception, which may or may not be mentioned within the help text for a given feat, was unimplemented. Having a number of people, both internal and external to the project, who are intimately familiar with the rules facilitates spotting little errors such as these before the game reaches store shelves. However, there is another issue when converting a pen-and-paper rule set to the computer. Not only do you have to adequately define, document and test what is being implemented, but you must describe the variances between that and the written rules. Our procedures were somewhat lacking in that regard: internal and external QA were reporting many bugs that were not bugs in NWN, and end users had different expectations based on reading the core rulebooks. Our technical design document, which the designers spent a lot of time writing, was not required reading for everyone who worked on NWN. And the programmers were no better than anyone else! One day, the designers got so frustrated they actually took away the core rulebooks from the programmers' desks to get us to read the document.
______________________________________________________
|
||||||||||||||||||||||||||||||||||||||||||||||
|
|