     |
 |
 |
 | | | | |
 |
|||
 |
||||||
          |
 |
 |
||||
|
Postmortem: Insomniac Games' Ratchet & Clank What Went Wrong 1. Poor disc-burning process. Making the switch from CD-ROMs on the PSX to DVDs on the PS2 sounded like it would be easy. After all, we survived the challenge of recording PSX discs with quirky burners and nonintuitive burning software. What we didn't account for was the incredible amount of time that building and burning the DVDs would take.
We had to first transfer the code and data to the PC on which we would generate the files necessary to create a playable disc. Next we'd have to transfer the files to the burner PC. Then the burner software would have to create a disc image, and finally we could burn the disc. By the end of the project we were working with 4GB of data. Combining those steps with slow connections and a burner that we had to use at only double speed to prevent errors, the entire process took more than four hours to generate one disc. And there were many, many places along the way where something could go wrong, forcing us to start over again. There were countless instances where a level would be out of memory or someone would change the memory card format, breaking everything. But we wouldn't know about it until the final disc had popped out of the tray and had been booted up on a test station. Two mistakes like this would cost an entire day. So why didn't we change the process? Based on our PSX-burning experience, where the system was extremely finicky, when we had things working on the PS2 we didn't want to touch it and risk breaking everything. This was especially true near the end of the project. As a result, a few of us didn't go home for days at a time near the end of the project. I remember promising our testers that if our first gold burns worked, I would do circuits of the office singing Britney Spears songs as loud as I could. Fortunately for everyone in the office, they didn't. The result of our disc-burning pain is that we've now completely overhauled our system. We believe we've halved the overall disc production time for our current project.
Fortunately, the animators had finished most of the in-game animations by the time the movies were in full swing. But it was still a real challenge. Furthermore, animating the scenes was just the first step. We had to add programmatic and 2D effects and convert many of the animations into MPEGs before alpha, which stretched many people to the limit. We got such a late start because we had to finalize the story, write the scripts, audition the actors, record the dialogue, and put the final sound files together before starting the animation. It helped somewhat that we took an iterative approach -- starting animations as soon as the first scenes were recorded -- but in general the tardy start created a lot of stress. 3. Immense level designs. Even though we tempered our ambitions for the macro design, sometimes we cut loose and created some absolutely huge level designs. We had a habit of wanting to make each level better than the last, and a few times this tendency resulted in layouts that made the artists want to kill the designers. Early on, we didn't have a good understanding of what "too big" meant. The first level designs we created were reasonable, but then we decided that we really needed to show off the power of the Ratchet technology. We also had some ambitious gameplay ideas involving a fight on a moving train and a hoverboard race. This resulted in the Metropolis and Blackwater City levels, two of the biggest in the game. When the artists saw the layouts they said, "Are you nuts? There's no way we can build this in six weeks!" So the designers went back to the designs and tried to edit them, but the levels still ended up being massive. To the artists' and gameplay programmers' credit they made these and other huge levels work, and they did it on time. And to the designers' credit, they continued to find better and better ways to put more gameplay into smaller areas without sacrificing creativity. In the end, our level design ambitions pushed the limits of time and resources we had allotted. Out of this stress came a more team-oriented approach to level design, where we now involve a large number of people -- artists, programmers, sound engineers, and others -- earlier in the design process. Whether or not levels in our future games will be smaller remains to be seen. But with more people involved at the beginning stages, we can find solutions sooner to balancing the need for gameplay space in levels with the time we have available to build them.
4. Maya issues. Maya is a superb tool for building polygonal environments and characters, and it's also great for animation and for prototyping particle effects, rendering, and many other things. However, early in the project we had decided to use Maya as our construction, texturing, lighting, and gameplay placement tool. We had abandoned our in-house tool, Karma, which we had used previously to do gameplay placement, texturing, and lighting. What we didn't realize was that with the size of our levels, we would push Maya past the breaking point. Even though we set people up with dual 1.2GHz Dells with superfast graphics cards and a gigabyte of RAM, Maya would still chug and frequently crash whenever our levels got up to around 40MB. And forget making all 500K polygons in a level visible. Fortunately, Al Hastings and T.J. Bdelon worked valiantly to create a suite of plug-ins and tools that worked with the Maya API. This solution didn't always prevent the crashes that plagued the artists or the occasionally corrupted level, but it kept us running and allowed us to create finished levels every six weeks. While Maya has always been and probably will still be our first choice for art creation, we're moving back to our original approach of using proprietary tools for things like gameplay setup, lighting, and texturing.
5. Localization woes. From the beginning we planned to include the NTSC and PAL versions of the game on one disc. This plan created two problems for us. First, we had to send all of our assets to Europe for localization in French, Italian, German, and Spanish as early as possible. In most cases this meant pre-alpha, which really put the squeeze on the animators who were working on the movies. Second, we knew that we would end up fixing both functionality and localization bugs at the same time. We anticipated that this would create even more chaos duri ng the last few weeks before we went gold. And we were right. Surprisingly, the biggest nightmare for us was the text localization. We had made the decision to allow subtitles for all of the movie scenes; plus we had a lot of text for the help system and a ton for the menus. We used spreadsheet databases to ensure some organization for all of the text (as opposed to entering localized text in the actual code, which we did on the Spyro series), and this allowed us make updates and changes quickly. But the system was also prone to user error when cutting and pasting changes into the database.
Because we were still fixing TRC (technical requirement checklist) bugs -- things like memory card messages -- we were making text changes up to a couple of weeks before gold. We had also added some text late in the process to support some of our postgame features. We made mistakes, and the localization folks in Europe made mistakes when putting fixes into the database. In addition, it took forever to transfer our discs to Europe once they were burned (eight hours to FTP if nothing crashed, 24 hours for a courier). These facts combined meant that we were still desperately trying to resolve some TRC issues hours before the gold disc was due. Fortunately, the game shipped on-time in all territories, but I think it prematurely aged our producer in Europe, as well as a few of us here. The Will to Kill With this project, we had to fail to succeed. Had it not been for the pain we went through on I5, Ratchet & Clank might have never emerged. In the six months of preproduction on I5 we learned how to make games on the PS2, and we were able to hit the ground running when we switched to Ratchet & Clank.
Most importantly, we were very fortunate to have an extremely supportive publisher in Sony. SCEA's Shuhei Yoshida and Connie Booth helped us make the agonizing decision to shoot I5 in the head. But they made sure we understood that if we wanted to continue down that dark path of developing I5 for release, they would still support us. Furthermore, Sony never once threatened to cancel the I5 project or sever our relationship. Instead, they helped us to develop what Mark Cerny calls "the will to kill" ? meaning we grew the balls to voluntarily throw out everything we had worked so hard on for six months and start over. The development process that Ratchet & Clank represents as a finished game is the ultimate example of how developer-publisher relationships can and should work. Sometimes good teams make games that aren't good. When a developer has the support of a great publisher and can cut off a nonperforming project in preproduction without fearing reprisals, everyone can save millions in production costs and apply the lessons learned to the next project. Doing so may cost money in the short term, but ultimately it may give birth to a blockbuster, strengthen the development team, and solidify the relationship between the developer and publisher.
[back to] Introduction |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Features
2.
Late start on cinematics. Ratchet & Clank has a much
more lengthy and involved story than any of our previous projects.
Oliver Wade, our animation director, compiled the scenes and found
that we've got more than 60 minutes of movies. Even though most
of them are about 30 seconds long, that's a lot of animation time.
The problem was that we only gave our team of seven animators five
months to animate them. That doesn't sound too bad until you consider
that the animators creating the movies were also responsible for
the in-game animations. Therefore they effectively had two-and-a-half
months. If you don't include weekends, that's about 10 seconds per
animator per day. And that's a lot.
|
|
|