What Went Wrong

1. Epic Code Complexity

We began work on the project almost two years from ship. At the start we planned to "ninja in, make some minor changes, replace the graphics system and ninja out." As we began to work with the code base, we found severe performance and stability issues with the engine. We traced some issues back to the core threading model of the Infinity engine.

Infinity was multi-threaded before there were processors capable of parallel code execution. As such, it was designed around a concept of threading that never really emerged. The main issue was that all of the threads shared the same set of data. The problem is all these threads were created and they all hit the same memory and as such, blocked each other, stalling all threads until the current one completed.

The code was also heavily laced with critical sections, which caused (in some cases) 70 percent of execution time spent in no-ops. Our long-term solution was to machete in and remove the threading, removing a couple hundred thousand lines of code from the game. The end result was more stable and had fewer performance irregularities. The "ninja-in" approach was tried in other areas, as well, and every time the end result was consistent. We always found a ton of complexity solving an era-specific problem that no longer applied. We continue to find roadblocks in the code and we're improving it as we go.

2. Intricate Code-Data Dependency

I have a development joke I often drop: "When a programmer believes he/she is being clever is when they create the greatest atrocities." The Infinity Engine is chock full of clever. For example: to render a character to the screen, the correct frame and orientation of a character sprite is first loaded out of a resource file using a very heavy resource-management system. The sprite is then color-mapped using a 256-color palette swap to enable player colors. Following the re-mapping, any number of further palette-manipulation code (stoneskin, anyone?) can step in and further change the actual data. Then the sprite is rendered against whatever potentially covering elements are nearby.

The final result is sent to the screen in 64x64 pixel tiles to be rendered. The entire system runs under a dynamic update system that flags 64x64 tiles as updated and renders them or, with no change, leaves the tile from the previous buffer. The volume of clever shifts and tweaks along the way make it nearly impossible to track down all the ways in which a simple sprite can be manipulated. In some cases, the data can reference many different .2da data files on how it can be manipulated, from equipment-changing animation frames to a data redirection to render a dwarf sprite instead of the default human. Complexity is par for the course in an RPG of this magnitude, but the intricate linking of assets and code really limited our ability to make the architectural improvements we wanted to make.

3. The Lost Source Art

We had drafted the original deal in the context of a Baldur's Gate: HD. Our plan was simple: grab the original artwork, clean it up, re-render it at higher resolutions and with better materials, thus creating stunning versions of the areas everyone remembers. We planned to take the character models and re-render them with many more frames of animation and add new orientations to the movement to make the game smoother. We nailed down the core terms, got everyone on the same page and we got our first drop of the assets from BioWare.

A few days later we noticed a large hole where the source art should be -- stuff like 3DS Max files and texture images. "No problem," I said; I contacted Derek French over at BioWare and he dug further and sent us more data. We again dug through and failed to find the source art. I made arrangements to visit BioWare with a removable drive and work with Derek and the IS department to find the assets.

After two days of searching we came to the horrible realization that the source artwork was stored on a departmental drive and not a project drive, and as such was not frequently backed up. We dug through tape backups to no avail. The source art was lost.

At this point we brought the information back to Atari and the deal was dead in the water. Cameron and I spent a few weeks re-thinking the concept and we re-pitched Atari with an "Enhanced Edition" as opposed to an "HD" version. We discussed the implications of a non-HD version and we had to renegotiate royalty terms to get the deal back on track. We basically had to completely discard our plans and start anew after almost a year of negotiation.
 
Click for larger version

The end result is a little less graphical flash for launch than we had initially planned, but we actually had a bit more time to make more widespread improvements as a result, so the project is different; in some ways larger, but still pretty awesome. The key learning here is: don't panic and always have a towel, because you never know when you are going to be surprised... and towels can be very useful.