|
[When the schedule is shot and a game needs to ship, programmers may employ some dirty coding tricks to get the game out the door. In an article originally published in Gamasutra sister publication Game Developer magazine earlier this year, here are nine real-life examples of just that.]
Programmers are often methodical and precise beasts who do their utmost to keep their code clean and pretty. But when the chips are down, the perfectly-planned schedule is shot, and the game needs to ship, "getting it done" can win out over elegance.
In a case like this, a frazzled and overworked programmer is far more likely to ignore best practices, and hack in a less desirable solution to get the game out the door. We have here compiled nine testimonials from working developers, which chronicle times when they weren't quite able to follow the script and had to pull some tricks to save a project.
- Brandon Sheffield
[If any readers have any dirty coding tricks of their own to share, please email them to Brandon Sheffield at bsheffield@gdmag.com. Illustrations for this article by Jonathan Kim.]
Shoot Me From My Good Side
Around four years ago I was working as a programmer on a multiplatform PlayStation 2, Xbox, and GameCube release. As development was coming to a close it should come as no surprise that some code hacks started creeping into the game to get the title shipped. The PS2 version in particular had a late, extremely hard-to-track-down issue: A long soak test of the player character standing still in the first level of the game would cause it to periodically crash. Unfortunately, the crash was limited to disc-only retail builds that included no debugging information. With fears of a Sony technical requirement check (TRC) failure looming we worked hard to find a solution.
To narrow down the issue we continuously rendered a border around the edge of the screen with a different color for different sections of the code. For example, blue represented render setup, and green was the player control update. Because the crash was intermittent, the lead engineer on the title and I would manually burn a bunch of discs and start them up on an array of PS2s. (Keep in mind this was an independent developer, with no IT team or fancy disc burning machines.) The test cycle -- between making a code change, burning discs, deploying, and waiting to whittle down some approximation of where the crash was -- took hours. It was the eleventh hour of development and there was a fixed number of these test cycles we could conduct.
Between stretches of passing the time over the weekend in the office with World of Warcraft while waiting for a crash, someone noticed that the game didn't crash if you rotated the camera 90 degrees to the right. Initially, the programming team correctly waved this off as some kind of fluke that was masking the underlying bug. This isn't the type of "fix" that would repair the root cause of a crash. That didn't stop us from shipping it though! As we ran out of time, we created the final PS2 disc with a rotated camera at level start -- the other two platforms were already on track -- and submitted everything. The game passed all the platform holders' TRC tests and was shipped to market on time.
I wouldn't call this a "coding trick," but it was definitely "dirty." The mysticism of the fix is disconcerting to this day, but thankfully I never heard of any reported user issues.
- Mark Cooke
Identity Crisis
This scene is familiar to all game developers: It's the day we're sending out the gold candidate for our Xbox 1 game. The whole team is playtesting the game all day long, making sure everything looks good. It's fun, it's solid, it's definitely a go in our minds.
In the afternoon, we make the last build with the last few game-balancing tweaks, and do one last playthrough session when disaster strikes: The game crashes hard! We all run to our workstations, fire up the debugger, and try to figure out what's going on. It's not something trivial, like an assert, or even something moderately hard to track down, like a divide by zero. It looks like memory is garbage in a few places, but the memory reporting comes out clean. What's going on?
One dinner and many hours later, our dreams of getting out on time shattered, we manage to track it down to one data file being loaded in with the wrong data. The wrong data? How's that possible? Our resource system boiled down every asset to a 64-bit identifier made out of the CRC32 of the full filename and the CRC32 of all the data contents. That was also our way of collapsing identical resource files into a single one in the game. With tens of thousands of files, and two years of development, we never had a conflict. Never.
Until now, that is.
It turns out that one of the innocent tweaks the designers had checked in that afternoon made it so a text file had the exact same filename and data CRC as another resource file, even though they were completely different!
Our hearts sank to our feet when we recognized the problem. There's no way we could change the resource indexing system in such a short period of time. Even if we pulled an all-nighter, there was no way to know for sure that everything would be stable in the morning.
Then, as quickly as despair swept over us, we realized how we could fix this on time for the gold candidate release. We opened up the text file responsible for the conflict, added a space at the end, and saved it. We looked at each other with huge grins on our faces and said:
"Ship it!"
- Noel Llopis
Driving Under the Influence
I've been tasked with taking care of a system dealing with vehicles, which is a fairly major part of our current game. Fortunately, we were able to grab the majority of the code from another studio. Unfortunately, the code isn't always perfect or well written, as is the perception with most code you don't write personally. I happened to find this nice bit of code that was simply trying to get a variable from the engine loop, but was going about it in the most backward way possible (See Listing 1).
Listing 1: Driving Under the Influence
//**************************************************
// Function: AGameVehicle::Debug_GetFrameCount
//
//! A very hacky method to get the current frame count; the variable is protected.
//!
//! \return The current frame number.
//**************************************************
UINT AGameVehicle::Debug_GetFrameCount()
{
BYTE* pEngineLoop = (BYTE*)(&GEngineLoop);
pEngineLoop += sizeof( Array<FLOAT> ) + sizeof(
DOUBLE );
INT iFrameCount = *((INT*)pEngineLoop);
return iFrameCount;
}
The funniest part about this chunk of incredibly gross code is that there was a simple function on that object to get the frame count. Even if there weren't, the person who wrote this code could have easily added one himself! Needless to say, this code was not added to my game, nor to the game this code was taken from, as soon as I pointed it out. If this isn't an example of why code reviews are good, I don't know what is.
- Austin McGee
|
en.wikipedia.org/wiki/Checksum
I was just about to tell the team the good news when the producer pulled me aside, and mimed a "shh!" gesture.
The team continued to crunch, none the wiser, until we reached the now fake deadline. After that, the producer acted dissappointed in us, and told us to keep going.
Boy, that producer really knew how to get work out of people who were demoralized and exhausted already! HAH HAH!
I feel strangely more human all of a sudden!
As for 'The Programming Antihero" - Well on that one I actually thought *everyone* did that. The real trick when coding solo is to make yourself forget about that array until the very last minute of the project.. ;)
There is a special place in hell for producers that give resources fake deadlines. It's a trick bad producers use to make themselves look good, at the expense of everyone else. In this case, your team suffered so your bad producer could be the hero to the client (and upper management).
Way to cover up for him/her!
Did you like seeing your other team members get extra demoralized and extra exhausted? I guess letting them also be "super relieved" was not your problem. They must have been amazed at your ability to handle the stress of the "unrealistically aggressive" fake deadline. Hopefully that "can-do" attitude got you a promotion.
Good producers work honestly with their team, and they get far more accomplished as a result.
The fake deadline trick tends to only work once (it always gets discovered).
Ever notice that teams miss deadlines for some producers more than others?
Ever wonder why that is? It's all about trust and respect.
HAH HAH!
As some have already mentioned, this is done fairly often with deadlines in quite a few fields. I've not ever seen it done with memory constraints, though!
I can see I'll be looking forward to more in the comments, too. ;)
As far as that goes, we were recently having problems where the player avatar would mysteriously disappear and become unresponsive due to the screen clipping against a non-square map. We found the fix readily enough, but the performance cost was so high, that we ended up just forcing the player to render if they were alive with the comment:
// HACK
// Players should ALWAYS be on screen if they exist.
There were also some related to how we had to implement our own renderer to get half-decent performance, but the real ugly stuff was in the grid optimisation. We lost about two solid weeks ironing out the bugs in that (which, considering our deadline, was actually about a quarter of the total development time; madness), and we still ended up with another avatar-specific hack so players wouldn't be dropped from the game state if they crossed a grid boundary. The code for this is still enclosed in
// HACK HACK HACK LOUSY HACK
and
// End of lousy hack.
But I love the trick in "The Programming Antihero"; I think I'm going to start using it. :D
Employees is a little vague though. Need a way of distinguishing between management, account managers, and senior leads (techincal or creative). Unless the producer is insane, those employees aren't given fake deadlines to motivate them. It's only the people underneath the producer, the creative and technical resources, that get abused. Not sure how else to group them.
What could be more obvious? Only render every second letter! But alternate, so that on even numbered frames, you render even-numbered letters, and on odd-numbered frames, you render odd-numbered letters!
This leaves you a flickery, but fast and usable, debug menu.
Stuff like this is a great read. Not only is it amusing but it also teaches you tricks you can implement when in a crisis, or more importantly what kind of problem might blow up in your face and how you can avoid them to begin with.
Sir, do not worry! You may have missed my sarcastic tones, and also I missed a key plot point: I did the exactly opposite of cover for him.
I was as shocked as you, and told my comrades the story behind the producer's back, because I'm a LOOSE CANNON. It was still a hard push, but they were able to pace themselves.
"static char buffer[1024*1024*2];"
That's just priceless.
i really enjoyed every one. I would love to see the second edition of this, or maybe more!
Profuse apologies, I TOTALLY did not realize you were joking.
Tone totally changes everything.
I applaud your approach.
I deem thee "Sir" Loose Cannon.
@Brandon
Forgot to mention I loved the article.
Great reading. I have gained wisdom.
Major kudos to Noel Llopis' old hand coder for that memory trick!
At university there was a team (not related to me, but these guys are the perfect example :P) that made a FPS flash game...
For some bizarre reason, the programmer instead of checking if you was colliding with the wall and not allow you go there, he made the inverse, he checked if there was a wall, and allowed you to move parallel to it...
This sparked a bizarre bug: In crossings, you could not actually cross, only turn to the passage on your left or right.
The deadline was closing, and they had no idea on how to fix it...
Then the team writer fixed the issue! He told the artist to draw a animation of hands touching the walls, and then he wrote in the story that the protagonist was blind and needed to touch the walls to know where he was going.
//@hack
//@remove
//@fixme!
And, well... they almost never get changed :)
Turned out the flash init code was dead and the carts could not save games properly!!! Studio went into meltdown trying to figure out how to ship 250'000 broken carts. Suggestions of production lines adding extra resistors and other hacks to every cart were tried and failed , then some figured out if you played some games in a weird order the flash memory would sort of work. So i extra leaflet was added to every box explaining how to use this "feature".
Job done
Chris Kirby
That's the best thing I've ever heard!
Yes, of course...if you are in need of many megabytes of memory, inspecting the code-size is definitely not the most obvious thing to do ;) but, at least for console/handheld-projects, it is somewhat of a standard procedure.
I have found similar (but unintended) stuff in earlier projects while inspecting the map-file once in a while.
------
Don't know how many remember Force 21, but it was an early 3D RTS which used a follow cam to observe your current platoon. Towards the end of the project we had a strange bug where the camera would stop following the platoon -- it would just stay where it was while your platoon moved on and nothing would budge it. The apparent cause was random because we couldn't find a decent repro case. Until, finally, one of the testers noticed that it happened more often when an air strike occurred near your vehicles. Using that info I was able to track it down.
Because the camera was using velocity and acceleration and was collidable, I derived it from our PhysicalObject class, which had those characteristics. It also had another characteristic: PhysicalObjects could take damage. The air strikes did enough damage in a large enough radius that they were quite literally "killing" the camera.
I did fix the bug by ensuring that cameras couldn't take damage, but just to be sure, I boosted their armor and hit points to ridiculous levels. I believe I can safely say we had the toughest camera in any game.
------
When I was a lead on R6 Lockdown PS2, I also used the "The Programming Antihero" trick -- whenever an engineer found a significant memory savings I told them to release enough to get the game under budget for the time being, but hold some back for the eventual future overrun. In the end we had a 1-2K buffer (IIRC) that we released just before ship. Most of the team never knew.
Well, this is basically what Microsoft uses as their WPARAM and HPARAM in WinProc, so it wasn't that bad =)
@Evan Bell: I am a programmer in the healthcare industry and hit that lottery once some years ago. I had to load a text file with over 100k lines, one per medicine, and two lines gave the same CRC32. Fortunately the update routine was run in our server and the clash was easily found. For two files with the same CRC32, check http://www.allegro.cc/forums/thread/585925
Although not game derived, I got a couple to share. Around 5 years ago we had to change the L&F of the application. That included switching menus over, changing background images, reshaping controls, etc. It went pretty well (at 16 hours per day, took us just a week). However, testers complained that the application started to randomly crash after some time of use, with an out of memory error. Just 4 hours before shipping we discovered the error: the new rounded buttons didn't free the normal/pushed bitmaps correctly, eating memory every time they appeared on screen. So we had to remove the new buttons, and shipped a Mac-like rounded application with gray square buttons.
Regarding fake deadlines, we once decided to fulfill it. Indeed, we reached it, had the product uploaded to our servers, fully tested and waiting for the approval of the CEO to be released. But he decided not to launch it. The thing was that he had put a very impossible goal so that we would be forced to delay the package, giving him time to suggest more features before the real launch date arrived.
Also, whenever the physician chose a drug, it would be stored into the database, linked to the patient and a recipe within a visit. However, one of the programmers really messed up and, instead of storing the NDC (national drug code as set by the FDA), he stored the drug index in the drug database. The drug database was usually modified once per month (removing drugs that had expired, adding new drugs alphabetically, etc) so that, when I checked, the IDs in our database were usually pointing to drugs that were not the chosen ones. Even though the contraindication check used names instead of
NDC (which made the routine work still), it was too risky to have random numbers as medicine codes, so we had to clean them up. None of them was salvageable, and I had to wipe out the entire column, but there was no way to explain that to the users without making them think they were in great risk. So, I placed a progress bar that lasted around 7 minutes with something like "Upgrading drug information" which did nothing at all, and notified the user that he would be forced to relink some (all!) of the medicines in the program database with the ones from the medicine database as he starts reusing them. None of the doctors complained, though, and everything has been going smooth since then.
Chris Kirby's one reminds me of a couple. A programmer had just installed a spell checker into our program, with the ability to add your own custom words, and he started testing with words like a**, f*ck, and similar. Unfortunately, he didn't clear the table and we got angry calls from one doctor who was showing his secretary the new spell checking functionality when f*ck popped up. So, always clear your databases before shipping the product!
Great feature! I look forward to "Dirty Coding Tricks II"!
One of the "impossible" errors were: "OH MY GOD, THE GAME IS TOTALLY FUCKED AND THE WORLD WILL EXPLODE BECAUSE SOME IMPOSSIBLE SHIT HAPPENED AND ACTIVATED THE IMPROBABILITY DRIVE CAUSING A TIME PARADOX THAT WILL SCREW THE UNIVERSE IN ALL FUCKING SEXUAL POSITIONS POSSIBLE"
After I typed that, I plainly forgot it, it was embedeed deeply in some initialization routines of the API, a code that once done I would never peek at, also it was the only case that I used profanity or that sort of stuff (I am a person that is not much into profanity, but that day was a bad day, so I ranted on that error message).
Unofortunally, one of the testers actually made that error happen... He called me all confused, and me too (since I don't remembered it) and I tought that he was joking, but I searched and actually found it...
Then I had two questions on my head:
How that error happened? (awnser: a mistype that I introduced on the lastest build caused a nasty chain reaction of bugs that made the execution stack go awry, making the impossible actually possible).
Why I did wrote so much profanity? (awnser: In fact I am still wondering the awnser for that...)
So this is a way to access private/protected members if there are no getter/setter.
Now I'm not saying I'd do that but in the case you're working on a program with an API and you only have access to the .h file, you could get around it this way. =)
I don't know why this ever worked but I had read that if you shift the mouse position left and then right it solved the problem. So I added that and it worked.
Later a programmer looked at this and thought it was just self cancelling and commented out the code and you could no longer click the button.
I explained the real problem was that we were not counting "Mickeys" and should move to a relative system. He asked if I knew how and that is how I moved from being a tester to a programmer.