Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
Optimizing Asset Processing
arrowPress Releases
May 26, 2019
Games Press
View All     RSS








If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

Optimizing Asset Processing


October 29, 2008 Article Start Previous Page 2 of 3 Next
 

Tune Your Machines

Antivirus software should be configured so that it does not scan the directories that your assets reside in, nor the actual tools. Poorly written antivirus and other security tools can significantly degrade the speed of a machine that performs a lot of file operations. Try running a build both with and without the antivirus software and see if there is any difference in speed. Then consider removing the antivirus software entirely.

If you have any form of distributed “farm” of machines in the asset pipeline, beware of any screensaver other than “turn off monitor.” Some screensavers use a significant chunk of processing power. You need to be especially careful of this problem when repurposing a machine; the previous user may have installed her favorite screensaver, which doesn’t kick in for several hours, and then slows the machine to a crawl.

Write Bad Code

In-house tools don’t always need to be up to the same code standards as the code you use in your commercially released games. Sometime you can get performance benefits by making certain dangerous assumptions about the data you’re processing and the hardware it will be running on.

Instead of constantly allocating buffers as needed, try allocating a “reasonable” chunk of memory as a general purpose buffer. If you have debugging code, make sure you can switch it off. Logging or other instrumenting functions can end up taking more time than the code they are logging. If earlier stages in the pipeline are robust enough, then (very carefully) consider removing error and bounds checking from later stages if you can see they are a significant factor.

If you have a bunch of separate programs, consider bunching them together into one uber-tool to cut the load times. All these are bad practices, but for their limited lifetime, the risks may be outweighed by the rewards.

Minimize I/O

Older programmers tend to write conversion tools using the standard C I/O functions: fopen, fread, fwrite, fclose, etc. The standard method is to open an input file and an output file, then read in chunks of data from the input file (with fread or fgetc), and write them to the output file (with fwrite or fputc).

This approach has the advantage of being simple, easy to understand, and easy to implement. It also uses very little memory, so quite often tools are written like this. The problem is it’s insanely slow. It’s a holdover from the (really) bad old days of computing, when processing large amounts of data meant reading from one spool of tape and writing to another.

Younger programmers learn to use C++ I/O “streams,” which are intended to make it easy for data structures to be read and written into a binary format. But when used to read and write files, they still suffer from the same problems that our older C programmer has. It’s still stuck in the same serial model of “read a bit, write a bit” that’s not only excessively slow, but also mostly unnecessary on modern hardware.

Unless you’re doing things like encoding .MPEG data, you will generally be dealing with files that are smaller than a few tens of megabytes. Most developers will now have a machine with at least 1GB of memory. If you’ll be processing the whole file a piece at a time, then there’s no reason you should not load the entire file into memory.

Similarly, there’s no reason you should have to write your output file a few bytes at a time. Build the file in memory, and write it out all at once.

You might counter that that’s what the file cache is for. It’s true: The OS will buffer reads and writes in memory, and very few of those reads or writes will actually cause physical disk access. But the overhead associated with using the OS to buffer your data versus simply storing it in a raw block of memory is very significant.

Listing 1 shows a simple file conversion program that takes a file and writes out a version of it with all the zero bytes replaced with 0xFF. It’s simple for illustration purposes, but many file format converters do not do significantly more CPU work than this simple example.


LISTING 1  Old-fashioned file I/O

FILE *f_in = fopen("IMAGE.JPG","rb");
FILE *f_out = fopen("IMAGE.BIN","wb");
fseek(f_in,0,SEEK_END);
long size = ftell(f_in);
rewind(f_in);
for (int b = 0;b<size;b++) {
    char c = fgetc(f_in);
    if (c == 0)    c = 0xff;       
    fputc(c,f_out);
}
fclose(f_in);
fclose(f_out);


Listing 2 shows the same program converted to read in the whole file into a buffer, process it, and write it out again. The code is slightly more complex, yet this version executes approximately ten times as fast as the version in Listing 1.


LISTING 2  Reading the Whole File into Memory

FILE *f_in = fopen("IMAGE.JPG","rb");
fseek(f_in,0,SEEK_END);
long size = ftell(f_in);
rewind(f_in);
char* p_buffer = (char*) malloc (size);
fread (p_buffer,size,1,f_in);
fclose(f_in);
unsigned char *p= (unsigned char*)p_buffer;
for (int x=0;x<size;x++,p++)
    if (*p == 0) *p = 0xff;
FILE *f_out = fopen("IMAGE.BIN","wb");
fwrite(p_buffer,size,1,f_out);
fclose(f_out);
free(p_buffer); 

Article Start Previous Page 2 of 3 Next

Related Jobs

Gear Inc.
Gear Inc. — Hanoi, Vietnam
[05.25.19]

Technical Director
Dream Harvest
Dream Harvest — Brighton, England, United Kingdom
[05.25.19]

Technical Game Designer
Deep Silver Volition
Deep Silver Volition — Champaign, Illinois, United States
[05.24.19]

Senior Animation Programmer
Ubisoft RedLynx
Ubisoft RedLynx — Helsinki, Finland
[05.24.19]

Senior/Lead Graphics Programmer





Loading Comments

loader image