Gamasutra is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.


Gamasutra: The Art & Business of Making Gamesspacer
Fast File Loading (Pt. 1)
View All     RSS
October 14, 2019
arrowPress Releases
October 14, 2019
Games Press
View All     RSS







If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 

Fast File Loading (Pt. 1)


April 19, 2007 Article Start Page 1 of 2 Next
 

Reading data efficiently from Hard Disk and DVD units is vital for video games and one of the more important problems to solve in the next generation of games. While we are getting 20x performance in processing power and memory size, we are only getting 4x performance improvement in data devices (dvd for consoles).

I describe in this article how to efficiently read raw data from disk (hdd, dvd) oriented towards streaming files in a realtime application (although the concepts are useful in other areas). The platform used is Win32, but all the topics covered could be easily ported to other platforms.

I have included a project for Visual Studio 2005 with all the code described here and all the framework to test the different techniques. You can download it here. The machine where I have done the tests is a 3.2GHz Pentium 4. The devices used for testing are:

  • A 7200rpm Hard Disk with an average read performance of 46.8 MB/s (measured using Hd Tach)

  • A DVD-Rom unit (using a DVD+RW media) with a peak performance of 2.40Mb (measured using Nero CD-DVD Speed)

Windows (and all the Operating Systems in general) uses part of the physical memory (not being used by processes) for file caching (you can view how much memory is being used for the File System Cache in the Task Manager). To avoid Windows caching my test files I implemented a function that flushes the cache reading big files before measuring times. The tests were executed 10 times, lasting each one several minutes.

So let’s start the travel… Objective: having a 100MB file in the CPU memory as fast as possible.

1. The Standard and Portable way

The first option is using portable code from the standard C library. All we know the benefits of portability. So we allocate a buffer and fread() the file.

FILE *fp = fopen(FileName, “rb”);
fread(&g_buffer[0], 1, FileSize, fp);
fclose(fp);

Stats

Min (MB/s)

Max (MB/s)

Average (MB/s)

HDD

47.847

48.828

48.527

DVD

2.381

2.386

2.383

2. The Win32 Native way

Trying to improve the native approach we go for the Win32 native file functions: CreateFile and ReadFile

HANDLE hFile = CreateFile(FileName, GENERIC_READ, 0, 0, OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN, 0);

DWORD dwNumberOfBytesRead = 0;
ReadFile(hFile, &g_buffer[0], FileSize, &dwNumberOfBytesRead, 0);

CloseHandle(hFile);

Stats

Min (MB/s)

Max (MB/s)

Average (MB/s)

HDD

47.483

48.852

48.497

DVD

2.383

2.390

2.386

Nearly the same performance. FILE_FLAG_SEQUENTIAL_SCAN is used to direct the Cache Manager to access the file sequentially. It is recommended to use it when reading large files with sequential access. I though that the FILE_FLAG_SEQUENTIAL_SCAN hint would give a better improvement than this but obviously the fread implementation in Win32 is doing a good job.

3. File Memory Mapping

Our next approach is trying memory mapped files where the system reads the data from disk on demand using the same mechanism that is used in Virtual Memory.

HANDLE hFile = CreateFile(FileName, GENERIC_READ, 0, 0, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL | FILE_FLAG_SEQUENTIAL_SCAN, 0);


HANDLE hFileMapping = CreateFileMapping(hFile, 0, PAGE_READONLY, 0, FileSize, 0);

int iPos = 0;
const unsigned int BlockSize = 128 * 1024;

while(iPos < FileSize)
{
int iLeft = FileSize - iPos;
int iBytesToRead = iLeft > BlockSize ? BlockSize: iLeft;

void *rawBuffer = MapViewOfFile(hFileMapping, FILE_MAP_READ, 0, iPos, iBytesToRead);
memcpy(&g_buffer[iPos], rawBuffer, iBytesToRead);
UnmapViewOfFile(rawBuffer);

iPos += iBytesToRead;
}

CloseHandle(hFileMapping);
CloseHandle(hFile);

Stats

Min (MB/s)

Max (MB/s)

Average (MB/s)

HDD

45.830

48.828

48.190

DVD

2.524

2.528

2.526

We are getting a significant improvement when reading from the DVD. Reading from Hard Disk is nearly the same performance.


Article Start Page 1 of 2 Next

Related Jobs

Disbelief
Disbelief — Cambridge, Massachusetts, United States
[10.11.19]

Senior Programmer, Cambridge, MA
University of Exeter
University of Exeter — Exeter, England, United Kingdom
[10.11.19]

Serious Games Developer
Square Enix Co., Ltd.
Square Enix Co., Ltd. — Tokyo, Japan
[10.10.19]

Experienced Game Developer
Juncture Media
Juncture Media — New York, New York, United States
[10.10.19]

Cloud-Based Solutions Programmer





Loading Comments

loader image