|
 Features
Streaming for Next Generation Games
 File Archive System
Console File Systems
The file systems used on consoles are rather simple, which has
some implications. For example, simply opening and closing files
takes a lot more time than you first might realize. When there
are too many files in a folder, we need extra seeks to read file
table data. There are hard limits on the number of files in a folder
and the length of filenames. We want to allow artists and level
designers to organize their data in a logical way, and we want
to be able to quickly find data during development. However, even
changing the folder being read from will cause an implicit seek.
Checking if
a file exists or a file's size may also cause an implicit seek.
Archives
Our solution to the overhead of handling files is to have a simple,
platform independent game file system. All files are combined
into several large archive files, and these files are kept open
at all
time. When the archive files are created, we hash filenames and
store the hash. The archive file's table of contents is always
kept in memory and contains the hashed filenames, the archive file
handle,
offsets of the files in the archive and the sizes of the files
in the archive.
Since we have all archive files open at all time,
we never need to use the expensive open/close system calls. As
we always have the table of contents in memory, we can quickly
return the size of every file. As all file request are simply
seeks inside
the already opened archive files, we do not cause extra seeks
when we change to read from another directory. Hashing filenames
alone
does require that all filenames must be unique, regardless of
where they are stored - something that needs to be verified by
the content
pipeline. We can use this to our advantage when we need to share
files for different models.
struct SArchiveToc
{
HANDLE ArchiveFile;
unsigned int offset;
unsigned int size;
};
std::map<unsigned int, SArchiveToc> TableOfContents;
Correct data alignment inside the archives is important to minimize reads. Each file must be aligned to a start on a sector, typically 2048 bytes on a DVD. If files were not aligned the system would in fact have to read extra sectors when reading files spanning sector boundaries. The drawback is that we waste space on the DVD.
Content Pipeline And Tools
In a production system, you do not want to be forced to build
new archive files each time your content changes. For development,
you need to be able to read files as separate files. The archive
system is for final builds and archives should be created by the
nightly build system. Another implication of the archive system
is that you can no longer use DVD emulator logs directly, as you
no longer know the seek/read time of the actual files only for "stuff" inside
one of your archives. You need to write a tool to convert the emulator
log files to a readable format, or add code to log streaming performance
yourself.
Streaming system
Goals And System Overview
A good streaming system should spend as much time as possible
to read data, as little as possible seeking, and never ever be
idle. It has to read data while at the same time feeding data into
the game. Feeding the game with data should finish before new data
is read.
Our solution is to have one reader thread that fire callbacks
when data is read. We use two buffers to be able to load
new data at the same time as we process the recently loaded
data.
Double Buffers
We use two large aligned static buffers of equal size. The size
is a multiple of the sector size, 2048 bytes. The size of the static
buffers sets the size limit of individual resources being read,
for example how large texture files can be when stream.
It is still possible to get around the limitation by having the
game
issue
several read requests and then combine the data chunks. By using
static buffers, we avoid memory fragmentation in the stream system,
as we never need to do dynamic memory allocation.
|
|
|
|
|
Previous