When developing games for consoles, satisfying
the console's memory requirements is one of the most challenging
tasks. It's a recurring problem: you put in a lot of effort to get
your memory requirements right, then a week later you must start
all over because of the changes in the game's content. Having a
tool that provides you the correct information quickly would be
invaluable. Not having found an off-the-shelf tool that meets my
needs, I set about creating my own. This article, the first of two
parts, describes the cross-platform tool I created to support our
Xbox and PS2 development.
The solution described here is not about monitoring
memory performance such as cache misses or page misses--it instead
focuses on three main aspects of memory usage:
- The amount of memory your application uses
(and by what code).
- Displaying the memory layout, to visualize
- Discovering memory leaks in the application,
and what caused them.
I do not know of any third-party tools for consoles
that monitor these very basic issues, and I find this to be rather
odd, because memory and performance issues are frequently on the
top of my to-do-list. Microsoft has taken a nice first step with
the Xbox development tools. XbMemdump is able to display the layout
of physical pages, but it is very basic: just a command-line tool
that outputs ASCII characters. There is a tool from Metrowerks that
covers these memory issues—CodeTEST—but it is not available
for game consoles [REF7]. Another tool that covers memory issues
is Boundschecker [REF8]. It finds memory leaks and as from version
7.1, it also has a memory and resource viewer. Again, this product
is not available for consoles.
I will show you how we built a tool, called
MemAnalyze, which monitors all three of the above memory issues
for Xbox and PS2. (Supporting the Gamecube is not covered, simply
because Playlogic does not develop for the platform.) After reading
this article, you should have enough ideas on how to extend the
tool for other platforms.
This article provides an overview of the tool
and how to make a memory snapshot of the game. In part two of the
article (which will be published on Gamasutra this Friday), I'll
show you how to interpret the data.
We will run the game, and at the press of a
button, have the game output a file that holds the current memory
status--the addresses and sizes of all the blocks currently allocated
in memory. The file will be stored on the console's hard disk, if
present. Otherwise, it will be stored on the PC's hard disk, which
means we can only output the file if we are running the game remotely
from a PC. This article does not cover any alternative ways to output
the file—it simply describes how to collect the right contents
for the file. Saving it to an appropriate location is up to you.
Besides the allocation's block information,
we will also provide a callstack for each allocated block, using
real-time callstack tracing. Real-time callstack tracing should
be possible on each platform. Why? Each function always needs to
return to the previous, so the return address must be stored somewhere
or somehow. We just need to figure out how each platform retrieves
If you have written your own allocation or heap
manager, gathering the correct information for the memory dump will
be an easier task. You probably have most of the information at
hand. For the tool, we need the following data:
- The address of each allocated block
- The allocation size
- The callstack per allocation, which is an
array of function addresses (not quite, but we'll get to that
Our tool will read the memory dump offline,
on a PC, and read symbol information from a map file or a program
database. The symbol information is then used to convert the function
addresses to function names. The tool will implement several views
of this data.
Where do we draw the line between platform dependence
and platform independence? That is mostly up to you. The platform
specific information includes:
- Heap information, as stored on the console
- Symbol information, in the form of a map
file or program database.
- Image location information, if needed.
You could let the game walk the heap, process
it and dump it to file in a platform-independent data structure.
You could also let the game itself parse its own symbols and immediately
replace the addresses in the memory dump with function names. This
scheme would output a single platform-independent file, and you
could make MemAnalyze completely platform-independent. While this
sounds great, there are a few disadvantages to this approach:
- Adding a function name to each dumped callstack
function creates a lot of overhead because the function names
will be duplicated numerous times.
- If the function addresses are replaced by
names, we have to convert the names to a unique value such as
a CRC32 in order to process (compare, collapse) the data.
- We are limited to the console's libraries
for parsing symbol information. For some platforms, this might
turn out to be a problem. If we want to parse our symbols from
a program database, there is a good chance we will need to write
our own PDB parser, which is quite complex and hard to maintain,
in terms of version changes.
- We need to load symbol information. This
data will also be displayed in our memory analysis. We can partially
work around this problem by reloading symbol information on each
We chose, instead, to output a platform-dependent
file from the game, excluding function names. For the PS2, we even
dump the entire heap to a file, then walk the heap completely offline.
Doing so, we can even dump the PS2 memory to file if a critical
assertion occurs, and do some postmortem debugging in MemAnalyze.
This also has the benefit of being able to display and compare the
memory's contents. Obviously, this shifts some of the platform-dependent
code to the tool.
The tool will include two platform-dependent
pieces of code:
- Reading of the platform-specific memory dump,
and converting it to an internal, platform-independent data structure.
- Reading of platform specific symbols, and
converting it to an internal, platform-independent data structure.
From this point on, everything should be multiplatform.
In the end we will have three different views
of the data. In terms of graphical views, I only implemented two:
One for displaying the layout of the memory and one that shows how
much memory each function has allocated. In MemAnalyze, we
can open multiple memory dumps in multiple windows. The third view
is simply a dialog that lists memory leaks by comparing multiple
A view that I have not yet implemented is a
Hierarchy view. It will display a hierarchy of the functions that
allocated memory. Using this view, we can have more of an overview
on the memory usage and zoom in and out on allocation hotspots.
More information on this will be covered in Part two.
Memory layout view
This view shows blocks of memory as they are
physically present on the console. It's a Microsoft Defrag-like
style of displaying. Moving the mouse cursor over a block causes
a tooltip to appear that shows the complete callstack of the function
that allocated it. This is very convenient for the PS2, where memory
fragmentation is a big issue. You will mostly be searching for scattered
small blocks that clutter your memory.
Unfortunately, this view is not that useful
for Xbox games. The Xbox uses virtual memory addressing and this
solves a lot of the heap fragmentation issues within the VMM. The
VMM can split large virtual allocations into separate, non-contiguous
4KB physical pages. The addresses we use in our programs are virtual
addresses and may be mapped onto multiple physical pages. We can
monitor the virtual addresses, but fragmentation in the virtual
address space is not really an issue, as we can map our 64MB onto
a 4GB address space.
I don't know if it is possible to track the
real physical pages on each virtual heap allocation. Maybe then
we could really show the physical mapping of our virtual allocations.
But I am not even sure if this would prove to be useful information.
Physical allocations, on the other hand, might
be useful to monitor in the tool.
This view shows a series of bars, one for each
function that allocated memory. Again, if you move over a bar, a
tooltip will display the name of the function, along with the exact
size of the allocation and the number of allocations.
We can sort the functions in several interesting
- The total size allocated to each.
- The number of allocations by each.
- On the function name.
Memory leaks view
The memory leaks view will compare two dumps
and display the differences in a dialog box, as seen in Figure 3.