When developing games for consoles, satisfying the console's memory requirements is one of the most challenging tasks. It's a recurring problem: you put in a lot of effort to get your memory requirements right, then a week later you must start all over because of the changes in the game's content. Having a tool that provides you the correct information quickly would be invaluable. Not having found an off-the-shelf tool that meets my needs, I set about creating my own. This article, the first of two parts, describes the cross-platform tool I created to support our Xbox and PS2 development.

The solution described here is not about monitoring memory performance such as cache misses or page misses--it instead focuses on three main aspects of memory usage:

• The amount of memory your application uses (and by what code).
• Displaying the memory layout, to visualize memory fragmentation.
• Discovering memory leaks in the application, and what caused them.

I do not know of any third-party tools for consoles that monitor these very basic issues, and I find this to be rather odd, because memory and performance issues are frequently on the top of my to-do-list. Microsoft has taken a nice first step with the Xbox development tools. XbMemdump is able to display the layout of physical pages, but it is very basic: just a command-line tool that outputs ASCII characters. There is a tool from Metrowerks that covers these memory issues—CodeTEST—but it is not available for game consoles [REF7]. Another tool that covers memory issues is Boundschecker [REF8]. It finds memory leaks and as from version 7.1, it also has a memory and resource viewer. Again, this product is not available for consoles.

I will show you how we built a tool, called MemAnalyze, which monitors all three of the above memory issues for Xbox and PS2. (Supporting the Gamecube is not covered, simply because Playlogic does not develop for the platform.) After reading this article, you should have enough ideas on how to extend the tool for other platforms.

This article provides an overview of the tool and how to make a memory snapshot of the game. In part two of the article (which will be published on Gamasutra this Friday), I'll show you how to interpret the data.

Overview

Theory

We will run the game, and at the press of a button, have the game output a file that holds the current memory status--the addresses and sizes of all the blocks currently allocated in memory. The file will be stored on the console's hard disk, if present. Otherwise, it will be stored on the PC's hard disk, which means we can only output the file if we are running the game remotely from a PC. This article does not cover any alternative ways to output the file—it simply describes how to collect the right contents for the file. Saving it to an appropriate location is up to you.

Besides the allocation's block information, we will also provide a callstack for each allocated block, using real-time callstack tracing. Real-time callstack tracing should be possible on each platform. Why? Each function always needs to return to the previous, so the return address must be stored somewhere or somehow. We just need to figure out how each platform retrieves that data.

If you have written your own allocation or heap manager, gathering the correct information for the memory dump will be an easier task. You probably have most of the information at hand. For the tool, we need the following data:

• The address of each allocated block
• The allocation size
• The callstack per allocation, which is an array of function addresses (not quite, but we'll get to that later).

Our tool will read the memory dump offline, on a PC, and read symbol information from a map file or a program database. The symbol information is then used to convert the function addresses to function names. The tool will implement several views of this data.

Where do we draw the line between platform dependence and platform independence? That is mostly up to you. The platform specific information includes:

• Heap information, as stored on the console in hardware.
• Symbol information, in the form of a map file or program database.
• Image location information, if needed.

You could let the game walk the heap, process it and dump it to file in a platform-independent data structure. You could also let the game itself parse its own symbols and immediately replace the addresses in the memory dump with function names. This scheme would output a single platform-independent file, and you could make MemAnalyze completely platform-independent. While this sounds great, there are a few disadvantages to this approach:

• Adding a function name to each dumped callstack function creates a lot of overhead because the function names will be duplicated numerous times.
• If the function addresses are replaced by names, we have to convert the names to a unique value such as a CRC32 in order to process (compare, collapse) the data.
• We are limited to the console's libraries for parsing symbol information. For some platforms, this might turn out to be a problem. If we want to parse our symbols from a program database, there is a good chance we will need to write our own PDB parser, which is quite complex and hard to maintain, in terms of version changes.
• We need to load symbol information. This data will also be displayed in our memory analysis. We can partially work around this problem by reloading symbol information on each memory dump.

We chose, instead, to output a platform-dependent file from the game, excluding function names. For the PS2, we even dump the entire heap to a file, then walk the heap completely offline. Doing so, we can even dump the PS2 memory to file if a critical assertion occurs, and do some postmortem debugging in MemAnalyze. This also has the benefit of being able to display and compare the memory's contents. Obviously, this shifts some of the platform-dependent code to the tool.

The tool will include two platform-dependent pieces of code:

• Reading of the platform-specific memory dump, and converting it to an internal, platform-independent data structure.
• Reading of platform specific symbols, and converting it to an internal, platform-independent data structure.

From this point on, everything should be multiplatform.

In the end we will have three different views of the data. In terms of graphical views, I only implemented two: One for displaying the layout of the memory and one that shows how much memory each function has allocated. In MemAnalyze, we can open multiple memory dumps in multiple windows. The third view is simply a dialog that lists memory leaks by comparing multiple memory dumps.

A view that I have not yet implemented is a Hierarchy view. It will display a hierarchy of the functions that allocated memory. Using this view, we can have more of an overview on the memory usage and zoom in and out on allocation hotspots. More information on this will be covered in Part two.

This view shows blocks of memory as they are physically present on the console. It's a Microsoft Defrag-like style of displaying. Moving the mouse cursor over a block causes a tooltip to appear that shows the complete callstack of the function that allocated it. This is very convenient for the PS2, where memory fragmentation is a big issue. You will mostly be searching for scattered small blocks that clutter your memory.

	Figure 1: The layout view. The top view displays a situation where RpGeometry causes fragmentation problems. In the bottom view we solved this problem. You can see that the small blocks are a lot less scattered here.

Unfortunately, this view is not that useful for Xbox games. The Xbox uses virtual memory addressing and this solves a lot of the heap fragmentation issues within the VMM. The VMM can split large virtual allocations into separate, non-contiguous 4KB physical pages. The addresses we use in our programs are virtual addresses and may be mapped onto multiple physical pages. We can monitor the virtual addresses, but fragmentation in the virtual address space is not really an issue, as we can map our 64MB onto a 4GB address space.

I don't know if it is possible to track the real physical pages on each virtual heap allocation. Maybe then we could really show the physical mapping of our virtual allocations. But I am not even sure if this would prove to be useful information.

Physical allocations, on the other hand, might be useful to monitor in the tool.

This view shows a series of bars, one for each function that allocated memory. Again, if you move over a bar, a tooltip will display the name of the function, along with the exact size of the allocation and the number of allocations.

We can sort the functions in several interesting ways:

• The total size allocated to each.
• The number of allocations by each.
• On the function name.

	Figure 2: The TopX view, sorted on total size allocated.

Memory leaks view

The memory leaks view will compare two dumps and display the differences in a dialog box, as seen in Figure 3.

	Figure 3: output of a memory compare, showing a memory leak.