Game Gauge 99 Specification

About the Game Gauge

Artificial benchmarks, as they exist today, are not sufficient to indicate accurately the performance of a complex 3D system such as Incoming or Quake. Consumers are tired of meaningless 3D benchmark scores that don’t correlate to real world performance gains on a given 3D card. Rather than artificially generate 3D benchmark tests, doesn’t it makes more sense to have the game software report the real-world performance based on the actual game data?

The game gauge is just such a 3D benchmark. It’s based on the real world performance of games that we know and love. The game’s score isn’t artificial – it’s what you feel when you are playing – the frame rate! Each game contributes its individual "Average Frames Per Second" to an overall 3D card score, yielding what we call the game gauge.

Game Gauge Score

Each game contributes three numbers to the game gauge score. The first number is the Average FPS number after a fixed length demo loop has been run at the specified game option settings. The second number is the MIN FPS encountered. The third number is the MAX FPS encountered.

For example, if the following games were tested under the game gauge :

Game Title Average FPS Min. FPS Max FPS

Quake 1 20 5 30

Quake 2 20 5 30

Turok 20 5 30

Hexen 2 20 5 30

Forsaken 20 5 30

Incoming 20 5 30

Average FPS Score = 120

Minimum FPS Score = 30

Maximum FPS Score = 180

Notice that the Average FPS score is the sum of all the average frame rates. Scoring high on one game and low on another will not yield an overall higher score in the average FPS number. This promotes creating better and better 3D products, and not just tuning a 3D card to a single game.

The MIN FPS score and Max FPS score are new additions to the game gauge for 1999. MIN FPS is the lowest number of frames per second recorded during the duration of the demo. Max FPS is the highest number of frames per second recorded during the duration of the demo. The addition of these two numbers allow us to better examine the consistency of the games performance during the demo. The MIN FPS number is calculated by adding up the MIN FPS numbers from all the games. The Max FPS number is calculated by adding up the Max FPS numbers from all the games.

Method of Calculating Average Frame Rate

While many methods exist to calculate the average frame rate of a demo loop, we require all game gauge participants to follow our guidelines. By having a single method, we guarantee all games will yield scores evenly across all hardware types.

The required method of calculating your game’s average frame rate is as follows:

NumFrames / Time = Average FPS

Where:

NumFrames = Number of frames in demo loop.

Time = Number of seconds the demo ran for. This should be carried out to a precision of at least 1/10 of a second.

Average FPS = Average number of frames per second in the demo loop.

Games should allow for the Average FPS to be numerically in the many hundreds or even thousands of frames per second range. If a quantum leap in technology is created, we don’t want the scores to be artificially low. Make certain that you don’t have a frame rate limiter in your code.

Method of Calculating the Min and Max Frame Rate

It’s very important that each game calculate the min/max frame rate exactly the same way. The method we require is as follows :

startup : (done at startup)

Initialize START_TIME to the current time in milliseconds.

Initialize FRAMES_ONE_SECOND counter to 0

Initialize MIN_ONE_SECOND counter to 2147483647 (Something really big like 0x7fffffff)

Initialize MAX_ONE_SECOND counter to 0

Per game loop: (done every game loop)

After each page flip in the game, add 1 to the FRAMES_ONE_SECOND counter.

Get the time in milliseconds and subtract the START_TIME. If the result is over 1,000 milliseconds, go to minmaxcheck

Minmaxcheck: (only done once every second)

If(FRAMES_ONE_SECOND

MIN_ONE_SECOND=FRAMES_ONE_SECOND

If(FRAMES_ONE_SECOND>MAX_ONE_SECOND)

MAX_ONE_SECOND=FRAMES_ONE_SECOND

FRAMES_ONE_SECOND=0

START_TIME = current time in milliseconds.

In essence, every 1,000 milliseconds we compare the number of frames that have passed against our min and max numbers and update accordingly. Like Average FPS, games should allow for the min and max numbers to be numerically in the many hundreds or even thousands of frames per second range. Once again, if a quantum leap in technology is created, we don’t want the scores to be artificially low. Make certain that you don’t have a frame rate limiter in your code.

Fixed Duration Demo Loop

Each game gauge game is required to have a demo loop of a fixed number of frames. This demo loop should be able to be run from a command option , or a single purpose .EXE. No user control or outside intervention should be allowed. At the conclusion of the demo loop, the Average FPS score, MAX FPS, and MAX FPS should be calculated and output to FPS.TXT.

Automation of testing with FPS.TXT

After the game or demo has run, we require an FPS.TXT file to be output to the root directory of the .EXE. This simple text file allows for automation when combined with a front end, and is a requirement for all games to be part of the game gauge.

Each game in the game gauge should allow the game to be run from using only command line options. To allow automation, we must be able to launch the game, have it run the tests, output FPS.TXT, and exit gracefully back to Windows 95/DOS. The "Game Gauge control software" will run all the games, collect all the FPS.TXT files, and output the final scores using the collected data.

The format of FPS.TXT is very simple. It consists of three lines of text. The first entry is the average FPS output in decimal with a max precision of 1/100 of a second followed by a space and the name / version of the game. The second and third lines are the min and max FPS numbers with the text "min" or "max" following the correct number.

A game’s FPS.TXT should look exactly like this :

287.04 FooBar v1.01

5 Min

100 Max

We prefer that games keep this order to allow for future enhancements to the FPS.TXT file.

The FPS.TXT should be output in the same directory as the EXE file is located.

Optional Graphing Data

We also have an optional set of data that we encourage be supported to enable a games performance be graphed. Every second of the game, a number of frames are displayed to the monitor. The number of frames displayed in a second is a number easily captured in two bytes, and can be saved into a small amount of memory for performance graphing. One minute of game playback is the same as 120 bytes (60 seconds * two bytes each). A three minute demo needs only 360 bytes of data to show the game’s performance over time. Once the game has completed running, this data should be appended to the FPS.TXT file as the following example illustrates.

A games FPS.TXT with the Optional Graphing Data should look exactly like this :

287.04 FooBar v1.01

5 Min

100 Max

20 Second 1

10 Second 2

5 Second 3

...

34 Second 305

The automation program will import the FPS.TXT file and graph the data over time. By comparing graphs from two different configurations we can examine the relative performance differences during the game playback.

Command Line Options

The automation of the game testing requires the use of command line options to select what would normally be decided using menus in the game. The following options are required for the testing to work across all manufacturers’ hardware, and must be supported.

• Selection of the graphics API to be used: D3D, OpenGL, or Glide as determined by your game.
• Board Selection for 3Dfx hardware: primary/secondary support.
• Resolution: All resolutions on all APIs up to 1600x1200 (or the highest your game supports)
• Double/triple-buffering: If triple-buffering is supported by your game, this switch would enable/disable.

Method of Testing Each Game

Each game that comprises the game gauge should have a detailed method for generating its game score. Preferably, the default option settings remain consistent across all 3D hardware cards, allowing for minimal errors in the process of testing games.

V-Sync. All game tests should be performed with vertical sync turned off and a refresh rate of 60Hz. If a 3D card doesn’t support disabling Vsync, then the game scores should be reported as returned with Vsync on. It’s up to the individual card manufacturer to create a publicly available driver that allows disabling Vsync and setting the refresh rate of the monitor. Higher refresh rates impact the available bandwidth for 3D graphics, so it was decided that 60Hz would be the test standard.

Audio. All game tests should be performed with audio enabled. We feel testing with audio enabled better represents the way consumers play games.