If you did a double-take when reading the title of this piece, never fear -- it's not a typo. "Benchmarketing" is a term that was long used as a term to describe the behavior of some graphics cards vendors who would tout their benchmarks as always being the faster than their competition. But that's not what I mean here.

I'm using the term as it was coined by my colleague, Andy Fischer, formerly of Jon Peddie Associates and now at Metabyte. He suggested that game developers could promote their games by building into their titles the capability to do performance tests. In the past, many game programmers have been loathe to do this. After all, if flight sim x runs at 40fps on a Pentium II 400MHz machine and flight sim y runs at 25fps, then users might feel that flight sim y is somehow inferior. (Or maybe it's just that the programmers might end up with an inferiority complex.)

Then came Quake, and after that, Quake II. id software built into their titles the capability to test performance, but unlike a lot of other companies, id never removed the performance test capability. Soon you began to see time demo results everywhere on the Internet, in print publications and even on the sides of graphics card boxes. This certainly made the Quake series of games highly visible. Even today, Quake II is still used as a performance metric, even though the game is well over a year old.

Synthetic versus Applications Tests

I do a lot of performance testing when reviewing products, be they systems, graphics cards, mass storage or even audio cards. At Computer Gaming World and Gamespot, the tools are both synthetic and applications-based.

A synthetic benchmark tries to create a "typical" workload, but also has the goal of increased granularity. By that, I mean that a synthetic benchmark allows you to examine vertically as well as horizontally. For example, 3D Winbench 99 allows you to enable or disable specific Direct3D features in order to see the effect of trilinear filtering. A good synthetic test also allows you to remove the effect of externalities. It's well known that at the refresh rate of your monitor can have an odd harmonic effect on the frame rate of a 3D title that's double-buffered. We're always very suspicious when we see a game peg at 42.5FPS when the refresh rate is 85Hz, for example. 3D Winbench allows you to either triple-buffer (the default) or render to the front buffer only.

Another type of synthetic benchmark is 3D Mark 99 (www.3dmark.com). 3D Mark has the benefit of using a real game engine, the one that will be used in Max Payne. It also has a fair amount of vertical granularity, such as the ability to set specific texture sizes. However, it's still based on a single game engine.

Synthetic benchmarks have their place, but it's a truism that the best performance tests are real applications. It is also true that different applications will behave and perform differently. Just because graphics card y runs great in first-person shooters doesn't necessarily mean they'll run as well in a flight simulator or sports title.