|
If you did
a double-take when reading the title of this piece, never fear -- it's
not a typo. "Benchmarketing" is a term that was long used as a term to
describe the behavior of some graphics cards vendors who would tout their
benchmarks as always being the faster than their competition. But
that's not what I mean here.
I'm using
the term as it was coined by my colleague, Andy Fischer, formerly of Jon
Peddie Associates and now at Metabyte. He suggested that game developers
could promote their games by building into their titles the capability
to do performance tests. In the past, many game programmers have been
loathe to do this. After all, if flight sim x runs at 40fps on
a Pentium II 400MHz machine and flight sim y runs at 25fps, then
users might feel that flight sim y is somehow inferior. (Or maybe
it's just that the programmers might end up with an inferiority complex.)
Then came
Quake, and after that, Quake II. id software built into
their titles the capability to test performance, but unlike a lot of other
companies, id never removed the performance test capability. Soon you
began to see time demo results everywhere on the Internet, in print publications
and even on the sides of graphics card boxes. This certainly made the
Quake series of games highly visible. Even today, Quake II
is still used as a performance metric, even though the game is well over
a year old.
Synthetic
versus Applications Tests
I do a lot
of performance testing when reviewing products, be they systems, graphics
cards, mass storage or even audio cards. At Computer Gaming World
and Gamespot, the tools are both synthetic and applications-based.
A synthetic
benchmark tries to create a "typical" workload, but also has the goal
of increased granularity. By that, I mean that a synthetic benchmark allows
you to examine vertically as well as horizontally. For example, 3D Winbench
99 allows you to enable or disable specific Direct3D features in order
to see the effect of trilinear filtering. A good synthetic test also allows
you to remove the effect of externalities. It's well known that at the
refresh rate of your monitor can have an odd harmonic effect on the frame
rate of a 3D title that's double-buffered. We're always very suspicious
when we see a game peg at 42.5FPS when the refresh rate is 85Hz, for example.
3D Winbench allows you to either triple-buffer (the default) or render
to the front buffer only.
Another
type of synthetic benchmark is 3D Mark 99 (www.3dmark.com).
3D Mark has the benefit of using a real game engine, the one that will
be used in Max Payne. It also has a fair amount of vertical granularity,
such as the ability to set specific texture sizes. However, it's still
based on a single game engine.
Synthetic
benchmarks have their place, but it's a truism that the best performance
tests are real applications. It is also true that different applications
will behave and perform differently. Just because graphics card y
runs great in first-person shooters doesn't necessarily mean they'll run
as well in a flight simulator or sports title.
|