Measuring Responsiveness in Video Games
July 16, 2008 Page 2 of 3
Calibration and Measuring
Plasma and LCD televisions introduce lag of their own, as they frequently do some additional processing to a picture before they display it. This can be as much as a few 60ths of a second, which can be quite significant. Ideally you would do your testing on a CRT Television, which should have no additional lag (the cheaper the better). Your flat panel TV might also have a "game mode" which will reduce (but not eliminate) the lag.
To find how much lag your TV adds, you need to compare an easily measured response with the same response on a CRT TV, or with a known value. To simplify things for you I've measured the response time of the PS3 system menus, which come out at a very solid 3/60ths on a CRT, and 5/60th on the plasma TV shown above.
Hence I need to subtract 2/60th of a frame to get an accurate measure of responsiveness when using the plasma for measurements. Make the appropriate adjustment for your TV.
This additional lag, however, should not be ignored, and it should be taken into account when developing the game. Many people will run your game on a TV that introduces additional lag. Steps should be taken to minimize the negative effects this unavoidable lag can have.
To measure the response time (lag) the simplest tool I found has been the free Quicktime viewer. With Quicktime installed, right click on the video file from your camera, and then quick on "Open with Quicktime". You can then use the arrow keys to frame advance 1/60th at a time.
For a particular response time, navigate the video to the first frame that shows the button fully pressed, and then count additional frames from there until you see the first indication of a response on screen. The number of frames you counted is your response time in 1/60ths of a second. You can convert to milliseconds by dividing by 60 and multiplying by 1000 (12/60ths is 200 ms)
Here are the frames for the PS3 system menu on the CRT TV:
In this first image (above), the thumb is descending to the "down" button on the D-Pad. It's important to press the buttons with the thumb starting off the pad and use a rapid hitting motion, so you can see the precise frame when the button is fully pressed. Note the slight motion blur on the thumb - this is also a useful indicator of if it is still moving or not.
Now the thumb is clearly stopped and is pressing the button. Since this is the first frame where we see the button pressed, then we start counting from here, at zero (if the response were visible here, you'd have zero lag, impossible with current systems).
Now (above) we are on the first frame of actual lag, note the same line on the menu is still highlighted.
Second frame of lag, still no change.
Third frame. The menu begins to move. This means that the PS3 system menu has a response time of 3/60ths of a second, or 50 ms (milliseconds). This is VERY GOOD, and is the fastest you can realistically expect from the PS3.
While it's theoretically possible to go to 2/60ths, this has severe performance problems. I don't think 1/60th is possible on the PS3. So - all games running at 60 fps should shoot for 3/60ths as an ideal response time. This also means games running at 30 fps should shoot for 6/60ths. That's in a perfect world, and as we shall see, actual results vary.
Results for Grand Theft Auto IV
The first thing I tried was shooting a gun in Grand Theft Auto IV for the PS3 on a plasma television. This is done with the right trigger button (R2), which feels (slightly) like pulling the trigger of a real gun.
In real life, there is zero delay between the trigger reaching the end of its travel, and the bullet leaving the gun. In GTAIV, this delay is somewhat longer. Here's the actual movie, so you can frame advance it yourself, and here are the relevant frames:
Note on the first frame (-1) , the finger is still moving, we start counting on the next frame (0) when the finger is fully on the button and the button is fully depressed. We then count until the first response, which comes at frame 12.
This indicates a response time of 12/60ths. Since it's measured on the plasma TV, we adjust this to 10/60ths. This gives us a raw response time for GTAIV of 166 ms (200 ms on flat panel TVs).
I repeated this test for turning and jumping, and got the same results. This is rather a long response time, and correlates with people reporting the game being sluggish and unresponsive. The delay in firing the gun after pulling the trigger is quite noticeable.
Page 2 of 3