Gamasutra: The Art & Business of Making Gamesspacer
Measuring Responsiveness in Video Games
View All     RSS
August 27, 2014
arrowPress Releases
August 27, 2014
PR Newswire
View All





If you enjoy reading this site, you might also want to check out these UBM Tech sites:


 
Measuring Responsiveness in Video Games

July 16, 2008 Article Start Previous Page 3 of 3
 

Other Results

I tested various other games, with various results. I'll list the results in 60ths of a second, and the results have been adjusted -2 to account for the plasma lag. All games are on the PS3 unless noted otherwise, and I've included the PS3 system menus and GTAIV for reference.

Games that run at 60 fps:

  • PS3 System menus: 3/60ths
  • Guitar Hero III (Xbox 360): 3/60th
  • Ridge Racer 7: 4/60ths
  • Virtua Tennis 3: 4/60ths
  • Ninja Gaiden Sigma: 4/60ths
  • PixelJunk Racers: 4/60ths

Games that run at 30 fps:

  • Genji: Days of the Blade: 6/60ths
  • Tony Hawk's Proving Ground: 8/60ths
  • BlackSite: Area 51: 8/60ths
  • Halo 3 (Xbox 360) : 8-10/60ths
  • Skate: 10/60ths
  • GTAIV: 10/60ths
  • Harry Potter and the Order of the Phoenix: 10-14/60ths
  • Heavenly Sword: 7-18/60ths

The first question that arises is: why are there no PS3 games with a response time of 3/60ths? The PS3 UI runs at 3/60ths response time, so it's quite possible. Why do all the 60fps games run at 4/60ths?

Still, that's still pretty good, and those games all feel quite responsive. Guitar Hero III on the Xbox 360 manages a very impressive 3/60ths - very important for that type of game, especially when it's actually 5/60ths on many TVs.

Next, why so much variance between the 30fps games - ranging from 6/60ths to 18/60ths?

It's worth noting that while EA's Skate seems like it would be a bit sluggish at 10/60ths, it actually feels quite responsive, in large part due to its use of stick gestures for input, as the movement starts before the gesture is complete, yet the player mentally synchronizes it with the end of the gesture, and so it actually seems very responsive.

Then what about the games where the lag varies? Harry Potter is 10-14 which is bad enough, but Heavenly Sword is an astonishing 7-18. It takes 7/60ths to start an attack, but 18/60ths to start to turn around (See movie: mvi_4263w). Clearly something is wrong there. I would consider that a bug.

It's sad that the programmer made the effort that allowed for a 7/60th response time, but then someone else messed up down the line, making the turn take nearly a third of a second. Halo 3 is another example, with the shooting and moving being 8, but the jumping being 10.

Particularly interesting here in the 30fps category is Genji: Days of the Blade. This is a very similar game to Ninja Gaiden, and yet Genji runs at 30, while Ninja Gaiden runs at 60.

However, the fact that Genji runs at 30 is barely noticeable, and does not detract from the game at all. In part this is because of the way the camera moves smoothly through the scene with very few rapid pans. It's also because of the low contrast graphics and motion blur. But it's also because Genji's response rate is 6/60ths, very similar to Ninja Gaiden's 4/60ths.

Ninja Gaiden is a faster paced game than Genji, with the main character jumping around rapidly, and the camera tightly following him - so the graphical benefits of 60fps are more apparent. The low response time feels good as well.

However, since Genji also has a low response time, it would benefit very little from running at 60 fps. As it runs at 30fps, this gives the designers the opportunity to put more graphics, enemies and special effects on screen, and reduces the pressure on programmers and artists to constantly strive to maintain 60, which can be a difficult factor in development.

Conclusions and Suggestions

Games that run at 60fps all seem to have a response time to 4/60ths, and while 3/60ths is possible, 4/60ths is a very good response time.

Some games running at 30fps have a response time of 8/60ths or 10/60ths (and some peak even higher). Genji shows us that a response time of 6/60ths is possible while running at 30 fps. 10/60ths can be too long, especially when combined with the processing delays in flat panel TVs which can push it up to 12/60ths. 1/5th of a second (200ms) is too long to wait for a gun to fire, and introduces annoying sluggishness when moving around or steering a car.

Some games have an inconsistent response time. Heavenly Sword varies from 7 to 18. If the system is capable of 7, then all moves should start in 7. Developers should verify ALL their response times, as other factors, such as animation, might be creating lag in specific places.

I suggest that game developers use this simple technique to measure the response time in their games, at least to verify that their assumptions are correct. If they are running at 60fps, then they should not be above 4/60ths. If they are running at 30fps, then they strive to duplicate Genji's responsiveness of 6/60ths, and certainly not slip below 8/60ths. I suggest that they keep in mind that flat panel TVs (which are probably a majority in gamers households, and certainly in game reviewers' households) add an additional 2 frames of lag, which makes it EVEN MORE important to keep programmed lag to a minimum.

I also suggest that game reviewers begin to use this technique to measure lag, and to include the measurement of lag in their reviews. While the subjective views of the reviewer are important and valuable, an objective measurement of response time would be a very useful additional piece of information for the person considering buying the game.

Placing this information in a game review would encourage developers to produce more responsive games, which benefits everyone.


Article Start Previous Page 3 of 3

Related Jobs

Cloud Imperium Games
Cloud Imperium Games — Austin, Texas, United States
[08.27.14]

Lead Network Engineer
Cloud Imperium Games
Cloud Imperium Games — Santa Monica, California, United States
[08.27.14]

Animation Programmer
Cloud Imperium Games
Cloud Imperium Games — Austin, Texas, United States
[08.27.14]

Lead Software Engineer
Cloud Imperium Games
Cloud Imperium Games — Austin, Texas, United States
[08.27.14]

Server/Backend Programmer






Comments


Martin Tremblay
profile image
Very good article!



I would love to see this done with MMORPG interfaces. Everyone claims that WoW's interface is the best, and all other games are sluggish compared. I wonder if WoW is just less laggy! I`m sure the stats on that could be pretty interesting :)

Bob McIntyre
profile image
Since when do game developers "have to choose" between 30 and 60 FPS? Forget about PC games, for the moment...almost no game runs solidly at a fixed framerate even on consoles, unless the console is grossly overpowered for the game and actually has idle time between frames.



Also, Martin, I don't think it will make WoW less laggy because the primary bottleneck for lag in an online game, especially an MMO, is almost always the network connection. Even playing Warcraft or Starcraft, you can notice a slight lag difference between single-player and BNet multiplayer.

Martin Tremblay
profile image
Yes, I agree Richard (Thank you for the comment! :)), that it being online creates its own set of lag. What I was trying to get at is compared to other MMO's, WoW's interface responds better, and seems a lot less sluggish. So if you play WoW and EQ2, both at 100ms Latency, and WoW's interface does respond faster, then it could be that WoW just is more responsive in the way some console games are more responsive. Since both games are online, with the same latency, online shouldn't be a factor.



Maybe I'm missing something though :) I'm not an expert or anything haha.

Jeremy Alessi
profile image
Another solid article that every game programmer should read. Thank you!

Daniel Lam
profile image
That's a pretty good idea! Though I have to point out that using the PS3 XMB may not be the most accurate calibration tool, unless it's official word that the XMB is indeed instantaneous.

In addition, TVs lag differently depending on how much processing is required at different resolutions too. Be sure to set your XMB to the resolution you are testing with (typically 720p)



The best way to measure the response time of your TV is to hook up a PC with DVI/HDMI-outputs in mirror mode, and run a video that has a countdown timer. Use the camera to take snapshots and you'll see the exact number differences between the PC outputting and the TV outputting.

Source : http://hdtvlag.googlepages.com/ourtest

James Williams
profile image
Hi, I have worked on both Goldeneye and Perfect dark franchise plus I'm an avid quake player who is used to playing his FPS games at 120 frames per second (and 120 hz on the monitor) so I was interested in this article as I feel the validity of your claims are missing from a lot of console games and are partly why I not longer work with game consoles.



I think the problem is a lot of devs will always sacrifice frame-rate for better visuals because those gains are a lot more obvious to publishers and consumers, it's also fair to say that some games can get away with being more sluggish in terms of response time.



FPS and Driving games (this includes tony hawk:P) where the life or death of a player is heavily focused on the performance of the game so yeah its vital there to keep it tight, however other games have mechanics where that level of continuity is not required and I guess this is where the games start to feel like a bag of nails.



I would like to see devs take a note from the PC world and put in more video options, so for example if your game uses real-time this and real-time that then at least put in the options so "real-gamers" can sacrifice graphics for a smoother gameplay expereince.



you can even make it cheat based so such options are only enabled afte the game has been finished, so you can still have your great looking game upon release:)



Lastly I would say that however you prsent your findings, anyone worth their salt in games should know if a game feels like crap and if they still release agame knowing this, then maybe its best they carry on and hopefully disappear :)



I find it scary that in this gay and age peple are aiming for 30, it should be 60 FPS and if you're dropping frames, then you should be rethinking your level deisgns and art in that area because unless you have crap game engine then it should not be hard for most people to obtain a crisp expereience.



lastly, I don't know how long this will take to filter into the console market, but in the pcmarket it is possible to increase the polling rate of usb devices.



They run at 125hs as standard, but you can overclock to 250, 750, 1000hz, which make the general movement feel a lot smoother.

Steve Huckle
profile image
How about measuring a game like Guitar Hero with the lag of the Audio? That would ignore frame boundaries, and give you a very accurate measurement of the lag. Plus it's a music game, so it might be important :0



Same method, just record the button press and the audio output of the game, then look at the difference in the attack portions of the sound.

Aaron Claussen
profile image
to James Williams,



While I see your point on frames per second being important in a fast twitch game, such as Quake, because they allow for a faster response time you have to take something else into account. YOU are able to run games at 120 fps but many gamers are not. This could, arguably, give you an edge against other players. While this may help you get kills it also kills the spirit of competativness. By keeping games capped at a certain target fps it helps to keep the playing field level by not allowing any one player to leverage their hardware in such a way as to give themselves a better response time. Granted there are other factors, especially things screen resolution, but from a purely "twitch movement" perspective it helps to keep the playing field level.

John Mawhorter
profile image
This is exactly the kind of hands-on article Gamasutra should run more often. You should do tests of a bunch of game genres comparing responsiveness to critical reception.

Kenneth Baird
profile image
On the pc, sometimes bus transactions can delay USB mouse information (not sure if it affects ps2 mice). I've seen it in particular with sli / crossfire rigs where buffers have to be copied across the link (not properly clearing an offscreen surface for instance), or if excessive texture uploads happen in a frame.



Sometimes your framerate will be fine but there will be a good half second lag in mouse movement to actual movement on the screen due to the above.

jessica winslet
profile image
Slowly but surely the top LCD and Plasma Televisions are getting to grips with combining TV PIctures and gaming technology. Something that should have been straight forward to do for some reason always got held back. Interesting article

http://www.site-de-bingo.fr

Wa Luigi
profile image
Useful information but I completely disagree that reviewers should start using these methods to measure lag and include this information in the review. That's like including the amount of film grain in a film review. While this data is very useful for the people creating, it is not for the player/viewer. For better or worse, lag is sometimes used for gameplay feel (i'm guessing this is the reason the turn in Heavenly Sword is slower than the attack) and it is my opinion that instant response is not the answer for every application. So this sort of "under the hood" data should stay with the developers... please GOD... don't let game reviewers bust out their Lag-O-Meters.

Mick West
profile image
Wa Luigi, while there are a few times when a developer may deliberately introduce a delay into a move, the majority of the time the player (and the developer) wants the response to be as quick as possible. Take for example firing a gun - it's very difficult to see any benefit from adding a 1/5th of a second delay between pulling the trigger and the gun firing (as we have in GTA-IV). The same applies to movements such as simply walking around.



And where a delay is inherent in the move, such as a power attack with a sword, the player still wants SOME response when they press the button, at the very least so they feel the game is responding to them - such as starting an animation.



I think you would be VERY hard pressed to find a game where the developers had DELIBERATELY added a short period of total unresponsiveness after particular button presses. In just about every case the lag is inadvertent, and perhaps unnoticed by the developers. Bringing lag forward as a measurable issue will help ensure developers pay attention to lag, and minimize it where possible.



Even if you accept that in a game such as Heavenly Sword there are particular moves that have additional lag, there is still the issue of the baseline lag - the shortest lag with which the game responds. This lag is often due to underlying technical issues I outlined in another article, and is generally something that is detrimental to the gameplay experience, and something the developers would prefer did not happen.



http://cowboyprogramming.com/2008/05/27/programming-responsivenes
s/


none
 
Comment: