Product Review of Physics Engines, Part One: The Stress Tests

Gama Network Presents:

Product Review of Physics Engines, Part One: The Stress Tests
By Jeff Lander and Chris Hecker
Gamasutra
September 13 , 2000
URL: http://www.gamasutra.com/features/20000913/lander_01.htm

As devoted readers of Game Developer, you are all aware of our belief (or hope!) that realistic physics can greatly improve the gameplay in interactive entertainment. You also know that creating a robust and flexible physical simulator is difficult work. Simulators are hard to design, challenging to code, and even more difficult to debug. These challenges cause many programmers and producers to look for external help for their physics development problems.

Lately we have seen the emergence of licensable "physics engines" from companies wishing to fill this need. We feel the market has matured to the point where it is time for us to take a hard look at these products to see how well they work.

We will be looking at three products that offer roughly the same level of technology and features: MathEngine's Dynamics Toolkit 2.0 and Collision Toolkit 1.0, the Havok GDK from Havok, and Ipion's Virtual Physics SDK. Our focus is on rigid body dynamics. We will not review cloth, particle, water, or other kinds of simulation, even though these packages support some of those features.

In the first of a two part series, we have created a series of tests that stress the capabilities of the simulations in difficult-to-solve situations where physical simulations typically break down. These tests give us a general feel for each engine's implementation of core features such as contact, constraints, and integration, as well as a working knowledge of each API. In the next part, we will take an in-depth look at each package with specific attention paid to how these packages will integrate into a large-scale game project.

Ground Rules & Disclaimers

First, it's important to point out that the stress tests are not exhaustive, nor directly representative of in-game situations. They are simulations of physical configurations specifically chosen to stress the engines in difficult numerical situations.

Our goal with these tests was to break the simulators. We are not necessarily expecting perfection (although it would be nice!). We chose this approach because users and artists have an annoying tendency to create physical configurations that break physics simulators, so it's better to find weaknesses before licensing than during final play-test.

Though the tests are difficult, they are not unrealistically complex. Artists could create configurations that correspond to these tests without knowing the configurations might be problematic. We did not tune the tests to exacerbate a discovered bug. Rather, we made a good faith programming effort to help each engine simulate the tests successfully, tuning parameters and requesting technical support in some cases.

When creating the score for each test, we evaluated a specific set of factors:

How physically plausible and consistent with our expectations were the results
How stable were the results? Did the simulation come to rest in a stable state or did the system continually jitter or explode?
Were the results accurate given the design of the situation?
How easy were the scenarios to design and adjust in each SDK?

The grading for each test ranges between 0.0 and 1.0, with 1.0 being perfect in all areas. The grades for each engine are absolute compared to a theoretical ideal, not relative to the other engines. A score of 0.5 is the minimum "acceptable" score given our expectations.

We designed the scenarios to be the same scale and with the same units (where possible) in each package. Scale is very important in a simulation, so we used standard units for measurement and chose a world scale that worked well in all three systems.

The final scores are not presented in an easy-to-compare table. This is a conscious decision, as we believe simple numbers are meaningless without a full understanding of the context. Choosing a physical simulator is not as simple as setting up a race between systems and see who wins. There are many other tests that might be more representative of your problem domain, and we encourage you to run them yourself before deciding.

Test 1:
A large cube is dropped on a small cube. This challenge was designed to test the collision detection and response of each system. Masses and sizes that vary by an order of magnitude can cause trouble for contact solvers and collision detectors.

The cubes were axially aligned and centered at the origin and translated up along the Y-axis. Both cubes were initially off the ground and dropped at the same time. The large cube was 10mx10mx10m with a mass of 5,000kg and the small cube was 1mx1mx1m with a mass of 5kg. The coefficient of restitution between all objects and the ground plane was set to 0. We expected the two cubes to drop straight on top of each other without bouncing and come to an immediate rest.

Results: Test One

MathEngine: Even though we tried everything to ensure the coefficients of restitution were 0.0, we could not get rid of the bounce when the boxes hit. In this test, the boxes bounced, then the smaller box shot out from beneath the large box. Score: 0.7

Ipion: The boxes dropped directly on each other without bouncing. However, the top box tipped over then jittered a bit before stabilizing. Score: 0.75

Havok: On contact, the top box slid back and forth in a very unrealistic manner before finally tipping over. Setting the friction even higher didn't help. The objects also went to sleep too fast, freezing as they were visibly moving.Score: 0.6

Figure 1: The Havok GDK undergoes Test 1, in which a large cube is dropped on a small cube.

Test 2:
A slightly larger cube is dropped on a smaller cube. This was a variation on the first test in that the difference between the cubes' sizes was much smaller. This test was meant to be a "gimme" and should just work.

The cubes were set at the origin and translated up along the Y-axis. Both cubes were initially off the ground and dropped at the same time. The large cube was 5mx5mx5m with a mass of 625kg and the small cube was 4mx4mx4m with a mass of 320kg. The coefficient of restitution between all objects and the ground plane was set to 0. We expected the two cubes to drop straight on top of each other without bouncing and come to an immediate rest.

Results: Test Two

MathEngine: Again, the initial bounce between the objects caused the top box to tip over. Score: 0.7

Ipion: The boxes dropped and stayed stacked. The system slept early while still jittering slightly. Score: 0.9

Havok: On contact, the boxes slid slightly. They stayed stacked and there was no jitter between them. Score: 0.9

Test 3:
A large cube is constrained to a small cube and both are dropped. This challenge was designed to test the constraint system as well as the collision detection and response of the systems. Again, orders of magnitude uncover numerical problems in constraint solvers and constrained collision and contact resolvers.

The cubes were set at the origin and translated up along the Y-axis. The two boxes were constrained together with a three-degree-of-freedom spherical constraint positioned exactly midway between the two cubes. Both cubes were initially off the ground and dropped at the same time. The large cube was 10mx10mx10m with a mass of 5,000kg and the small cube is 1mx1mx1m with a mass of 5kg. Again, we set the coefficient of restitution between all objects and the ground plane to 0. We expected the two cubes to drop straight on top of each other without bouncing and come to an immediate rest, with the top box balancing on the constraint. The joint should have stayed rigid, always maintaining the separation between the two boxes.

Results: Test Three

MathEngine: Again, the two boxes bounced on the drop. The constraint held the boxes apart. However, the box tipped over and the system continued to jitter without coming to a rest. Score: 0.4

Ipion: The joint collapsed on drop. The top box fell over and there was some jitter before the system stabilized to sleep. Score: 0.5

Havok: The joint collapsed on drop. The top box fell over and there was some jitter before the system stabilized to sleep. Applying a mouse force allowed the constraint to be completely violated, adding unrealistic energy to the system. Score: 0.4

Test 4:
A slightly larger cube is constrained to a smaller cube and both are dropped. This is a variation on Test 3 in that the difference between the cubes is much smaller. Like Test 2, this was another "gimme" with very similar sizes and masses, making for an easy-to-solve configuration.

The cubes were set at the origin and translated up along the Y-axis. The two boxes were constrained together with a spherical joint positioned exactly midway between the two cubes. Both cubes were initially off the ground and dropped at the same time. The large cube was 5mx5mx5m with a mass of 625kg and the small cube was 4mx4mx4m with a mass of 320kg. The coefficient of restitution between all objects and the ground plane was set to 0. We expected the two cubes to drop straight on top of each other without bouncing and come to an immediate, balanced rest. The constraint joint should have stayed rigid maintaining the separation between the two boxes.

Results: Test Four

MathEngine: The boxes bounced, but otherwise perfect. Score: 0.7

Ipion: There was a slight bounce. The top box finally fell over in a plausible manner. Score: 0.8

Havok: There was a slight bounce. The top box finally fell over and the system went to sleep a bit too early. We had to adjust default parameters to get this result. Score: 0.75

Test 5:
A large cube is constrained to a world anchor. A small cube is constrained to the large cube. This challenge was designed to test the constraint system with objects having a great difference in mass, including the infinite mass of the world constraint.

The cubes were set at the origin and translated up along the Y-axis. The two boxes were constrained together with spherical joint positioned exactly midway between the two cubes. The top large cube was constrained to a world anchor above it by another spherical joint. The large cube was 10mx10mx10m with a mass of 5,000kg and the small cube was 1mx1mx1m with a mass of 5kg. We expected the objects not to move, as the constraints would hold everything exactly in place.

Results: Test Five

MathEngine: Behaved as expected. Score: 0.95

Ipion: Behaved as expected. Due to the interface, we couldn't exert a strong mouse force on the small object to test stability. Score: 0.95

Havok: Behaved as expected. However, forces applied to the small box made the system slightly unstable. Score: 0.9

Test 6:
A small cube is constrained to a world anchor. A large cube is constrained to the small cube. This challenge was the same as the previous test with the two boxes reversed. It was designed to test the constraint system with objects having a great difference in mass, with the small mass between the large and infinite mass.

The cubes were set at the origin and translated up along the Y-axis. The two boxes were constrained together with a spherical joint positioned exactly midway between the two cubes. The top small cube was constrained to a world anchor above it by another spherical joint. The large cube was 10mx10mx10m with a mass of 5,000kg and the small cube was 1mx1mx1m with a mass of 5kg. We expected the objects not to move, as the constraints would hold everything exactly in place.

Results: Test Six

MathEngine: There was a slight energy addition, but it was very solid. Score: 0.9

Ipion: The constraint was violated in a springlike stretching manner. The system didn't blow up. Score: 0.45

Havok: With the default constraint stiffness parameter, tau, the constraint was stretched in a springlike manner and the system even went to sleep with the constraint violated. When we increased the tau, there was only a slight jitter. However, applying any force to the big cube caused the system to gain energy and explode. Score: 0.4

Test 7:
A small cube and large cube are dropped into a gap between two planes that narrows gradually. This challenge tested the collision detection system and the contact resolution system. The gradual slope of the planes caused the contact and collision system to exert large normal forces to stop the cubes from falling.

Two large and narrow boxes used as planes were positioned so they formed a vertical chute that narrowed to a point at the bottom at a 5.625-degree slope, and the two cubes were dropped into the gap. The large cube was 10mx10mx10m with a mass of 5,000kg and the small cube was 1mx1mx1m with a mass of 5kg. We expected the objects to fall into the gap, become wedged between the planes, and not move.

Results: Test Seven

MathEngine: There was an initial bounce, but otherwise it performed well. When the small box was moved with a force, energy was occasionally added. The lack of some basic math functions in the SDK made the test difficult to create. Score: 0.8

Ipion: This was a complete failure. Contacts were missed or incorrect impulses were applied and the boxes fell through the chute. Score: 0.0

Havok: Almost perfect. There was a small amount of bounce. Score: 0.9


Figure 2: Cubes drop into a chute in Havok (left) and Ipion (right) in Test 7.

Test 8:
One cube is dropped so that it will collide with another cube on collinear edges. This challenge tested the collision detection system. Degenerate collision manifolds can be problematic.

A slightly smaller cube was dropped onto a larger one so that they struck edge to edge. The small cube was 4mx4mx4m with a mass of 320kg and the large cube was 5mx5mx5m with a mass of 625kg. We expected the objects to fall and make contact at the edges without bouncing and then come to an immediate rest. Due to numerical inaccuracies with rotating the cubes to set up the test, we couldn't be totally sure the boxes hit exactly edge to edge.

Results: Test Eight

MathEngine: The dropped box slid off to the side in an unrealistic, slippery manner. Score: 0.7

Ipion: Almost perfect, but it went to sleep in dynamically unstable situation. A small epsilon gap was visible between the two cubes. Score: 0.95

Havok: Almost perfect, however there was a small amount of bounce. An epsilon gap was visible between the objects. Score: 0.9

Test 9:
One cube is dropped on another cube of the same size, so that they will collide exactly point-to-point. This challenge tested the collision detection system as another degenerate collision manifold test.

Two identical cubes were dropped so that they struck point-to-point. The cubes were each 4mx4mx4m with a mass of 320kg. We expected the objects to fall and contact at the point without bouncing and then come to an immediate rest. As in the previous test, due to numerical inaccuracies with rotating the cubes to set up the test, we couldn't be completely sure the boxes hit exactly point-to-point.

Results: Test Nine

MathEngine: The dropped box slid off in an unrealistic manner, exactly like in the previous test. Score: 0.7

Ipion: There was a little bounce and they went to sleep in dynamically unstable situation. Score: 0.85

Havok: Almost perfect. The boxes sat there stably, eventually tipping over. However, the default epsilon gap between objects was visible. Score: 0.9

Test 10:
Boxes are dropped to form an uneven stack. This challenge tested the collision detection system with massive numbers of contacts and collisions.

A series of boxes of various sizes were dropped onto each other with the initial rotations around the Y-axis, and slightly offset from the center. The goal was to try stacks of up to 40 boxes. We expected the objects to fall into a stable stack of blocks that behaved in a believable manner.

Results: Test Ten

MathEngine: When more then 10 to 12 boxes were dropped, the system crashed. Otherwise the boxes bounced badly, even adding energy. The system was very unstable, though no boxes appeared to interpenetrate. Score: 0.2

Ipion: Stable, but really started slowing as boxes were added to the system. Some boxes eventually fell through each other and the boxes at the bottom jiggled a bit. Parts of the stack slept while others jittered and interpenetrated. Score: 0.6

Havok: Perfectly stable for a while, but with the maximum stack, the boxes started falling through each other and the floor.
Score: 0.75


Figure 3: MathEngine (left) and Havok (right) handle stacking blocks (Test 10) with varying success.

Test 11:
A hinge joint with similar bodies. This test should have been easy to solve. A one-degree-of-freedom hinge constrained two similarly sized bodies.

The cubes were set at the origin and translated up along the Y-axis. The two boxes were constrained together with a hinge joint positioned exactly midway between the two cubes. Both cubes were initially off the ground and dropped at the same time. The large cube was 5mx5mx5m with a mass of 625kg and the small cube was 4mx4mx4m with a mass of 320kg. The coefficient of restitution between all objects and the ground plane was set to 0. We expected the two cubes to drop straight on top of each other without bouncing and come to an immediate, balanced rest. The constraint joint should have stayed rigid, maintaining the separation between the two boxes. We should have been able to move the bodies and the hinge should have only rotated around its axis.

Results: Test Eleven

MathEngine: The boxes bounced, but otherwise ran perfectly. The lack of mouse forces in the demo made it slightly hard to test the hinge. Score: 0.7

Ipion: There was a slight bounce. The top box finally fell over in a plausible manner. It was very stable. Score: 0.8

Havok: There was a nonstraight bounce. The top box finally fell over and system went to sleep a bit too early. Havok doesn't support hinges except through linearly dependent spherical joints.
Score: 0.7

Test 12:
A hinge joint with differing masses. This was the "hard" version of the previous test. The order of magnitude differences between the bodies can cause numerical trouble.

The cubes were set at the origin and translated up along the Y-axis. The two boxes were constrained together with a one-degree-of-freedom hinge constraint positioned exactly midway between the two cubes. Both cubes were initially off the ground and dropped at the same time. The large cube was 10mx10mx10m with a mass of 5,000kg and the small cube was 1mx1mx1m with a mass of 5kg. The coefficient of restitution between all objects and the ground plane was set to 0. We expected the two cubes to drop straight on top of each other without bouncing and come to an immediate rest, with the top box balancing on the constraint. The joint should have stayed rigid, always maintaining the separation between the two boxes.

Results: Test Twelve

MathEngine: The two boxes bounced on the drop. The constraint held the boxes apart. However, the box tipped over and the system continued to jitter without coming to a rest. Score: 0.45

Ipion: The joint collapsed on drop. The top box fell over and jittered. Score: 0.45

Havok: The joint collapsed on drop and the top box fell over. Applying a mouse force allowed the constraint to be completely violated and the system could blow up. Score: 0.4

Things We Didn't Test

Space and time limitations prevented us from testing more potential problems. Other things we could have tested include:

Other joint types; two-degree-of-freedom rotational joints are often problematic, prismatic joints with different moments of inertia, and so on.
Stiff springs and user-defined forces; jerking the constraints around was not really tested.
Concave collision detection, union of convex collision detection.
Sliding bodies over one another looking for inconsistent normals, especially as contact manifold collapses from a 2D patch to a 1D line segment and then to a 0D point.
Tunneling of fast-moving objects or objects strongly pulled into the ground (preliminary tests in this area indicated problems with some of the engines).

Conclusions

After conducting all of these tests on each system, we were impressed with the general engine stability across the board. With the exception of the box-stacking test on MathEngine, which was likely a memory problem, the simulators were able to handle every test without crashing. We had some difficulties with the access functions in some cases, which we will cover in depth in next month's conclusion. Nevertheless, we were able to get every test running fairly rapidly.

MathEngine is very capable at handling hard constraints. The bouncing problem on collisions hurt its score on most tests. After talking with MathEngine technical support about it, we adjusted the gamma and epsilon values and were able to improve several of the tests. However, this required case-by-case experimentation and never completely eliminated the bounce. The collision detection and resolution worked well; we never saw objects interpenetrating.

The Ipion engine was a solid performer as well, with the exception of Test 7, the narrowing chute. The rigid constraints were not exactly rigid but they were pretty stable. The collision detection and handling were reasonable, excepting the chute and box-stacking tests. The system also slowed down to unacceptable levels as the number of interacting objects increased.

The Havok engine posted generally strong scores. They clearly need to work on their rigid constraint system. The default constraint tau value (which is not well documented) resulted in springlike constraints that didn't perform well in our tests. Increasing tau caused the constraint to become more rigid, however the system then became unstable. We were also able to see object interpenetration problems when the system was really stressed. That said, the collision system was very robust and fast, handling even fairly large block stacks without slowing much.

So, which package is right for you? These stress tests are only the first part of an answer to that question. Next month we will have an in-depth review of each simulator, covering the complete feature sets, documentation, and API issues.

Note: As we were running these tests, Havok announced that they had purchased Ipion in a consolidation of this newly formed market. They will continue supporting both packages until they can consolidate them into one system.

Bio:
Jeff Lander (jeffl@darwin3d.com) is the Graphic Content columnist for Game Developer. Chris Hecker (checker@d6.com) is Game Developer's editor-at-large.