Along with Assemblies, our shift from centralized to distributed physics has allowed more players and more physics simulations in a shared space. Roblox initially used centralized physics simulation, a model under which physics calculations are processed on the game server, and the results are constantly fed to players and reflected on screen.
While a centralized model works on a small scale -- meaning few players, few physics calculations or few simulated parts -- Roblox's implementation wasn't made to stack up against our growth and increasingly complex physics. The server must simulate too many parts and player interactions, causing motion slow down rather than run in real time.
We solved this problem by evolving our centralized engine in 2010 to a distributed physics engine.
This graphic is a simplified illustration of the distributed physics model. In practice, client-simulated regions are not necessarily uniform in shape.
Distributed physics distributes the physics simulation among the server and players. In the graphic above, the blue area represents the server (server-simulated region) and all its physical bodies and players. In the case that no player is within an area, the server handles the physics simulation and relays data to players, as in the centralized model.
The difference is that each player has a dynamic region of physical bodies -- those within close proximity -- that they simulate locally (client-simulated region), taking the strain off the server. Such distributed architecture allows us to scale to an almost infinite number of physically simulated bodies without degrading physics performance.
We implemented load-balancing algorithms to shrink or grow the region size of each player, based on their computation power and network bandwidth. Players with slow machines or internet connections will generally simulate fewer physical objects; the opposite is true for players with fast connections and hefty rigs. The key challenge behind distributed physics is simulating objects at the border of two players' simulation regions.
A strict division works in very simple circumstances; for example, an object is in one player's simulation region and not interacting with other objects. The player runs the rigid-body simulation and sends the results to the other players.
In a more challenging -- and very likely -- scenario, object A and object B are colliding with each other. Each object is in a separate player's region. Theoretically, we could let one player simulate object A and the other player simulate object B, then exchange simulation results. With an immediate network update, this works. But the network update interval is usually lower (e.g., 15Hz) than the rendering rate (e.g., 30Hz) to conserve bandwidth. Assuming our physics engine simulates at 30Hz, twice as fast as the network update frequency, the network update only occurs at every other step of physics simulation. This causes problems during high-speed collisions.
Here's an example:
A is a vehicle simulated by player A
B is a brick simulated by player B
The scenario: Vehicle A runs into brick B. Vehicle A bounces back as though it ran into a rubber wall. (We wrote an article about this phenomenon, for more background.) Vehicle A should slow down and brick B should at least get knocked over. The following steps illustrate why it breaks:
Simulation A (Vehicle A/Player A):
A touches B at high speed and generates contact force, separating A from B
A bounces back and moves away from B; sends update of A to B
A receives update from B; B is not moving
Simulation B (Object B/Player B):
B receives update from A; A is two steps away from B
A is one step away from B; sends update of B to A.
B receives update of A. A moves away from B. B is not moving.
Simulation B doesn't get a chance to witness the collision before Simulation A processes the collision and separates the vehicle from the brick. As a result, brick B behaves as though it was anchored and vehicle A bounces back.
For slow collisions or cross-boundary interactions, this situation does not occur because both sides have a chance to witness the interaction and generate a more believable response. Increasing the network-update frequency or reducing network-link latency will improve the results of high-speed collisions, but will not solve the problem: The two parties involved in a collision are not simulated on the same simulator.