Arbitrating which properties are to be sent

When iterating over viewed entities sorted by descending priority, we compare the current value of the properties to the previous values stored during the last send. If they don't match, by a sufficient threshold, we include them in the update record to be sent. The mileage trick described in the previous section was retained for position arbitration. As this comparison is a critical part because it is called a large number of times, we used several tricks to optimise it. We later optimised the whole step by comparing only properties relevant to the entity type. For example, we know in advance that an intelligent plant is a still entity in Ryzom, so we don't need to compare the positions.

Quantitative Evaluation

Evaluating strategies and their implementation was not an easy task, because, based on a fixed bandwidth usage, they resulted in a subjective movement and animation credibility level in the client. Moreover, comparing different live moving scenes over time was not possible without any quantitative data. That is why we made measurements inside the front-end service and plotted some graphs to ease these tasks.

The Figure 4 shows an example in which we have measured the priority of the position of an entity moving at constant speed within a large group of entities, with a viewer who moves past it. In this graph, the higher the priority, the lower the y-value, the higher the probability that an update will be sent. The most common pattern looks like a saw tooth (a decreasing straight line, interrupted by a vertical line). Indeed, the watched entity is moving, thus the priority increases until an update is triggered.

Figure 4: Priorities of an entity’s position over time

The graph shows a couple of interesting phenomena: first, the first two patterns look bigger than the following ones. We can explain this by the fact that the server, at the beginning, has to send all initial properties to the client, while the bandwidth still has the same limit. The watched entity having more competitors of high priority, the sending will be triggered for a higher priority (lower y-value) than on average. Besides this competition, the first updates must compete for bandwidth with other network messages, called impulsions, used to transmit commands (unrelated with visual properties) from the server to the client.

Another visible cue is the dip in the middle of the graph. It occurred when the viewer went past the entity, in the middle of the group. With a reduced distance between viewer and viewed entity, the priority and the update frequency increased. Most of the time, the watched entity was at a similar distance from the viewer as the surrounding entities, but in the short time frame in the middle of the curve, it was closer to the watcher than the majority of the others.

Qualitative Evaluation

A light client, based on Snowballs (the technology demo for NeL), was developed to visually confirm the quantitative evaluation results, before a full Ryzom client was ready. As this prototype was on client side, it provided a test application for the front-end code implementing the frequency control. The update frequency of a position of an entity was represented by its colour.

Figure 5: visual check of update frequencies

The above screenshot (Figure 5) shows an example where the closer entities (in blue and green) to a viewer (in white) have higher update frequency than farther entities (in yellow and orange).

Optimizations

To reduce the number of operations to do in a game cycle, a system to split up computation over several game cycles was added. As a result, only an incremental subset of visibility pairs is prioritised at a time.

Another optimisation consisted in taking advantage of the multiprocessor machines we had (and later, hyper threaded multiprocessors): the computation part and the sending part were pipelined using several threads.

Low-level optimisations were also done: by displaying the assembly code generated by the C++ compiler, we could rework the data structures so that the processor had minimal overhead and cache misses when accessing them.

Conclusion

At the end, each instance of our front-end service was able to filter and prioritise dynamic properties, and to deliver them to roughly a thousand clients each. The scalability of this system, fed of properties by a mirror system, allowed to multiply the number of front-end services depending on the number of players to be supported in a single seamless environment (shard). We can run several instances of the Front End Service in parallel on a single multi-processor server and several such servers per game shard.

Of course, this system does not guarantee the order in which property change events are sent to the client. To synchronize some property changes, other information such as timestamps are used.