|
3) Core business attributes: The essential attributes related to the core of the business model of the company, for example, logging every time a user purchases a virtual item (and what that item is), establishes a friend connection in-game, or recommends the game to a Facebook friend -- or any other attributes related to revenue, retention, virality, and churn. For a mobile game, geolocation data can be very interesting to assist target marketing. In a traditional retail situation, none of these are of interest, of course.
4) Stakeholder requirements: In addition, there can be an assortment of stakeholder requirements that need to be considered. For example, management or marketing may place a high value on knowing the number of Daily Active Users (DAU). Such requirements may or may not align with the categories mentioned above.
5) QA and user research: Finally, if there is any interest in using telemetry data for user research/user testing and quality assurance (recording crashes and crash causes, hardware configuration of client systems, and notable game settings), it may be necessary to augment to attributes on the list of features accordingly.
When building the initial attribute set and planning the metrics that can be derived from them, you need to make sure that the selection process is as well informed as possible, and includes all the involved stakeholders. This minimizes the need to go back to the code and embed additional hooks at a later time -- which is a waste that can be eliminated with careful planning.
That being said, as the game evolves during production as well as following launch (whether a persistent game or through DLCs/patches), it will typically be necessary to some degree to embed new hooks in the code in order to track new attributes and thus sustain an evolving analytics practice. Sampling is another key consideration. It may not be necessary to track every time someone fires a gun, but only 1 percent of these. Sampling is a big issue in its own right, and we will therefore not delve further on this subject here, apart from noting that sampling can be an efficient way to cut resource requirements for game analytics.

Figure 2: The drivers of attribute selection for user behavior attributes. Given the broad scope of application of game analytics, a number of sources of requirements exist.
Preselecting Features
One important factor to consider during the feature selection process is the extent to which your attribute set selection can be driven by pre-planning, by defining the game metrics and analysis results (and thereby the actionable insights) we wish to obtain from user telemetry and select attributes accordingly.
Reducing complexity is necessary, but as you restrict the scope of the data-gathering process, you run the risk of missing important patterns in user behavior that cannot be detected using the preselected attributes. This problem is exasperated in situations where the game metrics and analyses are also predefined -- for example, relying on a set of Key Performance Indicators (such as DAU, MAU, ARPU, LTV, etc.) can eliminate your chance of finding any patterns in the behavioral data not detectable via the predefined metrics and analyses. In general, striking a balance between the two situations is the best solution, depending on available analytics resource. For example, focusing exclusively on KPIs will not tell you about in-game behavior, e.g., why 35 percent of the players drop out on level 8 -- for that we need to look at metrics related to design and performance.
It is worth noting that when it comes to user analytics, we are working with human behavior, which is notoriously unpredictable. This means that predicting user analytics requirements can be challenging. This emphasizes the need for the use of both explorative (we look at the user data to see what patterns they contain) and hypothesis-driven methods (we know what we want to measure and know the possible results, not just which one is correct).
|
In my experience that's one of the main constraints in game play related feature selection; QAing thousands of data points is simply unrealistic.
Developers who are aware that their analytics may produce false numbers trust data only if results are in line with their assumptions. That makes any analytics kind of pointless. Those who are not aware of that and trust their data usually end up making expensive mistakes. It's really way better to go blind without data and trust your team experience in both cases.
Of course there are ways to deal with that problem and get reliable results from analytics:
1. make sure that engineer instrumenting analytics service is working directly with someone experienced with that specific service - either someone in-house or a support guy from analytics vendor who will explain the process and audit the integration.
2. having a real time analytics during integration really helps as you can record your session and check results instantly. It's also important that you have ability to clean up database (or filter out your most recent activity) to make sure that you are checking the data from the last session only. If your analytics doesn't give you that comfort, log every outgoing data point on your side and do the math by yourself.
3. Even if you are very diligent about the integration, chances that you will get it right from the beginning are low. If you don't want to get into troubles due to data misinterpretation double check it using qualitative approach. Some of analytics services allows you to export data points by session id to excel and some others* have full set of features to analyze individual sessions or users. This will help you identify mistakes in data collection but also better understand correctly collected data before you jump into conclusions.
*I know only UseItBetter Analytics (disclosure: I'm co-founder) that does that for games but I might be totally wrong about it. Maybe Mixpanel or Playnomics?