Appearance
Skip to content This document describes the data format used in Analytics Events that are written with the Built-in Analytics sink.
Overview ​
When Metaplay Analytics Events are exported via the BigQuery Analytics Sink into a BigQuery Table, the event format is modified to roughly emulate the native export format of Firebase Analytics. The intention is to make integration easy with Firebase-aligned tooling. As with Firebase, Metaplay writes each Analytics Event as a single row in the BigQuery Table, and the values in an AnalyticsEventEnvelope are exported as follows:
| Event Field | BigQuery behavior |
|---|---|
Source | Mapped to source_id |
UniqueId | Mapped to event_id. This value is also used as the BigQuery deduplication key if native BigQuery deduplication is enabled. |
ModelTime | Mapped to event_timestamp |
EventType | Mapped to event_name |
SchemaVersion | Mapped to event_schema_version |
Payload | Flattened into event_params as key-value pairs. The process is described in more detail below. |
Context | Built-in fields of PlayerContext are mapped to built-in schema fields
|
Labels | Mapped to a labels list where each element is represented by a { "name": name, "string_value": value } pair |
Example event:
json
{
"source_id": "Player:123",
"event_id": "11223344AABBCCDD66778899EEFF0011",
"event_timestamp": "2020-10-10 14:40:12.000000",
"event_name": "PlayerEventClientDisconnected",
"event_schema_version": 1,
"event_params": [
{
"key": "SessionToken",
"string_value": "0000000000BC614E"
}
],
"player": {
"session_number": 4,
"experiments": [
{
"experiment": "experimentId",
"variant": "variantId"
}
]
},
"labels": [
{
"name": "Location",
"string_value": "FI"
}
]
}Payload Representation and Flattening ​
All Event Payload fields are represented as key-value pairs. The key is always a string, and the value is either an integer, a floating-point value, a string value, or a null value. This is represented as a struct of {"key": key, "string_value": value, "int_value": value, "double_value": value }. When exported, a single field will have only a single *_value field set, or none in the case of a null value.
When an Event is exported, each field in the Payload object is mapped to a single key-value pair in the exported row. For flat Payload objects, such as the example PlayerEventClientDisconnected, each field is trivially mapped to a key-value pair without any transformation. In the case of the example, the SessionToken field is mapped to a SessionToken key. But if the Payload contains either (1) Aggregate types, (2) Array types, or (3) Dynamic Types, the object tree is walked and flattened to key-value pairs as follows:
- Aggregate types, i.e., fields that contain multiple fields, are represented as multiple key-value pairs where each subfield uses a name of
<aggregate_field_name>:<subfield_name>. For example, aPair pairfield where Pair isstruct Pair { int A, int B }would be represented aspair:Aandpair:Bint-valued key-value pairs. If the field is null, it is interpreted as if all subfields were null. - Array types are represented as their elements where each element uses a name of
<array_field_name>:<array_index>. For example,int[] valuewould be represented asvalue:0,value:1, ... fields. - Dynamic types, i.e., fields that are declared with a base type and the contents depend on the runtime value of the field, are represented like Aggregate Types but with an additional synthetic
$tsubfield. The value$tstring is the runtime type of the field. For example, aBasetype fieldwith a value ofclass ConcreteType : BaseType { int A }would be exported asfield:$t = "ConcreteType"andfield:Asubfields. A null value is interpreted as if there are no fields.
The rules are applied recursively, and a single top-level field uses all of the rules. For example, a single MetaReward field named ResolvedContent might expand to:
json
{
"key": "ResolvedContent:$t",
"string_value": "ResolvedPurchaseMetaRewards"
},
{
"key": "ResolvedContent:Rewards:0:$t",
"string_value": "RewardGems"
},
{
"key": "ResolvedContent:Rewards:0:Amount",
"int_value": 123
},
{
"key": "ResolvedContent:Rewards:1:$t",
"string_value": "RewardGold"
},
{
"key": "ResolvedContent:Rewards:1:Amount",
"int_value": 123
},The flattening of individual types and members can be controlled with a BigQueryAnalyticsFormat attribute which allows choosing the flattening mode. The mode Ignore removes the annotated Type or Member from the BigQuery flattening result. This can be used as follows:
C#
class Event
{
//...
// this field is not exported via BigQuery
[BigQueryAnalyticsFormat(BigQueryAnalyticsFormatMode.Ignore)]
public int Field;
// this field is also not exported via BigQuery
public DataFragment fragment;
}
[BigQueryAnalyticsFormat(BigQueryAnalyticsFormatMode.Ignore)]
class DataFragment { }The mode ExtractDictionaryElements formats the annotated dictionary-like member as an object instead of an array. For example:
C#
class Event
{
public Dictionary<string, string> Normal = new() {{"foo","bar"}, {"baz","quz"}};
[BigQueryAnalyticsFormat(BigQueryAnalyticsFormatMode.ExtractDictionaryElements)]
public Dictionary<string, string> Extracted = new() {{"foo","bar"}, {"baz","quz"}};
}would result in:
json
{
"key": "Normal:0:Key",
"string_value": "foo"
},
{
"key": "Normal:0:Value",
"string_value": "bar"
},
{
"key": "Normal:1:Key",
"string_value": "baz"
},
{
"key": "Normal:1:Value",
"string_value": "quz"
},
{
"key": "Extracted:foo",
"string_value": "bar"
},
{
"key": "Extracted:baz",
"string_value": "quz"
},You can use the [BigQueryAnalyticsName(...)] attribute to override the name of an event when you want it to be different from the C# class:
C#
// Default: this event will have the name "MyEvent" in BigQuery analytics.
[AnalyticsEvent(PlayerEventCodes...)]
public class MyEvent : PlayerEventBase
{
// ... members ...
}
// Overridden event name: this event will have the name "my_other_event" in BigQuery analytics.
[BigQueryAnalyticsName("my_other_event")]
[AnalyticsEvent(PlayerEventCodes...)]
public class MyOtherEvent : PlayerEventBase
{
// ... members ...
}You can use the same attribute for overriding the name of an event parameter, which would otherwise get its name from the C# member:
C#
[AnalyticsEvent(PlayerEventCodes...)]
public class MyEvent : PlayerEventBase
{
// Default: this member will appear as "MyMember" in the BigQuery analytics event.
[MetaMember(1)]
public string MyMember { get; private set; }
// Overridden parameter name: this member will appear as "my_other_member" in the BigQuery analytics event.
[BigQueryAnalyticsName("my_other_member")]
[MetaMember(2)]
public int MyOtherMember { get; private set; }
// ...
}Analytics Context Representation ​
The default analytics context is mapped to the dedicated player.session_number and player.experiments columns in the BigQuery table. In the case of a Custom Analytics Context, the custom fields are flattened similarly to the analytics event fields. When a context field is flattened into event_params, it is given a $c: prefix to differentiate it from event fields potentially sharing the same name.
As an example if we have a context and an event such as:
C#
[MetaSerializableDerived(100)]
public class MyAnalyticsContext : PlayerAnalyticsContext
{
[BigQueryAnalyticsName("context_field")]
[MetaMember(201)]
public int ContextField;
MyAnalyticsContext() {}
public MyAnalyticsContext(int? sessionNumber, OrderedDictionary<string, string> experiments, int contextField) : base(sessionNumber, experiments)
{
ContextField = contextField;
}
}
[AnalyticsEvent(2000, "MyExampleEvent")]
[AnalyticsEventCategory("Example")]
[BigQueryAnalyticsName("MyEvent")]
public class MyExampleEvent : AnalyticsEventBase
{
[BigQueryAnalyticsNameAttribute("my_field")]
[MetaMember(1)] public int NamedField;
MyExampleEvent() { }
public MyExampleEvent(int namedField)
{
NamedField = namedField;
}
}Emitting the event with the context might result in:
json
{
// ...
"event_params": [
{
"key": "my_field",
"int_value": 0
},
{
"key": "$c:context_field",
"int_value": 1
}
]
}Example Event Generator ​
Due to the complexity of the key-value mapping, predicting the exact data format for an Event can be difficult. To help with inspecting and validating the mappings, the Metaplay LiveOps Dashboard ships with an example event generator. By selecting the desired Analytics Event on the Analytics Events page, an example JSON-formatted BigQuery Event can be inspected. The tool is only intended for inspecting the data format; the example contents in each field are mock values.
Appendices ​
BigQuery Table Schema:
Raw Schema JSON
json[ { "name": "source_id", "type": "STRING", "mode": "REQUIRED", "description": "EntityId of the source entity. For players, this is for example Player:0a23456789. Can be other entities as well, like Guild:XXX." }, { "name": "event_id", "type": "STRING", "mode": "REQUIRED", "description": "A unique ID of the event." }, { "name": "event_timestamp", "type": "TIMESTAMP", "mode": "REQUIRED", "description": "Timestamp of the event." }, { "name": "event_name", "type": "STRING", "mode": "REQUIRED", "description": "Name of the event. The name of the analytics event type in server code." }, { "name": "event_schema_version", "type": "INTEGER", "mode": "REQUIRED", "description": "Schema version of the event parameter data. Developer bumps this if the event class in code changes shape." }, { "name": "event_params", "type": "RECORD", "mode": "REPEATED", "description": "Event-specific key-to-value mapping", "fields": [ { "name": "key", "type": "STRING", "mode": "REQUIRED", "description": "Name of the event param." }, { "name": "string_value", "type": "STRING", "mode": "NULLABLE", "description": "String value of the event param if the param is a string." }, { "name": "int_value", "type": "INTEGER", "mode": "NULLABLE", "description": "Integer value of the event param if the param is an integer." }, { "name": "double_value", "type": "FLOAT", "mode": "NULLABLE", "description": "Floating point value of the event param if the param is a floating point." } ] }, { "name": "player", "type": "RECORD", "mode": "NULLABLE", "description": "The context of the player entity. Null for non-player events.", "fields": [ { "name": "session_number", "type": "INTEGER", "mode": "NULLABLE", "description": "The (nullable) session number of the player. For the first session of the player, this will be 1, and increase by one for each subsequent session. If there is no session, the value is null." }, { "name": "experiments", "type": "RECORD", "mode": "REPEATED", "description": "The active experiments of the player. Empty list for non-player events.", "fields": [ { "name": "experiment", "type": "STRING", "mode": "REQUIRED", "description": "The Analytics ID of the experiment." }, { "name": "variant", "type": "STRING", "mode": "REQUIRED", "description": "The Analytics ID of the experiment variant." } ] } ] }, { "name": "labels", "type": "RECORD", "mode": "REPEATED", "description": "The custom labels assigned by the source entity.", "fields": [ { "name": "name", "type": "STRING", "mode": "REQUIRED", "description": "The name of the custom label." }, { "name": "string_value", "type": "STRING", "mode": "NULLABLE", "description": "The string value of the custom label." } ] } ]The above schema can be used when creating the table via API or via Google Cloud Console. For Google Cloud Console, tap Edit as Text to paste the schema JSON.
Example dump for a simple session:
A player logs in, purchases an item, and logs out. The player is in a running experiment, and we can see the experiment metadata events.
Stream dump
json{ "source_id": "GlobalStateManager:0000000000", "event_id": "0000017C2E005FC46B684835D11C663D", "event_timestamp": "2021-09-28 20:02:28.932000", "event_name": "ServerEventExperimentInfo", "event_schema_version": 1, "event_params": [ { "key": "ExperimentId", "string_value": "EarlyGameFunnel" }, { "key": "ExperimentAnalyticsId", "string_value": "_egf" }, { "key": "IsActive", "int_value": 1 }, { "key": "IsRollingOut", "int_value": 0 }, { "key": "DisplayName", "string_value": "Faster early game funnel" }, { "key": "Description", "string_value": "If we tweak the early game funnel, then we can discover which settings work best to retain players, because players will either prefer a slower or faster early game experience." } ] } { "source_id": "GlobalStateManager:0000000000", "event_id": "0000017C2E005FC6C9BB324455A432B9", "event_timestamp": "2021-09-28 20:02:28.934000", "event_name": "ServerEventExperimentVariantInfo", "event_schema_version": 1, "event_params": [ { "key": "ExperimentId", "string_value": "EarlyGameFunnel" }, { "key": "ExperimentAnalyticsId", "string_value": "_egf" }, { "key": "VariantId", "string_value": "Fast" }, { "key": "VariantAnalyticsId", "string_value": "_fast" } ] } { "source_id": "GlobalStateManager:0000000000", "event_id": "0000017C2E005FC616FC1F5DD6A9CA26", "event_timestamp": "2021-09-28 20:02:28.934000", "event_name": "ServerEventExperimentVariantInfo", "event_schema_version": 1, "event_params": [ { "key": "ExperimentId", "string_value": "EarlyGameFunnel" }, { "key": "ExperimentAnalyticsId", "string_value": "_egf" }, { "key": "VariantId", "string_value": "Slow" }, { "key": "VariantAnalyticsId", "string_value": "_slow" } ] } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E0783BA084AD329DCEBCAAD", "event_timestamp": "2021-09-28 20:10:16.834000", "event_name": "PlayerEventClientConnected", "event_schema_version": 1, "event_params": [ { "key": "SessionToken", "string_value": "EECBC270F71FA6F6" }, { "key": "DeviceId", "string_value": "Vr0296eTOgX0JmagTlllBWseGIdtRgp5cCiNEvT2RVL8JQCk" }, { "key": "DeviceModel", "string_value": "Precision 7540 (Dell Inc.)" }, { "key": "LogicVersion", "int_value": 5 }, { "key": "TimeZoneInfo:CurrentUtcOffset:Milliseconds", "int_value": 10800000 }, { "key": "Location:Country:IsoCode", "string_value": null }, { "key": "ClientVersion", "string_value": "0.1" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E07A0F2A8C477A5C9C46B0E", "event_timestamp": "2021-09-28 20:10:24.134000", "event_name": "PlayerEventPendingStaticPurchaseContextAssigned", "event_schema_version": 1, "event_params": [ { "key": "ProductId", "string_value": "GemPackLarge" }, { "key": "DeviceId", "string_value": "Vr0296eTOgX0JmagTlllBWseGIdtRgp5cCiNEvT2RVL8JQCk" }, { "key": "GameProductAnalyticsId", "string_value": "GemPackLarge" }, { "key": "PurchaseContext:$t", "string_value": "IdlerPurchaseAnalyticsContext" }, { "key": "PurchaseContext:Placement", "string_value": "ShopTab" }, { "key": "PurchaseContext:Group", "string_value": "Basic" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E07A5436E376BBF50A37C96", "event_timestamp": "2021-09-28 20:10:25.234000", "event_name": "PlayerEventInAppValidationStarted", "event_schema_version": 1, "event_params": [ { "key": "ProductId", "string_value": "GemPackLarge" }, { "key": "Platform", "string_value": "Development" }, { "key": "TransactionId", "string_value": "fakeTxn449467" }, { "key": "PlatformProductId", "string_value": "dev.GemPackLarge" }, { "key": "ReferencePrice", "double_value": 9.989999999990687 }, { "key": "GameProductAnalyticsId", "string_value": "GemPackLarge" }, { "key": "PurchaseContext:$t", "string_value": "IdlerPurchaseAnalyticsContext" }, { "key": "PurchaseContext:Placement", "string_value": "ShopTab" }, { "key": "PurchaseContext:Group", "string_value": "Basic" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E07A95272F8B44CF259FE0C", "event_timestamp": "2021-09-28 20:10:25.234000", "event_name": "PlayerEventInAppValidationComplete", "event_schema_version": 1, "event_params": [ { "key": "Result", "string_value": "Valid" }, { "key": "ProductId", "string_value": "GemPackLarge" }, { "key": "Platform", "string_value": "Development" }, { "key": "TransactionId", "string_value": "fakeTxn449467" }, { "key": "PlatformProductId", "string_value": "dev.GemPackLarge" }, { "key": "ReferencePrice", "double_value": 9.989999999990687 }, { "key": "GameProductAnalyticsId", "string_value": "GemPackLarge" }, { "key": "PurchaseContext:$t", "string_value": "IdlerPurchaseAnalyticsContext" }, { "key": "PurchaseContext:Placement", "string_value": "ShopTab" }, { "key": "PurchaseContext:Group", "string_value": "Basic" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E07A98B966982742D5AD08E", "event_timestamp": "2021-09-28 20:10:26.334000", "event_name": "PlayerEventInAppPurchased", "event_schema_version": 2, "event_params": [ { "key": "ProductId", "string_value": "GemPackLarge" }, { "key": "Platform", "string_value": "Development" }, { "key": "TransactionId", "string_value": "fakeTxn449467" }, { "key": "PlatformProductId", "string_value": "dev.GemPackLarge" }, { "key": "ReferencePrice", "double_value": 9.989999999990687 }, { "key": "GameProductAnalyticsId", "string_value": "GemPackLarge" }, { "key": "PurchaseContext:$t", "string_value": "IdlerPurchaseAnalyticsContext" }, { "key": "PurchaseContext:Placement", "string_value": "ShopTab" }, { "key": "PurchaseContext:Group", "string_value": "Basic" }, { "key": "ResolvedContent:$t", "string_value": "ResolvedPurchaseGameContent" }, { "key": "ResolvedContent:NumGems", "int_value": 100 }, { "key": "ResolvedContent:NumGold", "int_value": 10000 } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } } { "source_id": "Player:X9Pq78AMqb", "event_id": "0000017C2E081B3B8CC57015FAFBB917", "event_timestamp": "2021-09-28 20:10:26.334000", "event_name": "PlayerEventClientDisconnected", "event_schema_version": 1, "event_params": [ { "key": "SessionToken", "string_value": "EECBC270F71FA6F6" } ], "player": { "session_number": 6, "experiments": [ { "experiment": "_egf", "variant": "_fast" } ] } }