Data rules

Last updated 10/05/2026

This chapter describes the data structure, data types, and data limits of the AE backend in detail. It shows you how to build data that follows the rules and how to troubleshoot data transfer issues.

If you upload data through LogBus or the RESTful API, you need to format the data according to the data rules in this chapter.

1. Data structure

The AE backend accepts JSON data that follows the rules. If you integrate through an SDK, the data is converted to JSON for transfer. If you upload data through LogBus or the POST method, the data must be JSON that follows the rules.

JSON data is organized by line: each line is one JSON record, which physically corresponds to one record of data and logically corresponds to one user action or one user property setting.

The data format and requirements are as follows (the data is formatted for readability; don't include line breaks in a real environment):

Sample event data:

{
  "#account_id": "ABCDEFG-123-abc",
  "#distinct_id": "F53A58ED-E5DA-4F18-B082-7E1228746E88",
  "#type": "track",
  "#ip": "192.168.171.111",
  "#uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "#time": "2017-12-18 14:37:28.527",
  "#event_name": "test",
  "properties": {
    "argString": "abc",
    "argNum": 123,
    "argBool": true
  }
}

Sample user property setting:

{
  "#account_id": "ABCDEFG-123-abc",
  "#distinct_id": "F53A58ED-E5DA-4F18-B082-7E1228746E88",
  "#type": "user_set",
  "#uuid": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "#time": "2017-12-18 14:37:28.527",
  "properties": {
    "userArgString": "abc",
    "userArgNum": 123,
    "userArgBool": true
  }
}

The value of "#type" can be replaced with "user_setOnce", "user_add", "user_unset", "user_append", or "user_del"

In terms of structure and function, a JSON record can be divided into two parts:

The other fields at the same level as properties make up the basic information of the record and include only the following:

The account ID #account_id and distinct ID #distinct_id of the user who triggered the data
The trigger time #time, accurate to the second or millisecond
#type, which indicates the data type (event or user property setting)
#event_name, which indicates the event name (only in event data)
#ip, which indicates the user's IP
#uuid, which indicates the uniqueness of the data

Note that apart from the items above, all other properties that start with "#" must be placed inside properties

The inner layer of properties is the content of the record, that is, the properties in the event or the user properties to set. They are used directly as properties or analysis objects in backend analysis.

Structurally, these two parts are somewhat similar to a message header (Header) and a message body (Content). The following sections describe the meaning of each field in these two parts in detail.

1.1 Data information section

As shown in the event data sample above, the fields at the same level as "properties" make up the information section of the record.

These fields contain information about the record, such as the triggering user and the trigger time. All of them start with "#". This section explains what each field means and how to configure it.

1.1.1 User information (`#account_id` and `#distinct_id`)

#account_id and #distinct_id are the two fields that the AE backend uses to identify users. #account_id is the user's ID when logged in, and #distinct_id is the user's identifier when not logged in. The AE backend uses these two fields to determine which user triggered the action, with #account_id taking precedence. For the detailed rules, see the user identification rules chapter.

You must pass in at least one of #account_id and #distinct_id. If all events are triggered while the user is logged in, passing in only #account_id works. If some events are triggered while the user is not logged in (including before registration), we recommend that you fill in both fields.

1.1.2 Data type information (`#type` and `#event_name`)

#type determines the type of the record: a user action record or an operation that modifies user properties. Every record must have the #type field. The values of #type fall into two categories: track means the record is a user action record, and values that start with user_ mean operations on user properties. Details:

track: Sends an event to the event table. All event uploads use track
user_set: Operates on the user table and overwrites one or more user properties. If a property already has a value, the previous value is overwritten
user_setOnce: Operates on the user table and initializes one or more user properties. If a property already has a value, this operation is ignored
user_add: Operates on the user table and accumulates one or more numeric user properties
user_unset: Operates on the user table and clears the values of one or more of the user's properties
user_del: Operates on the user table and deletes the user
user_append: Operates on the user table and adds elements to the user's list-type property values
user_uniq_append: Operates on the user table, adds elements to the user's list-type property values, and deduplicates the whole list once (deduplication keeps the original order of the elements)

When the value of #type is track, that is, the record is an action record, you must configure the event name #event_name. It must start with a letter and can contain only lowercase letters, digits, and underscores "_", up to 50 characters. Don't include spaces. If the record is an operation that modifies user properties, the #event_name field isn't needed.

View original image

Note that user properties are properties that mark milestones for a user, so we don't recommend modifying them frequently within a short period. We recommend putting properties that change frequently in events as event properties

1.1.3 Trigger time (`#time`)

#time is the time when the event occurred and is required. It must be a string accurate to the millisecond ("yyyy-MM-dd HH:mm:ss.SSS") or the second ("yyyy-MM-dd HH:mm:ss")

Although data that operates on the User table also needs #time, user property operations are performed in the order in which the backend receives the data.

For example, if a user resends User table operation data from a past day, both overwriting and initialization of properties proceed as usual and aren't decided by the #time field

1.1.4 Trigger location (`#ip`)

#ip is the IP address of the device and is optional. AE calculates the user's geographic location from the IP address. If you pass in geographic properties such as #country, #province, and #city in "properties", the values you pass in take precedence

1.1.5 Unique data ID (`#uuid`)

#uuid is a field that indicates the uniqueness of the data. It's optional and must be in the standard uuid format. Depending on the data volume, AE checks at the receiver, over a period of time, whether data with the same #uuid (that is, duplicate data) appears within a short time, and discards duplicates directly without storing them

Note that receiver-side validation through #uuid only checks data received in the last few hours. It mainly addresses short-term duplicates caused by network jitter and can't check received data against all data. To deduplicate data, contact ThinkingAI staff.

1.2 Data body section

The other part of the data is the data inside properties. properties is a JSON object whose data is expressed as key-value pairs. For user action data, it represents the properties and metrics of the action (equivalent to fields in the action table), which can be used directly in analysis. For user property operations, it represents the property content to set.

The key is the property name and is a string. Custom properties must start with a letter and can contain only lowercase letters, digits, and underscores "_", up to 50 characters. There are also AE preset properties that start with #, which you can learn more about in the preset properties chapter. Note that in most cases we recommend using only custom properties, without #.

The value is the property value and can be a number, text, time, Boolean, list, object, or object group. The data types are represented as shown in the following table:

AE data type	Sample value	Description	Data type
Numeric	123,1.23	The data range is -9E15 to 9E15	Number
Text	"ABC","Shanghai"	The default character limit is 2KB	String
Time	"2019-01-01 00:00:00","2019-01-01 00:00:00.000"	"yyyy-MM-dd HH:mm:ss.SSS" or "yyyy-MM-dd HH:mm:ss". To represent a date, you can use "yyyy-MM-dd 00:00:00"	String
Boolean	true,false	-	Boolean
List	["a","1","true"]	All elements in a list are converted to strings. A list can contain up to 500 elements	Array(String)
Object	{"hero_name":"Liu Bei","hero_level":22,"hero_equipment": ["Twin Swords","Dilu"],"hero_if_support":false}	Each child property (Key) in an object has its own data type. For value descriptions, see the regular property of the corresponding type above An object can contain up to 100 child properties	Object
Object group	[{"hero_name":"Liu Bei","hero_level":22,"hero_equipment": ["Twin Swords","Dilu"],"hero_if_support":false}, {"hero_name":"Liu Bei","hero_level":22,"hero_equipment": ["Twin Swords","Dilu"],"hero_if_support":false}]	Each child property (Key) in an object group has its own data type. For value descriptions, see the regular property of the corresponding type above An object group can contain up to 500 objects	Array(Object)

Note that the type of each property is determined by the type of the first value received for it. The type in subsequent data must match the type of the corresponding property. Properties with mismatched types are discarded (the other properties of the record with matching types are kept). AE doesn't perform compatible type conversion.

2. Data processing rules

After the AE server receives data, it processes the data. This section describes the processing rules of the AE backend with practical scenarios:

2.1 Receiving new event data

After receiving data for a new event, the AE backend automatically creates an association model between the new event and its properties. If a new property is received, its type when it's first received becomes the type of the property, and the type can't be changed afterward.

2.2 Add event properties

View original image

To add properties to an existing event, simply pass in the new properties when you upload data. The AE backend dynamically associates the event with the new properties, and no other configuration is needed.

2.3 Handling property inconsistencies

View original image

When an event record is received in which the type of a property differs from the type of that property stored in the backend, the value of that property is discarded (that is, its value is null).

2.4 Deprecate event properties

View original image

To deprecate an event property, simply hide the property in the Data management module of the AE backend, and subsequent data doesn't need to include it. The AE backend doesn't delete the data of the property, and hiding is reversible. If you still send the property after it's hidden, its values are still kept.

2.5 Properties shared by multiple events

View original image

Properties with the same name in different events are treated as the same property with the same type. Therefore, make sure that all properties with the same name have the same type, to avoid property values being discarded because of type mismatches.

2.6 User table operation logic

Data that modifies a user's data in the user table, that is, reported data whose #type field is user_set, user_setOnce, user_add, user_unset, user_append, or user_del, can essentially be seen as an instruction that operates on the user table data of the user that the record refers to. The #type field determines the type of operation, and the properties in properties determine its content.

The specific logic of the main user table property operations is as follows:

2.6.1 Overwrite user properties (user_set)

Determines the user to operate on from the user ID in the data, and then overwrites all properties based on the properties in properties. If a property doesn't exist, it's created.

2.6.2 Initialize user properties (user_setOnce)

Determines the user to operate on from the user ID in the data, and then, based on the properties in properties, sets the properties that have no value (are empty). If a property to set already has a value for the user, it isn't overwritten. If a property doesn't exist, it's created.

2.6.3 Accumulate user properties (user_add)

Determines the user to operate on from the user ID in the data, and then accumulates numeric properties based on the properties in properties. Passing in a negative value subtracts it from the original property value. If a property to set has no value (is empty) for the user, it's set to 0 by default before accumulation. If the property doesn't exist, it's created.

2.6.4 Clear user property values (user_unset)

Determines the user to operate on from the user ID in the data, and then clears all the properties in properties (that is, sets them to NULL). If a property doesn't exist, it isn't created.

2.6.5 Add elements to list user properties (user_append)

Determines the user to operate on from the user ID in the data, and then adds elements to list properties based on the properties in properties

2.6.6 Delete a user (user_del)

Determines the user to operate on from the user ID in the data and deletes the user from the user table. The event data of the user isn't deleted.

2.6.7 Add elements to deduplicated list user properties (user_uniq_append)

Determines the user to operate on from the user ID in the data, adds elements to list properties based on the properties in properties, and deduplicates the whole list once (deduplication keeps the original order of the elements)

3. Data limits

Limits on the number of event types and properties

For performance reasons, the AE backend limits the number of event types and properties in a project by default:

Limit	Max event types	Max event properties	Max user properties
Recommended max	100	300	100
Hard limit	500	1000	500

Admins can go to the Project Settings page to query the number of event types and properties that each project has used. To raise the limits on event types and properties, contact ThinkingAI staff.

Length limits of the account ID (#account_id) and distinct ID (#distinct_id)
- Projects created before version 3.1: 64 characters. To increase it to 128 characters, contact ThinkingAI staff
- Projects created in version 3.1 and later: 128 characters
Event and property name limits
- Event name: String type. Must start with a letter and can contain digits, lowercase letters, and underscores "_". Up to 50 characters long
- Property name: String type. Must start with a letter and can contain digits, lowercase letters, and underscores "_". Up to 50 characters long. Only preset properties can start with #.
Data ranges of text, numeric, list, object, and object group properties
- Text: Strings can be up to 2KB
- Numeric: The data range is -9E15 to 9E15
- List: Up to 500 elements. Each element is a string of up to 255 bytes
- Object: Up to 100 child properties
- Object group: Up to 500 objects
Data receiving time limits
- Accepted event time range for server-side data: from 3 years before to 3 days after the server time
- Accepted event time range for client-side data: from 10 days before to 3 days after the server time

4. Other rules

Encode data in UTF-8 to avoid garbled text
Property names in the AE backend support only lowercase letters. We recommend using "_" as the word separator
By default, the AE backend accepts only data from the last three years. Data older than three years can't be ingested. To ingest data from more than three years ago, contact ThinkingAI staff to relax the time limit

5. FAQ

This section summarizes common issues caused by data that doesn't follow the data rules. If you encounter data transfer issues, start troubleshooting with this section

5.1 The AE backend receives no data

If you transfer data through an SDK:

Confirm that the SDK is integrated successfully
Check whether the APPID and the transfer URL are set correctly, and whether the port number or the suffix for the transfer method is missing

If you transfer data through LogBus or the POST method:

Confirm that the APPID and the transfer URL are set correctly, and check whether the port number or the suffix for the transfer method is missing
Check whether the data is transferred in JSON format, and make sure that each line contains one JSON record
Check whether the keys in the data information section start with "#" and whether any required fields are missing
Check whether the types and formats (time format) of the values in the data information section are correct
Check whether the value of "#event_name" follows the rules and doesn't contain characters such as Chinese characters or spaces
Don't start the "properties" key itself with "#" (don't write it as "#properties")

Also note that setting user properties doesn't generate action records. If you upload only data such as user_set, you can't query the data directly in the behavior analysis models of the backend (except SQL IDE)
Pay attention to the time of the uploaded data. Data that's too old (more than three years) isn't ingested. If the uploaded data is historical, the query time range may not cover the time of the uploaded data; adjust the query time range

5.2 Data is missing and some properties weren't received

Make sure that the property keys in the data body section follow the rules and don't contain characters such as Chinese characters or spaces
Make sure that keys starting with "#" in the properties of the data body section are preset properties
Check whether the type of a missing property at upload matches the type of that property in the backend. You can view the types of received properties in Metadata management in the backend

5.3 Data was transferred incorrectly and you want to delete it

Users of on-premises deployments can delete data themselves with the data deletion tool. Cloud service users can contact ThinkingAI staff to delete data
If the data changes significantly, we recommend creating a new project directly. We also recommend that you fully test your data in a test project before you formally transfer data

Was this page helpful?

1. Data structure​

1.1 Data information section​

1.1.1 User information (#account_id and #distinct_id)​

1.1.2 Data type information (#type and #event_name)​

1.1.3 Trigger time (#time)​

1.1.4 Trigger location (#ip)​

1.1.5 Unique data ID (#uuid)​

1.2 Data body section​

2. Data processing rules​

2.1 Receiving new event data​

2.2 Add event properties​

2.3 Handling property inconsistencies​

2.4 Deprecate event properties​

2.5 Properties shared by multiple events​

2.6 User table operation logic​

2.6.1 Overwrite user properties (user_set)​

2.6.2 Initialize user properties (user_setOnce)​

2.6.3 Accumulate user properties (user_add)​

2.6.4 Clear user property values (user_unset)​

2.6.5 Add elements to list user properties (user_append)​

2.6.6 Delete a user (user_del)​

2.6.7 Add elements to deduplicated list user properties (user_uniq_append)​

3. Data limits​

4. Other rules​

5. FAQ​

5.1 The AE backend receives no data​

5.2 Data is missing and some properties weren't received​

5.3 Data was transferred incorrectly and you want to delete it​

1. Data structure

1.1 Data information section

1.1.1 User information (`#account_id` and `#distinct_id`)

1.1.2 Data type information (`#type` and `#event_name`)

1.1.3 Trigger time (`#time`)

1.1.4 Trigger location (`#ip`)

1.1.5 Unique data ID (`#uuid`)

1.2 Data body section

2. Data processing rules

2.1 Receiving new event data

2.2 Add event properties

2.3 Handling property inconsistencies

2.4 Deprecate event properties

2.5 Properties shared by multiple events

2.6 User table operation logic

2.6.1 Overwrite user properties (user_set)

2.6.2 Initialize user properties (user_setOnce)

2.6.3 Accumulate user properties (user_add)

2.6.4 Clear user property values (user_unset)

2.6.5 Add elements to list user properties (user_append)

2.6.6 Delete a user (user_del)

2.6.7 Add elements to deduplicated list user properties (user_uniq_append)

3. Data limits

4. Other rules

5. FAQ

5.1 The AE backend receives no data

5.2 Data is missing and some properties weren't received

5.3 Data was transferred incorrectly and you want to delete it