RESTful API user guide
This guide describes how to use the data ingestion API. With the data ingestion API, you can send data directly to the ThinkingAnalytics backend with the HTTP POST method, without relying on transfer tools or SDKs.
Before you start the integration, read Data rules. After you're familiar with the data format and data rules of AE, read this guide to complete the integration.
Data uploaded with the POST method must follow AE's data format
1. Data format conversion
Before uploading data, you first need to convert it to the AE data format. Each piece of AE data is a JSON object. A data sample is shown below (formatted for readability):
{
"#account_id": "ABCDEFG-123-abc",
"#distinct_id": "F53A58ED-E5DA-4F18-B082-7E1228746E88",
"#type": "track",
"#ip": "192.168.171.111",
"#time": "2017-12-18 14:37:28.527",
"#event_name": "test",
"properties": {
"#lib": "LogBus",
"#lib_version": "1.0.0",
"#screen_height": 1920,
"#screen_width": 1080,
"argString": "abc",
"argNum": 123,
"argBool": true
}
}
For the detailed data format specification, see the Data format section.
2. Data reporting
Once your JSON data is ready, you can report it. The AE backend accepts standard HTTP POST requests, and all API data uses UTF-8 character encoding. The calls are as follows:
2.1 Data receiving API (form-data submission)
If you use the cloud service, enter the following URL:
https://global-receiver-ta.thinkingdata.cn/sync_data
If you use an on-premises deployment, enter the following URL:
http://your-data-collection-address/sync_data
2.1.1 Request parameters (in the requestBody)
-
For a single JSON record:
- Parameter 1: appid=the APPID of your project
- Parameter 2: data=JSON data, UTF-8 encoded, must be urlencoded
- Parameter 3: client=0, defaults to 0. When set to 1, the IP of the reporting client is used as the #ip field (forced replacement)
-
For multiple records:
- Parameter 1: appid=the APPID of your project
- Parameter 2: data_list=data in JSONArray format containing multiple JSON records, UTF-8 encoded, must be urlencoded
- Parameter 3: client=0, defaults to 0. When set to 1, the IP of the reporting client is used as the #ip field (forced replacement)
Note: Libraries in some languages urlencode data automatically, in which case you don't need to urlencode it again, for example, the requests library in Python3 and request tests in Postman
The following uses curl to demonstrate how to call the RESTful API. The source data is as follows
{
"#account_id": "testing",
"#time": "2019-01-01 10:00:00.000",
"#type": "track",
"#event_name": "testing",
"properties": {
"test": "test"
}
}
First urlencode the data above
%7b%22%23account_id%22%3a%22testing%22%2c%22%23time%22%3a%222019-01-01+10%3a00%3a00.000%22%2c%22%23type%22%3a%22track%22%2c%22%23event_name%22%3a%22testing%22%2c%22properties%22%3a%7b%22test%22%3a%22test%22%7d%7d
Add the parameters and report the data
curl "http://receiver:9080/sync_data" --data "appid=test-sdk-appid&data=%7b%22%23account_id%22%3a%22testing%22%2c%22%23time%22%3a%222019-01-01+10%3a00%3a00.000%22%2c%22%23type%22%3a%22track%22%2c%22%23event_name%22%3a%22testing%22%2c%22properties%22%3a%7b%22test%22%3a%22test%22%7d%7d"
2.1.2 Response parameters
If the response contains code: 0, the data was transferred successfully
2.1.3 Debug mode
You can add the debug parameter to the upload parameters (that is, three upload parameters: appid, data\data_list, and debug). It is optional and off by default
Enable Debug mode only when uploading a small amount of test data. Don't enable Debug mode in the production environment
When debug=1, the response shows the detailed error reason, for example:
{"code":-1,"msg":"#time字段格式不对,需传递[yyyy-MM-dd HH:mm:ss]或者[yyyy-MM-dd HH:mm:ss.SSS]格式"}
2.2 Data receiving API (raw submission)
If you use the cloud service, enter the following URL:
https://global-receiver-ta.thinkingdata.cn/sync_json
If you use an on-premises deployment, enter the following URL:
http://your-data-collection-address/sync_json
2.2.1 Request parameters (in the requestBody)
- For a single JSON record:
{
"appid": "debug-appid",
"debug": 0,
"data": {
"#type": "track",
"#event_name": "test",
"#time": "2019-11-15 11:35:53.648",
"properties": { "a": "123", "b": 2 },
"#distinct_id": "1111"
}
}
- For multiple records:
[
{
"appid": "debug-appid",
"data": {
"#type": "track",
"#event_name": "test",
"#time": "2019-11-15 11:35:53.648",
"properties": { "a": "123", "b": 2 },
"#distinct_id": "1111"
}
},
{
"appid": "debug-appid",
"data": {
"#type": "track",
"#event_name": "test",
"#time": "2019-11-15 11:35:53.648",
"properties": { "a": "123", "b": 2 },
"#distinct_id": "1111"
}
}
]
2.2.2 Response parameters
If the response contains code: 0, the data was transferred successfully
2.2.3 Debug mode
You can add the debug parameter to the upload parameters (that is, add the debug parameter to the JSON; currently only single-record uploads support debug). It is optional and off by default
Enable Debug mode only when uploading a small amount of test data. Don't enable Debug mode in the production environment
When debug=1, the response shows the detailed error reason, for example:
{
"code": -1,
"msg": "#time字段格式不对,需传递[yyyy-MM-dd HH:mm:ss]或者[yyyy-MM-dd HH:mm:ss.SSS]格式"
}
2.2.4 Data compression
To upload compressed data, add the compress field to the RequestHeader. For example, if you pass compress=gzip, the server decompresses the data with gzip. Supported compression methods are gzip and snappy. Data is not compressed by default.
2.2.5 Get the reporting client IP
Add client=1 to the RequestHeader, and the server uses the IP of the reporting client as the #ip field (forced replacement). The default value is 0, which means the reporting client IP is not used for replacement
3. FAQ
See the data rules FAQ to troubleshoot data transfer errors caused by data format issues.