TopOn data integration solution
Last updated: 2022-08-17
1. Preparation before integration
Note that data generated by third-party data integration counts toward the cluster's data consumption
1. Overview
This document describes how to send TopOn data back to Agentic Engine (hereinafter the AE system). This solution supports the following three methods. Click an interface name to jump to the corresponding section:
| Interface | Description | API type | Productized | Data update frequency |
|---|---|---|---|---|
| Device-level data report API | Aggregated data | Pull | Yes | Only data from 2 days ago or earlier (T2) |
| Full report | Aggregated data | Pull | No | Same-day data available (T0) |
| Client SDK reporting | User-level details | Client SDKs | - | Real time |
Before you start connecting TopOn data, make sure you have read the AE system data rules and understand AE's data structure. We also recommend that you give the information needed to pull data to our customer success manager, using the format in the data integration configuration templates (Device-level data report API, Full report).
2. Device-level data report API (productized)
Basic interface information
| Interface | API type | Productized | Data granularity | Attribution | Cost | Revenue | Impressions | Clicks | Conversions |
|---|---|---|---|---|---|---|---|---|---|
| Device-level data report API | Pull | Yes | User level | Yes | Yes | Yes |
The device-level data report API gets aggregated metrics by user dimension, including users' total impressions, clicks, and revenue over a period of time.
2.1 API permissions
Before pulling data, ask your TopOn contact to enable the device-level data report API permission. After it is enabled, you can view the Publisher Key on the account information page of the developer dashboard. Send the Publisher Key and the app ID of the project in the TopOn dashboard to ThinkingAI staff, or enter them in the data integration configuration template.
Get the Publisher Key on the account information page
View the app ID of the TopOn project on the app page
2.2 Client SDK configuration
For cost reasons, the user_id property in the report data is not returned by default for Android apps. If you need it, first upgrade the SDK to 5.9.70 or later, and contact TopOn operations staff about the permission
Option 1 (automatic integration):
If the AE SDK version you integrate is 2.8.0~2.8.1, we recommend the automatic association option
If the AE SDK version you integrate is 2.8.2 or later, you also need to install the third-party data plugin
This option is an automatic integration option. After you initialize the AE client SDK, call the following code to enable it. For details, see Android SDK third-party data and iOS SDK third-party data
// Initialize the AE SDK
ThinkingAnalyticsSDK instance = ThinkingAnalyticsSDK.sharedInstance(this, TA_APP_ID, TA_SERVER_URL);
// Enable TopOn ID association
instance.enableThirdPartySharing(TDThirdPartyShareType.TD_TOP_ON);
// Initialize the TopOn SDK
// ...
// After you change the distinct ID, sync the data again (optional).
instance.identify("distinct_id");
instance.enableThirdPartySharing(TDThirdPartyShareType.TD_TOP_ON);
This option works by automatically calling the initCustomMap method of ATSDK internally and passing in ATCustomRuleKeys.USER_ID, with the distinct ID of the AE project as the value.
Option 2 (manual integration):
For manual integration, use TopOn's app-wide global custom rule settings to pass AE's #distinct_id into user_id in custom_rule of the TopOn SDK. For the calling method and code examples, see this document.
iOS client code sample:
[[ATAPI sharedInstance] setCustomData:@{kATCustomDataUserIDKey:self.TA_DISTINCT_ID}];
Android client code sample:
// Get the AE distinct ID, which corresponds to #distinct_id in AE
String te_distinct_id = instance.getDistinctId();
Map<String, String> customMap = new HashMap<>();
customMap.put(ATCustomRuleKeys.USER_ID, te_distinct_id);
ATSDK.initCustomMap(customMap);
Note (very important):
Reporting through the setCustomData (iOS) method or initCustomMap (Android) must be completed before the TopOn SDK is initialized; otherwise, some user_id values may fail to be sent back.
2.3 Data pull
2.3.1 Included fields
The device-level data report API includes the following fields
| Field | Type | Remarks |
|---|---|---|
| placement_id | String | Ad placement ID |
| placement_name | String | Ad placement name |
| placement_format | String | Ad type: 0: native; 1: rewarded_video; 2: banner; 3: interstitial; 4: splash |
| android_id | String | Device ID, androidid |
| gaid | String | Google advertising device ID |
| idfa | String | iOS device ID |
| area | String | Country |
| impression | Numeric | Number of impressions |
| click | Numeric | Clicks |
| revenue | Numeric | Revenue, split to the device level based on the revenue of the third-party ad platforms. The currency is the same as configured in the developer dashboard |
ecpm | Numeric | eCPM calculated by TopOn from the revenue split by device impressions based on the revenue API and the device impressions counted by TopOn. Formula: (device revenue / device impressions counted by TopOn) * 1000. Note: eCPM is provided with a 2-day delay |
| is_abtest | String | Control group or test group: 0: control group, or A/B testing not enabled; 1: test group |
| traffic_group_id | String | Control group or test group ID |
| segment_id | String | Traffic segment ID |
| segment_name | String | Traffic segment name |
| idfv | String | iOS device ID |
| oaid | String | Android device ID |
| user_id | String | Developer's custom user ID |
| network_firm_id | String | Ad platform ID |
| network_firm | String | Ad platform name |
| currency | String | Currency of the developer account. USD means US dollars, and CNY means Chinese yuan |
| os_version | String | OS version of the iOS device |
| att_status | String | ATT authorization status of the iOS device: 0: Not determined (authorization not yet decided); 1: Restricted; 2: Denied; 3: Authorized |
| imei | String | Android device identifier |
| device_type | String | iOS device type. Enum values: 0: non-iOS device; 1: iphone; 2: ipad |
| brand | String | Device brand name |
| model | String | Device model |
| app_vn | String | App version name |
| app_vc | String | App version code |
| new_user_type | String | New user type. Enum values: 1: new user; 2: not a new user |
| channel | String | Channel, passed in by the developer through the TopOn SDK |
| estimate_revenue | decimal(18,6) | Estimated revenue. For bidding ad sources, the estimated revenue is the sum of real-time ad impression prices; for non-bidding ad sources, it is the manually entered eCPM price * the impressions counted by TopOn |
2.3.2 API parameters
-
Time:
- Data is pulled with a specific day as the start time, and the start day must be at least 2 days ago
- The time zone can be UTC 0, -8, or +8. If it is not passed, the time zone of the developer account is used by default
-
App：
- You need to specify the App to pull data from by providing its app ID in the TopOn dashboard
2.3.3 Data ingestion rules
By default, we write the pulled data to the AE project as events:
- user_id in the data is used as the distinct ID of the data. This field should correspond to the distinct ID in the AE project
- The start time field of the pulled data is used as the #event_time of the event
- The event name is ta_ad_revenue_topon
- All other fields are ingested
2.4 Data integration configuration template
After reading the documentation above, we recommend that you fill in the following template and send it to your customer success manager at ThinkingAI:
Data interface: TopOn device-level data report API
--------
Company name: XXX
AE project environment: (SAAS/on-premises)
AE project name: XXX
AE project APP ID: XXX
Data receiving URL push_url: XXX
---------
TopOn App ID：XXX
TopOn Publisher Key：XXX
---------
Pull time zone: UTC+8 (enum values: UTC-8, UTC+8, UTC+0; if not passed, the time zone of the developer account is used by default)
Time range for historical data pull: starting from yyyy/mm/dd (at least 2 days ago)
Scheduled pull: pull the data of the past X days at X:00 every day
3. Full report
Basic interface information
| Interface | API type | Productized | Data granularity | Attribution | Cost | Revenue | Impressions | Clicks | Conversions |
|---|---|---|---|---|---|---|---|---|---|
| Full report | Pull | No | Aggregated data | Yes | Yes | Yes |
The full report refers to the full report data in the TopOn data report query API. It provides aggregated ad monetization data, including impression, click, and revenue metrics.
3.1 API permissions
Before pulling data, ask your TopOn contact to enable the data report query API permission. After it is enabled, you can view the Publisher Key on the account information page of the developer dashboard. Send the Publisher Key and the app ID of the project in the TopOn dashboard to ThinkingAI staff, or enter them in the data integration configuration template.
Get the Publisher Key on the account information page
View the app ID of the TopOn project on the app page
3.2 Data pull
3.2.1 Grouping dimensions
The following table shows all grouping dimensions supported by the full report query API. Note: When you query data from the last 10 days, you can select up to 6 grouping dimensions; when you query data from more than 10 days ago, you can select up to 3 grouping dimensions:
| Group-by dimension | Field | Type | Default | Remarks |
|---|---|---|---|---|
| date | date | String | Yes | Date, in the format YYYYmmdd |
app | app_id | String | Yes | App ID in the developer dashboard |
| app_name | String | Yes | App name | |
| app_platform | String | Yes | OS platform of the app | |
| app_pkg_name | String | Yes | Package name of the app | |
| placement | placement_id | String | Yes | Ad placement ID in the developer dashboard |
| placement_name | String | Yes | Ad placement name | |
adformat | adformat | String | Ad format. Enum values: Rewarded Video, Interstitial, Banner, Native, Splash | |
| area | area | String | Country (region) code | |
network | network | String | Ad platform account ID | |
| network_name | String | Ad platform account name | ||
adsource | adsource_network | String | Yes | Name of the ad platform that the ad source belongs to |
| adsource_token_position_id | String | Yes | Position ID of the ad source | |
| adsource_token_orientation | String | Yes | Orientation of the ad source | |
| adsource_token_video_muted | String | Yes | Whether the ad is muted | |
| adsource_token_app_id | String | Yes | App ID of the ad source | |
| adsource_token_app_name | String | Yes | App name of the ad source | |
| adsource_id | String | Yes | Ad source ID | |
| adsource_name | String | Yes | Ad source name | |
| network_firm_id | network_firm_id | String | Yes | Ad platform ID |
| network_firm | String | Yes | Ad platform name | |
| scenario | scenario_id | String | Ad scenario ID | |
| scenario_name | String | Ad scenario name | ||
traffic_group | traffic_group_id | String | Traffic group ID | |
| traffic_group_name | String | Traffic segment name | ||
| traffic_group_segment_id | String | Numeric segment ID of the traffic group. Note: For the default segment, segment_id = 0 and is not returned | ||
| channel | channel | String | Channel Name | |
| sdk_version | sdk_version | String | SDK version | |
| app_version | app_version | String | App version |
3.2.2 Metric fields
By default, we ingest all of the following fields. To adjust them, note the changes in the data integration configuration template:
| Field | Type | Remarks |
|---|---|---|
| time_zone | String | Time zone. Enum values: UTC+8, UTC+0, UTC-8 |
| currency | String | Currency of the developer account. The revenue formed by this field and the revenue field must match the revenue in the developer dashboard reports |
| new_users | Numeric | New users |
| new_user_rate | Numeric | Percentage of new users |
| day2_retention | Numeric | Day-1 retention |
| deu | Numeric | DEU |
| engaged_rate | Numeric | Penetration rate |
| imp_dau | Numeric | Impressions / DAU |
| imp_deu | Numeric | Impressions / DEU |
| impression_rate | Numeric | Impression rate |
| dau | Numeric | Returned only depending on the group_by conditions |
| arpu | Numeric | Returned only when dau is returned |
| request | Numeric | Requests |
| fillrate | Numeric | Fill rate |
| impression | Numeric | Number of impressions |
| click | Numeric | Clicks |
| ctr | Numeric | Click-through rate |
| ecpm | Numeric | eCPM calculated by TopOn from the actual revenue pulled from the ad platforms through the report API and the impressions counted by TopOn. Formula: (revenue / impressions counted by TopOn) *1000. Note: eCPM is provided with a 1-day delay |
| revenue | Numeric | Revenue of the third-party ad platform, in the currency of the developer account |
| request_api | Numeric | Requests of the third-party ad platform |
| fillrate_api | Numeric | Fill rate of the third-party ad platform |
| impression_api | Numeric | Impressions of the third-party ad platform |
| click_api | Numeric | Clicks of the third-party ad platform |
| ctr_api | Numeric | Click-through rate of the third-party ad platform |
| ecpm_api | Numeric | eCPM API calculated by TopOn from the actual revenue pulled from the ad platforms through the report API and the impression API. Formula: (revenue / impression API) *1000. Note: eCPM API is provided with a 1-day delay |
| estimate_revenue | Numeric | Estimated revenue, in US dollars |
estimate_revenue_ecpm | Numeric | Estimated eCPM calculated from the estimated revenue and the impressions counted by TopOn. Formula: (estimated revenue / impressions counted by TopOn) *1000. Note: 1. Estimated eCPM is provided on the same day; 2. For regular ad sources, it is calculated based on the manually entered eCPM price, and for bidding ad sources, it is calculated based on the real-time bidding price |
| ready_request | Numeric | Number of isReady calls |
| ready_rate | Numeric | isReady success rate |
| cy_estimate_revenue | Numeric | Estimated revenue returned in the currency of the developer account |
| cy_estimate_revenue_ecpm | Numeric | Estimated eCPM returned in the currency of the developer account, calculated in the same way as estimate_revenue_ecpm |
| load | Numeric | Traffic requests. Note: when certain group_by dimensions (such as network_firm_id) are selected, the response is "0" |
| load_fillrate | Numeric | Traffic fill rate. Note: when certain group_by dimensions (such as network_firm_id) are selected, this metric is not returned in the response |
3.2.3 API parameters
-
Time:
- Data is pulled by day
- The time zone can be UTC 0, -8, or +8. If it is not passed, the time zone of the developer account is used by default
3.2.4 Data ingestion rules
By default, we write the pulled data to the AE project as events:
- Because the full report query API returns aggregated data, we use a fixed value as its user identifier. You can think of all data as attached to a single virtual user
- The date field in the data, that is, the date of the data, is set as the #event_time of the aggregated data
- The event name is topon_fullreport
- All other fields are stored
3.3 Data integration configuration template
After reading the documentation above, we recommend that you fill in the following template and send it to your customer success manager at ThinkingAI:
Data interface: TopOn full report query API
--------
Company name: XXX
AE project environment: (SAAS/on-premises)
AE project name: XXX
AE project APP ID: XXX
Data receiving URL push_url: XXX
---------
TopOn App ID：XXX
TopOn Publisher Key：XXX
---------
Pull time zone: UTC+8 (enum values: UTC-8, UTC+8, UTC+0)
Analysis dimensions: xxx, xxx
Metrics to pull: xxx, xxx
Time range for historical data pull: starting from yyyy/mm/dd (at least 2 days ago)
Scheduled pull: pull the data of the last X days at X:00 every day
4. Client SDK reporting
Basic interface information
| Interface | API type | Productized | Data granularity | Attribution | Cost | Revenue | Impressions | Clicks | Conversions |
|---|---|---|---|---|---|---|---|---|---|
| ATAdInfo callback API | Client SDKs | No | User-level details | Yes | Yes |
The TopOn client SDK provides the ATAdInfo callback API. You can use methods such as getEcpm() to get the estimated eCPM value and other related dimensions and metrics, and then report them to the AE system through the AE SDK.
Taking the Android SDK as an example, the following code sample receives revenue data and passes it to AE:
ATBannerView mBannerView = new ATBannerView(this);
mBannerView.setBannerAdListener(new ATBannerExListener() {
@Override
public void onBannerShow(ATAdInfo entity) {
JSONObject properties = new JSONObject();
try{
properties.put("id", entity.getShowId());
properties.put("publisher_revenue", entity.getPublisherRevenue());
properties.put("currency", entity.getCurrency());
properties.put("country", entity.getCountry());
properties.put("adunit_id", entity.getTopOnPlacementId());
properties.put("adunit_format", entity.getTopOnAdFormat());
properties.put("precision", entity.getEcpmPrecision());
properties.put("network_type", entity.getAdNetworkType());
properties.put("network_placement_id", entity.getNetworkPlacementId());
properties.put("ecpm_level", entity.getEcpmLevel());
properties.put("segment_id", entity.getSegmentId());
properties.put("scenario_id", entity.getScenarioId());
properties.put("scenario_reward_name", entity.getScenarioRewardName());
properties.put("scenario_reward_number", entity.getScenarioRewardNumber());
properties.put("channel", entity.getChannel());
properties.put("sub_channel", entity.getSubChannel());
properties.put("custom_rule", entity.getCustomRule());
properties.put("network_firm_id", entity.getNetworkFirmId());
properties.put("adsource_id", entity.getAdsourceId());
properties.put("adsource_index", entity.getAdsourceIndex());
properties.put("adsource_price", entity.getEcpm());
properties.put("adsource_isheaderbidding", entity.isHeaderBiddingAdsource());
properties.put("ext_info", entity.getExtInfoMap());
}catch(JSONException e){
}
// Report the revenue data to AE with the event name topon_sdk_ad_info
instance.track("topon_sdk_ad_info", properties);
}
});
After the data is reported, you can analyze the related dimensions and metrics in the AE system based on the event named topon_sdk_ad_info.
5. Integration testing and data usage
5.1 Integration testing
You can look for the following events in Real-time saved data, Events Analysis, or SQL IDE:
- Device-level report API data: ta_ad_revenue_topon
- Full report query API data: topon_fullreport
- Client SDK real-time callback API: topon_sdk_ad_info
5.2 Data usage
- Add up the in-app purchases reported by the AE SDK and the ad monetization revenue obtained through TopOn for a more complete ROI analysis
Divide the revenue generated by non-organic users acquired through ads (including subscription payments, app download payments, in-app event payments, and ad monetization) by the user acquisition cost. Because the calculation requires user-level data, use the device-level data report API.
- Create the "Channel source" custom event property
Use custom properties to combine the attribution data in event properties and user properties into a custom event property, and then use it as a group-by item
channel_final = coalesce(channel(cost), network_name)
- Calculate user-level ad revenue in real time with the estimated eCPM
The accuracy of TopOn's estimated eCPM meets analysis needs, and its timeliness is also valuable. You can consider using the estimated eCPM returned by the TopOn client SDK real-time callback API to calculate user-level ad revenue in the AE system.
| adsource_price | String | Estimated eCPM calculated from the estimated revenue and the impressions counted by TopOn. Formula: (estimated revenue / impressions counted by TopOn) *1000. Note: 1. The estimated eCPM from the client SDK real-time callback API is provided on the same day; 2. For regular ad sources, it is calculated based on the manually entered eCPM price, and for bidding ad sources, it is calculated based on the real-time bidding price |
|---|
- Use the ad source ID (adsource_id) to calculate the revenue of each impression
- Data types of event properties after integration
To make data use more flexible, event properties are ingested as strings by default. You can use the custom property feature of the AE system to convert string fields to other types, for example:
- Convert the revenue property to a numeric type: "revenue" (select numeric as the data type)
- Apply a global filter where app_pkg_name has a value to topon_fullreport
When you use topon_fullreport data, you can apply a global filter where app_pkg_name has a value to exclude noise data
- Get the ATT authorization status of iOS devices
You can get the ATT authorization status of a device through the iOS device-level API:
0: Not determined (authorization not yet decided); 1: Restricted; 2: Denied; 3: Authorized
6. FAQ
How do I distinguish data from the TopOn device-level API and the full report API?
You can distinguish them by event name: ta_ad_revenue_topon is device-level data, and topon_fullreport is full report data