Secondary development tool
1. Overview
In version 2.8, the former data deletion tool was merged into the secondary development tool.
ta-tool is the secondary development tool for AE private deployment clusters. By default, it is deployed on the first three servers (on a single-node system, it is always deployed). It provides a range of secondary development features for your convenience. The current features are:
- Data deletion
- Property type modification and deletion
- Event data deduplication
- Import custom data, which can be joined with the event table and user table through SQL
- Delete custom tables
- Generate query secrets, with which you can call the query API using the POST method
- Set up scheduled scripts that query data in the AE backend through SQL and write it back to the AE database
- Set up scheduled scripts that use the DataX engine to write data from the data sources it supports to the AE database
- Data archiving and data reloading
2. Usage
Log in to a server where the secondary development tool is deployed (by default, the first three servers in a distributed deployment), run the su - ta command to switch to the ta user, and run ta-tool to view the command help. The output is as follows:
su - ta
ta-tool
usage: ta-tool <command> [options]
Commands:
user_event_import [-conf <configPath >|--configration <configPaht>] [-d <date>|--date <date>] Data backfill
data_transfer [-conf <configPath >|--configration <configPaht>] [-d <date>|--date <date>] Import custom data into the TA system
drop_table [-t <table>|--table <table>] Delete the specified table
generate_api_secret [-appid <appid>] Generate the api_secret for the appid
generate_root_secret Generate the root query secret
data_archive [start|stop|retry] [-jobid <jobid>] Archive command (start starts a job, stop stops a job, retry reruns a failed job and requires the jobid)
data_reload [start|stop|retry] [-jobid <jobid>] Reload command (start starts a job, stop stops a job, retry reruns a failed job and requires the jobid)
datax_engine [-conf <configPath >|--configration <configPaht>] Import data into the TA system using the DataX engine
update_user_by_foreignkey [-conf <configPath >|--configration <configPaht>] External user property association import
data_del Data deletion
dupevent_del [-hashNum <hashNum>|-needIncludeServerTime <true|false>] Duplicate event deduplication
modify_meta Metadata modification
Options:
-appid,--appid <appid> Set the appid parameter
-conf,--configration <configration> Set the configuration file path
-d,--date <date> Set the date parameter
-h,--help Show the help document and exit
-jobid,--jobid <jobid> Job ID, used to rerun the specified job when archiving/reloading fails
-t,--table <table> Set the table name to delete the specified table
Examples:
ta-tool --help Show help information
ta-tool user_event_import -conf /data/home/ta/import_configs/*.json [-d 2019-08-01] Import custom user and event data into the TA system
ta-tool data_transfer -conf /data/home/ta/dt_configs/*.json [--date 2019-08-01] Import a custom table into the TA system
ta-tool drop_table -t test Delete the test table
ta-tool generate_api_secret -appid test-appid Generate an api_secret
ta-tool generate_root_secret Generate the root query secret
ta-tool data_archive start Data archiving
ta-tool data_reload start Data reloading
ta-tool data_archive retry -jobid jobid_1234 Retry a failed data archiving job
ta-tool data_reload retry -jobid jobid_1234 Retry a failed data reloading job
ta-tool datax_engine -conf /data/home/ta/datax_configs/stream2stream.json Call the DataX engine
ta-tool data_del Data deletion
ta-tool dupevent_del Event data deduplication
ta-tool modify_meta Metadata modification
3. Feature list
The following are the main features of the secondary development tool. Click a link to view the documentation for that feature:
5.External user property association import