TaCustomReader plugin

Last updated 10/03/2026

1. Introduction

The TaCustomReader plugin enables DataX to read data from AE. Under the hood, TaCustomReader connects to the remote AE database through JDBC and runs the corresponding SQL statements to SELECT data from the AE database.

2. How it works

In short, TaCustomReader connects to the remote AE database through a JDBC connector, generates a SELECT SQL query statement based on the information you configure, and sends it to the AE cluster. It then assembles the results returned by the SQL into an abstract dataset using DataX's custom data types and passes the dataset to the downstream Writer for processing.

3. Features

3.1 Configuration example

Configure a job that reads data from the AE cluster and prints it to the console:

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1
      }
    },
    "content": [
      {
        "reader": {
          "name": "ta-custom-reader",
          "parameter": {
            "querySql": "select * from v_event_1 where \"$part_date\" = '2020-01-01'"
          }
        },
        "writer": {
          "name": "streamwriter",
          "parameter": {
            "print": true,
            "encoding": "UTF-8"
          }
        }
      }
    ]
  }
}

3.2 Parameters

• server

  • Description: The connection information for the peer AE database, in the form SERVER:PORT.
  • Required: No
  • Default value: Follows the cluster configuration

• querySql

  • Description: In some business scenarios, you can use this configuration item to customize the filtering SQL. Once you configure this item, its content is used directly to filter data. For example, to sync data after joining multiple tables, use select a,b from table_a join table_b on table_a.id = table_b.id
  • Required: Yes
  • Default: None

3.3 Type conversion

Currently, TaCustomReader supports most Presto types, but some individual types aren't supported. Make sure to check your types.

The following is the list of Presto type conversions that TaCustomReader uses for the AE cluster:

DataX internal type	Presto data type
Long	TINYINT, SMALLINT, INTEGER, BIGINT
Double	REAL, DOUBLE, DECIMAL
String	VARCHAR, CHAR, VARBINARY, JSON
Date	DATE, TIME, TIMESTAMP
Boolean	BOOLEAN

Note: Field types other than those listed above are not supported.