TaDataWriter plugin

Last updated 10/03/2026

1. Introduction

TaDataWriter enables DataX to transfer data to the AE cluster. The data is sent to the AE receiver.

2. Features and limitations

TaDataWriter converts data from the DataX protocol into the internal data format of the AE cluster. TaDataWriter has the following conventions:

1. Writes only to the AE cluster.
2. Supports data compression. The currently supported compression formats are gzip and snappy.
3. Supports multi-threaded transfer.
4. Can be used only on AE nodes.

3. Features

3.1 Configuration example

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1
      }
    },
    "content": [
      {
        "reader": {
          "name": "streamreader",
          "parameter": {
            "column": [
              {
                "value": "ABCDEFG-123-abc",
                "type": "string"
              },
              {
                "value": "F53A58ED-E5DA-4F18-B082-7E1228746E88",
                "type": "string"
              },
              {
                "value": "login",
                "type": "string"
              },
              {
                "value": "2020-01-01 01:01:01",
                "type": "date"
              },
              {
                "value": "abcdefg",
                "type": "string"
              },
              {
                "value": "2019-08-08 08:08:08",
                "type": "date"
              },
              {
                "value": 123456,
                "type": "long"
              },
              {
                "value": true,
                "type": "bool"
              }
            ],
            "sliceRecordCount": 1000
          }
        },
        "writer": {
          "name": "ta-data-writer",
          "parameter": {
            "type": "track",
            "appid": "34c703a885014208a737911748a7b51c",
            "column": [
              {
                "index": "0",
                "colTargetName": "#account_id",
                "type": "string"
              },
              {
                "index": "1",
                "colTargetName": "#distinct_id"
              },
              {
                "index": "2",
                "colTargetName": "#event_name"
              },
              {
                "index": "3",
                "colTargetName": "#time",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
              },
              {
                "index": "4",
                "colTargetName": "testString",
                "type": "string"
              },
              {
                "index": "5",
                "colTargetName": "testDate",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
              },
              {
                "index": "6",
                "colTargetName": "testLong",
                "type": "number"
              },
              {
                "index": "7",
                "colTargetName": "testBoolean",
                "type": "boolean"
              },
              {
                "colTargetName": "add_clo",
                "value": "addFlag",
                "type": "string"
              }
            ]
          }
        }
      }
    ]
  }
}

3.2 Parameters

• type

  • Description: The type of data to write: user_set or track.
  • Required: Yes
  • Default: None

• appid

  • Description: The appid of the corresponding project.
  • Required: Yes
  • Default: None

• thread

  • Description: The number of threads.
  • Required: No
  • Default value: 3

• compress

  • Description: The text compression type. If you leave it empty, no compression is applied. Supported compression types are gzip and snappy.
  • Required: No
  • Default value: No compression

• connType

  • Description: How data is received inside the cluster: through the receiver, or sent directly to Kafka.
  • Required: No
  • Default value: http

• column

  • Description: The list of fields to read. type specifies the data type. index specifies which column of the reader the current column corresponds to (starting from 0). value makes the current column a constant: instead of reading data from the reader, the column is generated automatically from the value.

You can specify the Column field information as follows:

[
  {
    "type": "Number",
    "colTargetName": "test_col", //Column name for the generated data
    "index": 0 //Get the Number field from the first column transferred from the reader to DataX
  },
  {
    "type": "string",
    "value": "testvalue",
    "colTargetName": "test_col" //Generate a string field with the value testvalue inside TaDataWriter and use it as the current field
  },
  {
    "index": 0,
    "type": "date",
    "colTargetName": "testDate",
    "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
  }
]

• When you specify Column information, you must set either index or value. type is optional. When you set the date type, you can optionally set dateFormat.

  • Required: Yes
  • Default value: All columns are read using the reader types

3.3 Type conversion

The types are defined by TaDataWriter:

DataX internal type	TaDataWriter data type
Int	Number
Long	Number
Double	Number
String	String
Boolean	Boolean
Date	Date