TaJsonWriter plugin

Last updated 10/03/2026

1. Introduction

TaJsonWriter writes files in AE JSON format to the local file system. TaJsonWriter is for users who need to restore AE cluster data as JSON text.

2. Features and limitations

TaJsonWriter converts data from the DataX protocol into local files in AE JSON format. TaJsonWriter has the following conventions:

1. Writes only JSON text files in AE format.
2. Supports multi-threaded writing. Each thread writes to a different sub-file.

3. Features

3.1 Configuration example

{
  "job": {
    "setting": {
      "speed": {
        "channel": 1
      }
    },
    "content": [
      {
        "reader": {
          "name": "streamreader",
          "parameter": {
            "column": [
              {
                "value": 123123,
                "type": "long"
              },
              {
                "value": "123123",
                "type": "string"
              },
              {
                "value": "login",
                "type": "string"
              },
              {
                "value": "2019-08-16 08:08:08",
                "type": "date"
              },
              {
                "value": "2019-08-16 08:08:08",
                "type": "date"
              },
              {
                "value": "2222",
                "type": "string"
              },
              {
                "value": "2019-08-16 08:08:08",
                "type": "date"
              },
              {
                "value": "test",
                "type": "bytes"
              },
              {
                "value": true,
                "type": "bool"
              }
            ],
            "sliceRecordCount": 100
          }
        },
        "writer": {
          "name": "ta-json-writer",
          "parameter": {
            "type": "event",
            "path": "/data/export/ta_datafile/",
            "filename": "test",
            "column": [
              {
                "index": "0",
                "colTargetName": "#user_id"
              },
              {
                "index": "1",
                "colTargetName": "#distinct_id"
              },
              {
                "index": "2",
                "colTargetName": "#event_name"
              },
              {
                "index": "3",
                "colTargetName": "#time",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
              },
              {
                "index": "4",
                "colTargetName": "#event_time",
                "type": "string"
              },
              {
                "index": "5",
                "colTargetName": "#account_id",
                "type": "string"
              },
              {
                "index": "6",
                "colTargetName": "timetest",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
              },
              {
                "index": "6",
                "colTargetName": "timetest2",
                "type": "date",
                "dateFormat": "yyyy-MM-dd HH:mm:ss"
              },
              {
                "index": "7",
                "colTargetName": "os_1",
                "type": "string"
              },
              {
                "index": "7",
                "colTargetName": "os_2",
                "type": "string"
              },
              {
                "index": "8",
                "colTargetName": "booleantest",
                "type": "boolean"
              },
              {
                "index": "0",
                "colTargetName": "testNumber",
                "type": "number"
              },
              {
                "colTargetName": "add_clo",
                "value": "123123",
                "type": "string"
              }
            ]
          }
        }
      }
    ]
  }
}

3.2 Parameters

• path

  • Description: The path in the local file system. TaJsonWriter writes multiple files under the Path directory.
  • Required: Yes
  • Default: None

fileName

• Description: The name of the file that TaJsonWriter writes. A random suffix is added to this file name to form the actual file name that each thread writes to.
• Required: Yes
• Default: None

• writeMode

  • Description: How TaJsonWriter cleans up data before writing:

    • truncate: Before writing, deletes all files in the directory whose names start with the fileName prefix.
    • append: Does nothing before writing. DataX TaJsonWriter writes directly using the filename and makes sure file names don't conflict.
    • nonConflict: Reports an error if the directory contains files with the fileName prefix.

  • Required: Yes

  • Default value: append

• encoding

  • Description: Encoding of the files to read.
  • Required: No
  • Default value: utf-8

• column

  • Description: The list of fields to read. type specifies the data type. index specifies which column of the reader the current column corresponds to (starting from 0). value makes the current column a constant: instead of reading data from the reader, the column is generated automatically from the value.

You can specify Column field information with the following configuration:

[
  {
    "type": "Number",
    "colTargetName": "test_col", //Column name for the generated data
    "index": 0 //Get the Number field from the first column transferred from the reader to DataX
  },
  {
    "type": "string",
    "value": "testvalue",
    "colTargetName": "test_col" //Generate a string field with the value testvalue inside TaDataWriter and use it as the current field
  },
  {
    "index": 0,
    "type": "date",
    "colTargetName": "testDate",
    "dateFormat": "yyyy-MM-dd HH:mm:ss.SSS"
  }
]

3.3 Type conversion

The types are defined by TaJsonWriter:

DataX internal type	TaJsonWriter data type
Int	Number
Long	Number
Double	Number
String	String
Boolean	Boolean
Date	Date