Configure an S3 data source
Amazon S3 (Simple Storage Service) is an object storage service provided by Amazon Web Services (AWS) and one of the most popular cloud storage solutions in the world.
Limitations
Currently, only the following configuration is supported:
- Reading from S3 data sources (offline read) and writing to AE Built-in Warehouse workspace tables
- Writing from AE Built-in Warehouse workspace tables to S3 (offline write)
Note:
- Writing directly from S3 to databases other than the AE Built-in Warehouse isn't supported yet
Supported field types
Offline reading (Reader)
When reading, field type support depends on the file format. The names in the table below are file field types. CSV, Text, JSON, ORC, and Parquet are file formats, not field types.
| File format | Supported field types for reading | Unsupported types or limitations |
|---|---|---|
| CSV, Text | Text type string | Fields are not parsed as separate numeric, date, Boolean, or complex types. |
| ORC | boolean, tinyint, smallint, int, bigint, float, double, decimal, string, varchar, char, date, timestamp, array, map, struct | Binary fields are not supported. Complex fields must have valid field structures. |
| Parquet | boolean, int_8, int_16, int32, int64, float, double, decimal, string, date, timestamp, int96, list, map, struct | Binary fields are not supported. int96 is a legacy time encoding; support for dates, timestamps, and complex fields depends on the actual parsing and validation results. |
| JSON | string, int, bigint, float, double | Nested objects or arrays are not read as native structured fields. For Boolean fields and other unlisted native types, use the actual parsing and validation results. |
Organize JSON files by line, with one complete object per line. Field parsing refers to the first line of the file. Ensure that field structures are consistent across lines, and check the parsing results and data preview.
Offline writing (Writer)
For writing, the table below lists configurable file field types. It does not mean that any source field can be written directly. Use types that can be selected in the field configuration and pass validation.
| File format | Supported field types for writing | Unsupported types or limitations |
|---|---|---|
| CSV, Text | Text type string | Separate numeric, date, Boolean, or complex field types are not provided. See the file writing rules below. |
| ORC | boolean, tinyint, smallint, int, bigint, float, double, decimal, string, varchar, char, date, timestamp, array, map, struct | Binary fields are not supported. Complex fields must have valid field structures. |
| Parquet | boolean, int_8, int_16, int32, int64, float, double, decimal, string, date, timestamp, int96, list, map, struct | Binary fields are not supported. int96 is a legacy time encoding; use the actual field configuration and validation results. |
| JSON | string, int, bigint, float, double, array, map, struct | Separate binary, date, timestamp, or Boolean field types are not provided. Complex fields must pass field configuration validation. |
Create an S3 data source
In the DataOps Platform - Integration module, you can choose to add an S3 data source.
Data source configuration parameters
Fill in the configuration required by the data source and pass the connectivity test to create the S3 data source.
Parameters whose names start with * are required; parameters without * are optional.
| Field name | Description |
|---|---|
| Basic Information | |
| *Datasource Name | Must be unique within the DataOps Platform space. Can contain only letters, digits, and underscores, and can't start with a digit or an underscore |
| Remarks | Optional |
| Data source configuration | |
| not distinguish environment / Independent setting | Choose one: not distinguish environment means the production and development environments share one configuration; Independent setting means the two environments are configured independently |
| *Endpoint | Access address of S3 |
| *Region | For example, ap-northeast-1 |
| *Bucket | Address of the S3 bucket, for example: S3://seatunnel-test |
| *Access Key | Access Key used for access |
| *Access Secret | Access Secret used for access |
| Advanced | Other advanced parameters required to connect to the database; you can customize them |
Parameters whose names start with * are required; parameters without * are optional.
Create an offline sync task
After you create the S3 data source and pass the connectivity test as described above, you can configure an S3 offline read task for your scenario.
S3 as the Data Source or Data Target
Select S3 as the Data Source and configure the following parameters:
| Field name | Description |
|---|---|
| *Source Type | Select S3 as the type of the Data Source |
| *Datasource Name | An S3 data source registered on the data source management page; select it from the drop-down list. If you haven't created the data source yet, click the Data sources management button to create an S3 data source. |
| *File path | Reads files stored in S3 object storage through the file path
|
| *File type | Supports the orc, csv, text, parquet, and json types |
| Advanced | Empty directory marked succeeded: When the switch is on, the task is considered successful if the file directory is empty |
Parameters whose names start with * are required; parameters without * are optional.
Select S3 as the Data Target and configure the following parameters:
| Field name | Description |
|---|---|
| Data Source | Select S3 as the type of the Data Target, and select an S3 data source registered on the data source management page from the drop-down list |
| Written Directory | The S3 directory to write data to. Enter it after the preset prefix. It can't start with /
|
| File type | Supports the orc, csv, text, parquet, and json types. If you select csv or text, you can configure file writing rules. See "CSV and Text file writing rules" below |
| Write mode | Defaults to Insert into directory. Options:
|
CSV and Text file reading rules
When OSS is the Data Source and you select the csv or text file type, expand File Reading Rules and configure Column delimiter and Skip header. The default column delimiter is a comma, and Skip header defaults to No. If the file contains a header that shouldn't be read as data, select Yes.
- csv: The column delimiter can be a single valid character, such as a comma, semicolon, or vertical bar; you can also enter
\tfor Tab. Except for\t, multi-character delimiters aren't supported, and double quotes, carriage returns, and line breaks can't be used as delimiters. For example, to split columns by a vertical bar, enter|, not\|. - text: The column delimiter is parsed as a regular expression; for example,
\tmeans Tab,\s+means consecutive whitespace, and::means two consecutive colons. To split columns by a vertical bar or a period, enter\|or\.respectively, so that they aren't interpreted as special characters in the regular expression.
CSV and Text handle delimiters differently. After you switch the file type or change the reading rules, check the data preview and field mapping again.
CSV and Text file writing rules
When OSS is the Data Target and you select the csv or text file type, expand File Written Rule. The page provides the Column delimiter and Skip header options, which default to a comma and No respectively. The column delimiter box accepts up to 8 characters.
Write settings are independent of read settings, so don't apply the regular expression rules of the read side directly to write settings. After you use a custom delimiter or adjust the header option, first check the output file through manual execution to confirm that the delimiter and header are as expected, and only then use the settings for production sync.
Field mapping
After configuring the data source and the target, create field mappings. The system automatically syncs data from source fields to target fields based on the mappings. You can configure field mappings in three ways:
- Method 1: Custom selection. Select a source table field, then select the target field in the target table
- Method 2: Name Mapping. The system automatically maps fields with the same name in the source and target tables
- Method 3: Line Mapping. The system automatically maps fields in the same row
Note that each target field can correspond to only one source field
Basic information settings
Finally, set the basic information of the integration plan, including the plan name, owner, synchronization rate, and remarks.
When you're done, click Save to create the integration plan.
Note: The plan name can't be changed after it's saved
The details page looks like this:
Mount an offline sync plan on a Flow
Mount on Flow
1. Start mounting
- On the integration plan details page, click Mount on task flow in the upper-right corner
2. Select a Flow
-
Select the target Flow from the drop-down menu
-
To create a new Flow:
- Click the New Flow shortcut button below the drop-down menu
- Or go to the Dev module to create one
-
Tip: If the target Flow isn't shown, click the ↻ refresh button on the right
3. Create a sync node
- In the Flow, create a node of the Offline sync plan type
- The node is linked to the current integration plan. When the node runs, it triggers a run of that integration plan
4. Complete mounting
- Click Create node and mount on it to complete the configuration
- After mounting succeeds, click Go to flow page to view the result right away
Note: The task node created by mounting is in the unreleased state. We recommend going to the Flow and releasing the node.
Unmount from a Flow
To unmount an offline sync plan from a Flow:
- If the Flow hasn't been released yet, go to the Flow that the plan is mounted on and delete the Offline sync plan task node in Dev Mode.
- If the Flow has already been released, after deleting the Offline sync plan task node in Dev Mode, release the Flow again. This also unmounts the node from the Production environment.

