Skip to main content

Configure an OSS data source

Last updated 11/03/2026

OSS (Object Storage Service) is a massive, secure, and highly reliable cloud storage service suitable for storing unstructured data.

Limitations​

Currently, only the following configuration is supported:

  • Reading from OSS (offline read) and writing to AE Built-in Warehouse workspace tables
  • Writing from AE Built-in Warehouse workspace tables to OSS (offline write)

Note:

  • Writing directly from OSS to databases other than the AE Built-in Warehouse isn't supported yet

Supported field types​

Offline reading (Reader)

When reading, field type support depends on the file format. The names in the table below are file field types. CSV, Text, JSON, ORC, and Parquet are file formats, not field types.

File formatSupported field types for readingUnsupported types or limitations
CSV, TextText type stringFields are not parsed as separate numeric, date, Boolean, or complex types.
ORCboolean, tinyint, smallint, int, bigint, float, double, decimal, string, varchar, char, date, timestamp, array, map, structBinary fields are not supported. Complex fields must have valid field structures.
Parquetboolean, int_8, int_16, int32, int64, float, double, decimal, string, date, timestamp, int96, list, map, structBinary fields are not supported. int96 is a legacy time encoding; support for dates, timestamps, and complex fields depends on the actual parsing and validation results.
JSONstring, int, bigint, float, doubleNested objects or arrays are not read as native structured fields. For Boolean fields and other unlisted native types, use the actual parsing and validation results.

Organize JSON files by line, with one complete object per line. Field parsing refers to the first line of the file. Ensure that field structures are consistent across lines, and check the parsing results and data preview.

Offline writing (Writer)

For writing, the table below lists configurable file field types. It does not mean that any source field can be written directly. Use types that can be selected in the field configuration and pass validation.

File formatSupported field types for writingUnsupported types or limitations
CSV, TextText type stringSeparate numeric, date, Boolean, or complex field types are not provided. See the file writing rules below.
ORCboolean, tinyint, smallint, int, bigint, float, double, decimal, string, varchar, char, date, timestamp, array, map, structBinary fields are not supported. Complex fields must have valid field structures.
Parquetboolean, int_8, int_16, int32, int64, float, double, decimal, string, date, timestamp, int96, list, map, structBinary fields are not supported. int96 is a legacy time encoding; use the actual field configuration and validation results.
JSONstring, int, bigint, float, double, array, map, structSeparate binary, date, timestamp, or Boolean field types are not provided. Complex fields must pass field configuration validation.

Create an OSS data source​

In the DataOps Platform - Integration module, you can choose to add an OSS data source.

Data source configuration parameters

Fill in the configuration required by the data source and pass the connectivity test to create the OSS data source.

Field nameDescription
Basic Information
*Datasource NameMust be unique within the DataOps Platform space. Can contain only letters, digits, and underscores, and can't start with a digit or an underscore
RemarksOptional
Data source configuration
not distinguish environment / Independent settingChoose one: not distinguish environment means the production and development environments share one configuration; Independent setting means the two environments are configured independently
*EndpointAccess address of OSS
*RegionFor example, oss-cn-shanghai
*BucketAddress of the OSS bucket, for example: oss://seatunnel-test
*Access KeyAccess Key used for access
*Access SecretAccess Secret used for access
AdvancedOther advanced parameters required to connect to the database; you can customize them

Parameters whose names start with * are required; parameters without * are optional.

Create an offline sync task​

After you create the OSS data source and pass the connectivity test as described above, you can configure an OSS offline read task for your scenario.

OSS as the Data Source or Data Target​

Select OSS as the Data Source and configure the following parameters:

Field nameDescription
*Source TypeSelect OSS as the type of the Data Source
*Datasource NameAn OSS data source registered on the data source management page; select it from the drop-down list.
If you haven't created the data source yet, click the Data sources management button to create an OSS data source.
*File path

Reads files stored in OSS object storage through the file path

  • The file path supports space parameters
  • The file path supports the wildcard * (not supported in Data Target paths yet)
  • You can view the full path information and preview data
*File typeSupports the orc, csv, text, parquet, and json types
AdvancedEmpty directory marked succeeded: When the switch is on, the task is considered successful if the file directory is empty

Parameters whose names start with * are required; parameters without * are optional.

Select OSS as the Data Target and configure the following parameters:

Field nameDescription
Data SourceSelect OSS as the type of the Data Target, and select an OSS data source registered on the data source management page from the drop-down list
Written Directory

The OSS directory to write data to. Enter it after the preset prefix. It can't start with /

  • The written directory supports space parameters (click Parameter)
  • The written directory doesn't support wildcards
  • Click Full Path preview to view the full path information
File typeSupports the orc, csv, text, parquet, and json types. If you select csv or text, you can configure file writing rules. See "CSV and Text file writing rules" below
Write mode

Defaults to Insert into directory. Options:

  • Insert into directory (Insert Into)
  • Overwrite directory (Insert Overwrite): Deletes all files and subdirectories in the written directory first, then writes new data files
  • Error if exist (Error If Exist)

CSV and Text file reading rules​

When OSS is the Data Source and you select the csv or text file type, expand File Reading Rules and configure Column delimiter and Skip header. The default column delimiter is a comma, and Skip header defaults to No. If the file contains a header that shouldn't be read as data, select Yes.

  • csv: The column delimiter can be a single valid character, such as a comma, semicolon, or vertical bar; you can also enter \t for Tab. Except for \t, multi-character delimiters aren't supported, and double quotes, carriage returns, and line breaks can't be used as delimiters. For example, to split columns by a vertical bar, enter |, not \|.
  • text: The column delimiter is parsed as a regular expression; for example, \t means Tab, \s+ means consecutive whitespace, and :: means two consecutive colons. To split columns by a vertical bar or a period, enter \| or \. respectively, so that they aren't interpreted as special characters in the regular expression.

CSV and Text handle delimiters differently. After you switch the file type or change the reading rules, check the data preview and field mapping again.

CSV and Text file writing rules​

When OSS is the Data Target and you select the csv or text file type, expand File Written Rule. The page provides the Column delimiter and Skip header options, which default to a comma and No respectively. The column delimiter box accepts up to 8 characters.

Write settings are independent of read settings, so don't apply the regular expression rules of the read side directly to write settings. After you use a custom delimiter or adjust the header option, first check the output file through manual execution to confirm that the delimiter and header are as expected, and only then use the settings for production sync.

Field mapping​

After configuring the data source and the target, create field mappings. The system automatically syncs data from source fields to target fields based on the mappings. You can configure field mappings in three ways:

  • Method 1: Custom selection. Select a source table field, then select the target field in the target table
  • Method 2: Name Mapping. The system automatically maps fields with the same name in the source and target tables
  • Method 3: Line Mapping. The system automatically maps fields in the same row

Note that each target field can correspond to only one source field

Basic information settings​

Finally, set the basic information of the integration plan, including the plan name, owner, synchronization rate, and remarks.

When you're done, click Save to create the integration plan.

Note: The plan name can't be changed after it's saved

The details page looks like this:

Mount an offline sync plan on a Flow​

Mount on Flow​

1. Start mounting

  • On the integration plan details page, click Mount on task flow in the upper-right corner

2. Select a Flow

  • Select the target Flow from the drop-down menu

  • To create a new Flow:

    • Click the New Flow shortcut button below the drop-down menu
    • Or go to the Dev module to create one
  • Tip: If the target Flow isn't shown, click the ↻ refresh button on the right

3. Create a sync node

  • In the Flow, create a node of the Offline sync plan type
  • The node is linked to the current integration plan. When the node runs, it triggers a run of that integration plan

4. Complete mounting

  • Click Create node and mount on it to complete the configuration
  • After mounting succeeds, click Go to flow page to view the result right away
tip

Note: The task node created by mounting is in the unreleased state. We recommend going to the Flow and releasing the node.

Unmount from a Flow​

To unmount an offline sync plan from a Flow:

  • If the Flow hasn't been released yet, go to the Flow that the plan is mounted on and delete the Offline sync plan task node in Dev Mode.
  • If the Flow has already been released, after deleting the Offline sync plan task node in Dev Mode, release the Flow again. This also unmounts the node from the Production environment.
Was this page helpful?