Configure a MaxCompute data source
The MaxCompute Data Source reads table data from Alibaba Cloud MaxCompute through Offline Sync and writes it to the platform's preset warehouse. This article covers connection settings, source Schema and partitions, Field Mapping, supported field types, sync validation, and Flow scheduling.
Limitations and prerequisites
Before you configure the Data Source, check the following:
- MaxCompute can only be used as a data source, with the platform's preset warehouse as the target. This plan cannot write data back to MaxCompute or directly to other external Data Sources.
- Prepare the MaxCompute project's End Point and Project name, together with the Access Key Id and Access Key Secret used to connect.
- The connection account must have access to the project, be able to view tables and their metadata, and be able to read the tables or partitions to be synced. Ask the project administrator to grant permissions for the required read scope. See MaxCompute permissions.
- Before reading tables in a custom Schema, confirm that the source Project supports Schemas and grant the connection account access to the relevant Schema and tables. For source-side preparation, see MaxCompute Schema operations.
- If download control is enabled for the project, the required Tunnel download permissions must also be granted according to the project's requirements. See Download permission control.
- The platform's runtime environment must be able to reach the configured MaxCompute service endpoint. If the project uses an IP allowlist, check the platform servers' actual outbound IP addresses.
- Prepare the target table in the preset warehouse, and check its field types and partition definitions. For creation instructions, see Create and manage space tables.
| Environment configuration | Requirements |
|---|---|
| not distinguish environment | The Development Environment and Production Environment share one set of project and credential settings |
| Independent setting | Configure the Development Environment and Production Environment separately; both configurations must have access to the Schema, tables, and partitions with the same names selected in the plan |
Data preview uses the Development Environment configuration, while manual execution uses the Production Environment configuration. Check the project, table permissions, and network access for each configuration.
The MaxCompute object hierarchy is Project > Schema > Table. Configure the Project in the Data Source connection, and select the Schema and table in the sync plan. Different Schemas within the same Project can contain tables with the same name, so check the Schema to confirm which table will be read. For a traditional two-level Project, select existing tables through the logical default Schema in the platform; selecting default does not upgrade the Project to support custom Schemas.
Transfer the Access Key Secret securely, and keep credentials out of screenshots, logs, and public documentation. After replacing credentials or changing project permissions, validate the Data Source connection and sync plan again.
Create a MaxCompute Data Source
- Go to DataOps Platform > Integration > Data Source and click + Data Source.
- Select MaxCompute, click Next, and enter a Data Source name and remarks.
- Select not distinguish environment or Independent setting. For independent configuration, enter the Development Environment and Production Environment settings separately.
- Enter the connection parameters below. Under Connectivity Test, click Test. After the test passes, click Finish.
| Configuration Item | Description |
|---|---|
| End Point | The service endpoint for the region where the MaxCompute project is located; confirm it with the project administrator |
| Access Key Id | The connection account's AccessKey ID |
| Access Key Secret | The AccessKey Secret corresponding to the AccessKey ID |
| Project name | The name of the MaxCompute Project to read from; the connection account must have permission to access it. Do not enter a Schema or table name here |
| Advanced | Enter any required extended settings in JSON format; leave this blank if no additional settings are needed |
After the connectivity test passes, load the source table, preview the data, and run the sync plan to validate table permissions, partition scope, and the actual data read.
The same Data Source can read from different Schemas that the connection account can access within the Project. Select the Schema under Source Table in each sync plan; there is no need to recreate the Data Source for each Schema.
Configure MaxCompute as the source
- Go to DataOps Platform > Integration > Offline Sync and click + Integration plan.
- Under Data Source, select MaxCompute and the Data Source you created.
- Under Source Table, select the Schema on the left, then select a table in that Schema on the right. When creating a plan, you must select the Schema manually first, even if
defaultis the only option. - If the source is a partitioned table, enter a value for each partition field under Partition Information. Partition values can use space parameters; wildcards are not supported. For parameter details, see Parameter management.
- Click Data Preview to check the fields, actual values, and read scope of the source table in the current Schema.
- To adjust the batch size, expand Advanced.
Select the Schema and table name separately. For example, to read project_a.ods.events, set the project name in the Data Source to project_a, select ods as the source Schema, and select events as the table. Do not enter the fully qualified name as the table name.
| Configuration Item | Description |
|---|---|
| Data source name | A MaxCompute Data Source already created in the current space |
| Schema | The Schema to read from; select default for a traditional two-level Project. Before reading from another Project, check the Data Source connection settings and permissions |
| Data table name | A table in the selected Schema; the list displays table names and marks partitioned tables |
| Partition Information | Enter each partition value when the source is a partitioned table; when using space parameters, confirm that the values resolved at execution time correspond to existing partitions |
| Batch size | An advanced setting with a default value of 1000; the supported range is 1000–10000 |
Changing the Schema clears the previously selected source table. Changing the Schema or table also clears the existing partition settings and source Field Mapping. Select the table again, enter the partition values, parse the fields, and check the preview results and mappings. Tables with the same name in different Schemas may have different fields, so do not reuse the previous mappings without checking them.
If fields are added, deleted, or changed in the source table, parse the fields again and review the mapping. When editing an existing plan, check the Schema, table, and partitions shown before saving your changes.
Configure the data target
Under Data Target, select the platform's Preset Warehouse, then select the target space table. If the target table has not been created, go to Tables and create it with the required field and partition definitions. See Create and manage space tables.
| Configuration Item | Description |
|---|---|
| Target table | The preset warehouse space table that receives the synced data |
| Partition Information | For a partitioned target table, configure partition values as required by the target table; check source and target partitions separately |
| Write mode | Select Append or Overwrite; Overwrite is selected by default, so adjust it to meet your data retention requirements |
Before execution, confirm the target table, target partition, and write mode again to avoid overwriting data that must be retained. Source and target field names can differ, but their correspondence must be defined through Field Mapping.
Field Mapping and supported field types
After configuring the source and target, use Field Mapping to map source fields to target fields. You can select fields individually, or use Map by Name or Map by Row and then check each mapping. Each target field can map to only one source field.
Field Mapping uses the fields parsed from the currently selected Schema and table. After switching the Schema or table, parse the fields again, establish their correspondences, and check whether the fields are supported.
The figure shows field correspondences from an existing plan. It does not indicate that live connectivity or this execution has been validated.
Supported field types
When MaxCompute is the Data Source, the following field types are supported. Type names are case-insensitive. For parameterized fields or nested structures, also check the complete definition.
| Type category | Supported MaxCompute field types |
|---|---|
| Boolean | BOOLEAN |
| Integer | TINYINT, SMALLINT, INT, BIGINT |
| Floating-point and fixed-point numbers | FLOAT, DOUBLE, DECIMAL |
| Strings and JSON | STRING, VARCHAR, CHAR, JSON |
| Date and time | DATE, TIMESTAMP, TIMESTAMP_NTZ, DATETIME |
| Complex types | ARRAY, MAP, and STRUCT; check the complete structure |
BINARY and other unrecognized types are not supported. This plan supports reading from MaxCompute only and does not provide a field type scope for writing to MaxCompute.
Field type limitations
- For
DECIMAL(p,s),VARCHAR(n), andCHAR(n), check the complete precision, scale, and length requirements. - For date-time fields, check the actual time meaning, time zone, and precision requirements. Support for the basic type alone does not establish that results will meet expectations.
- For
ARRAY<T>,MAP<K,V>, andSTRUCT<...>, check the complete element, key-value, or field definitions and null constraints. Support for a top-level type does not mean that arbitrary nested structures or element types are supported. - Schema and partitions determine the objects and scope to read; they do not change field type support conditions. After the source table structure changes, parse the fields again and check their correspondences.
- A field being visible in the data preview does not mean that its type is supported for synchronization. Run validation after saving, and check the actual results in the target table.
Sync validation and Flow scheduling
Save and validate the sync results
- Check the source Project, Schema, table, and partitions, as well as the target table and partitions, write mode, and Field Mapping.
- Enter the plan name, owner, and remarks, then click Save.
- Click Manual Execution, select the execution mode as prompted, and check the parameter values before starting. Manual execution uses the Production Environment configuration; the execution mode does not switch between the Development Environment and Production Environment.
- Check the execution status and logs under Manual Execution Records. After the task succeeds, check the data volume, field correspondence, and sample numeric, time, complex-type, and null values in the target table.
With independent environment configurations, successfully loading or previewing data during development does not mean that production credentials and read permissions are ready. Before the first execution, check both configurations and the Schemas, tables, partitions, and parameters in their respective Projects.
Mount on a Flow
After manual validation passes, you can mount the plan on a Flow and run it on a schedule.
- Open the saved plan, click Mount on task flow, and select the target Flow. If it has not been created, see Create a Flow.
- Enter the node name, execution mode, owner, and other details, then click Create and Mount.
- Click Go to Flow, check the node and its upstream and downstream dependencies, and verify the source partitions and space parameters used for scheduling.
- Complete the Flow schedule settings, save, and publish the Flow. Mounting the node does not mean that it has been published online.
After a plan is mounted on a Flow, it can no longer be run manually from the plan details page. Execute and validate the node from the Flow instead.
FAQ
| Symptom | What to check |
|---|---|
| Connectivity test fails | Check the End Point, project name, AccessKey, platform server network access, and IP allowlist |
| The connection passes, but the Schema list is empty or fails to load | Check whether the source Project supports Schemas, along with account permissions and network access. If the list fails to load, refresh and retry; do not assume that selecting default will resolve the failure |
| The source table cannot be found or previewed after a Schema is selected | Confirm that the table is in the current Schema, and check the account's project membership and permissions to access the Schema and read the table and metadata. If download control is enabled, also check the required download permissions |
| The fields or data in a table with the same name differ from expectations | Check the Project in the Data Source and the Schema in the plan, select the table again, parse the fields, and review the mappings |
| Partition read fails or returns no data | Check that the partitions exist, all partition values are complete, and the space parameters resolve to the intended values |
| Preview works, but execution fails | Check the Production Environment configuration and confirm that its Project contains the Schema, tables, and partitions with the same names. Then check credentials, permissions, the target table, and parameters, and review the execution log |

