Offline Sync
Overview
After you create a data source, you can create an offline sync plan in the Offline Sync module to bring data from the data source into the DataOps Platform.
You can sync data in the following ways:
- Sync now: Create a task in the Offline Sync module to import data from the data source into the platform
- Scheduled sync: Mount the task on a Flow and configure a scheduling strategy to sync automatically
Manage offline sync plans
The Offline Sync list page shows all offline sync plans in the current workspace, and you can also add new offline sync plans there.
The table on the Offline Sync list page has the following columns:
| Column Name | Description |
|---|---|
| Solution Name | User-defined name of the offline sync integration plan. Click the name to open the integration plan details page and view its configuration |
| Source | Source data source of the integration plan, shown with the data source type logo and name. Filtering is supported. Click the data source name to view the data source details |
| Target | Target data source of the integration plan, shown with the data source type logo and name. Filtering is supported. Click the data source name to view the data source details |
| Associated Table | Name of the workspace table that the integration plan reads from or writes to. Click it to view the table details |
| Mount on Flow | Name of the Flow that the integration plan is mounted on. Click it to view the Flow details. If no Flow is mounted, "-" is shown |
| Last Schedule Time | Time of the last scheduled execution after the plan is mounted on a Flow. If no Flow is mounted, "-" is shown |
| Last Schedule Status | Status of the last scheduled execution after the plan is mounted on a Flow. Filtering is supported. If no Flow is mounted, Not Executed is shown |
| Manual Execution Time | Time of the last execution while the plan isn't mounted on a Flow |
| Manual Execution Status | Status of the last execution while the plan isn't mounted on a Flow |
| Create Time (hidden by default) | Time the plan was created |
| Owner (hidden by default) | Owner of the plan |
| Last Modified Time (hidden by default) | Time the plan was last modified |
| Last Modifier (hidden by default) | User who last modified the plan |
| Remarks | User-defined remarks |
| Action - Edit | Click to open the integration plan edit page and modify its configuration |
| Action - Delete | Deletes the integration plan. Note that to keep data integration safe and stable, an integration plan mounted on a Flow can't be deleted. To delete it, unmount it first. For details, see Unmount from a Flow. |
Create an integration solution
The Create offline sync task section in each data source configuration guide describes the plan creation process in detail. This section outlines the general process.
Click + Integration plan in the upper-right corner of the list page to start adding an integration plan:
First, in the Data Source section, select the data source of the integration plan from the two-level Source Type + Data Source drop-down. If the data source you created isn't shown in the drop-down menu, click the refresh button on the right to refresh it.
After selecting a data source, specify how to read the data files or tables. These settings depend on the data source, and different data sources require different information. Complete the data source configuration based on your situation:
Note that some input boxes under Data Source (such as File path in the image above) support parameter expressions, such as the base time parameter ${bd}, so that each run reads the data for a specific day.
Next, configure the Data Target, that is, the target table to write data to: first select the target data source (such as Built-in Warehouse → Workspace Schema), then select the database and table in Target Space Table. Note that each target table can have only one integration plan, so some tables can't be selected as the target table. To create a new table, go to the Dev module or click the New Workspace Table shortcut at the bottom of the drop-down menu. If the target table you want doesn't appear in the drop-down menu, click the refresh button on the right to refresh the list:
After selecting the target table, configure the write rules. If the selected target table is a partitioned table, specify the partition to write to; for each partition field, you can choose a value acquisition method (constant / field / parameter, such as ${bd}). For the write mode, you can choose Overwrite or Insert New Data.
After configuring Data Source and Data Target, click the parse button in the Field Mapping section to read the table structures of the source and target. If your configuration is correct, you'll see the field information of the source table and target table and can configure the field mapping:
If the integration plan uses parameters:
- Base date parameter ${bd}: Assigned yesterday's date when the mapping is parsed;
- Theoretical scheduling time parameter ${st}: Assigned the current time when the mapping is parsed.
Click Name Mapping or Line Mapping to quickly create mappings between source and target fields, or click Cancel to remove all mappings. To configure a field mapping individually, click the plus sign after a source table field and set the target field in the pop-up window. Note that each target field can correspond to only one source field, so some fields can't be selected.
During data synchronization, the system uses the CAST() function to try to convert source table fields to the types of the target fields. If the conversion fails, the value of the target field is set to NULL
Finally, set the basic information of the integration plan, including the plan name, owner, synchronization rate, and remarks. When you're done, click the save button to create the integration plan.
Integration plan details page
After creating an integration plan, click its name on the list page to open its details page. On the details page, you can view the configuration of the integration plan and perform actions such as manual execution and mounting on a Flow.
Click Manual Execution on the right side of the card, select the required Execution mode (Exclusive mode / Shared mode) in the Manually execute offline sync dialog, and click start execution to trigger an offline sync task. You can view the execution records on the Manual Execution Records tab.
If the integration plan uses parameters, you can set the parameter values for this run when executing it manually. For example, Base Date in the image below corresponds to the parameter ${bd}:
Edit an integration plan
Click the edit button on the list page or details page to open the integration plan edit page. There, you can adjust the data source, Data Target, field mapping, and all other information except the integration plan name. Note that if fields in the original data source or Data Target become abnormal, for example because they were deleted, you must remove all abnormal mappings before you can save. You can fix them manually or click Empty incorrect mapping to remove all problematic field mappings.
Mount on Flow
After creating an integration plan, you can mount it on a Flow to sync on a schedule using the Flow's scheduling capability. On the integration plan details page, click Mount on task flow in the upper-right corner of the card to start mounting:
First, select the Flow to mount on from the drop-down menu. To create a new Flow, click the + Flow shortcut at the bottom of the drop-down menu, or go to the Dev module to create one. If the Flow you want doesn't appear in the drop-down menu, click the refresh button on the right to refresh it.
Next, create a new task node of the Offline sync plan type in the selected Flow. The node points to the current integration plan, and you also need to select the Execution mode (exclusive / shared) when creating it. When this task node is scheduled or run, it's equivalent to debugging or running the integration plan it points to. After configuring the sync node, click Create node and mount on it to complete the mounting.
When mounting is complete, the following dialog appears. Click Go to flow page to view the mounted Flow.
Note that the task node created by mounting hasn't been released yet, so we strongly recommend going to the Flow and releasing the node.
Unmount from a Flow
To unmount a plan from a Flow:
- Go to the Flow that the plan is mounted on, and in Dev Mode, delete the Offline sync plan task node that points to the plan
- Release the Offline sync plan task node deleted in the previous step. Once it's released, the plan is unmounted from the Flow