Partition table sync
Partition table sync updates the partition metadata of space tables with the partitions that can be discovered in external storage. With automatic sync on, the system checks for partition changes periodically; you can also start a sync for the current environment right away from the table details.
Feature description and scope
Partition sync applies to Hive external partition tables in Catalog of the DataOps Platform. The target table must meet all of the following conditions:
- The management mode is Workspace-Task table;
- The storage type is External Table;
- The partition type is Partition Table, and the table contains at least one partition field.
Internal tables, non-partitioned external tables, and views don't show the automatic sync setting or the manual sync entry. Manual sync also requires the current user to have permission to edit Catalog tables.
Enable automatic partition sync
When you create an eligible external partition table, Automatically synchronize partitions is on by default. You can also change this setting when you edit the table later.
- Go to DataOps Platform > Catalog, and create an external partition table, or open an existing external partition table and click Edit.
- In Form Mode, expand Advanced Setting; when editing with DDL, find the storage configuration area.
- Select or clear Automatically synchronize partitions.
- Click Save only or Release to apply the setting.
The automatic sync setting is shared by the development and production environments. When it's on, the system periodically checks for partition changes in both environments; when it's off, you can still sync partitions manually.
View the sync status
- Go to DataOps Platform > Catalog and open the external partition table you want to view.
- At the top of the page, select Dev environment or Production environment.
- Open the Partition tab.
The page shows whether automatic sync is on, as well as the result (success or failure) and start time of the most recent sync in the current environment. If a sync fails, hover over the tooltip icon to view the failure reason.
The development and production environments each keep their own most recent sync result. When you switch environments, the partition list and sync status switch accordingly, while the automatic sync setting stays the same.
Sync partitions manually
To update the partitions of the current environment immediately, click Sync now on the Partition tab of the table details.
After the request is submitted, the page shows Partition sync submitted, and the button changes to Syncing. This message means the system has accepted the request, not that the sync is complete. When the sync finishes, the page updates to Latest sync successful or Latest sync failed and refreshes the partition list of the current environment.
Manual sync only processes the environment currently selected on the page. Turning off automatic sync doesn't affect manual sync. If you don't have permission to edit Catalog tables, the page doesn't show Sync now. If a sync task is already running for the table, wait for it to finish and then try again.
Sync rules and notes
| Rule | Description |
|---|---|
| Sync scope | Sync registers partitions newly added in external storage and cleans up the metadata of partitions that no longer exist in external storage. |
| Data files | Sync only adjusts partition metadata, and doesn't move, modify, or delete data files in external storage. |
| Environment | The automatic sync setting is shared by the development and production environments; recent results and partition lists are shown separately for each environment; manual sync only applies to the current environment. |
| Timing | Saving the automatic sync setting doesn't trigger a sync immediately. To update partitions right away, click Sync now. |
| Successful result | A successful sync means that the currently discoverable partition directories are aligned with the partition metadata. It doesn't mean that external data writes are complete or that the data is readable. |
Partition sync may remove partition metadata whose directories no longer exist in external storage. Before you run a manual sync, confirm that external writes are complete and that the platform can access the target directories normally.