Configure a Databricks data source
The Databricks Data Source transfers table data between Databricks and the platform's preset warehouse through Offline Sync. This article covers Databricks-JDBC connection settings, source and target configuration, supported field types, sync validation, and Flow scheduling.
Limitations and prerequisites
Before you configure the Data Source, check the following:
- The supported directions are Databricks → preset warehouse and preset warehouse → Databricks. This plan does not support direct synchronization between Databricks and other external Data Sources.
- Prepare Databricks compute resources that support JDBC connections and obtain the corresponding JDBC URL and Personal Access Token. The connection details must match the compute resource, workspace, and driver configuration in use. See Databricks JDBC connection configuration.
- The account associated with the token must be allowed to use the compute resource, access the Catalog and database (Schema), and view table metadata. Reading a source table requires read permissions, and writing to a target table requires the corresponding write permissions. Ask the Databricks administrator to grant access for the required operations.
- The platform's runtime environment must be able to reach the service specified by the JDBC URL. If Databricks has network access restrictions, check the actual outbound address of the platform servers and the permitted access scope.
- Prepare the target table and check its field types and partition definitions. For a preset warehouse target, see Create and manage space tables. Prepare Databricks target tables in the target Catalog and database in advance.
| Environment configuration | Requirements |
|---|---|
| not distinguish environment | The Development Environment and Production Environment share one set of connection and credential settings |
| Independent setting | Configure the Development Environment and Production Environment separately; both configurations must have access to the tables selected in the plan |
Data preview uses the Development Environment configuration, while manual execution uses the Production Environment configuration. Check the compute resources, table permissions, and network access for each configuration.
Get the Databricks connection details
- Sign in to your Databricks workspace and open the compute resource to connect. For a cluster, find the connection details under Compute > target cluster > Configuration > Advanced options > JDBC/ODBC. For a SQL warehouse, use SQL Warehouses > target warehouse > Connection details.
- Obtain the JDBC URL and check its server hostname, port, and HTTP Path to ensure that it points to the intended compute resource. For the relevant settings, see Databricks JDBC connection configuration.
- Prepare a PAT that can be used in this workspace. If you need a new token, generate one under Settings > Developer > Access tokens > Manage and set an appropriate lifetime. If you cannot create or use tokens, contact the workspace administrator. See Create a Databricks personal access token.
Transfer tokens securely, and keep credentials out of screenshots, logs, and public documentation. After replacing a token or changing permissions, validate the Data Source connection and sync plan again.
Create a Databricks Data Source
- Go to DataOps Platform > Integration > Data Source and click + Data Source.
- Select Databricks-JDBC, click Next, and enter a Data Source name and remarks.
- Select not distinguish environment or Independent setting. For independent configuration, enter the Development Environment and Production Environment settings separately.
- Enter the connection parameters below. Under Connectivity Test, click Test. After the test passes, click Finish.
| Configuration Item | Description |
|---|---|
| JDBC URL | The JDBC connection address for the Databricks compute resource; confirm the service address and connection parameters with the administrator |
| Authentication mechanism (Acceptance mechanism in the UI) | Select PAT |
| Personal Access Token | Enter a valid token for the connecting account, with the permissions required to read or write the data for this sync |
| Advanced | Enter any required extended settings in JSON format; leave this blank if no additional settings are needed |
After the connectivity test passes, load the tables in the sync plan, preview the source data, and run the plan to validate actual table permissions and read and write results.
Configure Databricks as the source
- Go to DataOps Platform > Integration > Offline Sync and click + Integration plan.
- Under Data Source, select Databricks-JDBC and the Data Source you created.
- Select a Method and configure the source for an entire table or a custom query.
- Click Data Preview to check the source fields, actual values, and read scope.
- To adjust the batch size, expand Advanced.
This figure shows settings from an existing plan and does not replace validation of the current connection and data read. After configuring the plan, use data preview to check the actual results.
| Method | Configuration |
|---|---|
| Entire Table | Select the Catalog, database (Schema), and table in order. To read only part of the data, enable Data Filtering and enter a filter condition |
| Custom Query | Enter a read-only SQL query that returns the fields to synchronize, with any filter conditions included in the query. This method does not use the separate Catalog, database, table, or Data Filtering settings provided for Entire Table |
Data Filtering for Entire Table limits the read scope. Enter a WHERE clause as shown in the editor, for example, WHERE status = 'active', rather than a complete SELECT query. For Custom Query, ensure that the SQL can access the correct Catalog and database, with explicit result field names and recognizable types.
Batch Size defaults to 1000 and supports values from 1000 to 10000. After changing the Data Source, Method, Catalog, database, or table, check the read scope, preview results, and Field Mapping again. If the source structure or query results change, parse the fields again and review the mappings.
Configure the data target
Read Databricks data into the preset warehouse
Under Data Target, select the platform's Preset Warehouse, then select the target space table. If the target table has not been created, go to Tables and create it with the required field and partition definitions. See Create and manage space tables.
| Configuration Item | Description |
|---|---|
| Target table | The preset warehouse space table that receives the synced data |
| Partition Information | For a partitioned target table, enter each partition value as required by the table. You can use space parameters to configure the partitions used at execution time |
| Write mode | Select Append or Overwrite according to the target data retention requirements. Confirm the target table and partitions again before execution |
Write preset warehouse data to Databricks
Configure the preset warehouse source table under Data Source. Under Data Target, select Databricks-JDBC and the Data Source you created, then select the target Catalog, database (Schema), and table in order.
| Configuration Item | Description |
|---|---|
| Write Table | A target table that has already been prepared in Databricks and that the connecting account is allowed to write to |
| Partition Information | If target partition fields are displayed, enter every partition value and check the actual values used at execution time |
| Write mode | Only Append is supported. Data is appended to the target table; before running the plan again, assess whether this will create duplicate records |
| Pre-execution SQL | Optionally enter SQL to execute on the target before synchronization starts; leave this blank if it is not needed |
| Batch Size | Under Advanced; the default is 5000, and the supported range is 1–10000 |
Pre-execution SQL is actually executed on the target. If it fails, the sync is not submitted. The pre-execution SQL and subsequent data writes are not guaranteed to share a transaction, and a later sync failure is not guaranteed to undo SQL that has already executed. Before entering SQL or retrying, check the target environment, its data impact, and the outcome of repeated execution.
The Databricks target uses append mode and does not provide native overwrite or upsert modes. Do not treat a preliminary delete statement followed by an append as an overwrite operation with transaction-wide rollback.
Field Mapping and supported field types
After configuring the source and target, use Field Mapping to map source fields to target fields. You can select fields individually, or use Map by Name or Map by Row and then check each mapping. Each target field can map to only one source field.
The figure shows field correspondences from an existing plan. It does not indicate that live connectivity or this execution has been validated.
Supported field types for reading
When Databricks is the Data Source, the following field types are supported. Type names are case-insensitive. For parameterized fields or nested structures, also check the complete definition.
| Type category | Supported Databricks field types |
|---|---|
| Boolean | boolean |
| Integer | tinyint, smallint, int, integer, bigint |
| Floating-point and fixed-point numbers | float, double, decimal |
| String | string, varchar, char |
| Date and time | date, timestamp, timestamp_ntz |
| Complex types | array, map, and struct; check the complete structure |
binary, unlisted types such as interval, void, and variant, and other unrecognized types are not supported.
Supported field types for writing
When Databricks is the Data Target, the supported Databricks field types are the same as in the table above. binary, unlisted types, and other unrecognized types are not supported.
Create the target table in advance. Support for a field type does not mean that any data will meet the target field's complete definition and null constraints.
Field type limitations
- For
decimal(p,s),varchar(n), andchar(n), check the complete precision, scale, and length requirements. - For
timestampandtimestamp_ntz, check the actual time meaning, time zone, and precision requirements. A type name alone does not establish that results will meet expectations. - For
array<T>,map<K,V>, andstruct<...>, check the element, key-value, or field definitions, including field names and order, null constraints, and nesting levels. Support for a top-level type does not mean that every internal structure is supported. For example, support forarrayalone does not establish support forarray<binary>. - After the source or target table structure changes, parse the fields again and check their correspondences. A field being visible in the data preview does not mean that its type is supported for synchronization.
Saving, validation, and Workflow Schedule
Save and validate the sync results
- Check the source table or query, filter scope, target table and partitions, write mode, pre-execution SQL, and Field Mapping. First use the source Data Preview to check fields and sample values.
- Enter the plan name, owner, and remarks, then click Save.
- Click Manual Execution, select an execution mode as required by the dialog, check the current parameter values, and start execution. Manual execution uses the Production Environment configuration; the execution mode does not switch between the Development Environment and Production Environment.
- Check the status and logs under Manual Execution Records. After the task succeeds, check the data volume, field correspondence, and sample numeric, time, complex-type, and null values in the target table.
With independent environment configurations, successfully loading or previewing data in the Development Environment does not mean that production credentials and read/write permissions are ready. Before the first execution, check both configurations and the tables, partitions, and parameters in the target environment. For appends to Databricks or plans with pre-execution SQL, also assess how retries affect existing data.
Mount on a Flow
After manual validation passes, you can mount the plan on a Flow and run it on a schedule.
- Open the saved plan, click Mount on task flow, and select the target Flow. If it has not been created, see Create a Flow.
- Enter the node name, execution mode, owner, and other details, then click Create and Mount.
- Click Go to Flow, check the node and its upstream and downstream dependencies, and verify the tables, partitions, and space parameters used for scheduling.
- Complete the Flow schedule settings, save, and publish the Flow. Mounting the node does not mean that it has been published online.
After a plan is mounted on a Flow, it can no longer be run manually from the plan details page. Execute and validate the node from the Flow instead.
FAQ
| Symptom | What to check |
|---|---|
| Connectivity test fails | Check the JDBC URL, PAT validity, compute resource access permissions, and the network access scope of the platform servers |
| The connection passes, but the table cannot be found or previewed | Check the Catalog, database and table names, and metadata and read permissions. For Custom Query, also check the SQL and its access scope |
| Preview works, but execution fails | Check production credentials, target write permissions, tables and partitions, parameters, and pre-execution SQL, and review the execution log |
| Duplicate data appears after repeated execution | Check the append scope and previous execution results to determine whether the same batch of data was read and written more than once |

