Skip to main content

Configure a Databricks data source

Last updated 11/03/2026

The Databricks Data Source transfers table data between Databricks and the platform's preset warehouse through Offline Sync. This article covers Databricks-JDBC connection settings, source and target configuration, supported field types, sync validation, and Flow scheduling.

Limitations and prerequisites​

Before you configure the Data Source, check the following:

  • The supported directions are Databricks → preset warehouse and preset warehouse → Databricks. This plan does not support direct synchronization between Databricks and other external Data Sources.
  • Prepare Databricks compute resources that support JDBC connections and obtain the corresponding JDBC URL and Personal Access Token. The connection details must match the compute resource, workspace, and driver configuration in use. See Databricks JDBC connection configuration.
  • The account associated with the token must be allowed to use the compute resource, access the Catalog and database (Schema), and view table metadata. Reading a source table requires read permissions, and writing to a target table requires the corresponding write permissions. Ask the Databricks administrator to grant access for the required operations.
  • The platform's runtime environment must be able to reach the service specified by the JDBC URL. If Databricks has network access restrictions, check the actual outbound address of the platform servers and the permitted access scope.
  • Prepare the target table and check its field types and partition definitions. For a preset warehouse target, see Create and manage space tables. Prepare Databricks target tables in the target Catalog and database in advance.
Environment configurationRequirements
not distinguish environmentThe Development Environment and Production Environment share one set of connection and credential settings
Independent settingConfigure the Development Environment and Production Environment separately; both configurations must have access to the tables selected in the plan

Data preview uses the Development Environment configuration, while manual execution uses the Production Environment configuration. Check the compute resources, table permissions, and network access for each configuration.

Get the Databricks connection details​

  1. Sign in to your Databricks workspace and open the compute resource to connect. For a cluster, find the connection details under Compute > target cluster > Configuration > Advanced options > JDBC/ODBC. For a SQL warehouse, use SQL Warehouses > target warehouse > Connection details.
  2. Obtain the JDBC URL and check its server hostname, port, and HTTP Path to ensure that it points to the intended compute resource. For the relevant settings, see Databricks JDBC connection configuration.
  3. Prepare a PAT that can be used in this workspace. If you need a new token, generate one under Settings > Developer > Access tokens > Manage and set an appropriate lifetime. If you cannot create or use tokens, contact the workspace administrator. See Create a Databricks personal access token.

Transfer tokens securely, and keep credentials out of screenshots, logs, and public documentation. After replacing a token or changing permissions, validate the Data Source connection and sync plan again.

Create a Databricks Data Source​

  1. Go to DataOps Platform > Integration > Data Source and click + Data Source.
  2. Select Databricks-JDBC, click Next, and enter a Data Source name and remarks.
  3. Select not distinguish environment or Independent setting. For independent configuration, enter the Development Environment and Production Environment settings separately.
  4. Enter the connection parameters below. Under Connectivity Test, click Test. After the test passes, click Finish.
Configure the Databricks JDBC URL, PAT authentication, and advanced settings
Configuration ItemDescription
JDBC URLThe JDBC connection address for the Databricks compute resource; confirm the service address and connection parameters with the administrator
Authentication mechanism (Acceptance mechanism in the UI)Select PAT
Personal Access TokenEnter a valid token for the connecting account, with the permissions required to read or write the data for this sync
AdvancedEnter any required extended settings in JSON format; leave this blank if no additional settings are needed

After the connectivity test passes, load the tables in the sync plan, preview the source data, and run the plan to validate actual table permissions and read and write results.

Configure Databricks as the source​

  1. Go to DataOps Platform > Integration > Offline Sync and click + Integration plan.
  2. Under Data Source, select Databricks-JDBC and the Data Source you created.
  3. Select a Method and configure the source for an entire table or a custom query.
  4. Click Data Preview to check the source fields, actual values, and read scope.
  5. To adjust the batch size, expand Advanced.
Example Databricks source table and batch settings from an existing plan

This figure shows settings from an existing plan and does not replace validation of the current connection and data read. After configuring the plan, use data preview to check the actual results.

MethodConfiguration
Entire TableSelect the Catalog, database (Schema), and table in order. To read only part of the data, enable Data Filtering and enter a filter condition
Custom QueryEnter a read-only SQL query that returns the fields to synchronize, with any filter conditions included in the query. This method does not use the separate Catalog, database, table, or Data Filtering settings provided for Entire Table

Data Filtering for Entire Table limits the read scope. Enter a WHERE clause as shown in the editor, for example, WHERE status = 'active', rather than a complete SELECT query. For Custom Query, ensure that the SQL can access the correct Catalog and database, with explicit result field names and recognizable types.

Batch Size defaults to 1000 and supports values from 1000 to 10000. After changing the Data Source, Method, Catalog, database, or table, check the read scope, preview results, and Field Mapping again. If the source structure or query results change, parse the fields again and review the mappings.

Configure the data target​

Read Databricks data into the preset warehouse​

Under Data Target, select the platform's Preset Warehouse, then select the target space table. If the target table has not been created, go to Tables and create it with the required field and partition definitions. See Create and manage space tables.

Configuration ItemDescription
Target tableThe preset warehouse space table that receives the synced data
Partition InformationFor a partitioned target table, enter each partition value as required by the table. You can use space parameters to configure the partitions used at execution time
Write modeSelect Append or Overwrite according to the target data retention requirements. Confirm the target table and partitions again before execution

Write preset warehouse data to Databricks​

Configure the preset warehouse source table under Data Source. Under Data Target, select Databricks-JDBC and the Data Source you created, then select the target Catalog, database (Schema), and table in order.

Configuration ItemDescription
Write TableA target table that has already been prepared in Databricks and that the connecting account is allowed to write to
Partition InformationIf target partition fields are displayed, enter every partition value and check the actual values used at execution time
Write modeOnly Append is supported. Data is appended to the target table; before running the plan again, assess whether this will create duplicate records
Pre-execution SQLOptionally enter SQL to execute on the target before synchronization starts; leave this blank if it is not needed
Batch SizeUnder Advanced; the default is 5000, and the supported range is 1–10000

Pre-execution SQL is actually executed on the target. If it fails, the sync is not submitted. The pre-execution SQL and subsequent data writes are not guaranteed to share a transaction, and a later sync failure is not guaranteed to undo SQL that has already executed. Before entering SQL or retrying, check the target environment, its data impact, and the outcome of repeated execution.

The Databricks target uses append mode and does not provide native overwrite or upsert modes. Do not treat a preliminary delete statement followed by an append as an overwrite operation with transaction-wide rollback.

Field Mapping and supported field types​

After configuring the source and target, use Field Mapping to map source fields to target fields. You can select fields individually, or use Map by Name or Map by Row and then check each mapping. Each target field can map to only one source field.

Example Databricks Field Mapping from an existing plan

The figure shows field correspondences from an existing plan. It does not indicate that live connectivity or this execution has been validated.

When Databricks is the Data Source, the following field types are supported. Type names are case-insensitive. For parameterized fields or nested structures, also check the complete definition.

Type categorySupported Databricks field types
Booleanboolean
Integertinyint, smallint, int, integer, bigint
Floating-point and fixed-point numbersfloat, double, decimal
Stringstring, varchar, char
Date and timedate, timestamp, timestamp_ntz
Complex typesarray, map, and struct; check the complete structure

binary, unlisted types such as interval, void, and variant, and other unrecognized types are not supported.

Supported field types for writing​

When Databricks is the Data Target, the supported Databricks field types are the same as in the table above. binary, unlisted types, and other unrecognized types are not supported.

Create the target table in advance. Support for a field type does not mean that any data will meet the target field's complete definition and null constraints.

Field type limitations​

  • For decimal(p,s), varchar(n), and char(n), check the complete precision, scale, and length requirements.
  • For timestamp and timestamp_ntz, check the actual time meaning, time zone, and precision requirements. A type name alone does not establish that results will meet expectations.
  • For array<T>, map<K,V>, and struct<...>, check the element, key-value, or field definitions, including field names and order, null constraints, and nesting levels. Support for a top-level type does not mean that every internal structure is supported. For example, support for array alone does not establish support for array<binary>.
  • After the source or target table structure changes, parse the fields again and check their correspondences. A field being visible in the data preview does not mean that its type is supported for synchronization.

Saving, validation, and Workflow Schedule​

Save and validate the sync results​

  1. Check the source table or query, filter scope, target table and partitions, write mode, pre-execution SQL, and Field Mapping. First use the source Data Preview to check fields and sample values.
  2. Enter the plan name, owner, and remarks, then click Save.
  3. Click Manual Execution, select an execution mode as required by the dialog, check the current parameter values, and start execution. Manual execution uses the Production Environment configuration; the execution mode does not switch between the Development Environment and Production Environment.
  4. Check the status and logs under Manual Execution Records. After the task succeeds, check the data volume, field correspondence, and sample numeric, time, complex-type, and null values in the target table.

With independent environment configurations, successfully loading or previewing data in the Development Environment does not mean that production credentials and read/write permissions are ready. Before the first execution, check both configurations and the tables, partitions, and parameters in the target environment. For appends to Databricks or plans with pre-execution SQL, also assess how retries affect existing data.

Mount on a Flow​

After manual validation passes, you can mount the plan on a Flow and run it on a schedule.

  1. Open the saved plan, click Mount on task flow, and select the target Flow. If it has not been created, see Create a Flow.
  2. Enter the node name, execution mode, owner, and other details, then click Create and Mount.
  3. Click Go to Flow, check the node and its upstream and downstream dependencies, and verify the tables, partitions, and space parameters used for scheduling.
  4. Complete the Flow schedule settings, save, and publish the Flow. Mounting the node does not mean that it has been published online.

After a plan is mounted on a Flow, it can no longer be run manually from the plan details page. Execute and validate the node from the Flow instead.

FAQ​

SymptomWhat to check
Connectivity test failsCheck the JDBC URL, PAT validity, compute resource access permissions, and the network access scope of the platform servers
The connection passes, but the table cannot be found or previewedCheck the Catalog, database and table names, and metadata and read permissions. For Custom Query, also check the SQL and its access scope
Preview works, but execution failsCheck production credentials, target write permissions, tables and partitions, parameters, and pre-execution SQL, and review the execution log
Duplicate data appears after repeated executionCheck the append scope and previous execution results to determine whether the same batch of data was read and written more than once
Was this page helpful?