Federated Connectors

Federated Connectors are an Early Access feature that enables data to be queried at source across external databases — without moving data into the Harbr platform or into Databricks-managed storage.

The feature is built on Databricks Lakehouse Federation. When a Federated Connector is created, Harbr automatically provisions two Databricks objects using the credentials you provide: a foreign connection (which authenticates to the external source) and a federated catalog (a read-only Unity Catalog mirror of the external database's schema). Assets are then created by selecting tables from within that federated catalog. At no point is data copied or stored — all queries are executed against the source system in real time.

Databricks-to-Databricks federation is not supported. If your source data originates from an existing Databricks Workspace, use a Databricks-to-Databricks Delta Share to expose that content to the platform instead. Once shared, assets can be created in the standard way, giving a consistent experience across Databricks and external technology sources.

The following connector types support federation in the current Early Access release:

Microsoft SQL Server
MySQL
BigQuery
Snowflake

Architecture and Data Flow

When a user accesses a federated asset — through Spaces, Export, or a Delta Share — a query is pushed through the federated catalog to the source system via the foreign connection, and results are returned in real time. No data is staged or copied at any point. The execution path varies by consumption method:

Spaces: Spaces are configured with a compute-based SQL Warehouse and a compute-based Spark cluster within the platform's Application Plane. SQL tools execute queries within the SQL Warehouse; analytical tools execute Spark workloads on the cluster. Both reach source data via the Federated Catalog and Foreign Connection.
Delta Shares — Open Sharing: Queries are served via in-region Serverless SQL Warehouses. Share configurations are created via the Unity Catalog API.
Delta Shares — Databricks-to-Databricks: Queries are executed within compute-based SQL Warehouses in the recipient Databricks Workspace.
Export: Databricks jobs are configured via the Unity Catalog API and run on server-based compute clusters in the platform environment. The job reads from the federated source and writes to the target destination connector.

Enabling a Federated Connector

To enable federated access on a supported connector:

Create or edit a connector of a supported type (Microsoft SQL Server, MySQL, BigQuery, or Snowflake).
Select the Enable 'at source' assets option.
Harbr will automatically configure the underlying foreign connection and federated catalog in your Databricks environment using the credentials provided.
Once enabled, at-source assets can be created from this connector by selecting tables from within the federated catalog.

Note: Once the Enable 'at source' assets option is set on a connector, it cannot be unselected. The connector can otherwise be edited (name, description, credentials) while the feature remains in Early Access.

Deleting a Federated Connector

When a federated connector is deleted, Harbr also removes the underlying Databricks configuration — including the foreign connection and the federated catalog. Any at-source assets created from that connector will immediately become inaccessible across all consumption methods (Spaces, Delta Shares, Export). This action cannot be undone.

Limitations

Federated assets are read-only — data cannot be written back to the source system.
Only table assets can be created from federated connectors; file assets are not supported.
Federated assets cannot be exported to desktop.
Federation between Databricks Workspaces is not supported by Databricks — use a Databricks-to-Databricks Delta Share for this pattern.
Row and column filters applied via subscription plans are enforced at query time by Databricks, not at the source system.

Network Requirements

For federated connectivity to work, the source database must be network-accessible from the Databricks workspace in the platform's Application Plane. This means:

The source database host and port must be reachable from Databricks compute within the platform's cloud environment.
Databases that are only accessible within a private network (for example, behind a corporate firewall or VPN boundary) will not connect successfully unless explicit network peering, firewall rules, or a private endpoint configuration is in place between that network and the platform's Databricks environment.
Contact your Harbr support team for guidance on private network connectivity for your specific environment.

Delta Shares of Federated Assets

Federated assets can be included in Delta Shares in the same way as standard at-source assets. If you encounter issues creating or listing tables when sharing a federated asset, verify that the Databricks workspace default catalog is set to a Unity Catalog (not hive_metastore). This setting is found in the Databricks workspace under Settings > Advanced > Default Catalog.