Databricks

Databricks can be used in two ways on the Harbr platform:

1. Databricks Workspace Connector (Azure only) — connects Harbr to a Databricks workspace to bring data on and off the platform. Supports the full range of platform features including Export, Query and Spaces.

2. At-Source Connector (AWS) — configures Databricks as a source for At Source assets, enabling Delta Shares. As of version 5.31, Export and Spaces can also be enabled on certain environments.

1. Databricks Workspace Connector (Azure only)

Pre-requisites

• Unity Catalog enabled on your Databricks Workspace

• A Databricks Managed Service Account (Entra accounts are not supported)

• A Harbr account with one of the following roles: Default User, Organisation Admin, or Technician

Databricks account permissions required:

PAT token or OAuth credentials must also have:

• USE CATALOG on any catalogs you want to create assets from

• USE SCHEMA on relevant schemas

• SELECT at catalog, schema or table/view level

• READ VOLUME at catalog, schema or volume level

• Workspace access and Databricks SQL access (defaulted ON when creating a user from inside the workspace)

The service account must also have Can Use permissions for Use Tokens (Workspace Admin Settings → Permission Settings).

Create the Connector

1. Click Manage on the Navigation bar

2. Select Connectors → Create connector

3. Enter a Name and optional Description

4. Select Type → Databricks

5. Select authentication method:

   • PAT token — generated in Developer Settings in Databricks (guide). Only Databricks personal token authentication is supported.

   • OAuth — using a service account. Ensure required permissions are granted on catalogs, schemas and SQL Warehouses.

6. Enter Host — the workspace URL e.g. https://adb-1535878582058128.8.azuredatabricks.net (no trailing /)

7. Enter Client ID — generated in Azure (Service Principal UUID)

8. Enter Client Secret Value — visible once on generation in Azure

9. Set the httpPath:

   • In Databricks: Compute → SQL Warehouses → select warehouse → Connection Details → copy HTTP Path (e.g. /sql/1.0/warehouses/7cff6770269b80c7)

10. Click Create

Identity Configuration

The service account must:

1. Be added as a user (or in a group) in your Databricks workspace

2. Have Databricks SQL Access entitlement enabled

3. Have USE CATALOG on each relevant catalog

4. Have USE SCHEMA on relevant schemas

5. Have SELECT on each table to be read

6. Have Can Use permission on the designated SQL Warehouse

7. Have Can Use permissions for Use Tokens (Workspace Admin Settings → Permission Settings)

Integration Metadata Reference

Configure Organisations

Each organisation can use a different Databricks warehouse. To configure:

1. Go to Organisation Administration → Metadata

2. Add the following key/value pair:

Key: harbr.user_defaults

Value:

{
  "consumption": {
    "catalogs": [
      {
        "name": "",
        "id": "",
        "connector_id": "",
        "databricks_catalog": "assets",
        "databricks_schema": "managed",
        "databricks_table_name": { "naming_scheme": "PESSIMISTIC" },
        "default": true,
        "access": {
          "share": { "default_ttl_minutes": "1200" },
          "query": { "default_llm": "", "default_engine": "" },
          "iam": {}
        }
      }
    ]
  },
  "upload_platform": { "connector_id": "yourconnectorid" },
  "processing_platform": {
    "connector_id": "yourconnectorid",
    "default_job_cluster_definition": {}
  }
}

2. At-Source Connector (AWS)

At Source assets can be created from an appropriately configured Databricks connector and used to create Delta Shares, Exports and within Spaces (as of 5.31 - enabled independently per environment.

Limitations vs. the Workspace Connector:

• Sample Data returns the first rows of a table rather than a random sample

• Metadata size is estimated rather than calculated from a full table scan

Pre-requisites & Permissions

Both PAT and OAuth can be used. The service principal account requires the following metastore-level permissions:

Configuration Steps

1. Ensure Data Sharing is enabled on the platform

2. Set httpPath on the connector:

   • In Databricks: SQL Warehouses → select warehouse → Connection Details → HTTP Path (e.g. /sql/1.0/warehouses/024ad5865e6bc94d)

3. Spaces and Export on AWS (5.31)

At-source Databricks tabular assets can now be queried interactively in Spaces and included in Export jobs on platforms with an AWS dataplane, in addition to the existing Delta Sharing consumption path. Nothing is copied until a query runs or an export is triggered.

• In Spaces, queries run through Harbr's Trino engine directly against your Databricks SQL warehouse, so warehouse compute is only used while a query is running.

• In Export, a Glue job reads the data via the Databricks driver at export time.

• Spaces and Export are enabled independently — turning one on does not turn on the other.

• No Databricks account administrator action is required to enable this. Setup is cloud-side only (an AWS IAM role and trust policy); Harbr provisions the Unity Catalog storage credential automatically when the connector is saved.

• Both personal-access-token and OAuth machine-to-machine Databricks connectors are supported. For OAuth, Harbr mints a short-lived token so the client secret never leaves the platform.

• A reader-agnostic browser download is also introduced alongside this: an at-source asset (Databricks or Snowflake) can now be downloaded as a packaged ZIP.