Databricks

Databricks can be used in two ways on the Harbr platform:

1. Databricks Workspace Connector (Azure only) — connects Harbr to a Databricks workspace to bring data on and off the platform. Supports the full range of platform features including Export, Query and Spaces.

2. At-Source Connector (AWS) — configures Databricks as a source for At Source assets, enabling Delta Shares. Does not support Export, Query or Spaces.

1. Databricks Workspace Connector (Azure only)

Pre-requisites

• Unity Catalog enabled on your Databricks Workspace

• A Databricks Managed Service Account (Entra accounts are not supported)

• A Harbr account with one of the following roles: Default User, Organisation Admin, or Technician

Databricks account permissions required:

PAT token or OAuth credentials must also have:

• USE CATALOG on any catalogs you want to create assets from

• USE SCHEMA on relevant schemas

• SELECT at catalog, schema or table/view level

• READ VOLUME at catalog, schema or volume level

• Workspace access and Databricks SQL access (defaulted ON when creating a user from inside the workspace)

The service account must also have Can Use permissions for Use Tokens (Workspace Admin Settings → Permission Settings).

Create the Connector

1. Click Manage on the Navigation bar

2. Select Connectors → Create connector

3. Enter a Name and optional Description

4. Select Type → Databricks

5. Select authentication method:

   • PAT token — generated in Developer Settings in Databricks (guide). Only Databricks personal token authentication is supported.

   • OAuth — using a service account. Ensure required permissions are granted on catalogs, schemas and SQL Warehouses.

6. Enter Host — the workspace URL e.g. https://adb-1535878582058128.8.azuredatabricks.net (no trailing /)

7. Enter Client ID — generated in Azure (Service Principal UUID)

8. Enter Client Secret Value — visible once on generation in Azure

9. Add Integration Metadata — at minimum, set httpPath:

   • In Databricks: Compute → SQL Warehouses → select warehouse → Connection Details → copy HTTP Path (e.g. /sql/1.0/warehouses/7cff6770269b80c7)

10. Click Create

Identity Configuration

The service account must:

1. Be added as a user (or in a group) in your Databricks workspace

2. Have Databricks SQL Access entitlement enabled

3. Have USE CATALOG on each relevant catalog

4. Have USE SCHEMA on relevant schemas

5. Have SELECT on each table to be read

6. Have Can Use permission on the designated SQL Warehouse

7. Have Can Use permissions for Use Tokens (Workspace Admin Settings → Permission Settings)

Integration Metadata Reference

Configure Organisations

Each organisation can use a different Databricks warehouse. To configure:

1. Go to Organisation Administration → Metadata

2. Add the following key/value pair:

Key: harbr.user_defaults

Value:

{
  "consumption": {
    "catalogs": [
      {
        "name": "",
        "id": "",
        "connector_id": "",
        "databricks_catalog": "assets",
        "databricks_schema": "managed",
        "databricks_table_name": { "naming_scheme": "PESSIMISTIC" },
        "default": true,
        "access": {
          "share": { "default_ttl_minutes": "1200" },
          "query": { "default_llm": "", "default_engine": "" },
          "iam": {}
        }
      }
    ]
  },
  "upload_platform": { "connector_id": "yourconnectorid" },
  "processing_platform": {
    "connector_id": "yourconnectorid",
    "default_job_cluster_definition": {}
  }
}

2. At-Source Connector (AWS)

At Source assets can be created from an appropriately configured Databricks connector and used to create Delta Shares (for individual assets or as part of a product).

Limitations vs. the Workspace Connector:

• Does not support Export, Query or Spaces

• Sample Data returns the first rows of a table rather than a random sample

• Metadata size is estimated rather than calculated from a full table scan

Pre-requisites & Permissions

Both PAT and OAuth can be used. The service principal account requires the following metastore-level permissions:

Configuration Steps

1. Ensure Data Sharing is enabled on the platform

2. Set httpPath as Integration Metadata on the connector:

   • In Databricks: SQL Warehouses → select warehouse → Connection Details → HTTP Path (e.g. /sql/1.0/warehouses/024ad5865e6bc94d)