Integrating your data pipelines via platform connectors allows you to add data to the platform and take data off the platform for use within your own environments. Connectors form the bridge between your organization's data systems and the Harbr platform, enabling secure, controlled data movement.
It's often best to set up your cloud connectors early in your platform onboarding, as this process may require assistance from your organization's technical teams. Once configured, connectors make it possible for users to create assets and products that rely on live, managed, or imported data.
How Connectors Work
Connectors use secure credentials to reach out to a specified source location, receive data from that source, and make it available on the Harbr data plane for use within the platform.
Depending on the configuration, data may either be:
-
On-platform: copied into Harbr-managed storage, or
-
At-source: left in its original location and accessed remotely.
Both approaches make data accessible and usable through Harbr; the difference lies in where the data physically resides and how it's accessed and controlled.
When setting up a connector provision the identity with scoped permissions relevant to what you'd want on the platform - knowing if you granted an additional perm to that identity, it would be reflected once you go to create assets.
Connector Types
While availability varies by environment, the following connector types are currently supported on the Harbr platform.
On-Platform Connectors
Definition:
Data is copied from the source into Harbr-managed storage, allowing full control and visibility from within the platform.
Control:
The Harbr platform manages the catalog and indexing, ensuring consistent access across features like Query, Spaces, and Data Shares.
Security:
Credentials are managed securely by the platform.
Use Case:
Ideal for sharing, collaboration, or analytical scenarios where having a managed copy of data improves performance, accessibility, and governance.
At-Source Connectors
Definition:
Data remains in its original location, with Harbr securely managing the connection and credentials required to read it.
Control:
Operates through isolated infrastructure (such as Trino jobs and clusters) that query the data without copying it.
Security:
Credentials are managed exclusively by Harbr, ensuring they remain secret and inaccessible to users.
Use Case:
Ideal for high-security or compliance-sensitive scenarios where data must remain in place and only be accessed on demand.
To enable a connector to work in this manner select the 'Enable at-source assets' option when creating the connector.
In the case of Harbr platforms that leverage Databricks Data Planes, At-Source connectors are enabled via Federated Connectors
Behavioural Notes
At-source assets differ slightly from on-platform assets in how certain metadata is handled:
-
Data Dictionary: Loaded dynamically from the source's current schema; automatically reflects changes such as new columns.
-
Sample Data: Displays the first rows in the source table rather than a random sample.
-
Metadata Size: Estimated from the external table rather than scanned in full.
Connector Capability Checks
A connector test passing today tells you less than it seems to. Most integrations only confirm they can authenticate and read a shallow list of objects — not whether the environment your data is processed in can actually reach and use the source, whether the credentials carry the permissions a specific use case needs, or whether anything still works days or weeks later.
Connector Capability Checks run a series of checks from two vantage points before a connector can be finalized: from Harbr, and from the processing environment — the actual compute (for example, a Databricks SQL warehouse) that runs queries against your source. Harbr confirming a connector looks fine is not the same as that compute being able to reach and use it, which is why both are checked separately.
Note: For platforms deployed using a Databricks workspace, these improvements to connector tests are currently supported connectors are PostgreSQL, MySQL, Oracle, Microsoft SQL Server, Databricks, Azure Synapse, and Snowflake.
For platforms deployed using an AWS workspace, these improvements apply for the set-up of Databricks connectors only.
How it works
Checks run in four possible steps, in order. The first three are hard failures — if one test fails, testing stops and you cannot finish setup until it's resolved. The fourth step is entirely warnings — it never blocks setup, but flags things worth reviewing.
1. Confirm connection from the platform
|
State |
What you'll see |
|---|---|
|
Waiting / In progress |
"This test confirms that the credentials provided are valid and that we can connect to the source." |
|
Passed |
"We have confirmed that the credentials provided are valid and that we can connect to the source. See details" |
|
Failed |
"This test did not pass. This issue will need to be resolved before completing setup for this connector. See details" |
This step combines connectivity and credential validation into a single check:
-
Confirm connection and authentication — on failure: "Either the source couldn't be reached, or the credentials provided were not accepted. This means the connector will not work as required. We recommend you review the error details below, then check your connector details to ensure you have provided the correct information. Once this is done, please retry testing to confirm that the issue is resolved."
2. Verify required access and permissions
|
State |
What you'll see |
|---|---|
|
Waiting / In progress |
"This test checks we can access everything needed for the connector to work." |
|
Passed |
"We have confirmed we have access to everything we need for the connector to work. See details" |
|
Failed |
"This test did not pass. This issue will need to be resolved before completing setup for this connector. See details" |
|
Not run |
"This test was not run because earlier tests failed." |
This step covers two checks:
-
Check access to the required source objects — on failure: "The connector identity couldn't read one or more of the required source objects. This means the connector will not work as required. We recommend you review the error details, and try to resolve the issue or ask your administrator for support. Once this is done, please retry testing to confirm that the issue is resolved."
-
Workspace credential minting — on failure: "We were unable to create the credentials that processing jobs need to act on your behalf. This means the connector will not work as required. We recommend you review the error details, and try to resolve the issue or ask your administrator for support. Once this is done, please retry testing to confirm that the issue is resolved."
3. Confirm connection from the processing environment
This step checks the compute that runs your queries — for example, a Databricks SQL warehouse — rather than Harbr itself.
|
State |
What you'll see |
|---|---|
|
Waiting / In progress |
"This test confirms that the processing environment can connect to the source." |
|
Passed |
"We have confirmed the processing environment can connect to the source. See details" |
|
Failed |
"This test did not pass. This issue will need to be resolved before completing setup for this connector. See details" |
|
Not run |
"This test was not run because earlier tests failed." |
One check per compute environment tested (for example, per SQL warehouse) runs here; the specific warehouse name and whether it's serverless or non-serverless only appear in the detail view:
-
Check connection from each compute type — on failure: “We were unable to confirm that the processing environment can connect to the source. Please ask your administrator for support.”
4. Check possible capabilities
|
State |
What you'll see |
|---|---|
|
Waiting / In progress |
"This test checks additional capabilities that may affect how the connector can be used." |
|
Passed |
"We completed [N] checks, which raised no warnings. See details" |
|
Completed with [N] warnings |
"We completed [K] additional checks, which raised [N] issues that may affect how the connector can be used. We recommend you review these warnings before continuing." |
|
Not run |
"This test was not run because earlier tests failed." |
Every check in this step is a warning only — none of them block Finish Setup:
-
Check governance permissions — warning: "We were unable to confirm that the connector identity has all of the necessary governance permissions. For this connector to work as expected, we recommend that you review the warning details and try to resolve any issues or ask your administrator for support."
-
Check provisioning and compute permissions — warning: "We were unable to confirm that the connector identity has all of the necessary provisioning and compute permissions. For this connector to work as expected, we recommend that you review the warning details and try to resolve any issues."
-
Check processing compute availability — warning: "We were unable to list the existing clusters. For the connector to work as expected, we recommend that you review the warning details and try to resolve any issues."
-
Check resource quota headroom — warning: "One or more of the resource quotas are close to, or have reached, their limit — or we were not able to confirm the required capacity. This means that your connector may stop working if a limit is reached. To avoid any issues later on, we recommend that you review the warning details and ensure there is sufficient capacity."
-
Confirm write access to the source — currently only runs for Snowflake connectors. Warning: "We were unable to confirm that the platform can write to some of the required objects in the connector. For this connector to work as expected, we recommend that you review the warning details and ensure there is the necessary write access."
Not all connector types require every check. If a check doesn't apply to your connector type, it simply won't appear in the UI — there's no "not applicable" state to work through.
Step-By-Step Guidance
Testing starts automatically once you land on the setup screen (for connectors with a prerequisite setup step, such as WIF, testing starts once that step is complete). Each check appears immediately and updates in real time as it runs:
-
Waiting — hasn't started yet
-
In progress — currently running
-
Passed — completed with no issues
-
Passed with warnings — completed, no failures, but one or more things to be aware of; click through for detail
-
Failed — a hard failure occurred; the failing check is highlighted with guidance on the likely fix
-
Not run — testing stopped before this check was reached
Editing connector details is disabled while checks are running. On a hard failure, testing stops and an Edit connector details option becomes available, offering the full set of editable fields rather than isolating the one that failed — saving your changes restarts testing from the top.
Warnings will not stop testing or block Finish Setup. Instead, they are there so you can make an informed decision (for example, "I don't need Delta Shares for this connector, I'm fine to proceed").
A failure at the processing-environment check (the compute running your queries, e.g. a Databricks SQL warehouse) is almost always a network or firewall matter on the source side rather than something you can fix by editing the connector. If this error is shown, please contact your platform team.
Step 1 - Add the Connector Credentials.
For example, see below if you are completing this journey for the Snowflake Connector type.
Step 2 - Let the tests run.
Capability checks execute automatically against the connector once you click “Start Setup”.
Step 3 - Assess any failed tests.
Address the underlying issue for each failure, edit the connector if needed, then re-run the tests.
Step 4 - Assess any warnings.
For each one, determine whether it's acceptable for how this connector will actually be used.
Step 5 - Complete the connector setup
You may progress once all hard failures, and any warnings you've reviewed, have been addressed.
