Create a Connector
Integrating your data pipelines via platform connectors allows you to add data to the platform and take data off the platform for use within your own environments. Connectors form the bridge between your organization's data systems and the Harbr platform, enabling secure, controlled data movement.
It's often best to set up your cloud connectors early in your platform onboarding, as this process may require assistance from your organization's technical teams. Once configured, connectors make it possible for users to create assets and products that rely on live, managed, or imported data.
How Connectors Work
Connectors use secure credentials to reach out to a specified source location, receive data from that source, and make it available on the Harbr data plane for use within the platform.
Depending on the configuration, data may either be:
On-platform: copied into Harbr-managed storage, or
At-source: left in its original location and accessed remotely.
Both approaches make data accessible and usable through Harbr; the difference lies in where the data physically resides and how it's accessed and controlled.
Connector Types
While availability varies by environment, the following connector types are currently supported on the Harbr platform.
On-Platform Connectors
Definition:
Data is copied from the source into Harbr-managed storage, allowing full control and visibility from within the platform.
Control:
The Harbr platform manages the catalog and indexing, ensuring consistent access across features like Query, Spaces, and Data Shares.
Security:
Credentials are managed securely by the platform.
Use Case:
Ideal for sharing, collaboration, or analytical scenarios where having a managed copy of data improves performance, accessibility, and governance.
At-Source Connectors
Definition:
Data remains in its original location, with Harbr securely managing the connection and credentials required to read it.
Control:
Operates through isolated infrastructure (such as Trino jobs and clusters) that query the data without copying it.
Security:
Credentials are managed exclusively by Harbr, ensuring they remain secret and inaccessible to users.
Use Case:
Ideal for high-security or compliance-sensitive scenarios where data must remain in place and only be accessed on demand.
To enable a connector to work in this manner select the 'Enable at-source assets' option when creating the connector
In the case of Harbr platforms that leverage Databricks Data Planes, At-Source connectors are enabled via Federated Connectors
Behavioural Notes
At-source assets differ slightly from on-platform assets in how certain metadata is handled:
Data Dictionary: Loaded dynamically from the source's current schema; automatically reflects changes such as new columns.
Sample Data: Displays the first rows in the source table rather than a random sample.
Metadata Size: Estimated from the external table rather than scanned in full.