Designing Secure and Reliable Data Pipelines: The Role of Connectors for Data Engineers

Role-Based Playbooks

How Data Engineers Design Secure and Reliable Data Pipelines with Connectors

This guide is written to assist data engineers who are responsible for building and maintaining data pipelines that must be reliable, secure, and scalable over time. It is particularly relevant for teams moving toward a marketplace operating model, where data needs to be reused across multiple use cases, shared across teams or organizations, and consumed through self-service channels without risking governance or control.

Too often do data engineers fall into the trap of spending a bulk of their time correcting Python scripts, managing API key rotations, and rebuilding custom pipelines whenever upstream schemas change. This experience is common, and not to mention, costly. Custom data pipelines cost an average of $520,000 per year to maintain, and even simple pipelines can take three to four weeks to build. Much of this effort is repetitive, fragile, and difficult to scale.

On a platform that supports the data product framework, the role of the data engineer shifts monumentally. They are no longer simply transporting data from one system to another. They are responsible for making data reliable, secure and observable. The primary mission is to enable self-service data consumption while retaining governance, security, and operational stability. This guide outlines how data engineers can achieve this shift by using connectors to support marketplaces that are repeatable, scalable and commercially viable.

The Mission to Streamline Pipelines

In a traditional model, data engineers respond to a steady stream of tickets requesting datasets, pipelines, or one-off integrations. Each request introduces new logic, architectural weak spots and operational burdens. When entering an integrated data marketplace model, the focus must shift beyond the data pipeline. To be successful long-term, engineers must ensure that data can be trusted, accessed safely, and reused across multiple use cases without constant rework. Doing so requires a broader set of responsibilities.

Reliability and Observability

Reliable data pipelines require visibility. They are not configured and tested once, and then forgotten. Instead, successful engineers that power successful data marketplaces will monitor:

• Data freshness at the source

• Job success and failure

• Access credentials and security

• Latency and rate of delivery

• Schema changes and upstream breakages

Observability enables teams to identify issues early and prevent downstream impact on analytics, reporting, and consumer-facing products. Without this visibility, pipelines become silent liabilities rather than dependable infrastructure. The flow on effect here is damage to the reputability of the organization’s marketplace and offering as a whole. 

Environments and Compute Isolation

Engineers should design pipelines that separate environments such as development, testing, and production. This means that compute can be pre-provisioned, isolated across users and teams, or restricted to approved runtimes. This separation allows experimentation and iteration without compromising production systems or introducing unintended risk. By controlling environments and compute, engineers can enable innovation while maintaining operational stability.

Data Reuse to Reduce

A single data Asset, such as a Snowflake table or Power Bi visualization, can be re-purposed and re-packaged to power multiple data products. Engineers should endeavour to design pipelines so that data can be re-used safely, combined with other assets, and utilized without breaking existing workflows. This reduces duplication and lowers long-term monitoring strain and maintenance costs. Rather than building pipelines for each request, engineers benefit from utilizing, always-on channels that support multiple downstream use cases. Namely, connectors. 

Connectors as the Key 

Connectors provide standardized, repeatable ways to move or expose data in a secure and controlled manner. By configuring and using connectors, data engineers reduce the previously described bespoke pipeline development and lower operational risk. Connectors integrate with cloud data warehouses, on-prem databases, object storage and APIs, and allow for data to be moved on and off platforms, to and from existing environments. 

Rather than building custom pipelines for each integration, engineers can select from three core connector patterns depending on the use case.

On-platform Connectors

On-platform connectors copy data from the source system into platform-managed storage. This gives the platform full visibility and control over the data, allowing it to be indexed, cataloged, and governed consistently across consumption features. Given that the data is managed within the platform, access is governed through platform-level permissioning rather than direct system credentials. Importantly, the data’s source credentials are handled securely by the platform and are never exposed to end users.

On-platform connectors are well suited to sharing, collaboration, and analytical use cases. They are ideal when performance, discoverability, and reuse are important, or when data needs to be accessed repeatedly by different consumers in a governed environment.

At-source Connectors

At-source connectors allow data to remain in its original location, with Harbr securely managing the connection and credentials required to read it on demand. Data is queried through isolated infrastructure, such as Trino jobs and clusters, without being copied into the platform. Similar to on-platform connectors, credentials are managed exclusively by the platform system and are never exposed to users. Consumers access the data through governed paths while the Organization’s source location retains control.

At-source connectors are ideal for high-security or compliance-sensitive scenarios where data must remain in place. They enable discovery and controlled access without violating organizational regulations or best practice. 

Both on-platform and at-source connectors behave as secure and reliable pipelines to make data accessible. The difference lies in where the data resides and how it’s accessed and controlled.

Federated Connectors 

Federated connectors leverage technologies such as Databricks Lakehouse Federation to create read-only catalogs and enable querying across external systems without moving data. Federation reduces duplication while still supporting discovery across distributed environments.

In Practice, On Platform

1. Finding the Data

Data is often scattered across an organization in different formats and silos. Without clear visibility, data engineers can waste time trying to locate the right data or the correct connection details. Once the data is located, best practice is to streamline and organize its placement based on the wider business' policy and preference. Then, engineers should make it visible and document its location to ensure time is saved when the process is repeated. 

2. Gaining Access Approval

Even when the data is found, gaining access is often slow and frustrating. Manual requests, reviews and approvals can derail timelines, slow momentum and discourage Asset reuse by data producers. By managing governance on-platform, rather than the ad-hoc securitization of it before it gets there, an efficient, scalable permissioning system can ensure end-to-end governance. With role-based access control, data owners can approve requests promptly through the platform. Access is granted through governed paths rather than through copies in an unmonitored environment, ensuring that usage remains auditable and time-bound. 

3. Deploying Connectors

Once data access is eventually granted, the traditional user experience for data producers looking to upload organizational data is rarely self-service. Product managers and other producer roles typically rely on engineering teams to assist with connecting the required own storage, tools, and formats. Connectors change this reality. 

Once configured by an engineer, and validated with the required credentials, other users in the organization with the necessary roles and access levels are able to ingest data independently. Typifying the relationship between interoperability, cost and speed, whether the destination is Databricks, Snowflake, or BigQuery, connectors lessen the load on data engineers in the long-term. By relying on pre-built connectors, the architecture reduces a three-week build time by integrating reliable, repeatable data flows.

4. Customizing Data at its Source

Data needs vary by use case, and when a targeted consumer only requires a small subset of a large dataset, engineers can save compute, storage, time and money by leveraging flexible connector technology. By pointing each configured connector at a specific storage bucket, workspace or location, the noise can be reduced before the pipeline even executes. In downstream workflows, where a producer opts to use the connector to ingest a particular table, for example, they are shown and permitted to ingest only the data they are supposed to. 

Beyond the security benefit, this capability empowers non-technical producers with lesser knowledge of native storage systems, to locate and retrieve exactly what they need for their use case, and in turn, accelerate time-to-business-value. 

Summary

There is an intrinsic choice between speed and control that plagues one-off, data pipelines. While they may move quickly at first, and ultimately get the data where it needs to, engineers incidentally accumulate operational debt over time. In a more mature workflow that integrates connectors, teams can spend less time maintaining fragile pipelines and more time designing systems that scale. Pipelines are observable, access is governed, and data can be reused across products without manual rework. The success story is this: data engineers can enable others to move faster without relinquishing control, reliability or security.