SFTP
SFTP connectors are secure gateways through which data can be moved in and out of the Platform. It is referred to as a database connector. Users with the technician role are able to create and manage them.
Depending on their platform configuration, users have the option to create either an On-Platform or External SFTP connector. These two connector types require different set-up steps and prerequisites, and also support different consumption patterns. Please see below for the breakdown of these differences.
External SFTP
Pre-requisites
To use an External SFTP connector, you need:
Host
Port
Username
Password
Folder path (optional)
Create a Connector
Click Manage on the Navigation bar
Select Connectors to view the Manage Connectors screen
Click the Create connector button at the top right
Enter a Name for your Connector and a Description (optional)
Select the Type > SFTP (External)
Enter details for:
Host
Port
Username
Password
Folder path (optional) - this target folder this ensures data is placed in a location that includes the supplied folder name, to ensure data is exported to specific locations for external consumption.
Add any Integration Metadata needed for programmatic integrations
Click the Create button
Assets created from external SFTP connectors can be used the same way as any other assets and consumed in Query, Spaces and Datasharing as per product permissions. You can also setup a connector to your SFTP location and use it as a destination for Export.
The next steps will run the usual asset creation checks and create a new asset if successful.
The asset can then be used the same way as any other assets within products, and consumed in Query, Export and Spaces as per the product permissions.
Export to External SFTP
When creating a one-off or repeating export, users must choose a connector. This establishes the target server and location to which the data will be sent for external access and consumption.
Selecting one of the pre-configured External SFTP connectors ensures that the data is extracted to a location on the chosen SFTP server.
Where a folder location has been supplied during connector creation, this will be included as part of the name and location of the data. Once a data export job is complete, the data location will be supplied so that a consuming user can easily find the data.
When configuring an External SFTP export, it’s important to understand that Folder Path and Integration Metadata control two different aspects of delivery.
Integration Metadata (Overwrite Behaviour)
To enable overwrite behaviour, you must include the following Integration Metadata:
{{exportDestination: home}}This setting prevents the system from creating a new GUID-based subfolder for each export. Instead, files will be written to a fixed location and overwritten on subsequent deliveries.
If this metadata is not included, the system will default to creating a new GUID folder for every export, regardless of the Folder Path configuration.
Folder Path (Delivery Location)
The Folder Path defines where the files are delivered within the SFTP destination.
If left blank, files are delivered to the root location.
If specified (for example,
/Exports), files are delivered to that folder.
In summary, integration Metadata controls whether files overwrite or create new GUID folders, wheres the Folder Path controls where the files are delivered. To deliver to a specific folder and overwrite each time, you must configure both.
Note: If Parquet format is selected during export, it will generate multiple files that will have unique names (e.g. part-00000-<uuid>.snappy.parquet,part-00001-<uuid>.snappy.parquet).This means that during subsequent exports, the previous files will not be overwritten or removed.
On-Platform SFTP
For some Harbr platforms, there is an option to create an on-platform SFTP connector.
For security purposes, SSH keys are only accessible to the user that created the connector on behalf of the organization. If you are using an SFTP connector created by a different user, contact this user to obtain access to the SFTP server location to retrieve the exported data.
SSH Key are only accepted as an authentication option for internal SFTP connectors, whereas external SFTP connectors can only be configured using a password.
Create a Connector
Click Manage on the Navigation bar
Select Connectors to view the Manage Connectors screen
Click the Create connector button at the top right
Enter a Name for your Connector and a Description (optional)
Select the Type > SFTP (On-Platform)
Enter details for:
Host
Port
Username
Password or SSH Keys - SSH keys can be downloaded as .PEM files and used to access the SFTP server from the platform.
Folder path (optional) - this target folder this ensures data is placed in a location that includes the supplied folder name, to ensure data is exported to specific locations for external consumption.
Add any Integration Metadata needed for programmatic integrations
Click the Create button
Assets created from on-platform SFTP connectors can be added to products and used to create exports. Consumption options are more limited.
