Create a Cloud Connector to Connect to your Cloud Storage

Integrating your data pipelines via platform connectors allows you to add data to the platform and to take data off the platform, to use within your local environments.

It is often a good idea to setup your cloud connectors as soon as possible as sometimes it requires assistance from other departments in your company.

Note: Use the Snowflake web interface (Snowpipe) or any other supported Snowflake ETL software to integrate data from or to your cloud storage connector.

A direct connection to Snowflake is coming soon.

Note: You must have an Technician role to perform this action

Your connectors within your cloud accounts must have been created, else setup a cloud-agnostic SFTP connector

To create an connector:

1. Find and click the connectors button under the Manage menu in the main navigation bar.

2. Click Create connector.

3. Enter the connector Name.

4. Enter an connector Description.

5. Select the Type of connector you want to set up.
   Follow the instructions in the appropriate section below.

S3

1. Choose Type > S3.

2. Insert your S3 path.
   Example: s3://my-bucket/my-data
   This is the bucket name you set when you created your.

3. Select your AWS Region from the list.

4. Insert your IAM credentials (if requested)

   1. Insert your Amazon access key

   2. Insert your Amazon secret key

5. Click Create.

6. Follow the Your AWS S3 bucket policy steps

   • Navigate to the bucket in the AWS console S3 browser

   • Enter the `Permissions` tab and `Bucket Policy` option

   • Copy and paste the generated policy statement below into the text editor

   • Save

7. Click Close

Still not sure? Watch this video on how to set up an AWS S3 bucket.

GCS

1. Choose Type > GCS.

2. Enter your bucket name and optional subdirectory (path/to/dir). The Platform uses these locations when publishing and exporting to the connector.

3. Enter the GCS bucket name.

4. (Optional) Enter the GCS subdirectory.

5. Select GCP Key File (if requested)

6. Click Create.
   The New GCS connector information page appears.
   Help: This page informs you that the connector is ready and that the member account and roles are created. It also provides guidance on how to apply the required roles and access permissions to the bucket member account. Ensure bucket access control is set to Fine-grained.

7. Go back to Google Console and log in

8. Go to your bucket

9. Click on Roles > Add

10. Add New principals (This is the member email given to you on platform once you create your connector)

11. Add the following roles:

    1. Storage Legacy Bucket Writer

    2. Storage Legacy Object Owner

12. Click Close.

Still not sure? Watch this video on how to set up an GSC bucket.

Snowflake

We support Snowflake both as source for creating assets (requires only read access) or as a destination for exports (requires write access).

If a Snowflake reader account is used to setup the connector, it will not be possible to use it for export.

1. Choose Type > Snowflake .

2. Insert your Snowflake details

   1. Account URL

   2. Username

   3. Pasword

3. Click Create.

4. Click Close

BigQuery

1. Choose Type > BigQuery

2. Select and upload GCP Service Account Key File that will be used to manage access to your BigQuery project(s). See here for more details on how to set one up.

3. Project ID wil be populated automatically based on the file.

4. Click Create. Connection test will run and if successful, will show Connection Test Status as Successful.

5. Click Close.

Azure Blob Storage

1. Choose Type > Azure Blob Storage.

2. In the Shared Access Signature section:

   1. Enter the Blob SAS URL.

   2. (Optional) Insert a Blob prefix.

3. Click Create.
   The new connector ready message box appears.

4. Use the access permissions instructions to add the displayed member details to your permissions tab.

5. Click Close.

SFTP

(available to some customers only, contact Support if you would like to know more)

1. Choose Type > SFTP

2. Click + Create key to create an SSH key. You can create up to ten SSH keys. SSH keys can be downloaded as .PEM files and used to access the SFTP server.

Warning: For security purposes SFTP connector SSH keys are only accessible to the user that created the connector (the connector owner) on behalf of the organization. If you are using an SFTP connector created by a different user, contact this user to obtain access to the SFTP server location to retrieve the exported data.

Now you are ready to add data onto the Platform, or take data off the Platform, via a cloud connector.