Using GCS

‌

Google Cloud Storage (GCS) Connectors are secure gateways through which products can be moved in and out of the platform using GCS bucket transfers. It is referred to as a data storage connector. The first step in using this connector is setting up a storage bucket to store the data you intend to access.

You must have access to a GCS cloud account. See Create storage buckets for more information. To create a GCP account, please visit Google Cloud Console. If you already have a GCP account, please follow the steps below.

Create a GCS bucket

1. Log into your GCS account.

2. Click Create a storage bucket.

3. Insert a Bucket Name following GCP's naming guidelines.

4. Choose where to store your data.

   • For learning purposes, choose Multi-region; however pick whichever works best for you.

5. Choose a storage class for your data.

   • For this example, select Standard.

6. Choose how to control access to objects.

   • For this example, select Fine-grained, which will enforce public access prevention.

7. Choose how to protect object data.

   • For this example, select None.

8. Press Create.

Now you are ready to connect your storage to the platform via a Connector.

Connector Authentication Options

Option 1: GCP Service Account

Follow these steps if you are authenticating using a Service Account Key File. If you are using Workload Identity Federation, see Set Up Workload Identity Federation below.

Step 1: Creating the Service Account and Key

Using this option, the platform authenticates to your GCS bucket using a GCP service account JSON key file. You will need to create a service account and generate a key file before creating the connector.

Note: A paid GCP account is required to create service accounts and generate key files. Trial GCP accounts do not support this functionality.

1. In the Google Cloud Console, navigate to IAM & Admin > Service Accounts.

2. Click Create Service Account and follow the prompts to name and describe it.

3. Assign the following minimum roles to the service account:

   • storage.objectViewer (for read access)

   • storage.objectAdmin (for write access)

4. Once created, open the service account and navigate to the Keys tab.

5. Click Add Key > Create new key, select JSON, and click Create. The key file will download to your machine.

6. Ensure the service account has access to your GCS bucket by navigating to your bucket in Cloud Storage, clicking Permissions, and granting the service account the appropriate roles.

Step 2: Creating the Connector

1. Click Manage on the Navigation bar.

2. Select Connectors to view the Manage Connectors screen.

3. Click the Create Connector button at the top right.

4. Enter a Name for your Connector and a Description (optional).

5. Choose Type > GCS.

6. Enter the GCS bucket name.

7. (Optional) Enter the GCS subdirectory.

8. Upload your GCP Key File (the service account JSON file generated above).

9. Click Create. A connection test will run and, if successful, will show Connection Test Status as Successful.

10. Click Close.

Option 2: Workload Identity Federation

Workload Identity Federation (WIF) allows the platform to authenticate to your GCS bucket without a service account key file. Instead, the platform's AWS IAM role is trusted by your GCP project and exchanges its identity for a short-lived token at runtime.

This method is available to all customers on the AWS data plane from version 5.30. Existing key-based connectors are not affected and require no migration.

Step 1: Configuring the Workload Identity Pool

Before creating the connector in Harbr, you must configure a Workload Identity Pool in your GCP project to trust the Harbr AWS data plane. The exact GCP-side configuration steps will be provided to you during the connector setup flow in Harbr (see below).

At a high level, you will need to:

1. Create a Workload Identity Pool in your GCP project.

2. Add an AWS provider to the pool, trusting the Harbr AWS IAM role ARN provided to you.

3. Grant the Workload Identity Pool permission to impersonate a GCP service account that has access to your GCS bucket.

The exact commands or script required will be shown to you during the Harbr connector setup flow after you input your GCP project details.

Step 2: Creating the Connector

1. Click Manage on the Navigation bar.

2. Select Connectors to view the Manage Connectors screen.

3. Click the Create Connector button at the top right.

4. Enter a Name for your Connector and a Description (optional).

5. Choose Type > GCS.

6. Enter the GCS bucket name.

7. (Optional) Enter the GCS subdirectory.

8. Under Authentication Method, select Workload Identity Federation.

9. Click Start Setup. You will be prompted to enter your GCP Project ID and GCP Project Number.

10. Once entered, the platform will display the GCP-side configuration steps you must complete. Complete these steps in GCP before proceeding, and click Done.

1. After triggering the connection test, the Finish Setup button becomes available — you do not need to wait for the test to complete. If the test has not yet passed when you click Finish Setup, you will see the following warning: The connection test hasn't passed, so this connector may not be functional once it's live. You can finish setup now and run the test again later.

2. You can proceed past this warning. If the test failed because the bucket is empty, adding a file to the bucket and re-running the test should resolve it. If the test failed shortly after completing the GCP configuration steps, this is most likely the GCP setup delay described above — re-running the test after a short wait should resolve it.

3. Click Close.