Using S3

S3 Connectors are secure gateways through which data can be moved in and out of the Platform using S3 bucket transfers. It is referred to as a data storage connector. The first step in creating this connector is setting up a storage bucket to store the data you intend to access.

Note: You must have access to an AWS cloud account and know the AWS Account Number. See Creating a bucket for more information.

Create an S3 bucket

1. Log into your AWS account.

2. In the search bar type S3 or Bucket.
   Click enter.

3. Click on Create Bucket.

4. Under General Configuration:

   • Insert Bucket Name.

   • Select your AWS Region (any cross region transfers are handled appropriately if the bucket is in a different region from the platform)

   • (Optional) Copy settings from an existing bucket.

5. Under Object Owner:

   • Select ACLs Disabled (recommended).

6. Under Block Public Access settings for this bucket:

   • Select Block all Public Access.

7. Under Bucket Versioning:

   • Select Disable or Enable.
     Help: Versioning is a means of keeping multiple variants of an object in the same bucket. You can use versioning to preserve, retrieve, and restore every version of every object stored in your Amazon S3 bucket. With versioning, you can easily recover from both unintended user actions and application failures.

8. (Optional) Enable Tags.

9. Under Default encryption:

   • Select advanced settings if you want to Disable or Enable object lock.

10. Click Create bucket.

Now you are ready to connect your storage to the platform via a Connector.

Note: The S3 connector uses different authentication methods depending on the type of platform you are on:

• AWS native platforms use Bucket Policies as part of connector configuration (follow Step 11 and ignore Step 12).

• Databricks-based platforms (AWS or Azure) use Client ID / Secret Key. No bucket policy action is required on Databricks-based platforms (follow Step 12 and ignore Step 11).

Create the Connector

1. Click Manage on the Navigation bar.

2. Select Connectors to view the Manage Connectors screen

3. Click the Create Connector button at the top right

4. Enter a Name for your Connector and a Description (optional)

5. Choose Type > AWS S3.

6. Insert your S3 path.
   Example: s3://my-bucket/my-data
   This is the bucket name you set when you created your.

7. Select your AWS Region from the list.

8. Insert your IAM credentials (if requested)

   1. Insert your Amazon access key

   2. Insert your Amazon secret key

9. Add any Integration Metadata needed for programmatic integration.

10. Click Create. Connection test will run and if successful, will show Connection Test Status as Successful.

11. AWS native platforms only: Follow the Your AWS S3 bucket policy steps

    1. Navigate to the bucket in the AWS console S3 browser

    2. Enter the Permissions tab and Bucket Policy option

    3. Copy and paste the generated policy statement below into the text editor

    4. Save

12. Databricks-based platform: To apply this policy to the IAM user or IAM role associated with this connector:

    1. Navigate to the IAM user or IAM role in the AWS console IAM browser.

    2. Enter the 'Permissions' tab and select the 'Add inline policy' option.

    3. Enter the 'JSON' tab.

    4. Copy and paste the generated policy statement into the text editor.

    5. Select 'Review policy', enter a name for the policy and select 'Create policy'.

Note If you are on a Databricks-based platform, skip this step. The bucket policy page may appear but will be empty. No action is required.

12. Click Close.

‌

At-Source Tabular Assets

At-source cataloguing for tabular assets on object stores is supported on Databricks data planes from version 5.30. Operators can register and catalogue tabular assets directly from S3 storage without copying data into the platform.

• At-source assets are catalogued directly from cloud storage — no data movement required.

• Once catalogued, assets can be used in Spaces, Query, Export, and Data Sharing in the same way as on-platform assets.

This feature uses a cross-account IAM role for access and requires the Enable At-Source toggle to be turned on when configuring the Modern S3 connector.

Note: Delta Sharing is only supported for assets in Delta format via the S3 connector.

How it works

Provisioning happens in two phases:

1. Connector save — when the Enable At-Source toggle is turned on and the connector is saved, Harbr provisions a Unity Catalog Storage Credential and External Location in the Databricks data plane. The External ID required to complete the IAM trust policy is returned and displayed on the connector view.

2. Asset creation (DataSync) — when a producer creates a tabular asset from an at-source-enabled connector, the platform triggers a DataSync that provisions a Unity Catalog External Table at the asset's storage path.

Supported asset types

Note — Delta Share on private-network storage. Delta Sharing generates pre-signed URLs at query time. If the storage account has public network access disabled, Delta Sharing will fail for that asset. This is a known limitation and is not detected proactively at asset registration time.

Setup

1. Create a cross-account IAM role in your AWS account with: a permission policy granting s3:GetObject on <bucket>/*, s3:ListBucket and s3:GetBucketLocation on <bucket>, and sts:AssumeRole on the role's own ARN.

   1. Add a placeholder trust policy — you will patch it after the first connector save.

2. In the connector form, enable the Enable At-Source toggle, enter the IAM Role ARN, and save. The platform provisions the UC Storage Credential and External Location and returns the External ID.

3. Copy the UC AWS Principal ARN and External ID from the connector view. Patch the IAM trust policy to include both values — plus a self-assume statement on the role's own ARN — in a single Statement with the sts:ExternalId condition. A copyable trust statement is provided on the connector view.

4. Run Test Connector and confirm PASS before allowing producers to create assets.

Setting up the IAM role

A connector uses your access keys → to assume an IAM role → which accesses your S3 bucket. Four things must line up: (1) the role's trust policy allows your user, the platform role (with the External ID), and itself; (2) your user can assume the role; (3) the role can assume itself; (4) the role has S3 permissions. The error message always tells you which piece is missing.

The values you will need

This guide uses full ARNs as the unit you copy and paste, not composite parts like "account + name". An ARN already contains the account ID, so you never have to work out which account a value belongs to — important in AWS environments where several accounts are typically in play. The AWS Console shows the full ARN at the top of every IAM user, IAM role, and S3 bucket page, with a copy button.

There are two sources for the values. Replace every <...> placeholder below with your own.

Your own AWS resources

The user and the role are both yours — two separate IAM identities in your account. The user holds the access keys; the role is what actually reaches the data.

Provided by the connector dialog

Concepts to understand before you start

1. IAM users vs. roles — a user has long-lived access keys; a role is assumed temporarily. The connector uses a user's keys to assume a role.

2. A role has two policy sides, edited in different tabs:

   • Trust policy — who may assume the role.

   • Permissions (identity) policy — what the role may do once assumed.

3. sts:AssumeRole authorization:

   • Cross-account, or trusting …:root → you need both the trust policy and an identity policy granting sts:AssumeRole.

   • Same account + the exact user/role named in the trust policy → the trust policy alone is sufficient (and more secure — prefer it over …:root).

4. External ID — the unique value the platform generates per connector. It goes in the trust policy Condition and must match exactly (confused-deputy protection).

5. Self-assuming role — a Unity Catalog requirement (Databricks, not native AWS): the role must list itself in both its trust policy and a permissions policy.

6. S3 access lives on the role. A bucket policy is only needed if the bucket is in a different AWS account than the role.

Step 1 — Trust policy on the role

IAM → Roles → <your-iam-role-name> → Trust relationships → Edit trust policy, paste, Update policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": { "AWS": "<your-iam-user-arn>" },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "<platform-role-arn>",
          "<your-iam-role-arn>"
        ]
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "StringEquals": { "sts:ExternalId": "<your-external-id>" }
      }
    }
  ]
}

• Statement 1 trusts the connector user directly (scoped — preferred over …:root).

• Statement 2 trusts the platform role (with External ID) and the role itself (self-assuming).

Tip: the platform dialog shows a whole policy document. Do not paste it as a single list item — merge its statement into your Statement list, or use the policy above which already combines everything.

Step 2 — Allow the user to assume the role

Optional when the user is named directly in Step 1 and user + role are in the same account. Required if you trust …:root instead, or for cross-account.

IAM → Users → <your-iam-user-name> → Permissions → Add permissions → Create inline policy → JSON, paste, name it assume-connector-role, Create:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "<your-iam-role-arn>"
    }
  ]
}

Step 3 — Let the role assume itself

IAM → Roles → <your-iam-role-name> → Permissions → Add permissions → Create inline policy → JSON, paste, name it self-assume, Create:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "sts:AssumeRole",
      "Resource": "<your-iam-role-arn>"
    }
  ]
}

This plus the self-reference in Step 1 satisfies the UC_IAM_ROLE_NON_SELF_ASSUMING check.

Step 4 — Give the role access to the bucket

The connector can authenticate but will not read data until the role has S3 permissions.

IAM → Roles → <your-iam-role-name> → Permissions → Add permissions → Create inline policy → JSON, paste the S3 policy generated by the platform dialog (it lists s3:ListBucket, s3:GetObject, s3:PutObject, etc. on your bucket), name it s3-bucket-access, Create. Example shape:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation", "s3:ListBucket"],
      "Resource": ["arn:aws:s3:::<your-bucket-name>"]
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject", "s3:PutObject", "s3:PutObjectAcl", "s3:DeleteObject",
        "s3:GetObjectAttributes", "s3:GetObjectVersion",
        "s3:GetObjectVersionAttributes", "s3:GetObjectTagging"
      ],
      "Resource": ["arn:aws:s3:::<your-bucket-name>/*"]
    }
  ]
}

Bucket policy? Only needed if the bucket is in a different account than the role. For a same-account bucket, the identity policy above is sufficient — skip the bucket policy. If it is cross-account, also add a bucket policy that names <your-iam-role-arn> as Principal.

Step 5 — Test

1. Wait approximately 1 minute for IAM changes to propagate.

2. In the connector dialog, run Test connection.

3. Create the connector, then register an asset.

Error reference

Why name the user rather than …:root

Trusting …:root delegates the decision to IAM — the user then also needs an identity policy granting sts:AssumeRole (Step 2). Naming the exact user by ARN in the trust policy is a direct grant: in the same account it is sufficient on its own, and it is more secure because you are not trusting the entire account. Prefer naming the specific principal.

Useful AWS references

• Delegate access across accounts

• Create a role for an IAM user

• Grant a user permission to assume a role

• External ID / confused deputy

• Policy evaluation logic

Connection test

When the Enable At-Source toggle is on, the Test Connector function runs two probes:

• Base probe — the existing S3 connectivity check (unchanged).

• At-source probe — validates the Unity Catalog Storage Credential and External Location. Returns typed outcomes: PASS, CREDENTIAL_TRUST_FAILED, STORAGE_ACCESS_FAILED, NOT_PROVISIONED, or DATAPLANE_NOT_READY.

Run Test Connector before creating any assets to catch credential misconfiguration early. See your operator for guidance if the at-source probe fails.

Limitations

• File / binary at-source is not included — this covers tabular assets only.

• AWS-classic dataplane orchestration of catalogued at-source is not supported in this release.

• DataSync failure is terminal in v1. There is no user-initiated retry. If a DataSync fails, the recovery path is to decommission the asset and recreate it. Run Test Connector before creating assets to reduce this risk.

• Schema evolution in place is not supported. If the storage path or schema changes, the asset must be decommissioned and recreated.

• Disabling the Enable At-Source toggle retains the Unity Catalog Storage Credential and External Location — it does not revoke access to the storage. Full revocation requires deleting the connector in Harbr and removing the IAM trust policy on the customer side.

• Each storage location can only have one connector. If a connector already exists for a location, you must delete it before creating a new one.