Create a Data Asset

Assets are units of data that users can manage, distribute, and consume. Creating an asset allows users to choose from a multitude of configured connector types. The data can stay at source or be ingested to the platform. This means that no data movement is necessary and the data does not need to be stored on any particular cloud provider.

Depending on your platform configuration, Assets can be created from a range of internal and external sources and connectors:

• AWS S3

• GCP GCS

• Azure Blob

• Snowflake

• BigQuery

Assets can also be uploaded from your local drive, or created from internal sources within the platform, such as Spaces (see below) and Tasks (see here).

Note: You must have an Asset Creator role to perform this action.

Create an Asset using Connectors

1. Go to Manage > Assets.

2. Click Create New Asset.

3. Complete the setup:

   • Asset name

   • Connector source

   • Data type (ie. table or file)

   • Storage type (ie. at source or on platform)

     • For cloud storage sources (e.g. S3, Azure Blob or GCS), this will be path to data.

     • For data warehouse sources (e.g. Snowflake or BigQuery) this will be a selection that is dependent on source, such as a database and table/view.

   • Indicate whether the path is to a single file or folder containing files and folders

   • Select whether data is to be At Source or On Platform

Remember: Creating an At Source Asset means that the data will not be copied to the platform, it will remain and be accessed at source. There is no data movement occurring with this option.

Creating an On Platform Asset means that a copy is taken and stored on the Harbr platform.

Choosing At Source or On Platform is available optionality to support your specific data storage, location and movement requirements or restrictions.

Once this step is complete, the Status of the Asset is set to IN SETUP

Create an Asset using Drag and Drop

Users with asset creation permissions are able to kickoff an easy, simple, automated asset creation process by uploading or dragging and dropping a file from the My collection or Manage assets views.

You can do this from:

1. My collection

2. My collection or Manage asset views, users can drag and drop a file over the page.

3. By using the “Upload file” button on the My collection or Manage assets views.

If any of the above options are used, an upload dropdown will appear in the navigation bar which lists all of the in-progress, failed or successfully completed asset creation jobs.

As part of the process, the file type is identified and the data is automatically processed until the asset is ready for use.

If the user is managing the asset, it is available for editing and releasing. If the user would like to consume the asset, it is available for addition to space or export. In the unlikely event that the asset creation fails, each step of the process has a failure state.

Create an Asset Using Spaces

Having examined the contents of one or more assets or products in a Space you can create your own custom asset by combining multiple inputs which you can then share with other users on the platform. These assets use a Space as a Source.

You must have an Asset Creator role to perform this action.

You must have saved the results as a table in the publish_db of the Space.

As a simple example, execute this SQL (via SQL Lab) in a Space to create a custom data product”

DROP TABLE if exists publish_db.<example table>;

CREATE TABLE publish_db.<example table>

as

SELECT count (100) from <dbname.tablename>;

To create an asset from a space:

1. Click on the Assets icon in the manage from down found in the navigation bar.

2. Click on Create Asset.

3. Name the Asset

4. Select a specific Space as the source.

5. Complete the asset creation and set-up process.

Create an Asset using Tasks

Note: You must have an Asset Creator and Automation Creator or Administrator role to perform this action. You must also have successfully executed your Code Asset via a Task to create an an Asset in this way.

Having examined the contents of one or more products or assets in a Space, created a product or asset in publish_db, re-used your code as a code asset, scheduled its execution via a task you can share this updating asset with other users on the platform.

The subscription lineage of the asset from a task is determined by the subscriptions the publisher of the data product has to the contributing data products.

To create an asset from a task:

1. Click on the Assets icon in the manage from down found in the navigation bar.

2. Click on Create Asset.

3. Name the Asset

4. Select a specific Task as the source.

5. Complete the asset creation and set-up process.

Set up an Asset

1. Proceed to Asset Set Up, by clicking Start Setup.

2. View a summary of the values entered during Asset Creation.

3. Await completion of set up tasks, which may include:

   • Data access check (File and Table assets)

   • Data format check (Table assets)

   • Schema confirmation (CSV Table assets only)

     • In the case that the data format is found to be CSV, a schema review is required before setup can be completed. This is to allow any adjustments to the schema that have been discovered, for instance, to table properties or data types. Often, no adjustment is necessary. Once this has been reviewed and the correct structure is available, the schema must be saved.

4. Click Finish setup.

Once you have completed the setup of an asset, you will be directed to the Asset Overview page. The Status is updated from IN SETUP to DRAFT

Release an Asset

Releasing an Asset makes it available to be included within a Product and allows the data within the Asset to be consumed directly, for instance, in Spaces, Query, Export or via a Data Share.

Once you have completed the setup of an Asset it is moves from IN SETUP to DRAFT and you must then release the Asset to move it to a LIVE state. Only LIVE assets can be added to Products or Shared and made available to be consumed.

The release panel is found on the right of the asset overview page and the release button will be blue when its enabled and a release can be performed. All tabs on the Asset Overview page are accessible and editable while the asset is being prepared for release so you can make adjustments in the meantime.

To release a DRAFT Asset:

1. Go to Manage on the Navigation bar

2. Select Assets

3. Search for or filter the list to find a specific Asset

4. Click on your DRAFT Asset

5. Review the Key Information and Metadata.

6. Click on the Release Asset button on the right of the screen.

The Asset moves to a LIVE state immediately.

Share an Asset

Once an asset has been set up and is in a Draft or Live state, it can be shared with other users or organizations.

Note: Sharing an asset is only possible by users that have manage asset permissions.

1. Click the share button found on the asset overview page

2. A modal displays users that already have access to the asset

3. To add an new user or org, click add user/org

4. Search for the user or org you would like to add and select

5. Once the user or org is selected and listed, choose the permissions you would like to grant, there are two permissions available:

   1. Use - upon saving this will immediately grant the selected users/orgs the ability to consumer the asset in spaces and export

   2. Manage - upon saving this will immediately grant the selected users/orgs the ability to manage the asset. This includes the ability to share and delete the asset.

6. Click save to apply.