--- language: "en" --- # How can we help? ## User Documentation ### [Welcome](https://docs.harbrdata.com/v6/welcome.md) ### [Administer](https://docs.harbrdata.com/v6/administer.md) ### [Discover](https://docs.harbrdata.com/v6/discover.md) ### [Create](https://docs.harbrdata.com/v6/create.md) ### [Manage](https://docs.harbrdata.com/v6/manage.md) ### [Consume](https://docs.harbrdata.com/v6/consume.md) ### [Guides](https://docs.harbrdata.com/v6/other-resources.md) ### [Release Notes](https://docs.harbrdata.com/v6/release-notes.md) --- language: "en" --- # 4.10 Maintenance Release ## What's New Whilst we are working hard on our next major platform release we wanted to slip out a couple of small changes to help make your day more enjoyable. ### Faster page refreshes We know how frustrating it can be to wait for a page to refresh so we've optimised several pages that on average are up to **40%** faster on AWS and **25%**faster on GCP 💥. ### Faster Task executions We've also enabled Tasks to run more quickly so if you want to experiment with how to [Automate the Creation of your Custom Data Products](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2544764051/Automate+the+Creation+of+your+Custom+Data+Products), your first automatic execution of code will run much more quickly so you can [Check the Status of a Task](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2544764121/Check+the+Status+of+a+Task) to view your results in just a few minutes. ### Add more Data Products to a Space (4.10.2, 4.10.4) For our customers using an AWS platform we have raised the number of Data Products you can add to a space from 9 to 14. ### Presto has been replaced by Trino (4.10.3) For our customers using a GCP platform, Presto is no longer available to be used in a Space as it has been superceded by Trino - see also [How do I migrate from using Presto to Trino? (GCP Only)](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2684125217/How+do+I+migrate+from+Presto+to+Trino). ## Bug Fixes 1. Minor fixes that are so minor you won't even notice them. ## Release Start Date: 22FEB23 --- language: "en" --- # 4.11 Maintenance Release ## What's New Whilst we are building up our next major platform release we have some general platform improvements to share with you. ## Bug Fixes 1. To avoid a subsequent issue, you are restricted to selecting asset objects (tables/views) that are from a single Snowflake database only. We've updated this article [Create and Share a Remote Snowflake Data Asset](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2570616960/Create+and+Share+a+Remote+Snowflake+Data+Asset). 2. The description for your Space specification has been restricted to a maximum of 1000 characters. 3. Various niggling issues in Spaces have been fixed. ## Release Start Date: 21MAR23 It might take a few days before the release is available in your platform(s). --- language: "en" --- # 4.12 Maintenance Release ## What's New Whilst we are building up our next major platform release we have some general platform improvements to share with you. ## Bug Fixes 1. Editing an existing Task and adding a collaborator may cause an error ## Release Start Date: 18MAY23 It might take a few days before the release is available in your platform(s). --- language: "en" --- # 4.13 Maintenance Release ## What's New Whilst we are building up our next major platform release we have some general platform improvements to share with you ## Improvements 1. A security issue preventing large Snowflake assets being queryable in spaces has been resolved. 2. Users can now create code assets written using trino-sql and create a Task using these trino-sql based CAs 3. Users can now create code assets from Superset Lab ## Bug Fixes 1. Improved error handling for R and Python Automated Tasks. Previously, in some instances they could fail silently 2. Product icon is now visible in the dropdown when adding products to a Space 3. Fixed an intermittent JupyterLab 'directory not found error' issue ## Release Start Date: 29 June 2023 It might take a few days before the release is available in your platform(s). --- language: "en" --- # 4.7 Enabling SSO ## What's New ### Identity management and enablement of SSO integration We have implemented [Auth0](https://auth0.com/?utm_content=branded-homepage&utm_source=google&utm_campaign=emea_uki_gbr_all_ciam-all_dg-ao_auth0_search_google_text_kw_utm2&utm_medium=cpc&utm_term=auth0-c&utm_id=aNK4z0000004GJkGAM&gclid=Cj0KCQjwhY-aBhCUARIsALNIC062W02TlpVixLFKUGcokIGrGMvrc7RU_ScXJRKsb_gfIpWN0QQWcLEaAlobEALw_wcB) to manage identities and platform access for users and provide the option to enable secure yet frictionless entry to the Platform. If SSO hasn't been setup for you yet then you won't notice many changes. ### More Python Packages available in Spaces (AWS) PyPi is the Python package management framework that is used for installing packages into a python development and analytics environment. A private PyPi repository is now available for use within Spaces. This allows users to install Python packages that have been provisioned into the repository with standard `pip install `commands from JupyterLab notebooks: ![BBUEjPh4bMm0sV8hOL196dM1XjJIrojt4o_WGlCbgyrlVCfv1-56emPJD2YV_D2UldNPira4uLD1rG4sqoErOEdFqEtoaShslLlfRVKUs1FFoBWYzCZOz3lJLfsj4UFWKDfVqxGgvc88GoEcWarcgNz9gjqThveRWyHlOuASARLIR05U8I4kO74aLjAy](https://docs.harbrdata.com/__attachments/a_cb55a5e993d70f41b366c0028ebeaf7c0f1ee90499ff0e9b4a8ce47146dc90f9/BBUEjPh4bMm0sV8hOL196dM1XjJIrojt4o_WGlCbgyrlVCfv1-56emPJD2YV_D2UldNPira4uLD1rG4sqoErOEdFqEtoaShslLlfRVKUs1FFoBWYzCZOz3lJLfsj4UFWKDfVqxGgvc88GoEcWarcgNz9gjqThveRWyHlOuASARLIR05U8I4kO74aLjAy?cb=5cd790262d37743ad3d119a2d07f5d51) Example package installation from the Private Repo For more information see [Using Python in a Space](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2544763572/Using+Python+in+a+Space) ## Bug Fixes * When adding an Internal subscription plan with Enforced data lineage to an Engineered Data Product the subscription plan was displaying as Not Enforced. * When the name of an Organization changes, the user profiles of users from that Organization still displayed the old Organization name. * When viewing Space Collaborators the "Last used" date and "Hours Logged" fields are non-functional and have now been removed. --- language: "en" --- # 4.8 Get Ready for Snowflake Assets ## What's New ### Evaluating Snowflake Assets in Spaces Most of our work for 4.8 has been focussed on allowing users to evaluate Snowflake Assets in Spaces in GCP and then in AWS (due in 4.9) which is summarised in [Evaluating Snowflake Assets in Spaces](https://harbrgroup.atlassian.net/wiki/display/HDV5/Coming+soon%3A+Snowflake+Assets+in+Spaces). The feature is turned off by default so contact your Platform Owner to discuss having it enabled for your Organization. ### Enhanced platform legibility We are continuously looking for ways to improve the way you use the platform so we have adjusted all display colors to achieve a higher contrast and greater legibility. We have also removed the text size reduction hover effect from data product tiles on the exchange so that data product name now just remains the same larger size upon hover. ## Bug Fixes * Users can be re-invited post expiry even if the user had subscriptions allocated. * Depending on your SSO configuration, when you log out of the Platform your SSO session is also disabled. * Auto logout is enforced after 1 hour of inactivity on an open browser. When the user has been logged in and has closed the browser without logging out, the token is invalidated after 4 hours. * When applying a Trial subscription to a Data Product, the duration of the Trial must be at least 1 day * (GCP only) The help text link in the Spaces Help modal has been removed - the screenshot has been updated here [Access your Secure Desktop from your VDI](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2536571238/Access+your+Secure+Desktop+from+your+VDI). * (GCP only) It is once again possible to add more than 9 Data Products to a Space. Sorry about that one. ## Release Start Date: 16NOV22 --- language: "en" --- # 4.9 Enabling Snowflake Assets, Faster Queries and an Operators Portal ## What's New ### Evaluating Snowflake Assets in Spaces If you would like to use this feature and you do not see the *Assets* icon and the associated *Data Assets* tab then please contact Support as you must have the Asset Administrator or Asset Creator [User Roles](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2544764557/User+Roles+V4+5) After all the prep work we have been doing, we would now like to give you a warm [Welcome to Data Assets](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2571567105/Introducing+Products+and+Assets) (see Getting Started). Data Assets are our vision for the future and you can create and share them with others directly or you can package a single Data Asset into multiple different data products. Our first step on this journey is to allow you to create a Data Asset based on data objects held in a Snowflake instance where the objects remain remote, in Snowflake, and are not copied into the platform. You can [Create a Snowflake Data Asset and add to an Evaluation Space](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2570715225/Create+a+Snowflake+Data+Asset+and+add+to+an+Evaluation+Space) to compliment your analysis of other data assets within a Space. The introduction of Data Assets starts to appear throughout the platform in the following articles: [Understanding Spaces](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2536574597/Understanding+Spaces) [Specify the Data, People, Tools and Compute you Need](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2544763445/Specify+the+Data+People+Tools+and+Compute+you+Need) [Manage your Endpoints](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2544763311/Manage+your+Endpoints) ### Execute Faster Queries with Trino With this release Spaces have been upgraded to incorporate Trino as an additional and complimentary query engine to Hive and Spark within the environments. [Trino](https://trino.io/) is an open source distributed query engine that is highly optimised for query performance across multiple data sources and is visible when you [Activate your Tools and Compute](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2544763477/Activate+your+Tools+and+Compute). For Spaces this means : * 💥 💥 \~10x performance increase for Data Product queries compared to HIVE 💥 💥 * Provides secured connection to Snowflake to execute queries against Assets that have been added to a Space For guidance on using Trino within Spaces see [Using SQL in a Space](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2544763537/Using+SQL+in+a+Space) , [Using Python in a Space](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2544763572/Using+Python+in+a+Space) and [Combine Data Assets and Data Products in a Space](https://harbrgroup.atlassian.net/wiki/spaces/UJN/pages/2573041726/Combine+Data+Assets+and+Data+Products+in+a+Space) . ![:info:](https://docs.harbrdata.com/__attachments/a_f9a0d67d766c156284be3c0b4ab5c12159efd037ebe207ace4dc59ac1c9b4fd9/atlassian-info?cb=feab5cd71111204d6b52545f3027dd0c) Note : At the current time Trino can only be leveraged within Python based Automated Tasks. Support for SQL based Automated Tasks will be provided in up-coming releases. ### Operators Portal If you are part of the platform owners team then you will be invited to join the [Operators Documentation Portal](https://ops.harbrdata.com/pot/?l=en) which is behind a secure login. We think we've got everyone but if we missed you and you would like access then please submit a request via the Support link on the header of this portal. You may need to arrange for access to this portal, via Azure single sign on, to be authorised by your IT Administrator. ## Bug Fixes 1. Performance improvements have been made to the Products page where there is a need to load over 150 publishers and 650+ published Data Products. ## Release Start Date: 11JAN23 --- language: "en" --- # 5.0 Products & Assets **5.0 i**s a major release that changes how Products and data are managed and touches most of the areas in the platform. Key benefits are: * **Products** and their underlying data, i.e. **Assets**, can be managed independently, allowing different teams to work autonomously to launch and evolve a Product faster and more easily. * Assets can now be shared across **multiple Products, and Products can exist without any Assets,** streamlining the management of data and unlocking the capability to test demand of new Product lines. * Changes to both Products and Assets can now be **staged** and **released when ready**. This helps control the evolution of a Product, while maintaining the existing consumer experience. ## **Overview of changes** **PRODUCTS** * **Manage products** . The **Manage Products** view lists all **Products** accessible to a user. This may be for Product creation, editing, subscription management or release depending on the user, their roles or their relationship to the Products listed. The whole area has been reworked for better user experience and to separate data management into assets. * **Overview** page. The **Product** overview page is shown once a Product has been created. This page enables management, sharing, editing and releasing of the Product. * **Assets** . Products may contain data in the form of **Assets**, these are added in the Assets section. Users can gain access to these Assets via a subscription to a product. * **Packaging** . To make a product more easily discoverable and informative on the **exchange**, users can describe their Product, add visuals and select categories and tags. These will be visible on the product page. * **Subscription plans** give users the ability to consume Products and the Assets they may contain. * **Visibility** . The **visibility** setting allows Product managers to expose the Product to target users/organizations. * **Last refresh date.** The **last refresh date of a product** can be updated manually or set to update in sync with an identified asset that the product contains. * **Sharing product** for management. Users must have manage permissions for a Product to **share** it with other users or organizations. Product sharing grants **management** permissions only. * **Releasing a product.** Users must have a Product administrator role to **release** a Product. Releasing a Product is possible from the Product overview page using the release panel. Drafted changes can be identified using the yellow change markers. Releasing a Product transitions its state from **draft** or **draft with change** s to **live**. * **Deleting a product.** Users with a **Product administrator** role are able to delete products published by users in their organization. **Ecosystem administrators** are able to delete any product from any organization. * **Subscription management.** All subscription management permissions will be shifted away from an organization admin and grouped under a new role, **Subscription administrator**. Subscription administrators are able to view all products released by users in their organization and allocate, replace and expire subscriptions for users and organizations. Subscription management is available via the manage products view for users with this role. **ASSETS** * **Manage assets.** The **Manage Assets** view lists all **Assets** accessible to a user. This may be for Asset creation, editing, sharing or release depending on the user, their roles or their relationship to the Assets listed. * **Asset overview.** The**Asset overview**page is available once the Asset setup is complete. From this view, users with manage permissions are able to share and release the asset. The tabs available let an asset manager review details of the **source, review data dictionary, data sample, metadata and updates.** * **Asset sources** . Users can create Assets from a number of supported **connector** types. Where available, users are able to choose for their Asset to remain **at source** or to have it stored **on** **platform**. * **Data dictionary.** For **table** Assets, a **data dictionary** is generated. The information contained in this view includes all columns, their data types and a description that can be set. **Data sample.** For table Assets, a 10 row **data sample** is generated. A toggle is included on this tab for users to decide whether to have the data sample displayed on the Product page or not. * **Asset updates**. Asset updates allow users to setup periodic updates that refresh asset data. They can be toggled on from the updates tab. Once enabled, the user will be provided with: * **Data location** - users load the new data into a staging location on their connector storage * **Trigger file location** - once the new data has completed loading, users upload a trigger file that specifies where the new data is. The system will then ingest data from the specified location. * **Status file location** - the system will supply an end-of-job status file containing the outcome of the load. **Important:** Only assets that are created from connectors can be updated, meaning that those that are manually uploaded cannot have their data refreshed. * **Asset sharing.** Users must have manage permissions for an Asset to **share** it with other users or organizations. Asset sharing grants **Use** and/or **Management** permissions. * **Use** * Use in Space * Create Export * **Manage** * View and edit * Share * Release * **Release asset.** Users can complete the first release of an Asset using the release panel on the right of the overview page. Assets can only be released once all data copy or preparation processes are complete. * **Deleting asset** . **Assets** can be deleted from the manage assets view or the asset overview page. During deletion, the user will be informed of existing asset relationships before they confirm deletion. Some relationships block the user's ability to delete the asset and need to be managed first. If asset is in setup state, there will be no dependencies and it can be deleted straight away. * **Asset usage.** To understand how your asset is being used by consumers, an asset **usage** view is available on the asset overview page. **EXCHANGE AND NAVIGATION** * **Navigation** . The layout of platform features has been updated to clearly separate the activities performed by **producers** and **consumers**. Users performing activities in either of these categories see only what is applicable to them, depending on their roles. ![8Qw1t5etdRMZ2RXGe7vDOT3WrhSaa505bwoEQL4MUs9HLer_1cifPC24kGbnohYNl55qRx09Q54XGyZnU12Zo9deqWiUF7h6Vk3ezHTVXA18KvkdB1mPIcl9S_iB4f8dtB4neDWNtohFdxqJs0vRU96nrA=s2048](https://docs.harbrdata.com/__attachments/a_dcf1406325aec493cfd039be10c192f8907e2715f5bee80c77c3bf592e4c3e8c/8Qw1t5etdRMZ2RXGe7vDOT3WrhSaa505bwoEQL4MUs9HLer_1cifPC24kGbnohYNl55qRx09Q54XGyZnU12Zo9deqWiUF7h6Vk3ezHTVXA18KvkdB1mPIcl9S_iB4f8dtB4neDWNtohFdxqJs0vRU96nrA=s2048?cb=5a80c682be6f98631174ba2d108fe850) **Updated product page** **Producers** can provide information about their product to enable consumers to make informed decisions on whether to subscribe or contact the producer to negotiate a subscription. Product page shows: * Summary * Assets * Subscription plans * Usage permissions **MY COLLECTION** My collection is a single place where users can view the products for which they have **active subscriptions** and the assets for which they have **use**permissions. For each product and asset, users can see the type, owning-organization, the method through which they obtained access and the usage permissions they have. Sort, filter and pagination controls enable easy navigation of the table items. **USE PRODUCTS AND ASSETS** * **Use assets in spaces.**Users with space permissions are able to select products and/or assets when creating a space. * Where assets are contained in a product, the subscribed user is only able to select the product unit for addition to the space. This brings **all**assets contained in the product, into the space. * Where users have received asset usage permissions directly via a share, assets will be listed separately and can be added to a space as single units. * **Export assets and products.**Users with export permissions are able to select products or assets for export. Only one product or asset can be exported at a time. * Products and assets for export are selectable from a table with a number of filter and search components to help find the asset for export more easily. * Where products contain a mix of file and table assets, users must choose one asset type to export and create another export for the other. * At source asset exports are not yet supported. * **Automation** . Automation allows users to create code assets as a result of some space activity. These code assets contain some instruction in reference to various **products and/or assets** contained in a space, this is used to create some output data. * Using tasks, code assets can be set to run on a repeating basis thus enabling repeating derived outputs from a space. * The task and the code asset set up is frozen once linked with a product to ensure consistent data outputs that also uphold any enforced subscription lineage that may apply to the implicated **products and/or assets**. **SUBSCRIPTION LINEAGE** When users create new assets from some activity in a space (i.e from a space or task) the relationships between contributing products/assets and the newly created asset is tracked. This is known as **lineage**. In cases where producers would like to maintain control and transparency of who is consuming their data assets, they can make use of the **lineage enforcement setting.** It is set to either **enforced or not enforced** on a subscription plan. A subscription plan is then applied to a product. Subscription plans govern how users consume the data once they're subscribed. Setting subscription lineage to **enforced** means that wherever a product is used to create new assets/products, the consumers of those new assets/products will be **required to gain subscription** access to the contributing products too before they can use the new assets/products. Where subscription lineage is**not enforced,** any contributing products are simply part of the record of lineage but consumers of new assets/product don't require subscriptions to them too. **UPDATES TO ROLES** |--------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | **New/updated roles** | **Update** | | **Organization administrator** | Can no longer manage product subscriptions, can no longer view endpoints, can no longer delete products | | **Automation creator** | Name change from "Automation" -\> "Automation creator." To achieve consistency across role names | | **Product creator** | New role. Allows users to create new products. (these users receive management permissions for their products as a result of creation) | | **Product administrator** | New role and name update from "Release manager". Allows users to view, manage, release, share and delete (coming soon) products in their organization (deletion formerly reserved for org admins) | | **Product manager** | Will be phased out for 5.0. Migration of current product managers to persist permissions is necessary | | **Asset creator** | New role. Allows users to create assets. Allows users to view, manage, release, share and delete. | | **Asset administrator** | New role. Allows Allows users to view, manage, release, share and delete assets in their organization | | **Subscription administrator** | New role. Allows users to manage subscriptions for products in their organization. Formerly reserved for org admins | --- language: "en" --- # 5.1 Maintenance release ## Improvements * Space specifications can now be controlled at the organization or user level * Updated explanation for 'on platform / at source' asset options * **Asset creation** form has been reordered and starts with asset type selection (file/table) * **Asset updates** tab is not shown anymore for assets created from a space, task or upload sources * **'Location'** is shown when viewing finish asset setup page or key information tab. Location can be either 'On Platform' or 'At Source' * **Related products** are now enabled again and shown in a product page * For products that have been upgraded from 4.x and had no repeating updates, updates option will be disabled when viewing assets that are part of the product * Ecosystem admins viewing the**manage products** view will have the organization filter preset to their own org * File based asset file mounts * Space startup time improvements ## Bug Fixes 1. Eco admin cannot invite users to a new org when their org is non-interacting 2. Product Admin - Option to Manage Subscription present but doesn't have permissions to expire 3. Subscription expiry performed by ecosystem admin fails when attempting to expire a subscription for a product not owned by their own organisation ## Release Start Date: 2nd November 2023 It might take a few days before the release is available in your platform(s). --- language: "en" --- # 5.12 Asset section in Homepage * Add sections with assets to a Homepage (AWS, GCP, Azure) * **Other changes/fixes:** * Simplified the use of feature flags for connectors and how they are controlled --- language: "en" --- # 5.14 Export to SFTP and Activity Log * **Export to SFTP (Early Access, AWS).** * Setup a connector to your SFTP location and use it as a destination for export. * **Activity Log (Early Access, AWS, GCP, Azure)** * A single view of all the logged activity in the platform. Events are logged for actions across assets, products, organisations, connectors, users and more. * **Various bug fixes and improvements:** * The main logo dimensions have been updated to have 97x50 dimensions to allow for better legibility * Note: this new setting is disabled by default and will be enabled once there is a need. * Fixed an issue where asset data dictionary UI would overflow if there were multiple tags * Added support for additional iframe sandbox options in data product description embedding * Open XLSX assets directly (Early Access, AWS) --- language: "en" --- # 5.15 New signup form and profile changes, updated connector view * **Updates to user signup form and profile (AWS, Azure, GCS).** * User signup and profile have been streamlined with an additional option to set avatar image * **Connection tests and updated connector view (Azure)** * Added connection tests post asset creation for all connectors available on Azure platform * A connector list has been redesigned for better user experience * **Various bug fixes and improvements:** * Removed 'Description' label in asset details modal if no description is provided. * Display connector type next to an event in the Activity Log (Azure, AWS) * Added user filter to the Activity log * Improved logging for export errors --- language: "en" --- # 5.16 Multi-asset query, PowerBI connector and Content asset types * **Ask AI questions about multiple assets in one go (AWS, Early Access)** * **New, configurable asset types (e.g. Agent, Workspace) for storing content such as url (All, Private Preview)** * **PowerBI connector (AWS, Azure, Private Preview)** * Enables creating visualisation assets from PowerBI as a source * Reports are then viewed directly in the platform * **New 'Access' permission (AWS, Azure, GCP)** * New **Access** permission has been added to the available permissions * If turned on, it allows accessing visualisations or new 'content' asset types * **Various bug fixes and improvements: (AWS, Azure, GCP)** * Added a filter to exchange to filter products by organisation * Additional theme improvements for the Homepage * Export and task fixes to reduce failure rates --- language: "en" --- # 5.17 Product Page Redesign and Whitelisting Domains Harbr has redesigned the Product Page to improve the overall user experience, enhance data discoverability and consumption. **Some of the key enhancements for the first phase of the redesign:** * Introduction of a single-page scrolling layout, replacing the old tabs based layout. * Removal of some legacy elements to give users a more clean and less cluttered UI, and create space for full page width content in future. * Restyling of components for displaying assets, plans and information/metadata in the product page. * Redesign of the subscribe flow, with a clearer, single-modal review and checkout experience. This includes states for requesting a subscription, agreeing to T\&Cs, viewing an active subscription, and cancelling a subscription. * Ecosystem metadata and whitelisting domains for user invites where a new, optional capability has been added that allows Ecosystem admins to restrict invitations to only allowed email domains. --- language: "en" --- # 5.18 New design for Ecosystem and Org admin areas, Custom Metadata for products, Footer component * Both **Ecosystem** and **Organization** administration areas now feature a new, vertical menu design that has configuration options grouped by their section. * Products now have a capability to have and display **Custom Metadata** to allow adding additional metadata for Products that is visible in the exchange. * Custom Metadata is added in the packaging area in the product. One or more key/values pairs can be added and once saved, the product needs to be released for the changes to be visible. Key/Value pairs can also be removed or edited afterwards. * A new configurable **Footer** section is now available for the Ecosystem administration area. It allows Eco Admins to enable and configure the Footer component that is shown at the bottom of the platform for all pages. We recommend configuring and previewing the footer to ensure that the content is visible and usable. The configuration options are: * **Logo**. Logo is shown on the left side. * **Alternate text.**Optional descriptive text for accessibility reasons. * **Links**. Up to 5 links can be added, though we recommend keeping the number lower for usability. * **Content**. Additional text content that will be displayed in the footer (e.g. copyright information). The footer can be enabled or disabled at any time and the setting applies to all users. --- language: "en" --- # 5.19 Subscription Plan controls, Product Header Image, Desktop as a Controllable Source * Ecosystem controls that allows **disabling** certain options for subscription plan template configuration (for both Ecosystem and Organization) to prevent users creating subscription templates with these options selected. The options that can be disabled are: * **Trial** (Period) * **Multi-user**(Subscription options) * **Managed subscriptions**(this can be used to replace existing feature flag configuration) * Product header image has been re-introduced as an optional packaging setting for products. It can be set within manage products and is displayed at the top of the page before the rest of the content. * Desktop source can be optionally turned off by our support team if you prefer users to either not create assets or export to desktop as a target. --- language: "en" --- # 5.2 Snowflake and BigQuery connectivity ## Summary * **5.2** is focused on expanding connectivity of Harbr Data platform with **Snowflake** and **BiqQuery** as both sources for assets and as targets for export. You can now: * Create **on platform** assets for **Snowflake** and **BigQuery** * Set up **repeating updates** for those assets to keep data up to date * **Export** Products or Assets to **Snowflake** and **BigQuery** locations (applies to assets that store tables) * ***Note*** *: The above is available for AWS platform only at the moment* * **Changes in Spaces** * Engineering built 'Open Source' Space specs (replacing functionality like for like with the CS deployed ones) * Improved startup times * Superset dashboard import from File Asset; Customisable Superset Icon and favicon *** ** * ** *** ![image-20231206-123418.png](https://docs.harbrdata.com/__attachments/a_569d201be29ae135e799787f0bcc078567ad09f1a7e13bb31b2fc3cb1d28237f/image-20231206-123418.png?cb=986c99825ccfda5165f85ce51f8fb5cf) ## Create a 'On Platform' Snowflake asset In addition to**at source** Snowflake assets, you can now create **on platform** Snowflake assets. When creating an asset, you need to provide a **connector** , select **database** and **table/schema**that will be used to create an asset. These types of assets have all the same features as at source Snowflake assets but can also be **exported**, both individually and as part of a product. ## Update your Snowflake asset Once Snowflake 'On Platform' asset is created, there are two ways to ensure it has the latest data: * **Manual update**- by selecting Run Update * **Scheduling regular updates -** by enabling and editing schedule. Schedule can be set to repeat at a certain time and interval. Any update would update **'last refresh date'** for the asset and trigger any exports or tasks that are configured against the asset. *Note that the time for scheduled update is set at UTC timezone.* ## Export to Snowflake Any **table** asset or product that is **on platform** can now be exported to a **Snowflake** location, that is specified using the **Snowflake** connector and database/schema that data will be exported to. There will be a new table created on the specified database/schema, with a table name specified by the users before export. By default, the names are set to: * **Assets**: asset_provisionedname * **Product**: product_provisionedname_asset_provisionedname The data will be overwritten on repeat exports. Supported export size has been tested up to 400 GB. *Note that connector selected for export requires permissions that enable user to create tables and connection test will be done before the export.* *** ** * ** *** ![image-20231206-131230.png](https://docs.harbrdata.com/__attachments/a_b5a8cb7cfe8014d26b8cc5cb4128e3d03e87c6f1213a3b91fda33885a3b276d6/image-20231206-131230.png?cb=27daaa2728edd289a6db4b40cab16141) ## Create a BigQuery connector Connector is used to store the credentials that are used to connect to BigQuery when creating assets or exporting to a BigQuery location. For **BigQuery** **connector**, you will need to specify: * Basic details, such as name and description * Upload a GCP service account key file * Specify BQ Project ID if it wasn't available in the file Once connector is created, we will perform a connection test to ensure it is usable. ## Create a 'On Platform' BigQuery asset You can now create **on platform** BigQuery assets. When creating an asset, you need to provide a **connector** , select **dataset** and **table** that will be used to create an asset. Once created, the asset behaves the same way as any other asset and can be added to product or used in Spaces/Export. ## Update your BigQuery asset Once BigQuery 'On Platform' asset is created, there are two ways to ensure it has the latest data: * **Manual update**- by selecting Run Update * **Scheduling regular updates -** by enabling and editing schedule. Schedule can be set to repeat at a certain time and interval. Any update would update **'last refresh date'** for the asset and trigger any exports or tasks that are configured against the asset. *Note that the time for scheduled update is set at UTC timezone.* ## Export to BigQuery Any **table** asset or product that is **on platform** can now be exported to a **BigQuery** location, that is specified using the **BigQuery** connector and dataset that data will be exported to. There will be a new table created on the specified dataset, with a table name specified by the users before export. By default, the names are set to: * **Assets**: asset_provisionedname * **Product**: product_provisionedname_asset_provisionedname * Names cannot contain whitespace characters The data will be overwritten on repeat exports. Supported export size has been tested up to 400 GB. *Note that connector selected for export requires permissions that enable user to create tables and connection test will be done before the export.* *** ** * ** *** ![icon_spaces.png](https://docs.harbrdata.com/__attachments/a_1188ed3507b5999ee3d32512ad25237605931c62a1e584076e26ff4c2f9bf714/icon_spaces.png?cb=bdbd0a115c6b0e5849eb4e2685882ff4) ## Open Source Space specs * Cluster service upgrades (Dataproc 2.1, EMR 6.11) * Superset upgraded to 2.1 * Superset now supports custom icons and favicons * Trino upgraded to v423 * RStudio (2023.06.0+421) \& R (4.3.1) upgraded * New 'SQL' icon linking directly to Superset SQLLab * Tools installed via docker (increased stability post release preventing downstream library changes) * Jupyter running Spark 3.3.0 * New Spark context enabling Trino within Python * JupyterLab git integration * Hue, Zeppelin, Hadoop \& Spark removed as tool options from portal *** ** * ** *** ## Improvements * Ability to download asset dictionary to a CSV file * When creating assets from a task, available tables can now be selected from a dropdown instead of having to enter the path within a space. *** ** * ** *** ## Fixed bugs * Organization interaction rules no longer override an Ecosystem Administrator's ability to see organization names when viewing products on the platform in the manage products view. * When an eco admin visits the manage products view, their organization is selected in the organization filter by default. Should the eco admin choose a different organization to filter on, we have enabled that when they navigate in and out of a product, the selected organization on the manage products list is remembered and it doesn't keep reverting to their organization. * When managing subscriptions, the user details are now exposed. Previously this information was presenting 'Single user' and to find out who the user was, the subscription manager was having to open the subscription management view. ## Release Start Date: 7th December 2023 It might take a few days before the release is available in your platform(s). --- language: "en" --- # 5.20 Improved data view controls, enable or disable subscription prices, publishing org on product page * Updated category and tag filter design * We've restyled our data view control components and the way they are laid out in the header area, with a focus on improving usability and appearance. * We've also built some new components which are being added to our data view controls to further improve usability. These include filter chips to show your filter selections clearly. The chips provide an easier way to clear filters individually. * We've also built a new sort dropdown component. * New header and filtering controls for lists/tables/grids * Allow enabling/disabling displaying pricing for Subscriptions in the Exchange * A new Ecosystem setting has been added that allows controlling whether prices will be shown on subscriptions. This is disabled by default. * Show publishing organization on a product page in the updated UX * Added tooltip to explain subscription users option * Separate User and Service Accounts into separate screens --- language: "en" --- # 5.21 Product presentation, UI customization and enabling throttling jobs * Further roll out of improved data view controls * Product presentation improvements and more flexibility with images Includes as default: * Rectangular tile image with 10:7 ratio * A new placeholder tile image when no custom image is set for a product * Product name displayed in full * Publishing organization name * Asset type icons shown on the bottom left * CTA (Subscribe) or subscription 'statuses' on the bottom right - can be: * Subscribe button linking to Plans section of product page * Subscribed status - when there is an active subscription * Subscribed status with 'attention' badge - when user action is needed * 'Request pending' status when an approval request has been sent*(if approvals are enabled)* * Summary text hover state (now only shown if summary text exists) * UI Customizations: Platform administrators will now be able to customize parts of the UI for all users, initially via settings for: * Exchange * My Collection * Homepages * Updates to asset card in the Homepage * Enable throttling asset and export jobs if required * Display service accounts within an organisation view for Ecosystem Admin when non-parent org is viewed --- language: "en" --- # 5.22 Create At Source Databricks assets for Delta Sharing and Configure Platform Currency * **Configure Platform Currency** * A currency can now be set at platform level, in the Ecosystem admin area under Metadata. This platform currency will be shown wherever prices are displayed to users in the UI: * In the Subscription Plan templates screens * In Manage products where plans are applied * On the Product page * The display format that's users will see is dependent on their platform locale: * Prices are configured by ecosystem admins using the ISO 4217 currency codes system. * Seeing the locale symbol (e.g. £, $...) will depend on the platform locale. * Users whose platform locale is not set to match the currency will see the currency code directly, e.g. "USD 100". * If no platform currency is defined, the default will be USD ($). * It should be noted that this is not a currency conversion capability. Instead, we are displaying the *given*currency beside the number that is set for the product pricing. * **Create At-Source Databricks Assets for Delta Sharing** * If the Databricks connector is configured in the platform to support this asset location,**At Source** assets can be created from it and used to create Delta Shares (both for the specific asset and as part of the product). There are a number of differences for**At Source**assets compared to On Platform assets: * **Sample Data**. Returns the first set of rows in the table instead of a random sample * **Metadata**. We are estimated the size instead of doing the full scan of the entire table at source. * Assets will not support the following usage types: * Export * Query * Spaces --- language: "en" --- # 5.23 Homepage Builder and Search, Lakehouse Federation (EA), and new Events for Product and Subscription Approvals * Lakehouse Federation (Early Access) * As an early access feature, some environments can now enable Lakehouse Federation for the following connectors: Microsoft SQL, MySQL, BigQuery and Snowflake. * To enable '**At Source** ' assets to be created from these connectors, simply select the *'Enable 'at source' assets'*' option when creating or editing an existing connector. This will automatically set up a federated connection and catalog behind the scenes. * Homepage Builder * Ecosystem Admins can now use a simple form-style page builder to structure and populate homepages (replacing the previous json editor) * New Search section on Homepage * If search is turned on for a Homepage, users see a search input in the hero section that lets them find products. The user can either click on one of the listed products to navigate to that product's page, or view all results on the Exchange * Additional events for product release and subscription approvals * A number of new events have been added to aid with monitoring and actioning product release and subscription approval requests. These can be viewed in the **Activity Log**for your organisation. They include: * Subscription Requested * Subscription Request Approved * Subscription Request Rejected * Product Release Requested * Product Release Approved * Product Release Rejected --- language: "en" --- # 5.24 Looker Connector, Activity logging for events relating to Secrets, Homepage improvements, Improved error messages for connection test failures **New features** * Looker Connector * Activity logging for events relating to Secrets * Homepage improvements * Allow users to set whether the text in the hero section of a page is white or black in **Edit homepage** * Fix an issue that allowed the same section order number to be applied to multiple sections in **Edit homepage** * Add a warning modal to warn users they may lose changes when they click 'Cancel' in **Edit homepage** * Improved error messages for connection test failures **Improvements and Fixes** * Added ability for Support team to retrigger failed asset update jobs * Added new sort options for My Collection (Created, Released) * Improved error handling when user is opening product page they don't have access to * Simplified the names of subscription requests events --- language: "en" --- # 5.25 Export and Asset update improvements and Space spec improvements **New Features** * Improvements to export and asset update reliability * Allow UTF-8 characters for user first and last and org names * Automatically select space spec when creating a space if there is only one * New Synapse connector (if supported and configured on your platform) * Upgraded Superset version used in Spaces **Improvements and Fixes** * Do not show inactive users when sharing or collaborating platform objects * Fixed an issue where Space would not start if both owner and collaborator ran it at the same time * Fixed an issue where scrollbar did not appear on the Platform Admin Panel when in 'Exit Full Screen' --- language: "en" --- # 5.26 New Setting for Subscription Plan T&Cs and Improved result grouping in Global and Product search **New features** * Export products and assets to an external SFTP location * Allow changing Text on the user accept "Terms and conditions" for subscription plans **Fixes and Improvements** * Improved the performance listing exports in the Export screen * Fixed an issue where Data Sharing option was available in a Subscription template despite user setting data egress option to 'Do not allow' * Improved result grouping in Global and Product search --- language: "en" --- # 5.27 Subscription Plan Changes and Payment Gateways * Row and column filtering on Subscription Plan Improvements have been made to the Subscription Template screens for both Ecosystem and Org Admins, streamlining the experience, giving more control over pricing and enabling additional subscription methods. * Subscription template and plan management changes for products The changes to subscription are backwards compatible and will not break existing plan templates or plan, however certain values will be shown under different sections and with values that have different names. * Payment gateways If subscription plan is configured with subscription method **self-serve with payment**and payment gateway is configured for the platform, users will be able to pay for subscriptions directly. --- language: "en" --- # 5.28 AI Powered Discovery * AI Powered Discovery**(Private Preview)** * Data Loading and Export jobs can now run on job clusters * Product and Exchange Search Improvements * Row and Column filtering: Added support for Spaces and Tasks **(Early Access)** --- language: "en" --- # 5.29 Open Pages, Field renaming on Plan templates, Spaces Collaboration and Connector updates * **Platform Operators can now create publicly accessible Open Pages that allow unauthenticated visitors to browse and discover Data Products, without requiring an account or platform login.** *With this new feature, by removing the credential barrier to be able to view products, your Marketplace can engage potential consumers earlier in their journey, moving them seamlessly from curiosity to discovery, and ultimately, to subscription.* *Each Open Page is configured by a Platform Operator with a unique URL path, custom header and branding, and configurable conversion buttons (e.g. a link to your platform registration or login page).* *Data Producers control which products can be added to appear on Open Pages by setting the visibility on a product to "Open". The process of enabling this is outlined in the following slides.* *Unauthenticated consumers can view product name, description, publisher, asset names, types, and metadata (columns, rows, size). Harbr's* ***Product Search Bar*** *can also feature to allow visitors to search products specifically curated for that page.* *When a logged-in user visits an Open Page product link, they will be redirected directly to the full product page. Only after login, are asset metadata, data dictionaries, sample data, subscription plans, and the Subscribe button available to view.* * **In suitable environments, Users can now collaborate on content within Spaces, such as dashboards, charts, saved SQL queries and query history.** *Please note: If a Space was created prior to this release, the collaboration capability will not be enabled. For this change to apply, the Space will need to be recreated.* * **Where file extension or size validation is enabled, a clearer error is now shown to the use** * **Option to rename 'visibility' setting to 'eligibility' on plan templates** * **Where available, New PostgreSQL and EIDF Connectors** * **IP access restrictions for open delta shares** * **Improved error handling for Exports** --- language: "en" --- # 5.3 Additional At Source BigQuery and S3 asset sources ## Summary * Further expansion of connectivity of Harbr Data platform: * Create and use **at source** assets from **BigQuery**, without having to move the data * Create and use **at source** assets from **S3**, without having to move the data * Create and use **on platform** assets from **Snowflake** and **BigQuery** andexport Products or Assets to **Snowflake** and **BigQuery** locations (applies to assets that store tables)**(now also available on GCP platform)** * **Other improvements** * **At sourceSnowflake** assets now show sample data, dictionary and metrics ## Create and use **on platform** assets from **Snowflake** and **BigQuery** ## Export Products or Assets to **Snowflake** and **BigQuery** locations (applies to assets that store tables) Now also available on GCP platform. These features are identical to AWS version described in 5.2 release notes. ![image-20231206-131230.png](https://docs.harbrdata.com/__attachments/a_51f3e6c33e0438209d89c4ce6b731922f9d7785ec75f21a2d33f2f8ae67e49fa/image-20231206-131230.png?cb=27daaa2728edd289a6db4b40cab16141) ## Create a 'At Source' BigQuery asset In addition to**on platform** BigQuery assets, you can now create **at source platform** BigQuery assets without moving the data.\* When creating an asset, you need to provide a **connector** , select **dataset** and **table/view**that will be used to create an asset. Only table assets can be created from this source. These assets behave the same way as other at source assets - they can be queried in Spaces, added to product but cannot be exported. *\*Available on both AWS and GCP platforms.* ![image-20240206-140751.png](https://docs.harbrdata.com/__attachments/a_ef673a0045ee4a4f31c6b9871c793ee3a9b0010960e7e45cc18175b17451e248/image-20240206-140751.png?cb=f58f9c7319bd5173e4459a5dbb43d7fe) ## Create and use an 'At Source' S3 asset In addition to**on platform** S3 assets, you can now create **at source platform** S3 assets without moving the data.\* When creating an asset, you need to provide a **connector** and paththat will be used to create an asset. Both table and file assets can be created from this source. These assets behave the same way as other at source assets - they can be queried in Spaces, added to product but cannot be exported. Row count is not available for this type of asset. *\*Available on both AWS and GCP platforms.* ## Fixed bugs * **At source Snowflake** assets now show sample data, dictionary and metrics ## Release Start Date: 25th January 2024 It might take a few days before the release is available in your platform(s). --- language: "en" --- # 5.30 Release Notes Release notes for Harbr version 5.30 \| June 2026. Please select your platform deployment type for the features and fixes relevant to your environment. Guidance for any changes or improvements captured within the release are linked within the page. * [**5.30 Release Notes --- AWS Data plane**](https://docs.harbrdata.com/v6/5-30-release-notes-aws.md) * [**5.30 Release Notes --- Databricks Workspace**](https://docs.harbrdata.com/v6/5-30-release-notes-databricks.md) --- language: "en" --- # 5.30 Release Notes — AWS Release notes for Harbr version 5.30, covering features and fixes for platforms deployed using an AWS data plane. ## New Features ### [Workload Identity Federation (WIF) for the GCS Connector](https://docs.harbrdata.com/v6/using-a-gcs-bucket.md) General Availability **Workload Identity Federation (WIF)** is now supported for the Google Cloud Storage (GCS) connector. Operators can authenticate to GCS without provisioning or rotating long-lived service-account key files. * WIF is configured at the connector level during connector creation or update. * Existing GCS connectors using service-account key authentication continue to function; migration to WIF is optional. For more information, see [Using GCS](https://docs.harbrdata.com/v6/using-a-gcs-bucket.md). ### [Custom Plan Visibility](https://docs.harbrdata.com/v6/subscription-plans.md) General Availability **Custom Plan Visibility** gives platform operators control over which organizations and users can see a given plan. Plans can be restricted so they are visible only to intended audiences, rather than all users on the platform. * A new **plan visibility editor** is available in **Manage Products / Plans**. * Visibility can be scoped to specific organizations, specific users, or left open to all. * Visibility settings can be applied at plan creation or updated on existing plans. For more information, see [Subscription Plans](https://docs.harbrdata.com/v6/subscription-plans.md). ### Custom Plan Names General Availability Plans can now be given a display name, set individually for each plan in **Manage Products / Plans**, rather than inheriting the name from the plan template. The plan name is used as the main identifier wherever a plan is shown, across both the Harbr UI and the APIs. Previously, every plan created from the same template shared that template's name. When a producer used one template to create several plans --- for example, with different row and column filtering on each --- the plans were hard to tell apart. Giving each plan its own name makes it clearly identifiable to both providers and consumers. * A **Plan name** field is available in the plan editor under **Manage Products / Plans**. * The plan name is shown wherever a plan is referenced --- Exchange plan cards, subscription details, and the subscriptions tables --- in place of the template name. * A plan name can be set when a plan is first created, or added later and released as part of a subsequent product release. * The plan name is subject to release control in the same way as other plan settings in **Manage Products / Plans**. * A plan name change on an already-released plan applies to new subscribers only. Existing subscribers see no change --- a backfill migration is handled automatically as part of this release. ### Restrict Display of Asset Usage based on Capabilities General Availability Asset sharing and **My Collection** now display only the usage types the platform can fulfill for the requesting user or organization. Usage options that cannot be fulfilled are no longer surfaced in sharing flows, collection views, or available actions. * Asset sharing for usage now occurs after the asset has been released, and is restricted to the usage types the asset type supports. * A **Usage** column is now shown in **My Collection**: * For **products**, the usage shown is what is defined on the plan. Usage types that cannot be achieved by any single asset in the product are shown greyed out --- for example, if the plan includes exports but no assets in the product are of an exportable type. * For **assets**, the usage shown is the intersection of what the asset type supports and what the user has permission for. * Actions available on a product or asset are now aware of the usage the asset supports. For example, the **Create export** option is not shown for an asset if the asset type does not support export. *Note: This release updates what is shown and available in the UI --- it is surface-level filtering, not backend enforcement. Enforcement of fulfillment restrictions at the capability level will be introduced in a future release.* ‌ ## Fixes * Improved resiliency by fixing a queue behaviour that could cause exports to time out or not complete. * Extended file upload validation to cover additional file types, further protecting the platform against potentially harmful uploads. * Fixed logo picker replacing the original image with a degraded crop when re-opened after rotating. * Fixed logo picker preview going blank when rotation moved the image outside the crop boundary. * Improved logo upload handling when wide logos are enabled for an organization. ‌ ## Improvements * A Subset chip now appears next to product and plan names wherever row or column filtering is active on the plan. Clicking the chip opens the filtering details without leaving the current view. The chip appears in product views, subscription views, and selector screens. * Improved error handling to ensure clearer and more consistent responses when platform operations encounter failures. * Extended flexibility to allow users with non-eco-admin roles to create Subscription Plans with approval-based configurations. * Export events now identify the triggering organization in the Event stream and Activity Log. Previously, these events were logged anonymously. * Global search now handles UK/US spelling variants. Searching for a term such as "colour" will also return results containing "color", and vice versa. This behaviour is enabled via the `enableSpellingVariants` flag in the `globalSearchConfig` ecosystem metadata setting. * Updated AI-powered Discovery character handling, and [product search prompt configuration](https://ops.harbrdata.com/v5/configuring-long-description-search). --- language: "en" --- # 5.30 Release Notes — Databricks Release notes for Harbr version 5.30, covering features and fixes for platforms deployed using a Databricks workspace. ## New Features ### At-Source Tabular Asset Cataloguing for Object Stores General Availability At-source cataloguing for tabular assets on object stores is now supported on Databricks data planes. Operators can register and catalog tabular assets directly from cloud storage without copying data into the platform. * At-source assets are catalogued directly from cloud storage, such as via the S3 and Azure Blob Storages, with no data movement required. * Once catalogued, assets can be used in Spaces, Query, Export, and Data Sharing in the same way as on-platform assets. *Note: Support for at-source cataloguing depends on the cloud hosting your Databricks workspace. On Databricks workspaces hosted on Azure, both the S3 and Azure Blob Storage connectors are supported; for the Azure Blob Storage connector, the storage account and the Databricks workspace must be in the same Microsoft Entra tenant, as cross-tenant configurations are not yet supported.* *On Databricks workspaces hosted on AWS, support for the S3 and Azure Blob Storage connector are not yet supported. Please contact your Harbr Account Manager to confirm compatibility with your environment before enabling this feature.* *Note: Currently, these assets can only be consumed via Delta Sharing if the data is in Delta format and accessed via an AWS connector. Assets loaded via the Azure Blob Storage connector will be supported this way in a future release.* For more information, see [Using S3](https://docs.harbrdata.com/v6/using-an-s3-bucket.md), [Azure Blob Storage](https://docs.harbrdata.com/v6/using-azure-blob-storage.md) and [Databricks Workspace Permissions](https://ops.harbrdata.com/v5/databricks-workspace-permissions) ### [Custom Plan Visibility](https://docs.harbrdata.com/v6/subscription-plans.md) General Availability **Custom Plan Visibility** gives platform operators control over which organizations and users can see a given plan. Plans can be restricted so they are visible only to intended audiences, rather than all users on the platform. * A new **plan visibility editor** is available in **Manage Products / Plans**. * Visibility can be scoped to specific organizations, specific users, or left open to all. * Visibility settings can be applied at plan creation or updated on existing plans. For more information, see [Subscription Plans](https://docs.harbrdata.com/v6/subscription-plans.md). ### Custom Plan Names General Availability Plans can now be given a display name, set individually for each plan in **Manage Products / Plans**, rather than inheriting the name from the plan template. The plan name is used as the main identifier wherever a plan is shown, across both the Harbr UI and the APIs. Previously, every plan created from the same template shared that template's name. When a producer used one template to create several plans --- for example, with different row and column filtering on each --- the plans were hard to tell apart. Giving each plan its own name makes it clearly identifiable to both providers and consumers. * A **Plan name** field is available in the plan editor under **Manage Products / Plans**. * The plan name is shown wherever a plan is referenced --- Exchange plan cards, subscription details, and the subscriptions tables --- in place of the template name. * A plan name can be set when a plan is first created, or added later and released as part of a subsequent product release. * The plan name is subject to release control in the same way as other plan settings in **Manage Products / Plans**. * A plan name change on an already-released plan applies to new subscribers only. Existing subscribers see no change --- a backfill migration is handled automatically as part of this release. ### Restrict Display of Asset Usage based on Capabilities General Availability Asset sharing and **My Collection** now display only the usage types the platform can fulfill for the requesting user or organization. Usage options that cannot be fulfilled are no longer surfaced in sharing flows, collection views, or available actions. * Asset sharing for usage now occurs after the asset has been released, and is restricted to the usage types the asset type supports. * A **Usage** column is now shown in **My Collection**: * For **products**, the usage shown is what is defined on the plan. Usage types that cannot be achieved by any single asset in the product are shown greyed out --- for example, if the plan includes exports but no assets in the product are of an exportable type. * For **assets**, the usage shown is the intersection of what the asset type supports and what the user has permission for. * Actions available on a product or asset are now aware of the usage the asset supports. For example, the **Create export** option is not shown for an asset if the asset type does not support export. *Note: This release updates what is shown and available in the UI --- it is surface-level filtering, not backend enforcement. Enforcement of fulfillment restrictions at the capability level will be introduced in a future release.* ### [Federation Proxy](https://docs.harbrdata.com/v6/federation-proxy.md) General Availability Federation Proxy enables Federated Connectors to securely access source systems that are not directly reachable from Databricks Serverless compute, including on-premises databases, private corporate networks, and sources protected by IP allowlisting (e.g. MySQL). #### How it works * Routes connector traffic through a Harbr-managed proxy with fixed IP addresses. * Source system owners allowlist the proxy IPs instead of Databricks Serverless IP ranges. * Automatically applied to new federated connectors when enabled. #### Benefits * Enables Delta Share and other Serverless-powered consumption journeys for restricted data sources. * Simplifies network access management with a small, stable IP allowlist. * No changes to producer or consumer workflows. * Data remains in the source system and is queried in real time. *Note: This feature requires platform-level configuration by a Harbr operator. Please contact your Harbr Account Manager to discuss.* For more information, see [Configuring Federation Proxy](https://harbrgroup.atlassian.net/wiki/spaces/ODV5/pages/4531617793/Configuring+Federation+Proxy). ‌ ## Fixes * Fixed an issue preventing collaborators from activating Spaces when joining from a different organization. * Extended file upload validation to cover additional file types, further protecting the platform against potentially harmful uploads. * Fixed logo picker replacing the original image with a degraded crop when re-opened after rotating. * Fixed logo picker preview going blank when rotation moved the image outside the crop boundary. * Improved logo upload handling when wide logos are enabled for an organization. ‌ ## Improvements * A Subset chip now appears next to product and plan names wherever row or column filtering is active on the plan. Clicking the chip opens the filtering details without leaving the current view. The chip appears in product views, subscription views, and selector screens. * Extended flexibility to allow users with non-eco-admin roles to create Subscription Plans with approval-based configurations. * Export events now identify the triggering organization in the Event stream and Activity Log. Previously, these events were logged anonymously. * Updated AI-powered Discovery character handling, and [product search prompt configuration](https://ops.harbrdata.com/v5/configuring-long-description-search). * Improved resiliency by fixing a queue behaviour that could cause exports to time out or not complete. * Improved resiliency by hardening approval request processing so persistent failures are dead-lettered rather than retried indefinitely. * Improved resiliency by tuning platform auto-scaling and database connection handling to better manage high-volume workloads. * Improved error handling to ensure clearer and more consistent responses when platform operations encounter failures. * Gobal search now handles UK/US spelling variants. Searching for a term such as "colour" will also return results containing "color", and vice versa. This behaviour is enabled via the `enableSpellingVariants` flag in the `globalSearchConfig` ecosystem metadata setting. --- language: "en" --- # 5.31 Release Notes Release notes for Harbr version 5.31 \| August 2026. Please select your platform deployment type for the features and fixes relevant to your environment. Guidance for any changes or improvements captured within the release are linked within the page. * [++**5.31 Release Notes --- AWS Data plane**++](https://docs.harbrdata.com/v6/5-31-release-notes-aws.md) * [++**5.31 Release Notes --- Databricks Workspace**++](https://docs.harbrdata.com/v6/5-31-release-notes-databricks.md) --- language: "en" --- # 5.31 Release Notes - AWS Release notes for Harbr version 5.31, covering features and fixes for platforms deployed using an AWS data plane. ## New Features ### MCP Discovery Tools EARLY ACCESS Harbr's MCP server now exposes a read-only discovery surface, allowing users and AI agents to search, browse, and inspect products and assets on a user's behalf. * Search products by free text (eg. product name, summary and product description) using prompts related to what you can do with them or by asset type. * Open a single product to see the caller's own permissions on it, list its assets, and inspect an asset's schema. * Only what the signed-in user is actually allowed to see is ever returned --- the same organisation rules, product visibility, and plan eligibility that apply everywhere else on the platform are respected here too. *Note: this release covers discovery only (search, browse, inspect) --- subscribing, exporting, and sharing via MCP are not yet included.* For**user guidance,** please see . For **operator set-up** information, see [Configuring the MCP Server](https://ops.harbrdata.com/v5/configuring-the-mcp-server). ‌If you are interested in configuring this feature, please contact your Harbr Account Manager. ### AI Query Private Preview **AI Query** is a new asset type that allows a producer to provision access to an existing Databricks Genie Agent through the Harbr UI. The functionality currently facilitates question-and-answer with text and table responses only. * Producers register a Genie Agent they've already built by choosing a Databricks connector, selecting from the Genie Agents that connector can reach, and publish. * Consumers granted the ++'access' permission++on products, are able to ask questions in natural language via a chat modal that opens in place from the product page and My Collection. * No credentials are shared, no data is moved onto the platform. Rather, every question is proxied through Harbr using the connector's identity, so no user accesses Databricks directly, and each question is recorded for usage and cost visibility. * Conversations are persisted inside a session. Old sessions are not saved. For more information on how to create and consume this new asset type, please see . ‌If you are interested in configuring this feature, please contact your Account Manager. ## Improvements ### Databricks Connector Setup Checks Harbr now runs a series of capability checks on Databricks connectors before they can be finalized, checking from two vantage points --- from Harbr, and from the processing environment --- so a non-functional connector is more likely to be identified during setup rather than discovered later. For guidance, please see the relevant section within . For operator information, see [Configuring Connector Capability Checks](https://ops.harbrdata.com/v5/configuring-connector-capability-checks). ### Asset Usage Permission Enforcement This is the second phase of the asset usage permission changes introduced in[5.30](https://docs.harbrdata.com/v6/5-30-release-notes). The previous release updated **what was shown** in sharing flows and My Collection, by hiding usage options the platform couldn't fulfil for a given asset or plan. Usage permissions can now only be granted for usage types the asset actually supports --- enforcing, at grant time, the same restriction that has been surfaced since 5.30. Other related improvements to this feature include: * Asset creators automatically keep full usage permissions, including any new usage types added to the platform later. * An asset creator's own usage permissions can no longer be changed by other users. * Usage permissions can no longer be granted on an asset before it has been released. *Note: Each of these three rules is controlled independently via ecosystem metadata and is currently set to off by default. Please see* [*Configuring Usage Permission Enforcement*](https://ops.harbrdata.com/v5/configuring-usage-permission-enforcement)*for set-up guidance.* ### Other Improvements * New connectors list shows full detail parity for every connector type and Connector create/edit now uses a sticky page layout. * Introduced a scheduled sweep of delta share permissions to ensure accuracy. * Prevented editing the source database on a connector with 'at source' enabled. * Databricks connectors now specify their SQL warehouse via a dedicated field, rather than through integration metadata. * A batch of hardening improvements, including dependency updates and rebuilt images to patch known vulnerabilities, and a fix for an HTML injection issue in the Spaces description field. * Improvements to platform performance. ‌ ## Fixes * Fixed Jupyter app creation failures caused by a casing mismatch in Athena credentials. * Fixed bug where IAM Role ARN not showing for some S3 connectors. * Fixed row-expand control making table rows taller than others. * BigQuery connectors now return a clear message for a malformed service-account key. * Fixed asset update failures after a prior partial attempt caused by an SFTP folder-creation issue, and intermittent SFTP connection errors during asset read/pull. * Fixed intermittent timeouts on the endpoint of data products in Spaces. * Fixed the Databricks connector silently failing to re-authenticate after its secret was rotated. ‌ --- language: "en" --- # 5.31 Release Notes - Databricks Release notes for Harbr version 5.31, covering features and fixes for platforms deployed using an Databricks workspace. ## New Features ### MCP Discovery Tools EARLY ACCESS Harbr's MCP server now exposes a read-only discovery surface, allowing users and AI agents to search, browse, and inspect products and assets on a user's behalf. * Search products by free text (eg. product name, summary and product description) using prompts related to what you can do with them or by asset type. * Open a single product to see the caller's own permissions on it, list its assets, and inspect an asset's schema. * Only what the signed-in user is actually allowed to see is ever returned --- the same organisation rules, product visibility, and plan eligibility that apply everywhere else on the platform are respected here too. *Note: this release covers discovery only (search, browse, inspect) --- subscribing, exporting, and sharing via MCP are not yet included.* For**user guidance,** please see . For **operator set-up** information, see [Configuring the MCP Server](https://ops.harbrdata.com/v5/configuring-the-mcp-server). ‌If you are interested in configuring this feature, please contact your Harbr Account Manager. ### AI Query Private Preview **AI Query** is a new asset type that allows a producer to provision access to an existing Databricks Genie Agent through the Harbr UI. The functionality currently facilitates question-and-answer with text and table responses only. * Producers register a Genie Agent they've already built by choosing a Databricks connector, selecting from the Genie Agents that connector can reach, and publish. * Consumers granted the ++'access' permission++on products, are able to ask questions in natural language via a chat modal that opens in place from the product page and My Collection. * No credentials are shared, no data is moved onto the platform. Rather, every question is proxied through Harbr using the connector's identity, so no user accesses Databricks directly, and each question is recorded for usage and cost visibility. * Conversations are persisted inside a session. Old sessions are not saved. For more information on how to create and consume this new asset type, please see . ‌If you are interested in configuring this feature, please contact your Account Manager. ### Workload Identity Federation (WIF) for the GCS Connector Workload Identity Federation (WIF) is now supported for the Google Cloud Storage (GCS) connector. Operators can authenticate to GCS without provisioning or rotating long-lived service-account key files. * WIF is configured at the connector level during connector creation or update. * Existing GCS connectors using service-account key authentication continue to function; migration to WIF is optional. For user guidance and further information, please see [Using GCS](https://docs.harbrdata.com/v6/using-a-gcs-bucket). ### **At-Source Cataloguing for Azure Blob Storage** At-source cataloguing for Azure Blob Storage now works even when your storage account sits in a different Microsoft Entra tenant than your Databricks workspace. *Note: This extends the At-Source Tabular Asset Cataloguing for Object Stores feature shipped in* [*5.30*](https://docs.harbrdata.com/v6/5-30-release-notes-databricks)*, which noted that cross-tenant configurations were not yet supported for the Azure Blob Storage connector. That restriction has now been lifted.* * Setup is unchanged from your perspective --- enter your Azure service principal credentials as normal when creating the connector. Harbr handles the cross-tenant access on its side automatically. * Existing same-tenant Azure Blob Storage at-source connectors continue to work as before; this adds the cross-tenant case, it does not change the same-tenant path. For operator detail on how this is provisioned, please see ### Delta Table Detection for the Azure Crawler The Azure connector's crawler now detects Delta tables. The value this enables is that at-source Azure tables in delta format can be added to Delta shares *Note: This extends the At-Source Tabular Asset Cataloguing for Object Stores feature shipped in* [*5.30*](https://docs.harbrdata.com/v6/5-30-release-notes-databricks)*. While Delta detection for at-source assets previously worked via the S3 connector, it now also works via Azure Blob Storage.* ## Improvements ### Set-up Improvements for Various Connector Types Harbr now runs a series of capability checks on a connector before it can be finalized, checking from two vantage points --- from Harbr, and from the processing environment --- so a non-functional connector is more likely to be identified during setup rather than discovered later. For user guidance, see the relevant section within . For operator info, see [Configuring Connector Capability Checks](https://ops.harbrdata.com/v5/configuring-connector-capability-checks). ### Desktop Upload Improvements Desktop asset uploads have been improved for platforms using the upload proxy rather than pre-signed URLs --- most relevant where a customer's storage account must remain private. * Desktop asset uploads no longer time out after four minutes on larger files. * Fixed a validation bug affecting desktop uploads. * Finally, binary/desktop-download export zips now build in a way that works on serverless compute. For operator detail on how to configure this capability, please see [Configuring Desktop Upload](https://ops.harbrdata.com/v5/configuring-desktop-upload). ### Asset Usage Permission Enforcement This is the second phase of the asset usage permission changes introduced in[5.30](https://docs.harbrdata.com/v6/5-30-release-notes). The previous release updated **what was shown** in sharing flows and My Collection, by hiding usage options the platform couldn't fulfil for a given asset or plan. Usage permissions can now only be granted for usage types the asset actually supports --- enforcing, at grant time, the same restriction that has been surfaced since 5.30. Other related improvements to this feature include: * Asset creators automatically keep full usage permissions, including any new usage types added to the platform later. * An asset creator's own usage permissions can no longer be changed by other users. * Usage permissions can no longer be granted on an asset before it has been released. *Note: Each of these three rules is controlled independently via ecosystem metadata and is currently set to off by default. Please see* [*Configuring Usage Permission Enforcement*](https://ops.harbrdata.com/v5/configuring-usage-permission-enforcement) *for set-up guidance.* ### Other Improvements * New notification type for data-share updates needing attention. * Connector creation now stops cleanly at the Azure load-balancer limit. * New connectors list shows full detail parity for every connector type and Connector create/edit now uses a sticky page layout. * Introduced a scheduled sweep of delta share permissions to ensure accuracy. * Prevented editing the source database on a connector with 'at source' enabled. * Databricks connectors now specify their SQL warehouse via a dedicated field, rather than through integration metadata. * A batch of hardening improvements, including dependency updates and rebuilt images to patch known vulnerabilities, and a fix for an HTML injection issue in the Spaces description field. * Improvements to platform performance. ‌ ## Fixes * Magicbyte validation optimisation for file upload. * Fixed row-expand control making table rows taller than others. * Query page can no longer be reached by direct URL when the Query feature is off. * BigQuery connector now returns a clear message for a malformed service-account key. * Fixed asset update failures after a prior partial attempt caused by an SFTP folder-creation issue, and intermittent SFTP connection errors during asset read/pull. * Fixed intermittent timeouts on the Spaces data products endpoint. * Fixed the Databricks connector silently failing to re-authenticate after its secret was rotated. ‌ --- language: "en" --- # 5.32 Release Notes Release notes for Harbr version 5.32 \| September 2026. Please select your platform deployment type for the features and fixes relevant to your environment. Guidance for any changes or improvements captured within the release are linked within the page. * [++**5.32 Release Notes --- AWS Data plane**++](https://docs.harbrdata.com/v6/5-32-release-notes-aws.md) * [++**5.32 Release Notes --- Databricks Workspace**++](https://docs.harbrdata.com/v6/5-32-release-notes-databricks.md) --- language: "en" --- # 5.32 Release Notes - AWS ## New Features ### At-Source Databricks on AWS Databricks assets can now be queried in Spaces and exported on AWS-hosted platforms, with the data staying in your own Unity Catalog throughout. Harbr catalogues the metadata; the data itself never moves. * In Spaces, queries run live against your Databricks SQL warehouse, so results always reflect the current state of the data. * For Export, data is read at the point you export, to produce the export file. * You pay for Databricks compute only when you query or export. Nothing runs simply because the asset is catalogued in Harbr. Spaces and Export are enabled separately by an Ecosystem Admin, and Export requires an additional platform-side deployment. To have this configured, please contact your Harbr Account Manager. For **operator set-up,** see [Configuring Databricks At-Source Assets on AWS](https://ops.harbrdata.com/v5/configuring-databricks-at-source-assets-on-aws). ### New MCP Tools for Subscribing, Exporting and Data Sharing EARLY ACCESS Building on the Discovery MCP Tools introduced in 5.31, your AI assistant can now act on your behalf for several new workflows, all from within a normal chat conversation. * Subscribe to a product on a self-serve plan, request access to an approval-gated plan and track its status, or activate a subscription that is on hold pending terms acceptance. * Create a data share of a product or a standalone asset over Databricks, either an open, token-based share usable by any recipient, or a direct Databricks-to-Databricks share into the recipient's own Unity Catalog. * Export a product, setting the destination, name, description, refresh schedule, and email notifications. If you do not see the new tools right away, disconnect and reconnect your AI assistant to refresh the available tool list. For **user guidance** , please see [Using the Harbr MCP](https://docs.harbrdata.com/v6/using-the-harbr-mcp). For **operator set-up** , see [Configuring the MCP Server](https://ops.harbrdata.com/v5/configuring-the-mcp-server). If you are interested in configuring this feature, please contact your Harbr Account Manager. ## Improvements ### Additional Connector Types Supported by Capability Checks EARLY ACCESS Connector capability checks now run for Google Cloud Storage, Azure, Amazon S3, DB2, SFTP, Power BI and Looker, alongside the database connectors covered since 5.31. The checks test a connector from two places before you set it live: from Harbr, and from the processing environment where your jobs actually run. A connector that would have failed later is caught during setup instead. This release extends that same checking to further connector types: Google Cloud Storage, Azure, Amazon S3, DB2, SFTP, Power BI, and Looker. For **user guidance** , see the relevant section within [Create a Connector](https://docs.harbrdata.com/v6/create-a-connector). For **operator set-up** , see [Configuring Connector Capability Checks](https://ops.harbrdata.com/v5/configuring-connector-capability-checks). ### More Granular Control Over User Deletion EARLY ACCESS Ecosystem admins can now safely remove a user directly from the platform, with control over who can do so, whether that's a named list of admins or all ecosystem admins. Deletion is a soft deactivation rather than a hard delete: the person's access is revoked immediately, but everything they created is retained and re-attributed to "Name (Deactivated)", nothing they owned is transferred or lost. * A user cannot be deleted while they have a running Space, a running export, a scheduled or in-flight task, or own a Space with active collaborators, the platform explains exactly what needs resolving first before the deletion can proceed. * Deleting a user who was invited but never registered, or a service account, is out of scope for this release. For **operator set-up** , see [Delete a User](https://ops.harbrdata.com/v5/delete-a-user). ### Improvements to Connector Behaviour * The SFTP connector is more resilient to transient network conditions. --- language: "en" --- # 5.32 Release Notes - Databricks ## New Features ### Serverless compute for Jupyter tools. JupyterLab can now run on serverless compute, joining SQL Lab and Superset which already use serverless SQL warehouses. Your Space is ready sooner, and nothing changes about how you work once it is running. For **user guidance** , see the "Serverless in Spaces" section within [Spaces](https://docs.harbrdata.com/v6/spaces). ### Serverless for Asset Create EARLY ACCESS Asset creation and updates can now run on Databricks serverless compute rather than a dedicated cluster. * Available for Oracle, MySQL, SQL Server, and PostgreSQL connectors only. * Requires both a platform-level and a connector-level opt-in, and is intended for non-production workloads at this stage. For **operator set-up** , see [Configuring Serverless Compute for Asset Creation](https://ops.harbrdata.com/v5/configuring-serverless-compute-for-asset-creation). ### New MCP Tools for Subscribing, Exporting and Data Sharing EARLY ACCESS Building on the Discovery MCP Tools introduced in 5.31, your AI assistant can now act on your behalf for several new workflows, all from within a normal chat conversation. * Subscribe to a product on a self-serve plan, request access to an approval-gated plan and track its status, or activate a subscription that is on hold pending terms acceptance. * Create a data share of a product or a standalone asset over Databricks, either an open, token-based share usable by any recipient, or a direct Databricks-to-Databricks share into the recipient's own Unity Catalog. * Export a product, setting the destination, name, description, refresh schedule, and email notifications. If you do not see the new tools right away, disconnect and reconnect your AI assistant to refresh the available tool list. For **user guidance** , please see [Using the Harbr MCP](https://docs.harbrdata.com/v6/using-the-harbr-mcp). For **operator set-up** , see [Configuring the MCP Server](https://ops.harbrdata.com/v5/configuring-the-mcp-server). If you are interested in configuring this feature, please contact your Harbr Account Manager. ## Improvements ### Additional Connector Types Supported by Capability Checks EARLY ACCESS Connector capability checks now run for Google Cloud Storage, Azure, Amazon S3, DB2, SFTP, Power BI and Looker, alongside the database connectors covered since 5.31. The checks test a connector from two places before you set it live: from Harbr, and from the processing environment where your jobs actually run. A connector that would have failed later is caught during setup instead. This release extends that same checking to further connector types: Google Cloud Storage, Azure, Amazon S3, DB2, SFTP, Power BI, and Looker. For **user guidance** , see the relevant section within [Create a Connector](https://docs.harbrdata.com/v6/create-a-connector). For **operator set-up** , see [Configuring Connector Capability Checks](https://ops.harbrdata.com/v5/configuring-connector-capability-checks). ### More Granular Control Over User Deletion EARLY ACCESS Ecosystem admins can now safely remove a user directly from the platform, with control over who can do so, whether that's a named list of admins or all ecosystem admins. Deletion is a soft deactivation rather than a hard delete: the person's access is revoked immediately, but everything they created is retained and re-attributed to "Name (Deactivated)", nothing they owned is transferred or lost. * A user cannot be deleted while they have a running Space, a running export, a scheduled or in-flight task, or own a Space with active collaborators, the platform explains exactly what needs resolving first before the deletion can proceed. * Deleting a user who was invited but never registered, or a service account, is out of scope for this release. For **operator set-up** , see [Delete a User](https://ops.harbrdata.com/v5/delete-a-user). ### AI Query Assets --- Consistent Access Permissions AI Query assets now apply the same access permission checks used across the rest of the platform, so only users with the correct permissions can call a Space. ### Databricks Asset Update Retry and Timeout Asset updates now recover from transient failures on their own, and can be given a timeout so a stalled update does not run for hours. Retries cover network failures, connection stalls and Databricks job timeouts. The timeout is calculated from the asset's own typical run time. For **user guidance** , please see the relevant section in[Databricks.](https://docs.harbrdata.com/v6/databricks) For **operator set-up** , see [Configuring Databricks Asset Update Retry and Timeout](https://ops.harbrdata.com/v5/configuring-databricks-asset-update-retry-and-timeout). ### Improvements to Connector Behaviour * The SFTP connector is more resilient to transient network conditions. * When browsing catalogs, schemas, and tables during Databricks connector set-up, the platform now only shows resources the connector's account can actually use to create an asset. * Oracle connector listing shows only the objects your account can access, without needing elevated database privileges. For **user guidance** , please see the relevant section within [Oracle](https://docs.harbrdata.com/v6/oracle). ## Fixes * Organisation creation no longer completes silently when the required Unity Catalog schema fails to provision. * Complex data types from Databricks are now handled correctly. --- language: "en" --- # 5.4 New Workbench spaces that do not require VDI * New **Workbench** option for spaces and updated Spaces user experience. **Note that is this is available only on AWS.:** * Depending on the user's permissions to current and historical data in a Space, they will encounter either the Workbench or Sandbox spaces experience. * **Workbench** is a lightweight, fast way to access a space. Internet access remains in tact but there is no need to use the secure desktop. * **Sandbox** is the most secure way to access the data in a space. It requires that users log in to the virtual desktop to achieve the most secure environment to prevent unwanted data egress. * **My collection empty state updates**- when users first login to the platform and have yet to receive any consumption permissions for assets or subscriptions for products, they are met with a My Collection empty state that guides them towards what they can do. This gives action options and explanations on what the user can do next to start their data journey. Depending on permissions, users are encouraged to visit the exchange, create assets or drop files in My collection to begin automatic asset creation. * **Quick asset creation** from My collection or Manage assets views - Files can simply be dropped into the My Collection or Asset management views to initiate a fully automatic asset upload and creation process. * **User subscriptions** can be viewed by Eco and Org admins in the user profile * User can select **'Text'** as an export format * **At Source S3** assets are now enabled on AWS platform --- language: "en" --- # 5.5 Early access of Query, Global search and other improvements ## Summary **GENERAL AVAILABILITY** * **(AWS ONLY) Spaces** * Edit data, space name and description upgraded to a new design * Space creator can see correct permissions warnings during space creation * **(ALL) Quick asset creation** * User can cancel an in progress asset upload * User has access to Upload file button on My collection and Asset management views to initiate automated asset creation * **Other improvements** * Export to text format (added to GCP) * Feature flag that allows controlling whether users land on exchange or My Collection after login (set at ecosystem level) **EARLY ACCESS** *These features will be enabled at different times depending on the fit for the customer. Ask your account manager if you want to learn more.* * (AWS ONLY) Introducing a new way to use data - **Query**. Query allows users to use either SQL or natural language to query data on the platform. * (ALL) Global Search that allows easily finding what users need across products, assets and their metadata from one search area. * (ALL) Provision access to Harbr API using Service Accounts, that allow setting level of access and make calls to manage products and assets through an API. *** ** * ** *** ## Fixed bugs * REF 20325: 'Released by' value in asset created from quick asset upload is not populated * REF 20324: Deleting asset from manage assets while upload is in progress doesn't clear it from the upload dropdown. * REF 20262: There is no ability to cancel an in-progress asset * REF 20288: When space owner creates a space and collaborators are missing permissions, the modal lists only the products and omits the assets * REF 19926: Multiple account logins and VDI-less space access in the same browser results in users seeing the wrong data when accessing the tools of subsequent logins and subsequently activated spaces * REF 20345: Export of text format data is failing * REF 20042: File Assets created via Desktop Upload added to a Space in AWS are not mounted correctly * REF 20021: (GCP) Export fails with "Could not update export" after the connection test is successful * REF 19593: Migrating Products with similar table names give incorrect tables sizes. * REF 20049: Where data file or folder names contain spaces, Jupyter experiences file load errors. Non-VDI experience only * REF 20501: Spaces - Can't see data if adding two (or more) at-source Snowflake assets that use the same connector with tables in different databases * REF 20366: Parquet exports are failing as the data contains dates before 1582-10-15 * REF 20345: Export to Text format failing on GCP * REF 20338: On-Platform BigQuery asset cannot be created from a view ## Release Start Date: 5th April 2024 It might take a few days before the release is available in your platform(s). --- language: "en" --- # 5.6 Export notifications and updated Query experience ## Summary **GENERAL AVAILABILITY** * **(ALL)** Export * Export notifications for export success and failure, with additional notification settings available on each export * A warning during export to desktop to not download large assets * **(ALL)**Spaces improvements * Trino Upgrade * Users able to use UDFs via PySpark **EARLY ACCESS** * **(AWS Only) Query** * Save and reuse queries, view 'My Queries' list * Redesigned Query experience *** ** * ** *** ## Fixed bugs 1. REF 20772: BigQuery at-source assets not loading after Trino 443 upgrade 2. REF 20743: Cannot add links to other products in the exchange within data product description 3. REF 20737: Space containing only on platform assets causes cluster load and data proxy to fail 4. REF 20660: SQL Labs is displaying hidden schemas with Trino 443 when user is selecting databases in Spaces 5. REF 20586: Node workers and master node don't have the same version of python, this is causing failures. Update both to version 3.10 6. REF 20171: Cannot reference files in the Jupyter home area on GCP spaces due to path structure of data location 7. REF 20109: Code asset summary and saved code is truncated when saved as text type in the database 8. REF 19672: Related products are timing out 9. REF 19593: Migrated products with similar table names give incorrect asset table sizes ## Release Start Date: 7th May 2024 It might take a few days before the release is available in your platform(s). --- language: "en" --- # 5.7 Improved Export filtering and Query Products * **(All) Export improvements.** We have added a number of additional options that allow users to customize and filter the data in their exports: * Select a subset of assets from a product to export * Column filtering on exported assets * Row filtering on exported assets **GENERAL AVAILABILITY** * (All platforms) **Export improvements.** We have added a number of additional options that allow users to customize and filter the data in their exports: * Select a subset of assets from a product to export * Column filtering on exported assets * Row filtering on exported assets **EARLY ACCESS** * (AWS Only) **Subscription Plan Templates** * We have slightly updated the creation form by adding Query as a usage permissions controlled by subscriptions in addition to Spaces and Export * (AWS Only) **Query Products** * Subscription plan templates now have Query usage permissions enabled. Once user subscribes to a product they will be able to Query all assets within the product using the new Query fe *** ** * ** *** ## Fixed bugs 1. REF 20915: Export to BigQuery fails due to invalid characters in the Asset/Product name. 2. REF 21018 Sample data is not loaded for BigQuery assets if there is DATE or TIMESTAMP columns in source 3. REF 20843: BigQuery Connector: Access Checks require high level, broad permissions within a GCP Project. Minimum permissions set required needs to be more granular. 4. REF 20355: When there is a tasks failure caused by 'JSONDecodeError: Unterminated string', cannot download Task log 5. REF 20157: V5 Export was using the**data bucket** for the export cache location, it has been fixed to use the **system** bucket instead 6. REF 20957: Unable to Execute Tasks Running Python Code ## Release Start Date: 3rd June 2024 It might take a few days before the release is available in your platform(s). --- language: "en" --- # 5.9 Homepages, UI themes, new types of assets Coming to **Early Access (AWS and GCP)** * **Custom UI themes** - Customers can enjoy a fully branded platform experience, inclusive of typeface, brand colours and logos across the entire platform. * **New Homepage** - New option for fully customizable landing page once the user logs in. Home pages can now be managed through ecosystem admin area and assigned to organizations. * **Updated My Collection** - The products and assets in my collection have acquired a UI update. Mostly the styling and spacing to provide a new look. * **Adding images to categories and tags** - Ecosystem admins can upload images that represent their categories and tags, these are displayed alongside the selectable categories and tags for users filtering the exchange during product discovery. * **PDF assets** - As a sub type of file assets, PDF assets are now formally recognised during creation. They are specially labeled in management and consumption views. In consumption views, special action buttons allow users to open the PDF directly in line to view and/or download. * **Visualization assets** - Visualization assets mark the 3rd official asset type that can be created on the platform. These assets enable dashboards and charts to be packaged as assets. * **New exchange design**- Major updates have been made to the list and grid views of the exchange, introducing more relevant information, updating styling and including new action buttons to subscribe in line. **Other changes, AWS only** * Various changes to enable Platform API execution from Spaces * Configure a space spec to always allow access without a VDI **Other changes, AWS and GCP** * Allow enabling/disabling AI features in Query independently --- language: "en" --- # Activity Log The Harbr application populates an event stream of all key actions within the platform. This event stream populates the Activity Log in the UI. Please note that not all event types are shown on the Activity Log. The **Activity Log** is a list of key events within the organisation that can be used to troubleshoot issues and understand usage. Each **Event** has a date/time when it was raised, type, subject of the event (e.g. product or a user) and acting user. **Note**: You must have the necessary permissions associated with your user role. 1. Click on the *Organization logo* on the navigation bar. 2. Click on the **Activity Log**button at the top right of the screen. 3. Filter the list of Events by Date Range, Type and Subject ID 4. Click on the **\>**to expand the Event to show additional details that vary based on the Event Type. ![Screenshot 2025-01-28 at 12.53.20.png](https://docs.harbrdata.com/__attachments/a_293608cd2862117e15da6e5aa064b52059e8f0e7fca2a96bff3236f3af0aa04f/Screenshot%202025-01-28%20at%2012.53.20.png?cb=52ebd79b5fd9ea4c5747abf106b98765) Some events in the log may appear with either **SERVICE_USER** or **ANONYMOUS** as acting user. The "**SERVICE_USER**" is used for what we call "internal" calls, where one service (internal or external to the Harbr system) is calling another service, e.g. spaces calling into assets. The "**ANONYMOUS**" is used for scenarios where there is no user directly triggering the call. This could be either an event listener endpoint, or perhaps a cron-triggered endpoint (regular timer). We would expect to see this in the audit log and there are a number of user transactions that are handled asynchronously (via an event listener). This page provides an inventory of system-generated events within the Harbr platform. These events capture actions on assets, products, connectors, subscriptions, users, and more. For each event, the table below lists its group, name and description. ## Use of Event Inventory This event stream can be leveraged by consuming integrations and processes through: * Registered Integrations which are configured to be called for certain event types * Direct consumption of the event stream within Kubernetes (if consuming service hosted in same K8s cluster) * Consumption of the events from a Cloud Native event processing service (event forwarding integration required) * Query of the Activity Log API For more information on the mechanisms available for event consumption, see [here](https://harbrgroup.atlassian.net/wiki/pages/resumedraft.action?draftId=3915186178&draftShareId=f93dd9ff-6717-402b-afee-7ed36aab3241&atlOrigin=eyJpIjoiZTQxOGZkZDAwNzQ5NDI1ZjgyMDFkNTdkNWVhYjExYzMiLCJwIjoiYyJ9). ## Event Types | Group | Event Type | Name | Description | |------------------|------------------------------------|---------------------------------------|---------------------------------------------------------------------------------------------------------| | Asset | `asset.created` | Asset created | When a new asset is successfully created in the platform. | | Asset | `asset.updated` | Asset data updated | When an existing asset is successfully updated. | | Asset | `asset.update.failed` | Asset data update failed | When an attempt to update an asset fails. | | Asset | `asset.deleted` | Asset deleted | When an asset is permanently deleted. | | Asset | `asset.edited` | Asset edited | When edits are made to an asset's metadata or details. | | Asset | `asset.shared` | Asset permissions edited | When an asset's access permissions are shared with a user. | | Asset | `asset.released` | Asset released | When an asset is is released and available for consumption. | | Connector | `connector.created` | Connector created | When a data connector created and added to the platform. | | Connector | `connector.deleted` | Connector deleted | When a connector deleted and removed from the platform. | | Connector | `connector.updated` | Connector edited | When a connector's settings or metadata are updated. | | Connector | `connector.test.failed` | Connector test failed | When a connector test does not pass, e.g., due to authentication or network issues. | | Connector | `connector.test.succeeded` | Connector test succeeded | When a connector test completes successfully. | | Data Share | `asset.dataShare.created` | Asset data share created | When a new data share is created for an asset. | | Data Share | `asset.dataShare.deleted` | Asset data share deleted | When an existing asset data share is removed. | | Data Share | `asset.dataShare.updated` | Asset data share edited | When an existing asset data share is modified. | | Data Share | `asset.dataShare.keys-rotated` | Asset data share keys rotated | When access keys for a data share are rotated. | | Data Share | `product.productDataShare.created` | Product data share created | When a data share is created for a product. | | Data Share | `product.productDataShare.deleted` | Product data share deleted | When a product- data share is removed. | | Data Share | `product.productDataShare.updated` | Product data share edited | When an existing product data share is modified. | | Export | `export.completed` | Export completed | When an export job finishes successfully. | | Export | `export.failed` | Export failed | When a data export job fails to complete. | | Export | `export.task.attempt.completed` | Export task completed | Captures the successful completion of an export task. | | Export | `export.task.attempt.failed` | Export task failed | When a task in an export job fails. | | Organization | `organization.created` | Organization created | Triggered when a new organization is registered on the platform. | | Organization | `organization.edited` | Organization edited | When an organization or its metadata is updated. | | Payment | `payment.started` | Payment started | When a payment process is initiated. | | Payment | `payment.completed` | Payment completed | When a payment process is completed. | | Payment | `payment.expires` | Payment failed | When a payment process expires. | | Process Instance | `process-instance.dequeued` | Process instance dequeued | Generated when a process instance is removed from the execution queue. | | Process Instance | `process-instance.queued` | Process instance queued | Fired when a process instance is added to the execution queue. | | Product | `product.assets.edited` | Product assets edited | Occurs when assets linked to a product are modified, added, or removed. | | Product | `product.created` | Product created | Triggered when a new product is created. | | Product | `product.deleted` | Product deleted | Fired when a product is permanently removed from the platform. | | Product | `product.edited` | Product edited | Captures edits to product metadata or configuration. | | Product | `product.share.edited` | Product management permissions edited | Occurs when sharing or management permissions for a product are updated. | | Product | `product.released` | Product released | Fired when a product is released and available for consumption. | | Product | `product.release.requested` | Product Release requested | When a product release is requested by a user. | | Product | `product.release.approved` | Product Release Request Approved | When a product release request is approved by another user and the product enters the Exchange. | | Product | `product.release.rejected` | Product Release Request Rejected | When a product release request is rejected by another user and the product does not enter the Exchange. | | Secret | `secret.access.unAuthorized` | Secret access unauthorized | Triggered when a user attempts to access a secret without proper permissions. | | Secret | `secret.accessed` | Secret accessed | Occurs when a secret is successfully accessed. | | Secret | `secret.created` | Secret created | Fired when a new secret is generated. | | Secret | `secret.deleted` | Secret deleted | Triggered when a secret is removed from the system. | | Secret | `secret.updated` | Secret edited | Occurs when the metadata or content of a secret is modified. | | Secret | `secret.share.edit.unAuthorized` | Secret share edit unauthorized | Fired when an unauthorized attempt is made to modify a secret's sharing settings. | | Secret | `secret.share.get.unAuthorized` | Secret share view unauthorized | Triggered when an unauthorized attempt is made to view a secret's sharing settings. | | Secret | `secret.shared` | Secret shared | Occurs when a secret is successfully shared with other user. | | Service Account | `service-account.created` | Service account created | Fired when a new service account is created on the platform. | | Service Account | `service-account.deleted` | Service account deleted | Triggered when a service account is removed. | | Service Account | `service-account.edited` | Service account edited | Occurs when a service account's metadata, roles, or policies are updated. | | Service Account | `service-account.keys-rotated` | Service account keys rotated | When a service account's authentication keys are rotated. | | Subscription | `subscription.assigned` | Subscription assigned | Occurs when a subscription is assigned to a user or organization. | | Subscription | `subscription.expire.failed` | Subscription removal failed | When the removal of a subscription fails. | | Subscription | `subscription.requested` | Subscription requested | When a subscription is requested by a user. | | Subscription | `subscription.request.approved` | Subscription Request Approved | When a subscription request is approved by another user and the consumer gains access | | Subscription | `subscription.request rejected` | Subscription Request Rejected | When a subscription request is rejected by another user and they do not gain access. | | Subscription | `subscription.expired` | Subscription removed | When a subscription is removed after expiry. | | User | `user.invited` | User invited | Occurs when a new user invitation is sent. | | User | `user.registered` | User registered | Fired when a user completes registration on the platform. | | User | `user.roles.assigned` | User role edited | Triggered when a user's role assignments are updated. | --- language: "en" --- # Administer * [Your Organization](https://docs.harbrdata.com/v6/your-organization.md) * [Platform Personas](https://docs.harbrdata.com/v6/platform-personas.md) * [User Roles](https://docs.harbrdata.com/v6/user-roles.md) * [Subscription Plans](https://docs.harbrdata.com/v6/subscription-plans.md) * [Terms and Conditions](https://docs.harbrdata.com/v6/terms-and-conditions.md) * [UI Customizations](https://docs.harbrdata.com/v6/ui-customizations.md) * [Integrations](https://docs.harbrdata.com/v6/integrations.md) * [Activity Log](https://docs.harbrdata.com/v6/activity-log.md) * [Metadata](https://docs.harbrdata.com/v6/metadata.md) --- language: "en" --- # AI Query ## What is AI Query? AI Query is a new asset type that allows a producer to provision access to an existing Databricks Genie Agent through the Harbr UI. Users are able to register a Genie Agent they've already built, complete with the tables, instructions, and any pre-existing worked examples. **Note:** This feature is enabled per environment, so please contact your Harbr Account Manager if you are interested in using. ## Creating an AI Query asset A producer creates an AI Query asset by selecting the AI Query asset type, choosing a Databricks connector and selecting from the Genie Agents that connector can reach, then publishing it like any other asset. * No credentials are shared with Harbr beyond the connector's own access. * No data is moved onto the platform. * There is no crawl or schema step --- the asset simply references the existing Genie Agent. **Step 1 - Create a Databricks connector** (if one doesn't already exist) pointing to the Databricks instance where your Genie Agent is configured. For user guidance on how to create a new Databricks connector, please see [Create a Databricks Connector](https://docs.harbrdata.com/v6/databricks.md) **Step 2 - Start the asset creation process** and select **AI Query** as the asset type. **Step 3 - Select "Databricks"** as the connector type. ![image-20260803-220530.png](https://docs.harbrdata.com/__attachments/a_beecd4763742706dbfe0b16242ae308be50c9b7d88e8e6550917c49fb2bfdcc9/image-20260803-220530.png?cb=ed56bf137ba7e3c14ea597d8d7580d46) **Step 4 - Choose the connector** that points to your Genie Agent from the dropdown. ![image-20260803-220659.png](https://docs.harbrdata.com/__attachments/a_861bded222325ef4b6e0e1f7455bff16df02ca47376cdd950f9584acf991d07a/image-20260803-220659.png?cb=fbc480d9ea0fce5f7a61b6a04f1a551b) **Step 5 - Select the specific Genie Agent** you want to expose, from the dropdown. ![image-20260803-220815.png](https://docs.harbrdata.com/__attachments/a_397446c1c487e309b76848f1717cd01e1a2243d254190cdde6e26d50b50e19fd/image-20260803-220815.png?cb=1c6a4dd70fd9ec2287d3bdb472339be3) **Step 6 - Select "Start Set Up"** and finish the asset create journey. ![image-20260803-220931.png](https://docs.harbrdata.com/__attachments/a_c5b9e75a434fc0d69dc07989f357f0709b55247b774f0b1e2ff7489cd75257ff/image-20260803-220931.png?cb=233e5acaf0f7e2d5530b19942d7ebffb) **Step 7 - Release the AI Query asset.** ![image-20260803-221012.png](https://docs.harbrdata.com/__attachments/a_ba4013eea2594ba9ef8991e5ad90881c0b79705aa13b71d386125c8ac944d14e/image-20260803-221012.png?cb=c95ae5ffcb820a46c4ab5d9191531bd5) ## Consuming an AI Query asset Consumers with access to the asset, or are subscribed to a product that contains the AI Query asset, ask questions in natural language via a chat interface, all without leaving the Harbr Platform. You can open the AI Query asset place from: * The product page * My Collection * Home pages (if configured by your platform operator) Answers are returned as text and tables. The conversation is retained for the duration that the tab is open and the user is signed in. The session can be closed and be cleared to start again --- there is no server-side conversation history. Every question is proxied through Harbr using the connector's identity, so no consumer accesses Databricks directly, and each question is recorded for usage and cost visibility. **Note**: If the underlying Genie Agent becomes unreachable for whatever reason, consumers will be shown an error message when they try to open or query the asset. AI Query is an AI feature, AI can make mistakes. Please check the responses. --- language: "en" --- # Approval Requests Approval Requests allows the producer to Approve or Reject a request via an integration that is set up by the platform operator. The following step by step walkthrough takes you through consuming a product via Approval Requests. ## **1. Locate the Data Product** Navigate to the relevant **Data Product page** on the platform to view available options. ### **2. Select a Subscription Plan** Review the available plans and choose **plan** that fits your specific use case. #### **3. Initiate Subscription** On the selected plan card, click the **'Request Subscription'** button. ![image-20260317-092150.png](https://docs.harbrdata.com/__attachments/a_39ebdc7674e4c58f2d8b0f91d28af2600046251514cdde02ce791e3dacdaa0f5/image-20260317-092150.png?cb=822e46364bea0308e3aa7bc4431e5f97) #### **4. Review and Agree to Terms** Once the summary appears: * Carefully check the plan details. * Agree to the **Terms and Conditions**. * Add an optional **comment** to your request. * Click **'Subscribe'** ![image-20260317-092227.png](https://docs.harbrdata.com/__attachments/a_9ad54d6db9df3287e0e6d2d41065911deab633060f9ee32b9ad8df6edbdc3615/image-20260317-092227.png?cb=bb23edbe9563424b47abaa4fb33beb67) #### 5. Navigate to your profile, click on Request Status and select Requested. You'll see a list of all your pending requests. ![image-20260317-092448.png](https://docs.harbrdata.com/__attachments/a_437a4ea7ae40d85fbc6f573b36ebfd58a301e2cdbba279f734bf3181ec677c3d/image-20260317-092448.png?cb=6b9d1d95dbf089ed6b92662982b4ce72) #### 6. If you subscription request has been approved you'll see APproved and it will be visible in the subscription tab. ![image-20260317-101328.png](https://docs.harbrdata.com/__attachments/a_803c55fb5ffccb675aca1d4c1fb8d5a55bef4e29e3546842a392191da24474e5/image-20260317-101328.png?cb=ca48ec5f37eb9580d7cd068e70f9d117) #### 7. If you subscription request has been rejected you'll see rejected under status. ![image-20260317-101701.png](https://docs.harbrdata.com/__attachments/a_27fce8ad8746702e3bd56296cdd015d846b18494a64940b5b73460557ca5b5b6/image-20260317-101701.png?cb=a2742049f1dbc36b9ee470c9a23b2811) --- language: "en" --- # Automation Using the **Automation** feature in the main navigation bar, you can transform your existing code into an updating asset that refreshes automatically---either when contributing data products or assets are updated, or based on a defined schedule or trigger. To do this, select a **code asset** from a Space and define a **task** that runs the code on a recurring or event-driven basis. Once the task executes successfully, it can be used as the source for creating an updating asset. You can also configure the task to **automatically export** the resulting data to your local environment, enabling operational use of the output. **Note**: You must have an Automation Creator or Administrator role to perform these action. The Automation icon on the navigation bar is not visible without one of these roles. You must have saved and successfully executed a query in a Space. ## Create a Code Asset A Code Asset is code that has been written in a Space and has been transformed into a Code Asset, so it can be used as part of a Task. The code used in your Code Asset is a **copy** of the code you saved in your Space that created a table or tables in `publish_db`**.**Any changes you make to your saved code within the Space are not reflected in your code asset unless you refresh the Code Asset. For others to be able to edit the code for your Code Asset you they must be added to your Space as a collaborator. 1. Click on the Automation icon on the navigation bar. 2. Click on the Code Assets option. 3. Click **Create Code Asset** . The New Code Asset page appears. 4. Enter a **Title**. 5. Enter a **Description**. 6. Select the **Code**you want to turn into a code asset: * Select the relevant **Space**from the drop down list. * Select the codeyou saved. It must have run successfully in the Space to ensure it is available to be used as a Code Asset. 7. **Data Products and assets**display a list of all products and assets in your Space. * If your Code Asset does not reference all products or assets in your Space then consider removing those that are not required to avoid downstream lineage restrictions. 8. **Code Asset Sharing:** * Add collaborators who can edit the Code Asset title and description. 9. Click **Save**. ![:green_star:](https://docs.harbrdata.com/__attachments/a_4ddd77418f7de03ddc6f32c42fa1c142c1ae3fdfecd458c33a68b8719ff45c2b/atlassian-green_star?cb=3055763991e3275cb74b2dd3e1922b2d) Now you are able to schedule the execution of your code asset with a Task. ## Create a Task A Task is an automated process for executing a code asset against target products and/or assets to create an output that can be used to create a new asset. The process can be run on a scheduled basis or be triggered when one or more of the target products or assets update. Tasks can take several minutes to complete because behind the scenes a new, temporary compute instance is activated to execute the code in your Code Asset. As a user this is only really noticeable when the Task is run for the first time as all subsequent updates are executed automatically. You can return to check the status of your Task at any time. To schedule the execution of your Code Asset: 1. Click on the Automation icon on the navigation bar. 2. Click on the Tasks option. 3. Click **Create Task** . The Create Task dialog page appears. * In the Details section: * (Required) Enter a **title** for the task. * (Optional) Enter a **description** for the task. * In the Code Asset section: * Click **Select Code Asset**. The Code Asset selection page appears. * Select the required Code Asset. * Click **Save**. The dialog box closes and you return to the Create Task page. * In the Products section: select any additional products, if available. * In the Assets section: select any additional Assets, if available. * In the Task Sharing section: select any users you wish to add as collaborators on this Task. * In the Trigger section: select how the Task will trigger. Available options are: * **Manual**: your Task must be triggered by user input. * **Product or sset Update**: an update to a linked Product or asset will trigger this task. At least one contributing Product or asset must be updating. * **Schedule**: set a schedule to automatically trigger the Task repeatedly. * Select if a notification email should be sent for a successful run as well as for failed runs. If a Task fails then check the status of it to find out more information. * Click **Save** or **Save and Run**. * A Task must run successfully for it to be available as a **Source** to create an automated data product Now you are ready to create an asset from a Task, as shown [here](https://docs.harbrdata.com/v6/create-a-data-asset.md). ## Manage a Task **Note**: Again, you must have an Automation creator or Administrator role to perform these actions. The Automation icon on the navigation bar is not visible without one of these roles. ### Check the Status of a Task You can return to check on the status of a Task at any time. 1. Click on the Automation icon on the navigation bar. 2. Click on the Tasks option. 3. View Task details or search for a specific Task. Available details are: * Name * Description * Trigger * Creator * In Use (Yes or No) * Last Run 4. Click on a Task. The Detailstab appears. * Click on the **three dots** at the top right to edit the **Name** and **Description**of the Task. * Click on the **three dots** at the top right to **delete**the Task. * The Task Run Historytab: * Shows the date the task was executed and what triggered it. * Click **View Run Details**to see more information and download it to share with Support if there are any issues. * Check theUsagetab information matches your expectations. ### Edit or Delete a Task Certain descriptive attributes of a Task can be edited. A Task can also be deleted which means it no longer executes and the asset data no longer updates. 1. Click on the Automation icon on the navigation bar. 2. Click on the Tasks option. 3. For each Task you can: * Click **three dots** \>**Delete Task.** * Your Task is deleted and no longer executes. * Click **three dots** \>**Edit Task**: * Edit **Title** and**Description** 4. Click **Save**to save your changes. --- language: "en" --- # Best Practices ## Automation On the platform, you can easily update data your assets when your source data is refreshed through the use of a transfer notification file, or TNF. You can also assign data products or assets to be exported whenever they are updated, on an event (refresh) or schedule basis (for example once a week). Tasks allow you to turn what once were static assets into repeating engineered assets. This feature allows you to automate their Spaces code and dynamically create engineered data assets automatically. Automated Data Assets can update on a schedule, be triggered by an update of another Data Product/Asset, or be run manually. Using code assets and Tasks saves you countless hours of updating their engineered data assets as well as opening up new opportunities for discovery. We recommend the following when you set up automation for running tasks: * Structure your workflow in smaller, manageable Spaces, with a limited number of products and/or assets, instead of one large Space with all the automation inside which is very challenging to maintain. You can create multiple Spaces to suit your needs. * Separate your automation by tasks that are independent from each other. * If you are working on many use cases, combining several data sets, create one automation per use case instead of one large automation that tries to address everything at once. This is easier to maintain, to update, and to get support should you need help. * There are different ways to trigger an automated task on the platform : * Update manually; * Tie updates to data product or asset updates; * Schedule updates by date, time and interval. * Automating one task can also be used to trigger another. For example, you can trigger a task to update and then use this to trigger the automation of a use case. * You can always change when an automated asset updates in the **Trigger** section. ## Collaboration Collaborators work together in a Space to create insights and engineered assets. The specification of the Space i.e. the compute, data products, assets, tools and people, is defined by a user who defined the specification and this person is referred to as the Space owner. If you are not the Space owner yourself then make sure you know how to contact the owner as that person is responsible for inviting collaborators to join a Space and add or remove data products. Each collaborator has their own compute environment so there is never any contention with other collaborators so you can do your work at any time. * You do not need to wait for all collaborators to accept an invitation in order to start work in the Space. * Make sure you have the necessary subscriptions to the data products and / or access to standalone assets within the collaborative Space otherwise the Space appears as ***locked***. * Share your tables, notebooks, queries and other files with your fellow collaborators within a Space. As a collaborator you can leave that Space at any time. As the owner of a collaborative Space you must pass ownership to another user in your Organization before you can leave the Space. ## Security The following recommendations help you to keep your data secure and to manage your tasks efficiently. * **Limit the number of products and/or assets**in a Space to those that are related to your use case. You can create another, separate Space for your products and/or that are not related. This keeps related products/assets and use cases together in a manageable size, where you can more easily control access to them. * We recommend**keeping a small Space specialized by use case** with segregated access for users, so you don't have to grant access to all products and/or assets for all users. This way you can open data for collaboration without overexposing it. * Only the **owner of a Space can add products, assets and collaborators**. This person leads the work on that Space so make sure you know who the Space owner is by viewing the details of the Space specification. * When creating an engineered asset from a Space, the security of the asset is limited by the subscription plans associated with the products in that Space. This is another reason for why limiting the products in a Space is good practice. * You may need to a**dd more than one subscription plan template**to your product if the usage of that data product needs to be different for different consumers. * On your connector, be sure to **update your security credentials.** ## Integration The platform provides various integration capabilities so that you can connect your work on the platform with your own environment. To ensure that this integration remains simple and can grow with your needs, we recommend the following best practices * **Separate your inputs from your outputs** Each connector that you define in the platform is connected to one on your cloud storage. If the connector is used for a recurrent publication, the platform will scan the storage on a regular basis to look for new data revisions requests (TNF) and automatically trigger the import. By separating your inputs, you can apply the storage policy that works best for your environment and decide if you want to remove the temporary data after import. If the connector is used for a recurrent export, the platform will write a new dataset after each successful export.By separating your outputs, you can quickly identify that this data can be re-created by the platform and apply the appropriate storage policy based on your usage of data. * **Document your platform integrations.** When you create a connector, the platform provides you with instructions to update your cloud data storage security to allow data movement. Even if you can access all the configuration information in the administration of the platform, we recommend you store the security information in your documentation system and the part of your system that is connected to it. * **If you are multi-cloud, prioritize exchange from the native platform storage for high traffic.** Our platform is able to connect with the major cloud providers (e.g. AWS, Azure and GCP). As all the cloud providers charge a fee for the outgoing traffic, if you use multi-clouds (exchanging data from several cloud providers) and plan to have high volume exchange with the platform, we recommend using the same storage as our native platform to limit those cloud provider fees and ensuring a better transfer speed. Check with Support to find out which cloud provider the platform is hosted on. * **If you are not familiar with cloud provider storage, use our documentation articles.** We fully understand that the first cloud exchange setup can be intimidating. Our documentation provides an initial guide on how to use each of the cloud providers and links to resources to get more information. * **Look at the publishing or export information to understand your cloud traffic.** You can view the start/end and size of each data operation performed on the platform on the data product / asset or on the export. With this information, you can correlate the cloud traffic with your system logs and associate a data size to the time observed to transmit the data. Finally, do not hesitate to contact our support team. ## Image Sizes | **Element** | **Format** | **Recommended Size** | |------------------------------------------------------|------------|------------------------------------| | Tile Banner | JPG or PNG | 520px x 520px | | Organization Logo (Platform - Light Background) | SVG | 135px x 85px | | Organization Logo (Secure Desktop - Dark Background) | SVG | 135px x 85px | | Data Product Tile Images | JPG or PNG | 720px x 720px | | Data Product Icons | JPG or PNG | 480px x 480px | | Data Product Page Header Image | JPG or PNG | 1500px across (height as required) | --- language: "en" --- # Big Query Harbr BigQuery connectors leverage GCP Service Accounts to enable and authenticate the connections from the platform to BigQuery. Rather than a storage connector, that connects to a GCP or S3 bucket for example, a Big Query connector is referred to as a **database connector.** Your GCP Service Accounts must have the following minimum permission to enable the connector and asset configuration: * Read: * `bigquery.dataViewer` * `bigquery.jobUser` * Write: * `bigquery.dataEditor` *If you are creating assets against Big Query External Tables, the Service Account used in the connector must also be permissioned on the GCS Storage bucket location that the external table is built against.* To apply permissions to your service account navigate to your project in the Google Cloud Console and select **IAM -\> Permissions -\> View by Principles -\> \ -\> Edit -\> Assign Roles**. The first step in creating this connector is to set up a bucket to store the data you intend to access. ## Create the Connector 1. Click **Manage** on the Navigation bar. 2. Select **Connectors** to view the Manage Connectors screen 3. Click the **Create Connector**button at the top right 4. Enter a **Name** for your Connector and a **Description** (optional) 5. Choose Choose **Type** \>**BigQuery** 6. Select and upload ***GCP Service Account Key File*** that will be used to manage access to your BigQuery project(s). See [here](https://harbrgroup.atlassian.net/wiki/pages/resumedraft.action?draftId=2772993257) for more details on how to set one up. 7. **Project ID** will be populated automatically based on the file. 8. Add any **Integration Metadata** needed for programmatic integration. 9. Click **Create** . Connection test will run and if successful, will show***Connection Test Status***as Successful. 5. Click **Close**. ## Execution Patterns When BigQuery tables are exposed to the Harbr platform, different services within the asset / product usage life-cycle interact with them. The technical nature of these interactions are summarised below. |-----------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------------|--------------------------| | **Component** | **At-Source Asset** | **On-Platform Asset** | **Export** | | Data Copy Stored | No | Yes | Yes | | Connector Type | Trino BigQuery Connector | Spark BigQuery Connector | Spark BigQuery Connector | | BigQuery Query API Utilised | No | No | No | | BigQuery Storage Read API utilised | Yes | Yes | No | | BigQuery Storage Write API utilised | No | No | No | | Storage Optimisations Minimise the volume of data read via the BigQuery Storage API through SQL operations \*\* | Yes LIMIT - Returns only the numbers of rows specified WHERE - Returns only rows that match the where clause condition COLUMN SELECT - Reads data content only for the columns specified in a query | No | N/A | Note: For Azure versions of the Harbr platform where a BQ connector is intended to be used for 'at source' asset creation, the GCS key must have the 'universe_domain' value created. ## Service Account Permissions - Project Level Granularity BigQuery connectors on the Harbr platform leverage **GCP Service Accounts**to enable and authenticate the connections from the platform to BigQuery. In addition to performing the Harbr platform actions to configure the connector and assets, the Service Accounts used must have the following minimum permission to enable the platform journeys. ### **Create Asset / Use Asset in Spaces Tasks** The service account must have the permissions below at the GCP Project level: * BigQuery Data Viewer * BigQuery User **Note**: If you are creating assets against Big Query External Tables, the Service Account used in the connector must also be permissioned on the GCS Storage bucket location that the external table is built against. ### **Export a Product / Asset to BigQuery** The service account must have the permissions below at the GCP Project level: * BigQuery User To apply permissions to your service account navigate to your project in the Google Cloud Console and select **IAM -\> Permissions -\> View by Principles -\> \ -\> Edit -\> Assign Roles** and select the necessary roles. ## Service Account Permissions - Data Set level granularity To permission your service account at the finer grained dataset level, additional config is required on top of the default Google IAM roles. The best practise for this config is described below. ### **Create Asset / Use Asset in Spaces Tasks** Provide your service account **BigQuery Data Viewer** and **BigQuery Data User** permissions at the dataset level via either : * Sharing the dataset with the service account via the Google Console (**Console -\> BigQuery -\> Dataset -\> Sharing -\> Permissions -\> Add Principle**) * Applying an IAM condition on the project level BigQuery permissions of the form : * Condition Type = name * Operator = is * Value = projects/\/datasets/\.\ Additionally create a Custom IAM role with the permissions outlined below and apply this to your service account alongside the permissions above. Example Name : **CustomRoleBQAssetCreate** * bigquery.jobs.\* * bigquery.readsessions.\* ### **Export a Product / Asset to BigQuery** In addition to the permissions required for asset creation above, to use a dataset restricted service account to export to BigQuery, add the permissions below to the service account via a new custom role (e.g. CustomRoleBQExport) or by extending read role above : * bigquery.tables.create * bigquery.tables.update * bigquery.tables.updateData * bigquery.tables.delete (used to remove staging tables created during the loading flow) [Next Page](https://docs.harbrdata.com/llms-full.txt/1)