Best Practices

Automation

On the platform, you can easily update data your assets when your source data is refreshed through the use of a transfer notification file, or TNF. You can also assign data products or assets to be exported whenever they are updated, on an event (refresh) or schedule basis (for example once a week).

Tasks allow you to turn what once were static assets into repeating engineered assets. This feature allows you to automate their Spaces code and dynamically create engineered data assets automatically. Automated Data Assets can update on a schedule, be triggered by an update of another Data Product/Asset, or be run manually. Using code assets and Tasks saves you countless hours of updating their engineered data assets as well as opening up new opportunities for discovery.

We recommend the following when you set up automation for running tasks:

• Structure your workflow in smaller, manageable Spaces, with a limited number of products and/or assets, instead of one large Space with all the automation inside which is very challenging to maintain. You can create multiple Spaces to suit your needs.

• Separate your automation by tasks that are independent from each other.

• If you are working on many use cases, combining several data sets, create one automation per use case instead of one large automation that tries to address everything at once. This is easier to maintain, to update, and to get support should you need help.

• There are different ways to trigger an automated task on the platform :

  • Update manually;

  • Tie updates to data product or asset updates;

  • Schedule updates by date, time and interval.

• Automating one task can also be used to trigger another. For example, you can trigger a task to update and then use this to trigger the automation of a use case.

• You can always change when an automated asset updates in the Trigger section.

Collaboration

Collaborators work together in a Space to create insights and engineered assets. The specification of the Space i.e. the compute, data products, assets, tools and people, is defined by a user who defined the specification and this person is referred to as the Space owner. If you are not the Space owner yourself then make sure you know how to contact the owner as that person is responsible for inviting collaborators to join a Space and add or remove data products.

Each collaborator has their own compute environment so there is never any contention with other collaborators so you can do your work at any time.

• You do not need to wait for all collaborators to accept an invitation in order to start work in the Space.

• Make sure you have the necessary subscriptions to the data products and / or access to standalone assets within the collaborative Space otherwise the Space appears as locked.

• Share your tables, notebooks, queries and other files with your fellow collaborators within a Space.

As a collaborator you can leave that Space at any time. As the owner of a collaborative Space you must pass ownership to another user in your Organization before you can leave the Space.

Security

The following recommendations help you to keep your data secure and to manage your tasks efficiently.

• Limit the number of products and/or assets in a Space to those that are related to your use case. You can create another, separate Space for your products and/or that are not related. This keeps related products/assets and use cases together in a manageable size, where you can more easily control access to them.

• We recommend keeping a small Space specialized by use case with segregated access for users, so you don’t have to grant access to all products and/or assets for all users. This way you can open data for collaboration without overexposing it.

• Only the owner of a Space can add products, assets and collaborators. This person leads the work on that Space so make sure you know who the Space owner is by viewing the details of the Space specification.

• When creating an engineered asset from a Space, the security of the asset is limited by the subscription plans associated with the products in that Space. This is another reason for why limiting the products in a Space is good practice.

• You may need to add more than one subscription plan template to your product if the usage of that data product needs to be different for different consumers.

• On your connector, be sure to update your security credentials.

Integration

The platform provides various integration capabilities so that you can connect your work on the platform with your own environment. To ensure that this integration remains simple and can grow with your needs, we recommend the following best practices

• Separate your inputs from your outputs

Each connector that you define in the platform is connected to one on your cloud storage.

If the connector is used for a recurrent publication, the platform will scan the storage on a regular basis to look for new data revisions requests (TNF) and automatically trigger the import.
By separating your inputs, you can apply the storage policy that works best for your environment and decide if you want to remove the temporary data after import.

If the connector is used for a recurrent export, the platform will write a new dataset after each successful export.By separating your outputs, you can quickly identify that this data can be re-created by the platform and apply the appropriate storage policy based on your usage of data.

• Document your platform integrations.

When you create a connector, the platform provides you with instructions to update your cloud data storage security to allow data movement. Even if you can access all the configuration information in the administration of the platform, we recommend you store the security information in your documentation system and the part of your system that is connected to it.

• If you are multi-cloud, prioritize exchange from the native platform storage for high traffic.

Our platform is able to connect with the major cloud providers (e.g. AWS, Azure and GCP). As all the cloud providers charge a fee for the outgoing traffic, if you use multi-clouds (exchanging data from several cloud providers) and plan to have high volume exchange with the platform, we recommend using the same storage as our native platform to limit those cloud provider fees and ensuring a better transfer speed. Check with Support to find out which cloud provider the platform is hosted on.

• If you are not familiar with cloud provider storage, use our documentation articles.

We fully understand that the first cloud exchange setup can be intimidating. Our documentation provides an initial guide on how to use each of the cloud providers and links to resources to get more information.

• Look at the publishing or export information to understand your cloud traffic.

You can view the start/end and size of each data operation performed on the platform on the data product / asset or on the export. With this information, you can correlate the cloud traffic with your system logs and associate a data size to the time observed to transmit the data.

Finally, do not hesitate to contact our support team.

Image Sizes