{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "466126a6-174c-434e-a034-f090f7b4b4a6",
   "metadata": {},
   "source": [
    "### Harbr Platform SDK Example Notebook \n",
    "\n",
    "This notebook contains examples of utilising the Harbr Platform SDK to streamline the creation of outputs (Tabular Assets, File Assets and / or Tasks) from your Space via a guided interaction with the platform APIs\n",
    "\n",
    "For futher details on all methods within the SDK please see : https://ops.harbrdata.com/odv5/new-asset-management-via-harbr-sdk (Operators Portal Login Required)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b1e04c0-3fc1-46ab-9cd9-2ecd12e8d476",
   "metadata": {},
   "source": [
    "### Section 1 - Package and Session\n",
    "\n",
    "The Harbr Platform SDK (latest version) is prebuilt into your Spaces environment and can be import directly as below "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18a08130-84b4-4d6f-8e24-c40faf97a441",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import harbr_platform"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a705f637-b224-49c3-a68d-73e4cefbfdbf",
   "metadata": {},
   "source": [
    "The SDK has 3 main modules for users : \n",
    "- session : creates the API session that's used in the SDK methods\n",
    "- workflow : outcome based actions with simple parameterisation, wraps platform_actions methods\n",
    "- platform_actions : granular actions taken againts the platform API, mainly wrapped by workflow methods but available for cases where direct usage is approprtiate "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d49d4a9-9f16-4f3a-bc08-2905ba2d292b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from harbr_platform import session, workflow, platform_actions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4a0642c-2bb5-4850-9a3d-7b73b8a5b73b",
   "metadata": {},
   "source": [
    "### 1.1 - Create Session\n",
    "\n",
    "When using the SDK from a Space, your session cluster has been pre-configured with the necessary metadata and authentication to allow your user to interact with the platform API. The simple method below creates the API session"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8f92c70-27ee-4ea0-81fc-405fc039dc95",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session_details = session.create_session(session_source = \"space\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "361eaeca-98ec-4cb0-b7e8-9b52ece6820b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session_details"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "67a64aca-7126-4ced-a4ca-44caa403d0cb",
   "metadata": {},
   "source": [
    "For details on initiating a platofrm session from an external environment, including the use of Service Accounts in the SDK, please see : https://ops.harbrdata.com/odv5/new-asset-management-via-harbr-sdk#NEW:AssetManagementviaHarbrSDK-create_session()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55258102-01f6-40e0-8574-7268d8aa0b26",
   "metadata": {},
   "source": [
    "### 1.2 - Create a Tabular Asset from a table in publish_db\n",
    "\n",
    "List the tables in your publish_db using the pyspaces library and create an output tabular asset. For full details on the options within the create_asset method see : https://ops.harbrdata.com/odv5/new-asset-management-via-harbr-sdk#NEW:AssetManagementviaHarbrSDK-create_asset() \n",
    "\n",
    "Here we use the pyspaces library to list the tables within the publish_db of the space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "48b55e54-e28b-42c2-986e-2f80cd163962",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from pyspaces import spaces_trino\n",
    "import json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d758079-46a8-4257-9956-b948a804b309",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0817fab-aa9f-4b5f-a61d-19b319e4cf91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "conn = spaces_trino.trinoSession()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0d844f3-7f50-4a8b-944e-9e25f7609529",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "spaces_trino.listSchemaTables(conn,\"publish_db\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "991ac5e4-0d5b-4a5b-b14f-43acc17d12cc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "asset_table = 'location_intel_slim_avro'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ff85f1c-762e-4347-838c-4a1150520326",
   "metadata": {},
   "source": [
    "Get the Space ID from the session config"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cec5b9fe-26f1-4d92-9e04-df684810ad1d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "session_metadata_config_file = '/tmp/space_details.json'\n",
    "\n",
    "with open(session_metadata_config_file, 'r') as session_metadata_file:\n",
    "        session_metadata_config = json.load(session_metadata_file)\n",
    "\n",
    "space_id = session_metadata_config.get(\"space_id\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61469944-fa3c-4211-b8c0-674b094aacd5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "new_asset_details = workflow.create_asset(session_details,\n",
    "                                            name=\"Example Tabular Asset built from a Space\",\n",
    "                                            description=\"Example Tabular Asset built from a Space using the Harbr Platform SDK\",\n",
    "                                            asset_type=\"table\",\n",
    "                                            source_type=\"space\",\n",
    "                                            source_id=space_id,\n",
    "                                            source_path=\"publish/location_intel_slim_avro\",\n",
    "                                            source_path_type=\"folder\",\n",
    "                                            ingest_type=\"on-platform\",\n",
    "                                            release_on_completion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cd8ae15-3b4b-442c-a45d-6a441b432ab9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "new_asset_details"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e55489af-eb1e-4cdc-9c89-0e6cfad56706",
   "metadata": {},
   "source": [
    "### 1.3 - Create a Task from a Notebook stored in the Jupyter home area\n",
    "\n",
    "This action is achieved through the use of 2 workflow methods : **create_code_asset_from_space** and **create_and_validate_task_from_code_asset**\n",
    "\n",
    "**create_code_asset_from_space**\n",
    "For full details on the options within this method see : https://ops.harbrdata.com/odv5/new-asset-management-via-harbr-sdk#NEW:AssetManagementviaHarbrSDK-create_code_asset_from_space()\n",
    "\n",
    "**create_and_validate_task_from_code_asset**\n",
    "\n",
    "This action is a wrapper of several sub workflow methods (**create_task_from_code_asset**, **run_task** and **get_task_run_status**) that can be used individually if further control is required.  \n",
    "\n",
    "For full details on the options within this method see : https://ops.harbrdata.com/odv5/new-asset-management-via-harbr-sdk#NEW:AssetManagementviaHarbrSDK-create_and_validate_task_from_code_asset()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0fc83db8-22fa-47bc-917f-c06c359742ff",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "automation_notebook_name = \"hello_world.ipynb\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "059d9250-1686-475a-b0b0-f5395aead5e2",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "new_code_asset = workflow.create_code_asset_from_space(session_details,\n",
    "                                                            code_asset_name=\"Example code asset created from a Space : 14Nov\",\n",
    "                                                            code_asset_description='Example code asset created from a Space using the Harbr Platform SDK',\n",
    "                                                            space_id=space_id,\n",
    "                                                            code_name=automation_notebook_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a754130f-57af-493e-82e4-22d88afa2a96",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "new_code_asset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f16df28-707d-4f3a-95e1-430b0f09c4e5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "new_code_asset_id = new_code_asset.get(\"code_asset_id\")\n",
    "new_code_asset_id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94daa6f6-0521-4366-879a-77c71a908e3e",
   "metadata": {},
   "source": [
    "Create a task with a manual trigger to avoid repeating executions for this example. See the link above for further details on configuring Data Product / Asset updated event and schedule driven Task executions.\n",
    "\n",
    "Inherit the Task name from the code asset set in the actions above. This process may take upto 30 minutes depending on your environment configuration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aa57d84a-66ba-4d0b-bd8b-b234778f9d40",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "task_id, validation_run_status, validation_run_details = workflow.create_and_validate_task_from_code_asset(\n",
    "                                                                session_details,\n",
    "                                                                code_asset_id=new_code_asset_id,\n",
    "                                                                trigger_type=\"manual\",\n",
    "                                                                monitor_validation_run_to_completion=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ab69b65-23bd-4bc7-92e8-e589da2bc3da",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print('See your Task at : {}/assets/tasks/{}/details'.format(session_details.get(\"origin\").replace(\"marketplace-api.\",\"marketplace.\"),task_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef00b9cf-780b-424b-8c2f-9d28f617e566",
   "metadata": {},
   "source": [
    "**Note :** If your platform has a branded domain the link generted above may require editting. Please navigate to the created Task in the Exchange and inspect the URL pattern required for your environment if you wish to make a version of this resource available to your consumers. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PySpark",
   "language": "python",
   "name": "pyspark3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}