10. AML_pipeline_train.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.0 Libary and parameter settings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 1.0.1 Library import\n",
    "import configparser\n",
    "from azureml.core import Workspace\n",
    "from azureml.core.authentication import AzureCliAuthentication\n",
    "from azureml.core.compute import AmlCompute, ComputeTarget\n",
    "from azureml.core.runconfig import RunConfiguration\n",
    "from azureml.core.conda_dependencies import CondaDependencies\n",
    "from azureml.core.environment import Environment\n",
    "from azureml.pipeline.core._restclients.aeva.models.error_response import ErrorResponseException\n",
    "from azureml.pipeline.core import Pipeline, PipelineEndpoint, PipelineParameter\n",
    "from azureml.pipeline.steps import PythonScriptStep"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 1.0.2 Retrieve configuration file\n",
    "config_ini = configparser.ConfigParser()\n",
    "config_ini.read('./common/config.ini', encoding='utf-8')\n",
    "\n",
    "## 1.0.3 Basic Azure parameters\n",
    "subscription_id = config_ini.get('Azure', 'subscription_id')\n",
    "resource_group = config_ini.get('Azure', 'resource_group')\n",
    "workspace_name = config_ini.get('Azure', 'workspace_name')\n",
    "\n",
    "## 1.0.4 Data\n",
    "train_ratio = config_ini.get('data', 'train_ratio')\n",
    "dataset_name = config_ini.get('data', 'dataset_name')\n",
    "dataset_name_for_train = config_ini.get('data', 'dataset_name_for_train')\n",
    "dataset_name_for_test = config_ini.get('data', 'dataset_name_for_test')\n",
    "\n",
    "## 1.0.5 Azure ML parameters\n",
    "cluster_name = config_ini.get('AML', 'cluster_name')\n",
    "vm_size = config_ini.get('AML', 'vm_size')\n",
    "vm_location = config_ini.get('AML', 'vm_location')\n",
    "managed_id = config_ini.get('AML', 'managed_id')\n",
    "TASK_TYPE = config_ini.get('AML', 'TASK_TYPE')\n",
    "random_seed = config_ini.get('AML', 'random_seed')\n",
    "\n",
    "## 1.0.6 Parameters for image classification\n",
    "experiment_name = config_ini.get('AML', 'experiment_name')\n",
    "base_model = config_ini.get('AML', 'base_model')\n",
    "pipelineName = config_ini.get('AML', 'pipelineName')\n",
    "model_name = config_ini.get('AML', 'model_name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.1 Authentication"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 1.1.1 Retrieve AML workspace with CLI authentication\n",
    "cli_auth = AzureCliAuthentication()\n",
    "ws = Workspace(subscription_id=subscription_id,\n",
    "               resource_group=resource_group,\n",
    "               workspace_name=workspace_name,\n",
    "               auth=cli_auth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.2 Define compute target\n",
    "\n",
    "`compute_target` will be used in training with given image files under GPU-clusters.\n",
    "\n",
    "Also, you need `NC`-series VM instead of `NV`-series for automl library for images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 1.2.1 Define computer target\n",
    "try:\n",
    "    compute_target = ws.compute_targets[cluster_name]\n",
    "    print(\"Found existing compute target.\")\n",
    "except KeyError:\n",
    "    print(\"Creating a new compute target...\")\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size=vm_size,\n",
    "        idle_seconds_before_scaledown=600,\n",
    "        min_nodes=0,\n",
    "        max_nodes=4,\n",
    "        location=vm_location,\n",
    "        identity_type=managed_id,\n",
    "    )\n",
    "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "# Can poll for a minimum number of nodes and for a specific timeout.\n",
    "# If no min_node_count is provided, it will use the scale settings for the cluster.\n",
    "compute_target.wait_for_completion(\n",
    "    show_output=True, min_node_count=None, timeout_in_minutes=20\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.3 Run-Configuration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3.1 Run configuration\n",
    "\n",
    "It connects `compute_target` with the other settings in executing AML pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aml_run_config = RunConfiguration()\n",
    "aml_run_config.target = compute_target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.3.2 Add some packages for python environment\n",
    "\n",
    "Ideally, we want to use the same environment through training and infering processes.\n",
    "\n",
    "We referred [this environment](https://ml.azure.com/registries/azureml/environments/AzureML-AutoML-DNN-Vision-GPU/version/96) as basic idea to pick up which library and version are appropriate to deploy. \n",
    "\n",
    "You can find the more about the [package dependency](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.conda_dependencies.condadependencies?view=azure-ml-py#remarks).\n",
    "\n",
    "Note:\n",
    "- `azureml-automl-core`: The basic library in using `automl`.\n",
    "- `azureml-automl-dnn-vision==1.43.0`: We need this library to infer image files with the deep-learning model generated by `automl`.\n",
    "    - Associated to this library, these could be candidates to get successful results in AML pipelines:\n",
    "        - `python` version as `3.7`\n",
    "        - `numpy`, whose version as `1.20.1`\n",
    "        - `pycocotools`, whose version as `2.0.2`\n",
    "- You can explore the appropriate combination, in case that you need more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "aml_run_config.environment.python.conda_dependencies = CondaDependencies.create(\n",
    "    python_version='3.7'\n",
    "    ,conda_packages=['pandas'\n",
    "                ,'scikit-learn'\n",
    "                ,'numpy==1.20.1'\n",
    "                ,'pycocotools==2.0.2'\n",
    "                ]\n",
    "    ,pip_packages=['azureml-sdk'\n",
    "                ,'azureml-automl-core'\n",
    "                ,'azureml-automl-dnn-vision==1.43.0'\n",
    "                ]\n",
    "    ,pin_sdk_version=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1.4 Generate AML pipeline for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## 1.4.1 Define python script and its arguments to be used\n",
    "pipeline_param01 = PipelineParameter(name=\"pipeline_arg01\", default_value=\"dummy_subscription\")\n",
    "pipeline_param02 = PipelineParameter(name=\"pipeline_arg02\", default_value=\"dummy_resource_group\")\n",
    "pipeline_param03 = PipelineParameter(name=\"pipeline_arg03\", default_value=\"dummy_work_space\")\n",
    "pipeline_param04 = PipelineParameter(name=\"pipeline_arg04\", default_value=\"cluster_abc\")\n",
    "pipeline_param05 = PipelineParameter(name=\"pipeline_arg05\", default_value=\"image_classification_experiment\")\n",
    "pipeline_param06 = PipelineParameter(name=\"pipeline_arg06\", default_value=\"base_model\")\n",
    "pipeline_param07 = PipelineParameter(name=\"pipeline_arg07\", default_value=\"dataset\")\n",
    "pipeline_param08 = PipelineParameter(name=\"pipeline_arg08\", default_value=\"1\")\n",
    "pipeline_param09 = PipelineParameter(name=\"pipeline_arg09\", default_value=\"1\")\n",
    "pipeline_param10 = PipelineParameter(name=\"pipeline_arg10\", default_value=\"0.5\")\n",
    "pipeline_param11 = PipelineParameter(name=\"pipeline_arg11\", default_value=\"1\")\n",
    "pipeline_param12 = PipelineParameter(name=\"pipeline_arg12\", default_value=\"image_classification_model\")\n",
    "pipeline_param13 = PipelineParameter(name=\"pipeline_arg13\", default_value=\"image analysis type\")\n",
    "\n",
    "trainStep = PythonScriptStep(\n",
    "    script_name=\"train.py\",\n",
    "    arguments=[\n",
    "        \"--subscription_id\",         pipeline_param01,\n",
    "        \"--resource_group\",          pipeline_param02,\n",
    "        \"--workspace_name\",          pipeline_param03,\n",
    "        \"--cluster_name\",            pipeline_param04,\n",
    "        \"--experiment_name\",         pipeline_param05,\n",
    "        \"--base_model\",              pipeline_param06,\n",
    "        \"--dataset_name\",            pipeline_param07,\n",
    "        \"--dataset_name_for_train\",  pipeline_param08,\n",
    "        \"--dataset_name_for_test\",   pipeline_param09,\n",
    "        \"--train_ratio\",             pipeline_param10,\n",
    "        \"--random_seed\",             pipeline_param11,\n",
    "        \"--model_name\",              pipeline_param12,\n",
    "        \"--TASK_TYPE\",               pipeline_param13\n",
    "    ],\n",
    "    compute_target=compute_target,\n",
    "    source_directory='.',\n",
    "    runconfig=aml_run_config,\n",
    "    allow_reuse = True,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4.2.1 Define AML pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline(workspace=ws, steps=[trainStep])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4.2.2 Work-around in some error\n",
    "\n",
    "Try the following commands in the next cell, if you encounter the error:\n",
    "\n",
    "```sh\n",
    "While attempting to take snapshot of .\n",
    "Your total snapshot size exceeds the limit of 300.0 MB\n",
    "```\n",
    "\n",
    "ref. https://docs.microsoft.com/en-us/azure/machine-learning/how-to-save-write-experiment-files#storage-limits-of-experiment-snapshots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import azureml\n",
    "azureml._restclient.snapshots_client.SNAPSHOT_MAX_SIZE_BYTES = 1000000000"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4.3 Publish the pipeline\n",
    "\n",
    "Publish the pipeline, and you can use it from other Azure resources like [Azure Data Factory](https://azure.microsoft.com/en-us/services/data-factory/).\n",
    "Also, by using the following steps, the published URI will be kept."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    pipelineEndpoint = PipelineEndpoint.get(workspace=ws, name=pipelineName)\n",
    "except ErrorResponseException as ex:\n",
    "    if \"not found in workspace\" in ex.message:\n",
    "        pipelineEndpoint = None\n",
    "    else:\n",
    "        raise\n",
    "\n",
    "if pipelineEndpoint is None:\n",
    "    print('Pipeline does not exists, creating new: ' + pipelineName)\n",
    "    pipelineEndpoint = PipelineEndpoint.publish(workspace = ws\n",
    "                                        ,name = pipelineName\n",
    "                                        ,pipeline=pipeline\n",
    "                                        ,description=\"My Published Pipeline Description.\")\n",
    "else:\n",
    "    print('Found existing pipeline ' + pipelineName + ', adding new version.')\n",
    "    published_pipeline = pipeline.publish(name = pipelineName + \"_Pipeline\")\n",
    "    pipelineEndpoint.add_default(published_pipeline)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.4.4 Submit the pipeline\n",
    "\n",
    "You can execute the pipeline by giving the parameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline.submit(experiment_name=experiment_name,\n",
    "                pipeline_parameters={\"pipeline_arg01\": subscription_id\n",
    "                                    ,\"pipeline_arg02\": resource_group\n",
    "                                    ,\"pipeline_arg03\": workspace_name\n",
    "                                    ,\"pipeline_arg04\": cluster_name\n",
    "                                    ,\"pipeline_arg05\": experiment_name\n",
    "                                    ,\"pipeline_arg06\": base_model\n",
    "                                    ,\"pipeline_arg07\": dataset_name\n",
    "                                    ,\"pipeline_arg08\": dataset_name_for_train\n",
    "                                    ,\"pipeline_arg09\": dataset_name_for_test\n",
    "                                    ,\"pipeline_arg10\": train_ratio\n",
    "                                    ,\"pipeline_arg11\": random_seed\n",
    "                                    ,\"pipeline_arg12\": model_name\n",
    "                                    ,\"pipeline_arg13\": TASK_TYPE                           \n",
    "                                    })"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (py37_aml)",
   "language": "python",
   "name": "py37_aml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}