aml-pipelines-with-automated-machine-learning-step.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "Copyright (c) Microsoft Corporation. All rights reserved.  \n",
        "Licensed under the MIT License."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "![Impressions](https://PixelServer20190423114238.azurewebsites.net/api/impressions/NotebookVM/how-to-use-azureml/machine-learning-pipelines/intro-to-pipelines/aml-pipelines-with-automated-machine-learning-step.png)"
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Azure Machine Learning Pipeline with AutoMLStep (Udacity Course 2)\n",
        "This notebook demonstrates the use of AutoMLStep in Azure Machine Learning Pipeline."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Introduction\n",
        "In this example we showcase how you can use AzureML Dataset to load data for AutoML via AML Pipeline. \n",
        "\n",
        "If you are using an Azure Machine Learning Notebook VM, you are all set. Otherwise, make sure you have executed the [configuration](https://aka.ms/pl-config) before running this notebook.\n",
        "\n",
        "In this notebook you will learn how to:\n",
        "1. Create an `Experiment` in an existing `Workspace`.\n",
        "2. Create or Attach existing AmlCompute to a workspace.\n",
        "3. Define data loading in a `TabularDataset`.\n",
        "4. Configure AutoML using `AutoMLConfig`.\n",
        "5. Use AutoMLStep\n",
        "6. Train the model using AmlCompute\n",
        "7. Explore the results.\n",
        "8. Test the best fitted model."
      ],
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Azure Machine Learning and Pipeline SDK-specific imports"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import logging\n",
        "import os\n",
        "import csv\n",
        "\n",
        "from matplotlib import pyplot as plt\n",
        "import numpy as np\n",
        "import pandas as pd\n",
        "from sklearn import datasets\n",
        "import pkg_resources\n",
        "\n",
        "import azureml.core\n",
        "from azureml.core.experiment import Experiment\n",
        "from azureml.core.workspace import Workspace\n",
        "from azureml.train.automl import AutoMLConfig\n",
        "from azureml.core.dataset import Dataset\n",
        "\n",
        "from azureml.pipeline.steps import AutoMLStep\n",
        "\n",
        "# Check core SDK version number\n",
        "print(\"SDK version:\", azureml.core.VERSION)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "SDK version: 1.44.0\n"
        }
      ],
      "execution_count": 1,
      "metadata": {
        "gather": {
          "logged": 1663604482238
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Initialize Workspace\n",
        "Initialize a workspace object from persisted configuration. Make sure the config file is present at .\\config.json"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "ws = Workspace.from_config()\n",
        "print(ws.name, ws.resource_group, ws.location, ws.subscription_id, sep = '\\n')"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "quick-starts-ws-207945\naml-quickstarts-207945\nsouthcentralus\n976ee174-3882-4721-b90a-b5fef6b72f24\n"
        }
      ],
      "execution_count": 2,
      "metadata": {
        "gather": {
          "logged": 1663604482932
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Create an Azure ML experiment\n",
        "Let's create an experiment named \"automlstep-classification\" and a folder to hold the training scripts. The script runs will be recorded under the experiment in Azure.\n",
        "\n",
        "The best practice is to use separate folders for scripts and its dependent files for each step and specify that folder as the `source_directory` for the step. This helps reduce the size of the snapshot created for the step (only the specific folder is snapshotted). Since changes in any files in the `source_directory` would trigger a re-upload of the snapshot, this helps keep the reuse of the step when there are no changes in the `source_directory` of the step.\n",
        "\n",
        "*Udacity Note:* There is no need to create an Azure ML experiment, this needs to re-use the experiment that was already created\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Choose a name for the run history container in the workspace.\n",
        "# NOTE: update these to match your existing experiment name\n",
        "experiment_name = 'mlops-bankmarketing-experiment'\n",
        "project_folder = './pipeline-project'\n",
        "\n",
        "experiment = Experiment(ws, experiment_name)\n",
        "experiment"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 3,
          "data": {
            "text/plain": "Experiment(Name: mlops-bankmarketing-experiment,\nWorkspace: quick-starts-ws-207945)",
            "text/html": "<table style=\"width:100%\"><tr><th>Name</th><th>Workspace</th><th>Report Page</th><th>Docs Page</th></tr><tr><td>mlops-bankmarketing-experiment</td><td>quick-starts-ws-207945</td><td><a href=\"https://ml.azure.com/experiments/id/669a2585-1744-408c-808d-1c6d0d12665b?wsid=/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourcegroups/aml-quickstarts-207945/workspaces/quick-starts-ws-207945&amp;tid=660b3398-b80e-49d2-bc5b-ac1dc93b5254\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td><td><a href=\"https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.experiment.Experiment?view=azure-ml-py\" target=\"_blank\" rel=\"noopener\">Link to Documentation</a></td></tr></table>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 3,
      "metadata": {
        "gather": {
          "logged": 1663604502379
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Create or Attach an AmlCompute cluster\n",
        "You will need to create a [compute target](https://docs.microsoft.com/azure/machine-learning/service/concept-azure-machine-learning-architecture#compute-target) for your AutoML run. In this tutorial, you get the default `AmlCompute` as your training compute resource.\n",
        "\n",
        "**Udacity Note** There is no need to create a new compute target, it can re-use the previous cluster"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.core.compute import AmlCompute\n",
        "from azureml.core.compute import ComputeTarget\n",
        "from azureml.core.compute_target import ComputeTargetException\n",
        "\n",
        "# NOTE: update the cluster name to match the existing cluster\n",
        "# Choose a name for your CPU cluster\n",
        "amlcompute_cluster_name = \"compute-cluster-mlops\"\n",
        "\n",
        "# Verify that cluster does not exist already\n",
        "try:\n",
        "    compute_target = ComputeTarget(workspace=ws, name=amlcompute_cluster_name)\n",
        "    print('Found existing cluster, use it.')\n",
        "except ComputeTargetException:\n",
        "    compute_config = AmlCompute.provisioning_configuration(vm_size='STANDARD_D2_V2',# for GPU, use \"STANDARD_NC6\"\n",
        "                                                           #vm_priority = 'lowpriority', # optional\n",
        "                                                           max_nodes=4)\n",
        "    compute_target = ComputeTarget.create(ws, amlcompute_cluster_name, compute_config)\n",
        "\n",
        "compute_target.wait_for_completion(show_output=True, min_node_count = 1, timeout_in_minutes = 10)\n",
        "# For a more detailed view of current AmlCompute status, use get_status()."
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Found existing cluster, use it.\nSucceeded\nAmlCompute wait for completion finished\n\nMinimum number of nodes requested have been provisioned\n"
        }
      ],
      "execution_count": 4,
      "metadata": {
        "gather": {
          "logged": 1663604519458
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Data\n",
        "\n",
        "**Udacity note:** Make sure the `key` is the same name as the dataset that is uploaded, and that the description matches. If it is hard to find or unknown, loop over the `ws.datasets.keys()` and `print()` them.\n",
        "If it *isn't* found because it was deleted, it can be recreated with the link that has the CSV "
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Try to load the dataset from the Workspace. Otherwise, create it from the file\n",
        "# NOTE: update the key to match the dataset name\n",
        "found = False\n",
        "key = \"bankmarketing-dataset\"\n",
        "description_text = \"Bank Marketing DataSet for Udacity Course 2\"\n",
        "\n",
        "if key in ws.datasets.keys(): \n",
        "        found = True\n",
        "        dataset = ws.datasets[key] \n",
        "\n",
        "if not found:\n",
        "        # Create AML Dataset and register it into Workspace\n",
        "        example_data = 'https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_train.csv'\n",
        "        dataset = Dataset.Tabular.from_delimited_files(example_data)        \n",
        "        #Register Dataset in Workspace\n",
        "        dataset = dataset.register(workspace=ws,\n",
        "                                   name=key,\n",
        "                                   description=description_text)\n",
        "\n",
        "\n",
        "df = dataset.to_pandas_dataframe()\n",
        "df.describe()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 5,
          "data": {
            "text/plain": "                age      duration      campaign         pdays      previous  \\\ncount  32950.000000  32950.000000  32950.000000  32950.000000  32950.000000   \nmean      40.040212    257.335205      2.561730    962.174780      0.174780   \nstd       10.432313    257.331700      2.763646    187.646785      0.496503   \nmin       17.000000      0.000000      1.000000      0.000000      0.000000   \n25%       32.000000    102.000000      1.000000    999.000000      0.000000   \n50%       38.000000    179.000000      2.000000    999.000000      0.000000   \n75%       47.000000    318.000000      3.000000    999.000000      0.000000   \nmax       98.000000   4918.000000     56.000000    999.000000      7.000000   \n\n       emp.var.rate  cons.price.idx  cons.conf.idx     euribor3m   nr.employed  \ncount  32950.000000    32950.000000   32950.000000  32950.000000  32950.000000  \nmean       0.076228       93.574243     -40.518680      3.615654   5166.859608  \nstd        1.572242        0.578636       4.623004      1.735748     72.208448  \nmin       -3.400000       92.201000     -50.800000      0.634000   4963.600000  \n25%       -1.800000       93.075000     -42.700000      1.344000   5099.100000  \n50%        1.100000       93.749000     -41.800000      4.857000   5191.000000  \n75%        1.400000       93.994000     -36.400000      4.961000   5228.100000  \nmax        1.400000       94.767000     -26.900000      5.045000   5228.100000  ",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>age</th>\n      <th>duration</th>\n      <th>campaign</th>\n      <th>pdays</th>\n      <th>previous</th>\n      <th>emp.var.rate</th>\n      <th>cons.price.idx</th>\n      <th>cons.conf.idx</th>\n      <th>euribor3m</th>\n      <th>nr.employed</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>count</th>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n      <td>32950.000000</td>\n    </tr>\n    <tr>\n      <th>mean</th>\n      <td>40.040212</td>\n      <td>257.335205</td>\n      <td>2.561730</td>\n      <td>962.174780</td>\n      <td>0.174780</td>\n      <td>0.076228</td>\n      <td>93.574243</td>\n      <td>-40.518680</td>\n      <td>3.615654</td>\n      <td>5166.859608</td>\n    </tr>\n    <tr>\n      <th>std</th>\n      <td>10.432313</td>\n      <td>257.331700</td>\n      <td>2.763646</td>\n      <td>187.646785</td>\n      <td>0.496503</td>\n      <td>1.572242</td>\n      <td>0.578636</td>\n      <td>4.623004</td>\n      <td>1.735748</td>\n      <td>72.208448</td>\n    </tr>\n    <tr>\n      <th>min</th>\n      <td>17.000000</td>\n      <td>0.000000</td>\n      <td>1.000000</td>\n      <td>0.000000</td>\n      <td>0.000000</td>\n      <td>-3.400000</td>\n      <td>92.201000</td>\n      <td>-50.800000</td>\n      <td>0.634000</td>\n      <td>4963.600000</td>\n    </tr>\n    <tr>\n      <th>25%</th>\n      <td>32.000000</td>\n      <td>102.000000</td>\n      <td>1.000000</td>\n      <td>999.000000</td>\n      <td>0.000000</td>\n      <td>-1.800000</td>\n      <td>93.075000</td>\n      <td>-42.700000</td>\n      <td>1.344000</td>\n      <td>5099.100000</td>\n    </tr>\n    <tr>\n      <th>50%</th>\n      <td>38.000000</td>\n      <td>179.000000</td>\n      <td>2.000000</td>\n      <td>999.000000</td>\n      <td>0.000000</td>\n      <td>1.100000</td>\n      <td>93.749000</td>\n      <td>-41.800000</td>\n      <td>4.857000</td>\n      <td>5191.000000</td>\n    </tr>\n    <tr>\n      <th>75%</th>\n      <td>47.000000</td>\n      <td>318.000000</td>\n      <td>3.000000</td>\n      <td>999.000000</td>\n      <td>0.000000</td>\n      <td>1.400000</td>\n      <td>93.994000</td>\n      <td>-36.400000</td>\n      <td>4.961000</td>\n      <td>5228.100000</td>\n    </tr>\n    <tr>\n      <th>max</th>\n      <td>98.000000</td>\n      <td>4918.000000</td>\n      <td>56.000000</td>\n      <td>999.000000</td>\n      <td>7.000000</td>\n      <td>1.400000</td>\n      <td>94.767000</td>\n      <td>-26.900000</td>\n      <td>5.045000</td>\n      <td>5228.100000</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 5,
      "metadata": {
        "gather": {
          "logged": 1663604542171
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Review the Dataset Result\n",
        "\n",
        "You can peek the result of a TabularDataset at any range using `skip(i)` and `take(j).to_pandas_dataframe()`. Doing so evaluates only `j` records for all the steps in the TabularDataset, which makes it fast even against large datasets.\n",
        "\n",
        "`TabularDataset` objects are composed of a list of transformation steps (optional)."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dataset.take(5).to_pandas_dataframe()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 6,
          "data": {
            "text/plain": "   age          job  marital    education  default housing loan    contact  \\\n0   57   technician  married  high.school       no      no  yes   cellular   \n1   55      unknown  married      unknown  unknown     yes   no  telephone   \n2   33  blue-collar  married     basic.9y       no      no   no   cellular   \n3   36       admin.  married  high.school       no      no   no  telephone   \n4   27    housemaid  married  high.school       no     yes   no   cellular   \n\n  month day_of_week  ...  campaign  pdays  previous     poutcome emp.var.rate  \\\n0   may         mon  ...         1    999         1      failure         -1.8   \n1   may         thu  ...         2    999         0  nonexistent          1.1   \n2   may         fri  ...         1    999         1      failure         -1.8   \n3   jun         fri  ...         4    999         0  nonexistent          1.4   \n4   jul         fri  ...         2    999         0  nonexistent          1.4   \n\n   cons.price.idx  cons.conf.idx  euribor3m  nr.employed   y  \n0          92.893          -46.2      1.299       5099.1  no  \n1          93.994          -36.4      4.860       5191.0  no  \n2          92.893          -46.2      1.313       5099.1  no  \n3          94.465          -41.8      4.967       5228.1  no  \n4          93.918          -42.7      4.963       5228.1  no  \n\n[5 rows x 21 columns]",
            "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>age</th>\n      <th>job</th>\n      <th>marital</th>\n      <th>education</th>\n      <th>default</th>\n      <th>housing</th>\n      <th>loan</th>\n      <th>contact</th>\n      <th>month</th>\n      <th>day_of_week</th>\n      <th>...</th>\n      <th>campaign</th>\n      <th>pdays</th>\n      <th>previous</th>\n      <th>poutcome</th>\n      <th>emp.var.rate</th>\n      <th>cons.price.idx</th>\n      <th>cons.conf.idx</th>\n      <th>euribor3m</th>\n      <th>nr.employed</th>\n      <th>y</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>57</td>\n      <td>technician</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>mon</td>\n      <td>...</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.8</td>\n      <td>92.893</td>\n      <td>-46.2</td>\n      <td>1.299</td>\n      <td>5099.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>55</td>\n      <td>unknown</td>\n      <td>married</td>\n      <td>unknown</td>\n      <td>unknown</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>may</td>\n      <td>thu</td>\n      <td>...</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.1</td>\n      <td>93.994</td>\n      <td>-36.4</td>\n      <td>4.860</td>\n      <td>5191.0</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>33</td>\n      <td>blue-collar</td>\n      <td>married</td>\n      <td>basic.9y</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>may</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>1</td>\n      <td>999</td>\n      <td>1</td>\n      <td>failure</td>\n      <td>-1.8</td>\n      <td>92.893</td>\n      <td>-46.2</td>\n      <td>1.313</td>\n      <td>5099.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>36</td>\n      <td>admin.</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>no</td>\n      <td>no</td>\n      <td>telephone</td>\n      <td>jun</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>4</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.4</td>\n      <td>94.465</td>\n      <td>-41.8</td>\n      <td>4.967</td>\n      <td>5228.1</td>\n      <td>no</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>27</td>\n      <td>housemaid</td>\n      <td>married</td>\n      <td>high.school</td>\n      <td>no</td>\n      <td>yes</td>\n      <td>no</td>\n      <td>cellular</td>\n      <td>jul</td>\n      <td>fri</td>\n      <td>...</td>\n      <td>2</td>\n      <td>999</td>\n      <td>0</td>\n      <td>nonexistent</td>\n      <td>1.4</td>\n      <td>93.918</td>\n      <td>-42.7</td>\n      <td>4.963</td>\n      <td>5228.1</td>\n      <td>no</td>\n    </tr>\n  </tbody>\n</table>\n<p>5 rows × 21 columns</p>\n</div>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 6,
      "metadata": {
        "gather": {
          "logged": 1663604542479
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Train\n",
        "This creates a general AutoML settings object.\n",
        "**Udacity notes:** These inputs must match what was used when training in the portal. `label_column_name` has to be `y` for example."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "automl_settings = {\n",
        "    \"experiment_timeout_minutes\": 20,\n",
        "    \"max_concurrent_iterations\": 5,\n",
        "    \"primary_metric\" : 'AUC_weighted'\n",
        "}\n",
        "automl_config = AutoMLConfig(compute_target=compute_target,\n",
        "                             task = \"classification\",\n",
        "                             training_data=dataset,\n",
        "                             label_column_name=\"y\",   \n",
        "                             path = project_folder,\n",
        "                             enable_early_stopping= True,\n",
        "                             featurization= 'auto',\n",
        "                             debug_log = \"automl_errors.log\",\n",
        "                             **automl_settings\n",
        "                            )"
      ],
      "outputs": [],
      "execution_count": 7,
      "metadata": {
        "gather": {
          "logged": 1663604544366
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Create Pipeline and AutoMLStep\n",
        "\n",
        "You can define outputs for the AutoMLStep using TrainingOutput."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core import PipelineData, TrainingOutput\n",
        "\n",
        "ds = ws.get_default_datastore()\n",
        "metrics_output_name = 'metrics_output'\n",
        "best_model_output_name = 'best_model_output'\n",
        "\n",
        "metrics_data = PipelineData(name='metrics_data',\n",
        "                           datastore=ds,\n",
        "                           pipeline_output_name=metrics_output_name,\n",
        "                           training_output=TrainingOutput(type='Metrics'))\n",
        "model_data = PipelineData(name='model_data',\n",
        "                           datastore=ds,\n",
        "                           pipeline_output_name=best_model_output_name,\n",
        "                           training_output=TrainingOutput(type='Model'))"
      ],
      "outputs": [],
      "execution_count": 8,
      "metadata": {
        "gather": {
          "logged": 1663604547626
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create an AutoMLStep."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "automl_step = AutoMLStep(\n",
        "    name='automl_module',\n",
        "    automl_config=automl_config,\n",
        "    outputs=[metrics_data, model_data],\n",
        "    allow_reuse=True)"
      ],
      "outputs": [],
      "execution_count": 9,
      "metadata": {
        "tags": [
          "automlstep-remarks-sample1"
        ],
        "gather": {
          "logged": 1663604549572
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core import Pipeline\n",
        "pipeline = Pipeline(\n",
        "    description=\"pipeline_with_automlstep\",\n",
        "    workspace=ws,    \n",
        "    steps=[automl_step])"
      ],
      "outputs": [],
      "execution_count": 10,
      "metadata": {
        "tags": [
          "automlstep-remarks-sample2"
        ],
        "gather": {
          "logged": 1663604553238
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipeline_run = experiment.submit(pipeline)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Created step automl_module [0a1d2e75][9b8165ad-6e6d-48b9-8958-fae7562d0a86], (This step will run and generate new outputs)\nSubmitted PipelineRun 02294280-a906-470f-bb0e-2f4f4f9d29e5\nLink to Azure Machine Learning Portal: https://ml.azure.com/runs/02294280-a906-470f-bb0e-2f4f4f9d29e5?wsid=/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourcegroups/aml-quickstarts-207945/workspaces/quick-starts-ws-207945&tid=660b3398-b80e-49d2-bc5b-ac1dc93b5254\n"
        }
      ],
      "execution_count": 13,
      "metadata": {
        "gather": {
          "logged": 1663604575898
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.widgets import RunDetails\n",
        "RunDetails(pipeline_run).show()"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …",
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "72f9b08db00d4fadb81d9b2d62d7f020"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/runs/02294280-a906-470f-bb0e-2f4f4f9d29e5?wsid=/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourcegroups/aml-quickstarts-207945/workspaces/quick-starts-ws-207945&tid=660b3398-b80e-49d2-bc5b-ac1dc93b5254\", \"run_id\": \"02294280-a906-470f-bb0e-2f4f4f9d29e5\", \"run_properties\": {\"run_id\": \"02294280-a906-470f-bb0e-2f4f4f9d29e5\", \"created_utc\": \"2022-09-19T16:22:54.969537Z\", \"properties\": {\"azureml.runsource\": \"azureml.PipelineRun\", \"runSource\": \"SDK\", \"runType\": \"SDK\", \"azureml.parameters\": \"{}\", \"azureml.continue_on_step_failure\": \"False\", \"azureml.continue_on_failed_optional_input\": \"True\", \"azureml.pipelineComponent\": \"pipelinerun\"}, \"tags\": {}, \"end_time_utc\": \"2022-09-19T16:36:27.022583Z\", \"status\": \"Completed\", \"log_files\": {\"logs/azureml/executionlogs.txt\": \"https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.02294280-a906-470f-bb0e-2f4f4f9d29e5/logs/azureml/executionlogs.txt?sv=2019-07-07&sr=b&sig=2gN26jZ0RXaPiDQcY17hNW89VUKT7RsI2VqCc46qAPI%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A29%3A24Z&se=2022-09-20T00%3A39%3A24Z&sp=r\", \"logs/azureml/stderrlogs.txt\": \"https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.02294280-a906-470f-bb0e-2f4f4f9d29e5/logs/azureml/stderrlogs.txt?sv=2019-07-07&sr=b&sig=TEKJNok41SiHmdDSZx8MjjZtlsozrzdA%2F%2BtnTS78Pzw%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A29%3A24Z&se=2022-09-20T00%3A39%3A24Z&sp=r\", \"logs/azureml/stdoutlogs.txt\": \"https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.02294280-a906-470f-bb0e-2f4f4f9d29e5/logs/azureml/stdoutlogs.txt?sv=2019-07-07&sr=b&sig=RAib6g5D8AkIFx0%2BVwVFb75L1%2FdTFPuwlqBvqpOe2pc%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A29%3A24Z&se=2022-09-20T00%3A39%3A24Z&sp=r\"}, \"log_groups\": [[\"logs/azureml/executionlogs.txt\", \"logs/azureml/stderrlogs.txt\", \"logs/azureml/stdoutlogs.txt\"]], \"run_duration\": \"0:13:32\", \"run_number\": \"1663604574\", \"run_queued_details\": {\"status\": \"Finished\", \"details\": null}}, \"child_runs\": [{\"run_id\": \"9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2022-09-19T16:23:11.17281Z\", \"created_time\": \"2022-09-19T16:22:56.942033Z\", \"end_time\": \"2022-09-19T16:35:32.516528Z\", \"duration\": \"0:12:35\", \"run_number\": 1663604576, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2022-09-19T16:22:56.942033Z\", \"is_reused\": \"\"}], \"children_metrics\": {\"categories\": null, \"series\": null, \"metricName\": null}, \"run_metrics\": [], \"run_logs\": \"[2022-09-19 16:22:56Z] Submitting 1 runs, first five are: 0a1d2e75:9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13\\n[2022-09-19 16:36:26Z] Completing processing run id 9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13.\\n\\nRun is completed.\", \"graph\": {\"datasource_nodes\": {\"6883c63a\": {\"node_id\": \"6883c63a\", \"name\": \"bankmarketing-dataset\"}}, \"module_nodes\": {\"0a1d2e75\": {\"node_id\": \"0a1d2e75\", \"name\": \"automl_module\", \"status\": \"Finished\", \"_is_reused\": false, \"run_id\": \"9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13\"}}, \"edges\": [{\"source_node_id\": \"6883c63a\", \"source_node_name\": \"bankmarketing-dataset\", \"source_name\": \"data\", \"target_name\": \"training_data\", \"dst_node_id\": \"0a1d2e75\", \"dst_node_name\": \"automl_module\"}], \"child_runs\": [{\"run_id\": \"9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2022-09-19T16:23:11.17281Z\", \"created_time\": \"2022-09-19T16:22:56.942033Z\", \"end_time\": \"2022-09-19T16:35:32.516528Z\", \"duration\": \"0:12:35\", \"run_number\": 1663604576, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2022-09-19T16:22:56.942033Z\", \"is_reused\": \"\"}]}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.44.0\"}, \"loading\": false}"
          },
          "metadata": {}
        }
      ],
      "execution_count": 14,
      "metadata": {
        "gather": {
          "logged": 1663604602197
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipeline_run.wait_for_completion()"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "PipelineRunId: 02294280-a906-470f-bb0e-2f4f4f9d29e5\nLink to Azure Machine Learning Portal: https://ml.azure.com/runs/02294280-a906-470f-bb0e-2f4f4f9d29e5?wsid=/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourcegroups/aml-quickstarts-207945/workspaces/quick-starts-ws-207945&tid=660b3398-b80e-49d2-bc5b-ac1dc93b5254\nPipelineRun Status: Running\n\n\nStepRunId: 9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13\nLink to Azure Machine Learning Portal: https://ml.azure.com/runs/9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13?wsid=/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourcegroups/aml-quickstarts-207945/workspaces/quick-starts-ws-207945&tid=660b3398-b80e-49d2-bc5b-ac1dc93b5254\nStepRun( automl_module ) Status: Running\n\nStepRun(automl_module) Execution Summary\n=========================================\nStepRun( automl_module ) Status: Finished\n\nWarnings:\nNo scores improved over last 10 iterations, so experiment stopped early. This early stopping behavior can be disabled by setting enable_early_stopping = False in AutoMLConfig for notebook/python SDK runs.\n{'runId': '9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13', 'target': 'compute-cluster-mlops', 'status': 'Completed', 'startTimeUtc': '2022-09-19T16:23:11.17281Z', 'endTimeUtc': '2022-09-19T16:35:32.516528Z', 'services': {}, 'warnings': [{'source': 'JasmineService', 'message': 'No scores improved over last 10 iterations, so experiment stopped early. This early stopping behavior can be disabled by setting enable_early_stopping = False in AutoMLConfig for notebook/python SDK runs.'}], 'properties': {'ContentSnapshotId': '32b6b193-91d2-41f5-8199-c43468a78848', 'StepType': 'AutoMLStep', 'azureml.moduleid': '9b8165ad-6e6d-48b9-8958-fae7562d0a86', 'azureml.moduleName': 'automl_module', 'azureml.runsource': 'azureml.StepRun', 'azureml.nodeid': '0a1d2e75', 'azureml.pipelinerunid': '02294280-a906-470f-bb0e-2f4f4f9d29e5', 'azureml.pipeline': '02294280-a906-470f-bb0e-2f4f4f9d29e5', 'azureml.pipelineComponent': 'masterautomlcloud', 'num_iterations': '1000', 'training_type': 'TrainFull', 'acquisition_function': 'EI', 'metrics': 'accuracy', 'primary_metric': 'AUC_weighted', 'train_split': '0', 'MaxTimeSeconds': None, 'acquisition_parameter': '0', 'num_cross_validation': '', 'target': 'compute-cluster-mlops', 'RawAMLSettingsString': None, 'AMLSettingsJsonString': '{\"path\": null, \"name\": \"placeholder\", \"subscription_id\": \"976ee174-3882-4721-b90a-b5fef6b72f24\", \"resource_group\": \"aml-quickstarts-207945\", \"workspace_name\": \"quick-starts-ws-207945\", \"region\": \"southcentralus\", \"compute_target\": \"compute-cluster-mlops\", \"spark_service\": null, \"azure_service\": null, \"many_models\": false, \"pipeline_fetch_max_batch_size\": 1, \"enable_batch_run\": false, \"enable_run_restructure\": false, \"start_auxiliary_runs_before_parent_complete\": false, \"enable_code_generation\": true, \"iterations\": 1000, \"primary_metric\": \"AUC_weighted\", \"task_type\": \"classification\", \"positive_label\": null, \"data_script\": null, \"test_size\": 0.0, \"test_include_predictions_only\": false, \"validation_size\": 0.0, \"n_cross_validations\": null, \"y_min\": null, \"y_max\": null, \"num_classes\": null, \"featurization\": \"auto\", \"_ignore_package_version_incompatibilities\": false, \"is_timeseries\": false, \"max_cores_per_iteration\": 1, \"max_concurrent_iterations\": 5, \"iteration_timeout_minutes\": null, \"mem_in_mb\": null, \"enforce_time_on_windows\": false, \"experiment_timeout_minutes\": 20, \"experiment_exit_score\": null, \"partition_column_names\": null, \"whitelist_models\": null, \"blacklist_algos\": null, \"supported_models\": [\"MultinomialNaiveBayes\", \"LinearSVM\", \"BernoulliNaiveBayes\", \"ExtremeRandomTrees\", \"XGBoostClassifier\", \"TabnetClassifier\", \"DecisionTree\", \"SVM\", \"LightGBM\", \"LogisticRegression\", \"TensorFlowDNN\", \"AveragedPerceptronClassifier\", \"RandomForest\", \"GradientBoosting\", \"TensorFlowLinearClassifier\", \"KNN\", \"SGD\"], \"private_models\": [], \"auto_blacklist\": true, \"blacklist_samples_reached\": false, \"exclude_nan_labels\": true, \"verbosity\": 20, \"_debug_log\": \"automl_errors.log\", \"show_warnings\": false, \"model_explainability\": true, \"service_url\": null, \"sdk_url\": null, \"sdk_packages\": null, \"enable_onnx_compatible_models\": false, \"enable_split_onnx_featurizer_estimator_models\": false, \"vm_type\": \"STANDARD_DS3_V2\", \"telemetry_verbosity\": 20, \"send_telemetry\": true, \"enable_dnn\": false, \"scenario\": \"AutoML\", \"environment_label\": null, \"save_mlflow\": false, \"enable_categorical_indicators\": false, \"force_text_dnn\": false, \"enable_feature_sweeping\": true, \"enable_early_stopping\": true, \"early_stopping_n_iters\": 10, \"arguments\": null, \"dataset_id\": null, \"hyperdrive_config\": null, \"validation_dataset_id\": null, \"run_source\": null, \"metrics\": null, \"enable_metric_confidence\": false, \"enable_ensembling\": true, \"enable_stack_ensembling\": true, \"ensemble_iterations\": 15, \"enable_tf\": false, \"enable_subsampling\": null, \"subsample_seed\": null, \"enable_nimbusml\": false, \"enable_streaming\": false, \"force_streaming\": false, \"track_child_runs\": true, \"allowed_private_models\": [], \"label_column_name\": \"y\", \"weight_column_name\": null, \"cv_split_column_names\": null, \"enable_local_managed\": false, \"_local_managed_run_id\": null, \"cost_mode\": 1, \"lag_length\": 0, \"metric_operation\": \"maximize\", \"preprocess\": true}', 'DataPrepJsonString': '{\\\\\"training_data\\\\\": {\\\\\"datasetId\\\\\": \\\\\"d602c7f1-59b7-491e-9e8a-03512cc84166\\\\\"}, \\\\\"datasets\\\\\": 0}', 'EnableSubsampling': 'False', 'runTemplate': 'AutoML', 'Orchestrator': 'AutoML', 'ClientType': 'Others', '_aml_system_scenario_identification': 'Remote.Parent', 'ClientSdkVersion': '1.44.0', 'root_attribution': 'azureml.StepRun', 'snapshotId': '32b6b193-91d2-41f5-8199-c43468a78848', 'SetupRunId': '9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13_setup', 'SetupRunContainerId': 'dcid.9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13_setup', 'FeaturizationRunJsonPath': 'featurizer_container.json', 'FeaturizationRunId': '9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13_featurize', 'ProblemInfoJsonString': '{\"dataset_num_categorical\": 0, \"is_sparse\": true, \"subsampling\": false, \"has_extra_col\": true, \"dataset_classes\": 2, \"dataset_features\": 132, \"dataset_samples\": 29655, \"single_frequency_class_detected\": false}', 'ModelExplainRunId': '9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13_ModelExplain'}, 'inputDatasets': [{'dataset': {'id': 'd602c7f1-59b7-491e-9e8a-03512cc84166'}, 'consumptionDetails': {'type': 'RunInput', 'inputName': 'training_data', 'mechanism': 'Direct'}}], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13/logs/azureml/executionlogs.txt?sv=2019-07-07&sr=b&sig=bEdqxaJXKt9Ltss0c5bSWoFqBVNEYqVg4h91g57V5yU%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A23%3A16Z&se=2022-09-20T00%3A33%3A16Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13/logs/azureml/stderrlogs.txt?sv=2019-07-07&sr=b&sig=20t%2F65%2FGv56KjwbOCuJjixzvjpliwE0gk79Iwh2ltt4%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A23%3A16Z&se=2022-09-20T00%3A33%3A16Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13/logs/azureml/stdoutlogs.txt?sv=2019-07-07&sr=b&sig=jpPRzivwGBz%2FL5%2F7Pk86V4LRiEdSFOSiy4CNGoEm%2BP0%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A23%3A16Z&se=2022-09-20T00%3A33%3A16Z&sp=r'}, 'submittedBy': 'ODL_User 207945'}\n\n\n\nPipelineRun Execution Summary\n==============================\nPipelineRun Status: Finished\n{'runId': '02294280-a906-470f-bb0e-2f4f4f9d29e5', 'status': 'Completed', 'startTimeUtc': '2022-09-19T16:22:56.363591Z', 'endTimeUtc': '2022-09-19T16:36:27.022583Z', 'services': {}, 'properties': {'azureml.runsource': 'azureml.PipelineRun', 'runSource': 'SDK', 'runType': 'SDK', 'azureml.parameters': '{}', 'azureml.continue_on_step_failure': 'False', 'azureml.continue_on_failed_optional_input': 'True', 'azureml.pipelineComponent': 'pipelinerun'}, 'inputDatasets': [{'dataset': {'id': 'd602c7f1-59b7-491e-9e8a-03512cc84166'}, 'consumptionDetails': {'type': 'RunInput', 'inputName': 'training_data', 'mechanism': 'Direct'}}], 'outputDatasets': [], 'logFiles': {'logs/azureml/executionlogs.txt': 'https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.02294280-a906-470f-bb0e-2f4f4f9d29e5/logs/azureml/executionlogs.txt?sv=2019-07-07&sr=b&sig=ma8jaafxS%2BiwDES8sHnCUXN%2BhJcFPu6XvxJ8hLl45%2Fc%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A23%3A58Z&se=2022-09-20T00%3A33%3A58Z&sp=r', 'logs/azureml/stderrlogs.txt': 'https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.02294280-a906-470f-bb0e-2f4f4f9d29e5/logs/azureml/stderrlogs.txt?sv=2019-07-07&sr=b&sig=A7mn7y6hVq68vpx3OS4s95Ks2Z7bsN%2FCjfCDls2gTD0%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A23%3A58Z&se=2022-09-20T00%3A33%3A58Z&sp=r', 'logs/azureml/stdoutlogs.txt': 'https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.02294280-a906-470f-bb0e-2f4f4f9d29e5/logs/azureml/stdoutlogs.txt?sv=2019-07-07&sr=b&sig=sHEXH4Yf4vp2uzNPyyPcI0c%2BqHZUuJ4bZyzEFPssRrk%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A23%3A58Z&se=2022-09-20T00%3A33%3A58Z&sp=r'}, 'submittedBy': 'ODL_User 207945'}\n\n"
        },
        {
          "output_type": "stream",
          "name": "stderr",
          "text": "WARNING:urllib3.connectionpool:Retrying (Retry(total=2, connect=2, read=3, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7fc1da841dc0>: Failed to establish a new connection: [Errno 111] Connection refused')': /artifact/v2.0/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourceGroups/aml-quickstarts-207945/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-207945/artifacts/prefix/contentinfo/ExperimentRun/dcid.9b1d6d79-9f23-475d-aa0d-a2de5bd9ee13/user_logs\n"
        },
        {
          "output_type": "execute_result",
          "execution_count": 15,
          "data": {
            "text/plain": "'Finished'"
          },
          "metadata": {}
        }
      ],
      "execution_count": 15,
      "metadata": {
        "gather": {
          "logged": 1663605389593
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Examine Results\n",
        "\n",
        "### Retrieve the metrics of all child runs\n",
        "Outputs of above run can be used as inputs of other steps in pipeline. In this tutorial, we will examine the outputs by retrieve output data and running some tests."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "metrics_output = pipeline_run.get_pipeline_output(metrics_output_name)\n",
        "num_file_downloaded = metrics_output.download('.', show_progress=True)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import json\n",
        "with open(metrics_output._path_on_datastore) as f:\n",
        "    metrics_output_result = f.read()\n",
        "    \n",
        "deserialized_metrics_output = json.loads(metrics_output_result)\n",
        "df = pd.DataFrame(deserialized_metrics_output)\n",
        "df"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Retrieve the Best Model"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Retrieve best model from Pipeline Run\n",
        "best_model_output = pipeline_run.get_pipeline_output(best_model_output_name)\n",
        "num_file_downloaded = best_model_output.download('.', show_progress=True)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "\n",
        "with open(best_model_output._path_on_datastore, \"rb\" ) as f:\n",
        "    best_model = pickle.load(f)\n",
        "best_model"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "best_model.steps"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Test the Model\n",
        "#### Load Test Data\n",
        "For the test data, it should have the same preparation step as the train data. Otherwise it might get failed at the preprocessing step."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "dataset_test = Dataset.Tabular.from_delimited_files(path='https://automlsamplenotebookdata.blob.core.windows.net/automl-sample-notebook-data/bankmarketing_test.csv')\n",
        "df_test = dataset_test.to_pandas_dataframe()\n",
        "df_test = df_test[pd.notnull(df_test['y'])]\n",
        "\n",
        "y_test = df_test['y']\n",
        "X_test = df_test.drop(['y'], axis=1)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "#### Testing Our Best Fitted Model\n",
        "\n",
        "We will use confusion matrix to see how our model works."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from sklearn.metrics import confusion_matrix\n",
        "ypred = best_model.predict(X_test)\n",
        "cm = confusion_matrix(y_test, ypred)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "# Visualize the confusion matrix\n",
        "pd.DataFrame(cm).style.background_gradient(cmap='Blues', low=0, high=0.9)"
      ],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Publish and run from REST endpoint\n",
        "\n",
        "Run the following code to publish the pipeline to your workspace. In your workspace in the portal, you can see metadata for the pipeline including run history and durations. You can also run the pipeline manually from the portal.\n",
        "\n",
        "Additionally, publishing the pipeline enables a REST endpoint to rerun the pipeline from any HTTP library on any platform.\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "published_pipeline = pipeline_run.publish_pipeline(\n",
        "    name=\"Bankmarketing Train\", description=\"Training bankmarketing pipeline\", version=\"1.0\")\n",
        "\n",
        "published_pipeline\n"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 16,
          "data": {
            "text/plain": "Pipeline(Name: Bankmarketing Train,\nId: 5b46ca32-41ba-4c69-8606-cfdbffc06513,\nStatus: Active,\nEndpoint: https://southcentralus.api.azureml.ms/pipelines/v1.0/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourceGroups/aml-quickstarts-207945/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-207945/PipelineRuns/PipelineSubmit/5b46ca32-41ba-4c69-8606-cfdbffc06513)",
            "text/html": "<table style=\"width:100%\"><tr><th>Name</th><th>Id</th><th>Status</th><th>Endpoint</th></tr><tr><td>Bankmarketing Train</td><td><a href=\"https://ml.azure.com/pipelines/5b46ca32-41ba-4c69-8606-cfdbffc06513?wsid=/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourcegroups/aml-quickstarts-207945/workspaces/quick-starts-ws-207945\" target=\"_blank\" rel=\"noopener\">5b46ca32-41ba-4c69-8606-cfdbffc06513</a></td><td>Active</td><td><a href=\"https://southcentralus.api.azureml.ms/pipelines/v1.0/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourceGroups/aml-quickstarts-207945/providers/Microsoft.MachineLearningServices/workspaces/quick-starts-ws-207945/PipelineRuns/PipelineSubmit/5b46ca32-41ba-4c69-8606-cfdbffc06513\" target=\"_blank\" rel=\"noopener\">REST Endpoint</a></td></tr></table>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 16,
      "metadata": {
        "gather": {
          "logged": 1663605390217
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Authenticate once again, to retrieve the `auth_header` so that the endpoint can be used"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.core.authentication import InteractiveLoginAuthentication\n",
        "\n",
        "interactive_auth = InteractiveLoginAuthentication()\n",
        "auth_header = interactive_auth.get_authentication_header()\n",
        "\n"
      ],
      "outputs": [],
      "execution_count": 17,
      "metadata": {
        "gather": {
          "logged": 1663605390291
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Get the REST url from the endpoint property of the published pipeline object. You can also find the REST url in your workspace in the portal. Build an HTTP POST request to the endpoint, specifying your authentication header. Additionally, add a JSON payload object with the experiment name and the batch size parameter. As a reminder, the process_count_per_node is passed through to ParallelRunStep because you defined it is defined as a PipelineParameter object in the step configuration.\n",
        "\n",
        "Make the request to trigger the run. Access the Id key from the response dict to get the value of the run id.\n"
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "import requests\n",
        "\n",
        "rest_endpoint = published_pipeline.endpoint\n",
        "response = requests.post(rest_endpoint, \n",
        "                         headers=auth_header, \n",
        "                         json={\"ExperimentName\": \"pipeline-rest-endpoint\"}\n",
        "                        )"
      ],
      "outputs": [],
      "execution_count": 18,
      "metadata": {
        "gather": {
          "logged": 1663605390812
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "try:\n",
        "    response.raise_for_status()\n",
        "except Exception:    \n",
        "    raise Exception(\"Received bad response from the endpoint: {}\\n\"\n",
        "                    \"Response Code: {}\\n\"\n",
        "                    \"Headers: {}\\n\"\n",
        "                    \"Content: {}\".format(rest_endpoint, response.status_code, response.headers, response.content))\n",
        "\n",
        "run_id = response.json().get('Id')\n",
        "print('Submitted pipeline run: ', run_id)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Submitted pipeline run:  09e492de-4c7d-428d-b22c-e7e8854fad4d\n"
        }
      ],
      "execution_count": 19,
      "metadata": {
        "gather": {
          "logged": 1663605390883
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Use the run id to monitor the status of the new run. This will take another 10-15 min to run and will look similar to the previous pipeline run, so if you don't need to see another pipeline run, you can skip watching the full output."
      ],
      "metadata": {}
    },
    {
      "cell_type": "code",
      "source": [
        "from azureml.pipeline.core.run import PipelineRun\n",
        "from azureml.widgets import RunDetails\n",
        "\n",
        "published_pipeline_run = PipelineRun(ws.experiments[\"pipeline-rest-endpoint\"], run_id)\n",
        "RunDetails(published_pipeline_run).show()"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": "_PipelineWidget(widget_settings={'childWidgetDisplay': 'popup', 'send_telemetry': False, 'log_level': 'INFO', …",
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "9396089b287f47dea52fb3881de607bb"
            }
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/aml.mini.widget.v1": "{\"status\": \"Completed\", \"workbench_run_details_uri\": \"https://ml.azure.com/runs/09e492de-4c7d-428d-b22c-e7e8854fad4d?wsid=/subscriptions/976ee174-3882-4721-b90a-b5fef6b72f24/resourcegroups/aml-quickstarts-207945/workspaces/quick-starts-ws-207945&tid=660b3398-b80e-49d2-bc5b-ac1dc93b5254\", \"run_id\": \"09e492de-4c7d-428d-b22c-e7e8854fad4d\", \"run_properties\": {\"run_id\": \"09e492de-4c7d-428d-b22c-e7e8854fad4d\", \"created_utc\": \"2022-09-19T16:36:30.365785Z\", \"properties\": {\"azureml.runsource\": \"azureml.PipelineRun\", \"runSource\": \"Unavailable\", \"runType\": \"HTTP\", \"azureml.parameters\": \"{}\", \"azureml.continue_on_step_failure\": \"False\", \"azureml.continue_on_failed_optional_input\": \"True\", \"azureml.pipelineid\": \"5b46ca32-41ba-4c69-8606-cfdbffc06513\", \"azureml.pipelineComponent\": \"pipelinerun\"}, \"tags\": {}, \"end_time_utc\": \"2022-09-19T16:36:32.168585Z\", \"status\": \"Completed\", \"log_files\": {\"logs/azureml/executionlogs.txt\": \"https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.09e492de-4c7d-428d-b22c-e7e8854fad4d/logs/azureml/executionlogs.txt?sv=2019-07-07&sr=b&sig=nqPqTxd8I3cNg21DFle2d9H%2F1z2QI3dwrjc6dL8PyoU%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A26%3A52Z&se=2022-09-20T00%3A36%3A52Z&sp=r\", \"logs/azureml/stderrlogs.txt\": \"https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.09e492de-4c7d-428d-b22c-e7e8854fad4d/logs/azureml/stderrlogs.txt?sv=2019-07-07&sr=b&sig=2kE64hPgMWYR7%2F9ihLEIpH6ITwox20zrciQM7GvkUQQ%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A26%3A52Z&se=2022-09-20T00%3A36%3A52Z&sp=r\", \"logs/azureml/stdoutlogs.txt\": \"https://mlstrg207945.blob.core.windows.net/azureml/ExperimentRun/dcid.09e492de-4c7d-428d-b22c-e7e8854fad4d/logs/azureml/stdoutlogs.txt?sv=2019-07-07&sr=b&sig=uEuHw%2F6JiwR0SVrCQfNPxrFa9VFKrjnHL1GYUWekVOI%3D&skoid=e377abeb-c66f-4f50-9a04-7d4a90612e25&sktid=660b3398-b80e-49d2-bc5b-ac1dc93b5254&skt=2022-09-19T15%3A36%3A55Z&ske=2022-09-20T23%3A46%3A55Z&sks=b&skv=2019-07-07&st=2022-09-19T16%3A26%3A52Z&se=2022-09-20T00%3A36%3A52Z&sp=r\"}, \"log_groups\": [[\"logs/azureml/executionlogs.txt\", \"logs/azureml/stderrlogs.txt\", \"logs/azureml/stdoutlogs.txt\"]], \"run_duration\": \"0:00:01\", \"run_number\": \"1663605390\", \"run_queued_details\": {\"status\": \"Finished\", \"details\": null}}, \"child_runs\": [{\"run_id\": \"52a3c831-d998-49f9-969e-c7e38ddee4f6\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2022-09-19T16:36:31.777891Z\", \"created_time\": \"2022-09-19T16:36:31.777891Z\", \"end_time\": \"2022-09-19T16:36:31.845776Z\", \"duration\": \"0:00:00\", \"run_number\": 1663605391, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2022-09-19T16:36:31.777891Z\", \"is_reused\": \"Yes\"}], \"children_metrics\": {\"categories\": null, \"series\": null, \"metricName\": null}, \"run_metrics\": [], \"run_logs\": \"[2022-09-19 16:36:31Z] Completing processing run id 52a3c831-d998-49f9-969e-c7e38ddee4f6.\\n[2022-09-19 16:36:32Z] Finishing experiment: no runs left and nothing to schedule.\\n\\nRun is completed.\", \"graph\": {\"datasource_nodes\": {\"6883c63a\": {\"node_id\": \"6883c63a\", \"name\": \"bankmarketing-dataset\"}}, \"module_nodes\": {\"0a1d2e75\": {\"node_id\": \"0a1d2e75\", \"name\": \"automl_module\", \"status\": \"Finished\", \"_is_reused\": true, \"run_id\": \"52a3c831-d998-49f9-969e-c7e38ddee4f6\"}}, \"edges\": [{\"source_node_id\": \"6883c63a\", \"source_node_name\": \"bankmarketing-dataset\", \"source_name\": \"data\", \"target_name\": \"training_data\", \"dst_node_id\": \"0a1d2e75\", \"dst_node_name\": \"automl_module\"}], \"child_runs\": [{\"run_id\": \"52a3c831-d998-49f9-969e-c7e38ddee4f6\", \"name\": \"automl_module\", \"status\": \"Finished\", \"start_time\": \"2022-09-19T16:36:31.777891Z\", \"created_time\": \"2022-09-19T16:36:31.777891Z\", \"end_time\": \"2022-09-19T16:36:31.845776Z\", \"duration\": \"0:00:00\", \"run_number\": 1663605391, \"metric\": null, \"run_type\": \"azureml.StepRun\", \"training_percent\": null, \"created_time_dt\": \"2022-09-19T16:36:31.777891Z\", \"is_reused\": \"Yes\"}]}, \"widget_settings\": {\"childWidgetDisplay\": \"popup\", \"send_telemetry\": false, \"log_level\": \"INFO\", \"sdk_version\": \"1.44.0\"}, \"loading\": false}"
          },
          "metadata": {}
        }
      ],
      "execution_count": 20,
      "metadata": {
        "gather": {
          "logged": 1663605391151
        }
      }
    },
    {
      "cell_type": "code",
      "source": [],
      "outputs": [],
      "execution_count": null,
      "metadata": {}
    }
  ],
  "metadata": {
    "order_index": 11,
    "exclude_from_index": false,
    "task": "Demonstrates the use of AutoMLStep",
    "deployment": [
      "None"
    ],
    "authors": [
      {
        "name": "sanpil"
      }
    ],
    "vscode": {
      "interpreter": {
        "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
      }
    },
    "star_tag": [
      "featured"
    ],
    "kernel_info": {
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.9"
    },
    "compute": [
      "AML Compute"
    ],
    "kernelspec": {
      "display_name": "Python 3.8.9 64-bit",
      "language": "python",
      "name": "python3"
    },
    "tags": [
      "None"
    ],
    "datasets": [
      "Custom"
    ],
    "categories": [
      "how-to-use-azureml",
      "machine-learning-pipelines",
      "intro-to-pipelines"
    ],
    "category": "tutorial",
    "framework": [
      "Automated Machine Learning"
    ],
    "friendly_name": "How to use AutoMLStep with AML Pipelines",
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}