KW_CPC_pipeline_training_deployment.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "## Pipeline training and deployment"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "After having developed a machine learning model through with an exploratory approach in the previous notebook, the next step is to register the model to put it into production by deploying it on an online deployment. \n",
        "\n",
        "Although the previously developed model could have been logged via MLFlow and then registered and deployed, are trained again making use of pipeline jobs. Therefore, data preparation and training scripts are written to create yaml pipeline components that complete the training process in sequence. \n",
        "\n",
        "Finally the model logged by MLFlow during the pipeline job is registered and deployed to 100% traffic on an Azure managed online deployment."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "#### Import libraries"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import os\n",
        "from azure.ai.ml import MLClient\n",
        "from azure.identity import DefaultAzureCredential\n",
        "from azure.ai.ml import load_component\n",
        "from azure.ai.ml import Input\n",
        "from azure.ai.ml.constants import AssetTypes\n",
        "from azure.ai.ml.dsl import pipeline\n",
        "from azure.ai.ml.entities import AmlCompute"
      ],
      "outputs": [],
      "execution_count": 1,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717328741787
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Connect to the MLClient"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ml_client = MLClient(\n",
        "    DefaultAzureCredential(), \"e1f27d73-68d8-4f59-900b-77783d4c5b3b\", \"End2endCPC\", \"End2End_CPC\"\n",
        ")\n"
      ],
      "outputs": [],
      "execution_count": 1,
      "metadata": {
        "gather": {
          "logged": 1717238781863
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create a _src_ directory to store the generated data preparation and training scripts."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "script_folder = 'src'\n",
        "os.makedirs(script_folder, exist_ok=True)\n",
        "print(script_folder, 'folder created')"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "src folder created\n"
        }
      ],
      "execution_count": 2,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238782140
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Training via pipeline jobs"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Definition of scripts to prepare the data and train the model"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile src/prepare-data.py\n",
        "import argparse\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from pathlib import Path\n",
        "from sklearn.preprocessing import RobustScaler\n",
        "\n",
        "def main(args):\n",
        "    df = get_data(args.input_data)\n",
        "\n",
        "    cleaned_data = clean_data(df)\n",
        "\n",
        "    imputed_data = impute(cleaned_data)\n",
        "\n",
        "    scaled_data = scale_data(imputed_data)\n",
        "\n",
        "    output_df = scaled_data.to_csv((Path(args.output_data)), index = False)\n",
        "\n",
        "def get_data(path):\n",
        "    df = pd.read_csv(path)\n",
        "\n",
        "    print(f'Preparing {df.shape[1]} columns and {df.shape[0]} rows of data')\n",
        "\n",
        "    df = df.drop(\"keyword\",axis=1)\n",
        "    \n",
        "    return df\n",
        "\n",
        "def clean_data(df):\n",
        "    zero_mask = (df['lower_bid']==df['upper_bid'])|(df['lower_bid']==0)\n",
        "    df = df[~zero_mask]\n",
        "    lower_mask = (df['lower_bid']>df['upper_bid'])|(df['lower_bid']>df['CPC'])|(df['upper_bid']<df['CPC'])\n",
        "    df = df[~lower_mask]\n",
        "\n",
        "    return df\n",
        "\n",
        "def impute(df):\n",
        "    for column in df.columns:\n",
        "        if df[column].dtype in ['float64', 'int64']:\n",
        "            fill_value = df[column].median()\n",
        "        else:\n",
        "            fill_value = df[column].mode()[0]\n",
        "        \n",
        "        df[column].fillna(fill_value, inplace=True)\n",
        "    \n",
        "    return df\n",
        "\n",
        "def scale_data(df):\n",
        "    scaler = RobustScaler()\n",
        "    num_cols = df.select_dtypes(['float64', 'int64']).columns.to_list()\n",
        "    df[num_cols] = scaler.fit_transform(df[num_cols])\n",
        "\n",
        "    return df\n",
        "\n",
        "def parse_args():\n",
        "    parser = argparse.ArgumentParser()\n",
        "\n",
        "    parser.add_argument(\"--input_data\", dest='input_data',\n",
        "                        type=str)\n",
        "    parser.add_argument(\"--output_data\", dest='output_data',\n",
        "                        type=str)\n",
        "\n",
        "    args = parser.parse_args()\n",
        "\n",
        "    return args\n",
        "\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    print(\"\\n\\n\")\n",
        "    print(\"*\" * 60)\n",
        "\n",
        "    args = parse_args()\n",
        "\n",
        "    main(args)\n",
        "\n",
        "    print(\"*\" * 60)\n",
        "    print(\"\\n\\n\")"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Overwriting src/prepare-data.py\n"
        }
      ],
      "execution_count": 3,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile src/train-model.py\n",
        "import mlflow\n",
        "import glob\n",
        "import argparse\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.linear_model import LinearRegression\n",
        "from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor\n",
        "from sklearn.metrics import mean_absolute_error\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "def main(args):\n",
        "    mlflow.autolog()\n",
        "\n",
        "    df = get_data(args.training_data)\n",
        "\n",
        "    X_train, X_test, y_train, y_test = split_data(df,args.target_feature)\n",
        "\n",
        "    model = train_model(args.algorithm, X_train, X_test, y_train, y_test)\n",
        "\n",
        "    eval_model(model, X_test, y_test)\n",
        "\n",
        "def get_data(path):\n",
        "    df = pd.read_csv(path)\n",
        "\n",
        "    print(f'Modeling with {df.shape[1]} columns and {df.shape[0]} rows of data')\n",
        "    \n",
        "    return df\n",
        "\n",
        "def split_data(df,target_feature):\n",
        "    print(\"Splitting data...\")\n",
        "    X, y = df.drop(target_feature,axis=1), np.ravel(df[target_feature])\n",
        "\n",
        "    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, shuffle=True, random_state=99)\n",
        "\n",
        "    return X_train, X_test, y_train, y_test\n",
        "\n",
        "def train_model(algorithm,X_train, X_test, y_train, y_test):\n",
        "    print(\"Training model...\")\n",
        "    if algorithm == \"gradient-boosting\":\n",
        "        model = GradientBoostingRegressor()\n",
        "    if algorithm == \"random-forest\":\n",
        "        model = RandomForestRegressor()\n",
        "    else:\n",
        "        model = LinearRegression()\n",
        "    \n",
        "    model.fit(X_train, y_train)\n",
        "\n",
        "    mlflow.sklearn.save_model(model, args.model_output)\n",
        "\n",
        "    return model\n",
        "\n",
        "\n",
        "def eval_model(model, X_test, y_test):\n",
        "\n",
        "    y_pred = model.predict(X_test)\n",
        "    mae = mean_absolute_error(y_pred, y_test)\n",
        "    mlflow.log_param('MAE',mae)\n",
        "    print('MAE:', mae)\n",
        "\n",
        "\n",
        "def parse_args():\n",
        "    parser = argparse.ArgumentParser()\n",
        "\n",
        "    parser.add_argument(\"--training_data\", dest='training_data',\n",
        "                        type=str)\n",
        "    parser.add_argument(\"--algorithm\", dest='algorithm',\n",
        "                        type=str, default='linear-regression')\n",
        "    parser.add_argument(\"--target_feature\", dest='target_feature',\n",
        "                        type=str, default='CPC')\n",
        "    parser.add_argument(\"--model_output\", dest='model_output',\n",
        "                        type=str)\n",
        "\n",
        "    args = parser.parse_args()\n",
        "\n",
        "    return args\n",
        "\n",
        "if __name__ == \"__main__\":\n",
        "    print(\"\\n\\n\")\n",
        "    print(\"*\" * 60)\n",
        "\n",
        "    args = parse_args()\n",
        "\n",
        "    main(args)\n",
        "\n",
        "    print(\"*\" * 60)\n",
        "    print(\"\\n\\n\")\n"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Overwriting src/train-model.py\n"
        }
      ],
      "execution_count": 4,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Definition of pipeline components out of prepare and train scripts"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile prepare-data.yml\n",
        "$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json\n",
        "name: prep_data\n",
        "display_name: Prepare training data\n",
        "version: 1\n",
        "type: command\n",
        "inputs:\n",
        "  input_data: \n",
        "    type: uri_file\n",
        "outputs:\n",
        "  output_data:\n",
        "    type: uri_file\n",
        "code: ./src\n",
        "environment: azureml://registries/azureml/environments/sklearn-1.1/versions/34\n",
        "command: >-\n",
        "  python prepare-data.py \n",
        "  --input_data ${{inputs.input_data}}\n",
        "  --output_data ${{outputs.output_data}}"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Overwriting prepare-data.yml\n"
        }
      ],
      "execution_count": 5,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "%%writefile train-model.yml\n",
        "$schema: https://azuremlschemas.azureedge.net/latest/commandComponent.schema.json\n",
        "name: train_model\n",
        "display_name: Train a linear, gradient boosting or random forest regression model\n",
        "version: 1\n",
        "type: command\n",
        "inputs:\n",
        "  training_data: \n",
        "    type: uri_file\n",
        "  algorithm:\n",
        "    type: string\n",
        "    default: 'linear-regression'\n",
        "  target_feature:\n",
        "    type: string\n",
        "    default: 'CPC'\n",
        "outputs:\n",
        "  model_output:\n",
        "    type: mlflow_model\n",
        "code: ./src\n",
        "environment: azureml://registries/azureml/environments/sklearn-1.1/versions/34\n",
        "command: >-\n",
        "  python train-model.py \n",
        "  --training_data ${{inputs.training_data}} \n",
        "  --algorithm ${{inputs.algorithm}} \n",
        "  --target_feature ${{inputs.target_feature}} \n",
        "  --model_output ${{outputs.model_output}} "
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "Overwriting train-model.yml\n"
        }
      ],
      "execution_count": 6,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "##### Define, provision with compute and submit pipeline job"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Load components to use them as steps of the pipeline job."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prep_data = load_component(source= \"./prepare-data.yml\")\n",
        "train_regression = load_component(source=\"./train-model.yml\")"
      ],
      "outputs": [],
      "execution_count": 7,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238784326
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Define the pipeline job. It is designed to accept different target features and a model of choice between Linear, Random Forest and Gradient Boosting Regression."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "@pipeline()\n",
        "def kw_CPC_prediction(pipeline_job_input,algorithm='linear-regression',target_feature='CPC'):\n",
        "    clean_data = prep_data(input_data=pipeline_job_input)\n",
        "    train_model = train_regression(training_data=clean_data.outputs.output_data,algorithm=algorithm,target_feature=target_feature)\n",
        "\n",
        "    return {\n",
        "        \"pipeline_job_transformed_data\": clean_data.outputs.output_data,\n",
        "        \"pipeline_job_trained_model\": train_model.outputs.model_output,\n",
        "    }\n",
        "\n",
        "pipeline_job = kw_CPC_prediction(Input(type=AssetTypes.URI_FILE, path=\"azureml:kw-dataset:1\"),'random-forest')"
      ],
      "outputs": [],
      "execution_count": 8,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238784623
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Provision the compute cluster for the pipeline job."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "cpu_compute_target = \"aml-cluster2\"\n",
        "\n",
        "try:\n",
        "    cpu_cluster = ml_client.compute.get(cpu_compute_target)\n",
        "    print(\n",
        "        f\"There is already a cluster named {cpu_compute_target}. Reusing it.\"\n",
        "    )\n",
        "\n",
        "except Exception:\n",
        "    print(\"Creating a new cpu compute target...\")\n",
        "\n",
        "    cpu_cluster = AmlCompute(\n",
        "        name=cpu_compute_target,\n",
        "        type=\"amlcompute\",\n",
        "        size=\"Standard_D11_v2\",\n",
        "        min_instances=0,\n",
        "        max_instances=2,\n",
        "        idle_time_before_scale_down=60,\n",
        "        tier=\"Dedicated\",\n",
        "    )\n",
        "\n",
        "    cpu_cluster = ml_client.compute.begin_create_or_update(cpu_cluster)"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": "There is already a cluster named aml-cluster2. Reusing it.\n"
        }
      ],
      "execution_count": 10,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238785404
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Edit the pipeline job attribute values to make sure the output data and model upload to the workspace and use the provisioned compute cluster."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "\n",
        "pipeline_job.outputs.pipeline_job_transformed_data.mode = \"upload\"\n",
        "pipeline_job.outputs.pipeline_job_trained_model.mode = \"upload\"\n",
        "pipeline_job.settings.default_compute = \"aml-cluster2\"\n",
        "pipeline_job.settings.default_datastore = \"workspaceblobstore\""
      ],
      "outputs": [],
      "execution_count": 12,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238787022
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Submit the data for model training."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "pipeline_job = ml_client.jobs.create_or_update(\n",
        "    pipeline_job, experiment_name=\"pipeline_kw_CPC_prediction\"\n",
        ")\n",
        "pipeline_job"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": "pathOnCompute is not a known attribute of class <class 'azure.ai.ml._restclient.v2023_04_01_preview.models._models_py3.UriFileJobOutput'> and will be ignored\npathOnCompute is not a known attribute of class <class 'azure.ai.ml._restclient.v2023_04_01_preview.models._models_py3.MLFlowModelJobOutput'> and will be ignored\n"
        },
        {
          "output_type": "execute_result",
          "execution_count": 13,
          "data": {
            "text/plain": "PipelineJob({'inputs': {'pipeline_job_input': <azure.ai.ml.entities._job.pipeline._io.base.PipelineInput object at 0x7f3a0eff5730>, 'algorithm': <azure.ai.ml.entities._job.pipeline._io.base.PipelineInput object at 0x7f3a0eff5490>, 'target_feature': <azure.ai.ml.entities._job.pipeline._io.base.PipelineInput object at 0x7f3a0eef3a60>}, 'outputs': {'pipeline_job_transformed_data': <azure.ai.ml.entities._job.pipeline._io.base.PipelineOutput object at 0x7f3a0eef3580>, 'pipeline_job_trained_model': <azure.ai.ml.entities._job.pipeline._io.base.PipelineOutput object at 0x7f3a0eef3520>}, 'jobs': {}, 'component': PipelineComponent({'latest_version': None, 'intellectual_property': None, 'auto_increment_version': False, 'source': 'REMOTE.WORKSPACE.JOB', 'is_anonymous': True, 'auto_delete_setting': None, 'name': 'azureml_anonymous', 'description': None, 'tags': {}, 'properties': {}, 'print_as_yaml': False, 'id': None, 'Resource__source_path': None, 'base_path': '/mnt/batch/tasks/shared/LS_root/mounts/clusters/notebook-compute-cpc/code/Users/miguel_lopez_virues/azure_keyword_CPC', 'creation_context': None, 'serialize': <msrest.serialization.Serializer object at 0x7f3a0ef7f130>, 'version': '1', 'schema': None, 'type': 'pipeline', 'display_name': 'kw_CPC_prediction', 'is_deterministic': None, 'inputs': {'pipeline_job_input': {}, 'algorithm': {}, 'target_feature': {}}, 'outputs': {'pipeline_job_transformed_data': {}, 'pipeline_job_trained_model': {}}, 'yaml_str': None, 'other_parameter': {}, 'jobs': {'clean_data': Command({'parameters': {}, 'init': False, 'name': 'clean_data', 'type': 'command', 'status': None, 'log_files': None, 'description': None, 'tags': {}, 'properties': {}, 'print_as_yaml': False, 'id': None, 'Resource__source_path': '', 'base_path': '/mnt/batch/tasks/shared/LS_root/mounts/clusters/notebook-compute-cpc/code/Users/miguel_lopez_virues/azure_keyword_CPC', 'creation_context': None, 'serialize': <msrest.serialization.Serializer object at 0x7f3a0ee9aca0>, 'allowed_keys': {}, 'key_restriction': False, 'logger': <TraceLogger attr_dict (WARNING)>, 'display_name': None, 'experiment_name': None, 'compute': None, 'services': None, 'comment': None, 'job_inputs': {'input_data': '${{parent.inputs.pipeline_job_input}}'}, 'job_outputs': {'output_data': '${{parent.outputs.pipeline_job_transformed_data}}'}, 'inputs': {'input_data': <azure.ai.ml.entities._job.pipeline._io.base.NodeInput object at 0x7f3a0ef7fd30>}, 'outputs': {'output_data': <azure.ai.ml.entities._job.pipeline._io.base.NodeOutput object at 0x7f3a0ef7fa30>}, 'component': 'azureml_anonymous:727e2565-7275-4c37-8ada-379abb474752', 'referenced_control_flow_node_instance_id': None, 'kwargs': {'services': None}, 'instance_id': '23227056-e8d8-4d21-88d5-74dec0ba57e5', 'source': 'YAML.COMPONENT', 'validate_required_input_not_provided': True, 'limits': None, 'identity': None, 'distribution': None, 'environment_variables': {}, 'environment': None, 'resources': None, 'queue_settings': None, 'swept': False}), 'train_model': Command({'parameters': {}, 'init': False, 'name': 'train_model', 'type': 'command', 'status': None, 'log_files': None, 'description': None, 'tags': {}, 'properties': {}, 'print_as_yaml': False, 'id': None, 'Resource__source_path': '', 'base_path': '/mnt/batch/tasks/shared/LS_root/mounts/clusters/notebook-compute-cpc/code/Users/miguel_lopez_virues/azure_keyword_CPC', 'creation_context': None, 'serialize': <msrest.serialization.Serializer object at 0x7f3a0ef7f0a0>, 'allowed_keys': {}, 'key_restriction': False, 'logger': <TraceLogger attr_dict (WARNING)>, 'display_name': None, 'experiment_name': None, 'compute': None, 'services': None, 'comment': None, 'job_inputs': {'training_data': '${{parent.jobs.clean_data.outputs.output_data}}', 'algorithm': '${{parent.inputs.algorithm}}', 'target_feature': '${{parent.inputs.target_feature}}'}, 'job_outputs': {'model_output': '${{parent.outputs.pipeline_job_trained_model}}'}, 'inputs': {'training_data': <azure.ai.ml.entities._job.pipeline._io.base.NodeInput object at 0x7f3a0ef7f1f0>, 'algorithm': <azure.ai.ml.entities._job.pipeline._io.base.NodeInput object at 0x7f3a0ef7fbe0>, 'target_feature': <azure.ai.ml.entities._job.pipeline._io.base.NodeInput object at 0x7f3a0ef7f6a0>}, 'outputs': {'model_output': <azure.ai.ml.entities._job.pipeline._io.base.NodeOutput object at 0x7f3a0ef7fc70>}, 'component': 'azureml_anonymous:42265c5c-c4ba-4661-9c1b-47843a2c2bec', 'referenced_control_flow_node_instance_id': None, 'kwargs': {'services': None}, 'instance_id': '673b28c0-cc41-492a-951a-c89e63540b22', 'source': 'YAML.COMPONENT', 'validate_required_input_not_provided': True, 'limits': None, 'identity': None, 'distribution': None, 'environment_variables': {}, 'environment': None, 'resources': None, 'queue_settings': None, 'swept': False})}, 'job_types': {'command': 2}, 'job_sources': {'YAML.COMPONENT': 2}, 'source_job_id': None}), 'type': 'pipeline', 'status': 'NotStarted', 'log_files': None, 'name': 'bubbly_feast_ny2qlb9vj6', 'description': None, 'tags': {}, 'properties': {'mlflow.source.git.repoURL': 'https://github.com/MiguelLopezVirues/azure_keyword_CPC.git', 'mlflow.source.git.branch': 'main', 'mlflow.source.git.commit': '86e1c196fc004c86aa701461cd85a323c38eb8b8', 'azureml.git.dirty': 'True'}, 'print_as_yaml': False, 'id': '/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourceGroups/End2endCPC/providers/Microsoft.MachineLearningServices/workspaces/End2End_CPC/jobs/bubbly_feast_ny2qlb9vj6', 'Resource__source_path': '', 'base_path': '/mnt/batch/tasks/shared/LS_root/mounts/clusters/notebook-compute-cpc/code/Users/miguel_lopez_virues/azure_keyword_CPC', 'creation_context': <azure.ai.ml.entities._system_data.SystemData object at 0x7f3a0ef7fa00>, 'serialize': <msrest.serialization.Serializer object at 0x7f3a0eef30a0>, 'display_name': 'kw_CPC_prediction', 'experiment_name': 'pipeline_kw_CPC_prediction', 'compute': None, 'services': {'Tracking': {'endpoint': 'azureml://francecentral.api.azureml.ms/mlflow/v1.0/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourceGroups/End2endCPC/providers/Microsoft.MachineLearningServices/workspaces/End2End_CPC?', 'type': 'Tracking'}, 'Studio': {'endpoint': 'https://ml.azure.com/runs/bubbly_feast_ny2qlb9vj6?wsid=/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourcegroups/End2endCPC/workspaces/End2End_CPC&tid=0125b7aa-b6d6-4346-ad4d-12ed91eeca28', 'type': 'Studio'}}, 'settings': {}, 'identity': None, 'default_code': None, 'default_environment': None})",
            "text/html": "<table style=\"width:100%\"><tr><th>Experiment</th><th>Name</th><th>Type</th><th>Status</th><th>Details Page</th></tr><tr><td>pipeline_kw_CPC_prediction</td><td>bubbly_feast_ny2qlb9vj6</td><td>pipeline</td><td>NotStarted</td><td><a href=\"https://ml.azure.com/runs/bubbly_feast_ny2qlb9vj6?wsid=/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourcegroups/End2endCPC/workspaces/End2End_CPC&amp;tid=0125b7aa-b6d6-4346-ad4d-12ed91eeca28\" target=\"_blank\" rel=\"noopener\">Link to Azure Machine Learning studio</a></td></tr></table>"
          },
          "metadata": {}
        }
      ],
      "execution_count": 13,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238794305
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Model registration and deployment"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Register the model logged by MLFlow from the *train_data* child job inside the pipeline job. This model will be stored as a development version that can be modifided and continue to evolve."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azure.ai.ml.entities import Model\n",
        "from azure.ai.ml.constants import AssetTypes\n",
        "\n",
        "child_jobs = [child_job for child_job in ml_client.jobs.list(parent_job_name=pipeline_job.name)]\n",
        "child_job_name = child_jobs[1].name\n",
        "\n",
        "run_model = Model(\n",
        "    path=f\"azureml://jobs/{child_job_name}/outputs/artifacts/paths/model/\",\n",
        "    name=\"mlflow-kw-CPC\",\n",
        "    description=\"Model created from run.\",\n",
        "    type=AssetTypes.MLFLOW_MODEL,\n",
        ")\n",
        "\n",
        "ml_client.models.create_or_update(run_model)"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 15,
          "data": {
            "text/plain": "Model({'job_name': 'a0577f77-9fb1-433f-a19c-6e1dc735679a', 'intellectual_property': None, 'is_anonymous': False, 'auto_increment_version': False, 'auto_delete_setting': None, 'name': 'mlflow-kw-CPC', 'description': 'Model created from run.', 'tags': {}, 'properties': {}, 'print_as_yaml': False, 'id': '/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourceGroups/End2endCPC/providers/Microsoft.MachineLearningServices/workspaces/End2End_CPC/models/mlflow-kw-CPC/versions/11', 'Resource__source_path': '', 'base_path': '/mnt/batch/tasks/shared/LS_root/mounts/clusters/notebook-compute-cpc/code/Users/miguel_lopez_virues/azure_keyword_CPC', 'creation_context': <azure.ai.ml.entities._system_data.SystemData object at 0x7f3a0eccc5b0>, 'serialize': <msrest.serialization.Serializer object at 0x7f3a0ece85e0>, 'version': '11', 'latest_version': None, 'path': 'azureml://subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourceGroups/End2endCPC/workspaces/End2End_CPC/datastores/workspaceartifactstore/paths/ExperimentRun/dcid.1ff00290-6c78-4fbd-a47e-cf757ab8fbe0/model', 'datastore': None, 'utc_time_created': None, 'flavors': {'python_function': {'env': '{\\n  \"conda\": \"conda.yaml\",\\n  \"virtualenv\": \"python_env.yaml\"\\n}', 'loader_module': 'mlflow.sklearn', 'model_path': 'model.pkl', 'predict_fn': 'predict', 'python_version': '3.9.18'}, 'sklearn': {'code': '', 'pickled_model': 'model.pkl', 'serialization_format': 'cloudpickle', 'sklearn_version': '1.1.3'}}, 'arm_type': 'model_version', 'type': 'mlflow_model', 'stage': 'Development'})"
          },
          "metadata": {}
        }
      ],
      "execution_count": 15,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238834889
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Create the endpoint to deploy the model on."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from azure.ai.ml.entities import ManagedOnlineEndpoint, ManagedOnlineDeployment\n",
        "from azure.ai.ml.constants import AssetTypes\n",
        "import datetime\n",
        "\n",
        "online_endpoint_name = \"endpoint-CPC-prediction\"\n",
        "\n",
        "endpoint = ManagedOnlineEndpoint(\n",
        "    name=online_endpoint_name,\n",
        "    description=\"Online endpoint for MLflow keyword CPC prediction.\",\n",
        "    auth_mode=\"key\",\n",
        ")\n"
      ],
      "outputs": [],
      "execution_count": 16,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238835942
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Register a production version of the logged model that will be kept operational for real-time inference to predict CPCs. Then define the Azure managed online deployment that will receive the 100% of the traffic from the endpoint, until a new model is developed to be deployed in a green deployment before fully replacing the current one."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "production_model = Model(\n",
        "    path=f\"azureml://jobs/{child_job_name}/outputs/artifacts/paths/model/\",\n",
        "    name=\"mlflow-kw-CPC-blue-deployment\",\n",
        "    description=\"Model created from run.\",\n",
        "    type=AssetTypes.MLFLOW_MODEL,\n",
        ")\n",
        "\n",
        "blue_deployment = ManagedOnlineDeployment(\n",
        "    name=\"blue-kw-cpc\",\n",
        "    endpoint_name=online_endpoint_name,\n",
        "    description=\"Blue online deployment for keyword CPC prediction\",\n",
        "    model=production_model,\n",
        "    instance_type=\"Standard_F4s_v2\",\n",
        "    instance_count=1,\n",
        ")"
      ],
      "outputs": [],
      "execution_count": 17,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238836286
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ml_client.begin_create_or_update(endpoint).result()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 18,
          "data": {
            "text/plain": "ManagedOnlineEndpoint({'public_network_access': 'Enabled', 'provisioning_state': 'Succeeded', 'scoring_uri': 'https://endpoint-cpc-prediction.francecentral.inference.ml.azure.com/score', 'openapi_uri': 'https://endpoint-cpc-prediction.francecentral.inference.ml.azure.com/swagger.json', 'name': 'endpoint-cpc-prediction', 'description': 'Online endpoint for MLflow keyword CPC prediction.', 'tags': {}, 'properties': {'createdBy': 'miguel_lopez_virues', 'createdAt': '2024-05-31T10:55:45.535660+0000', 'lastModifiedAt': '2024-05-31T10:55:45.535660+0000', 'azureml.onlineendpointid': '/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourcegroups/end2endcpc/providers/microsoft.machinelearningservices/workspaces/end2end_cpc/onlineendpoints/endpoint-cpc-prediction', 'AzureAsyncOperationUri': 'https://management.azure.com/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/providers/Microsoft.MachineLearningServices/locations/francecentral/mfeOperationsStatus/oe:5fc20ec8-dbaf-49b8-a9c2-e6072f40b106:5a6d5a24-48c7-4230-b4e0-4887a558c469?api-version=2022-02-01-preview'}, 'print_as_yaml': False, 'id': '/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourceGroups/End2endCPC/providers/Microsoft.MachineLearningServices/workspaces/End2End_CPC/onlineEndpoints/endpoint-cpc-prediction', 'Resource__source_path': '', 'base_path': '/mnt/batch/tasks/shared/LS_root/mounts/clusters/notebook-compute-cpc/code/Users/miguel_lopez_virues/azure_keyword_CPC', 'creation_context': None, 'serialize': <msrest.serialization.Serializer object at 0x7f3a39354f70>, 'auth_mode': 'key', 'location': 'francecentral', 'identity': <azure.ai.ml.entities._credentials.IdentityConfiguration object at 0x7f3a244dac40>, 'traffic': {}, 'mirror_traffic': {}, 'kind': 'Managed'})"
          },
          "metadata": {}
        }
      ],
      "execution_count": 18,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717238870528
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "ml_client.online_deployments.begin_create_or_update(blue_deployment).result()"
      ],
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": "Check: endpoint endpoint-CPC-prediction exists\n"
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": ".............................................................................."
        },
        {
          "output_type": "execute_result",
          "execution_count": 19,
          "data": {
            "text/plain": "ManagedOnlineDeployment({'private_network_connection': None, 'package_model': False, 'provisioning_state': 'Succeeded', 'endpoint_name': 'endpoint-cpc-prediction', 'type': 'Managed', 'name': 'blue-kw-cpc', 'description': 'Blue online deployment for keyword CPC prediction', 'tags': {}, 'properties': {}, 'print_as_yaml': False, 'id': '/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourceGroups/End2endCPC/providers/Microsoft.MachineLearningServices/workspaces/End2End_CPC/onlineEndpoints/endpoint-cpc-prediction/deployments/blue-kw-cpc', 'Resource__source_path': '', 'base_path': '/mnt/batch/tasks/shared/LS_root/mounts/clusters/notebook-compute-cpc/code/Users/miguel_lopez_virues/azure_keyword_CPC', 'creation_context': None, 'serialize': <msrest.serialization.Serializer object at 0x7f3a0efe47c0>, 'model': '/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourceGroups/End2endCPC/providers/Microsoft.MachineLearningServices/workspaces/End2End_CPC/models/mlflow-kw-CPC-blue-deployment/versions/8', 'code_configuration': None, 'environment': '/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourceGroups/End2endCPC/providers/Microsoft.MachineLearningServices/workspaces/End2End_CPC/environments/DefaultNcdEnv-mlflow-ubuntu20-04-py38-cpu-inference/versions/20240510v6', 'environment_variables': {'MLFLOW_MODEL_FOLDER': 'model', 'AZUREML_EXTRA_CONDA_YAML_ABS_PATH': '/var/azureml-app/azureml-models/mlflow-kw-CPC-blue-deployment/8/model/conda.yaml', 'AZUREML_MODEL_DIR': '/var/azureml-app/azureml-models/mlflow-kw-CPC-blue-deployment/8'}, 'app_insights_enabled': False, 'scale_settings': <azure.ai.ml.entities._deployment.scale_settings.DefaultScaleSettings object at 0x7f3a0ec82100>, 'request_settings': <azure.ai.ml.entities._deployment.deployment_settings.OnlineRequestSettings object at 0x7f3a0ef8aee0>, 'liveness_probe': <azure.ai.ml.entities._deployment.deployment_settings.ProbeSettings object at 0x7f3a0ee93d30>, 'readiness_probe': <azure.ai.ml.entities._deployment.deployment_settings.ProbeSettings object at 0x7f3a0eed9700>, 'instance_count': 1, 'arm_type': 'online_deployment', 'model_mount_path': None, 'instance_type': 'Standard_F4s_v2', 'data_collector': None, 'egress_public_network_access': 'Enabled'})"
          },
          "metadata": {}
        }
      ],
      "execution_count": 19,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717239279905
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "endpoint.traffic = {\"blue-kw-cpc\": 100}\n",
        "ml_client.begin_create_or_update(endpoint).result()"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 20,
          "data": {
            "text/plain": "ManagedOnlineEndpoint({'public_network_access': 'Enabled', 'provisioning_state': 'Succeeded', 'scoring_uri': 'https://endpoint-cpc-prediction.francecentral.inference.ml.azure.com/score', 'openapi_uri': 'https://endpoint-cpc-prediction.francecentral.inference.ml.azure.com/swagger.json', 'name': 'endpoint-cpc-prediction', 'description': 'Online endpoint for MLflow keyword CPC prediction.', 'tags': {}, 'properties': {'createdBy': 'miguel_lopez_virues', 'createdAt': '2024-05-31T10:55:45.535660+0000', 'lastModifiedAt': '2024-05-31T10:55:45.535660+0000', 'azureml.onlineendpointid': '/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourcegroups/end2endcpc/providers/microsoft.machinelearningservices/workspaces/end2end_cpc/onlineendpoints/endpoint-cpc-prediction', 'AzureAsyncOperationUri': 'https://management.azure.com/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/providers/Microsoft.MachineLearningServices/locations/francecentral/mfeOperationsStatus/oe:5fc20ec8-dbaf-49b8-a9c2-e6072f40b106:414fda46-3a3d-4bce-84ac-65c98c58a829?api-version=2022-02-01-preview'}, 'print_as_yaml': False, 'id': '/subscriptions/e1f27d73-68d8-4f59-900b-77783d4c5b3b/resourceGroups/End2endCPC/providers/Microsoft.MachineLearningServices/workspaces/End2End_CPC/onlineEndpoints/endpoint-cpc-prediction', 'Resource__source_path': '', 'base_path': '/mnt/batch/tasks/shared/LS_root/mounts/clusters/notebook-compute-cpc/code/Users/miguel_lopez_virues/azure_keyword_CPC', 'creation_context': None, 'serialize': <msrest.serialization.Serializer object at 0x7f3a0eea47c0>, 'auth_mode': 'key', 'location': 'francecentral', 'identity': <azure.ai.ml.entities._credentials.IdentityConfiguration object at 0x7f3a0eaf3790>, 'traffic': {'blue-kw-cpc': 100}, 'mirror_traffic': {}, 'kind': 'Managed'})"
          },
          "metadata": {}
        }
      ],
      "execution_count": 20,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717239313702
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Sample data to test-invoke the deployed model"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import json \n",
        "sample_deployment_data = {\n",
        "  \"input_data\": {\n",
        "    \"columns\": [\n",
        "      \"competition\",\n",
        "      \"lower_bid\",\n",
        "      \"upper_bid\"\n",
        "    ],\n",
        "    \"index\": [1],\n",
        "    \"data\": [\n",
        "      [\n",
        "      30,2.1,5\n",
        "    ]\n",
        "    ]\n",
        "  }\n",
        "}\n",
        "\n",
        "sample_data_path = \"data/sample_deployment_data.json\"\n",
        "\n",
        "with open(sample_data_path, 'w') as json_file:\n",
        "    json.dump(sample_deployment_data, json_file)\n",
        "\n",
        "response = ml_client.online_endpoints.invoke(\n",
        "    endpoint_name=online_endpoint_name,\n",
        "    deployment_name=\"blue-kw-cpc\",\n",
        "    request_file=sample_data_path,\n",
        ")\n",
        "response"
      ],
      "outputs": [
        {
          "output_type": "execute_result",
          "execution_count": 21,
          "data": {
            "text/plain": "'[2.5990833333333323]'"
          },
          "metadata": {}
        }
      ],
      "execution_count": 21,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "gather": {
          "logged": 1717239314675
        }
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python38-azureml",
      "language": "python",
      "display_name": "Python 3.8 - AzureML"
    },
    "language_info": {
      "name": "python",
      "version": "3.8.5",
      "mimetype": "text/x-python",
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "pygments_lexer": "ipython3",
      "nbconvert_exporter": "python",
      "file_extension": ".py"
    },
    "kernel_info": {
      "name": "python38-azureml"
    },
    "microsoft": {
      "ms_spell_check": {
        "ms_spell_check_language": "en"
      },
      "host": {
        "AzureML": {
          "notebookHasBeenCompleted": true
        }
      }
    },
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}