AI4H_practical3b.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iru8mbpme2Ch"
      },
      "source": [
        "# Sklearn model building example\n",
        "\n",
        "Sklearn comes with many more functionalities than just training the parameters of a model. The main benefit of sklearn is actually in organizing the workflow *around* training a model. Think of functionalities such as splitting the data or cross-validation, but also preprocessing of data and the use of pipelines. \n",
        "\n",
        "Training the parameters of a machine learning model is not difficult. But ensuring that the process is done in such a way that your results give you the best possible representation of how well your model will perform for the next instance in your process can go wrong in many, many different ways.\n",
        "\n",
        "In this notebook we go through a pipeline example, where some fundamental functionalities of sklearn are unpacked, small-step by small-step, such that you will understand far better what is happening under the hood when training models using sklearn.\n",
        "\n",
        "Needless to say, there is much, much more to sklearn than shown below. Please do visit https://scikit-learn.org/stable/ to further investigate all the possibilities of this important package."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "C0J35f56OUDl"
      },
      "source": [
        "As always, we start with importing the relevant modules."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 85,
      "metadata": {
        "id": "2qB--SMu81rW"
      },
      "outputs": [],
      "source": [
        "#import relevant modules\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "#from sklearn.model_selection import cross_validate\n",
        "from sklearn.model_selection import StratifiedShuffleSplit\n",
        "from sklearn.model_selection import RepeatedKFold\n",
        "from sklearn.model_selection import GridSearchCV\n",
        "\n",
        "#from sklearn.metrics import plot_confusion_matrix\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from sklearn.metrics import ConfusionMatrixDisplay\n",
        "from sklearn.metrics import accuracy_score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WMQVPxDXjEzo"
      },
      "source": [
        "## Loading the Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "70vpmsAFO8Nj"
      },
      "source": [
        "We work with a toy dataset, containing sonar data which can be used to classify objects as being either a Mine or a Rock. More information on the dataset can be found at: https://datahub.io/machine-learning/sonar."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 86,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        },
        "id": "EtEXZweE9NPD",
        "outputId": "bfa56861-dd12-4aa8-bf14-cb2ac4367bfd"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>attribute_1</th>\n",
              "      <th>attribute_2</th>\n",
              "      <th>attribute_3</th>\n",
              "      <th>attribute_4</th>\n",
              "      <th>attribute_5</th>\n",
              "      <th>attribute_6</th>\n",
              "      <th>attribute_7</th>\n",
              "      <th>attribute_8</th>\n",
              "      <th>attribute_9</th>\n",
              "      <th>attribute_10</th>\n",
              "      <th>...</th>\n",
              "      <th>attribute_52</th>\n",
              "      <th>attribute_53</th>\n",
              "      <th>attribute_54</th>\n",
              "      <th>attribute_55</th>\n",
              "      <th>attribute_56</th>\n",
              "      <th>attribute_57</th>\n",
              "      <th>attribute_58</th>\n",
              "      <th>attribute_59</th>\n",
              "      <th>attribute_60</th>\n",
              "      <th>Class</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.0200</td>\n",
              "      <td>0.0371</td>\n",
              "      <td>0.0428</td>\n",
              "      <td>0.0207</td>\n",
              "      <td>0.0954</td>\n",
              "      <td>0.0986</td>\n",
              "      <td>0.1539</td>\n",
              "      <td>0.1601</td>\n",
              "      <td>0.3109</td>\n",
              "      <td>0.2111</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0027</td>\n",
              "      <td>0.0065</td>\n",
              "      <td>0.0159</td>\n",
              "      <td>0.0072</td>\n",
              "      <td>0.0167</td>\n",
              "      <td>0.0180</td>\n",
              "      <td>0.0084</td>\n",
              "      <td>0.0090</td>\n",
              "      <td>0.0032</td>\n",
              "      <td>Rock</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.0453</td>\n",
              "      <td>0.0523</td>\n",
              "      <td>0.0843</td>\n",
              "      <td>0.0689</td>\n",
              "      <td>0.1183</td>\n",
              "      <td>0.2583</td>\n",
              "      <td>0.2156</td>\n",
              "      <td>0.3481</td>\n",
              "      <td>0.3337</td>\n",
              "      <td>0.2872</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0084</td>\n",
              "      <td>0.0089</td>\n",
              "      <td>0.0048</td>\n",
              "      <td>0.0094</td>\n",
              "      <td>0.0191</td>\n",
              "      <td>0.0140</td>\n",
              "      <td>0.0049</td>\n",
              "      <td>0.0052</td>\n",
              "      <td>0.0044</td>\n",
              "      <td>Rock</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.0262</td>\n",
              "      <td>0.0582</td>\n",
              "      <td>0.1099</td>\n",
              "      <td>0.1083</td>\n",
              "      <td>0.0974</td>\n",
              "      <td>0.2280</td>\n",
              "      <td>0.2431</td>\n",
              "      <td>0.3771</td>\n",
              "      <td>0.5598</td>\n",
              "      <td>0.6194</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0232</td>\n",
              "      <td>0.0166</td>\n",
              "      <td>0.0095</td>\n",
              "      <td>0.0180</td>\n",
              "      <td>0.0244</td>\n",
              "      <td>0.0316</td>\n",
              "      <td>0.0164</td>\n",
              "      <td>0.0095</td>\n",
              "      <td>0.0078</td>\n",
              "      <td>Rock</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.0100</td>\n",
              "      <td>0.0171</td>\n",
              "      <td>0.0623</td>\n",
              "      <td>0.0205</td>\n",
              "      <td>0.0205</td>\n",
              "      <td>0.0368</td>\n",
              "      <td>0.1098</td>\n",
              "      <td>0.1276</td>\n",
              "      <td>0.0598</td>\n",
              "      <td>0.1264</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0121</td>\n",
              "      <td>0.0036</td>\n",
              "      <td>0.0150</td>\n",
              "      <td>0.0085</td>\n",
              "      <td>0.0073</td>\n",
              "      <td>0.0050</td>\n",
              "      <td>0.0044</td>\n",
              "      <td>0.0040</td>\n",
              "      <td>0.0117</td>\n",
              "      <td>Rock</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.0762</td>\n",
              "      <td>0.0666</td>\n",
              "      <td>0.0481</td>\n",
              "      <td>0.0394</td>\n",
              "      <td>0.0590</td>\n",
              "      <td>0.0649</td>\n",
              "      <td>0.1209</td>\n",
              "      <td>0.2467</td>\n",
              "      <td>0.3564</td>\n",
              "      <td>0.4459</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0031</td>\n",
              "      <td>0.0054</td>\n",
              "      <td>0.0105</td>\n",
              "      <td>0.0110</td>\n",
              "      <td>0.0015</td>\n",
              "      <td>0.0072</td>\n",
              "      <td>0.0048</td>\n",
              "      <td>0.0107</td>\n",
              "      <td>0.0094</td>\n",
              "      <td>Rock</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 61 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   attribute_1  attribute_2  attribute_3  attribute_4  attribute_5  \\\n",
              "0       0.0200       0.0371       0.0428       0.0207       0.0954   \n",
              "1       0.0453       0.0523       0.0843       0.0689       0.1183   \n",
              "2       0.0262       0.0582       0.1099       0.1083       0.0974   \n",
              "3       0.0100       0.0171       0.0623       0.0205       0.0205   \n",
              "4       0.0762       0.0666       0.0481       0.0394       0.0590   \n",
              "\n",
              "   attribute_6  attribute_7  attribute_8  attribute_9  attribute_10  ...  \\\n",
              "0       0.0986       0.1539       0.1601       0.3109        0.2111  ...   \n",
              "1       0.2583       0.2156       0.3481       0.3337        0.2872  ...   \n",
              "2       0.2280       0.2431       0.3771       0.5598        0.6194  ...   \n",
              "3       0.0368       0.1098       0.1276       0.0598        0.1264  ...   \n",
              "4       0.0649       0.1209       0.2467       0.3564        0.4459  ...   \n",
              "\n",
              "   attribute_52  attribute_53  attribute_54  attribute_55  attribute_56  \\\n",
              "0        0.0027        0.0065        0.0159        0.0072        0.0167   \n",
              "1        0.0084        0.0089        0.0048        0.0094        0.0191   \n",
              "2        0.0232        0.0166        0.0095        0.0180        0.0244   \n",
              "3        0.0121        0.0036        0.0150        0.0085        0.0073   \n",
              "4        0.0031        0.0054        0.0105        0.0110        0.0015   \n",
              "\n",
              "   attribute_57  attribute_58  attribute_59  attribute_60  Class  \n",
              "0        0.0180        0.0084        0.0090        0.0032   Rock  \n",
              "1        0.0140        0.0049        0.0052        0.0044   Rock  \n",
              "2        0.0316        0.0164        0.0095        0.0078   Rock  \n",
              "3        0.0050        0.0044        0.0040        0.0117   Rock  \n",
              "4        0.0072        0.0048        0.0107        0.0094   Rock  \n",
              "\n",
              "[5 rows x 61 columns]"
            ]
          },
          "execution_count": 86,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df = pd.read_csv('https://datahub.io/machine-learning/sonar/r/sonar.csv')\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 87,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "g-NnpFZ7Eu9p",
        "outputId": "5e30df67-77e2-4c99-8ac4-c33fd5bc9af9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "RangeIndex: 208 entries, 0 to 207\n",
            "Data columns (total 61 columns):\n",
            " #   Column        Non-Null Count  Dtype  \n",
            "---  ------        --------------  -----  \n",
            " 0   attribute_1   208 non-null    float64\n",
            " 1   attribute_2   208 non-null    float64\n",
            " 2   attribute_3   208 non-null    float64\n",
            " 3   attribute_4   208 non-null    float64\n",
            " 4   attribute_5   208 non-null    float64\n",
            " 5   attribute_6   208 non-null    float64\n",
            " 6   attribute_7   208 non-null    float64\n",
            " 7   attribute_8   208 non-null    float64\n",
            " 8   attribute_9   208 non-null    float64\n",
            " 9   attribute_10  208 non-null    float64\n",
            " 10  attribute_11  208 non-null    float64\n",
            " 11  attribute_12  208 non-null    float64\n",
            " 12  attribute_13  208 non-null    float64\n",
            " 13  attribute_14  208 non-null    float64\n",
            " 14  attribute_15  208 non-null    float64\n",
            " 15  attribute_16  208 non-null    float64\n",
            " 16  attribute_17  208 non-null    float64\n",
            " 17  attribute_18  208 non-null    float64\n",
            " 18  attribute_19  208 non-null    float64\n",
            " 19  attribute_20  208 non-null    float64\n",
            " 20  attribute_21  208 non-null    float64\n",
            " 21  attribute_22  208 non-null    float64\n",
            " 22  attribute_23  208 non-null    float64\n",
            " 23  attribute_24  208 non-null    float64\n",
            " 24  attribute_25  208 non-null    float64\n",
            " 25  attribute_26  208 non-null    float64\n",
            " 26  attribute_27  208 non-null    float64\n",
            " 27  attribute_28  208 non-null    float64\n",
            " 28  attribute_29  208 non-null    float64\n",
            " 29  attribute_30  208 non-null    float64\n",
            " 30  attribute_31  208 non-null    float64\n",
            " 31  attribute_32  208 non-null    float64\n",
            " 32  attribute_33  208 non-null    float64\n",
            " 33  attribute_34  208 non-null    float64\n",
            " 34  attribute_35  208 non-null    float64\n",
            " 35  attribute_36  208 non-null    float64\n",
            " 36  attribute_37  208 non-null    float64\n",
            " 37  attribute_38  208 non-null    float64\n",
            " 38  attribute_39  208 non-null    float64\n",
            " 39  attribute_40  208 non-null    float64\n",
            " 40  attribute_41  208 non-null    float64\n",
            " 41  attribute_42  208 non-null    float64\n",
            " 42  attribute_43  208 non-null    float64\n",
            " 43  attribute_44  208 non-null    float64\n",
            " 44  attribute_45  208 non-null    float64\n",
            " 45  attribute_46  208 non-null    float64\n",
            " 46  attribute_47  208 non-null    float64\n",
            " 47  attribute_48  208 non-null    float64\n",
            " 48  attribute_49  208 non-null    float64\n",
            " 49  attribute_50  208 non-null    float64\n",
            " 50  attribute_51  208 non-null    float64\n",
            " 51  attribute_52  208 non-null    float64\n",
            " 52  attribute_53  208 non-null    float64\n",
            " 53  attribute_54  208 non-null    float64\n",
            " 54  attribute_55  208 non-null    float64\n",
            " 55  attribute_56  208 non-null    float64\n",
            " 56  attribute_57  208 non-null    float64\n",
            " 57  attribute_58  208 non-null    float64\n",
            " 58  attribute_59  208 non-null    float64\n",
            " 59  attribute_60  208 non-null    float64\n",
            " 60  Class         208 non-null    object \n",
            "dtypes: float64(60), object(1)\n",
            "memory usage: 99.3+ KB\n"
          ]
        }
      ],
      "source": [
        "df.info()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fGa7Cpl0Dj1N"
      },
      "source": [
        "As we have no categorical features in our data (only the outcome variable Class is categorical), we add two categorical features to our data, such that we can demonstrate the functionalities relating to this data type."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JQRrVCwKPoLC"
      },
      "source": [
        "### Question 1\n",
        "- add a 'batch' feature to the dataframe, containing the levels 'first' (60 rows), 'second' (60 rows), 'third' (remaining rows).\n",
        "- add a 'location' feature to the dataframe, containing the levels 'loc_A' (20 rows), 'loc_B' (30 rows), 'loc_C' (40 rows), 'loc_D' (50 rows), 'loc_E' (remaining rows)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 88,
      "metadata": {},
      "outputs": [],
      "source": [
        "# add a 'batch' feature to the dataframe, containing the levels 'first' (60 rows), 'second' (60 rows), 'third' (remaining rows).\n",
        "df['batch'] = pd.cut(np.arange(len(df)), bins=[0, 60, 120, len(df)], labels=['first', 'second', 'third'])\n",
        "# add a 'location' feature to the dataframe, containing the levels 'loc_A' (20 rows), 'loc_B' (30 rows), 'loc_C' (40 rows), 'loc_D' (50 rows), 'loc_E' (remaining rows).\n",
        "df['location'] = pd.cut(np.arange(len(df)), bins=[0, 20, 50, 90, 140, len(df)], labels=['loc_A', 'loc_B', 'loc_C', 'loc_D', 'loc_E'])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cNq-OW3fhdhR"
      },
      "source": [
        "As this dataset does not have any missings, we will introduce some random missings such that we are forced to make use of more sklearn preprocessing functionalities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "k1gh5cTuQhWA"
      },
      "source": [
        "### Question 2\n",
        "Replace a random 10% of the dataframe by missing values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 89,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[False, False, False, ..., False, False, False],\n",
              "       [False, False, False, ..., False, False, False],\n",
              "       [False, False,  True, ..., False, False, False],\n",
              "       ...,\n",
              "       [False, False, False, ..., False,  True,  True],\n",
              "       [False, False, False, ..., False, False, False],\n",
              "       [False, False, False, ..., False, False, False]])"
            ]
          },
          "execution_count": 89,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# answer Q2\n",
        "np.random.seed(42)\n",
        "mask = np.random.choice([True, False], p=[0.1, 0.9], size=df.shape)\n",
        "mask"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 90,
      "metadata": {},
      "outputs": [],
      "source": [
        "# # replace a random 10% of the dataframe by missing value.\n",
        "\n",
        "# df.loc[df.sample(frac=0.1).index, :] = np.nan\n",
        "#df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 91,
      "metadata": {},
      "outputs": [],
      "source": [
        "# replace the missing values by the most occuring value in the dataframe.\n",
        "#df = df.fillna(df.mode().iloc[0])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 92,
      "metadata": {},
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>attribute_1</th>\n",
              "      <th>attribute_2</th>\n",
              "      <th>attribute_3</th>\n",
              "      <th>attribute_4</th>\n",
              "      <th>attribute_5</th>\n",
              "      <th>attribute_6</th>\n",
              "      <th>attribute_7</th>\n",
              "      <th>attribute_8</th>\n",
              "      <th>attribute_9</th>\n",
              "      <th>attribute_10</th>\n",
              "      <th>...</th>\n",
              "      <th>attribute_54</th>\n",
              "      <th>attribute_55</th>\n",
              "      <th>attribute_56</th>\n",
              "      <th>attribute_57</th>\n",
              "      <th>attribute_58</th>\n",
              "      <th>attribute_59</th>\n",
              "      <th>attribute_60</th>\n",
              "      <th>Class</th>\n",
              "      <th>batch</th>\n",
              "      <th>location</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.0200</td>\n",
              "      <td>0.0371</td>\n",
              "      <td>0.0428</td>\n",
              "      <td>0.0207</td>\n",
              "      <td>0.0954</td>\n",
              "      <td>0.0986</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.1601</td>\n",
              "      <td>0.3109</td>\n",
              "      <td>0.2111</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0159</td>\n",
              "      <td>0.0072</td>\n",
              "      <td>0.0167</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0084</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0032</td>\n",
              "      <td>Rock</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.0453</td>\n",
              "      <td>0.0523</td>\n",
              "      <td>0.0843</td>\n",
              "      <td>0.0689</td>\n",
              "      <td>0.1183</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.2156</td>\n",
              "      <td>0.3481</td>\n",
              "      <td>0.3337</td>\n",
              "      <td>NaN</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0048</td>\n",
              "      <td>0.0094</td>\n",
              "      <td>0.0191</td>\n",
              "      <td>0.0140</td>\n",
              "      <td>0.0049</td>\n",
              "      <td>0.0052</td>\n",
              "      <td>0.0044</td>\n",
              "      <td>Rock</td>\n",
              "      <td>first</td>\n",
              "      <td>loc_A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.0262</td>\n",
              "      <td>0.0582</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.1083</td>\n",
              "      <td>0.0974</td>\n",
              "      <td>0.2280</td>\n",
              "      <td>0.2431</td>\n",
              "      <td>0.3771</td>\n",
              "      <td>0.5598</td>\n",
              "      <td>0.6194</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0095</td>\n",
              "      <td>0.0180</td>\n",
              "      <td>0.0244</td>\n",
              "      <td>0.0316</td>\n",
              "      <td>0.0164</td>\n",
              "      <td>0.0095</td>\n",
              "      <td>0.0078</td>\n",
              "      <td>Rock</td>\n",
              "      <td>first</td>\n",
              "      <td>loc_A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.0100</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0623</td>\n",
              "      <td>0.0205</td>\n",
              "      <td>0.0205</td>\n",
              "      <td>0.0368</td>\n",
              "      <td>0.1098</td>\n",
              "      <td>0.1276</td>\n",
              "      <td>0.0598</td>\n",
              "      <td>0.1264</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0150</td>\n",
              "      <td>0.0085</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.0050</td>\n",
              "      <td>0.0044</td>\n",
              "      <td>0.0040</td>\n",
              "      <td>0.0117</td>\n",
              "      <td>Rock</td>\n",
              "      <td>first</td>\n",
              "      <td>loc_A</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.0762</td>\n",
              "      <td>0.0666</td>\n",
              "      <td>0.0481</td>\n",
              "      <td>0.0394</td>\n",
              "      <td>0.0590</td>\n",
              "      <td>0.0649</td>\n",
              "      <td>0.1209</td>\n",
              "      <td>NaN</td>\n",
              "      <td>0.3564</td>\n",
              "      <td>0.4459</td>\n",
              "      <td>...</td>\n",
              "      <td>0.0105</td>\n",
              "      <td>0.0110</td>\n",
              "      <td>0.0015</td>\n",
              "      <td>0.0072</td>\n",
              "      <td>0.0048</td>\n",
              "      <td>0.0107</td>\n",
              "      <td>NaN</td>\n",
              "      <td>NaN</td>\n",
              "      <td>first</td>\n",
              "      <td>loc_A</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 63 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   attribute_1  attribute_2  attribute_3  attribute_4  attribute_5  \\\n",
              "0       0.0200       0.0371       0.0428       0.0207       0.0954   \n",
              "1       0.0453       0.0523       0.0843       0.0689       0.1183   \n",
              "2       0.0262       0.0582          NaN       0.1083       0.0974   \n",
              "3       0.0100          NaN       0.0623       0.0205       0.0205   \n",
              "4       0.0762       0.0666       0.0481       0.0394       0.0590   \n",
              "\n",
              "   attribute_6  attribute_7  attribute_8  attribute_9  attribute_10  ...  \\\n",
              "0       0.0986          NaN       0.1601       0.3109        0.2111  ...   \n",
              "1          NaN       0.2156       0.3481       0.3337           NaN  ...   \n",
              "2       0.2280       0.2431       0.3771       0.5598        0.6194  ...   \n",
              "3       0.0368       0.1098       0.1276       0.0598        0.1264  ...   \n",
              "4       0.0649       0.1209          NaN       0.3564        0.4459  ...   \n",
              "\n",
              "   attribute_54  attribute_55  attribute_56  attribute_57  attribute_58  \\\n",
              "0        0.0159        0.0072        0.0167           NaN        0.0084   \n",
              "1        0.0048        0.0094        0.0191        0.0140        0.0049   \n",
              "2        0.0095        0.0180        0.0244        0.0316        0.0164   \n",
              "3        0.0150        0.0085           NaN        0.0050        0.0044   \n",
              "4        0.0105        0.0110        0.0015        0.0072        0.0048   \n",
              "\n",
              "   attribute_59  attribute_60  Class  batch  location  \n",
              "0           NaN        0.0032   Rock    NaN       NaN  \n",
              "1        0.0052        0.0044   Rock  first     loc_A  \n",
              "2        0.0095        0.0078   Rock  first     loc_A  \n",
              "3        0.0040        0.0117   Rock  first     loc_A  \n",
              "4        0.0107           NaN    NaN  first     loc_A  \n",
              "\n",
              "[5 rows x 63 columns]"
            ]
          },
          "execution_count": 92,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# answer Q2 continued\n",
        "# replace original df:\n",
        "df = df.mask(mask)\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Kp4yf_SIjN8L"
      },
      "source": [
        "The next step is to split the data into a random train (70%) and test (30%) set, such that we can perform model selection in the next section on the train set, and finally perform model assessment on the test set."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 93,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 388
        },
        "id": "iDRA7xvD_HUT",
        "outputId": "95cdd3ea-ff09-4991-aa83-c6e58987ade8"
      },
      "outputs": [
        {
          "ename": "ValueError",
          "evalue": "Input contains NaN",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[1;32m/Users/junda_huang/Work/radboudumc_projects/ai4h_course/AI4H_practical3b.ipynb Cell 18\u001b[0m line \u001b[0;36m2\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/junda_huang/Work/radboudumc_projects/ai4h_course/AI4H_practical3b.ipynb#X15sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m split \u001b[39m=\u001b[39m StratifiedShuffleSplit(n_splits\u001b[39m=\u001b[39m\u001b[39m1\u001b[39m, test_size\u001b[39m=\u001b[39m\u001b[39m0.3\u001b[39m, random_state\u001b[39m=\u001b[39m\u001b[39m42\u001b[39m)\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/junda_huang/Work/radboudumc_projects/ai4h_course/AI4H_practical3b.ipynb#X15sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m \u001b[39mfor\u001b[39;00m train_index, test_index \u001b[39min\u001b[39;00m split\u001b[39m.\u001b[39msplit(df, df[\u001b[39m'\u001b[39m\u001b[39mClass\u001b[39m\u001b[39m'\u001b[39m]):\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/junda_huang/Work/radboudumc_projects/ai4h_course/AI4H_practical3b.ipynb#X15sZmlsZQ%3D%3D?line=2'>3</a>\u001b[0m   strat_train_set \u001b[39m=\u001b[39m df\u001b[39m.\u001b[39mloc[train_index]\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/junda_huang/Work/radboudumc_projects/ai4h_course/AI4H_practical3b.ipynb#X15sZmlsZQ%3D%3D?line=3'>4</a>\u001b[0m   strat_test_set \u001b[39m=\u001b[39m df\u001b[39m.\u001b[39mloc[test_index]\n",
            "File \u001b[0;32m~/anaconda3/envs/ai4h_dl/lib/python3.11/site-packages/sklearn/model_selection/_split.py:2160\u001b[0m, in \u001b[0;36mStratifiedShuffleSplit.split\u001b[0;34m(self, X, y, groups)\u001b[0m\n\u001b[1;32m   2126\u001b[0m \u001b[39mdef\u001b[39;00m \u001b[39msplit\u001b[39m(\u001b[39mself\u001b[39m, X, y, groups\u001b[39m=\u001b[39m\u001b[39mNone\u001b[39;00m):\n\u001b[1;32m   2127\u001b[0m \u001b[39m    \u001b[39m\u001b[39m\"\"\"Generate indices to split data into training and test set.\u001b[39;00m\n\u001b[1;32m   2128\u001b[0m \n\u001b[1;32m   2129\u001b[0m \u001b[39m    Parameters\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m   2158\u001b[0m \u001b[39m    to an integer.\u001b[39;00m\n\u001b[1;32m   2159\u001b[0m \u001b[39m    \"\"\"\u001b[39;00m\n\u001b[0;32m-> 2160\u001b[0m     y \u001b[39m=\u001b[39m check_array(y, input_name\u001b[39m=\u001b[39m\u001b[39m\"\u001b[39m\u001b[39my\u001b[39m\u001b[39m\"\u001b[39m, ensure_2d\u001b[39m=\u001b[39m\u001b[39mFalse\u001b[39;00m, dtype\u001b[39m=\u001b[39m\u001b[39mNone\u001b[39;00m)\n\u001b[1;32m   2161\u001b[0m     \u001b[39mreturn\u001b[39;00m \u001b[39msuper\u001b[39m()\u001b[39m.\u001b[39msplit(X, y, groups)\n",
            "File \u001b[0;32m~/anaconda3/envs/ai4h_dl/lib/python3.11/site-packages/sklearn/utils/validation.py:921\u001b[0m, in \u001b[0;36mcheck_array\u001b[0;34m(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)\u001b[0m\n\u001b[1;32m    915\u001b[0m         \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\n\u001b[1;32m    916\u001b[0m             \u001b[39m\"\u001b[39m\u001b[39mFound array with dim \u001b[39m\u001b[39m%d\u001b[39;00m\u001b[39m. \u001b[39m\u001b[39m%s\u001b[39;00m\u001b[39m expected <= 2.\u001b[39m\u001b[39m\"\u001b[39m\n\u001b[1;32m    917\u001b[0m             \u001b[39m%\u001b[39m (array\u001b[39m.\u001b[39mndim, estimator_name)\n\u001b[1;32m    918\u001b[0m         )\n\u001b[1;32m    920\u001b[0m     \u001b[39mif\u001b[39;00m force_all_finite:\n\u001b[0;32m--> 921\u001b[0m         _assert_all_finite(\n\u001b[1;32m    922\u001b[0m             array,\n\u001b[1;32m    923\u001b[0m             input_name\u001b[39m=\u001b[39minput_name,\n\u001b[1;32m    924\u001b[0m             estimator_name\u001b[39m=\u001b[39mestimator_name,\n\u001b[1;32m    925\u001b[0m             allow_nan\u001b[39m=\u001b[39mforce_all_finite \u001b[39m==\u001b[39m \u001b[39m\"\u001b[39m\u001b[39mallow-nan\u001b[39m\u001b[39m\"\u001b[39m,\n\u001b[1;32m    926\u001b[0m         )\n\u001b[1;32m    928\u001b[0m \u001b[39mif\u001b[39;00m ensure_min_samples \u001b[39m>\u001b[39m \u001b[39m0\u001b[39m:\n\u001b[1;32m    929\u001b[0m     n_samples \u001b[39m=\u001b[39m _num_samples(array)\n",
            "File \u001b[0;32m~/anaconda3/envs/ai4h_dl/lib/python3.11/site-packages/sklearn/utils/validation.py:111\u001b[0m, in \u001b[0;36m_assert_all_finite\u001b[0;34m(X, allow_nan, msg_dtype, estimator_name, input_name)\u001b[0m\n\u001b[1;32m    109\u001b[0m \u001b[39mif\u001b[39;00m X\u001b[39m.\u001b[39mdtype \u001b[39m==\u001b[39m np\u001b[39m.\u001b[39mdtype(\u001b[39m\"\u001b[39m\u001b[39mobject\u001b[39m\u001b[39m\"\u001b[39m) \u001b[39mand\u001b[39;00m \u001b[39mnot\u001b[39;00m allow_nan:\n\u001b[1;32m    110\u001b[0m     \u001b[39mif\u001b[39;00m _object_dtype_isnan(X)\u001b[39m.\u001b[39many():\n\u001b[0;32m--> 111\u001b[0m         \u001b[39mraise\u001b[39;00m \u001b[39mValueError\u001b[39;00m(\u001b[39m\"\u001b[39m\u001b[39mInput contains NaN\u001b[39m\u001b[39m\"\u001b[39m)\n\u001b[1;32m    113\u001b[0m \u001b[39m# We need only consider float arrays, hence can early return for all else.\u001b[39;00m\n\u001b[1;32m    114\u001b[0m \u001b[39mif\u001b[39;00m X\u001b[39m.\u001b[39mdtype\u001b[39m.\u001b[39mkind \u001b[39mnot\u001b[39;00m \u001b[39min\u001b[39;00m \u001b[39m\"\u001b[39m\u001b[39mfc\u001b[39m\u001b[39m\"\u001b[39m:\n",
            "\u001b[0;31mValueError\u001b[0m: Input contains NaN"
          ]
        }
      ],
      "source": [
        "split = StratifiedShuffleSplit(n_splits=1, test_size=0.3, random_state=42)\n",
        "for train_index, test_index in split.split(df, df['Class']):\n",
        "  strat_train_set = df.loc[train_index]\n",
        "  strat_test_set = df.loc[test_index]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BGRVlbzbnceO"
      },
      "source": [
        "This is not working due to missings in the outcome variable, which causes StratifiedShuffleSplit to fail, because we ask it to perform a split where the distribution of our outcome variable Class is comparable across both the train and test set. And when the outcome variable Class contains missings, that procedure fails.\n",
        "\n",
        "This puts us in a Catch 22: we cannot impute the outcome variable using only the training data on outcomes, as our training data has to be defined based on our outcome variable. Please note this will not be a problem when the split between train and test data is not done randomly, but rather based on time, or location. In our case, however, it is a problem, and this is typically the point where you check whether the missing outcome data can be retrieved through other means, where you decide to drop missings, or decide to impute the outcome data. \n",
        "\n",
        "For now we will impute the missing *outcome* values with the most occurring value, and use the preprocessing functionality of sklearn for this task. We will use this functionality again after splitting the data to impute missings in the *predictors*, as this will ensure that we do the exact same preprocessing on the predictors in the test set."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7Wm9pxTaWX6d"
      },
      "source": [
        "### Question 3\n",
        "Missings in the outcome variable are more problematic than missings in the predictor variables. \n",
        "- Why is this the case? \n",
        "- What are the potential downsides of imputing the outcome variable?\n",
        "- What are the potential downsides of removing instances with missing outcome data?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bXydj9EuXR_s"
      },
      "source": [
        "For now we take a simple approach and impute the missing outcome values with the most frequent class."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 94,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "TTpgvSAHpgGE",
        "outputId": "40a8b23b-1c51-441b-8656-874a7466d6f7"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style>#sk-container-id-10 {color: black;background-color: white;}#sk-container-id-10 pre{padding: 0;}#sk-container-id-10 div.sk-toggleable {background-color: white;}#sk-container-id-10 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-10 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-10 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-10 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-10 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-10 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-10 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-10 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-10 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-10 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-10 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-10 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-10 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-10 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-10 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-10 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-10 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-10 div.sk-item {position: relative;z-index: 1;}#sk-container-id-10 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-10 div.sk-item::before, #sk-container-id-10 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-10 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-10 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-10 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-10 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-10 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-10 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-10 div.sk-label-container {text-align: center;}#sk-container-id-10 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-10 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-10\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SimpleImputer(strategy=&#x27;most_frequent&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" checked><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;most_frequent&#x27;)</pre></div></div></div></div></div>"
            ],
            "text/plain": [
              "SimpleImputer(strategy='most_frequent')"
            ]
          },
          "execution_count": 94,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from sklearn.impute import SimpleImputer\n",
        "imp_outcome = SimpleImputer(strategy = 'most_frequent')\n",
        "imp_outcome.fit(df['Class'].to_frame())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MrG_BLhQvv9I"
      },
      "source": [
        "The code above created an 'impute object' that has been fitted on df['Class'] to determine what the imputation value should be. The outcome Class has not been 'transformed' yet, and so it will still contain missing values."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 95,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J7hTPYk9wTAb",
        "outputId": "a1742010-c0e6-474b-aa6a-cf83e08010d2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "15"
            ]
          },
          "execution_count": 95,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "sum(df['Class'].isnull())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LuUL39-Ho5kD"
      },
      "source": [
        "The value that will be imputed when using the impute object to transform a variable, is saved in *.statistics_*, as shown below (and equals 'Mine', which is indeed the most_frequent value in df['Class'])."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 96,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "cBLqHSvLuqc9",
        "outputId": "b69a327b-3f4b-4b03-d5a3-cfeaaf87d04b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array(['Mine'], dtype=object)"
            ]
          },
          "execution_count": 96,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "imp_outcome.statistics_"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3tzc3DEgqJ9Y"
      },
      "source": [
        "The current distribution of df['Class'] is:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 97,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gDO7YbX4qIqq",
        "outputId": "77cdfd0b-9c8d-4729-d1c7-052291df861e"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Class\n",
              "Mine    101\n",
              "Rock     92\n",
              "Name: count, dtype: int64"
            ]
          },
          "execution_count": 97,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df['Class'].value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rMp1CLAxwF9B"
      },
      "source": [
        "We now impute the missing values of df['Class'] by applying 'transform' on df['Class']."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 100,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4yhQtbMMwd0n",
        "outputId": "c6d419da-9098-47f2-f2e4-823a40199984"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0"
            ]
          },
          "execution_count": 100,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# We now impute the missing values of df['Class'] by applying 'transform' on df['Class'].to_frame().\n",
        "\n",
        "df['Class'] = list(imp_outcome.transform(df['Class'].to_frame()))\n",
        "sum(df['Class'].isnull())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 101,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UQODfNSnql2l",
        "outputId": "17190480-d8b5-40cd-90e7-83e76d36052a"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "Class\n",
              "[Mine]    116\n",
              "[Rock]     92\n",
              "Name: count, dtype: int64"
            ]
          },
          "execution_count": 101,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "df['Class'].value_counts()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "euxwb1lpws8h"
      },
      "source": [
        "We could have done both the fitting and transforming in one go using the command imp_outcome.fit_transform().\n",
        "\n",
        "We are now ready to proceed with splitting the data into a train and test set"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 102,
      "metadata": {
        "id": "7LlSGNPzw0t0"
      },
      "outputs": [],
      "source": [
        "split = StratifiedShuffleSplit(n_splits=1, test_size=0.3, random_state=42)\n",
        "for train_index, test_index in split.split(df, df['Class']):\n",
        "  strat_train_set = df.loc[train_index]\n",
        "  strat_test_set = df.loc[test_index]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 103,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D5t8ibImAlV3",
        "outputId": "3084dc1b-eeaf-4fd0-9fa0-6b9901bc3b6e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "share of observations in the train set: 0.697\n",
            "share of observations in the test set: 0.303\n",
            "share of mines in the train set: 0.559\n",
            "share of mines in the test set: 0.556\n"
          ]
        }
      ],
      "source": [
        "print('share of observations in the train set:', str(round(len(strat_train_set)/len(df), 3)))\n",
        "print('share of observations in the test set:', str(round(len(strat_test_set)/len(df), 3)))\n",
        "print('share of mines in the train set:', str(round(len(strat_train_set.loc[strat_train_set['Class'] == 'Mine'])/len(strat_train_set), 3)))\n",
        "print('share of mines in the test set:', str(round(len(strat_test_set.loc[strat_test_set['Class'] == 'Mine'])/len(strat_test_set), 3)))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hLEYothCqyTt"
      },
      "source": [
        "We now split the features from the outcome data for both the train and the test set, to align with the input needed when using sklearn to train models."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 104,
      "metadata": {
        "id": "SSqWxbi24D5v"
      },
      "outputs": [],
      "source": [
        "# split features and outcome data\n",
        "X_train = strat_train_set.loc[:,strat_train_set.columns != 'Class']\n",
        "y_train = strat_train_set['Class']\n",
        "X_test = strat_test_set.loc[:,strat_test_set.columns != 'Class']\n",
        "y_test = strat_test_set['Class']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wASW59mlxHa2"
      },
      "source": [
        "## Setting up the pipeline\n",
        "\n",
        "Machine learning is about trial and error, with typically many iterations between different ways to prepare the data (data preparation) and assessing how that translates into performance (model building).\n",
        "\n",
        "Going through different iterations should require the least possible effort, which is done by setting up a pipeline, using sklearn's pipeline functionality. A pipeline explicitly operationalizes all the different steps to get from raw data into data prepared and ready to serve as modeling input.\n",
        "\n",
        "\n",
        "As such, the pipeline serves multiple purposes: \n",
        "- it encompasses every preprocessing step, coded within an overarching Python object, such that new data is really easy to preprocess consistently (the raw test data or tomorrow's raw process data). This approach prevents 'manual' adjustments to the data, that would get lost or would need to be repeated 'manually' when preparing new data; \n",
        "- it makes it really easy to tweak the preprocessing steps, trying out different ways to prepare the data, requiring minimal adjustment to your code (less time-consuming and less error-prone);\n",
        "- it helps in preventing mistakes that contribute to data leakage.\n",
        "\n",
        "Typical preprocessing steps captured in a pipeline include:\n",
        "- selecting the variables\n",
        "- imputing missing values\n",
        "- scaling numeric variables\n",
        "- creating dummy variables for categorical variables\n",
        "\n",
        "On top of that, there is full flexibility in defining your own preprocessing steps within a pipeline.\n",
        "\n",
        "Below, we will set up the typical pipeline step-by-step. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sZHs9z1IHwi7"
      },
      "source": [
        "The first thing to realize is that numeric and categorical features require different preprocessing steps, effectively meaning that our overarching pipeline should contain two parallel pipelines: one for preparing numeric features and one for preparing categorical features"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 105,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "KZS8TNN8Im9V",
        "outputId": "83bdd588-9faa-4d8c-df84-d2078dac57e0"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Index(['attribute_1', 'attribute_2', 'attribute_3', 'attribute_4',\n",
            "       'attribute_5', 'attribute_6', 'attribute_7', 'attribute_8',\n",
            "       'attribute_9', 'attribute_10', 'attribute_11', 'attribute_12',\n",
            "       'attribute_13', 'attribute_14', 'attribute_15', 'attribute_16',\n",
            "       'attribute_17', 'attribute_18', 'attribute_19', 'attribute_20',\n",
            "       'attribute_21', 'attribute_22', 'attribute_23', 'attribute_24',\n",
            "       'attribute_25', 'attribute_26', 'attribute_27', 'attribute_28',\n",
            "       'attribute_29', 'attribute_30', 'attribute_31', 'attribute_32',\n",
            "       'attribute_33', 'attribute_34', 'attribute_35', 'attribute_36',\n",
            "       'attribute_37', 'attribute_38', 'attribute_39', 'attribute_40',\n",
            "       'attribute_41', 'attribute_42', 'attribute_43', 'attribute_44',\n",
            "       'attribute_45', 'attribute_46', 'attribute_47', 'attribute_48',\n",
            "       'attribute_49', 'attribute_50', 'attribute_51', 'attribute_52',\n",
            "       'attribute_53', 'attribute_54', 'attribute_55', 'attribute_56',\n",
            "       'attribute_57', 'attribute_58', 'attribute_59', 'attribute_60'],\n",
            "      dtype='object')\n",
            "Index(['batch', 'location'], dtype='object')\n"
          ]
        }
      ],
      "source": [
        "# dividing the features into numeric and non-numeric\n",
        "num_features = X_train.select_dtypes(include=['float']).columns\n",
        "cat_features = X_train.select_dtypes(exclude=['float']).columns\n",
        "print(num_features)\n",
        "print(cat_features)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jMVKvJXELD41"
      },
      "source": [
        "We will now define a 'numeric pipeline' and a 'categorical pipeline', which is then combined into a 'full pipeline'.\n",
        "\n",
        "We start off with the 'numeric pipeline', containing the steps of imputing missing values, and standardizing the feature values (which is required for some machine learning algorithms, and speeds up estimation for others). As the imputation strategy within the numeric pipeline only applies to numeric features, imputation can be straightforwardly applied using median imputation (or comparable methods)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 106,
      "metadata": {
        "id": "37CxIKPVMT5B"
      },
      "outputs": [],
      "source": [
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "\n",
        "num_pipeline = Pipeline([\n",
        "        ('imputer', SimpleImputer(strategy=\"median\")),\n",
        "        ('std_scaler', StandardScaler()),\n",
        "    ])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iMF_u1Ce0_C1"
      },
      "source": [
        "### Question 4\n",
        "- make a copy of the dataframe (df_tmp = df.copy())\n",
        "- fit the SimpleImputer based on median values on the numeric features\n",
        "- fit a StandardScaler on the numeric features\n",
        "- check the fitted statistics of both objects"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 112,
      "metadata": {},
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[0.02285 0.0308  0.0344  0.0445  0.0597  0.08885 0.10695 0.1119  0.15375\n",
            " 0.185   0.23275 0.2555  0.2656  0.2821  0.2885  0.2956  0.31    0.36535\n",
            " 0.4416  0.5448  0.6177  0.6689  0.6987  0.6954  0.7221  0.7511  0.7552\n",
            " 0.7319  0.679   0.6152  0.492   0.4289  0.39375 0.3776  0.3032  0.3043\n",
            " 0.2934  0.315   0.2736  0.2811  0.258   0.2546  0.22255 0.17715 0.1459\n",
            " 0.12435 0.09905 0.0787  0.0447  0.01765 0.0139  0.01165 0.00935 0.0093\n",
            " 0.0075  0.0068  0.00575 0.00595 0.0067  0.0051 ] [0.02901442 0.03817452 0.04409615 0.0547625  0.07111394 0.09957692\n",
            " 0.12141827 0.13184712 0.17734423 0.20928077 0.23982788 0.25472067\n",
            " 0.27541683 0.29697644 0.32289423 0.36310913 0.40949904 0.43909471\n",
            " 0.50156346 0.56171971 0.60879471 0.62839712 0.65385673 0.67745721\n",
            " 0.68026635 0.70160817 0.71018894 0.69731106 0.64153173 0.588825\n",
            " 0.50417788 0.43817163 0.41610048 0.40676442 0.37043173 0.37388558\n",
            " 0.35344038 0.34165385 0.31812837 0.31172067 0.28438798 0.282525\n",
            " 0.24519038 0.20934087 0.18307981 0.15932837 0.11932596 0.09141058\n",
            " 0.05182788 0.01972981 0.01604375 0.01346923 0.01039327 0.0106851\n",
            " 0.00913606 0.0081149  0.00760385 0.00791683 0.0080851  0.00583558]\n"
          ]
        }
      ],
      "source": [
        "df_tmp =df.copy()\n",
        "\n",
        "# fit the SimpleImputer based on median values on the numeric features\n",
        "num_pipeline.fit(df_tmp[num_features])\n",
        "\n",
        "# fit a standerd scaler on the numeric features\n",
        "num_pipeline.fit_transform(df_tmp[num_features])\n",
        "\n",
        "# check the fitted statistics of both the imputer and the scaler\n",
        "print(num_pipeline['imputer'].statistics_,\n",
        "num_pipeline['std_scaler'].mean_)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4WUPiNCQiNSg"
      },
      "source": [
        "### Unpacking the pipeline\n",
        "\n",
        "Let's now unpack this part of our pipeline:\n",
        "num_pipeline is now a Pipeline object, which, as you would expect, simply passes on the input it receives through each of the elements of the pipeline. Each element of the Pipeline uses the output of the previous step as its input.\n",
        "\n",
        "Each element is given a user-defined name, and is accompanied by a specific preprocessing step, which is often a function imported from the sklearn.preprocessing module, but could also be a user-defined function. Each element in the Pipeline can be accessed individually, as shown below:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 113,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EAEEojV_jOsT",
        "outputId": "35a7a479-5a4c-4762-fe4a-f8387a0dfd3b"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "sklearn.pipeline.Pipeline"
            ]
          },
          "execution_count": 113,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "type(num_pipeline)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 114,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uRm6zA5GjSL3",
        "outputId": "af264853-0fe6-4292-98cb-1a7c8599d362"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style>#sk-container-id-13 {color: black;background-color: white;}#sk-container-id-13 pre{padding: 0;}#sk-container-id-13 div.sk-toggleable {background-color: white;}#sk-container-id-13 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-13 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-13 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-13 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-13 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-13 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-13 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-13 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-13 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-13 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-13 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-13 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-13 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-13 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-13 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-13 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-13 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-13 div.sk-item {position: relative;z-index: 1;}#sk-container-id-13 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-13 div.sk-item::before, #sk-container-id-13 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-13 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-13 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-13 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-13 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-13 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-13 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-13 div.sk-label-container {text-align: center;}#sk-container-id-13 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-13 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-13\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;imputer&#x27;, SimpleImputer(strategy=&#x27;median&#x27;)),\n",
              "                (&#x27;std_scaler&#x27;, StandardScaler())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;imputer&#x27;, SimpleImputer(strategy=&#x27;median&#x27;)),\n",
              "                (&#x27;std_scaler&#x27;, StandardScaler())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div>"
            ],
            "text/plain": [
              "Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),\n",
              "                ('std_scaler', StandardScaler())])"
            ]
          },
          "execution_count": 114,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "num_pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tX9gCmSHjXwc"
      },
      "source": [
        "Each functionality can be accessed individually as follows:\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 115,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "tSPircvQkApB",
        "outputId": "3138b23d-815e-4525-9512-60ede761d126"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style>#sk-container-id-14 {color: black;background-color: white;}#sk-container-id-14 pre{padding: 0;}#sk-container-id-14 div.sk-toggleable {background-color: white;}#sk-container-id-14 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-14 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-14 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-14 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-14 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-14 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-14 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-14 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-14 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-14 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-14 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-14 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-14 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-14 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-14 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-14 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-14 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-14 div.sk-item {position: relative;z-index: 1;}#sk-container-id-14 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-14 div.sk-item::before, #sk-container-id-14 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-14 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-14 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-14 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-14 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-14 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-14 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-14 div.sk-label-container {text-align: center;}#sk-container-id-14 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-14 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-14\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" checked><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div></div></div>"
            ],
            "text/plain": [
              "SimpleImputer(strategy='median')"
            ]
          },
          "execution_count": 115,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "num_pipeline['imputer']"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rg88owwjkMAX"
      },
      "source": [
        "Just as we saw above, this preprocessing functionality can be used to transform data, but it needs to be fitted on data first (or it can be done simultaneously). Once we fit the num_pipeline to our numerical training data, it will contain medians to be imputed and scaling factors to be applied, making it possible to use the pipeline for transforming data according to these preprocessing steps."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 116,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 352
        },
        "id": "gncOglbakGLs",
        "outputId": "eebe528f-280f-4f05-cb9e-26186edd5d19"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[ 1.04489737,  0.56458621,  2.07492148, ...,  0.37860039,\n",
              "         4.70397147,  3.85943531],\n",
              "       [-0.08924375, -0.68136411, -0.26658366, ...,  0.2356228 ,\n",
              "         0.40118938,  0.89350032],\n",
              "       [-0.14240662,  0.15965236,  0.28278998, ..., -0.25685558,\n",
              "        -0.54575494, -0.21550146],\n",
              "       ...,\n",
              "       [-0.27088354, -0.03969969, -1.06930923, ..., -0.79699316,\n",
              "         0.38457632, -0.18971072],\n",
              "       [-0.74048885, -1.12056159, -0.6532787 , ..., -0.33628758,\n",
              "        -0.562368  ,  0.32610406],\n",
              "       [-0.88668673, -0.99596656, -0.25858308, ..., -0.41571958,\n",
              "         1.79668627, -0.18971072]])"
            ]
          },
          "execution_count": 116,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# transforming will give an error, until the pipeline is being fitted (as stated in the NotFittedError below)\n",
        "num_pipeline.transform(X_train[num_features])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 117,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "yCJwS0ZymbWe",
        "outputId": "9d00e2ed-d71a-4061-8301-40d64675ad86"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<style>#sk-container-id-15 {color: black;background-color: white;}#sk-container-id-15 pre{padding: 0;}#sk-container-id-15 div.sk-toggleable {background-color: white;}#sk-container-id-15 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-15 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-15 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-15 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-15 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-15 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-15 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-15 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-15 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-15 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-15 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-15 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-15 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-15 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-15 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-15 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-15 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-15 div.sk-item {position: relative;z-index: 1;}#sk-container-id-15 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-15 div.sk-item::before, #sk-container-id-15 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-15 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-15 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-15 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-15 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-15 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-15 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-15 div.sk-label-container {text-align: center;}#sk-container-id-15 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-15 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-15\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;imputer&#x27;, SimpleImputer(strategy=&#x27;median&#x27;)),\n",
              "                (&#x27;std_scaler&#x27;, StandardScaler())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;imputer&#x27;, SimpleImputer(strategy=&#x27;median&#x27;)),\n",
              "                (&#x27;std_scaler&#x27;, StandardScaler())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-21\" type=\"checkbox\" ><label for=\"sk-estimator-id-21\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div>"
            ],
            "text/plain": [
              "Pipeline(steps=[('imputer', SimpleImputer(strategy='median')),\n",
              "                ('std_scaler', StandardScaler())])"
            ]
          },
          "execution_count": 117,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "num_pipeline.fit(X_train[num_features])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "icqwGekRmnzi"
      },
      "source": [
        "Now the imputer and standard scaler do contain fitted statistics that can be applied to transform data according to these preprocessing steps. These statistics can be accessed by going into the pipeline object, to the specific pipeline elements."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 118,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "H8Yl6EGVmhep",
        "outputId": "a6ce7842-1e2e-4cea-d5ed-6d9e727699f2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([0.0228 , 0.02925, 0.038  , 0.0454 , 0.06105, 0.0932 , 0.1122 ,\n",
              "       0.113  , 0.1553 , 0.18705, 0.2373 , 0.2551 , 0.27065, 0.2976 ,\n",
              "       0.3039 , 0.3047 , 0.3068 , 0.3771 , 0.4433 , 0.5425 , 0.6496 ,\n",
              "       0.6628 , 0.7052 , 0.69325, 0.72775, 0.7545 , 0.7654 , 0.7321 ,\n",
              "       0.634  , 0.6005 , 0.492  , 0.4241 , 0.3897 , 0.3682 , 0.3369 ,\n",
              "       0.31035, 0.2821 , 0.315  , 0.28905, 0.2883 , 0.2609 , 0.2633 ,\n",
              "       0.2207 , 0.1755 , 0.14955, 0.12785, 0.0947 , 0.08055, 0.049  ,\n",
              "       0.0179 , 0.014  , 0.0115 , 0.00865, 0.0093 , 0.0075 , 0.0065 ,\n",
              "       0.0057 , 0.0058 , 0.0069 , 0.0048 ])"
            ]
          },
          "execution_count": 118,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# the statistics_ for the imputer step contains the medians to be used for imputation\n",
        "num_pipeline['imputer'].statistics_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 119,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "xKEQwyBinPPh",
        "outputId": "53f369cf-4ad6-46fd-99ca-1b3acddfa741"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Means</th>\n",
              "      <th>Scale</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>attribute_1</th>\n",
              "      <td>0.030316</td>\n",
              "      <td>0.024040</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_2</th>\n",
              "      <td>0.038669</td>\n",
              "      <td>0.035053</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_3</th>\n",
              "      <td>0.047230</td>\n",
              "      <td>0.041599</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_4</th>\n",
              "      <td>0.057621</td>\n",
              "      <td>0.050468</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_5</th>\n",
              "      <td>0.074851</td>\n",
              "      <td>0.055196</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_6</th>\n",
              "      <td>0.101849</td>\n",
              "      <td>0.048008</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_7</th>\n",
              "      <td>0.126627</td>\n",
              "      <td>0.063411</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_8</th>\n",
              "      <td>0.136208</td>\n",
              "      <td>0.085086</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_9</th>\n",
              "      <td>0.183074</td>\n",
              "      <td>0.117458</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_10</th>\n",
              "      <td>0.215243</td>\n",
              "      <td>0.135830</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_11</th>\n",
              "      <td>0.248950</td>\n",
              "      <td>0.125070</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_12</th>\n",
              "      <td>0.256374</td>\n",
              "      <td>0.132509</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_13</th>\n",
              "      <td>0.283613</td>\n",
              "      <td>0.126429</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_14</th>\n",
              "      <td>0.310881</td>\n",
              "      <td>0.160377</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_15</th>\n",
              "      <td>0.341880</td>\n",
              "      <td>0.194530</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_16</th>\n",
              "      <td>0.371477</td>\n",
              "      <td>0.222879</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_17</th>\n",
              "      <td>0.418797</td>\n",
              "      <td>0.265304</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_18</th>\n",
              "      <td>0.458043</td>\n",
              "      <td>0.254017</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_19</th>\n",
              "      <td>0.505872</td>\n",
              "      <td>0.256024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_20</th>\n",
              "      <td>0.555277</td>\n",
              "      <td>0.248828</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_21</th>\n",
              "      <td>0.609820</td>\n",
              "      <td>0.247246</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_22</th>\n",
              "      <td>0.624209</td>\n",
              "      <td>0.251237</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_23</th>\n",
              "      <td>0.654730</td>\n",
              "      <td>0.238628</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_24</th>\n",
              "      <td>0.678239</td>\n",
              "      <td>0.229609</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_25</th>\n",
              "      <td>0.682419</td>\n",
              "      <td>0.229040</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_26</th>\n",
              "      <td>0.712124</td>\n",
              "      <td>0.224476</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_27</th>\n",
              "      <td>0.723590</td>\n",
              "      <td>0.226149</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_28</th>\n",
              "      <td>0.700014</td>\n",
              "      <td>0.215167</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_29</th>\n",
              "      <td>0.630289</td>\n",
              "      <td>0.231767</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_30</th>\n",
              "      <td>0.580569</td>\n",
              "      <td>0.199783</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_31</th>\n",
              "      <td>0.505533</td>\n",
              "      <td>0.201870</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_32</th>\n",
              "      <td>0.434770</td>\n",
              "      <td>0.203157</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_33</th>\n",
              "      <td>0.412819</td>\n",
              "      <td>0.196230</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_34</th>\n",
              "      <td>0.396540</td>\n",
              "      <td>0.221638</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_35</th>\n",
              "      <td>0.368626</td>\n",
              "      <td>0.225584</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_36</th>\n",
              "      <td>0.373233</td>\n",
              "      <td>0.255737</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_37</th>\n",
              "      <td>0.349441</td>\n",
              "      <td>0.235093</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_38</th>\n",
              "      <td>0.345731</td>\n",
              "      <td>0.211585</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_39</th>\n",
              "      <td>0.322130</td>\n",
              "      <td>0.187913</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_40</th>\n",
              "      <td>0.314211</td>\n",
              "      <td>0.168741</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_41</th>\n",
              "      <td>0.292407</td>\n",
              "      <td>0.159929</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_42</th>\n",
              "      <td>0.283801</td>\n",
              "      <td>0.161344</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_43</th>\n",
              "      <td>0.243272</td>\n",
              "      <td>0.132807</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_44</th>\n",
              "      <td>0.210315</td>\n",
              "      <td>0.129146</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_45</th>\n",
              "      <td>0.186814</td>\n",
              "      <td>0.144599</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_46</th>\n",
              "      <td>0.161016</td>\n",
              "      <td>0.126471</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_47</th>\n",
              "      <td>0.117938</td>\n",
              "      <td>0.086613</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_48</th>\n",
              "      <td>0.093636</td>\n",
              "      <td>0.063992</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_49</th>\n",
              "      <td>0.054905</td>\n",
              "      <td>0.036116</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_50</th>\n",
              "      <td>0.020219</td>\n",
              "      <td>0.012895</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_51</th>\n",
              "      <td>0.015806</td>\n",
              "      <td>0.011761</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_52</th>\n",
              "      <td>0.013355</td>\n",
              "      <td>0.009462</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_53</th>\n",
              "      <td>0.009907</td>\n",
              "      <td>0.006221</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_54</th>\n",
              "      <td>0.010981</td>\n",
              "      <td>0.007202</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_55</th>\n",
              "      <td>0.009450</td>\n",
              "      <td>0.007024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_56</th>\n",
              "      <td>0.007970</td>\n",
              "      <td>0.005882</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_57</th>\n",
              "      <td>0.007569</td>\n",
              "      <td>0.005725</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_58</th>\n",
              "      <td>0.007968</td>\n",
              "      <td>0.006671</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_59</th>\n",
              "      <td>0.008344</td>\n",
              "      <td>0.006434</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>attribute_60</th>\n",
              "      <td>0.005890</td>\n",
              "      <td>0.004134</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                 Means     Scale\n",
              "attribute_1   0.030316  0.024040\n",
              "attribute_2   0.038669  0.035053\n",
              "attribute_3   0.047230  0.041599\n",
              "attribute_4   0.057621  0.050468\n",
              "attribute_5   0.074851  0.055196\n",
              "attribute_6   0.101849  0.048008\n",
              "attribute_7   0.126627  0.063411\n",
              "attribute_8   0.136208  0.085086\n",
              "attribute_9   0.183074  0.117458\n",
              "attribute_10  0.215243  0.135830\n",
              "attribute_11  0.248950  0.125070\n",
              "attribute_12  0.256374  0.132509\n",
              "attribute_13  0.283613  0.126429\n",
              "attribute_14  0.310881  0.160377\n",
              "attribute_15  0.341880  0.194530\n",
              "attribute_16  0.371477  0.222879\n",
              "attribute_17  0.418797  0.265304\n",
              "attribute_18  0.458043  0.254017\n",
              "attribute_19  0.505872  0.256024\n",
              "attribute_20  0.555277  0.248828\n",
              "attribute_21  0.609820  0.247246\n",
              "attribute_22  0.624209  0.251237\n",
              "attribute_23  0.654730  0.238628\n",
              "attribute_24  0.678239  0.229609\n",
              "attribute_25  0.682419  0.229040\n",
              "attribute_26  0.712124  0.224476\n",
              "attribute_27  0.723590  0.226149\n",
              "attribute_28  0.700014  0.215167\n",
              "attribute_29  0.630289  0.231767\n",
              "attribute_30  0.580569  0.199783\n",
              "attribute_31  0.505533  0.201870\n",
              "attribute_32  0.434770  0.203157\n",
              "attribute_33  0.412819  0.196230\n",
              "attribute_34  0.396540  0.221638\n",
              "attribute_35  0.368626  0.225584\n",
              "attribute_36  0.373233  0.255737\n",
              "attribute_37  0.349441  0.235093\n",
              "attribute_38  0.345731  0.211585\n",
              "attribute_39  0.322130  0.187913\n",
              "attribute_40  0.314211  0.168741\n",
              "attribute_41  0.292407  0.159929\n",
              "attribute_42  0.283801  0.161344\n",
              "attribute_43  0.243272  0.132807\n",
              "attribute_44  0.210315  0.129146\n",
              "attribute_45  0.186814  0.144599\n",
              "attribute_46  0.161016  0.126471\n",
              "attribute_47  0.117938  0.086613\n",
              "attribute_48  0.093636  0.063992\n",
              "attribute_49  0.054905  0.036116\n",
              "attribute_50  0.020219  0.012895\n",
              "attribute_51  0.015806  0.011761\n",
              "attribute_52  0.013355  0.009462\n",
              "attribute_53  0.009907  0.006221\n",
              "attribute_54  0.010981  0.007202\n",
              "attribute_55  0.009450  0.007024\n",
              "attribute_56  0.007970  0.005882\n",
              "attribute_57  0.007569  0.005725\n",
              "attribute_58  0.007968  0.006671\n",
              "attribute_59  0.008344  0.006434\n",
              "attribute_60  0.005890  0.004134"
            ]
          },
          "execution_count": 119,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# the statistics of the standardscaler are the mean and std\n",
        "means = num_pipeline['std_scaler'].mean_\n",
        "scale = num_pipeline['std_scaler'].scale_\n",
        "pd.DataFrame({'Means':means, 'Scale':scale}, index=X_train[num_features].columns)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "di7-9vZB7XIG"
      },
      "source": [
        "## Question 5\n",
        "Compare the imputer and scaler statistics found here with the statistics found under question 4. Explain any differences."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RsfsHJnOqyfH"
      },
      "source": [
        "Now that the imputer and standard scaler have been fitted, we can use it to transform the numerical train data, and check whether it indeed results in fully imputed, fully standarized data:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 120,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gIKbhWjPrOtw",
        "outputId": "76ab25d7-8e87-4aba-f9cc-9b651d30a17d"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array([[ 0.92694488,  0.50297168,  1.79499411, ...,  0.34962688,\n",
              "         4.36066979,  3.60663451],\n",
              "       [-0.13792866, -0.63815596, -0.31562267, ...,  0.21471763,\n",
              "         0.33508158,  0.82479305],\n",
              "       [-0.18784461,  0.1321052 ,  0.17957899, ..., -0.24996978,\n",
              "        -0.55085868, -0.21537375],\n",
              "       ...,\n",
              "       [-0.30847482, -0.05047522, -1.03919403, ..., -0.75962693,\n",
              "         0.31953877, -0.2637536 ],\n",
              "       [-0.74939902, -1.04040346, -0.66418695, ..., -0.32491936,\n",
              "        -0.56640149,  0.29261469],\n",
              "       [-0.88666787, -0.92629069, -0.2218709 , ..., -0.39986894,\n",
              "         1.64067776, -0.2637536 ]])"
            ]
          },
          "execution_count": 120,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_num_prep = num_pipeline.transform(X_train[num_features])\n",
        "X_num_prep"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RPTO8StNr2Rm"
      },
      "source": [
        "You see that the pipeline has the typically undesired feature (though it does speed up computing time) of removing the column names. It is good practice to keep track of the column names, which becomes even more important when we apply preprocessing steps that will change the number of features that we have, as we will encounter when preprocessing the categorical features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 121,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        },
        "id": "m5JAka5EtEIr",
        "outputId": "8b526723-66f2-4e0f-e35a-c6009ff7f892"
      },
      "outputs": [
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>attribute_1</th>\n",
              "      <th>attribute_2</th>\n",
              "      <th>attribute_3</th>\n",
              "      <th>attribute_4</th>\n",
              "      <th>attribute_5</th>\n",
              "      <th>attribute_6</th>\n",
              "      <th>attribute_7</th>\n",
              "      <th>attribute_8</th>\n",
              "      <th>attribute_9</th>\n",
              "      <th>attribute_10</th>\n",
              "      <th>...</th>\n",
              "      <th>attribute_51</th>\n",
              "      <th>attribute_52</th>\n",
              "      <th>attribute_53</th>\n",
              "      <th>attribute_54</th>\n",
              "      <th>attribute_55</th>\n",
              "      <th>attribute_56</th>\n",
              "      <th>attribute_57</th>\n",
              "      <th>attribute_58</th>\n",
              "      <th>attribute_59</th>\n",
              "      <th>attribute_60</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.926945</td>\n",
              "      <td>0.502972</td>\n",
              "      <td>1.794994</td>\n",
              "      <td>1.247895</td>\n",
              "      <td>-0.910411</td>\n",
              "      <td>0.007312</td>\n",
              "      <td>-0.227513</td>\n",
              "      <td>-1.084874</td>\n",
              "      <td>0.391854</td>\n",
              "      <td>-0.383149</td>\n",
              "      <td>...</td>\n",
              "      <td>1.887135</td>\n",
              "      <td>2.171228</td>\n",
              "      <td>0.802669</td>\n",
              "      <td>3.127028</td>\n",
              "      <td>4.007493</td>\n",
              "      <td>1.603111</td>\n",
              "      <td>0.983535</td>\n",
              "      <td>0.349627</td>\n",
              "      <td>4.360670</td>\n",
              "      <td>3.606635</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>-0.137929</td>\n",
              "      <td>-0.638156</td>\n",
              "      <td>-0.315623</td>\n",
              "      <td>-0.652323</td>\n",
              "      <td>0.133136</td>\n",
              "      <td>0.494733</td>\n",
              "      <td>0.089465</td>\n",
              "      <td>0.260823</td>\n",
              "      <td>0.158578</td>\n",
              "      <td>-0.022404</td>\n",
              "      <td>...</td>\n",
              "      <td>0.331137</td>\n",
              "      <td>-0.196059</td>\n",
              "      <td>1.686764</td>\n",
              "      <td>-0.344468</td>\n",
              "      <td>-0.733223</td>\n",
              "      <td>0.209051</td>\n",
              "      <td>1.088333</td>\n",
              "      <td>0.214718</td>\n",
              "      <td>0.335082</td>\n",
              "      <td>0.824793</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>-0.187845</td>\n",
              "      <td>0.132105</td>\n",
              "      <td>0.179579</td>\n",
              "      <td>0.207629</td>\n",
              "      <td>-0.250042</td>\n",
              "      <td>0.482235</td>\n",
              "      <td>0.046886</td>\n",
              "      <td>0.431239</td>\n",
              "      <td>0.820094</td>\n",
              "      <td>0.590127</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.315070</td>\n",
              "      <td>-0.449697</td>\n",
              "      <td>-0.595808</td>\n",
              "      <td>-1.260943</td>\n",
              "      <td>-0.277659</td>\n",
              "      <td>-0.249970</td>\n",
              "      <td>-0.780564</td>\n",
              "      <td>-0.249970</td>\n",
              "      <td>-0.550859</td>\n",
              "      <td>-0.215374</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>-0.778517</td>\n",
              "      <td>-0.592511</td>\n",
              "      <td>-0.056002</td>\n",
              "      <td>1.029934</td>\n",
              "      <td>2.109716</td>\n",
              "      <td>-0.196821</td>\n",
              "      <td>-0.796811</td>\n",
              "      <td>-0.317415</td>\n",
              "      <td>-0.914150</td>\n",
              "      <td>-0.844020</td>\n",
              "      <td>...</td>\n",
              "      <td>-1.080315</td>\n",
              "      <td>0.311217</td>\n",
              "      <td>-0.001053</td>\n",
              "      <td>-0.358354</td>\n",
              "      <td>2.498437</td>\n",
              "      <td>3.354188</td>\n",
              "      <td>0.372213</td>\n",
              "      <td>-0.384879</td>\n",
              "      <td>-0.224460</td>\n",
              "      <td>0.800603</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>-0.587172</td>\n",
              "      <td>-0.381402</td>\n",
              "      <td>-0.505530</td>\n",
              "      <td>-0.376901</td>\n",
              "      <td>-0.187537</td>\n",
              "      <td>-1.138334</td>\n",
              "      <td>-0.331595</td>\n",
              "      <td>0.030468</td>\n",
              "      <td>-0.236458</td>\n",
              "      <td>-0.602542</td>\n",
              "      <td>...</td>\n",
              "      <td>-0.179026</td>\n",
              "      <td>-0.661062</td>\n",
              "      <td>-0.274319</td>\n",
              "      <td>1.696771</td>\n",
              "      <td>1.473417</td>\n",
              "      <td>-0.096963</td>\n",
              "      <td>-0.326440</td>\n",
              "      <td>0.304657</td>\n",
              "      <td>-0.550859</td>\n",
              "      <td>-0.965261</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "<p>5 rows × 60 columns</p>\n",
              "</div>"
            ],
            "text/plain": [
              "   attribute_1  attribute_2  attribute_3  attribute_4  attribute_5  \\\n",
              "0     0.926945     0.502972     1.794994     1.247895    -0.910411   \n",
              "1    -0.137929    -0.638156    -0.315623    -0.652323     0.133136   \n",
              "2    -0.187845     0.132105     0.179579     0.207629    -0.250042   \n",
              "3    -0.778517    -0.592511    -0.056002     1.029934     2.109716   \n",
              "4    -0.587172    -0.381402    -0.505530    -0.376901    -0.187537   \n",
              "\n",
              "   attribute_6  attribute_7  attribute_8  attribute_9  attribute_10  ...  \\\n",
              "0     0.007312    -0.227513    -1.084874     0.391854     -0.383149  ...   \n",
              "1     0.494733     0.089465     0.260823     0.158578     -0.022404  ...   \n",
              "2     0.482235     0.046886     0.431239     0.820094      0.590127  ...   \n",
              "3    -0.196821    -0.796811    -0.317415    -0.914150     -0.844020  ...   \n",
              "4    -1.138334    -0.331595     0.030468    -0.236458     -0.602542  ...   \n",
              "\n",
              "   attribute_51  attribute_52  attribute_53  attribute_54  attribute_55  \\\n",
              "0      1.887135      2.171228      0.802669      3.127028      4.007493   \n",
              "1      0.331137     -0.196059      1.686764     -0.344468     -0.733223   \n",
              "2     -0.315070     -0.449697     -0.595808     -1.260943     -0.277659   \n",
              "3     -1.080315      0.311217     -0.001053     -0.358354      2.498437   \n",
              "4     -0.179026     -0.661062     -0.274319      1.696771      1.473417   \n",
              "\n",
              "   attribute_56  attribute_57  attribute_58  attribute_59  attribute_60  \n",
              "0      1.603111      0.983535      0.349627      4.360670      3.606635  \n",
              "1      0.209051      1.088333      0.214718      0.335082      0.824793  \n",
              "2     -0.249970     -0.780564     -0.249970     -0.550859     -0.215374  \n",
              "3      3.354188      0.372213     -0.384879     -0.224460      0.800603  \n",
              "4     -0.096963     -0.326440      0.304657     -0.550859     -0.965261  \n",
              "\n",
              "[5 rows x 60 columns]"
            ]
          },
          "execution_count": 121,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "X_num_prep = pd.DataFrame(X_num_prep, columns=X_train[num_features].columns)\n",
        "X_num_prep.head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UMl3VGCL9H5T"
      },
      "source": [
        "## Question 6\n",
        "- check whether all missings have been imputed.\n",
        "- check whether means and standard deviations are 0 and 1 for each feature."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2wnXd1rOwXOv"
      },
      "source": [
        "### Categorical preprocessing\n",
        "\n",
        "We took very small steps when walking through the numeric preprocessing pipeline. Now that we have seen the general principles, we will go through the categorical preprocessing pipeline more quickly."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Oy5KNMP2Gqjn"
      },
      "source": [
        "Categorical data are typically preprocessed by imputing (in our example using the 'most_frequent' value), and by 'OneHotEncoding', defining dummy variables out of categorical features."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 122,
      "metadata": {
        "id": "BrBYJ-UwwWhm"
      },
      "outputs": [],
      "source": [
        "from sklearn.preprocessing import OneHotEncoder\n",
        "\n",
        "cat_pipeline = Pipeline([\n",
        "        ('cat_imputer', SimpleImputer(strategy = 'most_frequent')),\n",
        "        ('cat_encoder', OneHotEncoder(sparse=False))])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 123,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "6cQx6XYu2jOH",
        "outputId": "83c74fd9-141c-4c78-f551-48bc59bc641f"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/Users/junda_huang/anaconda3/envs/ai4h_dl/lib/python3.11/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<style>#sk-container-id-16 {color: black;background-color: white;}#sk-container-id-16 pre{padding: 0;}#sk-container-id-16 div.sk-toggleable {background-color: white;}#sk-container-id-16 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-16 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-16 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-16 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-16 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-16 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-16 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-16 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-16 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-16 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-16 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-16 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-16 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-16 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-16 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-16 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-16 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-16 div.sk-item {position: relative;z-index: 1;}#sk-container-id-16 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-16 div.sk-item::before, #sk-container-id-16 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-16 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-16 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-16 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-16 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-16 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-16 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-16 div.sk-label-container {text-align: center;}#sk-container-id-16 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-16 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-16\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;cat_imputer&#x27;, SimpleImputer(strategy=&#x27;most_frequent&#x27;)),\n",
              "                (&#x27;cat_encoder&#x27;,\n",
              "                 OneHotEncoder(sparse=False, sparse_output=False))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-22\" type=\"checkbox\" ><label for=\"sk-estimator-id-22\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;cat_imputer&#x27;, SimpleImputer(strategy=&#x27;most_frequent&#x27;)),\n",
              "                (&#x27;cat_encoder&#x27;,\n",
              "                 OneHotEncoder(sparse=False, sparse_output=False))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-23\" type=\"checkbox\" ><label for=\"sk-estimator-id-23\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;most_frequent&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-24\" type=\"checkbox\" ><label for=\"sk-estimator-id-24\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(sparse=False, sparse_output=False)</pre></div></div></div></div></div></div></div>"
            ],
            "text/plain": [
              "Pipeline(steps=[('cat_imputer', SimpleImputer(strategy='most_frequent')),\n",
              "                ('cat_encoder',\n",
              "                 OneHotEncoder(sparse=False, sparse_output=False))])"
            ]
          },
          "execution_count": 123,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "cat_pipeline.fit(X_train[cat_features])"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 124,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Xz1LHJtm2sTP",
        "outputId": "bd38fa30-b16e-44c9-8b40-1e5599a0dcc2"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "array(['third', 'loc_E'], dtype=object)"
            ]
          },
          "execution_count": 124,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "cat_pipeline['cat_imputer'].statistics_"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 125,
      "metadata": {
        "id": "Gi0BiNs85DXG"
      },
      "outputs": [],
      "source": [
        "X_cat_prep = cat_pipeline.transform(X_train[cat_features])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YfUyi-asH9NE"
      },
      "source": [
        "### Question 7\n",
        "- Compare the dimensions of the unpreprocessed and the preprocessed categorical data and explain the difference.\n",
        "- Apply .get_feature_names_out(cat_features) to the categorical encoder component of the pipeline to extract dummy names.\n",
        "- Create a pandas dataframe containing the prepared categorical features along with the appropriate column names."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nWNUKG_v9CtJ"
      },
      "source": [
        "### Full Pipeline\n",
        "\n",
        "Now that we have constructed the numeric and categorical pipeline, we can define the full pipeline, simply as the combination of both pipelines, wrapped in a ColumnTransformer, which allows for different subsets of features to be passed on to each pipeline."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 126,
      "metadata": {
        "id": "-TSGy1OQ9bWm"
      },
      "outputs": [],
      "source": [
        "from sklearn.compose import ColumnTransformer\n",
        "\n",
        "full_pipeline = ColumnTransformer([\n",
        "        (\"num\", num_pipeline, num_features),\n",
        "        (\"cat\", cat_pipeline, cat_features)])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "O5-yVLgtAm8t"
      },
      "source": [
        "By applying the fit_transform functionality, this could be applied straight away to the raw data and provide the fully prepared data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 127,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 301
        },
        "id": "n-CbifgwBrBJ",
        "outputId": "fc2bd381-df37-4f04-e30e-582da979eb6e"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/Users/junda_huang/anaconda3/envs/ai4h_dl/lib/python3.11/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "ename": "NameError",
          "evalue": "name 'cat_names' is not defined",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
            "\u001b[1;32m/Users/junda_huang/Work/radboudumc_projects/ai4h_course/AI4H_practical3b.ipynb Cell 71\u001b[0m line \u001b[0;36m2\n\u001b[1;32m      <a href='vscode-notebook-cell:/Users/junda_huang/Work/radboudumc_projects/ai4h_course/AI4H_practical3b.ipynb#Y121sZmlsZQ%3D%3D?line=0'>1</a>\u001b[0m X_train_prep \u001b[39m=\u001b[39m full_pipeline\u001b[39m.\u001b[39mfit_transform(X_train)\n\u001b[0;32m----> <a href='vscode-notebook-cell:/Users/junda_huang/Work/radboudumc_projects/ai4h_course/AI4H_practical3b.ipynb#Y121sZmlsZQ%3D%3D?line=1'>2</a>\u001b[0m pd\u001b[39m.\u001b[39mDataFrame(X_train_prep, columns\u001b[39m=\u001b[39m[num_features\u001b[39m.\u001b[39mtolist() \u001b[39m+\u001b[39m cat_names\u001b[39m.\u001b[39mtolist()])\u001b[39m.\u001b[39mhead()\n",
            "\u001b[0;31mNameError\u001b[0m: name 'cat_names' is not defined"
          ]
        }
      ],
      "source": [
        "X_train_prep = full_pipeline.fit_transform(X_train)\n",
        "pd.DataFrame(X_train_prep, columns=[num_features.tolist() + cat_names.tolist()]).head()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wV_hXT28GpJH"
      },
      "source": [
        "And that's it! Now the training data is fully prepared and ready for the phase of model building. Below you find the full code needed to set up the pipeline, without all the detours taken above, resulting in a compact block of code.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 130,
      "metadata": {
        "id": "N8_3Gol5IGxI"
      },
      "outputs": [
        {
          "name": "stderr",
          "output_type": "stream",
          "text": [
            "/Users/junda_huang/anaconda3/envs/ai4h_dl/lib/python3.11/site-packages/sklearn/preprocessing/_encoders.py:868: FutureWarning: `sparse` was renamed to `sparse_output` in version 1.2 and will be removed in 1.4. `sparse_output` is ignored unless you leave `sparse` to its default value.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "data": {
            "text/plain": [
              "array([[ 0.92694488,  0.50297168,  1.79499411, ...,  0.        ,\n",
              "         0.        ,  1.        ],\n",
              "       [-0.13792866, -0.63815596, -0.31562267, ...,  0.        ,\n",
              "         0.        ,  1.        ],\n",
              "       [-0.18784461,  0.1321052 ,  0.17957899, ...,  0.        ,\n",
              "         0.        ,  1.        ],\n",
              "       ...,\n",
              "       [-0.30847482, -0.05047522, -1.03919403, ...,  1.        ,\n",
              "         0.        ,  0.        ],\n",
              "       [-0.74939902, -1.04040346, -0.66418695, ...,  0.        ,\n",
              "         0.        ,  0.        ],\n",
              "       [-0.88666787, -0.92629069, -0.2218709 , ...,  0.        ,\n",
              "         0.        ,  0.        ]])"
            ]
          },
          "execution_count": 130,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "from sklearn.pipeline import Pipeline\n",
        "from sklearn.preprocessing import StandardScaler\n",
        "from sklearn.impute import SimpleImputer\n",
        "\n",
        "num_pipeline = Pipeline([\n",
        "       ('imputer', SimpleImputer(strategy=\"median\")),\n",
        "       ('std_scaler', StandardScaler()),\n",
        "   ])\n",
        "\n",
        "from sklearn.preprocessing import OneHotEncoder\n",
        "\n",
        "cat_pipeline = Pipeline([\n",
        "       ('cat_imputer', SimpleImputer(strategy = 'most_frequent')),\n",
        "       ('cat_encoder', OneHotEncoder(sparse=False))])\n",
        "\n",
        "from sklearn.compose import ColumnTransformer\n",
        "\n",
        "full_pipeline = ColumnTransformer([\n",
        "       (\"num\", num_pipeline, num_features),\n",
        "       (\"cat\", cat_pipeline, cat_features)])\n",
        "\n",
        "num_features = X_train.select_dtypes(include=['float']).columns\n",
        "cat_features = X_train.select_dtypes(exclude=['float']).columns\n",
        "X_train_prep = full_pipeline.fit_transform(X_train)\n",
        "X_train_prep\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jy4ACaEvxM9y"
      },
      "source": [
        "## Training and evaluating a machine learning model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dtJ7pFv27tB0"
      },
      "source": [
        "### Question 8\n",
        "- Train a LASSO model using the prepared train data and save the outcomes in a 'results' object (as in the previous notebook).\n",
        "- Assess the results on the training data by showing performance for different hyperparameter values, and by displaying the confusion matrix.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hBumiMMXRQj0"
      },
      "source": [
        "The accuracy obtained from predicting training data outcomes, using:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4yqY85FJR36O",
        "outputId": "296c496e-d228-4c37-b6d2-b9eeeb3af316"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.9655172413793104"
            ]
          },
          "execution_count": 49,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "accuracy_score(results.predict(X_train_prep), y_train)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ad9LdIthSBVT"
      },
      "source": [
        "is different from the accuracy reported in the 'best_score_':"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rSwiKNXWSMk-",
        "outputId": "e2cc3bfd-5707-4e06-8f0a-f94f1572dfb5"
      },
      "outputs": [
        {
          "data": {
            "text/plain": [
              "0.8873563218390805"
            ]
          },
          "execution_count": 50,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "results.best_score_"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5h2R8mQURMZt"
      },
      "source": [
        "### Question 9\n",
        "- explain the difference, considering that results.best_score_ is the maximum value in results.cv_results_['mean_test_score']\n",
        "- apply the best LASSO model to the test data using accuracy_score(results.predict(X_test), y_test)\n",
        "- prepare the test features using the full_pipeline and check the means and standard deviations of the prepared test features\n",
        "- provide the confusion matrix for the test data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rG219z6ohceY"
      },
      "source": [
        "# Conclusion\n",
        "\n",
        "This notebook took a 'small step by small step' approach to developing a model. Sklearn has extensive functionalities, and the risk in using such a powerful module is that you lose sight of what is happening and start leaning on the preprogrammed features too easily. \n",
        "\n",
        "Especially when you start out working with sklearn and its wide array of applications, it really pays off to take this step by step approach to keep track of what is happening as much as possible. "
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [
        "rG219z6ohceY"
      ],
      "name": "Scikit_learning_model_training_unpacked.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}