tdi_capstone_model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "53396a8b",
   "metadata": {},
   "source": [
    "This notebook contains the ML model that predicts log of the sales. It could just as well predict the sales themselves, but given the range of values for sales, it makes more sense to predict the log. An error of \\\\$50,000 on a 1.5 million dollars game would be less substantial than the same error on a \\\\$100,000 game."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "add8b3ee",
   "metadata": {},
   "source": [
    "The notebook assumes that the data set is already cleaned and ready to be used. See the data engineering notebook for the data preparation.\n",
    "\n",
    "There are three kinds of columns in the data set: numerical (e.g. sales), categorical (e.g. platform, genre), and free-form text (e.g. summary). The numerical data does not require any further preparation. The categorical data will be one-hot encoded. Some categorical columns contain singular values and will be transformed using OneHotEncoder. Other categorical columns contain lists that will use DictVectorizer. Note that the majority of the categorical columns contain numerical values, but these simply represent different classes of whatever data the columns hold. The free-form text will be transformed using TF-IDF vectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "c697d9c5",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "pd.set_option('display.max_columns', 50)\n",
    "pd.set_option('display.max_rows', 50)\n",
    "\n",
    "from tdi_capstone_common_functions import *\n",
    "# the two functions imported are: standardize_string and pseudo_list_parser"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "928488c2",
   "metadata": {},
   "source": [
    "The data, loaded into a pandas data frame, has a lot of columns. While the data engineering notebook dropped columns that were not useful for predictions, specifically a lot of meta-data (e.g., the url for the game on IGDB), there are still columns that may be useful, but may as well not be. They were retained up to this point, but are now dropped. They may be used in future versions of the model. Some, however, can and shouled be dropped regardless, such asname, release_year, closest_match, match_score, id, first_release_date, and release_dates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e5680923",
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the data\n",
    "drop_columns = [\n",
    "    'name',\n",
    "    'release_year',\n",
    "    'closest_match',\n",
    "    'match_score',\n",
    "    'id',\n",
    "    'first_release_date',\n",
    "    'release_dates',\n",
    "    'alternative_names',\n",
    "    'external_games',\n",
    "    'similar_games',\n",
    "    'language_supports',\n",
    "    'status',\n",
    "    'bundles',\n",
    "    'collections',\n",
    "    'parent_game',\n",
    "    'collection'\n",
    "    ]\n",
    "\n",
    "df = (pd.read_csv('data_complete.csv', index_col='index')\n",
    "      .drop(drop_columns, axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f98c323",
   "metadata": {},
   "source": [
    "Since different kinds of columns will be transformed using different transformers, it is easy to create lists of columns according to their kind."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e4edf26f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# columns with numerical data\n",
    "numeric_columns = ['sales_na', 'sales_eu', 'sales_jp', 'sales_other', 'sales_global']\n",
    "\n",
    "# columns with free-form text\n",
    "text_columns = ['summary', 'storyline']\n",
    "\n",
    "# columns that contain lists (imported as strings that look like lists)\n",
    "list_columns = ['age_ratings', 'game_modes', 'genres', 'themes', 'involved_companies', 'keywords',\n",
    "               'multiplayer_modes', 'franchises', 'game_engines', 'player_perspectives', 'game_localizations']\n",
    "\n",
    "# parse the columns that contain pseudo-lists into lists and populate those with a non-list (NaN) with an empty list\n",
    "df[list_columns] = df[list_columns].applymap(lambda x: pseudo_list_parser(x)).applymap(lambda x: x if isinstance(x, list) else [])\n",
    "\n",
    "# since most columns won't use OneHotEncoder, it is easier to exclude columns from df.columns than to explicitly list them out\n",
    "non_ohe_columns = numeric_columns + text_columns + list_columns\n",
    "# creates a list with the columns that do not appear in any of the above lists\n",
    "ohe_columns = [column_name for column_name in df.columns if column_name not in non_ohe_columns]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7d0746c",
   "metadata": {},
   "source": [
    "## Building the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3ebcc353",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "\n",
    "from sklearn.preprocessing import OneHotEncoder\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "\n",
    "from sklearn.decomposition import TruncatedSVD # used for dimensionality reduction\n",
    "\n",
    "from sklearn.linear_model import LinearRegression, Ridge\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.neighbors import KNeighborsRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "922aa8d1",
   "metadata": {},
   "source": [
    "As mentioned above, there are three kinds of columns: numerical, categorical, and free-form text. Numerical data does not require any transformation at this point. The categorical columns can be divided into two subtypes, columns with singular values that will be transformed using OneHotEncoder and columns with lists of values that will be transformed with DictVectorizer. Lastly, free-form text will use TfidfVectorizer. Each of these kinds of transformations will be stored as a list of transformers with their relevant columns, so they can all eventually be put together into a single list of transformers passed onto the ColumnTransformer of the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e5eddb7",
   "metadata": {},
   "source": [
    "First to be treated are the OneHotEncoder categorical columns. It is created as a tuple (per usual format of transformers) and then places into a list, so that it can easily be added to the other transformers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3d6dd6a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "ohe_transformers = [('categorical', OneHotEncoder(handle_unknown='ignore'), ohe_columns)]\n",
    "#ohe_transformer = ('categorical', OneHotEncoder(handle_unknown='ignore'), ohe_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0938d0e6",
   "metadata": {},
   "source": [
    "Then the categorical columns with lists of values. Each one of these lists is first encoded as a dictionary (using a custom class) and then vectorized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "1374c030",
   "metadata": {},
   "outputs": [],
   "source": [
    "class DictEncoder(BaseEstimator, TransformerMixin):\n",
    "\n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "#         X_trans = pd.DataFrame()\n",
    "        \n",
    "#         for column in X.columns:\n",
    "#             column_trans = [{key: 1 for key in item} for item in X[column]]\n",
    "#             X_trans[column] = column_trans\n",
    "        \n",
    "#         return X_trans\n",
    "        return [{key: 1 for key in row} for row in X] #, name=X.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6651fae8",
   "metadata": {},
   "source": [
    "Because the transformer first needs to encode and then vectorize the list, a pipeline is used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bc1bd779",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_column_vectorizer = Pipeline([\n",
    "    ('dict_encoder', DictEncoder()),\n",
    "    ('dict_vectorizer', DictVectorizer())\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a1d3512",
   "metadata": {},
   "source": [
    "Then a list is created, where every element is the tuple format of a transformer (name, transformer, column(s)) for every column. This way, it is possible to access specific transformers by its name via the named_steps method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a7ae72fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "list_column_transformers = [(f'list_vect_{column}', list_column_vectorizer, f'{column}') for column in list_columns]\n",
    "#list_column_transformer = ('list_columns', list_column_vectorizer, list_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "672de3d2",
   "metadata": {},
   "source": [
    "Lastly, the free-form text columns are treated, where the strings are first standardized and then vectorized using TF-IDF. Very rare words are excluded, represented by the min_df value. Additionally, a relatively low max_df is set to exclude words that appear in a lot of game descriptions, since we want to use the most impactful words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "c02816ed",
   "metadata": {},
   "outputs": [],
   "source": [
    "def text_column_preprocessing(series):\n",
    "    \"\"\"\n",
    "    Standardizes a series containing free-form strings.\n",
    "    \n",
    "    Parameter\n",
    "    ---------\n",
    "    series : pandas.Series()\n",
    "        A series of strings to be standardized, retaining only alphanumeric characters,\n",
    "        changing East Asian characters into Latin ones, and removing symbols and parentheses.\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    pandas.Series\n",
    "        A series of standardized strings.\n",
    "    \"\"\"\n",
    "    \n",
    "    return series.map(standardize_string)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ab35f807",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.preprocessing import FunctionTransformer\n",
    "from spacy.lang.en.stop_words import STOP_WORDS\n",
    "\n",
    "STOP_WORDS = STOP_WORDS.difference({'he','his','her','hers'}).union({'ll', 've'})\n",
    "\n",
    "text_column_vectorizer = Pipeline([\n",
    "    ('standardize_text', FunctionTransformer(text_column_preprocessing)),\n",
    "    ('tfidf', TfidfVectorizer(min_df=20, max_df=0.5, stop_words=list(STOP_WORDS)))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1f75d87",
   "metadata": {},
   "source": [
    "Much like in the case with the list columns, a list of tuples is created, each referring to an individual free-form text column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "b77814e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "text_column_transformers = [(f'text_{column}', text_column_vectorizer, f'{column}') for column in text_columns]\n",
    "#text_column_transformer = ('text_columns', text_column_vectorizer, text_columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27d858d8",
   "metadata": {},
   "source": [
    "It is now possible to build the model. All the features will undergo their respective transformations (remember that the numerical columns do not need any at this point)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "de23d512",
   "metadata": {},
   "outputs": [],
   "source": [
    "features = ColumnTransformer(\n",
    "    transformers = list_column_transformers + ohe_transformers + text_column_transformers,\n",
    "    remainder='drop')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f45cc60",
   "metadata": {},
   "source": [
    "The model was trained with several regressors alongside appropriate parameter grids. These include LinearRegression, Ridge, RandomForestRegressor, and KNeighborsRegressor. Out of all of them, Ridge seemed to performed the best. In order to streamline building different versions of the model, a function allows for a quick customization of the regressor."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a59134d",
   "metadata": {},
   "source": [
    "Note that the data set ends up having quite a lot of features, particularly due to the free-form text vectorizer. Given the number of observations, this needs to be taken into account. To balance that, some kind of dimensionality reduction can help to improve the model. Post-transformation, the data is stored in a sparse matrix, and since PCA doesn't work with sparse data, we'll use TruncatedSVD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "8c3a5486",
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_regressor(requested_model='ridge', hyperparameters=None, requested_dr='default', cv=5, n_jobs=2):\n",
    "    \"\"\"\n",
    "    Creates a regressor with grid search cross-validation.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    requested_model : string or sklearn model, default 'ridge'\n",
    "        The name of the model as a string or an instantiated model\n",
    "        object itself. Currently supported models: LinearRegression, Ridge,\n",
    "        RandomForestRegressor, KNeighborsRegressor. Defaults to Ridge\n",
    "        if anything else is given.\n",
    "    hyperparameters : dict, default None\n",
    "        The hyperparameters for grid search cross-validation based on\n",
    "        the regressor. If not passed, defaults to the built-in ones.\n",
    "    requested_dr : string or list, default 'default'\n",
    "        Indicates how dimensionality reduction using TruncatedSVD should\n",
    "        work. Accepted strings are 'default' for the default values based\n",
    "        on the model chosen, 'aggressive' to reduce to hundreds of features,\n",
    "        or 'relaxed' to reduce to tens of features. Alternatively, a list\n",
    "        can be passed with the possible values.\n",
    "    cv : int, default 5\n",
    "        The number of folds in the cross validation.\n",
    "    n_jobs : int, default 2\n",
    "        The number of parallel jobs for the cross validation.\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    GridSearchCV object\n",
    "        An object based on the requested model or the default if that\n",
    "        type of regressor is not supported by the function. The parameter\n",
    "        grid is also set for the specified model.\n",
    "    \n",
    "    Notes\n",
    "    -----\n",
    "    The currently supported models are linear regression, ridge, random\n",
    "    forest, and k-nearest neighbors. All but the last one use a relaxed\n",
    "    dimensionality reduction (n_components = [100, 250, 500]) by default,\n",
    "    with k-nearest neighbors using an aggressive one\n",
    "    (n_components = [10, 20, 30]). The built-in default hyperparameters\n",
    "    for each are as follows:\n",
    "        Ridge :\n",
    "            'alpha': [0.1, 1.0, 10.0]\n",
    "        Random Forest :\n",
    "            'n_estimators': [10, 50, 100, 200, 300],\n",
    "            'max_depth': [None, 10, 20, 50],\n",
    "            'min_samples_split': [2, 5, 10],\n",
    "            'min_samples_leaf': [1, 2, 4]\n",
    "        K-Nearest Neighbors :\n",
    "            'n_neighbors': [3, 5, 8, 10, 15]  \n",
    "    \"\"\"\n",
    "\n",
    "    #\n",
    "    if isinstance(requested_model, str):\n",
    "        supported_models = {\n",
    "            'linearregression': LinearRegression(),\n",
    "            'lr': LinearRegression(),\n",
    "            'linear': LinearRegression(),\n",
    "            'ridge': Ridge(),\n",
    "            'randomforestregressor': RandomForestRegressor(),\n",
    "            'randomforest': RandomForestRegressor(),\n",
    "            'kneighborsregressor': KNeighborsRegressor(),\n",
    "            'knn': KNeighborsRegressor()\n",
    "        }\n",
    "        \n",
    "        model = supported_models.get(requested_model.lower(), Ridge())\n",
    "        \n",
    "    # The default values for dimensionality reduction\n",
    "    dr_values = {\n",
    "        'relaxed': [100, 250, 500],\n",
    "        'aggressive': [10, 20, 30] # for, e.g., KNN\n",
    "    }\n",
    "    \n",
    "    # if the function was passed a specific list of dimensionality reduction values, it will use that list\n",
    "    if isinstance(requested_dr, list):\n",
    "        dr = requested_dr\n",
    "    else:\n",
    "        # if a string was passed, it will use one of the two built-in options, either aggressive or relaxed dimensionality\n",
    "        # reduction; or if the string was 'default' or another non-supported string, it will use a default set of values.\n",
    "        # These are based on the model requested, and so a function is used to determine which of the two built-in options\n",
    "        # to use.\n",
    "        def find_default_dr(model_for_dr):\n",
    "            aggressive_dr_models = [KNeighborsRegressor] # list of models which default to aggressive dimensionality reduction\n",
    "            \n",
    "            if model_for_dr in aggressive_dr_models:\n",
    "                return dr_values['aggressive']\n",
    "            \n",
    "            return dr_values['relaxed']\n",
    "        \n",
    "        # find the correct built-in value for dimensionality reduction\n",
    "        dr = dr_values.get(requested_dr, find_default_dr(model))\n",
    "\n",
    "    # Initiliazes the dimensionality reduction object\n",
    "    svd = TruncatedSVD()\n",
    "    \n",
    "    # Initializes the parameter grid with the dimensionality reduction values\n",
    "    param_grid = {'dim_reduction__n_components': dr}\n",
    "    \n",
    "    # If no parameter grid was passed, use the built-in values\n",
    "    if not hyperparameters:\n",
    "        \n",
    "        if isinstance(model, Ridge):\n",
    "            hyperparameters = {\n",
    "                'alpha': [0.1, 1.0, 10.0]\n",
    "            }\n",
    "        elif isinstance(model, RandomForestRegressor):\n",
    "            hyperparameters = {\n",
    "                'n_estimators': [10, 50, 100, 200, 300],\n",
    "                'max_depth': [None, 10, 20, 50],\n",
    "                'min_samples_split': [2, 5, 10],\n",
    "                'min_samples_leaf': [1, 2, 4]\n",
    "            }\n",
    "        elif isinstance(model, KNeighborsRegressor):\n",
    "            hyperparameters = {\n",
    "                'n_neighbors': [3, 5, 8, 10, 15]   \n",
    "            }\n",
    "        else: # default if model does not have hyperparameters, e.g. linear regression\n",
    "            hyperparameters = {}\n",
    "    \n",
    "    # renames the keys so they can be properly accessed\n",
    "    hyperparameters = {f'regressor__{k}' : v for k, v in hyperparameters.items()}\n",
    "        \n",
    "    # Updates the parameter grid (currently only with dimensionality reduction) with the regressor's hyperparameters.\n",
    "    param_grid.update(hyperparameters)\n",
    "        \n",
    "    estimator = Pipeline([\n",
    "        ('dim_reduction', svd),\n",
    "        ('regressor', model)\n",
    "    ])\n",
    "\n",
    "    gs = GridSearchCV(\n",
    "        estimator,\n",
    "        param_grid=param_grid,\n",
    "        cv=cv,\n",
    "        n_jobs=n_jobs\n",
    "    )\n",
    "\n",
    "    return gs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cb2398f",
   "metadata": {},
   "source": [
    "It is then possible to build the model with different kinds of regression algorithms or parameter grids. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef7bbb1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Pipeline([\n",
    "    ('features', features),\n",
    "    ('main_regressor', generate_regressor('ridge'))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c96b7298",
   "metadata": {},
   "source": [
    "## Training the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65a95e52",
   "metadata": {},
   "source": [
    "As mentioned above, the numerical columns are only the sales, which are also the (basis of the) labels, so these can be dropped from the feature matrix passed onto the model. The labels are going to be the log of the global sales. In the future, it would be possible to have the model predict localized sales (e.g. EU, US, etc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "343f0c6c",
   "metadata": {},
   "outputs": [],
   "source": [
    "X = df.drop(numeric_columns, axis=1)\n",
    "y = np.log(df['sales_global'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7742e7b7",
   "metadata": {},
   "source": [
    "The data is then split into a training set and a test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3481eab",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.05)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a0c0502",
   "metadata": {},
   "source": [
    "The model is then fit with the training set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c3e3021",
   "metadata": {},
   "outputs": [],
   "source": [
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1d81144",
   "metadata": {},
   "source": [
    "## Saving and loading the trained model\n",
    "The model can then be serialized and saved into a file so it doesn't have to be trained again. A container (list) object is created that will hold the model itself, as well as the train/test split of the data that was used in its training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "36fb7e1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca8d1650",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_container = [model, X_train, X_test, y_train, y_test]\n",
    "\n",
    "with open('model.pickle', 'wb') as f:\n",
    "    pickle.dump(data_container, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6dca4b72",
   "metadata": {},
   "source": [
    "If the model has already been pickled, it can be imported."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d4183dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('model', 'rb') as f:\n",
    "    loaded_container = pickle.load(f)\n",
    "\n",
    "model, X_train, X_test, y_train, y_test = loaded_container"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e89966d",
   "metadata": {},
   "source": [
    "## Making predictions\n",
    "Predictions can then be made:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92fb98d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aa16f5b",
   "metadata": {},
   "source": [
    "## Assessing model performance\n",
    "It's then possible to observe some metrics on the performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "31367437",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9129eef",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = pipe.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "666ff214",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Mean absolute error: {metrics.mean_absolute_error(y_test, y_pred)}\")\n",
    "print(f\"Mean squared error: {metrics.mean_squared_error(y_test, y_pred)}\")\n",
    "print(f\"R^2: {metrics.r2_score(y_test, y_pred)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "77b83903",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe.score(X_test, y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b38cc61",
   "metadata": {},
   "source": [
    "## Working with multiple models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9774e92b",
   "metadata": {},
   "source": [
    "So far only one model was used. But we can train multiple models to see the differences in performances and predictive power. To be consistent, the same train/test data split will be used. The trained models are then saved as a dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "348ad2be",
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithms = ['lr', 'ridge', 'knn', 'randomforest']\n",
    "# Note that these need to be strings accepted by the generate_regressor function. Note also that depending on the model requested\n",
    "# this might take a while to run (e.g., with this data set, the random forest regressor will take hours and hours to train).\n",
    "\n",
    "models = dict()\n",
    "\n",
    "for algorithm in algorithms:\n",
    "    model = Pipeline([\n",
    "        ('features', features),\n",
    "        ('main_regressor', generate_regressor(algorithm, cv=10, n_jobs=-1))\n",
    "    ])\n",
    "    model.fit(X_train, y_train)\n",
    "    models[algorithm] = model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83f7a8fa",
   "metadata": {},
   "source": [
    "Once we trained several models and saved them as a dictionary, we can then save them into different files."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fc3004f1",
   "metadata": {},
   "outputs": [],
   "source": [
    "for model_name, model in models.items():\n",
    "    with open(f'{model_name}.pickle', 'wb') as f:\n",
    "        data_container = [model, X_train, X_test, y_train, y_test]\n",
    "        pickle.dump(data_container, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c9e121b",
   "metadata": {},
   "source": [
    "Similarly, we can load the pickled models. This time, we are going to use an embedded dictionary. The top level's keys are the name of the model, and the values are embedded dictionaries that contain the trained model object itself ('model') as well as the train/test data split. This is done in order to retain the associated train/test split data used for the training of each model. Note, however, that in this case, all of models would have the same train/test split data, because that's how they were saved above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "3641dbcd",
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithms = ['lr', 'ridge', 'knn', 'randomforest']\n",
    "\n",
    "loaded_models = dict()\n",
    "\n",
    "for algorithm in algorithms:\n",
    "    with open(f'{algorithm}.pickle', 'rb') as f:\n",
    "        # The embedded dict is reset every loop as otherwise all loaded models would actually point to the last dict (here: randomforest)\n",
    "        trained_model = dict()\n",
    "        trained_model['model'], trained_model['X_train'], trained_model['X_test'], trained_model['y_train'], trained_model['y_test'] = pickle.load(f)\n",
    "        loaded_models[algorithm] = trained_model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2293b46",
   "metadata": {},
   "source": [
    "We can run predictions to each of them and save them to their corresponding model in the loaded_models dictionary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "443ecec5",
   "metadata": {},
   "outputs": [],
   "source": [
    "for model in loaded_models.values():\n",
    "    model['y_pred'] = model['model'].predict(model['X_test'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "d348f00c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "For lr:\n",
      "Mean absolute error: 0.8173556466817667\n",
      "Mean squared error: 1.149279655508038\n",
      "R^2: 0.5197002033564457\n",
      "\n",
      "For ridge:\n",
      "Mean absolute error: 0.8106476823277217\n",
      "Mean squared error: 1.171809867623642\n",
      "R^2: 0.5102845174129952\n",
      "\n",
      "For knn:\n",
      "Mean absolute error: 0.8923915907997727\n",
      "Mean squared error: 1.316066308424797\n",
      "R^2: 0.44999776401119607\n",
      "\n",
      "For randomforest:\n",
      "Mean absolute error: 0.8933907637672172\n",
      "Mean squared error: 1.2946976024871029\n",
      "R^2: 0.45892804052589997\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for model_name, model in loaded_models.items():\n",
    "    print(f'For {model_name}:')\n",
    "    print(f\"Mean absolute error: {metrics.mean_absolute_error(model['y_test'], model['y_pred'])}\")\n",
    "    print(f\"Mean squared error: {metrics.mean_squared_error(model['y_test'], model['y_pred'])}\")\n",
    "    print(f\"R^2: {metrics.r2_score(model['y_test'], model['y_pred'])}\")\n",
    "    print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6124d3c3",
   "metadata": {},
   "source": [
    "## Data visualization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "47e449b4",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "414e77f7",
   "metadata": {},
   "source": [
    "We can plot the actual vs predicted values of the single model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "98d688d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_predicted_vs_actual(actual_values, predicted_values, units='log of sales'):\n",
    "    \"\"\"\n",
    "    Plots a graph of the predicted labels against the actual values.\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    actual_values : pandas.Series\n",
    "        The series containing the label values that were kept for testing,\n",
    "        often called y_test.\n",
    "    predicted_values : numpy.ndarray\n",
    "        The array of predicted values for the model, often called y_pred.\n",
    "    units : string, default 'log of sales'\n",
    "        A string that designates the units of the graph, which will appear\n",
    "        on the plot itself.\n",
    "    \n",
    "    Returns\n",
    "    -------\n",
    "    None\n",
    "    \n",
    "    \"\"\"\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    sns.scatterplot(x=actual_values, y=predicted_values, alpha=0.5)\n",
    "    sns.lineplot(x=[min(actual_values), max(actual_values)], y=[min(actual_values), max(actual_values)], color='red', linestyle='--')\n",
    "    plt.xlabel(f'Actual values ({units})')\n",
    "    plt.ylabel(f'Predicted values ({units})')\n",
    "    plt.title(f'Actual vs predicted values ({units})')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5feff84b",
   "metadata": {},
   "source": [
    "The red line denotes perfect predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "88e8ced0",
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_predicted_vs_actual(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5f872d0",
   "metadata": {},
   "source": [
    "If we have multiple trained models in the loaded_models dictionary, we can plot a comparison between them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "25ce73da",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA04AAAIhCAYAAAB5deq6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy88F64QAAAACXBIWXMAAA9hAAAPYQGoP6dpAABKgklEQVR4nO3deVgVdf//8ddhkR1UVERFwTXcSTTRTNwwtxb11lvLXW93M1yKvHPLJdcvLqmZe6VZmWVlLpW7Zbm13FqZiiuGioKiosD8/uji/DoBDoJyJJ6P6zrX5XzmMzPvOed0H173Z+YzFsMwDAEAAAAAsuRg7wIAAAAA4GFHcAIAAAAAEwQnAAAAADBBcAIAAAAAEwQnAAAAADBBcAIAAAAAEwQnAAAAADBBcAIAAAAAEwQnAAAAADBBcAJQ4KxYsUIWi0UWi0Xbt2/PsN4wDFWsWFEWi0Xh4eH39dgWi0Xjx4+/5+1iYmJksVi0YsWK+1rP/bB27VpVq1ZNbm5uslgsOnz4sL1LQj4VGBionj172rsMAMiUk70LAAB78fLy0tKlSzOEox07duj48ePy8vKyT2H5yMWLF9WtWzc9+eSTWrBggVxcXFS5cmV7l4V8av369fL29rZ3GQCQKUacABRYnTt31rp165SYmGjTvnTpUoWFhals2bJ2quzhd/PmTRmGod9++0137tzR888/r8aNG6t+/fpyd3fP1b5v3Lhxn6pEfnHz5k1JUkhIiCpUqGDnagAgcwQnAAVWly5dJElr1qyxtiUkJGjdunXq3bt3ptvEx8dr0KBBKl26tAoVKqTy5ctrzJgxSk5OtumXmJiofv36ydfXV56ennryySf122+/ZbrPY8eOqWvXripRooRcXFwUHBysN954I0fntH37dlksFr3zzjuKjIxUyZIl5ebmpsaNG+vQoUMZ+u/fv19PPfWUihYtKldXV4WEhOj999+36ZN+aeOWLVvUu3dvFS9eXO7u7urSpYsef/xxSX+G0L9f2rhhwwaFhYXJ3d1dXl5eatGihb755hubfY8fP14Wi0UHDx5Ux44dVaRIEesfzoGBgWrbtq0+++wzhYSEyM3NTcHBwfrss8+sdQUHB8vDw0P16tXT/v37M5zbv//9bwUGBsrNzU2BgYHq0qWLTp06len5bdu2TQMHDlSxYsXk6+ur9u3b6/z58xnes9WrVyssLEyenp7y9PRU7dq1tXTpUps+X375pZo1ayZvb2+5u7urYcOG+uqrr+720VldvXpVI0aMUPny5eXi4qISJUqodevW+uWXX6x9svs9tFgsGjJkiJYvX64qVarIzc1NoaGh+vbbb2UYhmbMmKGgoCB5enqqadOm+v333222Dw8PV/Xq1bVr1y7Vr19fbm5uKl26tF599VWlpqba9J0wYYIee+wxFS1aVN7e3nr00Ue1dOlSGYZh0y/9c/3oo48UEhIiV1dXTZgwwbrur5fqpaWladKkSdbaCxcurJo1a2rOnDk2+9y9e7eaNWsmLy8vubu7q0GDBvr8889t+tzr5wwAf0dwAlBgeXt7q2PHjlq2bJm1bc2aNXJwcFDnzp0z9L9165aaNGmiVatWKTIyUp9//rmef/55TZ8+Xe3bt7f2MwxDzzzzjN5++22NGDFC69evV/369dWqVasM+zxy5Ijq1q2rn3/+WbNmzdJnn32mNm3aaNiwYdY/JnPilVde0YkTJ7RkyRItWbJE58+fV3h4uE6cOGHts23bNjVs2FBXr17VokWL9Mknn6h27drq3LlzpvdS9e7dW87Oznr77bf14YcfavLkydaAN2XKFH3zzTdasGCBpD/DxdNPPy1vb2+tWbNGS5cu1ZUrVxQeHq7du3dn2Hf79u1VsWJFffDBB1q0aJG1/YcfflBUVJReeuklffTRR/Lx8VH79u01btw4LVmyRFOmTNG7776rhIQEtW3b1jpyIf15X1iVKlUUHR2tzZs3a9q0aYqNjVXdunV16dKlDDX07dtXzs7OWr16taZPn67t27fr+eeft+kzduxYPffccypVqpRWrFih9evXq0ePHjZh7J133lFERIS8vb21cuVKvf/++ypatKhatmxpGp6uXbumxx9/XG+++aZ69eqlTz/9VIsWLVLlypUVGxsrKfvfw3SfffaZlixZotdff11r1qzRtWvX1KZNG40YMUJ79uzR/PnztXjxYh05ckQdOnTIEHQuXLigf//733ruuef0ySefqGPHjpo0aZJeeOEFm34xMTHq37+/3n//fX300Udq3769hg4dqtdeey1DTQcPHtSoUaM0bNgwbdq0SR06dMj0/Zg+fbrGjx+vLl266PPPP9fatWvVp08fXb161dpnx44datq0qRISErR06VKtWbNGXl5eateundauXZthn9n5nAEgUwYAFDDLly83JBnff/+9sW3bNkOS8fPPPxuGYRh169Y1evbsaRiGYVSrVs1o3LixdbtFixYZkoz333/fZn/Tpk0zJBlbtmwxDMMwvvjiC0OSMWfOHJt+kydPNiQZ48aNs7a1bNnSKFOmjJGQkGDTd8iQIYarq6sRHx9vGIZhnDx50pBkLF++/K7nln4+jz76qJGWlmZtj4mJMZydnY2+ffta2x555BEjJCTEuHPnjs0+2rZta/j7+xupqak271f37t2zPN4HH3xgbUtNTTVKlSpl1KhRw7oPwzCMa9euGSVKlDAaNGhgbRs3bpwhyRg7dmyGfZcrV85wc3Mzzp49a207fPiwIcnw9/c3kpKSrO0ff/yxIcnYsGFDlu9NSkqKcf36dcPDw8Pms0k/v0GDBtn0nz59uiHJiI2NNQzDME6cOGE4Ojoazz33XJbHSEpKMooWLWq0a9fOpj01NdWoVauWUa9evSy3NQzDmDhxoiHJ2Lp1a5Z9svs9NAzDkGSULFnSuH79urUt/b2qXbu2zXckOjrakGT8+OOP1rbGjRsbkoxPPvnE5lj9+vUzHBwcjFOnTmVaY2pqqnHnzh1j4sSJhq+vr81xypUrZzg6Ohq//vprhu3KlStn9OjRw7rctm1bo3bt2lm+F4ZhGPXr1zdKlChhXLt2zdqWkpJiVK9e3ShTpoz12Nn9nAEgK4w4ASjQGjdurAoVKmjZsmX66aef9P3332d5md7XX38tDw8PdezY0aY9/dKi9NGEbdu2SZKee+45m35du3a1Wb5165a++uorPfvss3J3d1dKSor11bp1a926dUvffvttjs6ra9euslgs1uVy5cqpQYMG1tp+//13/fLLL9Ya/37s2NhY/frrrzb7zGpU4O9+/fVXnT9/Xt26dZODw///mfH09FSHDh307bffZriPKat9165dW6VLl7YuBwcHS/rzErK/3kuV3v7XkZ/r16/rpZdeUsWKFeXk5CQnJyd5enoqKSlJR48ezXCsp556yma5Zs2aNvvcunWrUlNTNXjw4CzPfe/evYqPj1ePHj1s3tO0tDQ9+eST+v7775WUlJTl9l988YUqV66s5s2bZ9knu9/DdE2aNJGHh4d1Of29atWqlc13JLP3UPpzEpW/vzddu3ZVWlqadu7caVNX8+bN5ePjI0dHRzk7O2vs2LG6fPmy4uLibLavWbNmtiYRqVevnn744QcNGjRImzdvznA/YlJSkvbt26eOHTvK09PT2u7o6Khu3brp7NmzGb7HZp8zAGSFWfUAFGgWi0W9evXS3LlzdevWLVWuXFmNGjXKtO/ly5dVsmRJmz82JalEiRJycnLS5cuXrf2cnJzk6+tr069kyZIZ9peSkqJ58+Zp3rx5mR4zs0vKsuPvx0pv++GHHyRJf/zxhyRp5MiRGjlyZLaO7e/vn61jp78PmfUvVaqU0tLSdOXKFZvgk9W+ixYtarNcqFChu7bfunXL2ta1a1d99dVXevXVV1W3bl15e3vLYrGodevWNpf0pfv75+Xi4iLp/09ccPHiRUlSmTJlMq1V+v/v699DzV/Fx8fbBJm/unjxoumkJNn9HqbLzXsoSX5+fhlqSP9+pR/ru+++U0REhMLDw/XWW2+pTJkyKlSokD7++GNNnjw5w/ud3e9SVFSUPDw89M4772jRokVydHTUE088oWnTpik0NFRXrlyRYRhZftf+WmM6s88ZALJCcAJQ4PXs2VNjx47VokWLNHny5Cz7+fr6at++fTIMw+aP1ri4OKWkpKhYsWLWfikpKbp8+bLNH2kXLlyw2V+RIkWs/894VqMYQUFBOTqnvx8rvS29nvRao6KiMr0vRpKqVKlis/z3P9Szkn6M9Hty/ur8+fNycHBQkSJFcrTv7EpISNBnn32mcePG6eWXX7a2JycnKz4+Pkf7LF68uCTp7NmzCggIyLRP+vs6b9481a9fP9M+mQWRvx7j7Nmzd60ju9/D+yU9DP5V+vcr/bN+77335OzsrM8++0yurq7Wfh9//HGm+8zu5+3k5KTIyEhFRkbq6tWr+vLLL/XKK6+oZcuWOnPmjIoUKSIHB4csv2uS7vv7AaDg4lI9AAVe6dKlNWrUKLVr1049evTIsl+zZs10/fr1DH8Mrlq1yrpe+vPSKEl69913bfqtXr3aZtnd3V1NmjTRoUOHVLNmTYWGhmZ4/f3/Hc+uNWvW2Nzkf+rUKe3du9c6612VKlVUqVIl/fDDD5keNzQ0NMfPsapSpYpKly6t1atX29SQlJSkdevWWWfae5AsFosMw7COJqRbsmRJhtngsisiIkKOjo5auHBhln0aNmyowoUL68iRI1m+r+kjO5lp1aqVfvvtN3399ddZ9snu9/B+uXbtmjZs2GDTtnr1ajk4OOiJJ56Q9Of77eTkJEdHR2ufmzdv6u23375vdRQuXFgdO3bU4MGDFR8fr5iYGHl4eOixxx7TRx99ZDNilJaWpnfeeUdlypThuWIA7htGnABA0uuvv27ap3v37nrjjTfUo0cPxcTEqEaNGtq9e7emTJmi1q1bW+9LiYiI0BNPPKHRo0crKSlJoaGh2rNnT6Z/RM6ZM0ePP/64GjVqpIEDByowMFDXrl3T77//rk8//fSuf0DfTVxcnJ599ln169dPCQkJGjdunFxdXRUVFWXt8+abb6pVq1Zq2bKlevbsqdKlSys+Pl5Hjx7VwYMH9cEHH+To2A4ODpo+fbqee+45tW3bVv3791dycrJmzJihq1evZuu9zi1vb2898cQTmjFjhooVK6bAwEDt2LFDS5cuVeHChXO0z8DAQL3yyit67bXXdPPmTXXp0kU+Pj46cuSILl26pAkTJsjT01Pz5s1Tjx49FB8fr44dO6pEiRK6ePGifvjhB128ePGuwWv48OFau3atnn76ab388suqV6+ebt68qR07dqht27Zq0qRJtr+H94uvr68GDhyo06dPq3Llytq4caPeeustDRw40HpZYZs2bTR79mx17dpV//nPf3T58mXNnDkzQ3C9V+3atVP16tUVGhqq4sWL69SpU4qOjla5cuVUqVIlSdLUqVPVokULNWnSRCNHjlShQoW0YMEC/fzzz1qzZs19H80EUHARnAAgm1xdXbVt2zaNGTNGM2bM0MWLF1W6dGmNHDlS48aNs/ZzcHDQhg0bFBkZqenTp+v27dtq2LChNm7cqEceecRmn1WrVtXBgwf12muv6b///a/i4uJUuHBhVapUSa1bt85xrVOmTNH333+vXr16KTExUfXq1dN7771n83DRJk2a6LvvvtPkyZM1fPhwXblyRb6+vqpatao6deqU42NLf95f5OHhoalTp6pz585ydHRU/fr1tW3bNjVo0CBX+86u1atX64UXXtDo0aOVkpKihg0bauvWrWrTpk2O9zlx4kRVqlRJ8+bN03PPPScnJydVqlRJw4YNs/Z5/vnnVbZsWU2fPl39+/fXtWvXVKJECdWuXdvmGUWZ8fLy0u7duzV+/HgtXrxYEyZMUJEiRVS3bl395z//kZT97+H9UrJkSb3xxhsaOXKkfvrpJxUtWlSvvPKKzXT5TZs21bJlyzRt2jS1a9dOpUuXVr9+/VSiRAn16dMnx8du0qSJ1q1bpyVLligxMVElS5ZUixYt9Oqrr8rZ2VnSnxO8fP311xo3bpx69uyptLQ01apVSxs2bFDbtm1zff4AkM5iGH97YAMAIN/avn27mjRpog8++OCuExQA2REeHq5Lly7p559/tncpAGB33OMEAAAAACYITgAAAABggkv1AAAAAMAEI04AAAAAYILgBAAAAAAm7Bqcdu7cqXbt2qlUqVKyWCxZPmE83UcffaQWLVqoePHi8vb2VlhYmDZv3pw3xQIAAAAosOz6HKekpCTVqlVLvXr1UocOHUz779y5Uy1atNCUKVNUuHBhLV++XO3atdO+ffsUEhKSrWOmpaXp/Pnz8vLy4qF4AAAAQAFmGIauXbumUqVKycHh7mNKD83kEBaLRevXr9czzzxzT9tVq1ZNnTt31tixYzNdn5ycrOTkZOvyuXPnVLVq1dyUCgAAAOAf5MyZMypTpsxd+9h1xCm30tLSdO3aNRUtWjTLPlOnTrV5unm6M2fOyNvb+0GWBwAAAOAhlpiYqICAAHl5eZn2zdfBadasWUpKSlKnTp2y7BMVFaXIyEjrcvqb4+3tTXACAAAAkK1bePJtcFqzZo3Gjx+vTz75RCVKlMiyn4uLi1xcXPKwMgAAAAD/NPkyOK1du1Z9+vTRBx98oObNm9u7HAAAAAD/cPnuOU5r1qxRz549tXr1arVp08be5QAAAAAoAOw64nT9+nX9/vvv1uWTJ0/q8OHDKlq0qMqWLauoqCidO3dOq1atkvRnaOrevbvmzJmj+vXr68KFC5IkNzc3+fj42OUcAAAAkDcMw1BKSopSU1PtXQryEWdnZzk6OuZ6P3adjnz79u1q0qRJhvYePXpoxYoV6tmzp2JiYrR9+3ZJUnh4uHbs2JFl/+xITEyUj4+PEhISmBwCAAAgn7h9+7ZiY2N148YNe5eCfMZisahMmTLy9PTMsO5essFD8xynvEJwAgAAyF/S0tJ07NgxOTo6qnjx4ipUqFC2ZkEDDMPQxYsXdePGDVWqVCnDyNO9ZIN8OTkEAAAACo7bt28rLS1NAQEBcnd3t3c5yGeKFy+umJgY3blzJ1eX7OW7ySEAAABQMDk48Kcr7t39Gp3k2wcAAAAAJghOAAAAAGCC4AQAAAD8Q4WHh2v48OHZ7r9ixQoVLlz4gdWTnxGcAAAAAMAEwQkAAAAATBCcAAAAgDwWHh6uoUOHavjw4SpSpIj8/Py0ePFiJSUlqVevXvLy8lKFChX0xRdfWLfZsWOH6tWrJxcXF/n7++vll19WSkqKdX1SUpK6d+8uT09P+fv7a9asWRmOe/v2bY0ePVqlS5eWh4eHHnvsMW3fvj0vTjnfIzgBAAAAdrBy5UoVK1ZM3333nYYOHaqBAwfqX//6lxo0aKCDBw+qZcuW6tatm27cuKFz586pdevWqlu3rn744QctXLhQS5cu1aRJk6z7GzVqlLZt26b169dry5Yt2r59uw4cOGBzzF69emnPnj1677339OOPP+pf//qXnnzySR07diyvTz/fsRiGYdi7iLx0L08HBgAAgP3dunVLJ0+eVFBQkFxdXe1dzn0RHh6u1NRU7dq1S5KUmpoqHx8ftW/fXqtWrZIkXbhwQf7+/vrmm2/06aefat26dTp69Kj1uUQLFizQSy+9pISEBN24cUO+vr5atWqVOnfuLEmKj49XmTJl9J///EfR0dE6fvy4KlWqpLNnz6pUqVLWWpo3b6569eppypQpWrFihYYPH66rV6/m7RvyAN3t+3Mv2cDpQRYJAAAAIHM1a9a0/tvR0VG+vr6qUaOGtc3Pz0+SFBcXp6NHjyosLMzmYa4NGzbU9evXdfbsWV25ckW3b99WWFiYdX3RokVVpUoV6/LBgwdlGIYqV65sU0dycrJ8fX3v+/n90xCcAAAAADtwdna2WbZYLDZt6SEpLS1NhmHYhCZJSr9wzGKxKDsXkaWlpcnR0VEHDhyQo6OjzTpPT88cnUNBQnACAAAAHnJVq1bVunXrbALU3r175eXlpdKlS6tIkSJydnbWt99+q7Jly0qSrly5ot9++02NGzeWJIWEhCg1NVVxcXFq1KiR3c4lvyI4wS4CX/7c3iXkWszrbexdAgAAKCAGDRqk6OhoDR06VEOGDNGvv/6qcePGKTIyUg4ODvL09FSfPn00atQo+fr6ys/PT2PGjJGDw/+fC65y5cp67rnn1L17d82aNUshISG6dOmSvv76a9WoUUOtW7e24xk+/AhOAAAAwEOudOnS2rhxo0aNGqVatWqpaNGi6tOnj/773/9a+8yYMUPXr1/XU089JS8vL40YMUIJCQk2+1m+fLkmTZqkESNG6Ny5c/L19VVYWBihKRuYVQ92wYgTAADIrn/irHrIO8yqByBXaqysYd7pIfdTj5/sXQIAACggeAAuAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACR6ACwAAgHwr8OXP8/R4Ma+3uaf+4eHhql27tqKjox9MQcgzjDgBAAAAgAmCEwAAAACYIDgBAAAAeWTTpk3y8fHRqlWr1LNnTz3zzDOaOXOm/P395evrq8GDB+vOnTvW/oGBgZoyZYp69+4tLy8vlS1bVosXL7bjGRRcBCcAAAAgD7z33nvq1KmTVq1ape7du0uStm3bpuPHj2vbtm1auXKlVqxYoRUrVthsN2vWLIWGhurQoUMaNGiQBg4cqF9++cUOZ1CwEZwAAACAB2zBggUaMGCAPvnkEz399NPW9iJFimj+/Pl65JFH1LZtW7Vp00ZfffWVzbatW7fWoEGDVLFiRb300ksqVqyYtm/fnsdnAGbVAwAAAB6gdevW6Y8//tDu3btVr149m3XVqlWTo6Ojddnf318//fSTTZ+aNWta/22xWFSyZEnFxcU92KKRASNOAAAAwANUu3ZtFS9eXMuXL5dhGDbrnJ2dbZYtFovS0tLuuQ8ePIITAAAA8ABVqFBB27Zt0yeffKKhQ4fauxzkEJfqAQAAAA9Y5cqVtW3bNoWHh8vJyYkH4uZDBCcAAADkWzGvt7F3CdlWpUoVff311woPD7e5rwn5A8EJAAAAeED+PvtdcHCw/vjjjyz7/30kKiYmJkOfw4cP574w3DPucQIAAAAAE4w4Aci3jj4SbO8SciX4l6P2LgEAAGQTI04AAAAAYILgBAAAAAAmCE4AAAAAYIJ7nICcGu9j7wpyJ6isvSsAAADINxhxAgAAAAATBCcAAAAAMEFwAgAAAAAT3OMEAACA/Cuv7zken3BP3cPDw1W7dm1FR0c/mHqQZxhxAgAAAAATBCcAAADADm7fvm3vEnAPCE4AAABAHggMDNSkSZPUs2dP+fj4qF+/fvYuCfeA4AQAAADkkRkzZqh69eo6cOCAXn31VXuXg3vA5BAAAABAHmnatKlGjhxp7zKQA4w4AQAAAHkkNDTU3iUghwhOAAAAQB7x8PCwdwnIIYITAAAAAJggOAEAAACACSaHAAAAQP41PsHeFaCAIDgBAAAAD8j27dut/46JibFbHcg9LtUDAAAAABMEJwAAAAAwQXACAAAAABMEJwAAAAAwQXACAAAAABMEJwAAAAAwQXACAAAAABN2DU47d+5Uu3btVKpUKVksFn388cem2+zYsUN16tSRq6urypcvr0WLFj34QgEAAAAUaHYNTklJSapVq5bmz5+frf4nT55U69at1ahRIx06dEivvPKKhg0bpnXr1j3gSgEAAAAUZE72PHirVq3UqlWrbPdftGiRypYtq+joaElScHCw9u/fr5kzZ6pDhw4PqEoAAAA8rGqsrJGnx/upx095erx70bNnT129ejVbV3HlhmEY6t+/vz788ENduXJFhw4dUu3atR/oMR8G+eoep2+++UYRERE2bS1bttT+/ft1586dTLdJTk5WYmKizQsAAABAzmzatEkrVqzQZ599ptjYWFWvXt0udcTExMhisejw4cN5crx8FZwuXLggPz8/mzY/Pz+lpKTo0qVLmW4zdepU+fj4WF8BAQF5USoAAACQwe3bt+1dQq4dP35c/v7+atCggUqWLCknp3u/iM0wDKWkpDyA6h6cfBWcJMlisdgsG4aRaXu6qKgoJSQkWF9nzpx54DUCAAAAkhQeHq4hQ4YoMjJSxYoVU4sWLTR79mzVqFFDHh4eCggI0KBBg3T9+nXrNitWrFDhwoW1efNmBQcHy9PTU08++aRiY2OtfVJTUxUZGanChQvL19dXo0ePtv5dnC45OVnDhg1TiRIl5Orqqscff1zff/+9df327dtlsVi0efNmhYSEyM3NTU2bNlVcXJy++OILBQcHy9vbW126dNGNGzck/Xk54NChQ3X69GlZLBYFBgbe87FCQ0Pl4uKiXbt2yTAMTZ8+XeXLl5ebm5tq1aqlDz/80LrdlStX9Nxzz6l48eJyc3NTpUqVtHz5cklSUFCQJCkkJEQWi0Xh4eH350PLQr4KTiVLltSFCxds2uLi4uTk5CRfX99Mt3FxcZG3t7fNCwAAAMgrK1eulJOTk/bs2aM333xTDg4Omjt3rn7++WetXLlSX3/9tUaPHm2zzY0bNzRz5ky9/fbb2rlzp06fPq2RI0da18+aNUvLli3T0qVLtXv3bsXHx2v9+vU2+xg9erTWrVunlStX6uDBg6pYsaJatmyp+Ph4m37jx4/X/PnztXfvXp05c0adOnVSdHS0Vq9erc8//1xbt27VvHnzJElz5szRxIkTVaZMGcXGxlrDUXaPNXr0aE2dOlVHjx5VzZo19d///lfLly/XwoUL9b///U8vvviinn/+ee3YsUOS9Oqrr+rIkSP64osvdPToUS1cuFDFihWTJH333XeSpC+//FKxsbH66KOPcvtR3ZVdJ4e4V2FhYfr0009t2rZs2aLQ0FA5OzvbqSoAAAAgaxUrVtT06dOty4888oj130FBQXrttdc0cOBALViwwNp+584dLVq0SBUqVJAkDRkyRBMnTrSuj46OVlRUlHWCtEWLFmnz5s3W9UlJSVq4cKFWrFhhnYztrbfe0tatW7V06VKNGjXK2nfSpElq2LChJKlPnz6KiorS8ePHVb58eUlSx44dtW3bNr300kvy8fGRl5eXHB0dVbJkyXs+1sSJE9WiRQvrdrNnz9bXX3+tsLAwSVL58uW1e/duvfnmm2rcuLFOnz6tkJAQhYaGSpJ1hEuSihcvLkny9fW11vIg2TU4Xb9+Xb///rt1+eTJkzp8+LCKFi2qsmXLKioqSufOndOqVaskSQMGDND8+fMVGRmpfv366ZtvvtHSpUu1Zs0ae50CAAAAcFfpf/Sn27Ztm6ZMmaIjR44oMTFRKSkpunXrlpKSkuTh4SFJcnd3t4YmSfL391dcXJwkKSEhQbGxsdawIUlOTk4KDQ21Xq53/Phx3blzxxqIJMnZ2Vn16tXT0aNHbeqpWbOm9d9+fn5yd3e3hqb0tvTRnczcy7H++l4cOXJEt27dsgapdLdv31ZISIgkaeDAgerQoYMOHjyoiIgIPfPMM2rQoEGWtTxIdg1O+/fvV5MmTazLkZGRkqQePXpoxYoVio2N1enTp63rg4KCtHHjRr344ot64403VKpUKc2dO5epyAEAAPDQSg9DknTq1Cm1bt1aAwYM0GuvvaaiRYtq9+7d6tOnj80s0X+/mspisWS4h+luspoHwDCMDG1/PZbFYsn02GlpafflWH99L9L3+fnnn6t06dI2/VxcXCT9+fiiU6dO6fPPP9eXX36pZs2aafDgwZo5c2aW9Twodr3HKTw8XIZhZHitWLFC0p83xm3fvt1mm8aNG+vgwYNKTk7WyZMnNWDAgLwvHAAAAMiB/fv3KyUlRbNmzVL9+vVVuXJlnT9//p724ePjI39/f3377bfWtpSUFB04cMC6XLFiRRUqVEi7d++2tt25c0f79+9XcHBw7k/kL3J6rKpVq8rFxUWnT59WxYoVbV5/nQm7ePHi6tmzp9555x1FR0dr8eLFkqRChQpJ+nOijLyQr+5xAgAAAPKzChUqKCUlRfPmzVO7du20Z88eLVq06J7388ILL+j1119XpUqVFBwcrNmzZ+vq1avW9R4eHho4cKBGjRplvQ1m+vTpunHjhvr06XMfzyjnx/Ly8tLIkSP14osvKi0tTY8//rgSExO1d+9eeXp6qkePHho7dqzq1KmjatWqKTk5WZ999pk1jJUoUUJubm7atGmTypQpI1dXV/n4+NzXc/srghMAAADyrZ96/GTvEu5J7dq1NXv2bE2bNk1RUVF64oknNHXqVHXv3v2e9jNixAjFxsaqZ8+ecnBwUO/evfXss88qISHB2uf1119XWlqaunXrpmvXrik0NFSbN29WkSJF7vdp5fhYr732mkqUKKGpU6fqxIkTKly4sB599FG98sorkv4cVYqKilJMTIzc3NzUqFEjvffee5L+vK9r7ty5mjhxosaOHatGjRpluFrtfrIY93Kx5D9AYmKifHx8lJCQwNTkdhT48uf2LiHXYly72ruEXKkRVNbeJeTa+1Pz14Pz/i74l6PmnQAAunXrlk6ePKmgoCC5urrauxzkM3f7/txLNshXz3ECAAAAAHsgOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJhwsncBAAAAQE4dfSQ4T48X/MvRXO/DYrFo/fr1euaZZzJdHxMTo6CgIB06dEi1a9fO9fFwfxCcAAAAgDwUGxurIkWK2LsM3COCEwAAAJBHbt++rZIlS9q7DOQA9zgBAAAAD0h4eLiGDBmiyMhIFStWTC1atJDFYtHHH39s7fPdd98pJCRErq6uCg0N1aFDhzLsZ8OGDapUqZLc3NzUpEkTrVy5UhaLRVevXrX22bt3r5544gm5ubkpICBAw4YNU1JSUh6cZcFAcAIAAAAeoJUrV8rJyUl79uzRm2++abMuKSlJbdu2VZUqVXTgwAGNHz9eI0eOtOkTExOjjh076plnntHhw4fVv39/jRkzxqbPTz/9pJYtW6p9+/b68ccftXbtWu3evVtDhgx54OdXUHCpHgAAAPAAVaxYUdOnT8903bvvvqvU1FQtW7ZM7u7uqlatms6ePauBAwda+yxatEhVqlTRjBkzJElVqlTRzz//rMmTJ1v7zJgxQ127dtXw4cMlSZUqVdLcuXPVuHFjLVy4UK6urg/uBAsIghMAAADwAIWGhma57ujRo6pVq5bc3d2tbWFhYTZ9fv31V9WtW9emrV69ejbLBw4c0O+//653333X2mYYhtLS0nTy5EkFB+ft7IP/RAQnAAAA4AHy8PDIcp1hGKbbG4Yhi8Vy1+3S0tLUv39/DRs2LMP2ZcuWzWaluBuCEwAAAGAnVatW1dtvv62bN2/Kzc1NkvTtt9/a9HnkkUe0ceNGm7b9+/fbLD/66KP63//+p4oVKz7YggswJocAAAAA7KRr165ycHBQnz59dOTIEW3cuFEzZ8606dO/f3/98ssveumll/Tbb7/p/fff14oVKyTJOhL10ksv6ZtvvtHgwYN1+PBhHTt2TBs2bNDQoUPz+pT+sRhxAgAAQL4V/MtRe5eQK56envr00081YMAAhYSEqGrVqpo2bZo6dOhg7RMUFKQPP/xQI0aM0Jw5cxQWFqYxY8Zo4MCBcnFxkSTVrFlTO3bs0JgxY9SoUSMZhqEKFSqoc+fO9jq1fxyCEwAAAPCAbN++PUPb3+9Pql+/vg4fPnzXPk899ZSeeuop6/LkyZNVpkwZm9ny6tatqy1btuS+aGSK4AQAAAA85BYsWKC6devK19dXe/bs0YwZM3hGUx4jOAEAAAAPuWPHjmnSpEmKj49X2bJlNWLECEVFRdm7rAKF4AQAAAA85P7v//5P//d//2fvMgo0ZtUDAAAAABMEJwAAAOQL2XlYLPB39+t7Q3ACAADAQ83Z2VmSdOPGDTtXgvzo9u3bkiRHR8dc7Yd7nAAAAPBQc3R0VOHChRUXFydJcnd3tz74FbibtLQ0Xbx4Ue7u7nJyyl30ITgBAADgoVeyZElJsoYnILscHBxUtmzZXIdtghMAAAAeehaLRf7+/ipRooTu3Llj73KQjxQqVEgODrm/Q4ngBAAAgHzD0dEx1/eqADnB5BAAAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAm7B6cFixYoKCgILm6uqpOnTratWvXXfu/++67qlWrltzd3eXv769evXrp8uXLeVQtAAAAgILIrsFp7dq1Gj58uMaMGaNDhw6pUaNGatWqlU6fPp1p/927d6t79+7q06eP/ve//+mDDz7Q999/r759++Zx5QAAAAAKErsGp9mzZ6tPnz7q27evgoODFR0drYCAAC1cuDDT/t9++60CAwM1bNgwBQUF6fHHH1f//v21f//+PK4cAAAAQEFit+B0+/ZtHThwQBERETbtERER2rt3b6bbNGjQQGfPntXGjRtlGIb++OMPffjhh2rTpk2Wx0lOTlZiYqLNCwAAAADuhd2C06VLl5Samio/Pz+bdj8/P124cCHTbRo0aKB3331XnTt3VqFChVSyZEkVLlxY8+bNy/I4U6dOlY+Pj/UVEBBwX88DAAAAwD+f3SeHsFgsNsuGYWRoS3fkyBENGzZMY8eO1YEDB7Rp0yadPHlSAwYMyHL/UVFRSkhIsL7OnDlzX+sHAAAA8M/nZK8DFytWTI6OjhlGl+Li4jKMQqWbOnWqGjZsqFGjRkmSatasKQ8PDzVq1EiTJk2Sv79/hm1cXFzk4uJy/08AAAAAQIFhtxGnQoUKqU6dOtq6datN+9atW9WgQYNMt7lx44YcHGxLdnR0lPTnSBUAAAAAPAh2vVQvMjJSS5Ys0bJly3T06FG9+OKLOn36tPXSu6ioKHXv3t3av127dvroo4+0cOFCnThxQnv27NGwYcNUr149lSpVyl6nAQAAAOAfzm6X6klS586ddfnyZU2cOFGxsbGqXr26Nm7cqHLlykmSYmNjbZ7p1LNnT127dk3z58/XiBEjVLhwYTVt2lTTpk2z1ykAAAAAKAAsRgG7xi0xMVE+Pj5KSEiQt7e3vcspsAJf/tzeJeRajGtXe5eQKzWCytq7hFx7f2qKvUvIleBfjtq7BAAACrR7yQZ2n1UPAAAAAB52BCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATOQ5OV69e1ZIlSxQVFaX4+HhJ0sGDB3Xu3Ln7VhwAAAAAPAyccrLRjz/+qObNm8vHx0cxMTHq16+fihYtqvXr1+vUqVNatWrV/a4TAAAAAOwmRyNOkZGR6tmzp44dOyZXV1dre6tWrbRz5877VhwAAAAAPAxyFJy+//579e/fP0N76dKldeHChVwXBQAAAAAPkxwFJ1dXVyUmJmZo//XXX1W8ePFcFwUAAAAAD5McBaenn35aEydO1J07dyRJFotFp0+f1ssvv6wOHTrc1wIBAAAAwN5yFJxmzpypixcvqkSJErp586YaN26sihUrysvLS5MnT77fNQIAAACAXeVoVj1vb2/t3r1bX3/9tQ4ePKi0tDQ9+uijat68+f2uDwAAAADs7p6DU0pKilxdXXX48GE1bdpUTZs2fRB1AQAAAMBD454v1XNyclK5cuWUmpr6IOoBAAAAgIdOju5x+u9//6uoqCjFx8ff73oAAAAA4KGTo3uc5s6dq99//12lSpVSuXLl5OHhYbP+4MGD96U4AAAAAHgY5Cg4PfPMM/e5DAAAAAB4eOUoOI0bN+6+FbBgwQLNmDFDsbGxqlatmqKjo9WoUaMs+ycnJ2vixIl65513dOHCBZUpU0ZjxoxR796971tNAAAAAPBXOQpO6Q4cOKCjR4/KYrGoatWqCgkJuaft165dq+HDh2vBggVq2LCh3nzzTbVq1UpHjhxR2bJlM92mU6dO+uOPP7R06VJVrFhRcXFxSklJyc1pAAAAAMBd5Sg4xcXF6d///re2b9+uwoULyzAMJSQkqEmTJnrvvfdUvHjxbO1n9uzZ6tOnj/r27StJio6O1ubNm7Vw4UJNnTo1Q/9NmzZpx44dOnHihIoWLSpJCgwMzMkpAAAAAEC25WhWvaFDhyoxMVH/+9//FB8frytXrujnn39WYmKihg0blq193L59WwcOHFBERIRNe0REhPbu3ZvpNhs2bFBoaKimT5+u0qVLq3Llyho5cqRu3ryZ5XGSk5OVmJho8wIAAACAe5GjEadNmzbpyy+/VHBwsLWtatWqeuONNzIEoaxcunRJqamp8vPzs2n38/PThQsXMt3mxIkT2r17t1xdXbV+/XpdunRJgwYNUnx8vJYtW5bpNlOnTtWECROyeWYAAAAAkFGORpzS0tLk7Oycod3Z2VlpaWn3tC+LxWKzbBhGhra/Htdisejdd99VvXr11Lp1a82ePVsrVqzIctQpKipKCQkJ1teZM2fuqT4AAAAAyFFwatq0qV544QWdP3/e2nbu3Dm9+OKLatasWbb2UaxYMTk6OmYYXYqLi8swCpXO399fpUuXlo+Pj7UtODhYhmHo7NmzmW7j4uIib29vmxcAAAAA3IscBaf58+fr2rVrCgwMVIUKFVSxYkUFBQXp2rVrmjdvXrb2UahQIdWpU0dbt261ad+6dasaNGiQ6TYNGzbU+fPndf36dWvbb7/9JgcHB5UpUyYnpwIAAAAApnJ0j1NAQIAOHjyorVu36pdffpFhGKpataqaN29+T/uJjIxUt27dFBoaqrCwMC1evFinT5/WgAEDJP15md25c+e0atUqSVLXrl312muvqVevXpowYYIuXbqkUaNGqXfv3nJzc8vJqQAAAACAqVw9x6lFixZq0aJFjrfv3LmzLl++rIkTJyo2NlbVq1fXxo0bVa5cOUlSbGysTp8+be3v6emprVu3aujQoQoNDZWvr686deqkSZMm5eY0AAAAAOCuLIZhGPe60bBhw1SxYsUMU4/Pnz9fv//+u6Kjo+9XffddYmKifHx8lJCQwP1OdhT48uf2LiHXYly72ruEXKkRlPlDpvOT96fm74dfB/9y1N4lAABQoN1LNsjRPU7r1q1Tw4YNM7Q3aNBAH374YU52CQAAAAAPrRwFp8uXL9vMbJfO29tbly5dynVRAAAAAPAwyVFwqlixojZt2pSh/YsvvlD58uVzXRQAAAAAPExyNDlEZGSkhgwZoosXL6pp06aSpK+++kozZ87UnDlz7muBAAAAAGBvOQpOvXv3VnJysiZPnqzXXntNkhQUFKRFixape/fu97VAAAAAALC3HF2qd/PmTfXo0UNnz57VH3/8oR9//FFDhgyRn5/f/a4PAAAAAOwuR8Hp6aeftj6U1tnZWc2bN9fs2bP1zDPPaOHChfe1QAAAAACwtxwFp4MHD6pRo0aSpA8//FB+fn46deqUVq1apblz597XAgEAAADA3nIUnG7cuCEvLy9J0pYtW9S+fXs5ODiofv36OnXq1H0tEAAAAADsLcfTkX/88cc6c+aMNm/erIiICElSXFyc6RN3AQAAACC/yVFwGjt2rEaOHKnAwEA99thjCgsLk/Tn6FNISMh9LRAAAAAA7C1H05F37NhRjz/+uGJjY1WrVi1re7NmzfTss8/et+IAAAAA4GGQo+AkSSVLllTJkiVt2urVq5frggAAAADgYZOjS/UAAAAAoCAhOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACSd7FwAAAJBvjfexdwW5Nz7B3hUA+QIjTgAAAABgguAEAAAAACYITgAAAABggnucAAAAkG8dfSTY3iXkWvAvR+1dArKBEScAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMEFwAgAAAAATBCcAAAAAMOFk7wIAAABgPzVW1rB3Cbnyvr0LQIHBiBMAAAAAmCA4AQAAAIAJLtUDAAB2E/jy5/YuIVdiXO1dAYC8wogTAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACYITAAAAAJggOAEAAACACbsHpwULFigoKEiurq6qU6eOdu3ala3t9uzZIycnJ9WuXfvBFggAAACgwLNrcFq7dq2GDx+uMWPG6NChQ2rUqJFatWql06dP33W7hIQEde/eXc2aNcujSgEAAAAUZHYNTrNnz1afPn3Ut29fBQcHKzo6WgEBAVq4cOFdt+vfv7+6du2qsLCwPKoUAAAAQEFmt+B0+/ZtHThwQBERETbtERER2rt3b5bbLV++XMePH9e4ceOydZzk5GQlJibavAAAAADgXtgtOF26dEmpqany8/Ozaffz89OFCxcy3ebYsWN6+eWX9e6778rJySlbx5k6dap8fHysr4CAgFzXDgAAAKBgsfvkEBaLxWbZMIwMbZKUmpqqrl27asKECapcuXK29x8VFaWEhATr68yZM7muGQAAAEDBkr1hmwegWLFicnR0zDC6FBcXl2EUSpKuXbum/fv369ChQxoyZIgkKS0tTYZhyMnJSVu2bFHTpk0zbOfi4iIXF5cHcxIAAAAACgS7jTgVKlRIderU0datW23at27dqgYNGmTo7+3trZ9++kmHDx+2vgYMGKAqVaro8OHDeuyxx/KqdAAAAAAFjN1GnCQpMjJS3bp1U2hoqMLCwrR48WKdPn1aAwYMkPTnZXbnzp3TqlWr5ODgoOrVq9tsX6JECbm6umZoBwAAAID7ya7BqXPnzrp8+bImTpyo2NhYVa9eXRs3blS5cuUkSbGxsabPdAIAAACAB82uwUmSBg0apEGDBmW6bsWKFXfddvz48Ro/fvz9LwoAAAAA/sLus+oBAAAAwMOO4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJghOAAAAAGCC4AQAAAAAJuwenBYsWKCgoCC5urqqTp062rVrV5Z9P/roI7Vo0ULFixeXt7e3wsLCtHnz5jysFgAAAEBBZNfgtHbtWg0fPlxjxozRoUOH1KhRI7Vq1UqnT5/OtP/OnTvVokULbdy4UQcOHFCTJk3Url07HTp0KI8rBwAAAFCQ2DU4zZ49W3369FHfvn0VHBys6OhoBQQEaOHChZn2j46O1ujRo1W3bl1VqlRJU6ZMUaVKlfTpp5/mceUAAAAAChK7Bafbt2/rwIEDioiIsGmPiIjQ3r17s7WPtLQ0Xbt2TUWLFs2yT3JyshITE21eAAAAAHAv7BacLl26pNTUVPn5+dm0+/n56cKFC9nax6xZs5SUlKROnTpl2Wfq1Kny8fGxvgICAnJVNwAAAICCx+6TQ1gsFptlwzAytGVmzZo1Gj9+vNauXasSJUpk2S8qKkoJCQnW15kzZ3JdMwAAAICCxcleBy5WrJgcHR0zjC7FxcVlGIX6u7Vr16pPnz764IMP1Lx587v2dXFxkYuLS67rBQAAAFBw2W3EqVChQqpTp462bt1q075161Y1aNAgy+3WrFmjnj17avXq1WrTps2DLhMAAAAA7DfiJEmRkZHq1q2bQkNDFRYWpsWLF+v06dMaMGCApD8vszt37pxWrVol6c/Q1L17d82ZM0f169e3jla5ubnJx8fHbucBAAAA4J/NrsGpc+fOunz5siZOnKjY2FhVr15dGzduVLly5SRJsbGxNs90evPNN5WSkqLBgwdr8ODB1vYePXpoxYoVeV0+AAAAgALCrsFJkgYNGqRBgwZluu7vYWj79u0PviAAAAAA+Bu7z6oHAAAAAA87ghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmCA4AQAAAIAJghMAAAAAmLB7cFqwYIGCgoLk6uqqOnXqaNeuXXftv2PHDtWpU0eurq4qX768Fi1alEeVAgAAACio7Bqc1q5dq+HDh2vMmDE6dOiQGjVqpFatWun06dOZ9j958qRat26tRo0a6dChQ3rllVc0bNgwrVu3Lo8rBwAAAFCQ2DU4zZ49W3369FHfvn0VHBys6OhoBQQEaOHChZn2X7RokcqWLavo6GgFBwerb9++6t27t2bOnJnHlQMAAAAoSJzsdeDbt2/rwIEDevnll23aIyIitHfv3ky3+eabbxQREWHT1rJlSy1dulR37tyRs7Nzhm2Sk5OVnJxsXU5ISJAkJSYm5vYUkAtpyTfsXUKuJVoMe5eQK6k3U+1dQq5dT83f58D/DgH5//cgv/8WSPn/9yC//xZI/B7YU/p7bxjm/y3bLThdunRJqamp8vPzs2n38/PThQsXMt3mwoULmfZPSUnRpUuX5O/vn2GbqVOnasKECRnaAwICclE9IPnYu4BcO2rvAnKtnr0LyC2f/P8tAgq6f8Z/xfn79yDf/xZI/B48BK5duyYfk8/BbsEpncVisVk2DCNDm1n/zNrTRUVFKTIy0rqclpam+Ph4+fr63vU4wD9ZYmKiAgICdObMGXl7e9u7HACAnfB7gILOMAxdu3ZNpUqVMu1rt+BUrFgxOTo6ZhhdiouLyzCqlK5kyZKZ9ndycpKvr2+m27i4uMjFxcWmrXDhwjkvHPgH8fb25ocSAMDvAQo0s5GmdHabHKJQoUKqU6eOtm7datO+detWNWjQINNtwsLCMvTfsmWLQkNDM72/CQAAAADuB7vOqhcZGaklS5Zo2bJlOnr0qF588UWdPn1aAwYMkPTnZXbdu3e39h8wYIBOnTqlyMhIHT16VMuWLdPSpUs1cuRIe50CAAAAgALArvc4de7cWZcvX9bEiRMVGxur6tWra+PGjSpXrpwkKTY21uaZTkFBQdq4caNefPFFvfHGGypVqpTmzp2rDh062OsUgHzJxcVF48aNy3AZKwCgYOH3AMg+i5GdufcAAAAAoACz66V6AAAAAJAfEJwAAAAAwATBCQAAAABMEJwAAAAAwATBCQAAoIBavHixmjdvrlq1aqlly5aKj4+3d0nAQ4vgBPxD3L59294lAADymR49eujLL7/UDz/8oNTUVO3bt8/eJQEPLYITkE+Fh4dryJAhioyMVLFixdSiRQtZLBZt3rxZISEhcnNzU9OmTRUXF6cvvvhCwcHB8vb2VpcuXXTjxg3rfj788EPVqFFDbm5u8vX1VfPmzZWUlGRdv3z5cgUHB8vV1VWPPPKIFixYYI/TBQDcB3//7YiIiJAkLVu2TMWLF9eTTz5p5wqBh5ddH4ALIHdWrlypgQMHas+ePdq2bZt27typ8ePHa/78+XJ3d1enTp3UqVMnubi4aPXq1bp+/bqeffZZzZs3Ty+99JJiY2PVpUsXTZ8+Xc8++6yuXbumXbt2Kf3xbm+99ZbGjRun+fPnKyQkRIcOHVK/fv3k4eGhHj162PnsAQA58dffjuTkZL3wwgtyd3fXO++8I4vFYu/ygIcWD8AF8qnw8HAlJCTo0KFDkqTt27erSZMm+vLLL9WsWTNJ0uuvv66oqCgdP35c5cuXlyQNGDBAMTEx2rRpkw4ePKg6deooJiZG5cqVy3CMsmXLatq0aerSpYu1bdKkSdq4caP27t2bB2cJALif/v7b8cILL2jlypV65JFHJEkjR45Ux44d7Vki8NBixAnIx0JDQzO01axZ0/pvPz8/ubu7W0NTett3330nSapVq5aaNWumGjVqqGXLloqIiFDHjh1VpEgRXbx4UWfOnFGfPn3Ur18/6/YpKSny8fF5gGcFAHiQ/vrbMWfOHM2ZM8eO1QD5B8EJyMc8PDwytDk7O1v/bbFYbJbT29LS0iRJjo6O2rp1q/bu3astW7Zo3rx5GjNmjPbt2yd3d3dJf16u99hjj9nsw9HR8X6fCgAgj2T22wHAHJNDAAWcxWJRw4YNNWHCBB06dEiFChXS+vXr5efnp9KlS+vEiROqWLGizSsoKMjeZQMAAOQpRpyAAmzfvn366quvFBERoRIlSmjfvn26ePGigoODJUnjx4/XsGHD5O3trVatWik5OVn79+/XlStXFBkZaefqAQAA8g7BCSjAvL29tXPnTkVHRysxMVHlypXTrFmz1KpVK0lS37595e7urhkzZmj06NHy8PBQjRo1NHz4cPsWDgAAkMeYVQ8AAAAATHCPEwAAAACYIDgBAAAAgAmCEwAAAACYIDgBAAAAgAmCEwAAAACYIDgBAAAAgAmCEwAAAACYIDgBAAAAgAmCEwAA2TB+/HjVrl3b3mUAAOzEYhiGYe8iAACwh/DwcNWuXVvR0dGmfa9fv67k5GT5+vo++MIAAA8dJ3sXAADAw8wwDKWmpsrT01Oenp72LgcAYCdcqgcAyBfCw8M1dOhQDR8+XEWKFJGfn58WL16spKQk9erVS15eXqpQoYK++OIL6zZHjhxR69at5enpKT8/P3Xr1k2XLl2SJPXs2VM7duzQnDlzZLFYZLFYFBMTo+3bt8tisWjz5s0KDQ2Vi4uLdu3alemlesuWLVO1atXk4uIif39/DRkyJC/fEgBAHiI4AQDyjZUrV6pYsWL67rvvNHToUA0cOFD/+te/1KBBAx08eFAtW7ZUt27ddOPGDcXGxqpx48aqXbu29u/fr02bNumPP/5Qp06dJElz5sxRWFiY+vXrp9jYWMXGxiogIMB6rNGjR2vq1Kk6evSoatasmaGWhQsXavDgwfrPf/6jn376SRs2bFDFihXz7L0AAOQt7nECAOQL4eHhSk1N1a5duyRJqamp8vHxUfv27bVq1SpJ0oULF+Tv769vvvlGGzdu1L59+7R582brPs6ePauAgAD9+uuvqly5cqb3OG3fvl1NmjTRxx9/rKefftraPn78eH388cc6fPiwJKl06dLq1auXJk2a9OBPHgBgd9zjBADIN/468uPo6ChfX1/VqFHD2ubn5ydJiouL04EDB7Rt27ZM70s6fvy4KleufNdjhYaGZrkuLi5O58+fV7Nmze71FAAA+RTBCQCQbzg7O9ssWywWmzaLxSJJSktLU1pamtq1a6dp06Zl2I+/v7/psTw8PLJc5+bmlt2SAQD/EAQnAMA/0qOPPqp169YpMDBQTk6Z/9wVKlRIqamp97xvLy8vBQYG6quvvlKTJk1yWyoAIB9gcggAwD/S4MGDFR8fry5duui7777TiRMntGXLFvXu3dsalgIDA7Vv3z7FxMTo0qVLSktLy/b+x48fr1mzZmnu3Lk6duyYDh48qHnz5j2o0wEA2BnBCQDwj1SqVCnt2bNHqampatmypapXr64XXnhBPj4+cnD48+dv5MiRcnR0VNWqVVW8eHGdPn062/vv0aOHoqOjtWDBAlWrVk1t27bVsWPHHtTpAADsjFn1AAAAAMAEI04AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYILgBAAAAAAmCE4AAAAAYOL/AdfqDLcc8SssAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1000x600 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "          model      rmse        r²\n",
      "0            lr  1.072045  0.519700\n",
      "1         ridge  1.082502  0.510285\n",
      "2           knn  1.147199  0.449998\n",
      "3  randomforest  1.137848  0.458928\n"
     ]
    }
   ],
   "source": [
    "metrics_list = []\n",
    "\n",
    "for model_name, model in loaded_models.items():\n",
    "    rmse = np.sqrt(metrics.mean_squared_error(model['y_test'], model['y_pred']))\n",
    "    r2 = metrics.r2_score(model['y_test'], model['y_pred'])\n",
    "    metrics_list.append({'model': model_name, 'rmse': rmse, 'r²': r2})\n",
    "\n",
    "metrics_df = pd.DataFrame(metrics_list)\n",
    "\n",
    "# Changes the dataframe from wide to long format, creating a dataframe that adds a column ('metric') populated by the name of\n",
    "# the other columns in the original dataframe ('rmse' and r²'), and another column with their corresponding values ('score')\n",
    "metrics_melted = pd.melt(metrics_df, id_vars='model', var_name='metric', value_name='score')\n",
    "\n",
    "# Pivots the dataframe so that the index will be the metrics and the columns will be the models and their values\n",
    "pivot_df = metrics_melted.pivot(index='metric', columns='model', values='score')\n",
    "\n",
    "pivot_df.plot(kind='bar', figsize=(10, 6))\n",
    "plt.title('Model performance comparison')\n",
    "plt.xticks(rotation=0)\n",
    "plt.ylabel('score')\n",
    "plt.show()\n",
    "\n",
    "print(metrics_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9cc06b8",
   "metadata": {},
   "source": [
    "We can also plot the actual vs predicted values for each of the individual models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f652191c",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "for model_name, model in loaded_models.items():\n",
    "    print(type(model['model'].named_steps.main_regressor.estimator.named_steps.regressor))\n",
    "    plot_predicted_vs_actual(model['y_test'], model['y_pred'])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}