starter-notebook - 2 Nov.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "6c7e849a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-11T09:24:53.643384Z",
     "start_time": "2021-06-11T09:24:53.622385Z"
    }
   },
   "source": [
    "# Regression Predict Student Solution\n",
    "\n",
    "© Explore Data Science Academy\n",
    "\n",
    "---\n",
    "### Honour Code\n",
    "\n",
    "I {**YOUR NAME, YOUR SURNAME**}, confirm - by submitting this document - that the solutions in this notebook are a result of my own work and that I abide by the [EDSA honour code](https://drive.google.com/file/d/1QDCjGZJ8-FmJE3bZdIQNwnJyQKPhHZBn/view?usp=sharing).\n",
    "\n",
    "Non-compliance with the honour code constitutes a material breach of contract.\n",
    "\n",
    "### Predict Overview: Spain Electricity Shortfall Challenge\n",
    "\n",
    "The government of Spain is considering an expansion of it's renewable energy resource infrastructure investments. As such, they require information on the trends and patterns of the country's renewable sources and fossil fuel energy generation. Your company has been awarded the contract to:\n",
    "\n",
    "- 1. analyse the supplied data;\n",
    "- 2. identify potential errors in the data and clean the existing data set;\n",
    "- 3. determine if additional features can be added to enrich the data set;\n",
    "- 4. build a model that is capable of forecasting the three hourly demand shortfalls;\n",
    "- 5. evaluate the accuracy of the best machine learning model;\n",
    "- 6. determine what features were most important in the model’s prediction decision, and\n",
    "- 7. explain the inner working of the model to a non-technical audience.\n",
    "\n",
    "Formally the problem statement was given to you, the senior data scientist, by your manager via email reads as follow:\n",
    "\n",
    "> In this project you are tasked to model the shortfall between the energy generated by means of fossil fuels and various renewable sources - for the country of Spain. The daily shortfall, which will be referred to as the target variable, will be modelled as a function of various city-specific weather features such as `pressure`, `wind speed`, `humidity`, etc. As with all data science projects, the provided features are rarely adequate predictors of the target variable. As such, **you are required to perform feature engineering** to ensure that you will be able to accurately model Spain's three hourly shortfalls.\n",
    " \n",
    "On top of this, she has provided you with a starter notebook containing vague explanations of what the main outcomes are. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05600c92",
   "metadata": {},
   "source": [
    "<a id=\"cont\"></a>\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "<a href=#one>1. Importing Packages</a>\n",
    "\n",
    "<a href=#two>2. Loading Data</a>\n",
    "\n",
    "<a href=#three>3. Exploratory Data Analysis (EDA)</a>\n",
    "\n",
    "<a href=#four>4. Data Engineering</a>\n",
    "\n",
    "<a href=#five>5. Modeling</a>\n",
    "\n",
    "<a href=#six>6. Model Performance</a>\n",
    "\n",
    "<a href=#seven>7. Model Explanations</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "997462e2",
   "metadata": {},
   "source": [
    " <a id=\"one\"></a>\n",
    "## 1. Importing Packages\n",
    "<a href=#cont>Back to Table of Contents</a>\n",
    "\n",
    "---\n",
    "    \n",
    "| ⚡ Description: Importing Packages ⚡ |\n",
    "| :--------------------------- |\n",
    "| In this section you are required to import, and briefly discuss, the libraries that will be used throughout your analysis and modelling. |\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "475dbe93",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-23T10:30:53.800892Z",
     "start_time": "2021-06-23T10:30:50.215449Z"
    }
   },
   "outputs": [],
   "source": [
    "# Libraries for data loading, data manipulation and data visulisation\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import seaborn as sns\n",
    "\n",
    "# Libraries for data preparation and model building\n",
    "import sklearn\n",
    "\n",
    "# Setting global constants to ensure notebook results are reproducible\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f22a6718",
   "metadata": {},
   "source": [
    "<a id=\"two\"></a>\n",
    "## 2. Loading the Data\n",
    "<a class=\"anchor\" id=\"1.1\"></a>\n",
    "<a href=#cont>Back to Table of Contents</a>\n",
    "\n",
    "---\n",
    "    \n",
    "| ⚡ Description: Loading the data ⚡ |\n",
    "| :--------------------------- |\n",
    "| In this section you are required to load the data from the `df_train` file into a DataFrame. |\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "fbbb6c18",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-28T08:49:35.311495Z",
     "start_time": "2021-06-28T08:49:35.295494Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>time</th>\n",
       "      <th>Madrid_wind_speed</th>\n",
       "      <th>Valencia_wind_deg</th>\n",
       "      <th>Bilbao_rain_1h</th>\n",
       "      <th>Valencia_wind_speed</th>\n",
       "      <th>Seville_humidity</th>\n",
       "      <th>Madrid_humidity</th>\n",
       "      <th>Bilbao_clouds_all</th>\n",
       "      <th>Bilbao_wind_speed</th>\n",
       "      <th>...</th>\n",
       "      <th>Madrid_temp_max</th>\n",
       "      <th>Barcelona_temp</th>\n",
       "      <th>Bilbao_temp_min</th>\n",
       "      <th>Bilbao_temp</th>\n",
       "      <th>Barcelona_temp_min</th>\n",
       "      <th>Bilbao_temp_max</th>\n",
       "      <th>Seville_temp_min</th>\n",
       "      <th>Madrid_temp</th>\n",
       "      <th>Madrid_temp_min</th>\n",
       "      <th>load_shortfall_3h</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>2015-01-01 03:00:00</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>level_5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.666667</td>\n",
       "      <td>74.333333</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>265.938000</td>\n",
       "      <td>281.013000</td>\n",
       "      <td>269.338615</td>\n",
       "      <td>269.338615</td>\n",
       "      <td>281.013000</td>\n",
       "      <td>269.338615</td>\n",
       "      <td>274.254667</td>\n",
       "      <td>265.938000</td>\n",
       "      <td>265.938000</td>\n",
       "      <td>6715.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2015-01-01 06:00:00</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>level_10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.666667</td>\n",
       "      <td>78.333333</td>\n",
       "      <td>64.666667</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>266.386667</td>\n",
       "      <td>280.561667</td>\n",
       "      <td>270.376000</td>\n",
       "      <td>270.376000</td>\n",
       "      <td>280.561667</td>\n",
       "      <td>270.376000</td>\n",
       "      <td>274.945000</td>\n",
       "      <td>266.386667</td>\n",
       "      <td>266.386667</td>\n",
       "      <td>4171.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2015-01-01 09:00:00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>level_9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>71.333333</td>\n",
       "      <td>64.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>272.708667</td>\n",
       "      <td>281.583667</td>\n",
       "      <td>275.027229</td>\n",
       "      <td>275.027229</td>\n",
       "      <td>281.583667</td>\n",
       "      <td>275.027229</td>\n",
       "      <td>278.792000</td>\n",
       "      <td>272.708667</td>\n",
       "      <td>272.708667</td>\n",
       "      <td>4274.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>2015-01-01 12:00:00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>level_8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>65.333333</td>\n",
       "      <td>56.333333</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>281.895219</td>\n",
       "      <td>283.434104</td>\n",
       "      <td>281.135063</td>\n",
       "      <td>281.135063</td>\n",
       "      <td>283.434104</td>\n",
       "      <td>281.135063</td>\n",
       "      <td>285.394000</td>\n",
       "      <td>281.895219</td>\n",
       "      <td>281.895219</td>\n",
       "      <td>5075.666667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>2015-01-01 15:00:00</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>level_7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>59.000000</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>280.678437</td>\n",
       "      <td>284.213167</td>\n",
       "      <td>282.252063</td>\n",
       "      <td>282.252063</td>\n",
       "      <td>284.213167</td>\n",
       "      <td>282.252063</td>\n",
       "      <td>285.513719</td>\n",
       "      <td>280.678437</td>\n",
       "      <td>280.678437</td>\n",
       "      <td>6620.666667</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 49 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                 time  Madrid_wind_speed Valencia_wind_deg  \\\n",
       "0           0  2015-01-01 03:00:00           0.666667           level_5   \n",
       "1           1  2015-01-01 06:00:00           0.333333          level_10   \n",
       "2           2  2015-01-01 09:00:00           1.000000           level_9   \n",
       "3           3  2015-01-01 12:00:00           1.000000           level_8   \n",
       "4           4  2015-01-01 15:00:00           1.000000           level_7   \n",
       "\n",
       "   Bilbao_rain_1h  Valencia_wind_speed  Seville_humidity  Madrid_humidity  \\\n",
       "0             0.0             0.666667         74.333333        64.000000   \n",
       "1             0.0             1.666667         78.333333        64.666667   \n",
       "2             0.0             1.000000         71.333333        64.333333   \n",
       "3             0.0             1.000000         65.333333        56.333333   \n",
       "4             0.0             1.000000         59.000000        57.000000   \n",
       "\n",
       "   Bilbao_clouds_all  Bilbao_wind_speed  ...  Madrid_temp_max  Barcelona_temp  \\\n",
       "0                0.0           1.000000  ...       265.938000      281.013000   \n",
       "1                0.0           1.000000  ...       266.386667      280.561667   \n",
       "2                0.0           1.000000  ...       272.708667      281.583667   \n",
       "3                0.0           1.000000  ...       281.895219      283.434104   \n",
       "4                2.0           0.333333  ...       280.678437      284.213167   \n",
       "\n",
       "   Bilbao_temp_min  Bilbao_temp  Barcelona_temp_min  Bilbao_temp_max  \\\n",
       "0       269.338615   269.338615          281.013000       269.338615   \n",
       "1       270.376000   270.376000          280.561667       270.376000   \n",
       "2       275.027229   275.027229          281.583667       275.027229   \n",
       "3       281.135063   281.135063          283.434104       281.135063   \n",
       "4       282.252063   282.252063          284.213167       282.252063   \n",
       "\n",
       "   Seville_temp_min Madrid_temp  Madrid_temp_min  load_shortfall_3h  \n",
       "0        274.254667  265.938000       265.938000        6715.666667  \n",
       "1        274.945000  266.386667       266.386667        4171.666667  \n",
       "2        278.792000  272.708667       272.708667        4274.666667  \n",
       "3        285.394000  281.895219       281.895219        5075.666667  \n",
       "4        285.513719  280.678437       280.678437        6620.666667  \n",
       "\n",
       "[5 rows x 49 columns]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('C://Users//Rethabile Ntshabele//Documents//Predict//df_train.csv')\n",
    "df.head(5)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "07fa125f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>time</th>\n",
       "      <th>Madrid_wind_speed</th>\n",
       "      <th>Valencia_wind_deg</th>\n",
       "      <th>Bilbao_rain_1h</th>\n",
       "      <th>Valencia_wind_speed</th>\n",
       "      <th>Seville_humidity</th>\n",
       "      <th>Madrid_humidity</th>\n",
       "      <th>Bilbao_clouds_all</th>\n",
       "      <th>Bilbao_wind_speed</th>\n",
       "      <th>...</th>\n",
       "      <th>Barcelona_temp_max</th>\n",
       "      <th>Madrid_temp_max</th>\n",
       "      <th>Barcelona_temp</th>\n",
       "      <th>Bilbao_temp_min</th>\n",
       "      <th>Bilbao_temp</th>\n",
       "      <th>Barcelona_temp_min</th>\n",
       "      <th>Bilbao_temp_max</th>\n",
       "      <th>Seville_temp_min</th>\n",
       "      <th>Madrid_temp</th>\n",
       "      <th>Madrid_temp_min</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>8763</td>\n",
       "      <td>2018-01-01 00:00:00</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>level_8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>87.000000</td>\n",
       "      <td>71.333333</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>287.816667</td>\n",
       "      <td>280.816667</td>\n",
       "      <td>287.356667</td>\n",
       "      <td>276.150000</td>\n",
       "      <td>280.380000</td>\n",
       "      <td>286.816667</td>\n",
       "      <td>285.150000</td>\n",
       "      <td>283.150000</td>\n",
       "      <td>279.866667</td>\n",
       "      <td>279.150000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>8764</td>\n",
       "      <td>2018-01-01 03:00:00</td>\n",
       "      <td>4.666667</td>\n",
       "      <td>level_8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.333333</td>\n",
       "      <td>89.000000</td>\n",
       "      <td>78.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>284.816667</td>\n",
       "      <td>280.483333</td>\n",
       "      <td>284.190000</td>\n",
       "      <td>277.816667</td>\n",
       "      <td>281.010000</td>\n",
       "      <td>283.483333</td>\n",
       "      <td>284.150000</td>\n",
       "      <td>281.150000</td>\n",
       "      <td>279.193333</td>\n",
       "      <td>278.150000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8765</td>\n",
       "      <td>2018-01-01 06:00:00</td>\n",
       "      <td>2.333333</td>\n",
       "      <td>level_7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>89.000000</td>\n",
       "      <td>89.666667</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>284.483333</td>\n",
       "      <td>276.483333</td>\n",
       "      <td>283.150000</td>\n",
       "      <td>276.816667</td>\n",
       "      <td>279.196667</td>\n",
       "      <td>281.816667</td>\n",
       "      <td>282.150000</td>\n",
       "      <td>280.483333</td>\n",
       "      <td>276.340000</td>\n",
       "      <td>276.150000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>8766</td>\n",
       "      <td>2018-01-01 09:00:00</td>\n",
       "      <td>2.666667</td>\n",
       "      <td>level_7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5.333333</td>\n",
       "      <td>93.333333</td>\n",
       "      <td>82.666667</td>\n",
       "      <td>26.666667</td>\n",
       "      <td>5.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>284.150000</td>\n",
       "      <td>277.150000</td>\n",
       "      <td>283.190000</td>\n",
       "      <td>279.150000</td>\n",
       "      <td>281.740000</td>\n",
       "      <td>282.150000</td>\n",
       "      <td>284.483333</td>\n",
       "      <td>279.150000</td>\n",
       "      <td>275.953333</td>\n",
       "      <td>274.483333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>8767</td>\n",
       "      <td>2018-01-01 12:00:00</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>level_7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>8.666667</td>\n",
       "      <td>65.333333</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>26.666667</td>\n",
       "      <td>10.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>287.483333</td>\n",
       "      <td>281.150000</td>\n",
       "      <td>286.816667</td>\n",
       "      <td>281.816667</td>\n",
       "      <td>284.116667</td>\n",
       "      <td>286.150000</td>\n",
       "      <td>286.816667</td>\n",
       "      <td>284.483333</td>\n",
       "      <td>280.686667</td>\n",
       "      <td>280.150000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 48 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                 time  Madrid_wind_speed Valencia_wind_deg  \\\n",
       "0        8763  2018-01-01 00:00:00           5.000000           level_8   \n",
       "1        8764  2018-01-01 03:00:00           4.666667           level_8   \n",
       "2        8765  2018-01-01 06:00:00           2.333333           level_7   \n",
       "3        8766  2018-01-01 09:00:00           2.666667           level_7   \n",
       "4        8767  2018-01-01 12:00:00           4.000000           level_7   \n",
       "\n",
       "   Bilbao_rain_1h  Valencia_wind_speed  Seville_humidity  Madrid_humidity  \\\n",
       "0             0.0             5.000000         87.000000        71.333333   \n",
       "1             0.0             5.333333         89.000000        78.000000   \n",
       "2             0.0             5.000000         89.000000        89.666667   \n",
       "3             0.0             5.333333         93.333333        82.666667   \n",
       "4             0.0             8.666667         65.333333        64.000000   \n",
       "\n",
       "   Bilbao_clouds_all  Bilbao_wind_speed  ...  Barcelona_temp_max  \\\n",
       "0          20.000000           3.000000  ...          287.816667   \n",
       "1           0.000000           3.666667  ...          284.816667   \n",
       "2           0.000000           2.333333  ...          284.483333   \n",
       "3          26.666667           5.666667  ...          284.150000   \n",
       "4          26.666667          10.666667  ...          287.483333   \n",
       "\n",
       "   Madrid_temp_max  Barcelona_temp  Bilbao_temp_min  Bilbao_temp  \\\n",
       "0       280.816667      287.356667       276.150000   280.380000   \n",
       "1       280.483333      284.190000       277.816667   281.010000   \n",
       "2       276.483333      283.150000       276.816667   279.196667   \n",
       "3       277.150000      283.190000       279.150000   281.740000   \n",
       "4       281.150000      286.816667       281.816667   284.116667   \n",
       "\n",
       "   Barcelona_temp_min  Bilbao_temp_max Seville_temp_min  Madrid_temp  \\\n",
       "0          286.816667       285.150000       283.150000   279.866667   \n",
       "1          283.483333       284.150000       281.150000   279.193333   \n",
       "2          281.816667       282.150000       280.483333   276.340000   \n",
       "3          282.150000       284.483333       279.150000   275.953333   \n",
       "4          286.150000       286.816667       284.483333   280.686667   \n",
       "\n",
       "   Madrid_temp_min  \n",
       "0       279.150000  \n",
       "1       278.150000  \n",
       "2       276.150000  \n",
       "3       274.483333  \n",
       "4       280.150000  \n",
       "\n",
       "[5 rows x 48 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"C://Users//Rethabile Ntshabele//Documents//Predict//df_test.csv\")\n",
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81132ab3",
   "metadata": {},
   "source": [
    "<a id=\"three\"></a>\n",
    "## 3. Exploratory Data Analysis (EDA)\n",
    "<a class=\"anchor\" id=\"1.1\"></a>\n",
    "<a href=#cont>Back to Table of Contents</a>\n",
    "\n",
    "---\n",
    "    \n",
    "| ⚡ Description: Exploratory data analysis ⚡ |\n",
    "| :--------------------------- |\n",
    "| In this section, you are required to perform an in-depth analysis of all the variables in the DataFrame. |\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "e805134e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-06-28T08:52:37.824204Z",
     "start_time": "2021-06-28T08:52:37.811206Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2920 entries, 0 to 2919\n",
      "Data columns (total 48 columns):\n",
      " #   Column                Non-Null Count  Dtype  \n",
      "---  ------                --------------  -----  \n",
      " 0   Unnamed: 0            2920 non-null   int64  \n",
      " 1   time                  2920 non-null   object \n",
      " 2   Madrid_wind_speed     2920 non-null   float64\n",
      " 3   Valencia_wind_deg     2920 non-null   object \n",
      " 4   Bilbao_rain_1h        2920 non-null   float64\n",
      " 5   Valencia_wind_speed   2920 non-null   float64\n",
      " 6   Seville_humidity      2920 non-null   float64\n",
      " 7   Madrid_humidity       2920 non-null   float64\n",
      " 8   Bilbao_clouds_all     2920 non-null   float64\n",
      " 9   Bilbao_wind_speed     2920 non-null   float64\n",
      " 10  Seville_clouds_all    2920 non-null   float64\n",
      " 11  Bilbao_wind_deg       2920 non-null   float64\n",
      " 12  Barcelona_wind_speed  2920 non-null   float64\n",
      " 13  Barcelona_wind_deg    2920 non-null   float64\n",
      " 14  Madrid_clouds_all     2920 non-null   float64\n",
      " 15  Seville_wind_speed    2920 non-null   float64\n",
      " 16  Barcelona_rain_1h     2920 non-null   float64\n",
      " 17  Seville_pressure      2920 non-null   object \n",
      " 18  Seville_rain_1h       2920 non-null   float64\n",
      " 19  Bilbao_snow_3h        2920 non-null   int64  \n",
      " 20  Barcelona_pressure    2920 non-null   float64\n",
      " 21  Seville_rain_3h       2920 non-null   float64\n",
      " 22  Madrid_rain_1h        2920 non-null   float64\n",
      " 23  Barcelona_rain_3h     2920 non-null   float64\n",
      " 24  Valencia_snow_3h      2920 non-null   int64  \n",
      " 25  Madrid_weather_id     2920 non-null   float64\n",
      " 26  Barcelona_weather_id  2920 non-null   float64\n",
      " 27  Bilbao_pressure       2920 non-null   float64\n",
      " 28  Seville_weather_id    2920 non-null   float64\n",
      " 29  Valencia_pressure     2466 non-null   float64\n",
      " 30  Seville_temp_max      2920 non-null   float64\n",
      " 31  Madrid_pressure       2920 non-null   float64\n",
      " 32  Valencia_temp_max     2920 non-null   float64\n",
      " 33  Valencia_temp         2920 non-null   float64\n",
      " 34  Bilbao_weather_id     2920 non-null   float64\n",
      " 35  Seville_temp          2920 non-null   float64\n",
      " 36  Valencia_humidity     2920 non-null   float64\n",
      " 37  Valencia_temp_min     2920 non-null   float64\n",
      " 38  Barcelona_temp_max    2920 non-null   float64\n",
      " 39  Madrid_temp_max       2920 non-null   float64\n",
      " 40  Barcelona_temp        2920 non-null   float64\n",
      " 41  Bilbao_temp_min       2920 non-null   float64\n",
      " 42  Bilbao_temp           2920 non-null   float64\n",
      " 43  Barcelona_temp_min    2920 non-null   float64\n",
      " 44  Bilbao_temp_max       2920 non-null   float64\n",
      " 45  Seville_temp_min      2920 non-null   float64\n",
      " 46  Madrid_temp           2920 non-null   float64\n",
      " 47  Madrid_temp_min       2920 non-null   float64\n",
      "dtypes: float64(42), int64(3), object(3)\n",
      "memory usage: 1.1+ MB\n"
     ]
    }
   ],
   "source": [
    "# look at data statistics\n",
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "8543771e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>Madrid_wind_speed</th>\n",
       "      <th>Bilbao_rain_1h</th>\n",
       "      <th>Valencia_wind_speed</th>\n",
       "      <th>Seville_humidity</th>\n",
       "      <th>Madrid_humidity</th>\n",
       "      <th>Bilbao_clouds_all</th>\n",
       "      <th>Bilbao_wind_speed</th>\n",
       "      <th>Seville_clouds_all</th>\n",
       "      <th>Bilbao_wind_deg</th>\n",
       "      <th>...</th>\n",
       "      <th>Barcelona_temp_max</th>\n",
       "      <th>Madrid_temp_max</th>\n",
       "      <th>Barcelona_temp</th>\n",
       "      <th>Bilbao_temp_min</th>\n",
       "      <th>Bilbao_temp</th>\n",
       "      <th>Barcelona_temp_min</th>\n",
       "      <th>Bilbao_temp_max</th>\n",
       "      <th>Seville_temp_min</th>\n",
       "      <th>Madrid_temp</th>\n",
       "      <th>Madrid_temp_min</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "      <td>2920.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>10222.500000</td>\n",
       "      <td>2.457820</td>\n",
       "      <td>0.067517</td>\n",
       "      <td>3.012785</td>\n",
       "      <td>67.123516</td>\n",
       "      <td>62.644463</td>\n",
       "      <td>43.355422</td>\n",
       "      <td>2.283562</td>\n",
       "      <td>15.477283</td>\n",
       "      <td>162.643836</td>\n",
       "      <td>...</td>\n",
       "      <td>290.695462</td>\n",
       "      <td>288.888393</td>\n",
       "      <td>289.911289</td>\n",
       "      <td>284.920684</td>\n",
       "      <td>286.522375</td>\n",
       "      <td>289.124971</td>\n",
       "      <td>288.483641</td>\n",
       "      <td>290.152431</td>\n",
       "      <td>287.869763</td>\n",
       "      <td>286.618130</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>843.075718</td>\n",
       "      <td>1.774838</td>\n",
       "      <td>0.153381</td>\n",
       "      <td>1.996340</td>\n",
       "      <td>20.611292</td>\n",
       "      <td>24.138393</td>\n",
       "      <td>30.486298</td>\n",
       "      <td>1.654787</td>\n",
       "      <td>25.289197</td>\n",
       "      <td>97.749873</td>\n",
       "      <td>...</td>\n",
       "      <td>7.113599</td>\n",
       "      <td>9.089699</td>\n",
       "      <td>7.119411</td>\n",
       "      <td>6.803424</td>\n",
       "      <td>6.492355</td>\n",
       "      <td>7.168049</td>\n",
       "      <td>6.221324</td>\n",
       "      <td>7.906915</td>\n",
       "      <td>8.977511</td>\n",
       "      <td>8.733163</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>8763.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>11.666667</td>\n",
       "      <td>8.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>273.816667</td>\n",
       "      <td>269.816667</td>\n",
       "      <td>272.650000</td>\n",
       "      <td>266.483333</td>\n",
       "      <td>268.120000</td>\n",
       "      <td>271.483333</td>\n",
       "      <td>270.138667</td>\n",
       "      <td>271.150000</td>\n",
       "      <td>268.713333</td>\n",
       "      <td>267.816667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>9492.750000</td>\n",
       "      <td>1.333333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.666667</td>\n",
       "      <td>52.000000</td>\n",
       "      <td>43.000000</td>\n",
       "      <td>13.333333</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>86.666667</td>\n",
       "      <td>...</td>\n",
       "      <td>284.816667</td>\n",
       "      <td>281.483333</td>\n",
       "      <td>284.307500</td>\n",
       "      <td>280.150000</td>\n",
       "      <td>281.778333</td>\n",
       "      <td>283.483333</td>\n",
       "      <td>284.150000</td>\n",
       "      <td>284.483333</td>\n",
       "      <td>280.816667</td>\n",
       "      <td>279.816667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>10222.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2.333333</td>\n",
       "      <td>70.333333</td>\n",
       "      <td>63.000000</td>\n",
       "      <td>45.000000</td>\n",
       "      <td>1.666667</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>140.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>290.150000</td>\n",
       "      <td>287.483333</td>\n",
       "      <td>289.483333</td>\n",
       "      <td>284.483333</td>\n",
       "      <td>286.265000</td>\n",
       "      <td>288.816667</td>\n",
       "      <td>288.483333</td>\n",
       "      <td>289.150000</td>\n",
       "      <td>286.396667</td>\n",
       "      <td>285.483333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>10952.250000</td>\n",
       "      <td>3.333333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>85.000000</td>\n",
       "      <td>84.000000</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>3.333333</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>233.333333</td>\n",
       "      <td>...</td>\n",
       "      <td>296.483333</td>\n",
       "      <td>295.483333</td>\n",
       "      <td>295.816667</td>\n",
       "      <td>289.816667</td>\n",
       "      <td>291.119167</td>\n",
       "      <td>295.150000</td>\n",
       "      <td>292.816667</td>\n",
       "      <td>295.150000</td>\n",
       "      <td>294.452500</td>\n",
       "      <td>293.150000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>11682.000000</td>\n",
       "      <td>13.333333</td>\n",
       "      <td>1.600000</td>\n",
       "      <td>14.333333</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>97.333333</td>\n",
       "      <td>10.666667</td>\n",
       "      <td>93.333333</td>\n",
       "      <td>360.000000</td>\n",
       "      <td>...</td>\n",
       "      <td>309.483333</td>\n",
       "      <td>313.483333</td>\n",
       "      <td>308.150000</td>\n",
       "      <td>307.483333</td>\n",
       "      <td>308.966667</td>\n",
       "      <td>306.816667</td>\n",
       "      <td>310.816667</td>\n",
       "      <td>314.483333</td>\n",
       "      <td>312.223333</td>\n",
       "      <td>310.150000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>8 rows × 45 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "         Unnamed: 0  Madrid_wind_speed  Bilbao_rain_1h  Valencia_wind_speed  \\\n",
       "count   2920.000000        2920.000000     2920.000000          2920.000000   \n",
       "mean   10222.500000           2.457820        0.067517             3.012785   \n",
       "std      843.075718           1.774838        0.153381             1.996340   \n",
       "min     8763.000000           0.000000        0.000000             0.000000   \n",
       "25%     9492.750000           1.333333        0.000000             1.666667   \n",
       "50%    10222.500000           2.000000        0.000000             2.333333   \n",
       "75%    10952.250000           3.333333        0.000000             4.000000   \n",
       "max    11682.000000          13.333333        1.600000            14.333333   \n",
       "\n",
       "       Seville_humidity  Madrid_humidity  Bilbao_clouds_all  \\\n",
       "count       2920.000000      2920.000000        2920.000000   \n",
       "mean          67.123516        62.644463          43.355422   \n",
       "std           20.611292        24.138393          30.486298   \n",
       "min           11.666667         8.000000           0.000000   \n",
       "25%           52.000000        43.000000          13.333333   \n",
       "50%           70.333333        63.000000          45.000000   \n",
       "75%           85.000000        84.000000          75.000000   \n",
       "max          100.000000       100.000000          97.333333   \n",
       "\n",
       "       Bilbao_wind_speed  Seville_clouds_all  Bilbao_wind_deg  ...  \\\n",
       "count        2920.000000         2920.000000      2920.000000  ...   \n",
       "mean            2.283562           15.477283       162.643836  ...   \n",
       "std             1.654787           25.289197        97.749873  ...   \n",
       "min             0.000000            0.000000         0.000000  ...   \n",
       "25%             1.000000            0.000000        86.666667  ...   \n",
       "50%             1.666667            0.000000       140.000000  ...   \n",
       "75%             3.333333           20.000000       233.333333  ...   \n",
       "max            10.666667           93.333333       360.000000  ...   \n",
       "\n",
       "       Barcelona_temp_max  Madrid_temp_max  Barcelona_temp  Bilbao_temp_min  \\\n",
       "count         2920.000000      2920.000000     2920.000000      2920.000000   \n",
       "mean           290.695462       288.888393      289.911289       284.920684   \n",
       "std              7.113599         9.089699        7.119411         6.803424   \n",
       "min            273.816667       269.816667      272.650000       266.483333   \n",
       "25%            284.816667       281.483333      284.307500       280.150000   \n",
       "50%            290.150000       287.483333      289.483333       284.483333   \n",
       "75%            296.483333       295.483333      295.816667       289.816667   \n",
       "max            309.483333       313.483333      308.150000       307.483333   \n",
       "\n",
       "       Bilbao_temp  Barcelona_temp_min  Bilbao_temp_max  Seville_temp_min  \\\n",
       "count  2920.000000         2920.000000      2920.000000       2920.000000   \n",
       "mean    286.522375          289.124971       288.483641        290.152431   \n",
       "std       6.492355            7.168049         6.221324          7.906915   \n",
       "min     268.120000          271.483333       270.138667        271.150000   \n",
       "25%     281.778333          283.483333       284.150000        284.483333   \n",
       "50%     286.265000          288.816667       288.483333        289.150000   \n",
       "75%     291.119167          295.150000       292.816667        295.150000   \n",
       "max     308.966667          306.816667       310.816667        314.483333   \n",
       "\n",
       "       Madrid_temp  Madrid_temp_min  \n",
       "count  2920.000000      2920.000000  \n",
       "mean    287.869763       286.618130  \n",
       "std       8.977511         8.733163  \n",
       "min     268.713333       267.816667  \n",
       "25%     280.816667       279.816667  \n",
       "50%     286.396667       285.483333  \n",
       "75%     294.452500       293.150000  \n",
       "max     312.223333       310.150000  \n",
       "\n",
       "[8 rows x 45 columns]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "2fb74182",
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "'load_shortfall_3h'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\indexes\\base.py:3621\u001b[0m, in \u001b[0;36mIndex.get_loc\u001b[1;34m(self, key, method, tolerance)\u001b[0m\n\u001b[0;32m   3620\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[1;32m-> 3621\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_engine\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_loc\u001b[49m\u001b[43m(\u001b[49m\u001b[43mcasted_key\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   3622\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m err:\n",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\_libs\\index.pyx:136\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[1;34m()\u001b[0m\n",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\_libs\\index.pyx:163\u001b[0m, in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[1;34m()\u001b[0m\n",
      "File \u001b[1;32mpandas\\_libs\\hashtable_class_helper.pxi:5198\u001b[0m, in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[1;34m()\u001b[0m\n",
      "File \u001b[1;32mpandas\\_libs\\hashtable_class_helper.pxi:5206\u001b[0m, in \u001b[0;36mpandas._libs.hashtable.PyObjectHashTable.get_item\u001b[1;34m()\u001b[0m\n",
      "\u001b[1;31mKeyError\u001b[0m: 'load_shortfall_3h'",
      "\nThe above exception was the direct cause of the following exception:\n",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "Input \u001b[1;32mIn [48]\u001b[0m, in \u001b[0;36m<cell line: 9>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      9\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m index, column \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28menumerate\u001b[39m(df\u001b[38;5;241m.\u001b[39mcolumns):\n\u001b[0;32m     10\u001b[0m     axs[index\u001b[38;5;241m-\u001b[39m\u001b[38;5;241m1\u001b[39m]\u001b[38;5;241m.\u001b[39mset_title(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{}\u001b[39;00m\u001b[38;5;124m vs. load_shortfall_3h\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m.\u001b[39mformat(column),fontsize\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m16\u001b[39m)\n\u001b[1;32m---> 11\u001b[0m     axs[index\u001b[38;5;241m-\u001b[39m\u001b[38;5;241m1\u001b[39m]\u001b[38;5;241m.\u001b[39mscatter(x\u001b[38;5;241m=\u001b[39mdf[column],y\u001b[38;5;241m=\u001b[39m\u001b[43mdf\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mload_shortfall_3h\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m]\u001b[49m,color\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mblue\u001b[39m\u001b[38;5;124m'\u001b[39m,edgecolor\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mk\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[0;32m     13\u001b[0m fig\u001b[38;5;241m.\u001b[39mtight_layout(pad\u001b[38;5;241m=\u001b[39m\u001b[38;5;241m1\u001b[39m)\n",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\frame.py:3505\u001b[0m, in \u001b[0;36mDataFrame.__getitem__\u001b[1;34m(self, key)\u001b[0m\n\u001b[0;32m   3503\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mcolumns\u001b[38;5;241m.\u001b[39mnlevels \u001b[38;5;241m>\u001b[39m \u001b[38;5;241m1\u001b[39m:\n\u001b[0;32m   3504\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_getitem_multilevel(key)\n\u001b[1;32m-> 3505\u001b[0m indexer \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcolumns\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mget_loc\u001b[49m\u001b[43m(\u001b[49m\u001b[43mkey\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   3506\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m is_integer(indexer):\n\u001b[0;32m   3507\u001b[0m     indexer \u001b[38;5;241m=\u001b[39m [indexer]\n",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\indexes\\base.py:3623\u001b[0m, in \u001b[0;36mIndex.get_loc\u001b[1;34m(self, key, method, tolerance)\u001b[0m\n\u001b[0;32m   3621\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_engine\u001b[38;5;241m.\u001b[39mget_loc(casted_key)\n\u001b[0;32m   3622\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m err:\n\u001b[1;32m-> 3623\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m(key) \u001b[38;5;28;01mfrom\u001b[39;00m \u001b[38;5;21;01merr\u001b[39;00m\n\u001b[0;32m   3624\u001b[0m \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mTypeError\u001b[39;00m:\n\u001b[0;32m   3625\u001b[0m     \u001b[38;5;66;03m# If we have a listlike key, _check_indexing_error will raise\u001b[39;00m\n\u001b[0;32m   3626\u001b[0m     \u001b[38;5;66;03m#  InvalidIndexError. Otherwise we fall through and re-raise\u001b[39;00m\n\u001b[0;32m   3627\u001b[0m     \u001b[38;5;66;03m#  the TypeError.\u001b[39;00m\n\u001b[0;32m   3628\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_indexing_error(key)\n",
      "\u001b[1;31mKeyError\u001b[0m: 'load_shortfall_3h'"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA3UAAAFpCAYAAAA7hQHpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAueklEQVR4nO3dfbRld13n+ffHCrElgoApFfMgoSdChx5w4BoBn8KgWIli+TgmIHQi09VxEcdlOw4ZHWlGW21XL3spQzBWx5hO2xq7hwcjJkS6ERgNwVRceQSDZYCkurCTkAwOooTAd/7Y+5KTU+fW3XVr3zq/Xef9Wuusqr33b+/9Pbs+95z63r3PPqkqJEmSJEnT9EXLLkCSJEmStHU2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhmzZ1Sa5Mcn+SOzdYniRvTLI/ye1Jnj9+mVp15lAtMIdaNjOoFphDqT1DztRdBew6zPJzgTP7xx7g14++LOkQV2EOtXxXYQ61XFdhBrV8V2EOpaZs2tRV1fuAhw4zZDdwdXVuAp6S5OljFSiBOVQbzKGWzQyqBeZQas8Yn6k7BbhvZvpAP086lsyhWmAOtWxmUC0wh9IxdsII28iCebVwYLKH7jQ8J5100gue/exnj7B7HW9uueWWB6tq5xGuZg41qu3MoRnUEL4WqgXmUMu2xQyunDGaugPAaTPTpwIHFw2sqr3AXoC1tbXat2/fCLvX8SbJx7awmjnUqLYzh2ZQQ/haqBaYQy3bFjO4csa4/PJa4NX9nY5eCHyyqj4+wnalI2EO1QJzqGUzg2qBOZSOsU3P1CX5XeAc4OQkB4B/ATwBoKouB64DzgP2A58GLtquYrW6zKFaYA61bGZQLTCHUns2beqq6oJNlhfw2tEqkhYwh2qBOdSymUG1wBxK7Rnj8ktJkiRJ0pLY1EmSJEnShNnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShNnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShNnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShNnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShNnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShA1q6pLsSnJ3kv1JLl2w/MuS/EGS25LcleSi8UvVKjODaoE5VAvMoZbNDErt2bSpS7IDuAw4FzgLuCDJWXPDXgt8sKqeB5wD/EqSE0euVSvKDKoF5lAtMIdaNjMotWnImbqzgf1VdU9VPQJcA+yeG1PAk5IE+FLgIeDRUSvVKjODaoE5VAvMoZbNDEoNGtLUnQLcNzN9oJ83603APwIOAncAP15Vnx+lQskMqg3mUC0wh1o2Myg1aEhTlwXzam76O4Bbga8Gvg54U5InH7KhZE+SfUn2PfDAA0dYqlbYaBkEc6gt87VQLTCHWjbfk6UGDWnqDgCnzUyfSvebl1kXAW+tzn7gI8Cz5zdUVXuraq2q1nbu3LnVmrV6RssgmENtma+FaoE51LL5niw1aEhTdzNwZpIz+g+5ng9cOzfmXuClAEm+EngWcM+YhWqlmUG1wByqBeZQy2YGpQadsNmAqno0ySXADcAO4MqquivJxf3yy4GfB65KcgfdafnXVdWD21i3VogZVAvMoVpgDrVsZlBq06ZNHUBVXQdcNzfv8pm/HwReNm5p0mPMoFpgDtUCc6hlM4NSewZ9+bgkSZIkqU02dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YYOauiS7ktydZH+SSzcYc06SW5PcleS945apVWcG1QJzqBaYQy2bGZTac8JmA5LsAC4Dvh04ANyc5Nqq+uDMmKcAbwZ2VdW9Sb5im+rVCjKDaoE5VAvMoZbNDEptGnKm7mxgf1XdU1WPANcAu+fGvAJ4a1XdC1BV949bplacGVQLzKFaYA61bGZQatCQpu4U4L6Z6QP9vFlfCzw1yXuS3JLk1WMVKGEG1QZzqBaYQy2bGZQatOnll0AWzKsF23kB8FLgS4D3J7mpqj78uA0le4A9AKeffvqRV6tVNVoGwRxqy3wtVAvMoZbN92SpQUPO1B0ATpuZPhU4uGDMO6vqb6vqQeB9wPPmN1RVe6tqrarWdu7cudWatXpGyyCYQ22Zr4VqgTnUsvmeLDVoSFN3M3BmkjOSnAicD1w7N+b3gW9OckKSJwLfAHxo3FK1wsygWmAO1QJzqGUzg1KDNr38sqoeTXIJcAOwA7iyqu5KcnG//PKq+lCSdwK3A58HrqiqO7ezcK0OM6gWmEO1wBxq2cyg1KZUzV8GfWysra3Vvn37lrJvtS3JLVW1diz2ZQ61kWOVQzOojfhaqBaYQy3bsczglA368nFJkiRJUpts6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwgY1dUl2Jbk7yf4klx5m3Ncn+VySHxivRMkMqg3mUC0wh1o2Myi1Z9OmLskO4DLgXOAs4IIkZ20w7peBG8YuUqvNDKoF5lAtMIdaNjMotWnImbqzgf1VdU9VPQJcA+xeMO7HgLcA949YnwRmUG0wh2qBOdSymUGpQUOaulOA+2amD/TzviDJKcD3ApePV5r0BWZQLTCHaoE51LKZQalBQ5q6LJhXc9O/Cryuqj532A0le5LsS7LvgQceGFiiNF4GwRxqy3wtVAvMoZbN92SpQScMGHMAOG1m+lTg4NyYNeCaJAAnA+clebSq3j47qKr2AnsB1tbW5l8ApI2MlkEwh9oyXwvVAnOoZfM9WWrQkKbuZuDMJGcA/xU4H3jF7ICqOmP970muAt6x6AdX2iIzqBaYQ7XAHGrZzKDUoE2buqp6NMkldHcv2gFcWVV3Jbm4X+710tpWZlAtMIdqgTnUsplBqU1DztRRVdcB183NW/hDW1UXHn1Z0uOZQbXAHKoF5lDLZgal9gz68nFJkiRJUpts6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCBjV1SXYluTvJ/iSXLlj+yiS3948bkzxv/FK1ysygWmAO1QJzqGUzg1J7Nm3qkuwALgPOBc4CLkhy1tywjwDfWlXPBX4e2Dt2oVpdZlAtMIdqgTnUsplBqU1DztSdDeyvqnuq6hHgGmD37ICqurGqHu4nbwJOHbdMrTgzqBaYQ7XAHGrZzKDUoCFN3SnAfTPTB/p5G3kNcP2iBUn2JNmXZN8DDzwwvEqtutEyCOZQW+ZroVpgDrVsvidLDRrS1GXBvFo4MHkJ3Q/v6xYtr6q9VbVWVWs7d+4cXqVW3WgZBHOoLfO1UC0wh1o235OlBp0wYMwB4LSZ6VOBg/ODkjwXuAI4t6o+MU55EmAG1QZzqBaYQy2bGZQaNORM3c3AmUnOSHIicD5w7eyAJKcDbwVeVVUfHr9MrTgzqBaYQ7XAHGrZzKDUoE3P1FXVo0kuAW4AdgBXVtVdSS7ul18OvB74cuDNSQAeraq17Stbq8QMqgXmUC0wh1o2Myi1KVULL4Pedmtra7Vv376l7FttS3LLsXrxN4fayLHKoRnURnwtVAvMoZbtWGZwygZ9+bgkSZIkqU02dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YYOauiS7ktydZH+SSxcsT5I39stvT/L88UvVKjODaoE5VAvMoZbNDErt2bSpS7IDuAw4FzgLuCDJWXPDzgXO7B97gF8fuU6tMDOoFphDtcAcatnMoNSmIWfqzgb2V9U9VfUIcA2we27MbuDq6twEPCXJ00euVavLDKoF5lAtMIdaNjMoNeiEAWNOAe6bmT4AfMOAMacAH58dlGQP3W9sAD6T5M4jqnZ7nAw8uOwiaKOOFmoAeNbc9GgZBHPYeA3QTh3blkMzeFgt1NFCDXBoBsEcrkoN0E4dq/ae3Mpxt47HLHot1JwhTV0WzKstjKGq9gJ7AZLsq6q1AfvfVtbRVg3rdczPWjBsSxkEc9hyDa3VMT9rwTBfC4/DOlqoYb2ORbMXzDOHx1kNrdUxP2vBsOPmPbmFGqzj0BqWuf+pGHL55QHgtJnpU4GDWxgjbZUZVAvMoVpgDrVsZlBq0JCm7mbgzCRnJDkROB+4dm7MtcCr+7sdvRD4ZFUdcopd2iIzqBaYQ7XAHGrZzKDUoE0vv6yqR5NcAtwA7ACurKq7klzcL78cuA44D9gPfBq4aMC+92656nFZx2NaqAHm6tjGDB6yryVqoY4WaoBG6/C18JhpoY4WaoAFdZjDY6KFGqDROlbgPbmFGsA6ZrVQQ/NStfASZ0mSJEnSBAz68nFJkiRJUpts6iRJkiRpwralqUuyK8ndSfYnuXTB8iR5Y7/89iTPH7ruiDW8st/37UluTPK8mWUfTXJHkluP9jaqA+o4J8kn+33dmuT1Q9cduY6fmqnhziSfS/K0ftkoxyPJlUnuzwbfQTNmLlrI4MA6ViaHq5bBgc95ZXLYQgYH1mEOfU82h/ievF05NIOP28cxfS087lXVqA+6D83+FfBM4ETgNuCsuTHnAdfTfY/JC4EPDF13xBpeDDy1//u56zX00x8FTj5Gx+Ic4B1bWXfMOubGvxx49zYcj28Bng/cucHyUXLRQgbN4Wpn0By2l0Fz6HuyOVxODlvIYCs5NIPLyeCqPLbjTN3ZwP6quqeqHgGuAXbPjdkNXF2dm4CnJHn6wHVHqaGqbqyqh/vJm+i+Q2VsR/N8xjoWW9nWBcDvbnFfG6qq9wEPHWbIWLloIYOD6lihHK5aBhm4zqrksIUMbmVb5tD35DHWPdptTT2HLWRwUB2+Fm5o6hlcCdvR1J0C3DczfaCfN2TMkHXHqmHWa+h+E7CugD9KckuSPVvY/5HW8aIktyW5PslzjnDdMesgyROBXcBbZmaPdTw2M1YuWsjg0DpmHc85XLUMHm5b27W/o6lj1nbksIUMHtG2zKHvyebQ92R8LTxeMrgSNv2eui3Ignk1cMyQdceqoRuYvITuB/ebZmZ/Y1UdTPIVwLuS/EX/24TtqOPPga+pqk8lOQ94O3DmwHXHrGPdy4E/rarZ35yMdTw2M1YuWsjg0Dq6gcd/Dlctg4fb1nbt72jq6AZuXw5byODQOtaZQ9+TzeGhfE/2tRCml8GVsB1n6g4Ap81MnwocHDhmyLpj1UCS5wJXALur6hPr86vqYP/n/cDb6E7zbsWmdVTV31TVp/q/Xwc8IcnJQ5/DWHXMOJ+5U+wjHo/NjJWLFjI4tI5VyeGqZfBw29qu/R1NHdudwxYyOKiOGebQ92Rz6Huyr4XHRwZXQ438IT26s3/3AGfw2IcXnzM35jt5/Acf/2zouiPWcDqwH3jx3PyTgCfN/P1GYNc2Houvgi98CfzZwL39cRnlWBzJcQW+jO7a5pO243j023gGG38gdpRctJBBc7jaGTSH7WXQHPqebA6Xk8MWMthKDs3gcjK4Ko/t2Wh3t5oP092Z5mf6eRcDF/d/D3BZv/wOYO1w625TDVcADwO39o99/fxn9uG4DbjraGoYWMcl/X5uo/tQ7osPt+521dFPXwhcM7feaMeD7jc9Hwc+S/dbltdsVy5ayKA5XO0MmsP2MmgOfU82h8vJYQsZbCWHZnA5GVyFx/pvAiRJkiRJE7QtXz4uSZIkSTo2bOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZIkacJs6iRJkiRpwmzqJEmSJGnCbOokSZKWKMlVSQ5ssOycJJXk2451XS2ZOQ7nbHH9f5DkXyf5eJK/S/L+JN8ybpXj6J/nG47xPp/R7/fCEbd5zLKb5HuS/PMNlr0oyQeS/G1fz9cdwXbfk+Q9M9NHnMMkP5zkT5M8kOQzST6a5Iokp82Ne0O/7ROGbnuWTZ0kSZKOd78J/FPg9cB3AR8HbjiS/+Crad8DLGzq6P7tTwBeDrwI+PAxqmndlwP/BfifgZcBvwh8B3BjkieNtZMtdYKSJEnSFCR5HvAK4Eeq6rf6ee8F7gJ+DvjuJZano5Dki6vqM4dZ/kXAs4BfqKp3H7vKHlNVvzY3671JPga8k67Je8sY+/FMnSRJ0oT0l2/9dpLzk3yov6xsX5Jvmht3VZIDSf6HJP9Pkk8n+cskF8+N25nkN5J8uB9zX5LfSXLK3Lj1y8OeneSGfr/3JrmoX/6qJH+R5FNJ/jjJP1xQ+z9NcluSv0/yYJLfTPK0BfX8TpK/SfL/JrkaeMpRHLLvBj4L/N76jKp6FLgG+I4kX7zRiknuSnLIf7qTfEN/LL6nn/7aJG9Lcn//3O5N8p+2eindgv3t6i8Z/bskn0zy9iTPmhvzsiTX9ZeYfjrJnUl+MsmOuXFPTPLmJJ/o/62uBU7dQk1Dn/MTk7yp//d+oM/uU+a29eR+zMH+EsW7k/xEksyMWb/08fuS/NskDwD/LclVwD8BTumXV/8zciHwObp+52fX5/fb+vok/3f/8/F3/f5+McmXHOlx2KJP9H9+dsGyM5L8Yf9v87Ekr++b08OyqZMkSZqebwZ+EvhZ4IeAHcA75v+zDDwZ+B3gt4HdwM3Aryd5ycyYpwF/D/zvwC7gp4AzgT9N8g8W7Ps/AX9Id8nbLcCVSX4R+FHgUuAiurMjvzO7UpJ/BbwZ+M90jdZP9fu7fq7xeCvdJZI/3T+3R4H/a76IJBdm2OebngN8pKo+PTf/LuBE4L87zLr/HviuJE+dm//DwEPAdf30O4BT6I7Bd9Adh88wwv+1k+yiO96fojsePwr8Y+BP5hrvZ9Jd5vcjwHcC/w54A/ALc5v8DbpLAf8N8H3A3cz9Ww009Dn/GlB0Z0t/Dvj+ft768/ui/vldBPwK3WWS7+zrm68duiwEeBVwIfDzdP8OD9BdXvki4Hv7ba7/ouM3Z+YDnA7cClxMl8Ffoztuv3VER+AIJNmR5IuTPJfuuX0Q+KMFQ98GvJvu5+vtwP9J17QeXlX58OHDhw8fPnz4WNIDuAo4sMGyc+j+Q/xtM/M+CjwMPHVm3lo/7hVz2y3gJTPzvhh4ENh7mHp2AKf1637vzPw39PNePTPvqXRN1yeAJ8/M/1/6sV/TTz+D7qzJ6+f29Y39uO/pp7+9nz5/btz1/fxzZua9ut/3t25yfP8IuGnB/G/rt/nNh1n3tL7ufzYz7wl0DcSb++mT++1890h5KOANM9P7gL8ETpiZdwbdWZ5/s8E2Qvcxq5/ps/JF/fxn9c/n0rnxv97v98KBNW76nGey++/m5r+J7pcI6ae/a9G+gSvomsST57b3tqE/Q/0xeNzxPMyx+mHg88CXzyx7D/CeBc/pnI22d5j9PNivW3S/XHn63PI39Msumpt/B/BHm23fM3WSJEnT8/6qenhm+o7+z9Pnxn26qv54faK6zx/95fy4JD/aXxb5KbpG6d5+0eMu8etdP7O9h4H76Zqmv5kZ8xf9n+t3+Pt2ujM4/yHJCesP4APA3wDrd6J8EV3TMX/J4zXzRVTV1VV1QlW9d0GNj3t6dP9ZXjT/sKrqPuC9dGeF1u2ia2qu7qc/AdwD/Kv+8tIzN9vuUElOAp4P/F51l4yu1/UR4E+Bb50Z+/R0l9F+DHiErun7l3SXrn5FP+wb6P4d/uPcrg45vps4kuf8h3PTd9D9cuEr++lvoWumfndu3G/TnUl90dz8tx1hrYfoL/f85SR/Rdc4fpburGzozlJvh5cCLwZeQ/dv8q4FZ9bh0ON1J4f+XB/Cpk6SJGm5HqU7O7bIjpkxsx6anajHbhYxf7nkwxzqM7PjkvwYj10W+X3A2cALN9jeom0+ssG82fXXm4r9dP+Bnn08me4OgQBPBx6uqvnPGv23BXUM9RDdJabznjqz/HCuBr4xyRn99KuA/VV1E0B1p1O+ne6M2i8BH05yT5IfPYqaZ2sM3d065/01/fPqL2G8lu6s178E/kfg63ns8sX1f4en93/OH88jOr5H+Jznj+98Vp8GPFSH3vDkr2eWz1p0LI7Ub9FdevlGuufx9cBr5+oaVVXdVlXvr6or6W6QclZfw7xFx2vTmrz7pSRJ0nLdD5yc5MSqemRu2Vf3fx5NU7OZ84H/UlU/uT5jpoEZy/qNIV7G4kZzffnHgacmecJcY/eVC9YZ6i7ge5M8sR7/ubqz6JrP/Zus/xbgMuCHk/wa3We+fml2QFXdA7y6v7HH84BLgDcn+WhVXT+/wSPwMN1Zxq9asOyreOy4/UO6S3BfVVW/vT4gycvn1llviL6S7kwbM9NHZMTn/BDwtAX5X3/On5gbv+is62D950R3012SOfvZvv/+aLZ7JKrqI0ke4vCf5zwinqmTJElarj+m+0X7olvrfz/df8Tv3sb9P5FD78J30cj7eBfdJXanV9W+BY+P9OPeT3d28vvn1j//KPZ9Ld3n4H5wfUZ/6ecP0X1WacNb4gNU1f8H/D7dGbofpDtr8u83GFtVdSuPfWfaPz6Kuqmqv6W7Gc0Pzt5MJsnX0F3Kt37p6RP7Pz87M+YJwCvnNvkBun+H/2lu/paP7wjP+b10PckPzs1/JV3TfdOAbXwGGHrnyi+my9h85i8cuP5RS/IcurPTfzXWNj1TJ0mStFz/ma7puSrJs+n+4/0kuv9o76a7ccLnt3H/7wRel+SngT+ju3TvB8bcQVX9VZJfBt7U34r/vXQ3yziN7vK3K6rqj6vqXUn+BPiNJCfTff7vh1jQKCR5NXAl8NLDfa6uqm5N8nvAr/aNzkfo7th4Boc2PRu5GriA7k6EfzLThNLfzfDX6L4yYT9dw3Ah3SWz754Z9yjdTUNeM3Cf636W7nNW70jyZuBL+zo+SXe3SIAPAR8DfiHJ5+galp+Y31BV3Z3kd4Cf6y/ZvJnu+J93JAUNfc4DXQ/8CXB5kp10Z1bPo7tD5y9V1YMDtvFBurN9P0p3SejfV9UdiwZW1SeT3AT8ZJKP093A5Efo7uQ5uj7Pb6P7nOnfA8+lu3PtAeDfjrUfmzpJkqQlqqpK8t3A/0F3R8efpTtDcSvdXSF/f5tL+Dm6Gzf8BN1ZqPfS3aL+nsOsc8Sq6qeTfIjus0uvpbuM7j662/D/5czQ76P7rNMv0d005Vq6S/vePrfJL6JrJja94Qndmcdf4LEbh9wG7KqqPx9Y/rvoPuN1Ct3xmvXXdDeW+ed03/f293Q3A/muqrplZtwONv7s5Iaq6p1JvhP4F3Q3OHmE7q6M/1tVHezHPJLuO/PeRNeAPkTX8N7LoY3DP6P7eoT/le5GJO+m+7qBPzmCsoY+5yHP7/P98/tF4HV0Z7A+2m/7Vwdu5gq6z4H+It2/78fo7ri6kQvo7vh5GfB3dMf1x+m+pmFsH6BreJ9Bl9V76W4K868HNqyDrN9KVJIkSZI0QX6mTpIkSZImbNOmLsmVSe5PcucGy5PkjUn2J7k9yfPHL1OrzhyqBeZQy2YG1QJzeHzr//1OONxj2TW2oLXjNORM3VV0X7K4kXPpvqTvTGAP3fWp0tiuwhxq+a7CHGq5rsIMavmuwhwez76VQ79L8HGPJM9YWnXt+CdscpyOZTGbdpBV9b5N/uF2A1f3X0J4U5KnJHl6VY3xxYASYA7VBnOoZTODaoE5PO7dQvdl3Idz8FgU0rg/YPPjdMyMcVrwFLo7F6070M875Ac3yR6639hw0kknveDZz372CLvX8eaWW255sKp2HuFq5lCj2s4cmkEN4WuhWmAOV88LXvCCQePW1ta2uZLOFjO47arqExz6xehLM0ZTt+g2sgtvqVlVe4G9AGtra7Vv374Rdq/jTZKPbWW1BfPMobZsO3NoBjWEr4VqgTnUsm0xgytnjLtfHqD74sh1p+IpWR175lAtMIdaNjOoFphD6Rgbo6m7Fnh1fweYFwKf9JppLYE5VAvMoZbNDKoF5lA6xja9/DLJ7wLnACcnOUD3bfZPAKiqy4HrgPOA/cCngYu2q1itLnOoFphDLZsZVAvModSeIXe/vGCT5QW8drSKpAXMoVpgDrVsZlAtMIdSe8a4/FKSJEmStCQ2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YTZ1kiRJkjRhNnWSJEmSNGE2dZIkSZI0YYOauiS7ktydZH+SSxcs/7Ikf5DktiR3Jblo/FK1ysygWmAO1QJzqGUzg1J7Nm3qkuwALgPOBc4CLkhy1tyw1wIfrKrnAecAv5LkxJFr1Yoyg2qBOVQLzKGWzQxKbRpypu5sYH9V3VNVjwDXALvnxhTwpCQBvhR4CHh01Eq1ysygWmAO1QJzqGUzg1KDhjR1pwD3zUwf6OfNehPwj4CDwB3Aj1fV5+c3lGRPkn1J9j3wwANbLFkraLQMgjnUlvlaqBaYQy2b78lSg4Y0dVkwr+amvwO4Ffhq4OuANyV58iErVe2tqrWqWtu5c+cRlqoVNloGwRxqy3wtVAvMoZbN92SpQUOaugPAaTPTp9L95mXWRcBbq7Mf+Ajw7HFKlMygmmAO1QJzqGUzg1KDhjR1NwNnJjmj/5Dr+cC1c2PuBV4KkOQrgWcB94xZqFaaGVQLzKFaYA61bGZQatAJmw2oqkeTXALcAOwArqyqu5Jc3C+/HPh54Kokd9Cdln9dVT24jXVrhZhBtcAcqgXmUMtmBqU2bdrUAVTVdcB1c/Mun/n7QeBl45YmPcYMqgXmUC0wh1o2Myi1Z9CXj0uSJEmS2mRTJ0mSJEkTZlMnSZIkSRNmUydJkiRJE2ZTJ0mSJEkTZlMnSZIkSRNmUydJkiRJE2ZTJ0mSJEkTZlMnSZIkSRNmUydJkiRJE2ZTJ0mSJEkTZlMnSZIkSRNmUydJkiRJE2ZTJ0mSJEkTZlMnSZIkSRNmUydJkiRJE2ZTJ0mSJEkTNqipS7Iryd1J9ie5dIMx5yS5NcldSd47bpladWZQLTCHaoE51LKZQak9J2w2IMkO4DLg24EDwM1Jrq2qD86MeQrwZmBXVd2b5Cu2qV6tIDOoFphDtcAcatnMoNSmIWfqzgb2V9U9VfUIcA2we27MK4C3VtW9AFV1/7hlasWZQbXAHKoF5lDLZgalBg1p6k4B7puZPtDPm/W1wFOTvCfJLUlePVaBEmZQbTCHaoE51LKZQalBm15+CWTBvFqwnRcALwW+BHh/kpuq6sOP21CyB9gDcPrppx95tVpVo2UQzKG2zNdCtcAcatl8T5YaNORM3QHgtJnpU4GDC8a8s6r+tqoeBN4HPG9+Q1W1t6rWqmpt586dW61Zq2e0DII51Jb5WqgWmEMtm+/JUoOGNHU3A2cmOSPJicD5wLVzY34f+OYkJyR5IvANwIfGLVUrzAyqBeZQLTCHWjYzKDVo08svq+rRJJcANwA7gCur6q4kF/fLL6+qDyV5J3A78Hngiqq6czsL1+owg2qBOVQLzKGWzQxKbUrV/GXQx8ba2lrt27dvKftW25LcUlVrx2Jf5lAbOVY5NIPaiK+FaoE51LIdywxO2aAvH5ckSZIktcmmTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImbFBTl2RXkruT7E9y6WHGfX2SzyX5gfFKlMyg2mAO1QJzqGUzg1J7Nm3qkuwALgPOBc4CLkhy1gbjfhm4YewitdrMoFpgDtUCc6hlM4NSm4acqTsb2F9V91TVI8A1wO4F434MeAtw/4j1SWAG1QZzqBaYQy2bGZQaNKSpOwW4b2b6QD/vC5KcAnwvcPl4pUlfYAbVAnOoFphDLZsZlBo0pKnLgnk1N/2rwOuq6nOH3VCyJ8m+JPseeOCBgSVK42UQzKG2zNdCtcAcatl8T5YadMKAMQeA02amTwUOzo1ZA65JAnAycF6SR6vq7bODqmovsBdgbW1t/gVA2shoGQRzqC3ztVAtMIdaNt+TpQYNaepuBs5McgbwX4HzgVfMDqiqM9b/nuQq4B2LfnClLTKDaoE5VAvMoZbNDEoN2rSpq6pHk1xCd/eiHcCVVXVXkov75V4vrW1lBtUCc6gWmEMtmxmU2jTkTB1VdR1w3dy8hT+0VXXh0ZclPZ4ZVAvMoVpgDrVsZlBqz6AvH5ckSZIktcmmTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImzKZOkiRJkibMpk6SJEmSJsymTpIkSZImbFBTl2RXkruT7E9y6YLlr0xye/+4Mcnzxi9Vq8wMqgXmUC0wh1o2Myi1Z9OmLskO4DLgXOAs4IIkZ80N+wjwrVX1XODngb1jF6rVZQbVAnOoFphDLZsZlNo05Ezd2cD+qrqnqh4BrgF2zw6oqhur6uF+8ibg1HHL1Iozg2qBOVQLzKGWzQxKDRrS1J0C3DczfaCft5HXANcfTVHSHDOoFphDtcAcatnMoNSgEwaMyYJ5tXBg8hK6H95v2mD5HmAPwOmnnz6wRGm8DPZjzKG2wtdCtcAcatl8T5YaNORM3QHgtJnpU4GD84OSPBe4AthdVZ9YtKGq2ltVa1W1tnPnzq3Uq9U0WgbBHGrLfC1UC8yhls33ZKlBQ5q6m4Ezk5yR5ETgfODa2QFJTgfeCryqqj48fplacWZQLTCHaoE51LKZQalBm15+WVWPJrkEuAHYAVxZVXclubhffjnweuDLgTcnAXi0qta2r2ytEjOoFphDtcAcatnMoNSmVC28DHrbra2t1b59+5ayb7UtyS3H6sXfHGojxyqHZlAb8bVQLTCHWrZjmcEpG/Tl45IkSZKkNtnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShNnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShNnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShNnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShNnUSZIkSdKE2dRJkiRJ0oTZ1EmSJEnShNnUSZIkSdKEDWrqkuxKcneS/UkuXbA8Sd7YL789yfPHL1WrzAyqBeZQLTCHWjYzKLVn06YuyQ7gMuBc4CzggiRnzQ07Fzizf+wBfn3kOrXCzKBaYA7VAnOoZTODUpuGnKk7G9hfVfdU1SPANcDuuTG7gaurcxPwlCRPH7lWrS4zqBaYQ7XAHGrZzKDUoCFN3SnAfTPTB/p5RzpG2iozqBaYQ7XAHGrZzKDUoBMGjMmCebWFMSTZQ3caHuAzSe4csP/tdjLw4LKLoI06WqgB4Flz06NlEMxh4zVAO3VsWw7N4GG1UEcLNcChGQRzuCo1QDt1rNp7civH3Toes+i1UHOGNHUHgNNmpk8FDm5hDFW1F9gLkGRfVa0dUbXbwDraqmG9jrlZo2UQzGHLNbRWx9wsXwtXpI4WalivY8Fsc7gCNbRWx9ys4/o9uYUarOPQGpa5/6kYcvnlzcCZSc5IciJwPnDt3JhrgVf3dzt6IfDJqvr4yLVqdZlBtcAcqgXmUMtmBqUGbXqmrqoeTXIJcAOwA7iyqu5KcnG//HLgOuA8YD/waeCi7StZq8YMqgXmUC0wh1o2Myi1acjll1TVdXQ/oLPzLp/5ewGvPcJ97z3C8dvFOh7TQg2woI5tyuDCfS1JC3W0UAM0XIevhcdEC3W0UANsUIc53HYt1AAN13Gcvye3UANYx6wWamheup87SZIkSdIUDflMnSRJkiSpUdvS1CXZleTuJPuTXLpgeZK8sV9+e5LnD113xBpe2e/79iQ3JnnezLKPJrkjya1He8edAXWck+ST/b5uTfL6oeuOXMdPzdRwZ5LPJXlav2yU45HkyiT3Z4PbFY+ZixYyOLCOlcnhqmVw4HNemRy2kMGBdZhD35PNIb4nb1cOzeDj9nFMXwuPe1U16oPuQ7N/BTwTOBG4DThrbsx5wPV032PyQuADQ9cdsYYXA0/t/37ueg399EeBk4/RsTgHeMdW1h2zjrnxLwfevQ3H41uA5wN3brB8lFy0kEFzuNoZNIftZdAc+p5sDpeTwxYy2EoOzeByMrgqj+04U3c2sL+q7qmqR4BrgN1zY3YDV1fnJuApSZ4+cN1RaqiqG6vq4X7yJrrvUBnb0TyfsY7FVrZ1AfC7W9zXhqrqfcBDhxkyVi5ayOCgOlYoh6uWQQausyo5bCGDW9mWOfQ9eYx1j3ZbU89hCxkcVIevhRuaegZXwnY0dacA981MH+jnDRkzZN2xapj1GrrfBKwr4I+S3JJkzxb2f6R1vCjJbUmuT/KcI1x3zDpI8kRgF/CWmdljHY/NjJWLFjI4tI5Zx3MOVy2Dh9vWdu3vaOqYtR05bCGDR7Qtc+h7sjn0PRlfC4+XDK6EQV9pcISyYF4NHDNk3bFq6AYmL6H7wf2mmdnfWFUHk3wF8K4kf9H/NmE76vhz4Guq6lNJzgPeDpw5cN0x61j3cuBPq2r2NydjHY/NjJWLFjI4tI5u4PGfw1XL4OG2tV37O5o6uoHbl8MWMji0jnXm0Pdkc3go35N9LYTpZXAlbMeZugPAaTPTpwIHB44Zsu5YNZDkucAVwO6q+sT6/Ko62P95P/A2utO8W7FpHVX1N1X1qf7v1wFPSHLy0OcwVh0zzmfuFPuIx2MzY+WihQwOrWNVcrhqGTzctrZrf0dTx3bnsIUMDqpjhjn0Pdkc+p7sa+HxkcHVUCN/SI/u7N89wBk89uHF58yN+U4e/8HHPxu67og1nA7sB148N/8k4Ekzf78R2LWNx+Kr4AvfF3g2cG9/XEY5FkdyXIEvo7u2+aTtOB79Np7Bxh+IHSUXLWTQHK52Bs1hexk0h74nm8Pl5LCFDLaSQzO4nAyuymN7NtrdrebDdHem+Zl+3sXAxf3fA1zWL78DWDvcuttUwxXAw8Ct/WNfP/+ZfThuA+46mhoG1nFJv5/b6D6U++LDrbtddfTTFwLXzK032vGg+03Px4HP0v2W5TXblYsWMmgOVzuD5rC9DJpD35PN4XJy2EIGW8mhGVxOBlfhsf6bAEmSJEnSBG3Ll49LkiRJko4NmzpJkiRJmjCbOkmSJEmaMJs6SZIkSZowmzpJkiRJmjCbOkmSJEmaMJs6SZIkSZowmzpJkiRJmrD/H7USOsaVnud9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1008x432 with 10 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot relevant feature interactions\n",
    "# IGNORE FOR NOW - TT\n",
    "from statsmodels.graphics.correlation import plot_corr\n",
    "\n",
    "fig, axs = plt.subplots(2,5, figsize=(14,6),)\n",
    "fig.subplots_adjust(hspace = 0.5, wspace=.2)\n",
    "axs = axs.ravel()\n",
    "\n",
    "for index, column in enumerate(df.columns):\n",
    "    axs[index-1].set_title(\"{} vs. load_shortfall_3h\".format(column),fontsize=16)\n",
    "    axs[index-1].scatter(x=df[column],y=df['load_shortfall_3h'],color='blue',edgecolor='k')\n",
    "    \n",
    "fig.tight_layout(pad=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "6dd6ee8c",
   "metadata": {},
   "outputs": [
    {
     "ename": "KeyError",
     "evalue": "\"['load_shortfall_3h'] not found in axis\"",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "Input \u001b[1;32mIn [49]\u001b[0m, in \u001b[0;36m<cell line: 3>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;66;03m# evaluate correlation\u001b[39;00m\n\u001b[1;32m----> 3\u001b[0m corr \u001b[38;5;241m=\u001b[39m \u001b[43mdf\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdrop\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mload_shortfall_3h\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43maxis\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[38;5;124;43mcolumns\u001b[39;49m\u001b[38;5;124;43m'\u001b[39;49m\u001b[43m)\u001b[49m\u001b[38;5;241m.\u001b[39mcorr()\n\u001b[0;32m      4\u001b[0m fig\u001b[38;5;241m=\u001b[39mplot_corr(corr,xnames\u001b[38;5;241m=\u001b[39mcorr\u001b[38;5;241m.\u001b[39mcolumns)\n",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\util\\_decorators.py:311\u001b[0m, in \u001b[0;36mdeprecate_nonkeyword_arguments.<locals>.decorate.<locals>.wrapper\u001b[1;34m(*args, **kwargs)\u001b[0m\n\u001b[0;32m    305\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mlen\u001b[39m(args) \u001b[38;5;241m>\u001b[39m num_allow_args:\n\u001b[0;32m    306\u001b[0m     warnings\u001b[38;5;241m.\u001b[39mwarn(\n\u001b[0;32m    307\u001b[0m         msg\u001b[38;5;241m.\u001b[39mformat(arguments\u001b[38;5;241m=\u001b[39marguments),\n\u001b[0;32m    308\u001b[0m         \u001b[38;5;167;01mFutureWarning\u001b[39;00m,\n\u001b[0;32m    309\u001b[0m         stacklevel\u001b[38;5;241m=\u001b[39mstacklevel,\n\u001b[0;32m    310\u001b[0m     )\n\u001b[1;32m--> 311\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m func(\u001b[38;5;241m*\u001b[39margs, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\frame.py:4954\u001b[0m, in \u001b[0;36mDataFrame.drop\u001b[1;34m(self, labels, axis, index, columns, level, inplace, errors)\u001b[0m\n\u001b[0;32m   4806\u001b[0m \u001b[38;5;129m@deprecate_nonkeyword_arguments\u001b[39m(version\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m, allowed_args\u001b[38;5;241m=\u001b[39m[\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mself\u001b[39m\u001b[38;5;124m\"\u001b[39m, \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mlabels\u001b[39m\u001b[38;5;124m\"\u001b[39m])\n\u001b[0;32m   4807\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mdrop\u001b[39m(\n\u001b[0;32m   4808\u001b[0m     \u001b[38;5;28mself\u001b[39m,\n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   4815\u001b[0m     errors: \u001b[38;5;28mstr\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mraise\u001b[39m\u001b[38;5;124m\"\u001b[39m,\n\u001b[0;32m   4816\u001b[0m ):\n\u001b[0;32m   4817\u001b[0m     \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[0;32m   4818\u001b[0m \u001b[38;5;124;03m    Drop specified labels from rows or columns.\u001b[39;00m\n\u001b[0;32m   4819\u001b[0m \n\u001b[1;32m   (...)\u001b[0m\n\u001b[0;32m   4952\u001b[0m \u001b[38;5;124;03m            weight  1.0     0.8\u001b[39;00m\n\u001b[0;32m   4953\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[1;32m-> 4954\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43msuper\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdrop\u001b[49m\u001b[43m(\u001b[49m\n\u001b[0;32m   4955\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlabels\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlabels\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   4956\u001b[0m \u001b[43m        \u001b[49m\u001b[43maxis\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43maxis\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   4957\u001b[0m \u001b[43m        \u001b[49m\u001b[43mindex\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mindex\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   4958\u001b[0m \u001b[43m        \u001b[49m\u001b[43mcolumns\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mcolumns\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   4959\u001b[0m \u001b[43m        \u001b[49m\u001b[43mlevel\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlevel\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   4960\u001b[0m \u001b[43m        \u001b[49m\u001b[43minplace\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43minplace\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   4961\u001b[0m \u001b[43m        \u001b[49m\u001b[43merrors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43merrors\u001b[49m\u001b[43m,\u001b[49m\n\u001b[0;32m   4962\u001b[0m \u001b[43m    \u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:4267\u001b[0m, in \u001b[0;36mNDFrame.drop\u001b[1;34m(self, labels, axis, index, columns, level, inplace, errors)\u001b[0m\n\u001b[0;32m   4265\u001b[0m \u001b[38;5;28;01mfor\u001b[39;00m axis, labels \u001b[38;5;129;01min\u001b[39;00m axes\u001b[38;5;241m.\u001b[39mitems():\n\u001b[0;32m   4266\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m labels \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m-> 4267\u001b[0m         obj \u001b[38;5;241m=\u001b[39m \u001b[43mobj\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_drop_axis\u001b[49m\u001b[43m(\u001b[49m\u001b[43mlabels\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43maxis\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mlevel\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mlevel\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43merrors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43merrors\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   4269\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m inplace:\n\u001b[0;32m   4270\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_update_inplace(obj)\n",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\generic.py:4311\u001b[0m, in \u001b[0;36mNDFrame._drop_axis\u001b[1;34m(self, labels, axis, level, errors, consolidate, only_slice)\u001b[0m\n\u001b[0;32m   4309\u001b[0m         new_axis \u001b[38;5;241m=\u001b[39m axis\u001b[38;5;241m.\u001b[39mdrop(labels, level\u001b[38;5;241m=\u001b[39mlevel, errors\u001b[38;5;241m=\u001b[39merrors)\n\u001b[0;32m   4310\u001b[0m     \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m-> 4311\u001b[0m         new_axis \u001b[38;5;241m=\u001b[39m \u001b[43maxis\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mdrop\u001b[49m\u001b[43m(\u001b[49m\u001b[43mlabels\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43merrors\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43merrors\u001b[49m\u001b[43m)\u001b[49m\n\u001b[0;32m   4312\u001b[0m     indexer \u001b[38;5;241m=\u001b[39m axis\u001b[38;5;241m.\u001b[39mget_indexer(new_axis)\n\u001b[0;32m   4314\u001b[0m \u001b[38;5;66;03m# Case for non-unique axis\u001b[39;00m\n\u001b[0;32m   4315\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n",
      "File \u001b[1;32m~\\anaconda3\\lib\\site-packages\\pandas\\core\\indexes\\base.py:6644\u001b[0m, in \u001b[0;36mIndex.drop\u001b[1;34m(self, labels, errors)\u001b[0m\n\u001b[0;32m   6642\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m mask\u001b[38;5;241m.\u001b[39many():\n\u001b[0;32m   6643\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m errors \u001b[38;5;241m!=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mignore\u001b[39m\u001b[38;5;124m\"\u001b[39m:\n\u001b[1;32m-> 6644\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mKeyError\u001b[39;00m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mlist\u001b[39m(labels[mask])\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m not found in axis\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m   6645\u001b[0m     indexer \u001b[38;5;241m=\u001b[39m indexer[\u001b[38;5;241m~\u001b[39mmask]\n\u001b[0;32m   6646\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mdelete(indexer)\n",
      "\u001b[1;31mKeyError\u001b[0m: \"['load_shortfall_3h'] not found in axis\""
     ]
    }
   ],
   "source": [
    "# evaluate correlation\n",
    "\n",
    "corr = df.drop('load_shortfall_3h', axis='columns').corr()\n",
    "fig=plot_corr(corr,xnames=corr.columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "de51df85",
   "metadata": {},
   "outputs": [],
   "source": [
    "# have a look at feature distributions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3fa93ec6",
   "metadata": {},
   "source": [
    "<a id=\"four\"></a>\n",
    "## 4. Data Engineering\n",
    "<a class=\"anchor\" id=\"1.1\"></a>\n",
    "<a href=#cont>Back to Table of Contents</a>\n",
    "\n",
    "---\n",
    "    \n",
    "| ⚡ Description: Data engineering ⚡ |\n",
    "| :--------------------------- |\n",
    "| In this section you are required to: clean the dataset, and possibly create new features - as identified in the EDA phase. |\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "059c2f3e",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'df_train' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Input \u001b[1;32mIn [59]\u001b[0m, in \u001b[0;36m<cell line: 2>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      1\u001b[0m \u001b[38;5;66;03m# remove missing values/ features\u001b[39;00m\n\u001b[1;32m----> 2\u001b[0m \u001b[43mdf_train\u001b[49m\u001b[38;5;241m.\u001b[39misnull()\u001b[38;5;241m.\u001b[39msum()\n",
      "\u001b[1;31mNameError\u001b[0m: name 'df_train' is not defined"
     ]
    }
   ],
   "source": [
    "# remove missing values/ features\n",
    "df_train.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "84eea17b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create new features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "59692724",
   "metadata": {},
   "outputs": [],
   "source": [
    "# engineer existing features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43b2d523",
   "metadata": {},
   "source": [
    "<a id=\"five\"></a>\n",
    "## 5. Modelling\n",
    "<a class=\"anchor\" id=\"1.1\"></a>\n",
    "<a href=#cont>Back to Table of Contents</a>\n",
    "\n",
    "---\n",
    "    \n",
    "| ⚡ Description: Modelling ⚡ |\n",
    "| :--------------------------- |\n",
    "| In this section, you are required to create one or more regression models that are able to accurately predict the thee hour load shortfall. |\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2344b3e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# split data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c58df02",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create targets and features dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "20d073e0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create one or more ML models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a70c15d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# evaluate one or more ML models"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b530251",
   "metadata": {},
   "source": [
    "<a id=\"six\"></a>\n",
    "## 6. Model Performance\n",
    "<a class=\"anchor\" id=\"1.1\"></a>\n",
    "<a href=#cont>Back to Table of Contents</a>\n",
    "\n",
    "---\n",
    "    \n",
    "| ⚡ Description: Model performance ⚡ |\n",
    "| :--------------------------- |\n",
    "| In this section you are required to compare the relative performance of the various trained ML models on a holdout dataset and comment on what model is the best and why. |\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a69b5a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compare model performance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3874a7c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Choose best model and motivate why it is the best choice"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a8ad0c0d",
   "metadata": {},
   "source": [
    "<a id=\"seven\"></a>\n",
    "## 7. Model Explanations\n",
    "<a class=\"anchor\" id=\"1.1\"></a>\n",
    "<a href=#cont>Back to Table of Contents</a>\n",
    "\n",
    "---\n",
    "    \n",
    "| ⚡ Description: Model explanation ⚡ |\n",
    "| :--------------------------- |\n",
    "| In this section, you are required to discuss how the best performing model works in a simple way so that both technical and non-technical stakeholders can grasp the intuition behind the model's inner workings. |\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ff741c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# discuss chosen methods logic"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}