Pytorch_ANN_Regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Full Artificial Neural Network Code Along\n",
    "In the last section we took in four continuous variables (lengths) to perform a classification. In this section we'll combine continuous and categorical data to perform a regression. The goal is to estimate the cost of a New York City cab ride from several inputs. The inspiration behind this code along is a recent <a href='https://www.kaggle.com/c/new-york-city-taxi-fare-prediction'>Kaggle competition</a>.\n",
    "\n",
    "<div class=\"alert alert-success\"><strong>NOTE:</strong> In this notebook we'll perform a regression with one output value. In the next one we'll perform a binary classification with two output values.</div>\n",
    "\n",
    "## Working with tabular data\n",
    "Deep learning with neural networks is often associated with sophisticated image recognition, and in upcoming sections we'll train models based on properties like pixels patterns and colors.\n",
    "\n",
    "Here we're working with tabular data (spreadsheets, SQL tables, etc.) with columns of values that may or may not be relevant. As it happens, neural networks can learn to make connections we probably wouldn't have developed on our own. However, to do this we have to handle categorical values separately from continuous ones. Make sure to watch the theory lectures! You'll want to be comfortable with:\n",
    "* continuous vs. categorical values\n",
    "* embeddings\n",
    "* batch normalization\n",
    "* dropout layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform standard imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load the NYC Taxi Fares dataset\n",
    "The <a href='https://www.kaggle.com/c/new-york-city-taxi-fare-prediction'>Kaggle competition</a> provides a dataset with about 55 million records. The data contains only the pickup date & time, the latitude & longitude (GPS coordinates) of the pickup and dropoff locations, and the number of passengers. It is up to the contest participant to extract any further information. For instance, does the time of day matter? The day of the week? How do we determine the distance traveled from pairs of GPS coordinates?\n",
    "\n",
    "For this exercise we've whittled the dataset down to just 120,000 records from April 11 to April 24, 2010. The records are randomly sorted. We'll show how to calculate distance from GPS coordinates, and how to create a pandas datatime object from a text column. This will let us quickly get information like day of the week, am vs. pm, etc.\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pickup_datetime</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>fare_class</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>passenger_count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2010-04-19 08:17:56 UTC</td>\n",
       "      <td>6.5</td>\n",
       "      <td>0</td>\n",
       "      <td>-73.992365</td>\n",
       "      <td>40.730521</td>\n",
       "      <td>-73.975499</td>\n",
       "      <td>40.744746</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010-04-17 15:43:53 UTC</td>\n",
       "      <td>6.9</td>\n",
       "      <td>0</td>\n",
       "      <td>-73.990078</td>\n",
       "      <td>40.740558</td>\n",
       "      <td>-73.974232</td>\n",
       "      <td>40.744114</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2010-04-17 11:23:26 UTC</td>\n",
       "      <td>10.1</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.994149</td>\n",
       "      <td>40.751118</td>\n",
       "      <td>-73.960064</td>\n",
       "      <td>40.766235</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2010-04-11 21:25:03 UTC</td>\n",
       "      <td>8.9</td>\n",
       "      <td>0</td>\n",
       "      <td>-73.990485</td>\n",
       "      <td>40.756422</td>\n",
       "      <td>-73.971205</td>\n",
       "      <td>40.748192</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2010-04-17 02:19:01 UTC</td>\n",
       "      <td>19.7</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.990976</td>\n",
       "      <td>40.734202</td>\n",
       "      <td>-73.905956</td>\n",
       "      <td>40.743115</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           pickup_datetime  fare_amount  fare_class  pickup_longitude  \\\n",
       "0  2010-04-19 08:17:56 UTC          6.5           0        -73.992365   \n",
       "1  2010-04-17 15:43:53 UTC          6.9           0        -73.990078   \n",
       "2  2010-04-17 11:23:26 UTC         10.1           1        -73.994149   \n",
       "3  2010-04-11 21:25:03 UTC          8.9           0        -73.990485   \n",
       "4  2010-04-17 02:19:01 UTC         19.7           1        -73.990976   \n",
       "\n",
       "   pickup_latitude  dropoff_longitude  dropoff_latitude  passenger_count  \n",
       "0        40.730521         -73.975499         40.744746                1  \n",
       "1        40.740558         -73.974232         40.744114                1  \n",
       "2        40.751118         -73.960064         40.766235                2  \n",
       "3        40.756422         -73.971205         40.748192                1  \n",
       "4        40.734202         -73.905956         40.743115                1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('NYCTaxiFares.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count    120000.000000\n",
       "mean         10.040326\n",
       "std           7.500134\n",
       "min           2.500000\n",
       "25%           5.700000\n",
       "50%           7.700000\n",
       "75%          11.300000\n",
       "max          49.900000\n",
       "Name: fare_amount, dtype: float64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['fare_amount'].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this we see that fares range from \\\\$2.50 to \\\\$49.90, with a mean of \\\\$10.04 and a median of \\\\$7.70"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calculate the distance traveled\n",
    "The <a href='https://en.wikipedia.org/wiki/Haversine_formula'>haversine formula</a> calculates the distance on a sphere between two sets of GPS coordinates.<br>\n",
    "Here we assign latitude values with $\\varphi$ (phi) and longitude with $\\lambda$ (lambda).\n",
    "\n",
    "The distance formula works out to\n",
    "\n",
    "${\\displaystyle d=2r\\arcsin \\left({\\sqrt {\\sin ^{2}\\left({\\frac {\\varphi _{2}-\\varphi _{1}}{2}}\\right)+\\cos(\\varphi _{1})\\:\\cos(\\varphi _{2})\\:\\sin ^{2}\\left({\\frac {\\lambda _{2}-\\lambda _{1}}{2}}\\right)}}\\right)}$\n",
    "\n",
    "where\n",
    "\n",
    "$$\n",
    "\\begin{split}\n",
    "r&: \\text{radius of the sphere (Earth's radius averages 6371 km)}\\\\\n",
    "\\varphi_1, \\varphi_2&: \\text{latitudes of point 1 and point 2 in radians}\\\\\n",
    "\\lambda_1, \\lambda_2&: \\text{longitudes of point 1 and point 2 in radians}\n",
    "\\end{split}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def haversine_distance(df, lat1, long1, lat2, long2):\n",
    "    \"\"\"\n",
    "    Calculates the haversine distance between 2 sets of GPS coordinates in df\n",
    "    \"\"\"\n",
    "    r = 6371  # average radius of Earth in kilometers\n",
    "       \n",
    "    phi1 = np.radians(df[lat1])\n",
    "    phi2 = np.radians(df[lat2])\n",
    "    \n",
    "    delta_phi = np.radians(df[lat2]-df[lat1])\n",
    "    delta_lambda = np.radians(df[long2]-df[long1])\n",
    "     \n",
    "    a = np.sin(delta_phi/2)**2 + np.cos(phi1) * np.cos(phi2) * np.sin(delta_lambda/2)**2\n",
    "    c = 2 * np.arctan2(np.sqrt(a), np.sqrt(1-a))\n",
    "    d = (r * c) # in kilometers\n",
    "\n",
    "    return d"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pickup_datetime</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>fare_class</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>dist_km</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2010-04-19 08:17:56 UTC</td>\n",
       "      <td>6.5</td>\n",
       "      <td>0</td>\n",
       "      <td>-73.992365</td>\n",
       "      <td>40.730521</td>\n",
       "      <td>-73.975499</td>\n",
       "      <td>40.744746</td>\n",
       "      <td>1</td>\n",
       "      <td>2.126312</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010-04-17 15:43:53 UTC</td>\n",
       "      <td>6.9</td>\n",
       "      <td>0</td>\n",
       "      <td>-73.990078</td>\n",
       "      <td>40.740558</td>\n",
       "      <td>-73.974232</td>\n",
       "      <td>40.744114</td>\n",
       "      <td>1</td>\n",
       "      <td>1.392307</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2010-04-17 11:23:26 UTC</td>\n",
       "      <td>10.1</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.994149</td>\n",
       "      <td>40.751118</td>\n",
       "      <td>-73.960064</td>\n",
       "      <td>40.766235</td>\n",
       "      <td>2</td>\n",
       "      <td>3.326763</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2010-04-11 21:25:03 UTC</td>\n",
       "      <td>8.9</td>\n",
       "      <td>0</td>\n",
       "      <td>-73.990485</td>\n",
       "      <td>40.756422</td>\n",
       "      <td>-73.971205</td>\n",
       "      <td>40.748192</td>\n",
       "      <td>1</td>\n",
       "      <td>1.864129</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2010-04-17 02:19:01 UTC</td>\n",
       "      <td>19.7</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.990976</td>\n",
       "      <td>40.734202</td>\n",
       "      <td>-73.905956</td>\n",
       "      <td>40.743115</td>\n",
       "      <td>1</td>\n",
       "      <td>7.231321</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           pickup_datetime  fare_amount  fare_class  pickup_longitude  \\\n",
       "0  2010-04-19 08:17:56 UTC          6.5           0        -73.992365   \n",
       "1  2010-04-17 15:43:53 UTC          6.9           0        -73.990078   \n",
       "2  2010-04-17 11:23:26 UTC         10.1           1        -73.994149   \n",
       "3  2010-04-11 21:25:03 UTC          8.9           0        -73.990485   \n",
       "4  2010-04-17 02:19:01 UTC         19.7           1        -73.990976   \n",
       "\n",
       "   pickup_latitude  dropoff_longitude  dropoff_latitude  passenger_count  \\\n",
       "0        40.730521         -73.975499         40.744746                1   \n",
       "1        40.740558         -73.974232         40.744114                1   \n",
       "2        40.751118         -73.960064         40.766235                2   \n",
       "3        40.756422         -73.971205         40.748192                1   \n",
       "4        40.734202         -73.905956         40.743115                1   \n",
       "\n",
       "    dist_km  \n",
       "0  2.126312  \n",
       "1  1.392307  \n",
       "2  3.326763  \n",
       "3  1.864129  \n",
       "4  7.231321  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['dist_km'] = haversine_distance(df,'pickup_latitude', 'pickup_longitude', 'dropoff_latitude', 'dropoff_longitude')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add a datetime column and derive useful statistics\n",
    "By creating a datetime object, we can extract information like \"day of the week\", \"am vs. pm\" etc.\n",
    "Note that the data was saved in UTC time. Our data falls in April of 2010 which occurred during Daylight Savings Time in New York. For that reason, we'll make an adjustment to EDT using UTC-4 (subtracting four hours)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pickup_datetime</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>fare_class</th>\n",
       "      <th>pickup_longitude</th>\n",
       "      <th>pickup_latitude</th>\n",
       "      <th>dropoff_longitude</th>\n",
       "      <th>dropoff_latitude</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>dist_km</th>\n",
       "      <th>EDTdate</th>\n",
       "      <th>Hour</th>\n",
       "      <th>AMorPM</th>\n",
       "      <th>Weekday</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2010-04-19 08:17:56 UTC</td>\n",
       "      <td>6.5</td>\n",
       "      <td>0</td>\n",
       "      <td>-73.992365</td>\n",
       "      <td>40.730521</td>\n",
       "      <td>-73.975499</td>\n",
       "      <td>40.744746</td>\n",
       "      <td>1</td>\n",
       "      <td>2.126312</td>\n",
       "      <td>2010-04-19 04:17:56</td>\n",
       "      <td>4</td>\n",
       "      <td>am</td>\n",
       "      <td>Mon</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2010-04-17 15:43:53 UTC</td>\n",
       "      <td>6.9</td>\n",
       "      <td>0</td>\n",
       "      <td>-73.990078</td>\n",
       "      <td>40.740558</td>\n",
       "      <td>-73.974232</td>\n",
       "      <td>40.744114</td>\n",
       "      <td>1</td>\n",
       "      <td>1.392307</td>\n",
       "      <td>2010-04-17 11:43:53</td>\n",
       "      <td>11</td>\n",
       "      <td>am</td>\n",
       "      <td>Sat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2010-04-17 11:23:26 UTC</td>\n",
       "      <td>10.1</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.994149</td>\n",
       "      <td>40.751118</td>\n",
       "      <td>-73.960064</td>\n",
       "      <td>40.766235</td>\n",
       "      <td>2</td>\n",
       "      <td>3.326763</td>\n",
       "      <td>2010-04-17 07:23:26</td>\n",
       "      <td>7</td>\n",
       "      <td>am</td>\n",
       "      <td>Sat</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2010-04-11 21:25:03 UTC</td>\n",
       "      <td>8.9</td>\n",
       "      <td>0</td>\n",
       "      <td>-73.990485</td>\n",
       "      <td>40.756422</td>\n",
       "      <td>-73.971205</td>\n",
       "      <td>40.748192</td>\n",
       "      <td>1</td>\n",
       "      <td>1.864129</td>\n",
       "      <td>2010-04-11 17:25:03</td>\n",
       "      <td>17</td>\n",
       "      <td>pm</td>\n",
       "      <td>Sun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2010-04-17 02:19:01 UTC</td>\n",
       "      <td>19.7</td>\n",
       "      <td>1</td>\n",
       "      <td>-73.990976</td>\n",
       "      <td>40.734202</td>\n",
       "      <td>-73.905956</td>\n",
       "      <td>40.743115</td>\n",
       "      <td>1</td>\n",
       "      <td>7.231321</td>\n",
       "      <td>2010-04-16 22:19:01</td>\n",
       "      <td>22</td>\n",
       "      <td>pm</td>\n",
       "      <td>Fri</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           pickup_datetime  fare_amount  fare_class  pickup_longitude  \\\n",
       "0  2010-04-19 08:17:56 UTC          6.5           0        -73.992365   \n",
       "1  2010-04-17 15:43:53 UTC          6.9           0        -73.990078   \n",
       "2  2010-04-17 11:23:26 UTC         10.1           1        -73.994149   \n",
       "3  2010-04-11 21:25:03 UTC          8.9           0        -73.990485   \n",
       "4  2010-04-17 02:19:01 UTC         19.7           1        -73.990976   \n",
       "\n",
       "   pickup_latitude  dropoff_longitude  dropoff_latitude  passenger_count  \\\n",
       "0        40.730521         -73.975499         40.744746                1   \n",
       "1        40.740558         -73.974232         40.744114                1   \n",
       "2        40.751118         -73.960064         40.766235                2   \n",
       "3        40.756422         -73.971205         40.748192                1   \n",
       "4        40.734202         -73.905956         40.743115                1   \n",
       "\n",
       "    dist_km             EDTdate  Hour AMorPM Weekday  \n",
       "0  2.126312 2010-04-19 04:17:56     4     am     Mon  \n",
       "1  1.392307 2010-04-17 11:43:53    11     am     Sat  \n",
       "2  3.326763 2010-04-17 07:23:26     7     am     Sat  \n",
       "3  1.864129 2010-04-11 17:25:03    17     pm     Sun  \n",
       "4  7.231321 2010-04-16 22:19:01    22     pm     Fri  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['EDTdate'] = pd.to_datetime(df['pickup_datetime'].str[:19]) - pd.Timedelta(hours=4)\n",
    "df['Hour'] = df['EDTdate'].dt.hour\n",
    "df['AMorPM'] = np.where(df['Hour']<12,'am','pm')\n",
    "df['Weekday'] = df['EDTdate'].dt.strftime(\"%a\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2010-04-11 00:00:10')"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['EDTdate'].min()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Timestamp('2010-04-24 23:59:42')"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['EDTdate'].max()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Separate categorical from continuous columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['pickup_datetime', 'fare_amount', 'fare_class', 'pickup_longitude',\n",
       "       'pickup_latitude', 'dropoff_longitude', 'dropoff_latitude',\n",
       "       'passenger_count', 'dist_km', 'EDTdate', 'Hour', 'AMorPM', 'Weekday'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cat_cols = ['Hour', 'AMorPM', 'Weekday']\n",
    "cont_cols = ['pickup_latitude', 'pickup_longitude', 'dropoff_latitude', 'dropoff_longitude', 'passenger_count', 'dist_km']\n",
    "y_col = ['fare_amount']  # this column contains the labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\"><strong>NOTE:</strong> If you plan to use all of the columns in the data table, there's a shortcut to grab the remaining continuous columns:<br>\n",
    "<pre style='background-color:rgb(217,237,247)'>cont_cols = [col for col in df.columns if col not in cat_cols + y_col]</pre>\n",
    "\n",
    "Here we entered the continuous columns explicitly because there are columns we're not running through the model (pickup_datetime and EDTdate)</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Categorify\n",
    "Pandas offers a <a href='https://pandas.pydata.org/pandas-docs/stable/user_guide/categorical.html'><strong>category dtype</strong></a> for converting categorical values to numerical codes. A dataset containing months of the year will be assigned 12 codes, one for each month. These will usually be the integers 0 to 11. Pandas replaces the column values with codes, and retains an index list of category values. In the steps ahead we'll call the categorical values \"names\" and the encodings \"codes\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Convert our three categorical columns to category dtypes.\n",
    "for cat in cat_cols:\n",
    "    df[cat] = df[cat].astype('category')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pickup_datetime              object\n",
       "fare_amount                 float64\n",
       "fare_class                    int64\n",
       "pickup_longitude            float64\n",
       "pickup_latitude             float64\n",
       "dropoff_longitude           float64\n",
       "dropoff_latitude            float64\n",
       "passenger_count               int64\n",
       "dist_km                     float64\n",
       "EDTdate              datetime64[ns]\n",
       "Hour                       category\n",
       "AMorPM                     category\n",
       "Weekday                    category\n",
       "dtype: object"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that <tt>df['Hour']</tt> is a categorical feature by displaying some of the rows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0     4\n",
       "1    11\n",
       "2     7\n",
       "3    17\n",
       "4    22\n",
       "Name: Hour, dtype: category\n",
       "Categories (24, int64): [0, 1, 2, 3, ..., 20, 21, 22, 23]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Hour'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here our categorical names are the integers 0 through 23, for a total of 24 unique categories. These values <em>also</em> correspond to the codes assigned to each name.\n",
    "\n",
    "We can access the category names with <tt>Series.cat.categories</tt> or just the codes with <tt>Series.cat.codes</tt>. This will make more sense if we look at <tt>df['AMorPM']</tt>:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    am\n",
       "1    am\n",
       "2    am\n",
       "3    pm\n",
       "4    pm\n",
       "Name: AMorPM, dtype: category\n",
       "Categories (2, object): [am, pm]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['AMorPM'].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['am', 'pm'], dtype='object')"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['AMorPM'].cat.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0\n",
       "1    0\n",
       "2    0\n",
       "3    1\n",
       "4    1\n",
       "dtype: int8"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['AMorPM'].head().cat.codes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Fri', 'Mon', 'Sat', 'Sun', 'Thu', 'Tue', 'Wed'], dtype='object')"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Weekday'].cat.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    1\n",
       "1    2\n",
       "2    2\n",
       "3    3\n",
       "4    0\n",
       "dtype: int8"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['Weekday'].head().cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\"><strong>NOTE: </strong>NaN values in categorical data are assigned a code of -1. We don't have any in this particular dataset.</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to combine the three categorical columns into one input array using <a href='https://docs.scipy.org/doc/numpy/reference/generated/numpy.stack.html'><tt>numpy.stack</tt></a> We don't want the Series index, just the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 4,  0,  1],\n",
       "       [11,  0,  2],\n",
       "       [ 7,  0,  2],\n",
       "       [17,  1,  3],\n",
       "       [22,  1,  0]], dtype=int8)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hr = df['Hour'].cat.codes.values\n",
    "ampm = df['AMorPM'].cat.codes.values\n",
    "wkdy = df['Weekday'].cat.codes.values\n",
    "\n",
    "cats = np.stack([hr, ampm, wkdy], 1)\n",
    "\n",
    "cats[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\"><strong>NOTE:</strong> This can be done in one line of code using a list comprehension:\n",
    "<pre style='background-color:rgb(217,237,247)'>cats = np.stack([df[col].cat.codes.values for col in cat_cols], 1)</pre>\n",
    "\n",
    "Don't worry about the dtype for now, we can make it int64 when we convert it to a tensor.</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Convert numpy arrays to tensors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 4,  0,  1],\n",
       "        [11,  0,  2],\n",
       "        [ 7,  0,  2],\n",
       "        [17,  1,  3],\n",
       "        [22,  1,  0]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert categorical variables to a tensor\n",
    "cats = torch.tensor(cats, dtype=torch.int64) \n",
    "# this syntax is ok, since the source data is an array, not an existing tensor\n",
    "\n",
    "cats[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can feed all of our continuous variables into the model as a tensor. Note that we're not normalizing the values here; we'll let the model perform this step.\n",
    "<div class=\"alert alert-info\"><strong>NOTE:</strong> We have to store <tt>conts</tt> and <tt>y</tt> as Float (float32) tensors, not Double (float64) in order for batch normalization to work properly.</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 40.7305, -73.9924,  40.7447, -73.9755,   1.0000,   2.1263],\n",
       "        [ 40.7406, -73.9901,  40.7441, -73.9742,   1.0000,   1.3923],\n",
       "        [ 40.7511, -73.9941,  40.7662, -73.9601,   2.0000,   3.3268],\n",
       "        [ 40.7564, -73.9905,  40.7482, -73.9712,   1.0000,   1.8641],\n",
       "        [ 40.7342, -73.9910,  40.7431, -73.9060,   1.0000,   7.2313]])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert continuous variables to a tensor\n",
    "conts = np.stack([df[col].values for col in cont_cols], 1)\n",
    "conts = torch.tensor(conts, dtype=torch.float)\n",
    "conts[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'torch.FloatTensor'"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conts.type()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 6.5000],\n",
       "        [ 6.9000],\n",
       "        [10.1000],\n",
       "        [ 8.9000],\n",
       "        [19.7000]])"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Convert labels to a tensor\n",
    "y = torch.tensor(df[y_col].values, dtype=torch.float).reshape(-1,1)\n",
    "\n",
    "y[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([120000, 3])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cats.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([120000, 6])"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conts.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([120000, 1])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set an embedding size\n",
    "The rule of thumb for determining the embedding size is to divide the number of unique entries in each column by 2, but not to exceed 50."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(24, 12), (2, 1), (7, 4)]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This will set embedding sizes for Hours, AMvsPM and Weekdays\n",
    "cat_szs = [len(df[col].cat.categories) for col in cat_cols]\n",
    "emb_szs = [(size, min(50, (size+1)//2)) for size in cat_szs]\n",
    "emb_szs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define a TabularModel\n",
    "This somewhat follows the <a href='https://docs.fast.ai/tabular.models.html'>fast.ai library</a> The goal is to define a model based on the number of continuous columns (given by <tt>conts.shape[1]</tt>) plus the number of categorical columns and their embeddings (given by <tt>len(emb_szs)</tt> and <tt>emb_szs</tt> respectively). The output would either be a regression (a single float value), or a classification (a group of bins and their softmax values). For this exercise our output will be a single regression value. Note that we'll assume our data contains both categorical and continuous data. You can add boolean parameters to your own model class to handle a variety of datasets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-info\"><strong>Let's walk through the steps we're about to take. See below for more detailed illustrations of the steps.</strong><br>\n",
    "\n",
    "1. Extend the base Module class, set up the following parameters:\n",
    "   * <tt>emb_szs: </tt>list of tuples: each categorical variable size is paired with an embedding size\n",
    "   * <tt>n_cont:  </tt>int: number of continuous variables\n",
    "   * <tt>out_sz:  </tt>int: output size\n",
    "   * <tt>layers:  </tt>list of ints: layer sizes\n",
    "   * <tt>p:       </tt>float: dropout probability for each layer (for simplicity we'll use the same value throughout)\n",
    "   \n",
    "<tt><font color=black>class TabularModel(nn.Module):<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;def \\_\\_init\\_\\_(self, emb_szs, n_cont, out_sz, layers, p=0.5):<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;super().\\_\\_init\\_\\_()</font></tt><br>\n",
    "\n",
    "2. Set up the embedded layers with <a href='https://pytorch.org/docs/stable/nn.html#modulelist'><tt><strong>torch.nn.ModuleList()</strong></tt></a> and <a href='https://pytorch.org/docs/stable/nn.html#embedding'><tt><strong>torch.nn.Embedding()</strong></tt></a><br>Categorical data will be filtered through these Embeddings in the forward section.<br>\n",
    "<tt><font color=black>&nbsp;&nbsp;&nbsp;&nbsp;self.embeds = nn.ModuleList([nn.Embedding(ni, nf) for ni,nf in emb_szs])</font></tt><br><br>\n",
    "3. Set up a dropout function for the embeddings with <a href='https://pytorch.org/docs/stable/nn.html#dropout'><tt><strong>torch.nn.Dropout()</strong></tt></a> The default p-value=0.5<br>\n",
    "<tt><font color=black>&nbsp;&nbsp;&nbsp;&nbsp;self.emb_drop = nn.Dropout(emb_drop)</font></tt><br><br>\n",
    "4. Set up a normalization function for the continuous variables with <a href='https://pytorch.org/docs/stable/nn.html#batchnorm1d'><tt><strong>torch.nn.BatchNorm1d()</strong></tt></a><br>\n",
    "<tt><font color=black>&nbsp;&nbsp;&nbsp;&nbsp;self.bn_cont = nn.BatchNorm1d(n_cont)</font></tt><br><br>\n",
    "5. Set up a sequence of neural network layers where each level includes a Linear function, an activation function (we'll use <a href='https://pytorch.org/docs/stable/nn.html#relu'><strong>ReLU</strong></a>), a normalization step, and a dropout layer. We'll combine the list of layers with <a href='https://pytorch.org/docs/stable/nn.html#sequential'><tt><strong>torch.nn.Sequential()</strong></tt></a><br>\n",
    "<tt><font color=black>&nbsp;&nbsp;&nbsp;&nbsp;self.bn_cont = nn.BatchNorm1d(n_cont)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;layerlist = []<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;n_emb = sum((nf for ni,nf in emb_szs))<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;n_in = n_emb + n_cont<br>\n",
    "<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;for i in layers:<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;layerlist.append(nn.Linear(n_in,i)) <br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;layerlist.append(nn.ReLU(inplace=True))<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;layerlist.append(nn.BatchNorm1d(i))<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;layerlist.append(nn.Dropout(p))<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;n_in = i<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;layerlist.append(nn.Linear(layers[-1],out_sz))<br>\n",
    "<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;self.layers = nn.Sequential(*layerlist)</font></tt><br><br>\n",
    "6. Define the forward method. Preprocess the embeddings and normalize the continuous variables before passing them through the layers.<br>Use <a href='https://pytorch.org/docs/stable/torch.html#torch.cat'><tt><strong>torch.cat()</strong></tt></a> to combine multiple tensors into one.<br>\n",
    "<tt><font color=black>&nbsp;&nbsp;&nbsp;&nbsp;def forward(self, x_cat, x_cont):<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;embeddings = []<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;for i,e in enumerate(self.embeds):<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;embeddings.append(e(x_cat[:,i]))<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;x = torch.cat(embeddings, 1)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;x = self.emb_drop(x)<br>\n",
    "<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;x_cont = self.bn_cont(x_cont)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;x = torch.cat([x, x_cont], 1)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;x = self.layers(x)<br>\n",
    "&nbsp;&nbsp;&nbsp;&nbsp;return x</font></tt>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-danger\"><strong>Breaking down the embeddings steps</strong> (this code is for illustration purposes only.)</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 4,  0,  1],\n",
       "        [11,  0,  2],\n",
       "        [ 7,  0,  2],\n",
       "        [17,  1,  3]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This is our source data\n",
    "catz = cats[:4]\n",
    "catz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(24, 12), (2, 1), (7, 4)]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This is passed in when the model is instantiated\n",
    "emb_szs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "ModuleList(\n",
       "  (0): Embedding(24, 12)\n",
       "  (1): Embedding(2, 1)\n",
       "  (2): Embedding(7, 4)\n",
       ")"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This is assigned inside the __init__() method\n",
    "selfembeds = nn.ModuleList([nn.Embedding(ni, nf) for ni,nf in emb_szs])\n",
    "selfembeds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, Embedding(24, 12)), (1, Embedding(2, 1)), (2, Embedding(7, 4))]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(enumerate(selfembeds))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[tensor([[ 0.7031,  0.5132, -1.0168,  1.3594,  0.7909, -1.2648,  0.9565,  0.6827,\n",
       "           0.4922,  0.1282, -0.9863,  0.2622],\n",
       "         [ 0.0402, -0.8209,  2.5186,  0.5037,  0.0925,  0.0058,  1.0398,  1.7655,\n",
       "           0.0216, -0.1349, -1.0968,  1.4534],\n",
       "         [ 0.5162, -0.6815,  1.4828,  0.1271,  1.1672,  0.8569,  1.1472,  0.7672,\n",
       "           0.4720,  1.3629,  1.2446,  0.3470],\n",
       "         [ 0.1047,  0.2153, -1.4652, -1.6907, -0.0670,  0.9863,  0.7836, -1.3762,\n",
       "           1.6670,  0.1276,  1.2241,  0.1415]], grad_fn=<EmbeddingBackward>),\n",
       " tensor([[-1.1156],\n",
       "         [-1.1156],\n",
       "         [-1.1156],\n",
       "         [-0.7577]], grad_fn=<EmbeddingBackward>),\n",
       " tensor([[-0.8579, -0.4516,  0.3814,  0.9935],\n",
       "         [ 0.6264, -1.1347,  1.9039,  1.1867],\n",
       "         [ 0.6264, -1.1347,  1.9039,  1.1867],\n",
       "         [ 0.1609, -0.1231,  0.5787, -0.3180]], grad_fn=<EmbeddingBackward>)]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This happens inside the forward() method\n",
    "embeddingz = []\n",
    "for i,e in enumerate(selfembeds):\n",
    "    embeddingz.append(e(catz[:,i]))\n",
    "embeddingz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.7031,  0.5132, -1.0168,  1.3594,  0.7909, -1.2648,  0.9565,  0.6827,\n",
       "          0.4922,  0.1282, -0.9863,  0.2622, -1.1156, -0.8579, -0.4516,  0.3814,\n",
       "          0.9935],\n",
       "        [ 0.0402, -0.8209,  2.5186,  0.5037,  0.0925,  0.0058,  1.0398,  1.7655,\n",
       "          0.0216, -0.1349, -1.0968,  1.4534, -1.1156,  0.6264, -1.1347,  1.9039,\n",
       "          1.1867],\n",
       "        [ 0.5162, -0.6815,  1.4828,  0.1271,  1.1672,  0.8569,  1.1472,  0.7672,\n",
       "          0.4720,  1.3629,  1.2446,  0.3470, -1.1156,  0.6264, -1.1347,  1.9039,\n",
       "          1.1867],\n",
       "        [ 0.1047,  0.2153, -1.4652, -1.6907, -0.0670,  0.9863,  0.7836, -1.3762,\n",
       "          1.6670,  0.1276,  1.2241,  0.1415, -0.7577,  0.1609, -0.1231,  0.5787,\n",
       "         -0.3180]], grad_fn=<CatBackward>)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We concatenate the embedding sections (12,1,4) into one (17)\n",
    "z = torch.cat(embeddingz, 1)\n",
    "z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# This was assigned under the __init__() method\n",
    "selfembdrop = nn.Dropout(.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensor([[ 0.0000,  0.8553, -1.6946,  0.0000,  1.3182, -2.1080,  1.5942,  1.1378,\n",
       "          0.8204,  0.2137, -1.6438,  0.0000, -0.0000, -1.4298, -0.0000,  0.6357,\n",
       "          0.0000],\n",
       "        [ 0.0000, -1.3682,  0.0000,  0.8395,  0.0000,  0.0000,  0.0000,  2.9425,\n",
       "          0.0359, -0.0000, -1.8280,  2.4223, -0.0000,  0.0000, -1.8912,  3.1731,\n",
       "          0.0000],\n",
       "        [ 0.0000, -1.1359,  0.0000,  0.0000,  0.0000,  0.0000,  0.0000,  1.2786,\n",
       "          0.7867,  2.2715,  0.0000,  0.5783, -0.0000,  1.0440, -1.8912,  3.1731,\n",
       "          0.0000],\n",
       "        [ 0.0000,  0.3589, -2.4420, -0.0000, -0.1117,  1.6438,  1.3059, -0.0000,\n",
       "          2.7783,  0.2127,  2.0402,  0.0000, -0.0000,  0.0000, -0.0000,  0.0000,\n",
       "         -0.5300]], grad_fn=<MulBackward0>)"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = selfembdrop(z)\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-danger\"><strong>This is how the categorical embeddings are passed into the layers.</strong></div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "class TabularModel(nn.Module):\n",
    "\n",
    "    def __init__(self, emb_szs, n_cont, out_sz, layers, p=0.5):\n",
    "        super().__init__()\n",
    "        self.embeds = nn.ModuleList([nn.Embedding(ni, nf) for ni,nf in emb_szs])\n",
    "        self.emb_drop = nn.Dropout(p)\n",
    "        self.bn_cont = nn.BatchNorm1d(n_cont)\n",
    "        \n",
    "        layerlist = []\n",
    "        n_emb = sum((nf for ni,nf in emb_szs))\n",
    "        n_in = n_emb + n_cont\n",
    "        \n",
    "        for i in layers:\n",
    "            layerlist.append(nn.Linear(n_in,i)) \n",
    "            layerlist.append(nn.ReLU(inplace=True))\n",
    "            layerlist.append(nn.BatchNorm1d(i))\n",
    "            layerlist.append(nn.Dropout(p))\n",
    "            n_in = i\n",
    "        layerlist.append(nn.Linear(layers[-1],out_sz))\n",
    "            \n",
    "        self.layers = nn.Sequential(*layerlist)\n",
    "    \n",
    "    def forward(self, x_cat, x_cont):\n",
    "        embeddings = []\n",
    "        for i,e in enumerate(self.embeds):\n",
    "            embeddings.append(e(x_cat[:,i]))\n",
    "        x = torch.cat(embeddings, 1)\n",
    "        x = self.emb_drop(x)\n",
    "        \n",
    "        x_cont = self.bn_cont(x_cont)\n",
    "        x = torch.cat([x, x_cont], 1)\n",
    "        x = self.layers(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "torch.manual_seed(33)\n",
    "model = TabularModel(emb_szs, conts.shape[1], 1, [200,100], p=0.4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TabularModel(\n",
       "  (embeds): ModuleList(\n",
       "    (0): Embedding(24, 12)\n",
       "    (1): Embedding(2, 1)\n",
       "    (2): Embedding(7, 4)\n",
       "  )\n",
       "  (emb_drop): Dropout(p=0.4)\n",
       "  (bn_cont): BatchNorm1d(6, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (layers): Sequential(\n",
       "    (0): Linear(in_features=23, out_features=200, bias=True)\n",
       "    (1): ReLU(inplace)\n",
       "    (2): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (3): Dropout(p=0.4)\n",
       "    (4): Linear(in_features=200, out_features=100, bias=True)\n",
       "    (5): ReLU(inplace)\n",
       "    (6): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (7): Dropout(p=0.4)\n",
       "    (8): Linear(in_features=100, out_features=1, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Define loss function & optimizer\n",
    "PyTorch does not offer a built-in <a href='https://en.wikipedia.org/wiki/Root-mean-square_deviation'>RMSE Loss</a> function, and it would be nice to see this in place of MSE.<br>\n",
    "For this reason, we'll simply apply the <tt>torch.sqrt()</tt> function to the output of MSELoss during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "criterion = nn.MSELoss()  # we'll convert this to RMSE later\n",
    "optimizer = torch.optim.Adam(model.parameters(), lr=0.001)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Perform train/test splits\n",
    "At this point our batch size is the entire dataset of 120,000 records. This will take a long time to train, so you might consider reducing this. We'll use 60,000. Recall that our tensors are already randomly shuffled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "batch_size = 60000\n",
    "test_size = int(batch_size * .2)\n",
    "\n",
    "cat_train = cats[:batch_size-test_size]\n",
    "cat_test = cats[batch_size-test_size:batch_size]\n",
    "con_train = conts[:batch_size-test_size]\n",
    "con_test = conts[batch_size-test_size:batch_size]\n",
    "y_train = y[:batch_size-test_size]\n",
    "y_test = y[batch_size-test_size:batch_size]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "48000"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(cat_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12000"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(cat_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train the model\n",
    "Expect this to take 30 minutes or more! We've added code to tell us the duration at the end."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:   1  loss: 12.49953079\n",
      "epoch:  26  loss: 11.52666759\n",
      "epoch:  51  loss: 10.47162533\n",
      "epoch:  76  loss: 9.53090382\n",
      "epoch: 101  loss: 8.72838020\n",
      "epoch: 126  loss: 7.81538534\n",
      "epoch: 151  loss: 6.70782852\n",
      "epoch: 176  loss: 5.48520994\n",
      "epoch: 201  loss: 4.37029028\n",
      "epoch: 226  loss: 3.70538783\n",
      "epoch: 251  loss: 3.51656818\n",
      "epoch: 276  loss: 3.44707990\n",
      "epoch: 300  loss: 3.42467499\n",
      "\n",
      "Duration: 730 seconds\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "start_time = time.time()\n",
    "\n",
    "epochs = 300\n",
    "losses = []\n",
    "\n",
    "for i in range(epochs):\n",
    "    i+=1\n",
    "    y_pred = model(cat_train, con_train)\n",
    "    loss = torch.sqrt(criterion(y_pred, y_train)) # RMSE\n",
    "    losses.append(loss)\n",
    "    \n",
    "    # a neat trick to save screen space:\n",
    "    if i%25 == 1:\n",
    "        print(f'epoch: {i:3}  loss: {loss.item():10.8f}')\n",
    "\n",
    "    optimizer.zero_grad()\n",
    "    loss.backward()\n",
    "    optimizer.step()\n",
    "\n",
    "print(f'epoch: {i:3}  loss: {loss.item():10.8f}') # print the last line\n",
    "print(f'\\nDuration: {time.time() - start_time:.0f} seconds') # print the time elapsed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Plot the loss function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzt3Xd8VfUd//HX595sSAibsEFEQGSG7WqtWwtURRAREEQFtdpqrb9qtz9bbcUFCg4ExYmKWitqqYqijLD3EGRDwg4r8/v7Ixd/SEkIIfeeO97Px+M+cu/JSe77cJK8Oet7zDmHiIjELp/XAURExFsqAhGRGKciEBGJcSoCEZEYpyIQEYlxKgIRkRinIhARiXEqAhGRGKciEBGJcXFeByiPWrVquaZNm3odQ0QkosybN2+nc672yeaLiCJo2rQpWVlZXscQEYkoZrahPPNp15CISIxTEYiIxDgVgYhIjFMRiIjEOBWBiEiMUxGIiMQ4FYGISIyL6iKYvW4XL3y1juJi3Y5TRKQ0UV0E/1q8jb9+tIL+z89i465DXscREQlLUV0Ef+59No9d244VW/dz2ZMzeGXWBpzT1oGIyLGiugjMjOsyG/HJPefTuUl1Hpq6lEEvzmHL3sNeRxMRCRtRXQRH1U9PZtLNXXm4b1vmb9zDZaNn8HbWJm0diIgQI0UAJVsHA7s1Ydovz6dN/TTum7KY30xZzJGCIq+jiYh4KmaK4KjGNVN47Zbu3PXTFrw9bzPXPvcNm/foQLKIxK6YKwIAv8/41SVn8cJNmWzYeYirn/6ar9bkeB1LRMQTQSsCM3vJzLLNbOkx0x4zs5VmttjM3jOz9GC9f3n8rE1dPrjzXGqnJjL4pTmM/WKtjhuISMwJ5hbBy8Blx037DGjrnGsHrAYeCOL7l0uzWlV4b2Qvrjgng0enreL2V+eTe6TA61giIiETtCJwzs0Adh837VPnXGHg5SygYbDe/1RUSYzj6QEdefDK1ny2Ygd9xsxkbfYBr2OJiISEl8cIbgY+9vD9f8TMGH5ec14d1o29hwro/czXzFy70+tYIiJB50kRmNnvgEJgchnzjDCzLDPLyskJ3YHcHmfU5MM7z6Vh9RSGvjyXT5dtD9l7i4h4IeRFYGaDgauAga6MI7POufHOuUznXGbt2rVDF5CSC9DeGNGd1hlp3D55PlMXbAnp+4uIhFJIi8DMLgPuB37unAvrk/erV0lg8vBudG1ag3veWsgrszZ4HUlEJCiCefro68C3wFlmttnMhgHPAKnAZ2a20MyeC9b7V4aqiXFMGNqFi1rV4aGpSxn7xVqvI4mIVLq4YH1j59yAE0x+MVjvFyxJ8X6evbEz9769iEenrQJg5IUtPE4lIlJ5glYE0STe72N0vw4APDptFQeOFPLrS87C7zOPk4mInD4VQTn5fMY/rmtPSoKfsV98x7Z9R/jHde1VBiIS8VQEpyDe7+ORX7SjfrVk/vnZaoqd45/XtSfOH5NDNolIlFARVMCdF52Jz2c89skqnIPH+6kMRCRyqQgqaNRPWmBWcszgUH4hTw/oRHKC3+tYIiKnTP+NPQ0jL2zBX3qfzfSV2dzwwix2H8z3OpKIyClTEZymQT2a8uzAzizbup9rn/2GTbvD+jo5EZH/oSKoBJe1rcfk4d3YeSCP/uNnsTY71+tIIiLlpiKoJF2a1uC1W7pzuKCIq5+eycdLtnkdSUSkXFQElahtg2r8+67zaJWRyqjX5vN21iavI4mInJSKoJLVq5bE5OHd6NWiFvdNWczLM9d7HUlEpEwqgiBISYjjhcGZXNKmLn/8cDmPf7pK90IWkbClIgiSxDg/Ywd2ol9mQ57671p++cZCjhQUeR1LROR/6IKyIIrz+/j7Ne1oWqsKj05bxZa9hxk/qDM1qyZ6HU1E5AfaIggyM2PkhS0Yc0Mnlm7ZR5+xM/ku54DXsUREfqAiCJEr22XwxojuHM4v4sYXZrNl72GvI4mIACqCkOrYuDqTbu7GgSOF9H5mJlnf7/Y6koiIiiDU2tRP452RPUlNimPohLks3bLP60giEuNUBB5oWTeVycO7kZoUx/XjvmX6ih1eRxKRGKYi8Ej99GTeGdmT5rWrMnxSFhN04ZmIeERF4KGMasm8dWsPLmlTlz99uJynpq+hqFgXnolIaKkIPJac4OeZGzrRp0N9Hv9sNcMmzuVQfqHXsUQkhqgIwkC838fo6zvw1z5tmbE6h8EvzeFwvq5CFpHQUBGECTPjxu5NeLJ/R7I27GHIhDnsPJDndSwRiQEqgjBzdfv6jO7XgYWb9vKLsd/owjMRCToVQRjq07EBb4zozp5D+Vz77Des3L7f60giEsVUBGGqY+PqvDGiO8XOcd2z3/L1mp1eRxKRKKUiCGNn16/GeyN70aB6MkMmzGHqgi1eRxKRKKQiCHP105N567YeZDatzt1vLuT5Geu8jiQiUUZFEAHSkuKZeHNXrmyXwcP/XsGfP1yuC89EpNLoxjQRIjHOz9P9O1I3NYmXZq5n3c4DPDWgI2lJ8V5HE5EIpy2CCOLzGb+/ug0P923L12t20nfMTNbvPOh1LBGJcCqCCDSwWxNeGdaNXQfz6TNmJjPX6owiEak4FUGE6nFGTT4YdS510xIZ/NIc3pm32etIIhKhVAQRrHHNFKbc3pNuzWvw67cXafRSEamQoBWBmb1kZtlmtvSYaTXM7DMzWxP4WD1Y7x8r0pLimTCkK307NuDxz1YzYPwsdh/M9zqWiESQYG4RvAxcdty03wLTnXNnAtMDr+U0JcT5eLxfe/55XXsWbt7Ldc9pjCIRKb+gFYFzbgZw/N3ZewMTA88nAn2C9f6xxsy4pnNDXrm5K9m5eVwz9huWbdX9kEXk5EJ9jKCuc24bQOBjnRC/f9Tr1rwmb93aAzO49tlveXXWBpzTcQMRKV3YHiw2sxFmlmVmWTk5OV7HiSitM9J4/45eZDatzoNTlzJ8YpbueiYipQp1EewwswyAwMfs0mZ0zo13zmU65zJr164dsoDRok5qEhOHduX3V7Xh81XZ3PzyXLJzj3gdS0TCUKiL4ANgcOD5YOD9EL9/TPH5jJvPbcbj/Towf+NeLn58Bp8s2+51LBEJM8E8ffR14FvgLDPbbGbDgL8BF5vZGuDiwGsJsj4dG/DxL8+jcY0Ubn1lHn/8YBlHCnRPZBEpYZFwIDEzM9NlZWV5HSPi5RUW8bePVzJh5vecVTeVCUO7UD892etYIhIkZjbPOZd5svnC9mCxVL7EOD9/uPpsXh7aha17DzPg+Vls36fjBiKxTkUQgy48qw4Th3Vl14F8rh//LZNnb6CwqNjrWCLiERVBjOrUuDovD+2C34zfvbeU+99ZQoHKQCQmqQhiWGbTGvz33gu552cteWf+Zi4dPYN5G46/GFxEop2KQPjlz87kxcGZFBQX02/cLEZNns+O/Tp2IBIrVAQCwEWt6/LRXecx/NxmfL4qm2ue/YalWzRWkUgsUBHID9KS4nngita8fkt38gqL6TNmJqM/W61jByJRTkUg/6N9o3Q+u+d8rm5fnyenr2HIhDm6x4FIFFMRyAmlpyQw+voO/OO69sxdv4dLRn+p4SlEopSKQMp0beeGvH9HL+qkJnHrK/O4582F7DtU4HUsEalEKgI5qdYZaUwd1Yu7LjqTDxZt5ZInvuTzVaUOHCsiEUZFIOWSEOfjVxe3ZOrIXlRLjmfohLncP2UxuUe0dSAS6VQEckrOaViND+88l9svPIO3523i0tEzmL9xj9exROQ0qAjklCXG+bn/slZMub0ncX4fA5+fzaRvv+dwvoa2FolEJy0CM+tlZlUCz280s8fNrEnwo0m469S4OlNu70Gb+mn8/v1lXPLEl8xet8vrWCJyisqzRfAscMjM2gO/ATYAk4KaSiJGndQkptzWg8nDu+E3Y8Dzs3j2i+8oLg7/+1yISInyFEGhK7l7TW/gSefck0BqcGNJJDEzerWoxb/uOo/L22bw92krufSJGUxdsIVIuPGRSKwrTxHkmtkDwI3AR2bmB+KDG0siUdXEOJ65oSP/vK49ifE+7n5zIXe8voC9h3RVskg4K08RXA/kAcOcc9uBBsBjQU0lEcvMuKZzQ94fdS73XXoWnyzdzmVPfMXXa3Z6HU1ESlGuLQJKdgl9ZWYtgQ7A68GNJZHO7zNG/aQFU0f1okqinxtfnM0D7y4hJzfP62gicpzyFMEMINHMGgDTgaHAy8EMJdGjbYNq/OvO8xh2bjPeztpEnzEz2bDroNexROQY5SkCc84dAn4BPO2c6wucHdxYEk2SE/w8dFUb3hvZi4P5hVz99Nd8sGir17FEJKBcRWBmPYCBwEeBaf7gRZJodU7Dakwd2Ysz6lTlrtcXcNfrCzSAnUgYKE8R3A08ALznnFtmZs2Bz4MbS6JV01pVePvWHvz64pb8e8k2zn30v4z5fK1OMxXxkJX3F9DMUgHnnDsQ3Ej/KzMz02VlZYX6bSXIlm/dz+j/rOaz5Tu4un19/tqnLdWSdWaySGUxs3nOucyTzVeeISbOMbMFwFJguZnNMzMdI5DT1qZ+GuMHdea+S8/i30u20fuZr9m855DXsURiTnl2DY0DfuWca+Kcawz8Gng+uLEkVpiVnGb65oju7D6Yz6WjZ/DIxyt0n2SRECpPEVRxzv1wTMA59wVQJWiJJCZlNq3BuyN7cXGbuoz7ch1DJszRgWSREClPEawzs4fMrGng8SCwPtjBJPa0qFOVJ/p35LFr2zFn/W76jp3JupyQH5ISiTnlKYKbgdrAu4FHLWBIEDNJjLsusxGv3dKdvYcL6P3MTF1zIBJkJy0C59we59xdzrlOgcfdwIMhyCYxrEvTGnxwRy/OrFtyzcF9by/iYF6h17FEolJF71DWr1JTiJxAw+opvHVrD+78aQumzN9M7zEz2b7viNexRKJORYvAKjWFSCni/D5+fclZTB7WjW17D9Nv3Les2ZHrdSyRqFJqEZhZjVIeNVERSIj1bFGLV4d341B+EX3HfsP0FTu8jiQSNcraIpgHZAU+HvvIAk7rTiNmdo+ZLTOzpWb2upklnc73k9jQsXF1PrijF01qpjB8UhbPfvGdhqYQqQSlFoFzrplzrnng4/GP5hV9w8Bw1ncBmc65tpQMYNe/ot9PYkv99GSm3NaTK84puSXmPW8u5EhBkdexRCJanIfvm2xmBUAKoPMDpdySE/w8M6Ajreul8o9PV7N+50HGDcqkXjVtWIpUREUPFleYc24L8A9gI7AN2Oec+zTUOSSymRl3/PRMxg3qzJrsA/z8ma9ZuGmv17FEIlLIi8DMqgO9gWZAfaCKmd14gvlGmFmWmWXl5OSEOqZEiEvPrse7I3uSEOej37hveW/BZq8jiUScss4a+ukxz5sd97lfnMZ7/gxY75zLcc4VUHK1cs/jZ3LOjXfOZTrnMmvXrn0abyfRrlW9ND6441w6NkrnnjcX8cjHKygq1kFkkfIqa4vgH8c8f+e4z53OlcUbge5mlmJmBlwErDiN7ydCjSoJvDq8GwO7NWbcl+u4ZVIWuUc0aJ1IeZRVBFbK8xO9Ljfn3GxgCjAfWBLIML6i30/kqHi/j4f7nsNf+rRlxuoc+o79ho27dH8DkZMpqwhcKc9P9PqUOOf+4Jxr5Zxr65wb5JzLO53vJ3KsQd2b8Mqwbuw8kEffsTOZt2GP15FEwlpZRdDczD4wsw+PeX70dbMyvk7Ecz3OqMm7t/ckNSmOAc/P4qPF27yOJBK2Sr1nsZldUNYXOue+DEqiE9A9i6Widh/MZ8SkLLI27OH+y1px2wXNKTk0JRL9ynvP4lIvKDv+D72ZxQNtgS3OuezTjygSfEcPIt83ZTF/n7aSg3mF3HvpWV7HEgkrZZ0++tzRm9SbWTVgETAJWGBmA0KUT+S0JcX7efL6Dgzo2ohnPl/LA+8uJq9Qw1KIHFXWMYLznHPLAs+HAqudc+cAnYHfBD2ZSCXy+YyH+5zDyAvP4PU5m+g/fha7D57W2IkiUaOsIjj2t+RiYCqAc257UBOJBInPZ/zmslaMHdiJZVv3c+srWRzO15aBSFlFsNfMrjKzjkAvYBqAmcUByaEIJxIMV5yTweP92pO1YQ9XPvUVy7fu9zqSiKfKKoJbgTuACcDdx2wJXAR8FOxgIsF0Vbv6TB7WjYP5hVz73DfMWb/b60ginin19NFwotNHJViy9x+h//Oz2HuogDdGdKdl3VSvI4lUmvKePlrWdQRPlfWFzrm7KpjtlKkIJJjW5Ryg37hvOZhXxBP9O3Dp2fW8jiRSKcpbBGXtGroNOJeSm8ac6JaVIlGhee2qfHTXebSsl8rtr87j1VkbvI4kElJl3aEsA7gOuB4oBN4E3nHOaeAWiTp105J4/ZZujJo8nwenLuVgXiG3XnCG17FEQqKsexbvcs4955z7CTAESAeWmdmgUIUTCaWUhDievymTq9pl8MjHK7VlIDHjpPcsNrNOwABKriX4GO0WkigW5/cx+voOHM4v4qH3l3I4v4jh5zXT+EQS1coaYuJPZjYP+BXwJZDpnBvmnFsesnQiHoj3+xgzsBOXtKnLw/9ewZ//pR95iW5lHSx+CKgGtAceAeab2WIzW2Jmi0OSTsQjSfF+nruxM0N6NmXCzO8Z9+V3RMKp1iIVUdauId1zQGKamfHgla3Jzj3CIx+vZPehfH57WSvtJpKoU9Yw1Cc8UmZmfqA/oCNpEvXi/D6eGdCJh1KWMu7LdSTG+fnVxS29jiVSqUotAjNLA0YBDYAPgM8oGXLiXmAhMDkUAUW85vMZf+ndloKiYp6avobEOB+jftLC61gilaasXUOvAHuAb4HhwH1AAtDbObcwBNlEwobPZzzyi3YUFDke+2QViXE+hp/X3OtYIpWirCJoHrj/AGb2ArATaOycyw1JMpEw4/cZj13bjvzCYv760QoS4nzc1KOp17FETltZRVBw9IlzrsjM1qsEJNbF+X080b8D+UXF/P79ZST4ffTv2tjrWCKnpazTR9ub2f7AIxdod/S5mWkAd4lZ8X4fz9zQkQta1uaB95bw1txNXkcSOS1lDTHhd86lBR6pzrm4Y56nhTKkSLhJjPMzblBnzm1Ri9+8s5jxM77zOpJIhZW1RSAiZUiK9/Pi4C5c2S6D//vvlbwxZ6PXkUQqREUgchoS4nw83q89vVrU5LfvLuHBqUt0BbJEHBWByGlKjPMzcWhXRpzfnFdnbeRv01Z6HUnklJx09FERObk4v48HLm/FkYIixn25jvTkBG6/UPczkMigIhCpJGbGH68+m32HC/j7tJVUS47nhm46tVTCn4pApBL5fMY/rmtP7pFCfjd1CalJcVzdvr7XsUTKpGMEIpUs3u9j7MBOdGlag3veXMjnq7K9jiRSJhWBSBAkxft5YXAmrTJSuXXSPKav2OF1JJFSqQhEgiQtKZ5Xh3WjVUYqo16bz/KtuiBfwpOKQCSI0lMSeGFwJunJCdz00hwWbdrrdSSR/6EiEAmyOqlJvDKsK0nxPm58YTZLt+zzOpLIj3hSBGaWbmZTzGylma0wsx5e5BAJlTPrpvL2bT1IS45nyIS57Nh/xOtIIj/waovgSWCac64V0B5Y4VEOkZDJqJbMhKFdOJRfyLCJc9l5IM/rSCKAB0UQuAXm+cCLAM65fOecdpxKTGhZN5UxN3RibfYB+j33LbtUBhIGvNgiaA7kABPMbIGZvWBmVY6fycxGmFmWmWXl5OSEPqVIkPykVR1eHdaNLXsPc/PELA7lF3odSWKcF0UQB3QCnnXOdQQOAr89fibn3HjnXKZzLrN27dqhzigSVJlNa/D0gI4s2byXO19bQGFRsdeRJIZ5UQSbgc3OudmB11MoKQaRmHLJ2fX4S5+2TF+Zzb1vL+JAnrYMxBshLwLn3HZgk5mdFZh0EbA81DlEwsHAbk349cUtmbpwK1c+9RW7D+Z7HUlikFdnDd0JTDazxUAH4P96lEPEc3dedCav3dKNbfuOMGryfAq0m0hCzJMicM4tDOz/b+ec6+Oc2+NFDpFw0fOMWjzS9xy+XbeLv/5rue5yJiGlYahFwsQ1nRuycvt+nv9qPekpCdxzcUuvI0mMUBGIhJEHLm/N3kMFPDl9DQlxPkb9pIXXkSQGqAhEwojPZ/ztmnYUFBXz2CerSPD7uOX85l7HkiinIhAJM/7AXc4KihwP/3sF8X5jSK9mXseSKKYiEAlDcX4fT/TvQEFRMX/8cDkJcX7d/1iCRsNQi4SpeL+Pp2/oyE9b1eH/vLeEt7I2eR1JopSKQCSMJcb5GTuwE+edWYv731nMu/M3ex1JopCKQCTMJcX7GT8okx7Na3Lv24t4f+EWryNJlFERiESA5AQ/Lw7uQtdmNbjnzYV8sGir15EkiqgIRCJEcoKfl4Z0IbNpSRl8tHib15EkSqgIRCJISkIcE4Z0oVPjdO56YwEfL1EZyOlTEYhEmCqJcUwY2pUOjdK58/UFzNuw2+tIEuFUBCIRqGpiHBOGdiEjPYm7Xl/Ixl2HvI4kEUxFIBKh0pLieXpAJ/YfKeDyJ2eQ9b22DKRiVAQiEaxDo3Sm3X0+ddOSuPnluazcvt/rSBKBVAQiEa5BejKThnUlJSGOQS/OYW12rteRJMKoCESiQMPqKUwa1hXn4Bdjv2Hpln1eR5IIoiIQiRIt66by3siepCbFM2TCHB1AlnJTEYhEkUY1Uph4c1cKix03vTSb7fuOeB1JIoCKQCTKtKhTlRcHZ5KTm8fPn/maBRt1S3Apm4pAJAp1blKDd0b2JDHex/XjZzFh5nqKi53XsSRMqQhEolSremm8P+pcep1Rkz99uJy/T1vpdSQJUyoCkShWo0oCLw3pwqDuTRg3Yx1jPl+Lc9oykB/TrSpFopyZ8Yer27D/SAGPfbKK77IP8Ne+bUlJ0K+/lNBPgkgMiPP7GN2vA81qVeHJ6WvYsvcwE4Z2URkIoF1DIjHD5zPu/llLnri+A3O/383NL8/lUH6h17EkDKgIRGJM7w4NGH19B+as382wl7M4nF/kdSTxmIpAJAYdLYPZ63dpy0BUBCKxqneHBjzer6QM+o75huVbNXJprFIRiMSwPh0bMGFoV3YdzOOqp7/iD+8vJfdIgdexJMRUBCIx7oKWtfnPry7gxu5NeGXWBkZMmkdhUbHXsSSEVAQiQnpKAn/u3ZZHr23Pt+t2cdcbC9h3WFsGsUJFICI/uLZzQx64vBWfLNvBFU9+xTdrd3odSUJARSAiP3LrBWfwzu09ifcbN7wwm0c1RlHU86wIzMxvZgvM7F9eZRCREzt6L+TrMxsx9ovvGDJhDjNW53gdS4LEyy2CXwIrPHx/ESlDUryfh/u25aYeTVi+dT+3vTqP73IOeB1LgsCTIjCzhsCVwAtevL+IlE+c38efe7fl/Tt6kRjn44bnZ/Hhoq2s2ZFLgc4sihpebRE8AfwG0E+SSATIqJbMa7d0Jynez52vL+Di0TPo/cxMnVkUJUJeBGZ2FZDtnJt3kvlGmFmWmWXl5GjfpIjXWmek8Z9fXcA7t/fkz73PZk12LgNfmMWny7brquQIZ6G+SYWZPQIMAgqBJCANeNc5d2NpX5OZmemysrJClFBEyuPzldnc8dp8DgYGrbvynAz+2qct1askeJxMjjKzec65zJPO5+XdiszsQuBe59xVZc2nIhAJT9v2HWbznsPMXLuTMZ+vJcHvo2+nBoy8sAX105O9jhfzylsEuiuFiFRYRrVkMqol06VpDS5rW48Xv1rPm3M38eGibYy5oRO9WtTEzLyOKSfh6RZBeWmLQCRyfL/zIENfnsv6nQdpXrsKfTs04BedG9JAWwghFxG7hspLRSASWQ7kFfLhoq28t2ALc9bvJinex32XtqJpzRQAftqqjrYUQkBFICJhYdPuQ9z/zmK++W7XD9POaVCNpHgfXZvV4LYLziA1Kd7DhNFLRSAiYcM5x8rtuew5lM+iTfuYtmw7cT5j/sY9VE2Io3X9NG49vzk1qiTQoVG6thYqiYpARMLego17mDJvM9NXZLN9/xEAmteuAkCtKonccn5zqiXHk5Lg56Ml2+jQKJ2fta6L36eiKA8VgYhEjNwjBczfuJeNuw7y6fIdpCXFs3DTXrbsPfw/855ZpyrnNKjGdzkHOKteKsnxfupVS8bhaF0vjZpVE9ixP4/2DatRJy3ph6/Lyc2jZpUEfIESWb51PxnVkqL6ugcVgYhEtEP5hcxat4uiYtiw6yA/b1+f2et389LM9WzafZhmtVJYk32AomJH7pHC//l6n8FZ9dJoWbcqS7fs47ucg3RrVoNrOjXky9U5fLRkG7WqJtK3Y31SEuLYd7iAs+unsWDTXqok+KlRJZG05DjqpCZRVOyI9xs7D+TRqEYKiXE+0lMSOKN21R/eL6+wiA27DuEzqFctmaqJ3p+dryIQkZhxIK8Q5xzLt+5n3+EC0lMSmLl2Jws37WXV9lxa1KnK2Q3SmPD19+QXFZOWFEe/zEbM/X43q3bkcqSgmIQ4H/mFxfgMHFCeP40Z1ZLwmVE7NZHNew6z80AeUFJCjWukUFDkSIr3cd6ZtamblsTG3QdZtT2Xjo2rM6BrI75YlcO0pdvp1aIWDripRxN2Hchn2dZ9dG5SnZlrd9G7Q32qVLBUVAQiIsfJyc3jUH4hDaun/Og4Q1Gxw4APF2+lfnoytaom4jPYuPsQBUXFxPl8FDlHi9pV2bj7EPlFxazansvqHbngIDs3jyqJfi5vm4EZrM0+wLqdB0mK85Ode4Q563eTV1hMlQQ/rTLSWLJ5H/mB0VsbpCf/sAsszmcUFv/4b/KYGzpxZbuMCi2vikBEJEw45zhcUERyvB8zY+OuQ3y+Kps29dPIbFKdvYcK2L7/CK/N3kib+mm0rJvK3O9306N5Tdo1rFbhs6hUBCIiMa68RaB7FouIxDgVgYhIjFMRiIjEOBWBiEiMUxGIiMQ4FYGISIxTEYiIxDgVgYhIjIuIC8rMLAfYUMEvrwXsrMQ4XtKyhCctS3jSskAT51ztk80UEUVwOswsqzxX1kUCLUt40rKEJy1L+WnXkIhIjFMRiIjEuFgogvFeB6hEWpbwpGUKyWdLAAAF40lEQVQJT1qWcor6YwQiIlK2WNgiEBGRMkR1EZjZZWa2yszWmtlvvc5zqszsezNbYmYLzSwrMK2GmX1mZmsCH6t7nfNEzOwlM8s2s6XHTDthdivxVGA9LTazTt4l/7FSluOPZrYlsF4WmtkVx3zugcByrDKzS71JfWJm1sjMPjezFWa2zMx+GZgeieultGWJuHVjZklmNsfMFgWW5U+B6c3MbHZgvbxpZgmB6YmB12sDn2962iGcc1H5APzAd0BzIAFYBLTxOtcpLsP3QK3jpj0K/Dbw/LfA373OWUr284FOwNKTZQeuAD4GDOgOzPY6/0mW44/AvSeYt03g5ywRaBb4+fN7vQzH5MsAOgWepwKrA5kjcb2UtiwRt24C/75VA8/jgdmBf++3gP6B6c8BtweejwSeCzzvD7x5uhmieYugK7DWObfOOZcPvAH09jhTZegNTAw8nwj08TBLqZxzM4Ddx00uLXtvYJIrMQtIN7OK3aS1kpWyHKXpDbzhnMtzzq0H1lLycxgWnHPbnHPzA89zgRVAAyJzvZS2LKUJ23UT+Pc9EHgZH3g44KfAlMD049fL0fU1BbjIKnovy4BoLoIGwKZjXm+m7B+UcOSAT81snpmNCEyr65zbBiW/DEAdz9KdutKyR+K6uiOwu+SlY3bPRcxyBHYndKTkf58RvV6OWxaIwHVjZn4zWwhkA59RssWy1zlXGJjl2Lw/LEvg8/uAmqfz/tFcBCdqyEg7RaqXc64TcDkwyszO9zpQkETaunoWOAPoAGwD/hmYHhHLYWZVgXeAu51z+8ua9QTTwmp5TrAsEblunHNFzrkOQENKtlRan2i2wMdKX5ZoLoLNQKNjXjcEtnqUpUKcc1sDH7OB9yj5AdlxdPM88DHbu4SnrLTsEbWunHM7Ar+4xcDz/P9dDGG/HGYWT8kfzsnOuXcDkyNyvZxoWSJ53QA45/YCX1ByjCDdzOICnzo27w/LEvh8Ncq/+/KEorkI5gJnBo68J1ByUOUDjzOVm5lVMbPUo8+BS4CllCzD4MBsg4H3vUlYIaVl/wC4KXCWSndg39FdFeHouP3kfSlZL1CyHP0DZ3U0A84E5oQ6X2kC+5FfBFY45x4/5lMRt15KW5ZIXDdmVtvM0gPPk4GfUXLM43Pg2sBsx6+Xo+vrWuC/LnDkuMK8PmIezAclZz2spmR/2++8znOK2ZtTcpbDImDZ0fyU7AucDqwJfKzhddZS8r9OyaZ5ASX/gxlWWnZKNnXHBNbTEiDT6/wnWY5XAjkXB34pM46Z/3eB5VgFXO51/uOW5VxKdiEsBhYGHldE6HopbVkibt0A7YAFgcxLgd8HpjenpKzWAm8DiYHpSYHXawOfb366GXRlsYhIjIvmXUMiIlIOKgIRkRinIhARiXEqAhGRGKciEBGJcSoCkSAzswvN7F9e5xApjYpARCTGqQhEAszsxsC48AvNbFxgILADZvZPM5tvZtPNrHZg3g5mNiswuNl7x4zh38LM/hMYW36+mZ0R+PZVzWyKma00s8mnO1qkSGVSEYgAZtYauJ6Sgf46AEXAQKAKMN+VDP73JfCHwJdMAu53zrWj5ErWo9MnA2Occ+2BnpRclQwlo2PeTcm4+M2BXkFfKJFyijv5LCIx4SKgMzA38J/1ZEoGXysG3gzM8yrwrplVA9Kdc18Gpk8E3g6MDdXAOfcegHPuCEDg+81xzm0OvF4INAW+Dv5iiZycikCkhAETnXMP/Gii2UPHzVfWmCxl7e7JO+Z5EfrdkzCiXUMiJaYD15pZHfjhPr5NKPkdOToC5A3A1865fcAeMzsvMH0Q8KUrGQ9/s5n1CXyPRDNLCelSiFSA/lciAjjnlpvZg5TcEc5HyWijo4CDwNlmNo+SO0FdH/iSwcBzgT/064ChgemDgHFm9ufA97guhIshUiEafVSkDGZ2wDlX1escIsGkXUMiIjFOWwQiIjFOWwQiIjFORSAiEuNUBCIiMU5FICIS41QEIiIxTkUgIhLj/h+Q0PMT1e35WwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(epochs), losses)\n",
    "plt.ylabel('RMSE Loss')\n",
    "plt.xlabel('epoch');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validate the model\n",
    "Here we want to run the entire test set through the model, and compare it to the known labels.<br>\n",
    "For this step we don't want to update weights and biases, so we set <tt>torch.no_grad()</tt>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RMSE: 3.34590030\n"
     ]
    }
   ],
   "source": [
    "# TO EVALUATE THE ENTIRE TEST SET\n",
    "with torch.no_grad():\n",
    "    y_val = model(cat_test, con_test)\n",
    "    loss = torch.sqrt(criterion(y_val, y_test))\n",
    "print(f'RMSE: {loss:.8f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This means that on average, predicted values are within &plusmn;$3.31 of the actual value.\n",
    "\n",
    "Now let's look at the first 50 predicted values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   PREDICTED   ACTUAL     DIFF\n",
      " 1.   2.5379   2.9000   0.3621\n",
      " 2.  25.1634   5.7000  19.4634\n",
      " 3.   6.3749   7.7000   1.3251\n",
      " 4.  13.4677  12.5000   0.9677\n",
      " 5.   4.4992   4.1000   0.3992\n",
      " 6.   4.8968   5.3000   0.4032\n",
      " 7.   3.1796   3.7000   0.5204\n",
      " 8.  17.7814  14.5000   3.2814\n",
      " 9.   6.1348   5.7000   0.4348\n",
      "10.  12.0325  10.1000   1.9325\n",
      "11.   6.1323   4.5000   1.6323\n",
      "12.   6.9208   6.1000   0.8208\n",
      "13.   5.9448   6.9000   0.9552\n",
      "14.  13.4625  14.1000   0.6375\n",
      "15.   5.9277   4.5000   1.4277\n",
      "16.  27.5778  34.1000   6.5222\n",
      "17.   3.2774  12.5000   9.2226\n",
      "18.   5.7506   4.1000   1.6506\n",
      "19.   8.1940   8.5000   0.3060\n",
      "20.   6.2858   5.3000   0.9858\n",
      "21.  13.6693  11.3000   2.3693\n",
      "22.   9.6759  10.5000   0.8241\n",
      "23.  16.0568  15.3000   0.7568\n",
      "24.  19.3310  14.9000   4.4310\n",
      "25.  48.6873  49.5700   0.8827\n",
      "26.   6.3257   5.3000   1.0257\n",
      "27.   6.0392   3.7000   2.3392\n",
      "28.   7.1921   6.5000   0.6921\n",
      "29.  14.9567  14.1000   0.8567\n",
      "30.   6.7476   4.9000   1.8476\n",
      "31.   4.3447   3.7000   0.6447\n",
      "32.  35.6969  38.6700   2.9731\n",
      "33.  13.9892  12.5000   1.4892\n",
      "34.  12.8934  16.5000   3.6066\n",
      "35.   6.3164   5.7000   0.6164\n",
      "36.   5.9684   8.9000   2.9316\n",
      "37.  16.1289  22.1000   5.9711\n",
      "38.   7.6541  12.1000   4.4459\n",
      "39.   8.6153  10.1000   1.4847\n",
      "40.   4.0447   3.3000   0.7447\n",
      "41.  10.2168   8.5000   1.7168\n",
      "42.   8.8325   8.1000   0.7325\n",
      "43.  15.2500  14.5000   0.7500\n",
      "44.   6.3571   4.9000   1.4571\n",
      "45.   9.7002   8.5000   1.2002\n",
      "46.  12.1134  12.1000   0.0134\n",
      "47.  24.3001  23.7000   0.6001\n",
      "48.   2.8357   3.7000   0.8643\n",
      "49.   6.8266   9.3000   2.4734\n",
      "50.   8.2082   8.1000   0.1082\n"
     ]
    }
   ],
   "source": [
    "print(f'{\"PREDICTED\":>12} {\"ACTUAL\":>8} {\"DIFF\":>8}')\n",
    "for i in range(50):\n",
    "    diff = np.abs(y_val[i].item()-y_test[i].item())\n",
    "    print(f'{i+1:2}. {y_val[i].item():8.4f} {y_test[i].item():8.4f} {diff:8.4f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So while many predictions were off by a few cents, some were off by \\\\$19.00. Feel free to change the batch size, test size, and number of epochs to obtain a better model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save the model\n",
    "We can save a trained model to a file in case we want to come back later and feed new data through it. The best practice is to save the state of the model (weights & biases) and not the full definition. Also, we want to ensure that only a trained model is saved, to prevent overwriting a previously saved model with an untrained one.<br>For more information visit <a href='https://pytorch.org/tutorials/beginner/saving_loading_models.html'>https://pytorch.org/tutorials/beginner/saving_loading_models.html</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Make sure to save the model only after the training has happened!\n",
    "if len(losses) == epochs:\n",
    "    torch.save(model.state_dict(), 'TaxiFareRegrModel.pt')\n",
    "else:\n",
    "    print('Model has not been trained. Consider loading a trained model instead.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading a saved model (starting from scratch)\n",
    "We can load the trained weights and biases from a saved model. If we've just opened the notebook, we'll have to run standard imports and function definitions. To demonstrate, restart the kernel before proceeding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "def haversine_distance(df, lat1, long1, lat2, long2):\n",
    "    r = 6371\n",
    "    phi1 = np.radians(df[lat1])\n",
    "    phi2 = np.radians(df[lat2])\n",
    "    delta_phi = np.radians(df[lat2]-df[lat1])\n",
    "    delta_lambda = np.radians(df[long2]-df[long1])\n",
    "    a = np.sin(delta_phi/2)**2 + np.cos(phi1) * np.cos(phi2) * np.sin(delta_lambda/2)**2\n",
    "    c = 2 * np.arctan2(np.sqrt(a), np.sqrt(1-a))\n",
    "    return r * c\n",
    "\n",
    "class TabularModel(nn.Module):\n",
    "    def __init__(self, emb_szs, n_cont, out_sz, layers, p=0.5):\n",
    "        super().__init__()\n",
    "        self.embeds = nn.ModuleList([nn.Embedding(ni, nf) for ni,nf in emb_szs])\n",
    "        self.emb_drop = nn.Dropout(p)\n",
    "        self.bn_cont = nn.BatchNorm1d(n_cont)\n",
    "        layerlist = []\n",
    "        n_emb = sum((nf for ni,nf in emb_szs))\n",
    "        n_in = n_emb + n_cont\n",
    "        for i in layers:\n",
    "            layerlist.append(nn.Linear(n_in,i)) \n",
    "            layerlist.append(nn.ReLU(inplace=True))\n",
    "            layerlist.append(nn.BatchNorm1d(i))\n",
    "            layerlist.append(nn.Dropout(p))\n",
    "            n_in = i\n",
    "        layerlist.append(nn.Linear(layers[-1],out_sz))\n",
    "        self.layers = nn.Sequential(*layerlist)\n",
    "    def forward(self, x_cat, x_cont):\n",
    "        embeddings = []\n",
    "        for i,e in enumerate(self.embeds):\n",
    "            embeddings.append(e(x_cat[:,i]))\n",
    "        x = torch.cat(embeddings, 1)\n",
    "        x = self.emb_drop(x)\n",
    "        x_cont = self.bn_cont(x_cont)\n",
    "        x = torch.cat([x, x_cont], 1)\n",
    "        return self.layers(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now define the model. Before we can load the saved settings, we need to instantiate our TabularModel with the parameters we used before (embedding sizes, number of continuous columns, output size, layer sizes, and dropout layer p-value). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "emb_szs = [(24, 12), (2, 1), (7, 4)]\n",
    "model2 = TabularModel(emb_szs, 6, 1, [200,100], p=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the model is set up, loading the saved settings is a snap."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TabularModel(\n",
       "  (embeds): ModuleList(\n",
       "    (0): Embedding(24, 12)\n",
       "    (1): Embedding(2, 1)\n",
       "    (2): Embedding(7, 4)\n",
       "  )\n",
       "  (emb_drop): Dropout(p=0.4)\n",
       "  (bn_cont): BatchNorm1d(6, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "  (layers): Sequential(\n",
       "    (0): Linear(in_features=23, out_features=200, bias=True)\n",
       "    (1): ReLU(inplace)\n",
       "    (2): BatchNorm1d(200, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (3): Dropout(p=0.4)\n",
       "    (4): Linear(in_features=200, out_features=100, bias=True)\n",
       "    (5): ReLU(inplace)\n",
       "    (6): BatchNorm1d(100, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
       "    (7): Dropout(p=0.4)\n",
       "    (8): Linear(in_features=100, out_features=1, bias=True)\n",
       "  )\n",
       ")"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model2.load_state_dict(torch.load('TaxiFareRegrModel.pt'));\n",
    "model2.eval() # be sure to run this step!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next we'll define a function that takes in new parameters from the user, performs all of the preprocessing steps above, and passes the new data through our trained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def test_data(mdl): # pass in the name of the new model\n",
    "    # INPUT NEW DATA\n",
    "    plat = float(input('What is the pickup latitude?  '))\n",
    "    plong = float(input('What is the pickup longitude? '))\n",
    "    dlat = float(input('What is the dropoff latitude?  '))\n",
    "    dlong = float(input('What is the dropoff longitude? '))\n",
    "    psngr = int(input('How many passengers? '))\n",
    "    dt = input('What is the pickup date and time?\\nFormat as YYYY-MM-DD HH:MM:SS     ')\n",
    "    \n",
    "    # PREPROCESS THE DATA\n",
    "    dfx_dict = {'pickup_latitude':plat,'pickup_longitude':plong,'dropoff_latitude':dlat,\n",
    "         'dropoff_longitude':dlong,'passenger_count':psngr,'EDTdate':dt}\n",
    "    dfx = pd.DataFrame(dfx_dict, index=[0])\n",
    "    dfx['dist_km'] = haversine_distance(dfx,'pickup_latitude', 'pickup_longitude',\n",
    "                                        'dropoff_latitude', 'dropoff_longitude')\n",
    "    dfx['EDTdate'] = pd.to_datetime(dfx['EDTdate'])\n",
    "    \n",
    "    # We can skip the .astype(category) step since our fields are small,\n",
    "    # and encode them right away\n",
    "    dfx['Hour'] = dfx['EDTdate'].dt.hour\n",
    "    dfx['AMorPM'] = np.where(dfx['Hour']<12,0,1) \n",
    "    dfx['Weekday'] = dfx['EDTdate'].dt.strftime(\"%a\")\n",
    "    dfx['Weekday'] = dfx['Weekday'].replace(['Fri','Mon','Sat','Sun','Thu','Tue','Wed'],\n",
    "                                            [0,1,2,3,4,5,6]).astype('int64')\n",
    "    # CREATE CAT AND CONT TENSORS\n",
    "    cat_cols = ['Hour', 'AMorPM', 'Weekday']\n",
    "    cont_cols = ['pickup_latitude', 'pickup_longitude', 'dropoff_latitude',\n",
    "                 'dropoff_longitude', 'passenger_count', 'dist_km']\n",
    "    xcats = np.stack([dfx[col].values for col in cat_cols], 1)\n",
    "    xcats = torch.tensor(xcats, dtype=torch.int64)\n",
    "    xconts = np.stack([dfx[col].values for col in cont_cols], 1)\n",
    "    xconts = torch.tensor(xconts, dtype=torch.float)\n",
    "    \n",
    "    # PASS NEW DATA THROUGH THE MODEL WITHOUT PERFORMING A BACKPROP\n",
    "    with torch.no_grad():\n",
    "        z = mdl(xcats, xconts)\n",
    "    print(f'\\nThe predicted fare amount is ${z.item():.2f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feed new data through the trained model\n",
    "For convenience, here are the max and min values for each of the variables:\n",
    "<table style=\"display: inline-block\">\n",
    "<tr><th>Column</th><th>Minimum</th><th>Maximum</th></tr>\n",
    "<tr><td>pickup_latitude</td><td>40</td><td>41</td></tr>\n",
    "<tr><td>pickup_longitude</td><td>-74.5</td><td>-73.3</td></tr>\n",
    "<tr><td>dropoff_latitude</td><td>40</td><td>41</td></tr>\n",
    "<tr><td>dropoff_longitude</td><td>-74.5</td><td>-73.3</td></tr>\n",
    "<tr><td>passenger_count</td><td>1</td><td>5</td></tr>\n",
    "<tr><td>EDTdate</td><td>2010-04-11 00:00:00</td><td>2010-04-24 23:59:42</td></tr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<strong>Use caution!</strong> The distance between 1 degree of latitude (from 40 to 41) is 111km (69mi) and between 1 degree of longitude (from -73 to -74) is 85km (53mi). The longest cab ride in the dataset spanned a difference of only 0.243 degrees latitude and 0.284 degrees longitude. The mean difference for both latitude and longitude was about 0.02. To get a fair prediction, use values that fall close to one another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "What is the pickup latitude?  40.5\n",
      "What is the pickup longitude? -73.9\n",
      "What is the dropoff latitude?  40.52\n",
      "What is the dropoff longitude? -73.92\n",
      "How many passengers? 2\n",
      "What is the pickup date and time?\n",
      "Format as YYYY-MM-DD HH:MM:SS     2010-04-15 16:00:00\n",
      "\n",
      "The predicted fare amount is $13.86\n"
     ]
    }
   ],
   "source": [
    "z = test_data(model2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}