1-0_Prelude.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 1-0 - Prelude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll start with the Kaggle Titanic dataset. The dataset is a list of Titanic passengers with features such as class, age, sex and fare. We'll use these features to train a model, and use the model to predict whether or not each passenger survived. A more detailed treatment of the dataset can be found here:\n",
    "\n",
    "https://github.com/savarin/python_for_ml"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from time import time\n",
    "\n",
    "np.random.seed(1337)\n",
    "\n",
    "df = pd.read_csv('data/titanic.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Survived</th>\n",
       "      <th>Class</th>\n",
       "      <th>Sex</th>\n",
       "      <th>Age</th>\n",
       "      <th>Fare</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>22.0</td>\n",
       "      <td>7.2500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>38.0</td>\n",
       "      <td>71.2833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>26.0</td>\n",
       "      <td>7.9250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>35.0</td>\n",
       "      <td>53.1000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>35.0</td>\n",
       "      <td>8.0500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Survived  Class  Sex   Age     Fare\n",
       "0         0      3    1  22.0   7.2500\n",
       "1         1      1    0  38.0  71.2833\n",
       "2         1      3    0  26.0   7.9250\n",
       "3         1      1    0  35.0  53.1000\n",
       "4         0      3    1  35.0   8.0500"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We split the data 80-20 into training and test sets. In addition, we scale the data so that each column has mean 0 and standard deviation 1, and create one-hot vectors with the labels (analogous to dummy variables)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df.iloc[:712, :]\n",
    "\n",
    "scaler = StandardScaler()\n",
    "features = ['Class', 'Sex', 'Age', 'Fare']\n",
    "\n",
    "X_train = scaler.fit_transform(df_train[features].values)\n",
    "y_train = df_train['Survived'].values\n",
    "y_train_onehot = pd.get_dummies(df_train['Survived']).values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.83290956,  0.74926865, -0.61259594, -0.51933199],\n",
       "       [-1.55353553, -1.33463478,  0.6184268 ,  0.79718222],\n",
       "       [ 0.83290956, -1.33463478, -0.30484025, -0.5054541 ],\n",
       "       [-1.55353553, -1.33463478,  0.38761004,  0.42333654],\n",
       "       [ 0.83290956,  0.74926865,  0.38761004, -0.50288412]])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[:5, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 1, 1, 1, 0])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1, 0],\n",
       "       [0, 1],\n",
       "       [0, 1],\n",
       "       [0, 1],\n",
       "       [1, 0]], dtype=uint8)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_onehot[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test = df.iloc[712:, :]\n",
    "\n",
    "X_test = scaler.transform(df_test[features].values)\n",
    "y_test = df_test['Survived'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ## Benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a basis for comparison, we train a Random Forest model and record the accuracy on the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "building tree 1 of 100\n",
      "building tree 2 of 100\n",
      "building tree 3 of 100\n",
      "building tree 4 of 100\n",
      "building tree 5 of 100\n",
      "building tree 6 of 100\n",
      "building tree 7 of 100\n",
      "building tree 8 of 100\n",
      "building tree 9 of 100\n",
      "building tree 10 of 100\n",
      "building tree 11 of 100\n",
      "building tree 12 of 100\n",
      "building tree 13 of 100\n",
      "building tree 14 of 100\n",
      "building tree 15 of 100\n",
      "building tree 16 of 100\n",
      "building tree 17 of 100\n",
      "building tree 18 of 100\n",
      "building tree 19 of 100\n",
      "building tree 20 of 100\n",
      "building tree 21 of 100\n",
      "building tree 22 of 100\n",
      "building tree 23 of 100\n",
      "building tree 24 of 100\n",
      "building tree 25 of 100\n",
      "building tree 26 of 100\n",
      "building tree 27 of 100\n",
      "building tree 28 of 100\n",
      "building tree 29 of 100\n",
      "building tree 30 of 100\n",
      "building tree 31 of 100\n",
      "building tree 32 of 100\n",
      "building tree 33 of 100\n",
      "building tree 34 of 100\n",
      "building tree 35 of 100\n",
      "building tree 36 of 100\n",
      "building tree 37 of 100\n",
      "building tree 38 of 100\n",
      "building tree 39 of 100\n",
      "building tree 40 of 100\n",
      "building tree 41 of 100\n",
      "building tree 42 of 100\n",
      "building tree 43 of 100\n",
      "building tree 44 of 100\n",
      "building tree 45 of 100\n",
      "building tree 46 of 100\n",
      "building tree 47 of 100\n",
      "building tree 48 of 100\n",
      "building tree 49 of 100\n",
      "building tree 50 of 100\n",
      "building tree 51 of 100\n",
      "building tree 52 of 100\n",
      "building tree 53 of 100\n",
      "building tree 54 of 100\n",
      "building tree 55 of 100\n",
      "building tree 56 of 100\n",
      "building tree 57 of 100\n",
      "building tree 58 of 100\n",
      "building tree 59 of 100\n",
      "building tree 60 of 100\n",
      "building tree 61 of 100\n",
      "building tree 62 of 100\n",
      "building tree 63 of 100\n",
      "building tree 64 of 100\n",
      "building tree 65 of 100\n",
      "building tree 66 of 100\n",
      "building tree 67 of 100\n",
      "building tree 68 of 100\n",
      "building tree 69 of 100\n",
      "building tree 70 of 100\n",
      "building tree 71 of 100\n",
      "building tree 72 of 100\n",
      "building tree 73 of 100\n",
      "building tree 74 of 100\n",
      "building tree 75 of 100\n",
      "building tree 76 of 100\n",
      "building tree 77 of 100\n",
      "building tree 78 of 100\n",
      "building tree 79 of 100\n",
      "building tree 80 of 100\n",
      "building tree 81 of 100\n",
      "building tree 82 of 100\n",
      "building tree 83 of 100\n",
      "building tree 84 of 100\n",
      "building tree 85 of 100\n",
      "building tree 86 of 100\n",
      "building tree 87 of 100\n",
      "building tree 88 of 100\n",
      "building tree 89 of 100\n",
      "building tree 90 of 100\n",
      "building tree 91 of 100\n",
      "building tree 92 of 100\n",
      "building tree 93 of 100\n",
      "building tree 94 of 100\n",
      "building tree 95 of 100\n",
      "building tree 96 of 100\n",
      "building tree 97 of 100\n",
      "building tree 98 of 100\n",
      "building tree 99 of 100\n",
      "building tree 100 of 100\n",
      "\n",
      "accuracy 0.8324022346368715\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:    0.1s finished\n",
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done   1 out of   1 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done   2 out of   2 | elapsed:    0.0s remaining:    0.0s\n",
      "[Parallel(n_jobs=1)]: Done 100 out of 100 | elapsed:    0.0s finished\n"
     ]
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "model = RandomForestClassifier(random_state=0, verbose=3)\n",
    "model = model.fit(X_train, y_train)\n",
    "\n",
    "y_prediction = model.predict(X_test)\n",
    "print(\"\\naccuracy\", np.sum(y_prediction == y_test) / float(len(y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1-layer Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the input, we have a vector of length 4 that represents each passenger's features. As an example, we consider the first passenger in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.83290956  0.74926865 -0.61259594 -0.51933199]\n"
     ]
    }
   ],
   "source": [
    "print(X_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the output, we want a vector of length 2 to represent the survival probabilities. A simple way to create this mapping is by using a 2x4 matrix. To start off, we generate a random matrix representing feature weights and apply the matrix to our input. We'll also add a bias term."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[0.00262025 0.00158684 0.00278127 0.00459317]\n",
      " [0.00321001 0.00518393 0.00261943 0.00976085]]\n"
     ]
    }
   ],
   "source": [
    "W = np.random.rand(2, 4) * 0.01\n",
    "\n",
    "print(W)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.00732815 0.00115274]\n"
     ]
    }
   ],
   "source": [
    "b = np.random.rand(2,) * 0.01\n",
    "\n",
    "print(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.00661037 0.00103677]\n"
     ]
    }
   ],
   "source": [
    "result = np.dot(W, X_train[0]) + b\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get the output vector to sum to 1, we apply a softmax mapping. The first element would now represent the probability that the passenger did not survive, and the second element represents the probability the passenger survives."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def softmax(x):\n",
    "    return np.exp(x) / np.exp(x).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.5013934 0.4986066]\n"
     ]
    }
   ],
   "source": [
    "result = softmax(result)\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then compare the output vector to the actual label. We would have a 'good' model if the probability for the correct label was close to 1, and a 'bad' one if it was close to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 0]\n"
     ]
    }
   ],
   "source": [
    "print(y_train_onehot[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n"
     ]
    }
   ],
   "source": [
    "label_index = np.argmax(y_train_onehot[0])\n",
    "\n",
    "print(label_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predicted label-0 probability 0.5013933977819183\n"
     ]
    }
   ],
   "source": [
    "print(\"predicted label-0 probability\", result[label_index])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define loss to be the negative logarithm of the probability of the correct label. Taking the logarithm penalizes the model for having a high probability associated with the wrong label. Here we have the loss associated with the first passenger."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss for first passenger 0.6903642609116132\n"
     ]
    }
   ],
   "source": [
    "loss = -np.log(result[label_index])\n",
    "\n",
    "print(\"loss for first passenger\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run the calculation through all the passengers in the training set, and divide the total loss by the number of passengers to obtain the average loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "average loss across all passengers 0.6938940425068784\n"
     ]
    }
   ],
   "source": [
    "for j in range(X_train.shape[0]):\n",
    "    result = np.dot(W, X_train[j]) + b\n",
    "    result = softmax(result)\n",
    "        \n",
    "    label_index = np.argmax(y_train_onehot[j])\n",
    "    loss += -np.log(result[label_index])\n",
    "\n",
    "loss = loss / float(X_train.shape[0])\n",
    "\n",
    "print(\"average loss across all passengers\", loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we iterate through 1000 iterations for random values of W and b, and keep the pair which minimizes the average loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.711 accuracy 0.317 loop 0\n",
      "loss 0.698 accuracy 0.393 loop 1\n",
      "loss 0.67 accuracy 0.772 loop 2\n",
      "loss 0.663 accuracy 0.785 loop 4\n",
      "loss 0.661 accuracy 0.749 loop 156\n",
      "loss 0.659 accuracy 0.725 loop 452\n",
      "loss 0.658 accuracy 0.787 loop 715\n",
      "\n",
      "time taken 9.314749240875244 seconds\n"
     ]
    }
   ],
   "source": [
    "min_loss = 1000\n",
    "best_weights = ()\n",
    "\n",
    "start = time()\n",
    "\n",
    "for i in range(1000):\n",
    "    W = np.random.rand(2, 4) / 10\n",
    "    b = np.random.rand(2,) / 10\n",
    "\n",
    "    scores = []\n",
    "    loss = 0\n",
    "    \n",
    "    for j in range(X_train.shape[0]):\n",
    "        result = np.dot(W, X_train[j]) + b\n",
    "        result = softmax(result)\n",
    "        scores.append(list(result))\n",
    "        \n",
    "        label_index = np.argmax(y_train_onehot[j])\n",
    "        loss += -np.log(result[label_index])\n",
    "\n",
    "    loss = loss / float(X_train.shape[0])\n",
    "    y_prediction = np.argmax(np.array(scores), axis=1)\n",
    "    accuracy = np.sum(y_prediction == y_train) / float(len(y_train))\n",
    "    \n",
    "    if loss < min_loss:\n",
    "        min_loss = loss\n",
    "        best_weights = (W, b)\n",
    "        print(\"loss %s accuracy %s loop %s\" % (round(loss, 3), round(accuracy, 3), i))\n",
    "\n",
    "print(\"\\ntime taken %s seconds\" % str(time() - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy 0.7821229050279329\n"
     ]
    }
   ],
   "source": [
    "W, b = best_weights\n",
    "scores = []\n",
    "\n",
    "for j in range(X_test.shape[0]):\n",
    "    result = np.dot(W, X_test[j]) + b\n",
    "    result = softmax(result)\n",
    "    scores.append(list(result))\n",
    "\n",
    "y_prediction = np.argmax(np.array(scores), axis=1)\n",
    "\n",
    "print(\"accuracy\", np.sum(y_prediction == y_test) / float(len(y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each passenger, predictions for the test set were made by selecting the label with the highest probability. Despite the naïve approach, we obtain a prediction accuracy of 78%!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-layer Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With the 1-layer neural network, we had a 2x4 weight matrix. To create more degrees of freedom, we can introduce intermediary matrices or 'layers'. For example, instead of having a mapping from a vector of length 4 to a vector of length 2, we'll have two mappings - first from 4 to 100, followed by 100 to 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_1 = np.random.rand(100, 4) * 0.01\n",
    "b_1 = np.random.rand(100,) * 0.01\n",
    "W_2 = np.random.rand(2, 100) * 0.01\n",
    "b_2 = np.random.rand(2,) * 0.01"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 1.13174863e-02  3.27517674e-03  6.64636413e-03  6.67664294e-03\n",
      "  3.43098480e-03  1.36640330e-02  9.18247683e-03  1.17094685e-02\n",
      "  1.01565364e-03 -1.60432219e-03  9.35207856e-03  7.47293230e-03\n",
      "  1.52676682e-02  7.97936749e-03  6.55563928e-04  6.00362175e-03\n",
      "  7.23496560e-03  9.92625256e-03  4.14720990e-03  5.41832887e-03\n",
      "  1.18845724e-02  1.60057818e-02  7.64679793e-03  6.94767477e-03\n",
      "  1.33371235e-02  6.07771410e-03  1.96233598e-02  1.57180776e-02\n",
      "  9.84855859e-03  3.82086941e-03  3.77595128e-04  2.26082278e-03\n",
      "  1.45080114e-03  1.45319158e-02  7.90452470e-03  8.92775565e-03\n",
      "  1.61055072e-02  8.93604299e-03  1.02217040e-02  9.01988122e-03\n",
      "  1.02962602e-02  4.56788321e-03  1.14912002e-02 -2.16470496e-04\n",
      "  7.17207370e-03  1.32500054e-02  2.73586936e-03  4.19898797e-03\n",
      "  5.70306531e-03  9.32144683e-03  8.41222049e-03  1.03885822e-02\n",
      "  1.64928832e-02 -2.05164035e-03  1.72807609e-02  3.94223376e-03\n",
      " -3.86111353e-03  1.54912269e-02  7.65819757e-03  2.88205757e-05\n",
      "  8.11385633e-03  1.10081834e-02  1.88604176e-03  1.22567360e-02\n",
      "  5.81988306e-03  6.61609950e-03  1.30486432e-02  1.84395962e-03\n",
      "  1.78840358e-02 -1.46500280e-03  1.40330891e-02  8.37979073e-03\n",
      "  1.12545050e-02 -1.93523942e-03  1.10487202e-03  1.59472488e-02\n",
      "  2.13196859e-03  1.51578949e-02  7.99031206e-03  1.33981824e-02\n",
      "  9.99760788e-04  7.04023165e-03  1.21942983e-02  1.33117699e-02\n",
      "  1.14257228e-02  1.31578297e-02  1.03026592e-02  4.83051271e-03\n",
      "  1.40758009e-02  7.29455432e-03  1.50803904e-03  7.33072912e-03\n",
      "  8.18612611e-03  1.45065360e-02 -5.13711067e-04  7.61443534e-03\n",
      " -2.83954620e-04  7.66773697e-03  2.88720973e-03  8.91764272e-03]\n"
     ]
    }
   ],
   "source": [
    "result = np.dot(W_1, X_train[0]) + b_1\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.00977321 0.00541989]\n"
     ]
    }
   ],
   "source": [
    "result = np.dot(W_2, result) + b_2\n",
    "\n",
    "print(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before we iterate through 1000 iterations of random values for W_1, b_1, W_2 and b_2, and keep the one which minimizes loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.693 accuracy 0.375 loop 0\n",
      "loss 0.693 accuracy 0.61 loop 1\n",
      "loss 0.692 accuracy 0.61 loop 3\n",
      "loss 0.692 accuracy 0.61 loop 22\n",
      "loss 0.692 accuracy 0.61 loop 59\n",
      "loss 0.692 accuracy 0.61 loop 144\n",
      "loss 0.692 accuracy 0.61 loop 229\n",
      "loss 0.692 accuracy 0.61 loop 511\n",
      "loss 0.692 accuracy 0.61 loop 808\n",
      "loss 0.692 accuracy 0.61 loop 809\n",
      "\n",
      "time taken 10.838479995727539 seconds\n"
     ]
    }
   ],
   "source": [
    "min_loss = 1000\n",
    "best_weights = ()\n",
    "\n",
    "start = time()\n",
    "\n",
    "for i in range(1000):\n",
    "    W_1 = np.random.rand(100, 4) * 0.01\n",
    "    b_1 = np.random.rand(100,) * 0.01\n",
    "    W_2 = np.random.rand(2, 100) * 0.01\n",
    "    b_2 = np.random.rand(2,) * 0.01\n",
    "    \n",
    "    scores = []\n",
    "    loss = 0\n",
    "\n",
    "    for j in range(X_train.shape[0]):\n",
    "        result = np.dot(W_1, X_train[j]) + b_1\n",
    "        result = np.dot(W_2, result) + b_2\n",
    "        result = softmax(result)\n",
    "        scores.append(list(result))\n",
    "        \n",
    "        label_index = np.argmax(y_train_onehot[j])\n",
    "        loss += -np.log(result[label_index])\n",
    "\n",
    "    loss = loss / float(X_train.shape[0])\n",
    "    y_prediction = np.argmax(np.array(scores), axis=1)\n",
    "    accuracy = np.sum(y_prediction == y_train) / float(len(y_train))\n",
    "        \n",
    "    if loss < min_loss:\n",
    "        min_loss = loss\n",
    "        best_weights = (W_1, b_1, W_2, b_2)\n",
    "        print(\"loss %s accuracy %s loop %s\" % (round(loss, 3), round(accuracy, 3), i))\n",
    "\n",
    "print(\"\\ntime taken %s seconds\" % str(time() - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy 0.6424581005586593\n"
     ]
    }
   ],
   "source": [
    "W_1, b_1, W_2, b_2 = best_weights\n",
    "scores = []\n",
    "\n",
    "for j in range(X_test.shape[0]):\n",
    "    result = np.dot(W_1, X_test[j]) + b_1\n",
    "    result = np.dot(W_2, result) + b_2\n",
    "    result = softmax(result)\n",
    "    scores.append(list(result))\n",
    "    \n",
    "y_prediction = np.argmax(np.array(scores), axis=1)\n",
    "\n",
    "print(\"accuracy\", np.sum(y_prediction == y_test) / float(len(y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Despite the greater degree of freedom, we get an accuracy score of only 64%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3-layer Neural Network"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We take this a step further by adding an additional layer, and similarly review model performance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss 0.693 accuracy 0.61 loop 0\n",
      "loss 0.693 accuracy 0.61 loop 1\n",
      "loss 0.693 accuracy 0.61 loop 20\n",
      "loss 0.692 accuracy 0.61 loop 58\n",
      "loss 0.692 accuracy 0.61 loop 74\n",
      "loss 0.692 accuracy 0.61 loop 563\n",
      "loss 0.692 accuracy 0.61 loop 809\n",
      "loss 0.692 accuracy 0.61 loop 990\n",
      "\n",
      "time taken 15.281288862228394 seconds\n"
     ]
    }
   ],
   "source": [
    "min_loss = 1000\n",
    "best_weights = ()\n",
    "\n",
    "start = time()\n",
    "\n",
    "for i in range(1000):\n",
    "    W_1 = np.random.rand(100, 4) * 0.01\n",
    "    b_1 = np.random.rand(100,) * 0.01\n",
    "    W_2 = np.random.rand(100, 100) * 0.01\n",
    "    b_2 = np.random.rand(100,) * 0.01\n",
    "    W_3 = np.random.rand(2, 100) * 0.01\n",
    "    b_3 = np.random.rand(2,) * 0.01\n",
    "    \n",
    "    scores = []\n",
    "    loss = 0\n",
    "\n",
    "    for j in range(X_train.shape[0]):\n",
    "        result = np.dot(W_1, X_train[j]) + b_1\n",
    "        result = np.dot(W_2, result) + b_2\n",
    "        result = np.dot(W_3, result) + b_3\n",
    "        result = softmax(result)\n",
    "        scores.append(list(result))\n",
    "        \n",
    "        label_index = np.argmax(y_train_onehot[j])\n",
    "        loss += -np.log(result[label_index])\n",
    "        \n",
    "    loss = loss / float(X_train.shape[0])\n",
    "    y_prediction = np.argmax(np.array(scores), axis=1)\n",
    "    accuracy = np.sum(y_prediction == y_train) / float(len(y_train))          \n",
    "        \n",
    "    if loss < min_loss:\n",
    "        min_loss = loss\n",
    "        best_weights = (W_1, b_1, W_2, b_2, W_3, b_3)\n",
    "        print(\"loss %s accuracy %s loop %s\" % (round(loss, 3), round(accuracy, 3), i))\n",
    "\n",
    "print(\"\\ntime taken %s seconds\" % str(time() - start))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy 0.6424581005586593\n"
     ]
    }
   ],
   "source": [
    "W_1, b_1, W_2, b_2, W_3, b_3 = best_weights\n",
    "scores = []\n",
    "\n",
    "for j in range(X_test.shape[0]):\n",
    "    result = np.dot(W_1, X_test[j]) + b_1\n",
    "    result = np.dot(W_2, result) + b_2\n",
    "    result = np.dot(W_3, result) + b_3    \n",
    "    result = softmax(result)\n",
    "    scores.append(list(result))\n",
    "    \n",
    "y_prediction = np.argmax(np.array(scores), axis=1)\n",
    "\n",
    "print(\"accuracy\", np.sum(y_prediction == y_test) / float(len(y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We run into the same problem of suboptimal model performance, but this is not unexpected given the simplistic approach taken for illustrative purposes. We look to build on this in the next section by taking a systematic approach to optimizing the weight matrices and biases."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}