mlp_classification_with_mnist.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "mlp_classification_with_mnist.ipynb",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Joycechidi/Deep-Learning-/blob/master/mlp_classification_with_mnist.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mi61-aCfZckv",
        "colab_type": "text"
      },
      "source": [
        "# Multi-Layer Perceptron, MNIST\n",
        "---\n",
        "In this notebook, we will train an MLP to classify images from the [MNIST database](http://yann.lecun.com/exdb/mnist/) hand-written digit database.\n",
        "\n",
        "The process will be broken down into the following steps:\n",
        ">1. Load and visualize the data\n",
        "2. Define a neural network\n",
        "3. Train the model\n",
        "4. Evaluate the performance of our trained model on a test dataset!\n",
        "\n",
        "Before we begin, we have to import the necessary libraries for working with data and PyTorch.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qduBBNT3ZOA0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# import libraries\n",
        "import torch\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pkUNZa8HZp4i",
        "colab_type": "text"
      },
      "source": [
        "## Load and Visualize the [Data](http://pytorch.org/docs/stable/torchvision/datasets.html)\n",
        "\n",
        "Downloading may take a few moments, and you should see your progress as the data is loading. You may also choose to change the `batch_size` if you want to load more data at a time.\n",
        "\n",
        "This cell will create DataLoaders for each of our datasets."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Fr7xxtMZ2GZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from torchvision import datasets\n",
        "import torchvision.transforms as transforms\n",
        "from torch.utils.data.sampler import SubsetRandomSampler\n",
        "\n",
        "# number of subprocesses to use for data loading\n",
        "num_workers = 0\n",
        "# how many samples per batch to load\n",
        "batch_size = 20\n",
        "\n",
        "# percentage of training set to use as validation\n",
        "valid_size = 0.2\n",
        "\n",
        "# convert data to torch.FloatTensor\n",
        "transform = transforms.ToTensor()\n",
        "\n",
        "# choose the training and test datasets\n",
        "train_data = datasets.MNIST(root='data', train=True,\n",
        "                                   download=True, transform=transform)\n",
        "test_data = datasets.MNIST(root='data', train=False,\n",
        "                                  download=True, transform=transform)\n",
        "#get training indices that will be used for validation\n",
        "num_train = len(train_data)\n",
        "indices = list(range(num_train))\n",
        "np.random.shuffle(indices)\n",
        "split = int(np.floor(valid_size * num_train))\n",
        "train_idx, valid_idx = indices[split:], indices[:split]\n",
        "\n",
        "# define samplers for obtaining training and validation batches\n",
        "train_sampler = SubsetRandomSampler(train_idx)\n",
        "valid_sampler = SubsetRandomSampler(valid_idx)\n",
        "\n",
        "# prepare data loaders\n",
        "train_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, sampler=train_sampler, num_workers=num_workers)\n",
        "valid_loader = torch.utils.data.DataLoader(train_data, batch_size=batch_size, sampler=valid_sampler, num_workers=num_workers)\n",
        "test_loader = torch.utils.data.DataLoader(test_data, batch_size=batch_size, num_workers=num_workers)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ux3MuFbAZ8gN",
        "colab_type": "text"
      },
      "source": [
        "### Visualize a Batch of Training Data\n",
        "\n",
        "The first step in a classification task is to take a look at the data, make sure it is loaded in correctly, then make any initial observations about patterns in that data."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YgnGLS5pZ1-S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline\n",
        "    \n",
        "# obtain one batch of training images\n",
        "dataiter = iter(train_loader)\n",
        "images, labels = dataiter.next()\n",
        "images = images.numpy()\n",
        "\n",
        "# plot the images in the batch, along with the corresponding labels\n",
        "fig = plt.figure(figsize=(25, 4))\n",
        "for idx in np.arange(20):\n",
        "    ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])\n",
        "    ax.imshow(np.squeeze(images[idx]), cmap='gray')\n",
        "    # print out the correct label for each image\n",
        "    # .item() gets the value contained in a Tensor\n",
        "    ax.set_title(str(labels[idx].item()))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W6If35gaaEVk",
        "colab_type": "text"
      },
      "source": [
        "### View an Image in More Detail"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ayJ5iex1Z10h",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "img = np.squeeze(images[1])\n",
        "\n",
        "fig = plt.figure(figsize = (12,12)) \n",
        "ax = fig.add_subplot(111)\n",
        "ax.imshow(img, cmap='gray')\n",
        "width, height = img.shape\n",
        "thresh = img.max()/2.5\n",
        "for x in range(width):\n",
        "    for y in range(height):\n",
        "        val = round(img[x][y],2) if img[x][y] !=0 else 0\n",
        "        ax.annotate(str(val), xy=(y,x),\n",
        "                    horizontalalignment='center',\n",
        "                    verticalalignment='center',\n",
        "                    color='white' if img[x][y]<thresh else 'black')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zmx1Io8zaMZW",
        "colab_type": "text"
      },
      "source": [
        "---\n",
        "## Define the Network [Architecture](http://pytorch.org/docs/stable/nn.html)\n",
        "\n",
        "The architecture will be responsible for seeing as input a 784-dim Tensor of pixel values for each image, and producing a Tensor of length 10 (our number of classes) that indicates the class scores for an input image. This particular example uses two hidden layers and dropout to avoid overfitting."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aRPnGJBBZ1Cq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "\n",
        "## TODO: Define the NN architecture\n",
        "class Net(nn.Module):\n",
        "    def __init__(self):\n",
        "        super(Net, self).__init__()\n",
        "        hidden_layer_1 = 512\n",
        "        hidden_layer_2 = 512\n",
        "        \n",
        "        \n",
        "        # linear layer1 (784 -> 1 hidden node)\n",
        "        self.fc1 = nn.Linear(28 * 28, hidden_layer_1)\n",
        "         # linear layer 2\n",
        "        self.fc2 = nn.Linear(hidden_layer_1, hidden_layer_2)\n",
        "         # linear layer 3\n",
        "        self.fc3 = nn.Linear(hidden_layer_2, 10) # 10 is number of digits from 0 to 9\n",
        "        \n",
        "        #dropout layer with 20% dropout rate\n",
        "        self.dropout = nn.Dropout(0.2)\n",
        "\n",
        "    def forward(self, x):\n",
        "        # flatten image input\n",
        "        x = x.view(-1, 28 * 28)\n",
        "        # add hidden layer, with relu activation function\n",
        "        x = F.relu(self.fc1(x))\n",
        "        #add hidden droupout layer\n",
        "        x = self.dropout(x)\n",
        "        \n",
        "        #add 2nd hidden layer with relu activation\n",
        "        x = F.relu(self.fc2(x))\n",
        "        #add hidden droupout layer\n",
        "        x = self.dropout(x)\n",
        "    \n",
        "        #add hidden droupout layer       \n",
        "        x = self.fc3(x)\n",
        "        return x\n",
        "\n",
        "# initialize the NN\n",
        "model = Net()\n",
        "print(model)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YyNV-KkwaVrG",
        "colab_type": "text"
      },
      "source": [
        "###  Specify [Loss Function](http://pytorch.org/docs/stable/nn.html#loss-functions) and [Optimizer](http://pytorch.org/docs/stable/optim.html)\n",
        "\n",
        "It's recommended that you use cross-entropy loss for classification. If you look at the documentation (linked above), you can see that PyTorch's cross entropy function applies a softmax funtion to the output layer *and* then calculates the log loss."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KniOXggTaIz7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## TODO: Specify loss and optimization functions\n",
        "\n",
        "# specify loss function\n",
        "criterion = nn.CrossEntropyLoss()\n",
        "\n",
        "# specify optimizer (Stochastic Gradient Descent)\n",
        "optimizer = torch.optim.SGD(model.parameters(), lr=0.01)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RY11sbYNapPM",
        "colab_type": "text"
      },
      "source": [
        "## Train the Network\n",
        "\n",
        "The steps for training/learning from a batch of data are described in the comments below:\n",
        "1. Clear the gradients of all optimized variables\n",
        "2. Forward pass: compute predicted outputs by passing inputs to the model\n",
        "3. Calculate the loss\n",
        "4. Backward pass: compute gradient of the loss with respect to model parameters\n",
        "5. Perform a single optimization step (parameter update)\n",
        "6. Update average training loss\n",
        "\n",
        "The following loop trains for 60 epochs;  I'll take a look at how the values for the training loss decrease over time. I want it to decrease while also avoiding overfitting the training data. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mPYzvrfYaIxB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# number of epochs to train the model\n",
        "n_epochs = 60  # suggest training between 20-50 epochs\n",
        "\n",
        "# initialize tracker for minimum validation loss\n",
        "valid_loss_min = np.Inf # set initial \"min\" to infinity\n",
        "\n",
        "for epoch in range(n_epochs):\n",
        "    # monitor training loss\n",
        "    train_loss = 0.0\n",
        "    valid_loss = 0.0\n",
        "    \n",
        "    ###################\n",
        "    # train the model #\n",
        "    ###################\n",
        "    model.train() # prep model for training\n",
        "    for data, target in train_loader:\n",
        "        # clear the gradients of all optimized variables\n",
        "        optimizer.zero_grad()\n",
        "        # forward pass: compute predicted outputs by passing inputs to the model\n",
        "        output = model(data)\n",
        "        # calculate the loss\n",
        "        loss = criterion(output, target)\n",
        "        # backward pass: compute gradient of the loss with respect to model parameters\n",
        "        loss.backward()\n",
        "        # perform a single optimization step (parameter update)\n",
        "        optimizer.step()\n",
        "        # update running training loss\n",
        "        train_loss += loss.item()*data.size(0)\n",
        "        \n",
        "        ######################    \n",
        "        # validate the model #\n",
        "        ######################\n",
        "    model.eval() # prep model for evaluation\n",
        "    for data, target in valid_loader:\n",
        "        # forward pass: compute predicted outputs by passing inputs to the model\n",
        "        output = model(data)\n",
        "        # calculate the loss\n",
        "        loss = criterion(output, target)\n",
        "        # update running validation loss \n",
        "        valid_loss += loss.item()*data.size(0)\n",
        "        \n",
        "    # print training statistics \n",
        "    # calculate average loss over an epoch\n",
        "    train_loss = train_loss/len(train_loader.sampler)\n",
        "    valid_loss = valid_loss/len(valid_loader.sampler)\n",
        "\n",
        "    print('Epoch: {} \\tTraining Loss: {:.6f} \\fValidation Loss: {:.6f}'.format(\n",
        "        epoch+1, \n",
        "        train_loss,\n",
        "        valid_loss))\n",
        "    \n",
        "    # save model if validation loss has decreased\n",
        "    if valid_loss <= valid_loss_min:\n",
        "        print('Validation loss decreased ({:.6f} --> {:.6f}).Saving model ...'.format(\n",
        "        valid_loss_min,\n",
        "            valid_loss\n",
        "        ))\n",
        "        torch.save(model.state_dict(), 'model.pt')\n",
        "        valid_loss_min = valid_loss\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jxv3Kdk3ayjh",
        "colab_type": "text"
      },
      "source": [
        "# Load the Model with the Lowest Validation Loss"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "K0e3tfdJaIt2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.load_state_dict(torch.load('model.pt'))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kqDe_OqZa6E6",
        "colab_type": "text"
      },
      "source": [
        "---\n",
        "## Test the Trained Network\n",
        "\n",
        "Finally, we test our best model on previously unseen **test data** and evaluate it's performance. Testing on unseen data is a good way to check that our model generalizes well. It may also be useful to be granular in this analysis and take a look at how this model performs on each class as well as looking at its overall loss and accuracy.\n",
        "\n",
        "#### `model.eval()`\n",
        "\n",
        "`model.eval(`) will set all the layers in your model to evaluation mode. This affects layers like dropout layers that turn \"off\" nodes during training with some probability, but should allow every node to be \"on\" for evaluation!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tV_1GeUQaIqw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# initialize lists to monitor test loss and accuracy\n",
        "test_loss = 0.0\n",
        "class_correct = list(0. for i in range(10))\n",
        "class_total = list(0. for i in range(10))\n",
        "\n",
        "model.eval() # prep model for *evaluation*\n",
        "\n",
        "for data, target in test_loader:\n",
        "    # forward pass: compute predicted outputs by passing inputs to the model\n",
        "    output = model(data)\n",
        "    # calculate the loss\n",
        "    loss = criterion(output, target)\n",
        "    # update test loss \n",
        "    test_loss += loss.item()*data.size(0)\n",
        "    # convert output probabilities to predicted class\n",
        "    _, pred = torch.max(output, 1)\n",
        "    # compare predictions to true label\n",
        "    correct = np.squeeze(pred.eq(target.data.view_as(pred)))\n",
        "    # calculate test accuracy for each object class\n",
        "    for i in range(batch_size):\n",
        "        label = target.data[i]\n",
        "        class_correct[label] += correct[i].item()\n",
        "        class_total[label] += 1\n",
        "\n",
        "# calculate and print avg test loss\n",
        "test_loss = test_loss/len(test_loader.dataset)\n",
        "print('Test Loss: {:.6f}\\n'.format(test_loss))\n",
        "\n",
        "for i in range(10):\n",
        "    if class_total[i] > 0:\n",
        "        print('Test Accuracy of %5s: %2d%% (%2d/%2d)' % (\n",
        "            str(i), 100 * class_correct[i] / class_total[i],\n",
        "            np.sum(class_correct[i]), np.sum(class_total[i])))\n",
        "    else:\n",
        "        print('Test Accuracy of %5s: N/A (no training examples)' % (classes[i]))\n",
        "\n",
        "print('\\nTest Accuracy (Overall): %2d%% (%2d/%2d)' % (\n",
        "    100. * np.sum(class_correct) / np.sum(class_total),\n",
        "    np.sum(class_correct), np.sum(class_total)))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rhgvl2AwbI9k",
        "colab_type": "text"
      },
      "source": [
        "### Visualize Sample Test Results\n",
        "\n",
        "This cell displays test images and their labels in this format: `predicted (ground-truth)`. The text will be green for accurately classified examples and red for incorrect predictions."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PLDoFwx8bP-Q",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# obtain one batch of test images\n",
        "dataiter = iter(test_loader)\n",
        "images, labels = dataiter.next()\n",
        "\n",
        "# get sample outputs\n",
        "output = model(images)\n",
        "# convert output probabilities to predicted class\n",
        "_, preds = torch.max(output, 1)\n",
        "# prep images for display\n",
        "images = images.numpy()\n",
        "\n",
        "# plot the images in the batch, along with predicted and true labels\n",
        "fig = plt.figure(figsize=(25, 4))\n",
        "for idx in np.arange(20):\n",
        "    ax = fig.add_subplot(2, 20/2, idx+1, xticks=[], yticks=[])\n",
        "    ax.imshow(np.squeeze(images[idx]), cmap='gray')\n",
        "    ax.set_title(\"{} ({})\".format(str(preds[idx].item()), str(labels[idx].item())),\n",
        "                 color=(\"green\" if preds[idx]==labels[idx] else \"red\"))"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}