SD_TSIA203_MLP_for_classification.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Student Name: Mohamad Jawad ABOU ALEIWI"
      ],
      "metadata": {
        "id": "1SuV1oc4yar2"
      }
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WpYK9fIlyZCq"
      },
      "source": [
        "# Lab Deep Learning / Multi-Layer Perceptron for classification / in pytorch\n",
        "\n",
        "**Author: geoffroy.peeters@telecom-paris.fr**\n",
        "\n",
        "For any remark or suggestion, please feel free to contact me.\n",
        "\n",
        "Last edit: 2020/04/27 peeters\n",
        "\n",
        "Read the \"introduction to pytorch\" slides first: https://perso.telecom-paristech.fr/gpeeters/doc/pytorch/\n",
        "\n",
        "**Is is recommended that you first perform the ```Lab Deep Learning / Multi-Layer Perceptron for regression / in pytorch``` Lab to understand how to write a Neural Network in pytorch.**\n",
        "\n",
        "\n",
        "## Objective:\n",
        "\n",
        "The objective of this lab is to develop a two hidden layers MLP to perform image **classification**.\n",
        "\n",
        "While the previous Lab ```Lab Deep Learning / Multi-Layer Perceptron for regression / in pytorch``` focused on how to write a Neural Network, the present Lab focuses on managing datasets and splitting training and testing .\n",
        "\n",
        "We will use MNIST for the image dataset.\n",
        "\n",
        "## Your task:\n",
        "\n",
        "You need to add the missing parts in the code (parts between ```# --- START CODE HERE``` and ```# --- END CODE HERE```)\n",
        "\n",
        "## Documentation:\n",
        "- NN: https://pytorch.org/docs/stable/nn.html\n",
        "- Autograd: https://pytorch.org/docs/stable/autograd.html\n",
        "- Optim: https://pytorch.org/docs/stable/optim.html\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h6o3NZfNyZCt"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torchvision\n",
        "import torchvision.transforms as transforms\n",
        "import torch.nn as nn\n",
        "import torch.nn.functional as F\n",
        "import torch.optim as optim\n",
        "from torchvision import datasets, transforms\n",
        "import matplotlib.pyplot as plt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Vy7u-IVkyZCu"
      },
      "source": [
        "## Data management\n",
        "\n",
        "It is common to separate the datasets into\n",
        "- a **training** part (used to find the network parameters which minimize the loss)\n",
        "- a **test part** (used to estimate the performances).\n",
        "\n",
        "The training part can itself be further splitted into a training and a validation part (the validation part is then used to fix the hyper-parameters of the system).\n",
        "\n",
        "When performing ```batch gradient descent``` the whole training set is used to compute the gradients (which itself is used for the parameter optimization).\n",
        "However when performing ```stochastic or mini-batch gradient descent``` only a part of the training data is used at each iteration.\n",
        "\n",
        "We therefore needs to write all the necessary code, to get these mini-batches of data, be sure that all data have been seen, potentially randomize the order of appearance of these data and potentially perform some modifications of the data before giving them to the network.\n",
        "\n",
        "Fortunatelly, pytorch has associated to it (but not included in it) a nice package (```torchvision```) which allows to do all that for us.\n",
        "\n",
        "This package allows to perform these process for any datasets but has a pre-tuned method for the ```MNIST```datasets.\n",
        "\n",
        "### Datasets\n",
        "\n",
        "In the following we will us ```datasets.MNIST``` to load the train and test data.\n",
        "\n",
        "We need to tell it to download the data, where to dowload them and if we want the training or the testing part of it.\n",
        "\n",
        "We then tell it if we want to apply some transformations to the data.\n",
        "\n",
        "For this lab, we will transform the data by\n",
        "- 1) transforming them to torch tensors (using ```transforms.ToTensor()```); since by default they are not torch tensors\n",
        "- 2) normalize them to zero mean and unit standard deviation (using ```transforms.Normalize()```).\n",
        "\n",
        "It is of course possible to also apply data augmentation.\n",
        "\n",
        "Since we have several transforms to be performed, we will compose them using ```transforms.Compose([transform1, transform2])```\n",
        "\n",
        "The corresponding datasets is a class which contains the pairs of [input tensors $X$, corresponding ground-truth label $y$].\n",
        "\n",
        "### Dataloader\n",
        "\n",
        "Another nice pytorch facility is the ```Dataloader``` which transforms the dataset to be used as an iterator (i.e. within ```for data in myDataloader: ```).\n",
        "\n",
        "Moreover, ```Dataloader``` allows to specify\n",
        "- how many data will be provided each time it is called (```batch_size```)\n",
        "- if the order of appearance of the data with be random (````shuffle````).\n",
        "\n",
        "We therefore convert our datasets (train and test) to ```DataLoader``` that are directly used as iterator during the iterations (it DataLoader manage directly the mini-batch and shuffling)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zsSEvakPyZCv"
      },
      "outputs": [],
      "source": [
        "# --- START CODE HERE\n",
        "\n",
        "transform = transforms.Compose([\n",
        "    transforms.ToTensor(),\n",
        "    transforms.Normalize((0.5,), (0.5,))\n",
        "])\n",
        "\n",
        "# Download and load the datasets\n",
        "train_set = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=transform)\n",
        "train_loader = torch.utils.data.DataLoader(train_set, batch_size=64, shuffle=True)\n",
        "test_set = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=transform)\n",
        "test_loader = torch.utils.data.DataLoader(test_set, batch_size=64, shuffle=False)\n",
        "# --- END CODE HERE"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FFuEXr0tyZCw",
        "outputId": "f8a134ce-454c-429f-c8e2-76e830b15c22",
        "colab": {
          "base_uri": "https://localhost:8080/"
        }
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "dataset type: <class 'torchvision.datasets.mnist.MNIST'>\n",
            "dataset size: 60000\n",
            "X data: torch.Size([1, 28, 28])\n",
            "y data: 5\n"
          ]
        }
      ],
      "source": [
        "print(\"dataset type: {}\".format(type(train_set)))\n",
        "print(\"dataset size: {}\".format(len(train_set)))\n",
        "print(\"X data: {}\".format(train_set[0][0].size()))\n",
        "print(\"y data: {}\".format(train_set[0][1]))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "piAf25dxyZCx"
      },
      "source": [
        "### Display the first 5 data of MNIST dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Jsa2MAEhyZCx",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 168
        },
        "outputId": "271ab4b6-b237-477b-96bf-4bbb1ffe0747"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 5 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAACXCAYAAAC1ITlNAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAj3klEQVR4nO3deXhU1d0H8O8kZCYBsrBlI4msEQhgNIQYUN5oUyNUFEmQam2xL7IZEAxUpCoWRKmlFESComWzaGnBV7YCLQYMIntKrCwhoFgiJEGRLASyzn3/iJxzB2ZIJpm5c2fy/TxPnuc3956ZOZlf7p2Te+45x6AoigIiIiIijXi5ugJERETUsrDxQURERJpi44OIiIg0xcYHERERaYqNDyIiItIUGx9ERESkKTY+iIiISFNsfBAREZGm2PggIiIiTbHxQURERJpi48NOhw8fxuTJkxETE4M2bdogKioKjz32GPLz811dNQJQVVWFmTNnIjw8HH5+fkhISMDOnTtdXa0W7fjx4xg1ahS6deuG1q1bo2PHjhgyZAi2bNni6qq1eMyNfnn6d42Ba7vYJy0tDZ9//jlGjRqF/v37o6ioCEuXLsWVK1dw4MAB9O3b19VVbNEef/xxbNiwAdOmTUPPnj2xevVqHD58GLt378Y999zj6uq1SNu2bcOSJUuQmJiI8PBwXL16FR999BE+++wzLF++HOPHj3d1FVss5ka/PP27ho0PO+3btw8DBgyA0WgU206fPo1+/fohLS0Na9eudWHtWrZDhw4hISEBCxYswIwZMwAAlZWV6Nu3L4KDg7Fv3z4X15Cuq6urQ1xcHCorK5GXl+fq6pAKc6MPnv5dw24XOw0aNMjijwEAevbsiZiYGJw8edJFtSIA2LBhA7y9vS3+W/P19cXYsWOxf/9+FBQUuLB2pObt7Y3IyEiUlJS4uip0A+ZGHzz9u6aVqyvgCRRFQXFxMWJiYlxdlRbt6NGjiI6ORkBAgMX2gQMHAgByc3MRGRnpiqoRgIqKCly7dg2lpaXYvHkztm/fjtGjR7u6WgTmxl140ncNGx8O8MEHH+D8+fOYO3euq6vSohUWFiIsLOym7de3XbhwQesqkcr06dOxfPlyAICXlxdGjhyJpUuXurhWBDA37sKTvmvY+GimvLw8pKenIzExEWPGjHF1dVq0a9euwWQy3bTd19dX7CfXmTZtGtLS0nDhwgX8/e9/R11dHaqrq11dLQJz4w487buGN5w2Q1FREQYPHoyamhocOHAA4eHhrq5Si9a3b1+EhIQgKyvLYvuJEycQExODd955BxMmTHBR7ehGDzzwAEpKSnDw4EEYDAZXV4dUmBt98cTvGt5w2kSlpaUYOnQoSkpKsGPHDo/4Y3B3YWFhKCwsvGn79W3Mkb6kpaXh8OHDHjNvgSdhbvTDU79r2PhogsrKSgwfPhz5+fnYunUr+vTp4+oqEYDY2Fjk5+ejrKzMYvvBgwfFftKP691gpaWlLq4J3Yi50QdP/q5h48NOdXV1GD16NPbv34/169cjMTHR1VWiH6WlpaGurg7vvvuu2FZVVYVVq1YhISGBI11c5OLFizdtq6mpwfvvvw8/Pz+POqG6G+ZGvzz9u4Y3nNpp+vTp2Lx5M4YPH44ffvjhpolennzySRfVjBISEjBq1CjMmjULFy9eRI8ePbBmzRp88803WLFihaur12JNmDABZWVlGDJkCDp37oyioiJ88MEHyMvLw8KFC9G2bVtXV7HFYm70y9O/a3jDqZ2SkpKQnZ1tcz8/TteqrKzEyy+/jLVr1+Ly5cvo378/Xn31VaSkpLi6ai3WunXrsGLFCnz55Ze4dOkS/P39ERcXhylTpuDhhx92dfVaNOZGvzz9u4aNDyIiItIU7/kgIiIiTbHxQURERJpi44OIiIg0xcYHERERacppjY/MzEx06dIFvr6+SEhIwKFDh5z1VmQH5kW/mBv9Ym70iXlxY4oTrFu3TjEajcrKlSuV48ePK+PGjVOCgoKU4uJiZ7wdNRLzol/MjX4xN/rEvLg3pwy1TUhIQHx8vFiS2Ww2IzIyElOmTMELL7xwy+eazWZcuHAB/v7+XNDIgRRFQVJSEgYNGoTMzEwA9uXlennmxrEURUF5eTlSU1ObfMxcL8/cOJYjcsO8OAfPZ/p0/ZgJDw+Hl9etO1YcPsNpdXU1cnJyMGvWLLHNy8sLycnJ2L9//03lq6qqUFVVJR6fP3+eU/o6UXp6uohvlReAudGSt7d3o48ZgLnRkj25YV60xfOZPhUUFCAiIuKWZRze+Pj+++9RV1eHkJAQi+0hISHIy8u7qfz8+fMxZ86cm7bfg2FoBR9HV6/FqkA5DmMXbrvtNovttvICMDdaqEUN9mKbXccMwNxooSm5YV60wfOZPl0/Zvz9/Rss6/K1XWbNmoWMjAzxuKysDJGRkWgFH7Qy8A/CUVop9am25/Iic6OBJnZ6MjcaaEJumBdt8HymUz8eM43Ji8MbHx07doS3tzeKi4stthcXFyM0NPSm8iaTCSaTydHVoBv4oP4zvnEVS1t5AZgbLdlzzADMjZZ4PtMfns/cn8OH2hqNRsTFxSErK0tsM5vNyMrK8rglgd2J14+pVi9UxLzoR2xsLI8ZnWJu9IfnM/fnlG6XjIwMjBkzBgMGDMDAgQOxePFiVFRU4Ne//rUz3o7ssGbNGgwaNIh50Zn09HRMmjSJx4wOMTf6xfOZ+3JK42P06NH47rvvMHv2bBQVFSE2NhY7duy46aYt0t68efOYFx1KTU1FRUUFc6NDzI1+8Xzmvpwyz0dzlJWVITAwEEl4hDcBOVCtUoNPsQmlpaUICAho0mswN47niLwAzI0z8JjRL+ZGn+zJC9d2ISIiIk2x8UFERESaYuODiIiINOXySca0ogyOtXh8OdrParmfPvu5iHcuGWy1jH9BtYh9PslpfuWIWphWt0WK+NxoGac/tUnE4wILZJnaqyIe86ycKMpvE1cxJde49s+uIv5hV5iIB6cetSiXtecOEXeK+U7E5Z/KG2MjdpWL+Nv75eygkcu+FLG5XJbxBLzyQURERJpi44OIiIg05dHdLoYBfUX80Lu7LfZNafffBp//+mv/sbr971cCRTzny4dEXHtSDi3q8pL1lRXJNbz7RIs4b1I7EU//yTYRf3guXsRB42TXWm3Bt06uXctQOH2QiP859Q8iDvZu3eBzo1rJMjP+uFbEmZuirRX3SJfGyZk7581cabHvmX2/EHHER/K0zm4p50kMPivi15/9yHbBxz+3vr2/DD8aK89JqW0ui3jeE/I7bOPyJBEHL9vX+IrqFK98EBERkabY+CAiIiJNeXS3y9XO8lJtY7pZGuuxtqUyTvxAvl+CvFQf036yiLuvrxWx9+5/O6wedLNWkREi9l4rP/ffRv5VxHeazCL2UrW/n+73tYjnbI4T8RfD5WiM2m/PO66yLcDppQkyfnSpak/DXS22GA11zaiR+2qXXyni+/yuWOw7+ZPlIl4bL/9ed8/sJeIfquRnfja7i4i9r8nXMV2WE177F8jjx7T9cNMq7cE2bZbdiK8/3bzzurqrRe2ljsdEPP4F2YU2ovw3Ig76i3t28fPKBxEREWmKjQ8iIiLSlEd3u7Q9eUnEY8/dY7Fv95EYEfftJ7tkNvfc0eT3a+1lFPHZEe+K+PMH5WX+52dOkvVbf7DJ79XSeQfJEUcXfilzmfOCvLRvhryE7AWDaruX1e3qtvirwbkiTuk5QL4vu13sEhVd3GCZb1QTiLX3kjkI8PJ1Sp3clfGb7xtV7sTVcBGvuG2n9UI2BgmpuyEvm2U3z+kaOSnjhNxfijjidXn8KEdkF0FLcNscef5+6I17bJYz3NZZxOdeM1otM6OPzNN3tXKSsYx2p0WsHhW2e/6bIr639VQRd1zuPl0wvPJBREREmmLjg4iIiDTFxgcRERFpyqPv+ajL/0rERanhFvuiv8+V5doFiXho8M+tvpZi8hHx6anyYzOobhn448D1Ih7RRg6FG+wr23jvLlgs4jcyUkRcnFhm9X3Juqq7eoj44Auy/1N9P4cZZtUz5PbMku4iTg/6qsHy1HSFh+SCW38K7yni947JRRtD/ibvJ1i9eKGIA2yk4LcnRoi4E045oJbuwfydvIet356nLfbVVMpzUvc18l6n0XM6ifhvPbba9X6BqnvYBpjk8OachNWykFwHEPF/kvcehC10/xk4G2SWn4n56lXb5U7K+zY6j7ReZN2dPxWxV7Uc4rxqpPyO2DFOzgrcWXX/x4CxuSIu2NBexHWXfrBdJx3gGZaIiIg0xcYHERERacqju13Uas9fsL2vSDUcsKjhoYE9nrS+/Z07HxHx0dW5Ip7T6biIY4zyEvOMsH+JeOSC50QcvdJySF2d6rJdS6aevfThpfKz82rE0Nno7RNEHLlFbn9zuOxqaXVJdq3lPZkp4m+elpexu1uuT0gNUC+w+MlLcghh9ztqROz3VqEs38r6zKdbrspFG0N+IY9ls7XCHkp9ab/r41806jkFH8jF6PCKo2tkade0BSJ+7ItnRezzSY5z39gDKEfld4R6/t5IuRnDy54X8b+fl1MKLOssF6574G8jRHxlze0i1uMsqHZf+dizZw+GDx+O8PBwGAwGbNy40WK/oiiYPXs2wsLC4Ofnh+TkZJw+zS9PZ7usfIdc5XPsUbbiE2UDLiqW81EoP855ER0dzbxorKHcAMBrr73GY0ZjPGb0i7nxfHY3PioqKnDHHXcgMzPT6v4//OEPWLJkCd555x0cPHgQbdq0QUpKCiorK62WJ8eoQy3aIhC9cKfV/QU4AwBYtGgR86KxhnIDAMuXL+cxozEeM/rF3Hg+u7tdhg4diqFDh1rdpygKFi9ejJdeegmPPFLfBfH+++8jJCQEGzduxM9/bn0kiadQXzo7/LicdfPOpP8R8aYX5B3L/Y1tRXzmF2+LeOBxOQsqALRrRLdLR0MYOuLHkQWK5T5FUfAt6kd1/OxnP0NAQIBb5uW/T0SJeHygvM1ePUrl9o/kgn4nU+WlyahN1tvZ0U8fEfH5mXKhKPXsqPd2PyNi2513tt0yNz9umDFjhkceM94Bsruk+OfymHhq6jYRPxN01upzc6rlBehFzz0hYt/yQ9aK260lHDMd35WX2+P95WiUukFycczcu98Xsa1uS7XtV2X32ZmqUBFPbSePk7YvyysV1dmWs3oqNdVoSEvIjb0iNquu/jxvvcy/em8UcY/BE0Uc9BcnVaoZHHrD6dmzZ1FUVITk5GSxLTAwEAkJCdi/33qfU1VVFcrKyix+yLGuoQLVqLLY1lBeAOZGC5Wo78dPSkoS25gb1+Mxo1/MjWdwaOOjqKgIABASEmKxPSQkROy70fz58xEYGCh+IiMjrZajpquG9UuRt8oLwNxo4fpJNDg42GI7c+NaPGb0i7nxDC4f7TJr1ixkZGSIx2VlZR7xR1F3Il/Ewao4reI3Il41508iVo+CWfU7uR0AMlYnwhX0kBv1AnITfvUPERfXXRPxo7+Tn2nPlfI/n4efjRexLxq+VN/5DTkxktez8vLzXf7nRHwB7RpTbafTQ27UvDt2EPHJeXIStxlDtot4YuCndr3m43vHi7jHVsd0tTib3vKiFrFWdou8mC4X0LQ1uZ56+0vFA0V84hG5UFptwbcinnJe9o+s77FFxA/+5BmLehh3HLaz5o6h59y0RA5tfISG1vf/FRcXIyxMzmxYXFyM2NhYq88xmUwwmUyOrAbdwAjrq4PeKi8Ac6MFI+o/34sXLyI6Wi41yty4Fo8Z/WJuPINDu126du2K0NBQZGVliW1lZWU4ePAgEhNd8987AX5oI77krmNe9MEX9fNaZGdni23MjevxmNEv5sYz2H3l48qVKzhzRl6+O3v2LHJzc9G+fXtERUVh2rRpmDdvHnr27ImuXbvi5ZdfRnh4OEaMGOHIerutdmtkt8DH0+UwspiOec163VqlFtcg15O5hgqUKyXwgRG+htaIULrja5zAtm3bEBMT4zZ5yXu1l4g/DvpExHMuypEp7Vc6ZgKdS2PlicsMOTHS+CD5974V8bDXrXLTCvUTmy1YsAD9+vVzq2OmJjlOxA+9KXOzJWinQ15/SeJfRbzoPjnaxXv3vx3y+p56zNhyebUcpaJeq0XtXK3szhy2VnZndl9wQsR1Jd/CHgU/9bZ43H2HjYIqLS03LZHdjY8jR47gvvvuE4+v96GNGTMGq1evxvPPP4+KigqMHz8eJSUluOeee7Bjxw74+lq/VEaOUYYf8G/sEY9P4z8AgDDchhjEIxI98DVOYOrUqSgtLWVeNHSr3NyOWADAhAkTeMxojMeMfjE3ns/uxkdSUhIURbG532AwYO7cuZg7d26zKkb2aW8IRjLSbO43/Dh+//Tp0whQzb1Azner3NQq9dOMv/jii3jjjTe0rFaLx2NGv5gbz+fy0S4t2ee/uks+2Na8bhdPNf0ncjIq9Rou/1oql2TvAMd0u3RYIV/Ha27Dky21dOf/R04eZWuisOZ40E+uZfK312XXVzG79ZvkwfCTDZaZ/PA4EXf5Qh4P1jtpgMLpg1SPuIaLM11Y7NdwITfCsyoRERFpio0PIiIi0hS7XVzozPNGq9tHrMuweNzNQd0K7mh84DciziyRk1epu0icQb22i7lFLdzeeG1sLHRTq7pIn3J8lIhLN4aL2LtKfr7Pz/xQxKltLlt9zTnhqrVg7pBdA+YvGu5KoHpb3pJrTL0w5wurZRrzeXr37CbiP056r8HyQScMDZYh6648dreIl/Zb1mB59XnLu0Lf1xb0XTsiIiLyOGx8EBERkabY7aIB706dRHz+vY4izotfKeKNFXK4WNfN8i7/lk69xPfSXDm/THccdfh7nZ8p79z3gpzIanzB/apS5Q5/X3fV6Z0DIn7o/XutljFVfCPiYHxjtcxbl+US6KlvvW21TFSr1iLOmyQny4qeaK00WRPyDzkiKXPa7Q2WV5+3/vt0TxHnTn7Lankfg5xMrMfup0Xc3cldpJ5s4e8zRRxvarj76kyNXO23+/QDtyjperzyQURERJpi44OIiIg0xW4XDZxdFirikwP/IuKcqloRL576uIhN+1yz5LQeqe/eXpO4QsRzcZe14s1S0bPa6vt+9lUPETuju8dtqWY6NldUNPll2vwjV8SfLZCnpHt9a62UBmCwPcMy2VZbWCTif/aV3bzlP5cjKnrsl2u4xPp/JeL0dnK0ka2xX123TRBx7xfPidjWBGUkefnLrsT8t+X55k6T+rvAco2c68YXDBFx4a9CVHu+urmwjvDKBxEREWmKjQ8iIiLSFLtdmskrto+ICx4MEvGy8XJCmESTXPPgX1flqovzvnpUxH7b2NVijXq0y0CT4y+3ewcFijgjUS4Fr35f8yXrk8F5qstj5OIpUU+ftlmu9LeRIvb6rOndUSVpd4p4oOkz1R7rp6fAYz5Nfi9P1SpMdu2ar8guMENr1XogHYJE2G3Nf0X8WtgiEbc2yL91W5PrlZtl9+R9b/5GxL2WfyniunKOCrOHuXcXEefft0K1x3pXi9pXZXIEpSnfvq4Wr9atLR4r1TK3Sq2Nbk8H4ZUPIiIi0hQbH0RERKQpdrs0ksFkEnH+wlgRT79f3gWeHlRg9bnqrpbXpz4lYr9/sKulIc5eYyXv1V4i/jjoExEP+XK0iHu9JNe78NQ7971j5KRT82e/K+Ik3xqbzxnzhpx87YdhsvuqrqS0wferSEsQ8ZJ5ctIqk8H6KWljRZCIw1bKdUm46k69n+zMF/EPtW1EPDQgW8QDTLb+ehv+GlB3tdy/WHa1hC/cJ2Lmwj6G+H4iHrVm5y1K3lrrqbIb0laGvW+XI2i++pWcPC4svtCinM/cIBF77c1tcp0ag1c+iIiISFNsfBAREZGm2O0CyzvFr/WLEPGji1SjHwzyomJ60MEGX7PbernoxO0rSkTc5tJ5ESt9okVcd0JeNiVJPerEUW3lVpEyx6dHyrVEzKrXbztbXrquK/vaIe+rZ1d6yG6TW3W1qK25bZeI7xn2jIgD1snuRK/+sjsnf4bsujxx31IRt7JxR/8BuUwF5v/hFyLuUMG1Qm60bHuKiE8+IT9bR3VV+hjksVEVxEneHOHMY21F/FTABbueOyzvYRFXxLQXcV2snDDu6dkbRRzqkyfiB/1srx32IP7Xrno0h11n8/nz5yM+Ph7+/v4IDg7GiBEjcOrUKYsylZWVSE9PR4cOHdC2bVukpqaiuLjYoZWmm51V8nBIycJuZSOylS34QtmHCuXm4W7Tp09nbjTEvOgXc6NfzI3ns6vxkZ2djfT0dBw4cAA7d+5ETU0NHnjgAVSoplZ+7rnnsGXLFqxfvx7Z2dm4cOECRo4c6fCKk6USfIcIdEc87sNduBdmmHEUn6FOsRyrvWPHDuZGQ8yLfjE3+sXceD67ul127Nhh8Xj16tUIDg5GTk4OhgwZgtLSUqxYsQIffvgh7r+//k74VatWoXfv3jhw4ADuvvtuay/rNAUvySXS63xtXyrsfa+8rP6Pnn92yHuH335RxF8/Jufbv/M+uX14R9l98/7tcsKmprjTYLmkeYwSjz3YgjJcRjt0Qi3qL6W/9tprushNYzlqtIu6q+WOzXLdCfXr374hXcQ9DzlmOWp3yYt3lfwcqlQneFujT270x3lyUr3tz98h4jmd1tp6R6tby8yVIp6UOUPEYX/eZ614s7hLbhqj+29kV9Sg3nKdqLYm2XdleENORuXziZz4UBkcK8t8niti9Zov//rjYhEf+1/ZrdPHOFnE3WY6rjvMk3Jjyz8fW6B6JCf78lZ1cdUp1s9523ptlg/etO99ryly5FJafqrFPuMZOfrFuVOMNbMTvbS0fkhd+/b1fU45OTmoqalBcnKyKNOrVy9ERUVh/37rf5hVVVUoKyuz+KHmu35w+qB+xsJylAAAkpKSRBnmRnuOyAvA3DgDjxn9Ym48T5MbH2azGdOmTcPgwYPRt29fAEBRURGMRiOCgoIsyoaEhKCoqMjKq9TfRxIYGCh+IiObdwWAAEVRkI9cBKID2hrqbySsRv1/QMyN6zgqLwBz42g8ZvSLufFMTR7tkp6ejmPHjmHv3r3NqsCsWbOQkZEhHpeVlTXrj+LrN+S6FMeelNejTAZt14PY2///RJzfW94T8+DuZ0X83eyuIm4FeRm0ufJwFFdQhgFIatbrODo3TeFjkJfna+y8yf7aIwNFPHr+dhGPD/xGxNHb5TLg0VMd09Vii6PyAjg+N8YdcoTK1qtyEqLUNpcb9fy7Taq40xdWy9i6nLyiTHaJvbPkERGHve34rhZbPOmYaf+QrZFz/7W6Vd3Voua/Th4Pd/V/TsRfjlki4mNPynjEX8eI2Jx7ohE1bRxPyk1j2OpqaY6sa/IA/f3EcSJWd78Bzu9qUWtS42Py5MnYunUr9uzZg4gIeeIIDQ1FdXU1SkpKLFqkxcXFCA0NtfJKgMlkgkk1eyg1T55yFN+jEAOQBF+D7Ec0ov4zLikpQUBAgNjO3GjDkXkBmBtH4jGjX8yN57Kr20VRFEyePBkff/wxdu3aha5du1rsj4uLg4+PD7KyssS2U6dO4dy5c0hMTLzx5ciBFEVBnnIU3+E84jAEfoY2Fvv9EQSgfsTSdcyN8zEv+sXc6Bdz4/nsuvKRnp6ODz/8EJs2bYK/v7/oWwsMDISfnx8CAwMxduxYZGRkoH379ggICMCUKVOQmJjoFncfu7NTOIoiFOAODII3fFCl1I8aaAUfeBu80Qr13U4vvvgiIiIimBuNMC/6xdzoF3Pj+exqfLz9dv1skOo7jIH6IU5PPfUUAGDRokXw8vJCamoqqqqqkJKSgmXLlkErxh7yDmZH3udRpchZH2P32TcLXNfJcuKbnsWOu7dD7VvUDxfOQbbF9j4YgHB0EY9TUlJclpumqFHkUknqobb5fx4g4j6/k8PDCpbKS7Cv9Fkn4odV9y6oX0f9XGf0d7pjXn6/6AkRx8z8k8W+Xj5Nv2xdar4m4tjNU0V8+3PyHpFOVdrNXuqOudGSkiiHTPdItH6/iNq5YUEijsht3nu3hNz8bI1coO/42Eynvte0XLlQZuQnzvkOspddjQ9FafiOP19fX2RmZiIz07kfJllKNqQ1qtzChQvx3nvvObk2dB3zol/MjX4xN56PC8sRERGRplrswnJ935QLYbUuaviKjmpdOUT9xb5Lw3UNFyEb4n8/RcSHX3hLxGeGviti81CZP/VCdOrZS1+5eKeIP58t+4R9vz3kuMp6iE5vy7/vjJzxFvu+HikXw3pi6B4Rz+74pYjVM6TGbJYzYPZefEnE0fnyc+cyZfphiIsRcdnsKyL+NHpTg89tXchM2qPrBtkVPDFFzuh6YMMd1orj8V/KgRwzO5wU8Vsl3US8YvUwq8/tsvq0iPXyfcQrH0RERKQpNj6IiIhIUx7X7RKRelzEKYi1Wa4ztJs9kZoufM0xEQ9UZBfMoVmyC0Y9eiWnSranf71Glu+2Qt6tz64WOxz60uJhN9VHd+AFOZpsGO6y+vRoyCfo5XIvWbr4jFyAc9VvFom4t7Hh/037/kXO2NxtlXYjlTyB+T95Ij6XILeH2/huyl7gJ2Mbx5ut5+rx2OOVDyIiItIUGx9ERESkKY/rdiHPUqda9jp4qbyk+NDSuAafG6W6BKnlgklE7iR4mTxOZi5LuEXJm3UDu1qoaXjlg4iIiDTFxgcRERFpio0PIiIi0hQbH0RERKQpNj6IiIhIU2x8EBERkabY+CAiIiJN6W6eD0WpXxmxFjVc7tKBalEDQH6+TcHcOJ4j8qJ+PnPjODxm9Iu50Sd78qK7xkd5eTkAYC+2ubgmnqm8vByBgYFNfi7A3DhDc/Jy/fkAc+MMPGb0i7nRp8bkxaA0918uBzObzbhw4QIURUFUVBQKCgoQEBDg6mppoqysDJGRkU75nRVFQXl5OcLDw+Hl1bTeNrPZjFOnTqFPnz4tKi+A83LjiLwALTc37nDM8Hym39zwmHFdXnR35cPLywsREREo+3Fa7YCAgBbzR3Gds37n5vxnDdTnpnPnzgBaZl4A5/zezc0LwNzo+Zjh+Uy/ueEx47q88IZTIiIi0hQbH0RERKQp3TY+TCYTXnnlFZhMJldXRTPu8Du7Qx2dwR1+b3eoo6O5y+/sLvV0JHf4nd2hjo6ml99ZdzecEhERkWfT7ZUPIiIi8kxsfBAREZGm2PggIiIiTbHxQURERJpi44OIiIg0pcvGR2ZmJrp06QJfX18kJCTg0KFDrq6Sw8yfPx/x8fHw9/dHcHAwRowYgVOnTlmUqaysRHp6Ojp06IC2bdsiNTUVxcXFLqqxJeaGudEa86JfzI1+6T43is6sW7dOMRqNysqVK5Xjx48r48aNU4KCgpTi4mJXV80hUlJSlFWrVinHjh1TcnNzlWHDhilRUVHKlStXRJmJEycqkZGRSlZWlnLkyBHl7rvvVgYNGuTCWtdjbpgbV2Be9Iu50S+950Z3jY+BAwcq6enp4nFdXZ0SHh6uzJ8/34W1cp6LFy8qAJTs7GxFURSlpKRE8fHxUdavXy/KnDx5UgGg7N+/31XVVBSFuWFu9IF50S/mRr/0lhtddbtUV1cjJycHycnJYpuXlxeSk5Oxf/9+F9bMeUpLSwEA7du3BwDk5OSgpqbG4jPo1asXoqKiXPoZMDfMjV4wL/rF3OiX3nKjq8bH999/j7q6OoSEhFhsDwkJQVFRkYtq5TxmsxnTpk3D4MGD0bdvXwBAUVERjEYjgoKCLMq6+jNgbpgbPWBe9Iu50S895qaV09+BbEpPT8exY8ewd+9eV1eFbsDc6BPzol/MjX7pMTe6uvLRsWNHeHt733S3bXFxMUJDQ11UK+eYPHkytm7dit27dyMiIkJsDw0NRXV1NUpKSizKu/ozYG6YG1djXvSLudEvveZGV40Po9GIuLg4ZGVliW1msxlZWVlITEx0Yc0cR1EUTJ48GR9//DF27dqFrl27WuyPi4uDj4+PxWdw6tQpnDt3zqWfAXPD3LgK86JfzI1+6T43Tr+l1U7r1q1TTCaTsnr1auXEiRPK+PHjlaCgIKWoqMjVVXOISZMmKYGBgcqnn36qFBYWip+rV6+KMhMnTlSioqKUXbt2KUeOHFESExOVxMREF9a6HnPD3LgC86JfzI1+6T03umt8KIqivPXWW0pUVJRiNBqVgQMHKgcOHHB1lRwGgNWfVatWiTLXrl1TnnnmGaVdu3ZK69atlUcffVQpLCx0XaVVmBvmRmvMi34xN/ql99wYfqwkERERkSZ0dc8HEREReT42PoiIiEhTbHwQERGRptj4ICIiIk2x8UFERESaYuODiIiINMXGBxEREWmKjQ8iIiLSFBsfREREpCk2PoiIiEhTbHwQERGRpv4fHGQykVsdgAkAAAAASUVORK5CYII=\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "# ------------------------------------\n",
        "for num in range(0,5):\n",
        "    one_data = next(iter(train_loader))\n",
        "    plt.subplot(1,5,num+1)\n",
        "    plt.imshow(one_data[0][0][0,:,:])\n",
        "    plt.title(one_data[1][0].item())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bo6-KX8UyZCy"
      },
      "source": [
        "## Network setting\n",
        "\n",
        "While image classification, is usually performed using Convolutional Neural Networks, we will use here a Multi-Layer-Perceptron (also named ```fully-connected```).\n",
        "For this, we will flatten (i.e. convert the iamge 2D matrix to a 1D vector) the input images.\n",
        "\n",
        "- The input images are of size 28*28 and are converted to vectors of size ```n_in```=784.\n",
        "\n",
        "- The two hidden layers have````n_h1````=500 and ```n_h2```=256 unit/neurons each and ```Relu```activations.\n",
        "\n",
        "- The output are the ```n_out```=10 classes (the 10 digits to be recognized).\n",
        "\n",
        "Since we deal with a muti-class problem (10 classes), the output activations is therefore a ```softmax```.\n",
        "\n",
        "Note that since the ```softmax```output will be given to a ```cross-entropy```, the ```log```et the latter can be directly added to the ```softmax```. This leads to the pytorch function ```F.log_softmax```.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H0uIIQb9yZCy"
      },
      "outputs": [],
      "source": [
        "class Net(nn.Module):\n",
        "    def __init__(self):\n",
        "        n_in = 28*28\n",
        "        n_h1 = 500\n",
        "        n_h2 = 256\n",
        "        n_out = 10\n",
        "        super(Net, self).__init__()\n",
        "        # --- START CODE HERE\n",
        "        self.fc1 = nn.Linear(n_in, n_h1)\n",
        "        self.fc2 = nn.Linear(n_h1, n_h2)\n",
        "        self.fc3 = nn.Linear(n_h2, n_out)\n",
        "        # --- END CODE HERE\n",
        "    def forward(self, x):\n",
        "        # --- START CODE HERE\n",
        "        x = self.fc3(F.relu(self.fc2(F.relu(self.fc1(x.view(-1, 28*28))))))\n",
        "        # --- END CODE HERE\n",
        "        return F.log_softmax(x, dim=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "x7qcgKFwyZCz"
      },
      "source": [
        "## Defining the train and test functions\n",
        "\n",
        "In the present lab, we distinguish between train and test data.\n",
        "We therefore write\n",
        "- a train function (to learn the network parameters on the train data),\n",
        "- a test function (to evaluate the network's performances on the test data).\n",
        "\n",
        "The ```train``` and ```test```function will perform the computation for a given ```epoch```.\n",
        "The functions therefore only need to ```loop over``` the mini-batches.\n",
        "This is now easy since we defined our ```Dataloader```for train and test.\n",
        "\n",
        "For the train function, for each mini-batch, we  \n",
        "- compute the forward pass by passing the data to the model: haty = model(x)\n",
        "- compute the the loss (the criterion)\n",
        "- putting at zero the gradients of all the parameters of the network (this is important since, by default, pytorch accumulate the gradients over time)\n",
        "- computing the backpropagation (using as before .backward())\n",
        "- performing one step of optimization (using .step())\n",
        "\n",
        "For the train function, we set the model to train mode (```model.train()```). This allows to set some specific behaviours for training (such as activate the dropout).\n",
        "\n",
        "For the test function, we set the model to train mode (```model.eval()```). This allows to set some specific behaviours for training (such as removing the dropout).\n",
        "\n",
        "\n",
        "Since we are dealing with a multi-class problem (10 classes), we will minimize as loss the ```cross-entropy``` (named negative-log-likelihood ```nll_loss```in pytorch).\n",
        "Since ```nll_loss```does not contains the ```log``` we have added it directly to the softmax by using the ```log_softmax```as output activation of our network.\n",
        "\n",
        "Note that we could use the loss from the nn package (```torch.nn.NLLLoss```) it is then a class which needs to be first instanciated.\n",
        "We can also use the same but as a function (```F.nll_loss()```) which can be used directly without instanciating the class."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3dGuPt96yZCz"
      },
      "outputs": [],
      "source": [
        "def train(model, train_loader, optimizer, epoch):\n",
        "    model.train()\n",
        "    for batch_idx, (data, target) in enumerate(train_loader):\n",
        "        # --- START CODE HERE\n",
        "        optimizer.zero_grad()\n",
        "        output = model(data)\n",
        "        loss = F.nll_loss(output, target)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "        # --- END CODE HERE\n",
        "\n",
        "        if batch_idx % 100 == 0:\n",
        "            print('epoch {}, loss {}'.format(epoch, loss.item()))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zI8QtzV2yZC0"
      },
      "outputs": [],
      "source": [
        "def test(model, test_loader):\n",
        "    model.eval()\n",
        "    test_loss = 0\n",
        "    correct = 0\n",
        "    with torch.no_grad():\n",
        "        for data, target in test_loader:\n",
        "            # --- START CODE HERE\n",
        "            output = model(data)\n",
        "            test_loss += F.nll_loss(output, target, reduction='sum').item()\n",
        "            pred = output.argmax(dim=1, keepdim=True)\n",
        "            correct += pred.eq(target.view_as(pred)).sum().item()\n",
        "            # --- STOP CODE HERE\n",
        "\n",
        "    test_loss /= len(test_loader.dataset)\n",
        "    correct /= len(test_loader.dataset)\n",
        "    print('\\nTest set: Average loss: {}, Accuracy: {}\\n'.format(test_loss, correct))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9MXQIbcJyZC0"
      },
      "source": [
        "## Looping over epochs\n",
        "\n",
        "\n",
        "We finally define, how we will optimize our parameters by defining an optimizer.\n",
        "We will use here a very simple one: the SGD (```optim.SGD```) with a learning rate of 0.01.\n",
        "\n",
        "At each iteration, we then call the train and the test function."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ofYcX05ZyZC0",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9922ef54-cee7-45b0-a026-8abb3e4fc489"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "epoch 1, loss 2.330418825149536\n",
            "epoch 1, loss 0.7129853963851929\n",
            "epoch 1, loss 0.47872835397720337\n",
            "epoch 1, loss 0.20551152527332306\n",
            "epoch 1, loss 0.25926628708839417\n",
            "epoch 1, loss 0.21244096755981445\n",
            "epoch 1, loss 0.193203404545784\n",
            "epoch 1, loss 0.1196436807513237\n",
            "epoch 1, loss 0.14615444839000702\n",
            "epoch 1, loss 0.1626826971769333\n",
            "\n",
            "Test set: Average loss: 0.17659655505120755, Accuracy: 0.9476\n",
            "\n",
            "epoch 2, loss 0.2104191780090332\n",
            "epoch 2, loss 0.46984541416168213\n",
            "epoch 2, loss 0.15836305916309357\n",
            "epoch 2, loss 0.14481483399868011\n",
            "epoch 2, loss 0.09653306752443314\n",
            "epoch 2, loss 0.23504862189292908\n",
            "epoch 2, loss 0.27321621775627136\n",
            "epoch 2, loss 0.1926669180393219\n",
            "epoch 2, loss 0.1447644978761673\n",
            "epoch 2, loss 0.1369144767522812\n",
            "\n",
            "Test set: Average loss: 0.11371979680880905, Accuracy: 0.9645\n",
            "\n",
            "epoch 3, loss 0.1305980235338211\n",
            "epoch 3, loss 0.28861626982688904\n",
            "epoch 3, loss 0.06336481124162674\n",
            "epoch 3, loss 0.04569612443447113\n",
            "epoch 3, loss 0.02325832098722458\n",
            "epoch 3, loss 0.028504999354481697\n",
            "epoch 3, loss 0.1301739364862442\n",
            "epoch 3, loss 0.12193935364484787\n",
            "epoch 3, loss 0.10334882140159607\n",
            "epoch 3, loss 0.05559484660625458\n",
            "\n",
            "Test set: Average loss: 0.09783120120614767, Accuracy: 0.9714\n",
            "\n",
            "epoch 4, loss 0.06695770472288132\n",
            "epoch 4, loss 0.05316079780459404\n",
            "epoch 4, loss 0.09876632690429688\n",
            "epoch 4, loss 0.14202669262886047\n",
            "epoch 4, loss 0.05546392500400543\n",
            "epoch 4, loss 0.07981416583061218\n",
            "epoch 4, loss 0.08284562826156616\n",
            "epoch 4, loss 0.014550426974892616\n",
            "epoch 4, loss 0.14528986811637878\n",
            "epoch 4, loss 0.09564005583524704\n",
            "\n",
            "Test set: Average loss: 0.11375958805400878, Accuracy: 0.9641\n",
            "\n",
            "epoch 5, loss 0.024953218176960945\n",
            "epoch 5, loss 0.02963925339281559\n",
            "epoch 5, loss 0.1143077090382576\n",
            "epoch 5, loss 0.07820191979408264\n",
            "epoch 5, loss 0.00486468942835927\n",
            "epoch 5, loss 0.050286564975976944\n",
            "epoch 5, loss 0.08597496896982193\n",
            "epoch 5, loss 0.0393160842359066\n",
            "epoch 5, loss 0.0476682111620903\n",
            "epoch 5, loss 0.08880358189344406\n",
            "\n",
            "Test set: Average loss: 0.0795729123183526, Accuracy: 0.9737\n",
            "\n",
            "epoch 6, loss 0.007288909517228603\n",
            "epoch 6, loss 0.04419029504060745\n",
            "epoch 6, loss 0.07474154233932495\n",
            "epoch 6, loss 0.05272499471902847\n",
            "epoch 6, loss 0.01773441955447197\n",
            "epoch 6, loss 0.1480395793914795\n",
            "epoch 6, loss 0.03697076067328453\n",
            "epoch 6, loss 0.04991431534290314\n",
            "epoch 6, loss 0.06037521734833717\n",
            "epoch 6, loss 0.1322825700044632\n",
            "\n",
            "Test set: Average loss: 0.07027522012246773, Accuracy: 0.9772\n",
            "\n",
            "epoch 7, loss 0.11125492304563522\n",
            "epoch 7, loss 0.03970026969909668\n",
            "epoch 7, loss 0.09950043261051178\n",
            "epoch 7, loss 0.05332077294588089\n",
            "epoch 7, loss 0.030030639842152596\n",
            "epoch 7, loss 0.023029664531350136\n",
            "epoch 7, loss 0.0369420051574707\n",
            "epoch 7, loss 0.11011134833097458\n",
            "epoch 7, loss 0.04814409837126732\n",
            "epoch 7, loss 0.009668558835983276\n",
            "\n",
            "Test set: Average loss: 0.07243044229373335, Accuracy: 0.9775\n",
            "\n",
            "epoch 8, loss 0.052699219435453415\n",
            "epoch 8, loss 0.027510881423950195\n",
            "epoch 8, loss 0.05820038542151451\n",
            "epoch 8, loss 0.046886373311281204\n",
            "epoch 8, loss 0.06508824229240417\n",
            "epoch 8, loss 0.007292310241609812\n",
            "epoch 8, loss 0.053694214671850204\n",
            "epoch 8, loss 0.048347439616918564\n",
            "epoch 8, loss 0.036164768040180206\n",
            "epoch 8, loss 0.007122025825083256\n",
            "\n",
            "Test set: Average loss: 0.07651011487003416, Accuracy: 0.9749\n",
            "\n",
            "epoch 9, loss 0.03002133034169674\n",
            "epoch 9, loss 0.025357674807310104\n",
            "epoch 9, loss 0.016104435548186302\n",
            "epoch 9, loss 0.013228025287389755\n",
            "epoch 9, loss 0.024324610829353333\n",
            "epoch 9, loss 0.015322617255151272\n",
            "epoch 9, loss 0.013575563207268715\n",
            "epoch 9, loss 0.019707847386598587\n",
            "epoch 9, loss 0.07084708660840988\n",
            "epoch 9, loss 0.08340459316968918\n",
            "\n",
            "Test set: Average loss: 0.06581364430298563, Accuracy: 0.9805\n",
            "\n",
            "epoch 10, loss 0.034517839550971985\n",
            "epoch 10, loss 0.008823007345199585\n",
            "epoch 10, loss 0.04403822124004364\n",
            "epoch 10, loss 0.012472919188439846\n",
            "epoch 10, loss 0.04710007086396217\n",
            "epoch 10, loss 0.04858027771115303\n",
            "epoch 10, loss 0.008604472503066063\n",
            "epoch 10, loss 0.06415381282567978\n",
            "epoch 10, loss 0.009609373286366463\n",
            "epoch 10, loss 0.06042081117630005\n",
            "\n",
            "Test set: Average loss: 0.06283481874247082, Accuracy: 0.9805\n",
            "\n"
          ]
        }
      ],
      "source": [
        "model = Net()\n",
        "optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)\n",
        "nb_epoch = 10\n",
        "for epoch in range(1, nb_epoch + 1):\n",
        "        # --- START CODE HERE\n",
        "        train(model, train_loader, optimizer, epoch)\n",
        "        test(model, test_loader)\n",
        "        # --- END CODE HERE"
      ]
    }
  ],
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.5"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {
        "height": "calc(100% - 180px)",
        "left": "10px",
        "top": "150px",
        "width": "165px"
      },
      "toc_section_display": true,
      "toc_window_display": false
    },
    "colab": {
      "provenance": []
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}