PyTorch_Transfer_Learning_Exercise.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.3"
    },
    "colab": {
      "name": "PyTorch_Transfer_Learning_Exercise.ipynb",
      "provenance": [],
      "toc_visible": true,
      "include_colab_link": true
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/ravi-gopalan/dog_breed_classifier_udacity/blob/master/PyTorch_Transfer_Learning_Exercise.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "D_YNPcJyrj0G",
        "colab_type": "text"
      },
      "source": [
        "# Transfer Learning\n",
        "\n",
        "In this notebook, you'll learn how to use pre-trained networks to solved challenging problems in computer vision. Specifically, you'll use networks trained on [ImageNet](http://www.image-net.org/) [available from torchvision](http://pytorch.org/docs/0.3.0/torchvision/models.html). \n",
        "\n",
        "ImageNet is a massive dataset with over 1 million labeled images in 1000 categories. It's used to train deep neural networks using an architecture called convolutional layers. I'm not going to get into the details of convolutional networks here, but if you want to learn more about them, please [watch this](https://www.youtube.com/watch?v=2-Ol7ZB0MmU).\n",
        "\n",
        "Once trained, these models work astonishingly well as feature detectors for images they weren't trained on. Using a pre-trained network on images not in the training set is called transfer learning. Here we'll use transfer learning to train a network that can classify our cat and dog photos with near perfect accuracy.\n",
        "\n",
        "With `torchvision.models` you can download these pre-trained networks and use them in your applications. We'll include `models` in our imports now."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZWY7LZODrofd",
        "colab_type": "code",
        "outputId": "31da2214-a2f0-4a18-d004-4fb3c7fecf26",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 124
        }
      },
      "source": [
        "from google.colab import drive\n",
        "drive.mount('/content/gdrive')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=email%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdocs.test%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.photos.readonly%20https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpeopleapi.readonly&response_type=code\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "09mpRtGetlKn",
        "colab_type": "code",
        "outputId": "3803900e-2e30-4f7d-8e8d-1d5850c05371",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "cd /root"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/root\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "lp-M69L1tsmO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!mkdir .kaggle"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OSpvP6MDtymq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!cp '/content/gdrive/My Drive/Colab_stuff/kaggle.json' /root/.kaggle/kaggle.json\n",
        "!chmod 600 /root/.kaggle/kaggle.json"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AcYdn2D9xOe3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!ls"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TkbWfQgBuCaX",
        "colab_type": "code",
        "outputId": "47fea0b6-8b60-4c0e-a692-75f7bfbac6c8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "cd /content"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "/content\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X4bx7Sdht1N3",
        "colab_type": "code",
        "outputId": "7555593a-4b0d-4b4a-a80d-246df5f4257e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 191
        }
      },
      "source": [
        "!kaggle competitions download -c dogs-vs-cats"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Warning: Looks like you're using an outdated API Version, please consider updating (server 1.5.6 / client 1.5.4)\n",
            "Downloading sampleSubmission.csv to /content\n",
            "  0% 0.00/86.8k [00:00<?, ?B/s]\n",
            "100% 86.8k/86.8k [00:00<00:00, 35.6MB/s]\n",
            "Downloading test1.zip to /content\n",
            " 95% 257M/271M [00:06<00:00, 35.9MB/s]\n",
            "100% 271M/271M [00:06<00:00, 44.5MB/s]\n",
            "Downloading train.zip to /content\n",
            "100% 541M/543M [00:13<00:00, 22.7MB/s]\n",
            "100% 543M/543M [00:13<00:00, 42.0MB/s]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MCbj_2TevLWQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!unzip -q /content/train.zip"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RRqdHz4wvaQp",
        "colab_type": "code",
        "outputId": "0cd0e592-0eb5-47f2-946e-6442935edeb3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "!ls"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "gdrive\tsample_data  sampleSubmission.csv  test1.zip  train  train.zip\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YhNS8uwM9XmS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# organize dataset into a useful structure\n",
        "from os import makedirs\n",
        "from os import listdir\n",
        "from shutil import copyfile\n",
        "from random import seed\n",
        "from random import random\n",
        "# create directories\n",
        "dataset_home = '/content/data_home/'\n",
        "subdirs = ['train/', 'test/']\n",
        "for subdir in subdirs:\n",
        "\t# create label subdirectories\n",
        "\tlabeldirs = ['dogs/', 'cats/']\n",
        "\tfor labldir in labeldirs:\n",
        "\t\tnewdir = dataset_home + subdir + labldir\n",
        "\t\tmakedirs(newdir, exist_ok=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "v5IXhu32oiU7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "0c8bdd21-5e93-4810-a1dc-36d6a64c8d20"
      },
      "source": [
        "!ls"
      ],
      "execution_count": 16,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "gdrive\tsample_data  sampleSubmission.csv  test1.zip  train  train.zip\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vOYgSR4A9r08",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# seed random number generator\n",
        "seed(1)\n",
        "# define ratio of pictures to use for validation\n",
        "val_ratio = 0.25\n",
        "# copy training dataset images into subdirectories\n",
        "src_directory = '/content/train/'\n",
        "for file in listdir(src_directory):\n",
        "\tsrc = src_directory + '/' + file\n",
        "\tdst_dir = 'train/'\n",
        "\tif random() < val_ratio:\n",
        "\t\tdst_dir = 'test/'\n",
        "\tif file.startswith('cat'):\n",
        "\t\tdst = dataset_home + dst_dir + 'cats/'  + file\n",
        "\t\tcopyfile(src, dst)\n",
        "\telif file.startswith('dog'):\n",
        "\t\tdst = dataset_home + dst_dir + 'dogs/'  + file\n",
        "\t\tcopyfile(src, dst)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tNMZJ0ju6sv3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 86
        },
        "outputId": "29cbe6c1-4d07-41e5-fb63-db2c44e55b18"
      },
      "source": [
        "base_dir = '/content'\n",
        "train_cat_dir = '/content/data_home/train/cats'\n",
        "train_dog_dir = '/content/data_home/train/dogs'\n",
        "test_cat_dir = '/content/data_home/test/cats'\n",
        "test_dog_dir = '/content/data_home/test/dogs'\n",
        "\n",
        "print(len(listdir(train_cat_dir)))\n",
        "print(len(listdir(train_dog_dir)))\n",
        "\n",
        "print(len(listdir(test_cat_dir)))\n",
        "print(len(listdir(test_dog_dir)))\n"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "9354\n",
            "9343\n",
            "3146\n",
            "3157\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hz_--kkTrj0I",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "%matplotlib inline\n",
        "%config InlineBackend.figure_format = 'retina'\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "\n",
        "import torch\n",
        "from torch import nn\n",
        "from torch import optim\n",
        "import torch.nn.functional as F\n",
        "from torchvision import datasets, transforms, models"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PCCDtYR6rj0P",
        "colab_type": "text"
      },
      "source": [
        "Most of the pretrained models require the input to be 224x224 images. Also, we'll need to match the normalization used when the models were trained. Each color channel was normalized separately, the means are `[0.485, 0.456, 0.406]` and the standard deviations are `[0.229, 0.224, 0.225]`."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ivt6YNl2rj0R",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# TODO: Define transforms for the training data and testing data\n",
        "train_transforms = transforms.Compose([transforms.RandomRotation(30),\n",
        "                                       transforms.RandomResizedCrop(224),\n",
        "                                       transforms.RandomHorizontalFlip(),\n",
        "                                       transforms.ToTensor(),\n",
        "                                       transforms.Normalize([0.485, 0.456, 0.406],\n",
        "                                                            [0.229, 0.224, 0.225])])\n",
        "\n",
        "test_transforms = transforms.Compose([transforms.Resize(255),\n",
        "                                      transforms.CenterCrop(224),\n",
        "                                      transforms.ToTensor(),\n",
        "                                      transforms.Normalize([0.485, 0.456, 0.406],\n",
        "                                                            [0.229, 0.224, 0.225])])\n",
        "\n",
        "# Pass transforms in here, then run the next cell to see how the transforms look\n",
        "train_data = datasets.ImageFolder(dataset_home + '/train', transform=train_transforms)\n",
        "test_data = datasets.ImageFolder(dataset_home + '/test', transform=test_transforms)\n",
        "\n",
        "trainloader = torch.utils.data.DataLoader(train_data, batch_size=64, shuffle=True)\n",
        "testloader = torch.utils.data.DataLoader(test_data, batch_size=64)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4lPVXQwIqFlo",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "b7c0ab34-6ff0-4895-c987-e5f436fe8cfd"
      },
      "source": [
        "type(trainloader.dataset.imgs[10000][0])"
      ],
      "execution_count": 33,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "str"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 33
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BPU4wAi_rj0W",
        "colab_type": "text"
      },
      "source": [
        "We can load in a model such as [DenseNet](http://pytorch.org/docs/0.3.0/torchvision/models.html#id5). Let's print out the model architecture so we can see what's going on."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "scrolled": true,
        "id": "7zkAZ95Qrj0X",
        "colab_type": "code",
        "outputId": "5aecdfce-2377-4c60-b51a-929f002d5064",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "model = models.densenet121(pretrained=True)\n",
        "model"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Downloading: \"https://download.pytorch.org/models/densenet121-a639ec97.pth\" to /root/.cache/torch/checkpoints/densenet121-a639ec97.pth\n",
            "100%|██████████| 32342954/32342954 [00:01<00:00, 31837791.52it/s]\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "DenseNet(\n",
              "  (features): Sequential(\n",
              "    (conv0): Conv2d(3, 64, kernel_size=(7, 7), stride=(2, 2), padding=(3, 3), bias=False)\n",
              "    (norm0): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "    (relu0): ReLU(inplace)\n",
              "    (pool0): MaxPool2d(kernel_size=3, stride=2, padding=1, dilation=1, ceil_mode=False)\n",
              "    (denseblock1): _DenseBlock(\n",
              "      (denselayer1): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(64, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer2): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(96, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(96, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer3): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer4): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(160, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer5): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(192, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer6): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(224, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(224, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (transition1): _Transition(\n",
              "      (norm): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace)\n",
              "      (conv): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)\n",
              "    )\n",
              "    (denseblock2): _DenseBlock(\n",
              "      (denselayer1): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(128, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer2): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(160, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(160, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer3): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(192, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(192, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer4): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(224, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(224, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer5): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer6): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(288, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer7): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer8): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(352, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(352, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer9): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(384, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer10): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(416, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer11): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(448, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(448, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer12): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(480, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (transition2): _Transition(\n",
              "      (norm): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace)\n",
              "      (conv): Conv2d(512, 256, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)\n",
              "    )\n",
              "    (denseblock3): _DenseBlock(\n",
              "      (denselayer1): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(256, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(256, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer2): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(288, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(288, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer3): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(320, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(320, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer4): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(352, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(352, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer5): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(384, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(384, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer6): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(416, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(416, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer7): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(448, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(448, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer8): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(480, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(480, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer9): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer10): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(544, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(544, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer11): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer12): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(608, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer13): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(640, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(640, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer14): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(672, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer15): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(704, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(704, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer16): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(736, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(736, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer17): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer18): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(800, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer19): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(832, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer20): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(864, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(864, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer21): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(896, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(896, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer22): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(928, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(928, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer23): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(960, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer24): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(992, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(992, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (transition3): _Transition(\n",
              "      (norm): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "      (relu): ReLU(inplace)\n",
              "      (conv): Conv2d(1024, 512, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "      (pool): AvgPool2d(kernel_size=2, stride=2, padding=0)\n",
              "    )\n",
              "    (denseblock4): _DenseBlock(\n",
              "      (denselayer1): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(512, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(512, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer2): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(544, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(544, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer3): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(576, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(576, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer4): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(608, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(608, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer5): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(640, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(640, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer6): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(672, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(672, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer7): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(704, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(704, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer8): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(736, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(736, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer9): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(768, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer10): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(800, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(800, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer11): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(832, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(832, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer12): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(864, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(864, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer13): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(896, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(896, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer14): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(928, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(928, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer15): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(960, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(960, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "      (denselayer16): _DenseLayer(\n",
              "        (norm1): BatchNorm2d(992, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu1): ReLU(inplace)\n",
              "        (conv1): Conv2d(992, 128, kernel_size=(1, 1), stride=(1, 1), bias=False)\n",
              "        (norm2): BatchNorm2d(128, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "        (relu2): ReLU(inplace)\n",
              "        (conv2): Conv2d(128, 32, kernel_size=(3, 3), stride=(1, 1), padding=(1, 1), bias=False)\n",
              "      )\n",
              "    )\n",
              "    (norm5): BatchNorm2d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)\n",
              "  )\n",
              "  (classifier): Linear(in_features=1024, out_features=1000, bias=True)\n",
              ")"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rhmpgXDLrj0d",
        "colab_type": "text"
      },
      "source": [
        "This model is built out of two main parts, the features and the classifier. The features part is a stack of convolutional layers and overall works as a feature detector that can be fed into a classifier. The classifier part is a single fully-connected layer `(classifier): Linear(in_features=1024, out_features=1000)`. This layer was trained on the ImageNet dataset, so it won't work for our specific problem. That means we need to replace the classifier, but the features will work perfectly on their own. In general, I think about pre-trained networks as amazingly good feature detectors that can be used as the input for simple feed-forward classifiers."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xyMBAjJCrj0e",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Freeze parameters so we don't backprop through them\n",
        "for param in model.parameters():\n",
        "    param.requires_grad = False\n",
        "\n",
        "from collections import OrderedDict\n",
        "classifier = nn.Sequential(OrderedDict([\n",
        "                          ('fc1', nn.Linear(1024, 500)),\n",
        "                          ('relu', nn.ReLU()),\n",
        "                          ('fc2', nn.Linear(500, 2)),\n",
        "                          ('output', nn.LogSoftmax(dim=1))\n",
        "                          ]))\n",
        "    \n",
        "model.classifier = classifier"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QzGIxtyGrj0i",
        "colab_type": "text"
      },
      "source": [
        "With our model built, we need to train the classifier. However, now we're using a **really deep** neural network. If you try to train this on a CPU like normal, it will take a long, long time. Instead, we're going to use the GPU to do the calculations. The linear algebra computations are done in parallel on the GPU leading to 100x increased training speeds. It's also possible to train on multiple GPUs, further decreasing training time.\n",
        "\n",
        "PyTorch, along with pretty much every other deep learning framework, uses [CUDA](https://developer.nvidia.com/cuda-zone) to efficiently compute the forward and backwards passes on the GPU. In PyTorch, you move your model parameters and other tensors to the GPU memory using `model.to('cuda')`. You can move them back from the GPU with `model.to('cpu')` which you'll commonly do when you need to operate on the network output outside of PyTorch. As a demonstration of the increased speed, I'll compare how long it takes to perform a forward and backward pass with and without a GPU."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-06iJRpWrj0j",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import time"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qnmMlMYurj0l",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "for device in ['cpu', 'cuda']:\n",
        "\n",
        "    criterion = nn.NLLLoss()\n",
        "    # Only train the classifier parameters, feature parameters are frozen\n",
        "    optimizer = optim.Adam(model.classifier.parameters(), lr=0.001)\n",
        "\n",
        "    model.to(device)\n",
        "\n",
        "    for ii, (inputs, labels) in enumerate(trainloader):\n",
        "\n",
        "        # Move input and label tensors to the GPU\n",
        "        inputs, labels = inputs.to(device), labels.to(device)\n",
        "\n",
        "        start = time.time()\n",
        "\n",
        "        outputs = model.forward(inputs)\n",
        "        loss = criterion(outputs, labels)\n",
        "        loss.backward()\n",
        "        optimizer.step()\n",
        "\n",
        "        if ii==3:\n",
        "            break\n",
        "        \n",
        "    print(f\"Device = {device}; Time per batch: {(time.time() - start)/3:.3f} seconds\")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cImTTH6Qrj0o",
        "colab_type": "text"
      },
      "source": [
        "You can write device agnostic code which will automatically use CUDA if it's enabled like so:\n",
        "```python\n",
        "# at beginning of the script\n",
        "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "...\n",
        "\n",
        "# then whenever you get a new Tensor or Module\n",
        "# this won't copy if they are already on the desired device\n",
        "input = data.to(device)\n",
        "model = MyModule(...).to(device)\n",
        "```\n",
        "\n",
        "From here, I'll let you finish training the model. The process is the same as before except now your model is much more powerful. You should get better than 95% accuracy easily.\n",
        "\n",
        ">**Exercise:** Train a pretrained models to classify the cat and dog images. Continue with the DenseNet model, or try ResNet, it's also a good model to try out first. Make sure you are only training the classifier and the parameters for the features part are frozen."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "78EuW-DVrj0p",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "## TODO: Use a pretrained model to classify the cat and dog images"
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}