Object_Detection_E2E_TF2.0.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/anshupandey/Computer-Vision/blob/master/Object_Detection_E2E_TF2.0.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c2zIuU2I70fY"
      },
      "source": [
        "#### 1. Install TensorFlow Object Detection API\n",
        "- For latest install instructions, check the [github](https://github.com/tensorflow/models/tree/master/research/object_detection) page or [readthedocs](https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html) site for this API."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yj_ujSb4C_mB"
      },
      "outputs": [],
      "source": [
        "#Clone githun repository\n",
        "!git clone https://github.com/tensorflow/models"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3sFSNT21DpnL"
      },
      "outputs": [],
      "source": [
        "#Upgrade pip package\n",
        "!pip install --upgrade pip"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tah1nb6bDHcZ"
      },
      "source": [
        "Install TensorFlow Object Detection API"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bBVJ65JPCnpJ"
      },
      "outputs": [],
      "source": [
        "%%bash\n",
        "cd models/research\n",
        "# Compile protos.\n",
        "protoc object_detection/protos/*.proto --python_out=.\n",
        "# Install TensorFlow Object Detection API.\n",
        "cp object_detection/packages/tf2/setup.py .\n",
        "python -m pip install --use-feature=2020-resolver ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "rTX7CyFED2zZ"
      },
      "outputs": [],
      "source": [
        "#Check if Object Detection API is installed\n",
        "!pip list | grep object-detection"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vx8_pjTsGorG"
      },
      "source": [
        "#### 2. Download dataset"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "34udzBTrJCMs"
      },
      "source": [
        "Create a directory for the project (e.g detection) and move to project directory."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sUGWdmsuIcz5"
      },
      "outputs": [],
      "source": [
        "#Come to the home directory\n",
        "%cd /content"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZkuVzkeK2zj1"
      },
      "outputs": [],
      "source": [
        "#Create a folder for your project e.g in this case, detection. You can choose any name for the folder \n",
        "!mkdir detection\n",
        "\n",
        "#Go to the project folder\n",
        "%cd detection"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6HQKojVNk1SR"
      },
      "outputs": [],
      "source": [
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mUHZZqXaJGoe"
      },
      "source": [
        "Download dataset : Information on dataset is available [here](http://host.robots.ox.ac.uk/pascal/VOC/). We are downloading Pascal VOC 2007 dataset here (VOC 2012 is also available)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HpHu3zFnGsG0"
      },
      "outputs": [],
      "source": [
        "#Get PASCAL VOC dataset\n",
        "!wget http://pjreddie.com/media/files/VOCtrainval_06-Nov-2007.tar --quiet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zpG0cpt3654S"
      },
      "outputs": [],
      "source": [
        "#Check the project folder\n",
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "B_bAxET0JMnO"
      },
      "source": [
        "Extract files from the tar file"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jvUMs49VI266"
      },
      "outputs": [],
      "source": [
        "!tar -xf VOCtrainval_06-Nov-2007.tar"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FbTDKJTKJoQC"
      },
      "outputs": [],
      "source": [
        "#Check the current folder for extracted data\n",
        "!ls -l"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "vALZ5plf7jF9"
      },
      "outputs": [],
      "source": [
        "!ls -l VOCdevkit"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QEb7XN6H7nOz"
      },
      "outputs": [],
      "source": [
        "!ls -l VOCdevkit/VOC2007"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "G-AV6P4pL5d7"
      },
      "outputs": [],
      "source": [
        "!ls -l VOCdevkit/VOC2007/Annotations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f91nND2h7y26"
      },
      "source": [
        "JPEGImages folder has actual images and 'Annotations' folder has Class labels and bounding box information for each image."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3m_taPbM9wSx"
      },
      "outputs": [],
      "source": [
        "!ls -l VOCdevkit/VOC2007/JPEGImages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "h8eLYfwPKSVf"
      },
      "outputs": [],
      "source": [
        "#Number of images\n",
        "!ls -l VOCdevkit/VOC2007/JPEGImages| wc -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3G9MOpFoJV1k"
      },
      "source": [
        "Review image annotation in XML"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uo-quvQx8KDu"
      },
      "outputs": [],
      "source": [
        "#Number of XML files - it should one for each image\n",
        "!ls -l VOCdevkit/VOC2007/Annotations | wc -l"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "NORj-5vEKVh3"
      },
      "outputs": [],
      "source": [
        "#Lets check\n",
        "!cat VOCdevkit/VOC2007/Annotations/008422.xml"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CTTo8kvtJvDg"
      },
      "source": [
        "Set images folder and XMLs folder"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DFBOWStNJ1sK"
      },
      "outputs": [],
      "source": [
        "img_path = 'VOCdevkit/VOC2007/JPEGImages'\n",
        "xml_path = 'VOCdevkit/VOC2007/Annotations'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MY5yXLB-K9Dl"
      },
      "source": [
        "#### 3. Data Pre-processing : Convert XML to CSV"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LSw6T4rjJt4N"
      },
      "source": [
        "Use xml_to_csv.py file provided. This script will read all XML files and save the information in a CSV file.\n",
        " Here we will download the file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "hc-4ZmtG8i3g"
      },
      "outputs": [],
      "source": [
        "# Download xml to csv\n",
        "!wget -q https://raw.githubusercontent.com/anshupandey/Computer-Vision/master/object%20detection/TF2.0%20OD%20API%20-%20PASCAL%20VOC/xml_to_csv.py"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "DbpWk4c5PDyE"
      },
      "outputs": [],
      "source": [
        "#We should have 'xml_to_csv.py' script in our current folder\n",
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "o3nrBOyXKTUO"
      },
      "source": [
        "Generate CSV file from all XML files using copied script. The script requires two parameters\n",
        "1. -i <xml_files_folder_name> : indicating XML files are stored\n",
        "2. -o <output_file_path> : indicating what name should be given to CSV output file and where it should be stored."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "X6cANEVgNhKc"
      },
      "outputs": [],
      "source": [
        "#Running the script with options\n",
        "!python xml_to_csv.py -i {xml_path} -o detection_data.csv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "mmv5Yhi7KGl-"
      },
      "outputs": [],
      "source": [
        "#We should have CSV file in current folder\n",
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "l10k7pYNKWOg"
      },
      "source": [
        "Load csv file as dataframe"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FsY8C01fRDjR"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "df = pd.read_csv('detection_data.csv')\n",
        "df.head(5)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FS4qN4dA-dnG"
      },
      "outputs": [],
      "source": [
        "#How many object across all images\n",
        "df.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uhWYQeTlRTht"
      },
      "outputs": [],
      "source": [
        "#List of labels\n",
        "df['class'].unique()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fazU9o5iRYt7"
      },
      "outputs": [],
      "source": [
        "#Number of labels\n",
        "len(df['class'].unique())"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FbCMi1ssUCVa"
      },
      "source": [
        "We will need to Label encode classes e.g assign a unique index number for each class"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "shN8PgyTUFFv"
      },
      "outputs": [],
      "source": [
        "#Use Label encoder available in Scikit Learn\n",
        "from sklearn import preprocessing"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1KfwMfHoUGKd"
      },
      "outputs": [],
      "source": [
        "#Label Encode class and add a 'label' column to the dataframe\n",
        "le = preprocessing.LabelEncoder()\n",
        "df['label'] = le.fit_transform(df['class'])\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HcQQgy3DUc7a"
      },
      "outputs": [],
      "source": [
        "#unique values in the label\n",
        "df.label.unique()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dHD8fUDRaBi-"
      },
      "outputs": [],
      "source": [
        "#Object detection API expects index to start from 1 (and not 0)\n",
        "df['label'] = df['label'] + 1\n",
        "df.label.unique()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jZiy8isyCCZU"
      },
      "outputs": [],
      "source": [
        "#Dataframe should have label column now\n",
        "df.head()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "wna0FoqMZw2h"
      },
      "outputs": [],
      "source": [
        "#Create a dictionary of Label and Class. This will be useful for building our second input to Model training\n",
        "label_class_dict = dict(zip(df['label'], df['class']))\n",
        "print(label_class_dict)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HyfANFw0Krpy"
      },
      "source": [
        "**Split data between training and test**\n",
        "\n",
        "First we have to split images between training and test. Then we can use that information to split dataframe between training and test. This will make sure objects from same image are not split between training and test."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HlgaYomMES2H"
      },
      "outputs": [],
      "source": [
        "#Get information on all images\n",
        "all_classes = df['filename'].unique()\n",
        "all_classes.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "evGM8W8tEzKA"
      },
      "outputs": [],
      "source": [
        "#Split images between training and test\n",
        "import numpy as np\n",
        "\n",
        "#80% of the data will be used for training\n",
        "mask = np.random.rand(all_classes.shape[0]) < 0.8\n",
        "\n",
        "#Get Training and Test images\n",
        "train_images = all_classes[mask]\n",
        "test_images = all_classes[~mask] "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EPiuv_H8FO21"
      },
      "outputs": [],
      "source": [
        "#Check number of images in training and test\n",
        "train_images.shape, test_images.shape"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "PyE474ukPQkA"
      },
      "outputs": [],
      "source": [
        "train_images[:10]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "y48z0ExeLBUR"
      },
      "outputs": [],
      "source": [
        "#Split dataframe between training and test\n",
        "train_df = df[df['filename'].isin(train_images)]\n",
        "test_df = df[df['filename'].isin(test_images)]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "SrGU4aqFLQlB"
      },
      "outputs": [],
      "source": [
        "train_df.shape, test_df.shape"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4NJq-pBELXlX"
      },
      "source": [
        "**Visualizing the data**"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4i2iym_4OEaF"
      },
      "outputs": [],
      "source": [
        "#We will use opencv and matplotlib\n",
        "from matplotlib import pyplot as plt\n",
        "import cv2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "q5ANh3SOLTzZ"
      },
      "outputs": [],
      "source": [
        "#Pickup a random image number\n",
        "img_num = np.random.randint(0, df.shape[0])\n",
        "\n",
        "#Read the image\n",
        "img_file = df.loc[img_num,'filename']\n",
        "img = cv2.imread(img_path + '/' + img_file)\n",
        "\n",
        "#Find all rows which have same file name\n",
        "rows_with_file = df[df['filename'] == img_file].index.tolist()\n",
        "\n",
        "#Draw rectangle(s) as per bounding box information\n",
        "for i in rows_with_file:\n",
        "\n",
        "    #Get bounding box\n",
        "    xmin, ymin, xmax, ymax = df.loc[i, ['xmin', 'ymin', 'xmax', 'ymax']]\n",
        "    #Get Label\n",
        "    label = df.loc[i, 'class']\n",
        "    #Add bounding box\n",
        "    cv2.rectangle(img, (xmin,ymin), (xmax, ymax), (0,255,0), 2)\n",
        "    #Add text\n",
        "    cv2.putText(img,label,(xmin, ymin-5),cv2.FONT_HERSHEY_SIMPLEX, 0.7, (255, 255, 255), 2)\n",
        "\n",
        "#Convert BGR format (used by opencv to RGB format used by matplotlib)\n",
        "img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
        "\n",
        "#Draw image using matplotlib\n",
        "plt.figure(figsize=(10,7))\n",
        "plt.imshow(img)\n",
        "plt.show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p5_wHuLqVxb3"
      },
      "source": [
        "Save training and test data as csv"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "A5mKamauV0Je"
      },
      "outputs": [],
      "source": [
        "train_df.to_csv('train.csv', index=False)\n",
        "test_df.to_csv('test.csv', index=False)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "lue3NVQtc8vV"
      },
      "outputs": [],
      "source": [
        "#We should have training and test csv files in current directory\n",
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sgYaNAGEQThJ"
      },
      "source": [
        "#### 4. Generate tfrecord from CSV\n",
        "Tensorflow object detection API requires data in tfrecord format. This can be done using generate_tfrecord.py file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AAy-ZUcJQwSy"
      },
      "outputs": [],
      "source": [
        "# Downloading generate_tfrecord.py\n",
        "!wget -q https://raw.githubusercontent.com/anshupandey/Computer-Vision/master/object%20detection/TF2.0%20OD%20API%20-%20PASCAL%20VOC/generate_tfrecord.py"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "myfindIvSFE-"
      },
      "outputs": [],
      "source": [
        "#Make sure the script file is now available\n",
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OR1-LBNXHIwx"
      },
      "source": [
        "The script file requires 3 inputs\n",
        "\n",
        "1. --csv_input=<csv_file_path> : where is csv file located which was prepared in previous step\n",
        "2. --img_path=<images_folder> : where are the actual images stored\n",
        "3. --output_path=<output_file_path> : where the script can save the generated tfrecord file and what should be file name.\n",
        "\n",
        "We will run script for training and test csv separately to create two tfrecord files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9AhiyJ3BRZVv"
      },
      "outputs": [],
      "source": [
        "#generate tfrecord for training data\n",
        "!python generate_tfrecord.py --csv_input=train.csv  --img_path={img_path} --output_path=train.tfrecord"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "UI6E0ogZYDr7"
      },
      "outputs": [],
      "source": [
        "#generate tfrecord for test data\n",
        "!python generate_tfrecord.py --csv_input=test.csv  --img_path={img_path} --output_path=test.tfrecord"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Vpy-h2voYIs_"
      },
      "outputs": [],
      "source": [
        "#train.record and test.record files should be available now\n",
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2xCyipZzYZr_"
      },
      "source": [
        "#### 5. Create Label Mapping File"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TV2dytODhW0X"
      },
      "outputs": [],
      "source": [
        "#Dict which was created earlier will be used for building Label Mapping file\n",
        "label_class_dict"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XxomQ0LyYeB5"
      },
      "outputs": [],
      "source": [
        "#Build a pbtxt label file using label and class name\n",
        "#This is required by Object detection API\n",
        "#You can prepare it manually as well.\n",
        "\n",
        "pbtxt_file_txt = ''\n",
        "for label in sorted(label_class_dict.keys()):\n",
        "    \n",
        "    pbtxt_file_txt += \"item {\\n  id: \" + str(label) + \"\\n  name: '\" +  label_class_dict[label] + \"'\\n}\\n\\n\"\n",
        "\n",
        "with open('label_map.txt','w') as pbfile:\n",
        "    pbfile.write(pbtxt_file_txt)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ApGs7F-Lcf4B"
      },
      "outputs": [],
      "source": [
        "!ls -l"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZM7b5TnvcK9_"
      },
      "outputs": [],
      "source": [
        "#Review the file content\n",
        "!cat label_map.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kjxG56jeeFXk"
      },
      "source": [
        "#### 6. Download a pre-trained model\n",
        "\n",
        "A list of pre-trained models is available at [TensorFlow model zoo](https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/tf1_detection_zoo.md). We will use 'ssd_mobilenet_v1_coco' model for transfer learning."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZNE5_ol7eJCE"
      },
      "outputs": [],
      "source": [
        "#Download the model from zoo\n",
        "!wget -q http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_320x320_coco17_tpu-8.tar.gz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7ICDRMr_e2iR"
      },
      "outputs": [],
      "source": [
        "!ls -l"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cqmqI0fGiI5D"
      },
      "outputs": [],
      "source": [
        "#Extract tar file content\n",
        "!tar -xf ssd_mobilenet_v2_320x320_coco17_tpu-8.tar.gz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YfcAjfqnjbfd"
      },
      "outputs": [],
      "source": [
        "!ls -l"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "oCBC5Xlajf5t"
      },
      "outputs": [],
      "source": [
        "#Check the extracted folder\n",
        "!ls -l ssd_mobilenet_v2_320x320_coco17_tpu-8"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-2Wff6vPV6nw"
      },
      "outputs": [],
      "source": [
        "#Check the extracted folder\n",
        "!ls -l ssd_mobilenet_v2_320x320_coco17_tpu-8/checkpoint"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Yfu0k9k0e-CV"
      },
      "source": [
        "#### 7. Prepare Training configuration file\n",
        "\n",
        "1. Change num_classes parameter to 20 (as we have 20 categories in pascal voc dataset)\n",
        "2. For 'train_input_reader' change 'input_path' to filepath of train.record file.\n",
        "3. For 'train_input_reader' change 'label_map_path' to filepath of pascal_voc.pbtxt file.\n",
        "4. Repeat above two steps for 'eval_input_reader'.\n",
        "5. Change fine_tune_checkpoint to filepath where pre-trained model.ckpt file is available e.g ssd_mobilenet_v1_coco_2018_01_28/model.ckpt\n",
        "6. Change 'batch_size' accordingly to available memory.\n",
        "7. Change 'num_steps' to indicate how long the training will done e.g. 200000. For demo purpose, we are keeping it to 20 so that we can finish training quickly.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Zd2oF4RATqCe"
      },
      "source": [
        "You can copy a sample configuration for the chosen pre-trained model (SSD MobileNet v2 320x320) in this case from [Configs](https://github.com/tensorflow/models/tree/master/research/object_detection/configs/tf2) folder. Here are things which need to be changed at a minimum."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0dH298sAfFVq"
      },
      "outputs": [],
      "source": [
        "# get config file\n",
        "!wget -q https://raw.githubusercontent.com/anshupandey/Computer-Vision/master/object%20detection/TF2.0%20OD%20API%20-%20PASCAL%20VOC/ssd_mobilenet_v2_modified.config"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RUFO3V4jZqjn"
      },
      "outputs": [],
      "source": [
        "!cat  ssd_mobilenet_v2_modified.config"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z5qSjVS2NNFV"
      },
      "source": [
        "Set Config file name"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fuPDh-FcNMJZ"
      },
      "outputs": [],
      "source": [
        "config_file = 'ssd_mobilenet_v2_modified.config'"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MbCourM9jMGd"
      },
      "source": [
        "#### 8. Training the model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Z1k412IGjP_d"
      },
      "outputs": [],
      "source": [
        "#Copy training file from 'models/research/object_detection' folder to current folder\n",
        "!cp /content/models/research/object_detection/model_main_tf2.py ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_B82kA4CjkWM"
      },
      "outputs": [],
      "source": [
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-cNtnArWwKiZ"
      },
      "source": [
        "Start training \n",
        "\n",
        "- Please note that Object detection take long time to train. The training may take few days if run on single GPU machine (depending on num of steps indicated). Try to keep training the model till loss comes close to 1 (or goes below 1). The script takes 3 inputs\n",
        "\n",
        "1. --train_dir=<folder_name> : where model will be saved periodically as training progresses\n",
        "2. --pipeline_config_path=<config_file_path> :where is model training configuration file located."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "YYSZ3lU8jrlq"
      },
      "outputs": [],
      "source": [
        "#Create a training folder to store model checkpoints/snapshots as training progresses\n",
        "!mkdir training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "RQLMnw-B43BR"
      },
      "outputs": [],
      "source": [
        "#Check training folder\n",
        "!ls -l training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qWU775b3cYsL"
      },
      "source": [
        "Start Tensorboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "EEc2xP2fbCCQ"
      },
      "outputs": [],
      "source": [
        "%load_ext tensorboard"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H9tm1l5BasDb"
      },
      "outputs": [],
      "source": [
        "%tensorboard --logdir training"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6sNQzqpxjnB4"
      },
      "outputs": [],
      "source": [
        "#start training\n",
        "!python model_main_tf2.py --model_dir=training/ --pipeline_config_path={config_file} --checkpoint_every_n=100 --alsologtostderr"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R1bNNQ2pzxW8"
      },
      "source": [
        "Model Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aiiARwj32t9y"
      },
      "outputs": [],
      "source": [
        "#start Evaluation - THIS WILL KEEP RUNNING.\n",
        "!python model_main_tf2.py --model_dir=training/ --pipeline_config_path={config_file} --checkpoint_dir=training/ --alsologtostderr "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ea3CVJqnGWg7"
      },
      "source": [
        "#### Training and Evaluation in Background"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ust7kYHQ0mh6"
      },
      "source": [
        "If we want to evaluate our model on training data regularly, we have to run both training and evaluation script in parallel. Model evaluation on test data gets done everytime model checkpoint is saved during training."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qYh6I60xCWLX"
      },
      "outputs": [],
      "source": [
        "#Start training\n",
        "!nohup /usr/bin/python3 model_main_tf2.py --model_dir=training/ --pipeline_config_path={config_file} --checkpoint_every_n=500 --alsologtostderr --eval_training_data --sample_1_of_n_eval_on_train_examples=10  > train.txt 2>&1 &"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "dczrQx0OC4Ki"
      },
      "outputs": [],
      "source": [
        "!nohup /usr/bin/python3 model_main_tf2.py --model_dir=training/ --pipeline_config_path={config_file} --checkpoint_dir=training/ --alsologtostderr > eval.txt 2>&1 &"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "uc444JgNDYzu"
      },
      "outputs": [],
      "source": [
        "!cat train.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "JvLi43NUDGX3"
      },
      "outputs": [],
      "source": [
        "!cat eval.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aznLkMtLoGuF"
      },
      "outputs": [],
      "source": [
        "#Check the training folder\n",
        "!ls -l training"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uxbue-P4kEvi"
      },
      "source": [
        "#### 9. Export trained model\n",
        "\n",
        "From the saved model checkpoints, we will create a frozen trained model. Frozen here means to remove model nodes which are no longer needed in prediction. This reduces model size."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "P7MVmgSQkLov"
      },
      "outputs": [],
      "source": [
        "#Copy export_inference_graph.py file from models/research/object_detection to current directory\n",
        "!cp /content/models/research/object_detection/exporter_main_v2.py ."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_59ZQxnmkay5"
      },
      "outputs": [],
      "source": [
        "!ls -l"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "45hd528oMlpy"
      },
      "source": [
        "The export_inference_graph.py script file requires the following input:\n",
        "\n",
        "1. --input_type <input_node_name> : This will be used during prediction to set model input\n",
        "2. --pipeline_config_path <model_training_config_file_path> : where is model training config file located.\n",
        "3. --trained_checkpoint_prefix <file_path__model_checkpoint> : Which checkpoint should be used to create final model.\n",
        "4. --output_directory <frozen_model_directory> : where should the frozen model created by script should be stored."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bwKWqRcwkHz7"
      },
      "outputs": [],
      "source": [
        "#Provide input name, config file location, training folder\n",
        "!python exporter_main_v2.py --input_type \"image_tensor\" --pipeline_config_path {config_file} --trained_checkpoint_dir training/ --output_directory detection_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zw8HOKCloQgp"
      },
      "outputs": [],
      "source": [
        "#Check if model is saved in current directory\n",
        "!ls -l detection_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "CIlaUfPifCDD"
      },
      "outputs": [],
      "source": [
        "#Check if model is saved in current directory\n",
        "!ls -l detection_model/saved_model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "OdQK1xwqL__e"
      },
      "outputs": [],
      "source": [
        "!ls -l detection_model/saved_model/variables/"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "VkSf2y-vMEN_"
      },
      "outputs": [],
      "source": [
        "!ls -l detection_model/saved_model/assets"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5G2ScbsvHrYT"
      },
      "source": [
        "#### Move the trained model to Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "TtyngLYIHvL_"
      },
      "outputs": [],
      "source": [
        "!cp -r detection_model '/gdrive/My Drive/AI-ML/models/pascal_voc_tf2/' "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7nuGe4twJ0iT"
      },
      "source": [
        "Save label dictionary as well for model prediction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L6kF6U09J5dh"
      },
      "outputs": [],
      "source": [
        "print(label_class_dict)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6D5PhdlGJ9cG"
      },
      "outputs": [],
      "source": [
        "import pickle\n",
        "\n",
        "label_file_path = '/gdrive/My Drive/AI-ML/models/pascal_voc_tf2/pascal_voc_label.pkl'\n",
        "\n",
        "with open(label_file_path,'wb') as file:\n",
        "    pickle.dump(label_class_dict, file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FtZnLR1ckdbS"
      },
      "source": [
        "#### Model Prediction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Sk6AdwT7kfPE"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9W9CfkGn9viy"
      },
      "source": [
        "Load Saved model"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "sGcnbCBGklP2"
      },
      "outputs": [],
      "source": [
        "saved_model_path = 'detection_model/saved_model'\n",
        "model = tf.saved_model.load(saved_model_path)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2vhAv2id1xk9"
      },
      "source": [
        "Function to get model prediction"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "W0i6SCaplWqP"
      },
      "outputs": [],
      "source": [
        "#Function to get predictions from a Detection model\n",
        "def detector_prediction(image_file, confidence_threshold=0.5):\n",
        "\n",
        "    \"\"\"\n",
        "    image_file: File path of the image for which prediction needs to be done\n",
        "    confidence_threshold: Minimum confidence/probability for prediction to be considered\n",
        "    \"\"\"\n",
        "    #Load image\n",
        "    img = tf.keras.preprocessing.image.load_img(image_file)\n",
        "    \n",
        "    #Convert to numpy array\n",
        "    img_array = tf.keras.preprocessing.image.img_to_array(img).astype('uint8')\n",
        "    #Make it a batch of one example\n",
        "    img_array = tf.expand_dims(img_array, axis=0)\n",
        "\n",
        "    #Prediction\n",
        "    output = model(img_array) #get list of tensors discussed above as output\n",
        "    #print(output)\n",
        "    detection_scores = output['detection_scores'].numpy()[0] #get detection scores\n",
        "    detection_classes = output['detection_classes'].numpy()[0]\n",
        "    detection_boxes = output['detection_boxes'].numpy()[0]\n",
        "\n",
        "    #Select predictions for which probability is higher than confidence_threshold\n",
        "    selected_predictions = detection_scores >= confidence_threshold\n",
        "\n",
        "    selected_prediction_scores = detection_scores[selected_predictions]\n",
        "    selected_prediction_classes = detection_classes[selected_predictions]\n",
        "    selected_prediction_boxes = detection_boxes[selected_predictions]\n",
        "\n",
        "    #De-normalize box co-ordinates (multiply x-coordinates by image width and y-coords by image height)\n",
        "    img_w, img_h = img.size\n",
        "\n",
        "    for i in range(selected_prediction_boxes.shape[0]):\n",
        "        \n",
        "        selected_prediction_boxes[i,0] *= img_h #ymin * img_w\n",
        "        selected_prediction_boxes[i,1] *= img_w #xmin * img_h\n",
        "        selected_prediction_boxes[i,2] *= img_h #ymax * img_w\n",
        "        selected_prediction_boxes[i,3] *= img_w #xmax * img_h\n",
        "\n",
        "    #Make all co-ordinates as integer\n",
        "    selected_prediction_boxes= selected_prediction_boxes.astype(int)\n",
        "\n",
        "    #Convert class indexes to actual class labels\n",
        "    predicted_classes = []\n",
        "    for i in range(selected_prediction_classes.shape[0]):\n",
        "        predicted_classes.append(label_class_dict[int(selected_prediction_classes[i])])\n",
        "\n",
        "    #Number of predictions\n",
        "    selected_num_predictions = selected_prediction_boxes.shape[0]\n",
        "\n",
        "    return {'Total Predictions': selected_num_predictions,\n",
        "            'Scores': selected_prediction_scores, \n",
        "            'Classes': predicted_classes, \n",
        "            'Box coordinates': selected_prediction_boxes}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ULqXIGX512ri"
      },
      "source": [
        "Copy an image to predict on"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZdGHNALJ1pKo"
      },
      "outputs": [],
      "source": [
        "!wget -q https://raw.githubusercontent.com/anshupandey/Computer-Vision/master/object%20detection/TF2.0%20OD%20API%20-%20PASCAL%20VOC/person_with_bike.jpg"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4jDTBVhc150o"
      },
      "outputs": [],
      "source": [
        "!ls -l"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "O7MEewOAmLUb"
      },
      "outputs": [],
      "source": [
        "#Model output\n",
        "detector_prediction('person_with_bike.jpg', confidence_threshold=0.7)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XvZqpVaF1Tpo"
      },
      "source": [
        "Visualize model output"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "03U3uDUO1NDA"
      },
      "outputs": [],
      "source": [
        "import cv2\n",
        "from matplotlib import pyplot as plt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "gmInmPGz1WsW"
      },
      "outputs": [],
      "source": [
        "def visualize_output(image_file, confidence_threshold=0.5):\n",
        "\n",
        "    #Call model prediction function above\n",
        "    output = detector_prediction(image_file, confidence_threshold=confidence_threshold)\n",
        "\n",
        "    #Read image\n",
        "    img = cv2.imread(image_file)\n",
        "\n",
        "    #Draw rectangle for predicted boxes, also add predicted classes\n",
        "    for i in range(output['Box coordinates'].shape[0]):\n",
        "\n",
        "        box = output['Box coordinates'][i]\n",
        "        \n",
        "        #Draw rectangle - (ymin, xmin, ymax, xmax)\n",
        "        img = cv2.rectangle(img, (box[1], box[0]), (box[3], box[2]), (0,255,0), 2)\n",
        "        \n",
        "        #Add Label - Class name and confidence level\n",
        "        label = output['Classes'][i] + ': ' + str(round(output['Scores'][i],2))\n",
        "        img = cv2.putText(img, label, (box[1], box[0]-10), cv2.FONT_HERSHEY_SIMPLEX, 1, (255,255,255), 2, cv2.LINE_AA)\n",
        "    \n",
        "    #Conver BGR image to RGB to use with Matplotlib\n",
        "    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n",
        "\n",
        "    #Display image\n",
        "    plt.figure(figsize=(10,6))\n",
        "    plt.imshow(img)\n",
        "    plt.show()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QKTH6Zys1YQ2"
      },
      "outputs": [],
      "source": [
        "#Visualize on image\n",
        "visualize_output('person_with_bike.jpg', confidence_threshold=0.7)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        ""
      ],
      "metadata": {
        "id": "65_i1qIUb3Vh"
      },
      "execution_count": null,
      "outputs": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "name": "1. Object_Detection_E2E_TF2.ipynb",
      "private_outputs": true,
      "provenance": [],
      "machine_shape": "hm",
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.4"
    },
    "latex_envs": {
      "LaTeX_envs_menu_present": true,
      "autoclose": false,
      "autocomplete": true,
      "bibliofile": "biblio.bib",
      "cite_by": "apalike",
      "current_citInitial": 1,
      "eqLabelWithNumbers": true,
      "eqNumInitial": 1,
      "hotkeys": {
        "equation": "Ctrl-E",
        "itemize": "Ctrl-I"
      },
      "labels_anchors": false,
      "latex_user_defs": false,
      "report_style_numbering": false,
      "user_envs_cfg": false
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}