8.3 to 8.10 - Building a CNN for handwritten digits - MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Our First CNN in Keras \n",
    "### Creating a model based on the MNIST Dataset of Handwrittent Digits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Lets load our dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 2s 0us/step\n",
      "(60000, 28, 28)\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist\n",
    "\n",
    "# loads the MNIST dataset\n",
    "(x_train, y_train), (x_test, y_test)  = mnist.load_data()\n",
    "\n",
    "print (x_train.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2A: Examine the size and image dimenions (not required but good practice)\n",
    "- Check the number of samples, dimenions and whether images are color or grayscale\n",
    "- We see that our training data consist of **60,000** samples of training data, **10,000** samples of test data\n",
    "- Our labels are appropriately sized as well\n",
    "- Our Image dimenions are **28 x 28**, with **no color channels** (i.e. they are grayscale, so no BGR channels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Initial shape or dimensions of x_train (60000, 28, 28)\n",
      "Number of samples in our training data: 60000\n",
      "Number of labels in our training data: 60000\n",
      "Number of samples in our test data: 10000\n",
      "Number of labels in our test data: 10000\n",
      "\n",
      "Dimensions of x_train:(28, 28)\n",
      "Labels in x_train:(60000,)\n",
      "\n",
      "Dimensions of x_test:(28, 28)\n",
      "Labels in y_test:(10000,)\n"
     ]
    }
   ],
   "source": [
    "# printing the number of samples in x_train, x_test, y_train, y_test\n",
    "print(\"Initial shape or dimensions of x_train\", str(x_train.shape))\n",
    "\n",
    "print (\"Number of samples in our training data: \" + str(len(x_train)))\n",
    "print (\"Number of labels in our training data: \" + str(len(y_train)))\n",
    "print (\"Number of samples in our test data: \" + str(len(x_test)))\n",
    "print (\"Number of labels in our test data: \" + str(len(y_test)))\n",
    "print()\n",
    "print (\"Dimensions of x_train:\" + str(x_train[0].shape))\n",
    "print (\"Labels in x_train:\" + str(y_train.shape))\n",
    "print()\n",
    "print (\"Dimensions of x_test:\" + str(x_test[0].shape))\n",
    "print (\"Labels in y_test:\" + str(y_test.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2B - Let's take a look at some of images in this dataset\n",
    "- Using OpenCV\n",
    "- Using Matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Using OpenCV\n",
    "# import opencv and numpy\n",
    "import cv2 \n",
    "import numpy as np\n",
    "\n",
    "# Use OpenCV to display 6 random images from our dataset\n",
    "for i in range(0,6):\n",
    "    random_num = np.random.randint(0, len(x_train))\n",
    "    img = x_train[random_num]\n",
    "    window_name = 'Random Sample #' + str(i)\n",
    "    cv2.imshow(window_name, img)\n",
    "    cv2.waitKey(0)\n",
    "\n",
    "cv2.destroyAllWindows()     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's do the same thing but using matplotlib to plot 6 images "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVEAAACvCAYAAABEme2fAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAF/BJREFUeJzt3XmQFOX9x/H3V8QrIEJU5JKNSH4J8UwUUZTCoBYmKphCShMtrKBYMSQYLSMQ72iCZYgStaJQIZAKicQD1EpSikQjGtQFEuUSQQTkEg8OAxrc5fn9MfN0z7C77Mz2TM907+dVNbU93T3T353v7jNPdz+HOecQEZGW2a/SAYiIJJkKURGRCFSIiohEoEJURCQCFaIiIhGoEBURiUCFqIhIBJEKUTMbbGYrzGyVmY0tVVBSWcpreim3pWctbWxvZm2At4FzgfVALXCZc25Z6cKTuCmv6aXclsf+EV7bF1jlnFsNYGaPAkOAJhNiZq29e9SHzrkjKh1EM5TX4iUhr1BkbpXXwvIa5XS+G/BezvP12XV5zGyUmS0wswURjpUWaysdQAGU1+IlIa9QQG6V1zwF5TVKTbQgzrnJwGTQN1uaKK/ppLwWL0pNdAPQI+d59+w6STblNb2U2zKIUojWAr3N7EtmdgBwKfB0acKSClJe00u5LYMWn8475+rMbDTwLNAGmOqcW1qyyKQilNf0Um7Lo8VNnFp0MF1jWeicO6XSQZSa8qq8plRBeVWPJRGRCFSIiohEoEJURCQCFaIiIhGoEBURiUCFqIhIBCpERUQiKHvf+Wq0//7hr/3ss88CUFNTA0CvXr0qEZJEcOeddwJwyy23BOsGDx4MhPkVKRfVREVEIkhtTfQLX/hCsHzRRRcBcMkllwD5NdGzzz4bgD179gBwyimZDgoLFmgksGp07LHHBstz5swB4OijjwYgt/edz3XXrl0B+P3vfx9XiBKjE088EYDDDz88WHfttdcC+f/nzfn3v/8dLN9+++1FxaCaqIhIBKmriX7jG98A4PHHHw/W9ezZE4B169YB0K1bgzGG2W+/zPeJvzaqmmh1ufjiiwGYMWNGsO6ggw4C4J577gHg7rvvDrbNmjULgCuvvBKArVu3AjB79uyyxyrl88tf/hKACy64AIAePTIj+7Vv3z7S+/7rX/9q8WtVExURiUCFqIhIBM2ezpvZVOACYItz7rjsuk7ATKAGWAMMd85tLV+YzWvTpg0ADz/8MBDeUAD4wQ9+AMC0adMAuO6664Jt/vTAO/XUU4H8ywFplJS83nvvvQBcc801AGzevDnY9sADDwDw4IMPAvD5558H2+rr64HwMs0ZZ5wBtI7T+aTkFqBt27YAdOrUKVh39dVXA9C9e3cAvve97wXbDj74YCD8f//f//4HwJYtW4J9fvvb3wL5fw9eXV0dAA899FDe+s8++6zFv0MhNdFpwOC91o0F5jrnegNzs88lWaahvKbVNJTb2DRbE3XOvWRmNXutHgIMzC5PB14EbiphXEWbMGECACeddBIAV111VbBt+vTpeft27Nixwet98xh/QyLtqj2vv/rVrwAYM2YMEN4U/Pa3vx3s89ZbbxX8fhdeeCEAP/3pT0sVYtWq5tweeeSRAHzta18DYOzYTFl+zjnnFPU+b775JhCeVb7wwgulCrFoLb0739k5tym7vBno3NSOZjYKGNXC40i8lNf0Kii3ymvxIjdxcs65fU0jUO4pWI844ggAfvzjHwPw2GOPAQ1rn7n69+/fYN2TTz4JwKuvvlrqEBOpEnm9/PLLg2VfA/XXvr773e8CxdU+pXH7ym058jpgwIBg2Z9h+KaIjVm2bBkA//jHP4D8/8lHHnkECGuwlayBei29O/++mXUByP7c0sz+kgzKa3opt2XS0pro08AIYEL251Mli6hIvovXxo0bAbjiiiua3Nd36WysJuprsK1cRfN6+umnB8u+BupbT+R2yyu1Aw88EAgbbgOsWrWqbMerkNhz61tG3HjjjcE6XwP19yD8GaAfRAbg3XffBWDnzp0N3vO5554D4KOPPipDxC3TbE3UzP4MzAf+z8zWm9lIMok418xWAudkn0uCKK/ppdzGq5C785c1sWlQiWORGCmv6aXcxivxfeeHDRsGwOuvvw6Ejawbc8ABBzRY51+n0/nK8WO45l6K8U2aJk2aBMDu3bsjHcPfkMod3cv3xx83bhwQ3rQA+M1vfhPpeBJ+1r7RfC4/loH/7N95552C3rOaTuM9dfsUEYkg8TXRo446CghrKr5bWG6XL9/Q+r777mvwen9hO3csSonX9ddfD0C7du2CdX5EptzufM3p27dvsJw7viTA8ccfD8D8+fODdccddxwAS5cuBWDt2rXFhC3N+OSTT4DwrALghBNOAMJunn60tIULFwb7+A4RixYtiiXOqFQTFRGJwOKsgZWjsb2/pnLXXXcB4bWVXbt2Bfv40a/9N2NujWfkyJFAbCOfL3TOnRLHgeIUNa87duwA8vPiayi1tbVAeF1t7xpmLt+VEODQQw9tdJ/cgSb8/Et+gJrcwU2KpLzuQ+fOYeeoUaMynaH8eKC+2WGu7du3A+FgQuPHjy9FGC1RUF5VExURiUCFqIhIBIk/nfe+8pWvAGGzlVxvv/02EPaI8TcyIJw65L333itXaLl02teIxYsXA/mn46Xie734yz65076UcJwE5bWFzj//fCA8vYdw7FjfDMrvA7FP26PTeRGRcktNTbQQkydPBvLHGvVNLbZt2xZHCKqxNMKfDfizCQibufgJyJYsWQLAzJkzm3yfPn36BMsTJ04EwnEn/TizZaK8ltCUKVMA+P73vw/AM888E2wbOnRonKGoJioiUm6Jb2xfjLPOOgvIH5MytymUVIZv5J7b2N03PyrGf//735LFJJWzevXqvOfHHHNMsNyhQwcgbAZVDVQTFRGJoFXURP23l2+AvWLFimBb1IEtpHpceumlDdZ9+OGHFYhESil34CA/Rmk1KWQ80R5m9oKZLTOzpWY2Jru+k5nNMbOV2Z8NZ3+TqqW8ppPyGr9CivU64AbnXB+gH/BDM+uDpmBNOuU1nZTXmBUyKPMmYFN2+RMzWw50o0qmYC1Ely5d8n7mns63VmnIq+dH7mps8rPWNi5omvLq5Y6X4HPtG+JXg6KuiWbnsj4ZeA1NwZoayms6Ka/xKLgQNbN2wBPAdc65HWYWbIt7ClYpnTTktV+/fnk/pTry6s8MckfH2rBhQ6P7nnfeecHyiBEj8ra9/PLLwbKfkLKaFHSry8zakknIDOfck9nVmoI14ZTXdFJe49VsTdQyX2G/A5Y7536ds6lqpk2W4qU9r3V1dUDra8JWTXmdO3cuAE89FR7Kd3Q55JBDALj66qsB6NgxbCyw//6ZYumDDz4A4Oabby53qJEUcjrfH7gCWGxm/8muG08mGX/JTse6FhhenhClTJTXdFJeY1bI3fmXAWtis6ZgTSjlNZ2U1/i1ih5Le5szZ06lQ5AS2rNnD5A/XbY/FVyzZk0lQhLCKbBnz55d1Ov8+LJ33HEHEI7gVa2qrw+ViEiCtMqaqG+wK+nwz3/+E4B58+YF6/zYpNXY17q1eP7554FwXFAIZ57wo6f5sUNzvfLKK0BybgrqL0xEJIJWURP140z6uZYOOuigSoYjZeKb1AAMHDgQgGOPPRaAZcuWVSKkVu3TTz8FYPr06cG63OW0UE1URCSCVjXHUhXQXDzppLymk+ZYEhEpNxWiIiIRqBAVEYlAhaiISARxN3H6ENiZ/Zk0hxM97p6lCKQKKa/ppLwWINa78wBmtiCJdzKTGndckvr5JDXuuCT184kzbp3Oi4hEoEJURCSCShSikytwzFJIatxxSernk9S445LUzye2uGO/JioikiY6nRcRiUCFqIhIBLEVomY22MxWmNkqMxsb13GLZWY9zOwFM1tmZkvNbEx2fSczm2NmK7M/Ozb3Xq1FEnKrvBZPeS0whjiuiZpZG+Bt4FxgPVALXOacq7pBHrNzcndxzi0ys/bAQmAocCXwsXNuQvYPqqNz7qYKhloVkpJb5bU4ymvh4qqJ9gVWOedWO+d2A48CQ2I6dlGcc5ucc4uyy58Ay4FuZOL1I8pOJ5MoSUhuldeiKa8FilSIFlHd7wa8l/N8fXZdVTOzGuBk4DWgs3NuU3bTZqBzhcIquyJP4xKX29aaV0j3/2yl8triQjRb3X8IOB/oA1xmZn1KFVilmVk74AngOufcjtxtLnMNJJVtw5TXdOYV0p3biubVOdeiB3A68GzO83HAuH3tm/1FWvPjg5Z+3nE9islrzv6V/lwr/aj6vLbwf7bSn2ulHwXlNcooTo1V90/beyczGwWMAo6PcKy0WFvpAApQbF4lGXmFAnKrvOYpKK9lv7HknJvsMqOpXFzuY0l8fF5dAkf4kaYpr8WLUohuAHrkPO+eXdco59zfIhxL4lNUXiVRlNsyiFKI1gK9zexLZnYAcCnwdGnCkgpSXtNLuS2DFl8Tdc7VmdloMjeM2gBTnXNLSxaZVITyml7KbXlo3vl4aX7ydFJe00nzzouIlJsKURGRCOKe7VOk5GbPng3AhRdeGKwbP348APfcc09FYpLWQzVREZEIdGMpXroBUQb19fUA5P4tz58/H4CzzjorjhCU1zK4/fbbAbj11luDdWYGhLm+4YYbALjvvvvKEYJuLImIlJsKURGRCHRjSRKrpqamyW3t27cH4Itf/CIAH330URwhSQsddthhwfLf/pbpIX7aaZmxUVasWBFs8zcR+/fvD8Cdd94JwI4d4eh3CxYsAOCNN94oY8Qh1URFRCJIbU20Z8+ewfLo0aMBGD58OAA9eoRjMDzyyCMATJw4EYBVq1bFFaJE5PPamN69ewPQt29fAP7+97/HEpO0zP333x8s9+vXD4C7774bgDvuuCPYVldXB8CNN94IwJlnngnAlClTgn0efvhhAK699toyRhxSTVREJILU1UT9N9OMGTOCdbk1z71dc801QFhzHTZsGAC7du0qV4gSg0WLFgGqgSbFwIEDg+U5c+YAcNtttwGwZ8+eZl+/e/fuBq+Pi2qiIiIRNFuImtlUM9tiZkty1nUyszlmtjL7s2N5w5RSU17TS7mNVyGn89OAB4E/5KwbC8x1zk3ITrs6Frip9OE1r0+fzGSFDzzwABD2UNl//4a/mm/msnHjxmDd8cdnpn4aPHgwEPa/njlzZpkirhrTqOK8FsL3Xtlvv0xdoJDTvlZiGgnO7Ysvvgg0ns+DDz4YgGOOOSZvfe4YCbNmzSpfcI1otibqnHsJ+Hiv1UOA6dnl6cDQEsclZaa8ppdyG6+W3ljq7JzblF3eDHQuUTxFmzZtGgCnnJLfxTX3W8w3n/jrX/8KwC9+8Ysm3+/UU08FWkVNtDFVk9emdO3aNVgeOXIkEOY6znEgEqjqc+t179497/nRRx8dLE+aNAmAIUOGALB9+3YgbKpYCZHvzjvn3L4GKtAUrMmkvKbXvnKrvBavpYXo+2bWxTm3ycy6AFua2tE5NxmYDKUbFaZz5/BLNLdRPcCmTZkv20suuSRYt2FDZkJDX7v0DbAbc/nllwNw7733Buvef//9iBEnRkXzWog2bdoEy+3atWtyP3+NXAIF5bZSec118sknA3DccccB+bXM008/PW/fdevWAfn3OeLW0iZOTwMjsssjgKdKE45UmPKaXsptmTRbEzWzPwMDgcPNbD1wGzAB+IuZjQTWAsPLGeTejjjiiEaXAQ488EAg/9vLN7Y/9NBDm33vI488ssH7prEmWo15LUSHDh2a3ObHEAWYO3duHOFUpSTmduvWrcGyv/Pu/4dza5+vvPIKAF/96lcBeP755+MKsUnNFqLOucua2DSoxLFIjJTX9FJu46UeSyIiESSy7/w777zTYLlXr14AdOrUKe9nY5YvXx4s+z63J554YsnjlNK75ZZbmty2fv36YFnjhybL1KlTg2XfjMlfWss91ff/p/u6qRg31URFRCJIZE30008/DZYff/xxIJywqrHunp5vbO9HbgL42c9+BoTfcG+99RYA7777bgkjlnLYu9unfy7J0aVLFwAuuuiiJvf505/+FCyfdNJJAGzevBnIH0e0UlQTFRGJIJE10Vzjxo0D4KmnMs3e9tWM6bnnnmuwLrfxNsAf/pAZs2Hnzp2lClHKxHfzVLfP5DnkkEMA+OMf/wjA2WefHWzzA5CcccYZABx11FHBtpimwC6KaqIiIhGoEBURiSDxp/Peq6++2qLX5faxh/CCtYiUnh8P1E/f40/jf/7znwf7+FHWbr31VgB+8pOfBNu+/vWvA+H0L9VANVERkQhSUxMtRm7D+r0b7c6bNy/ucKRE/E1BqV4333wzEI4HOn16ZpxoPz0yhB1g7rrrLgBOOOGEYNujjz4KwKhRmdH6/E2oSlJNVEQkglZZEx09enSw3LZtWwBWrlwJ6JpotaqpqQHyayV7e/PNN2OKRlqqW7duec99Z5fcKY8936nG11YhvJZ6/vnnA6qJiogkXiHjifYgM2tgZ8ABk51zk8ysEzATqAHWAMOdc1ubep9qsncDe4DVq1cD8Nlnn8UdTkUkLa++wfWXv/zlBtt8t0+p/ryed955Rb/Gd+0GGD48MwzqN7/5zZLFFFUhf311wA3OuT5AP+CHZtaHcArW3sDc7HNJDuU1nZTXmBUyZfIm59yi7PInwHKgG5qCNdGU13RSXuNX1I0lM6sBTgZeI0FTsHodO3YE4OKLL26wzU9iV19fH2tM1SAJefUNrxvrH+9H3sod3UuqM6+PPfYYAD/60Y+A8MZuoV5//XUATjvtNCDsg79r165ShVi0ggtRM2sHPAFc55zbkTvsmKZgTS7lNZ2U1xg555p9AG2BZ4Hrc9atALpkl7sAKwp4H1fJx5QpU9yUKVPcnj17Gjy6du3qunbtWu4YFhTyecf1SEJeBw0a5AYNGuS2bdvmtm3b5urq6ho8amtrXW1trevQoUPwiPlvS3kt8DFgwAA3YMAAV19f7+rr6928efPcvHnz3GGHHRY89vX6Xr16uV69egX/txMnTnQTJ06saF6bvSZqma+w3wHLnXO/ztmkKVgTTHlNJ+U1foWczvcHrgAWm9l/suvGU+VTsDZm78FGIGys+8EHH8QcTcUlIq9+rqx9zalz1VVXAbB9+/ZYYqpyVZ3Xl156CYAxY8YA4XxKzzzzTLCP37Z48WIg/z5F7969AYJr4/79KqmQKZNfBpqad0FTsCaU8ppOymv81EpZRCSCVtF3fujQTJO4xqYO8SP/fP7557HGJIV54403AFi7di0APXv2DLb5KWH8PpIcDz74IBBObT1+/Phg24IFC4Cw6drGjRuDbb6n0po1a4Dwb6CSVBMVEYnA/AXaWA7WRNu0cps1axYQ1kj9iE0QTsEaU2Pdhc65U+I4UJwqldcqorxG5DvCQDgN8ne+8x0AduzYEWzzjfX9DaklS5aUM6yC8qqaqIhIBK3imqjna91+DheobHcxEcnYujUcUGrYsGEVjKR4qomKiETQKmqi69aty3vu7/SKiESlmqiISAQqREVEImgVTZyqiJrCpJPymk5q4iQiUm5x31j6ENiZ/Zk0hxM97p7N75JIyms6Ka8FiPV0HsDMFiTx1CepccclqZ9PUuOOS1I/nzjj1um8iEgEKkRFRCKoRCE6uQLHLIWkxh2XpH4+SY07Lkn9fGKLO/ZroiIiaaLTeRGRCGIrRM1ssJmtMLNVZjY2ruMWy8x6mNkLZrbMzJaa2Zjs+k5mNsfMVmZ/dmzuvVqLJORWeS2e8lpgDHGczptZG+Bt4FxgPVALXOacW1b2gxfJzLqQmZ97kZm1BxYCQ4ErgY+dcxOyf1AdnXM3VTDUqpCU3CqvxVFeCxdXTbQvsMo5t9o5txt4FBgS07GL4pzb5JxblF3+BFgOdCMT7/TsbtPJJEoSklvltWjKa4HiKkS7Ae/lPF+fXVfVzKwGOBl4DejsnNuU3bQZ6FyhsKpN4nKrvBZEeS2Qbiw1wczaAU8A1znnduRuc5lrIGrWkEDKazpVMq9xFaIbgB45z7tn11UlM2tLJiEznHNPZle/n73+4q/DbKlUfFUmMblVXouivBYorkK0FuhtZl8yswOAS4GnYzp2UczMgN8By51zv87Z9DQwIrs8Aqj8hNfVIRG5VV6LprwWGkNcje3N7FvA/UAbYKpz7u5YDlwkMzsTmAcsBvZkV48nc53lL8DRwFpguHPu44oEWWWSkFvltXjKa4ExqMeSiEjL6caSiEgEKkRFRCJQISoiEoEKURGRCFSIiohEoEJURCQCFaIiIhGoEBURieD/Aa2MoDoSWCnRAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 6 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# importing matplot lib\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Plots 6 images, note subplot's arugments are nrows,ncols,index\n",
    "# we set the color map to grey since our image dataset is grayscale\n",
    "plt.subplot(331)\n",
    "random_num = np.random.randint(0,len(x_train))\n",
    "plt.imshow(x_train[random_num], cmap=plt.get_cmap('gray'))\n",
    "\n",
    "plt.subplot(332)\n",
    "random_num = np.random.randint(0,len(x_train))\n",
    "plt.imshow(x_train[random_num], cmap=plt.get_cmap('gray'))\n",
    "\n",
    "plt.subplot(333)\n",
    "random_num = np.random.randint(0,len(x_train))\n",
    "plt.imshow(x_train[random_num], cmap=plt.get_cmap('gray'))\n",
    "\n",
    "plt.subplot(334)\n",
    "random_num = np.random.randint(0,len(x_train))\n",
    "plt.imshow(x_train[random_num], cmap=plt.get_cmap('gray'))\n",
    "\n",
    "plt.subplot(335)\n",
    "random_num = np.random.randint(0,len(x_train))\n",
    "plt.imshow(x_train[random_num], cmap=plt.get_cmap('gray'))\n",
    "\n",
    "plt.subplot(336)\n",
    "random_num = np.random.randint(0,len(x_train))\n",
    "plt.imshow(x_train[random_num], cmap=plt.get_cmap('gray'))\n",
    "\n",
    "# Display out plots\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3A - Prepare our dataset for training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "60000 train samples\n",
      "10000 test samples\n"
     ]
    }
   ],
   "source": [
    "# Lets store the number of rows and columns\n",
    "img_rows = x_train[0].shape[0]\n",
    "img_cols = x_train[0].shape[1]\n",
    "\n",
    "# Getting our date in the right 'shape' needed for Keras\n",
    "# We need to add a 4th dimenion to our date thereby changing our\n",
    "# Our original image shape of (60000,28,28) to (60000,28,28,1)\n",
    "x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "\n",
    "# store the shape of a single image \n",
    "input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "# change our image type to float32 data type\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "\n",
    "# Normalize our data by changing the range from (0 to 255) to (0 to 1)\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3B - One Hot Encode Our Labels (Y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of Classes: 10\n"
     ]
    }
   ],
   "source": [
    "from keras.utils import np_utils\n",
    "\n",
    "# Now we one hot encode outputs\n",
    "y_train = np_utils.to_categorical(y_train)\n",
    "y_test = np_utils.to_categorical(y_test)\n",
    "\n",
    "# Let's count the number columns in our hot encoded matrix \n",
    "print (\"Number of Classes: \" + str(y_test.shape[1]))\n",
    "\n",
    "num_classes = y_test.shape[1]\n",
    "num_pixels = x_train.shape[1] * x_train.shape[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 0., 0., 0., 1., 0., 0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4 - Create Our Model\n",
    "- We're constructing a simple but effective CNN that uses 32 filters of size 3x3\n",
    "- We've added a 2nd CONV layer of 64 filters of the same size 3x2\n",
    "- We then downsample our data to 2x2, here he apply a dropout where p is set to 0.25\n",
    "- We then flatten our Max Pool output that is connected to a Dense/FC layer that has an output size of 128\n",
    "- How we apply a dropout where P is set to 0.5\n",
    "- Thus 128 output is connected to another FC/Dense layer that outputs to the 10 categorical units"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_3 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "conv2d_4 (Conv2D)            (None, 24, 24, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_2 (MaxPooling2 (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "dropout_3 (Dropout)          (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "flatten_2 (Flatten)          (None, 9216)              0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 128)               1179776   \n",
      "_________________________________________________________________\n",
      "dropout_4 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 1,199,882\n",
      "Trainable params: 1,199,882\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "from keras.optimizers import SGD \n",
    "\n",
    "# create model\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss = 'categorical_crossentropy',\n",
    "              optimizer = SGD(0.01),\n",
    "              metrics = ['accuracy'])\n",
    "\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 5 - Train our Model\n",
    "- We place our formatted data as the inputs and set the batch size, number of epochs\n",
    "- We store our model's training results for plotting in future\n",
    "- We then use Kera's molel.evaluate function to output the model's fina performance. Here we are examing Test Loss and Test Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 189s 3ms/step - loss: 0.2901 - acc: 0.9124 - val_loss: 0.1392 - val_acc: 0.9581\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 316s 5ms/step - loss: 0.2177 - acc: 0.9348 - val_loss: 0.1016 - val_acc: 0.9709\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 319s 5ms/step - loss: 0.1699 - acc: 0.9494 - val_loss: 0.0790 - val_acc: 0.9737\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 371s 6ms/step - loss: 0.1393 - acc: 0.9581 - val_loss: 0.0673 - val_acc: 0.9788\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 244s 4ms/step - loss: 0.1199 - acc: 0.9635 - val_loss: 0.0596 - val_acc: 0.9811\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 179s 3ms/step - loss: 0.1072 - acc: 0.9681 - val_loss: 0.0516 - val_acc: 0.9830\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 213s 4ms/step - loss: 0.0968 - acc: 0.9711 - val_loss: 0.0481 - val_acc: 0.9847\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 194s 3ms/step - loss: 0.0874 - acc: 0.9748 - val_loss: 0.0470 - val_acc: 0.9853\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 193s 3ms/step - loss: 0.0823 - acc: 0.9749 - val_loss: 0.0434 - val_acc: 0.9854\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 158s 3ms/step - loss: 0.0752 - acc: 0.9769 - val_loss: 0.0413 - val_acc: 0.9860\n",
      "Test loss: 0.041306651647796386\n",
      "Test accuracy: 0.986\n"
     ]
    }
   ],
   "source": [
    "batch_size = 32\n",
    "epochs = 10\n",
    "\n",
    "history = model.fit(x_train,\n",
    "                    y_train,\n",
    "                    batch_size = batch_size,\n",
    "                    epochs = epochs,\n",
    "                    verbose = 1,\n",
    "                    validation_data = (x_test, y_test))\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 6 - Ploting our Loss and Accuracy Charts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xd4VVXWwOHfSg9JSEIJLUgQQidACL1LEXQEC4pYsWEZlcGKMzo6jPPJqGPBsSGDlTKO6MBIU5EIiPQOAUIn1CSUFAIhYX9/nJvkJgRSb85Nst7nOU9Ov+seMSt777P3FmMMSiml1JV42B2AUkop96fJQimlVJE0WSillCqSJgullFJF0mShlFKqSJoslFJKFUmThVJKqSJpslBKKVUklyYLERkqIjtFZLeITCjk+CMiskVENorIchFp43TsBcd1O0XkWlfGqZRS6srEVT24RcQT2AUMBhKANcBoY8x2p3NqGmNSHOvDgceMMUMdSWMm0BVoCPwEtDDGZF/u8+rUqWMiIiJc8l0qSnp6OgEBAXaH4Tb0eeSnzyOPPov8yvI81q1bl2SMqVvUeV6lunvxdAV2G2P2AojILGAEkJsschKFQwCQk7lGALOMMeeBfSKy23G/3y73YREREaxdu7Z8v0EFi42NpX///naH4Tb0eeSnzyOPPov8yvI8RORAcc5zZbJoBBxy2k4AuhU8SUR+DzwF+ADXOF27ssC1jQq5diwwFqBevXrExsaWR9y2SUtLq/TfoTzp88hPn0cefRb5VcTzcGWyKBZjzPvA+yJyB/AicG8Jrp0CTAGIiYkxlf0vDf1rKT99Hvnp88ijzyK/ingermzgPgw0dtoOd+y7nFnAjaW8VimllAu5smSxBogUkaZYv+hvB+5wPkFEIo0x8Y7N64Gc9bnADBF5C6uBOxJY7cJYlaoSLly4QEJCAufOnbM7FJcKDg4mLi7O7jDcRnGeh5+fH+Hh4Xh7e5fqM1yWLIwxWSLyOLAI8ASmGWO2ichEYK0xZi7wuIgMAi4Ap3BUQTnO+xqrMTwL+P2V3oQqk+VvQ4c7IKieS26vVEVKSEggKCiIiIgIRMTucFwmNTWVoKAgu8NwG0U9D2MMycnJJCQk0LRp01J9hkvbLIwx84H5Bfb92Wl93BWu/RvwN9dF55B2An59F4b+n8s/SilXO3fuXJVPFKrkRITatWuTmJhY6ntoD+5e42DTDEg5anckSpULTRSqMGX9d6HJ4vRBEA/47yN2R6KUbd7+cZfdISg3p8nizCE4mwx7Y+GV4PzLktfsjk6pCvHu4viiTyqGAQMGsGjRonz73nnnHR599NErXhcYGAjAkSNHGDlyZKHn9O/fv8iOt++88w5nz57N3b7uuus4ffp0cUIv1MqVK2natCkdO3akY8eOBAYG0rJlSzp27Mg999xTontdvHiRSZMmXfZ4eHh4mWJ1NU0WkdeCt7+13u0ReOVM3jLgBXtjU6qSGT16NLNmzcq3b9asWYwePbpY1zds2JBvvvmm1J9fMFnMnz+fkJCQUt9vwYIFvPnmm2zcuJGNGzcSExPD9OnT2bhxI1988UWJ7lVUsnB3mizmPwtXD7CqolZ/Ase3F32NUqpQI0eOZN68eWRmZgKwf/9+jhw5Qp8+fUhLS2PgwIFER0fTvn175syZc8n1+/fvp127dgBkZGRw++2307p1a2666SYyMjJyzxs/fjwxMTG0bduWl19+GYDJkydz5MgRBgwYwIABAwBrGKCkpCQA3nrrLdq1a0e7du145513cj+vdevWPPTQQ7Rt25YhQ4bk+5zFixczaNCgy37frKwsnnrqKbp27UpUVBRTp04F4PDhw/Tu3ZuOHTvSrl07VqxYwYQJE0hNTS1RqSQpKYnhw4cTFRVFz5492bp1KwA///wzHTp0oGPHjkRHR5Oenl7oZ5Yn23tw22rDdDiyHh76GX58GdZ8Agueg3v/B9pIqCq5iAnzXHL+/knXX/ZYrVq16Nq1KwsWLGDEiBHMmjWL2267DRHBz8+P7777jpo1a5KUlET37t0ZPnz4ZRteP/zwQ2rUqEFcXBybN28mOjo699hLL71EkyZNyM7OZuDAgWzevJknn3ySt956iyVLllCnTp1891q3bh2ffvopq1atwhhDt27d6NevH6GhocTHxzNz5kw++eQTbrvtNmbPns1dd91FUlIS3t7eBAcHX/b7TpkyhbCwMFavXs358+fp3r07Q4YMYebMmdxwww08//zzZGdnk5GRQdeuXZk6dSobN24s1nPO+Z7dunVj7ty5/PDDD4wZM4a1a9fyxhtvMGXKFLp160ZaWhpZWVl8/PHHl3xmeareJYv0RLj1M/AJgAF/BP9asH8ZbL/0Lx6lVPE4V0U5V0EZY/jjH/9IVFQUgwYN4vDhwxw/fvyy91m6dCl33XUXAFFRUURFReUe++6774iOjqZTp05s27aN7duvXCOwfPlybrrpJgICAggMDOTmm29m2bJlALltEgCdO3dm//79APzwww8MGTLkivf94Ycf+PTTT+nYsSPdunXj9OnTxMfH06VLF6ZOncpf/vIXtm7dmtsmU1LLly/n7rvvBmDIkCEcOXKE9PR0evXqxbhx43jvvfdISUnB09Oz3D7zcqp3yaL3H/LWa9SCgS/B9+Phhxchcgj41LAvNqXK6EolgIIiJswr0flXMmLECMaPH8/69es5e/YsnTt3BmD69OkkJiaybt06vL29iYiIKFVP83379jF58mTWrVtHaGgoY8aMKVOPdV9f39x1T0/P3L/IFyxYwFNPPXXFa40xfPDBBwwcOPCSY7GxscybN4977rmH5557jlGjRpU6xoJefPFFhg8fzrx58+jevTtz5szhmmuuueQz77zzznL7zOpdsigo+l6oH2W9IfXrO3ZHo1SlFBgYyIABA7j//vvzNWyfOXOGsLAwvL29WbJkCQcOXHlk7L59+zJjxgwAtm7dyubNmwFISUkhICCA4OBgjh8/zoIFC3KvCQoKIjU19ZJ79enTh//+97+cPXuW9PR0vvvuO/r06XPZzzbGsHnz5twSx+Vce+21fPDBB2RlZQGwc+dOMjIyOHDgAPXr12fs2LHcd999bNiwAS8v62/znHOLo0+fPkyfPh2An376iUaNGhEQEMCePXuIiorihRdeIDo6mvj4+EI/szxV75JFQR6ecN0bMO1aWP4OdLwDQiPsjkqpSmf06NHcdNNN+d6MuvPOO7nhhhto3749MTExtGrV6or3ePTRR7nvvvto3bo1rVu3zi2hdOjQgaioKFq1akXjxo3p1atX7jVjx45l6NChNGzYkCVLluTuj46OZsyYMXTt2hWABx98kE6dOuVWORW0bt06OnXqVGRHtocffpiDBw/mJpWwsDDmzJnD4sWLeeutt/D29iYoKIgvv/wSgAceeICoqChiYmIKfZuqbdu2uZ95xx13MHHiRO6//36ioqIIDAzk008/BeDNN99k2bJleHh4EBUVxcCBA5kzZ06hn1lujDFVYuncubMpN988aMzLNY2ZeUf53bMYlixZUqGf5+70eeRXnOexffv2Ut27yfPfl+o6u6SkpLj0/n/961/NzJkzXfoZ5am4z6Owfx9YY/UV+TtWSxaFGTwRdsyDHd/Dnp+h2TVFX6NUJTZuYKTdIbiVF1980e4Q3I62WRSmZgPo96y1vuB5yL5gbzxKudj4wS3sDkG5OU0Wl9P9MajVDJJ2weopdkejlFK20mRxOV6+MNTRNT92kjWUuVJKVVOaLK6kxRBoMRTOp8BPf7E7GqWUso0mi6Jc+3/g6QMbv4KEK494qZRSVZUmi6LUbgY9HrfW5z8LFy/aG49S5Wn525B6+SE3Sio5OTl3OO/69evTqFGj3O2cwQWLct9997Fz584rnjNlypTczmpl1bt37xKN11Rd6auzxdHnadg0yxp0cON0iL7b7oiUKh/lPK1w7dq1c3/xvvLKKwQGBvLMM8/kOyfnvX0Pj8L/Vs3peHYlY8eO1Tm4K5iWLIrDNxCG/NVa/+kVyHDfCUqUKpGcaYXLsXRRmN27d9OmTRvuvPNO2rZty9GjRxk7dmzuMOMTJ07MPTfnL/2srCxCQkKYMGECHTp0oEePHpw4Yb1oMnHixNxhxnv37s2ECRPo2rUrLVu2zB2aOz09nVtuuYU2bdowcuRIYmJiil2CyMjI4N5776V9+/ZER0ezdOlSALZs2UKXLl3o2LEjUVFR7N27l9TUVIYNG0aHDh1o165dmebjcGdasiiudrfAmn/BwRXwy99hqM6ip9zcK5cfWvsS/yhBP4tXzpQ8FmDHjh188cUXxMTEADBp0iRq1apFVlYWAwYMYOTIkbRp0ybfNWfOnKFfv35MmjSJp556imnTpjFhwoRL7m2MYfXq1cydO5eJEyeycOFC3nvvPerXr8/s2bPZtGlTviHOizJ58mR8fX3ZsmUL27Zt47rrriM+Pp4PPviAZ555hlGjRnH+/HmMMcyZM4eIiIjcMarOnCnd83F3WrIoLhEY9ndrkqRVH8OJHXZHpFSl0qxZs9xEATBz5kyio6OJjo4mLi6u0GHG/f39GTZsGJB/+PCCbr755kvOWb58ObfffjtgjSfVtm3bYse6fPny3OHR27ZtS8OGDdm9ezc9e/bk1Vdf5fXXX+fQoUP4+fkRFRXFwoULmTBhAr/++usV57+ozLRkURINoqDzfbD2X9YkSffM0UmSlPsqTgkgMx2mDLCG6+94h0vDCQgIyF2Pj4/n3XffZfXq1YSEhHDXXXcVOsy4j49P7rqnp+dlR2zNGWb8SueUh7vvvpsePXowb948hg4dyrRp0+jbty9r165l/vz5TJgwgWHDhvHHP/7RZTHYRUsWJXXNi+AfCvt+gbj/2R2NUmUz7xkIj3F5oigoJSWFoKAgatasydGjR1m0aFG5f0avXr34+uuvAautoagJkpw5Dw0eFxfH0aNHad68OXv37qV58+aMGzeO3/3ud2zevJnDhw8TGBjI3XffzdNPP8369evL/bu4Ay1ZlFSNWlbCmPc0LPoTNB+kkySpysl5WuEKFh0dTZs2bWjVqhVNmjTJN8x4eXniiSe45557aNOmTe5yuSqia6+9Fm9vb8BKFNOmTePhhx+mffv2eHt788UXX+Dj48OMGTOYOXMm3t7eNGzYkFdeeSV3fm0PDw98fHz46KOPyv27uIXiDE1bGZZyHaK8KNlZxnzQyxrGfMlr5XZbHZI7P30e+ZX7EOXL3jbmeOmGNLdbcYbkvnDhgsnIyDDGGLNr1y4TERFhLly44OrQbKFDlLsrD0+47nX4dJjVqanDaAhtYndUSpWM87TCVVBaWhoDBw4kKysLYwwff/xx7mx1quT0yZVWk57QbiRs/caas3tUOc9KpZQqk5CQENatW2d3GFWGNnCXxZC/gncAxM2FvbF2R6MUYFUtK1VQWf9daLIoi5oNoe/T1rpOkqTcgJ+fH8nJyZowVD7GGJKTk/Hz8yv1PVxaDSUiQ4F3AU9gqjFmUoHjTwEPAllAInC/MeaA41g2sMVx6kFjzHBXxlpqPR6H9V9C4g5Y/Qn0eMzuiFQ1Fh4eTkJCAomJiXaH4lLnzp0r0y++qqY4z8PPz4/w8PBSf4bLkoWIeALvA4OBBGCNiMw1xji/7LwBiDHGnBWRR4HXgVGOYxnGmI6uiq/c5EySNHMUxL4G7W+FwLp2R6WqKW9vb5o2bWp3GC4XGxtLp06d7A7DbVTE83BlNVRXYLcxZq8xJhOYBYxwPsEYs8QYc9axuRIofdqzU8uhEDnEmiRp8St2R6OUUuXOlcmiEXDIaTvBse9yHgAWOG37ichaEVkpIje6IsByde1r4OENG76CBH0DQylVtbjFq7MichcQA/Rz2t3EGHNYRK4GfhaRLcaYPQWuGwuMBahXrx6xsbEVFXKhrm50A1cd+paUfz/C+mjHoIMlkJaWZvt3cCf6PPLT55FHn0V+FfE8XJksDgONnbbDHfvyEZFBwJ+AfsaY8zn7jTGHHT/3ikgs0AnIlyyMMVOAKQAxMTGmf//+5fsNSqpHZ3hvBTVTd9E/5Ch0urNEl8fGxmL7d3Aj+jzy0+eRR59FfhXxPFxZDbUGiBSRpiLiA9wOzHU+QUQ6AR8Dw40xJ5z2h4qIr2O9DtALKP4oYHbxDco/SdK5qjmuvVKq+nFZsjDGZAGPA4uAOOBrY8w2EZkoIjmvwb4BBAL/EZGNIpKTTFoDa0VkE7AEmFTgLSr31f5WaNwd0k/AL6/bHY1SSpULl7ZZGGPmA/ML7Puz0/qgy1y3AmjvythcRsQaN+rjfrDqI4i+B+q2tDsqpZQqE+3B7QoNOkDnMXAxy+rZrb1plVKVnCYLV7nmJfALgb1LYMc8u6NRSqky0WThKgG1rUmSABa9ABcy7I1HKaXKQJOFK3W+D+q1g9MHYcV7dkejlFKlpsnClTy9YNjfrfVlb1lJQymlKiFNFq4W0Rva3QJZGfDDS3ZHo5RSpaLJoiIM/it414Dt/4W9v9gdjVJKlZgmi4oQ3Aj6OE+SlGVvPEopVUKaLCpKj8chNAIS42DNVLujUUqpEtFkUVG8/axhzAGW/B+kJ9kbj1JKlYAmi4rUchg0HwTnz8Div9gdjVJKFZsmi4okYk3B6uFtzdt9eL3dESmlVLFosqhodSKh+6OAgQXPwcWLdkeklFJF0mRhh77PQmA9SFgDm/9tdzRKKVUkTRZ28KsJgyda6z+9DOdS7I1HKaWKoMnCLu1vg/CukHYcluokSUop96bJwi4eHtYkSQis/BASd9kdkVJKXZYmCzs17GTNpHcxC74fr5MkKaXcliYLuw38M3j6wIHl1E5ebXc0SilVKE0WdguoA32fA6B5/Cdw4ZzNASml1KU0WbiD3uPBvxb+5xNh0R+1Okop5XY0WbgDTy/oP8FaX/sv+EsIvBJsLUteszc2pZQCvOwOQAGZ6bDmXyQ0vI7wIwsAAzd+BB1H2x2ZUkoBWrJwD/OegfAYdrd4OG8a1rmPw+7F9sallFIOmizstmE6HFkP171hbXd7GHqNs16n/foeOLLR3viUUgpNFvZLT4RbPwOfgLx9A1+xenhnpsH0W+HUfpuCU0opiyYLu/X+A4S1zr/PwwNGvA9N+0H6CfjqFkhPtic+pZRCk4X78vKBUV9BvfaQvBtmjoLMs3ZHpZSqpjRZuDO/mnDnfyD4Kms489kPQHaW3VEppaohTRburmYDuGs2+IfCzvkw/2nttKeUqnCaLCqDui1g9Czw8oN1n8HSN+2OSClVzbg0WYjIUBHZKSK7RWRCIcefEpHtIrJZRBaLSBOnY/eKSLxjudeVcVYKV3WHW/4F4gFLXoUNX9kdkVKqGnFZshART+B9YBjQBhgtIm0KnLYBiDHGRAHfAK87rq0FvAx0A7oCL4tIqKtirTRa/y6vP8bcJyH+R3vjUUpVG64sWXQFdhtj9hpjMoFZwAjnE4wxS4wxOa/4rATCHevXAj8aY04aY04BPwJDXRhr5dHlQejzNJhsq9Pe4XV2R6SUqgZcmSwaAYecthMc+y7nAWBBKa+tXq55CTqMhgtnYfptcHKv3REppao4txhIUETuAmKAfiW8biwwFqBevXrExsaWf3AVKC0trdjfQYJH0j40jlqnNpIxZRjro//OBZ8Q1wZYwUryPKoDfR559FnkVxHPw5XJ4jDQ2Gk73LEvHxEZBPwJ6GeMOe90bf8C18YWvNYYMwWYAhATE2P69+9f8JRKJTY2lhJ9h17d4NPr8D+2mV7734Ux3+cfNqSSK/HzqOL0eeTRZ5FfRTwPV1ZDrQEiRaSpiPgAtwNznU8QkU7Ax8BwY8wJp0OLgCEiEupo2B7i2Kec+QbBnd9AyFXWYIT/GaOd9pRSLuGyZGGMyQIex/olHwd8bYzZJiITRWS447Q3gEDgPyKyUUTmOq49CfwVK+GsASY69qmCgurBXd+Cfy2I/wG+/4N22lNKlTuXtlkYY+YD8wvs+7PT+qArXDsNmOa66KqQOpFwx9fw+Q2w4Uuo2QgGvGB3VEqpKkR7cFcVjbvAyGlWp71fJlk9vZVSqpxosqhKWl0H1//DWv9+POxcaG88SqkqQ5NFVRNzP/R9DsxFq8E7Ya3dESmlqgBNFlXRgD9Cx7sgKwNm3AZJu+2OSClVyWmyqIpE4IZ3oPlgOJsMX90MaSeKvk4ppS5Dk0VV5eltze3dsBOcPgDTR8L5VLujUkpVUposqjLfQLjjPxDaFI5ugq/vhewLdkellKqENFlUdYF1rZn2atSBPYutoc21055SqoQ0WVQHtZtZnfa8a8CmGfDzq3ZHpJSqZDRZVBfhna02DPGEZW/Cmql2R6SUqkQ0WVQnLa613pICmP8sxH1vbzxKqUpDk0V1E30P9P+j1Wlv9gNwcJXdESmlKgFNFtVRv+cg+l7IOgczR0HiLrsjUkq5uWIlCxFpJiK+jvX+IvKkiFStadmqExG4/i1oMRQyTsFXt0DqMbujUkq5seKWLGYD2SLSHGtmusbADJdFpVzP08sapbZRDJw5aHXaO5did1RKKTdV3GRx0TGZ0U3Ae8aYZ4EGrgtLVQifALjj31CrGRzbAlP6welDdkellHJDxU0WF0RkNHAvkPMKjbdrQlIVKqCO1WkvoC6c3Atf3gwXL9odlVLKzRQ3WdwH9AD+ZozZJyJNgS9dF5aqULWawp3/AS9/SN4FC56zOyKllJspVrIwxmw3xjxpjJkpIqFAkDHm7y6OTVWkhp1g1FeAwJpP4LtHdeBBpVSu4r4NFSsiNUWkFrAe+ERE3nJtaKrCRQ6CPk9b65tmwGvh8EqwtSx5zd7YlFK2Km41VLAxJgW4GfjCGNMNGOS6sJQtMtMh7n8w8M9Qr721z8PL2u6nVVNKVWfFTRZeItIAuI28Bm5V1cx7BsJjrNLFQ4uh++/hYhYsnghfjIAzh+2OUCllk+Imi4nAImCPMWaNiFwNxLsuLFXhNkyHI+vhujesbS9fGPp/jjelwmD/MviwJ2yfa2+cSilbFLeB+z/GmChjzKOO7b3GmFtcG5qqUOmJ1qi0PgH59zcfBI+ugMghcO40fH23NSdGZrotYSql7FHcBu5wEflORE44ltkiEu7q4FQF6v0HCGtd+LHAutZ8GMNeB09fWP85fNzPmn1PKVUtFLca6lNgLtDQsfzPsU9VFyLQ7WEYuwTqtobkePhkIKx4TzvxKVUNFDdZ1DXGfGqMyXIsnwF1XRiXclf12loJo8tDcPEC/PAifHWzDkSoVBVX3GSRLCJ3iYinY7kLSHZlYMqNefvD9W/C6FlQozbsXWI1fu9caHdkSikXKW6yuB/rtdljwFFgJDDGRTGpyqLlMKvx++oBcDbZmhtj3jNwIcPuyJRS5ay4b0MdMMYMN8bUNcaEGWNuBPRtKAVB9eGub2HIq+DhbQ0VMmUAHN9md2RKqXJUlpnyniq3KFTl5uEBPZ+AB3+C2s0hMc5KGKumgDF2R6eUKgdlSRZS5AkiQ0Vkp4jsFpEJhRzvKyLrRSRLREYWOJYtIhsdi/YEqwwadoSHl1rzfGefhwXPwoxRkJZod2RKqTIqS7K44p+MIuIJvA8MA9oAo0WkTYHTDmK1fRQ2616GMaajYxlehjhVRfIJgOHvwW1fgF8IxC+yGr93/2R3ZEqpMrhishCRVBFJKWRJxepvcSVdgd2O3t6ZwCxghPMJxpj9xpjNgL6oX9W0GQGP/gpNekP6CWue70V/gqzzdkemlCqFKyYLY0yQMaZmIUuQMcariHs3Apzn6Exw7CsuPxFZKyIrReTGElyn3EVwONw7F655CcQTfvsnTB0IiTvtjkwpVUJF/cK3UxNjzGHHoIU/i8gWY8we5xNEZCwwFqBevXrExsbaEGb5SUtLq/TfoXAxBHV6jTbb38L/2BayP+zN7uYPcrTBEKtn+GVU3edROvo88uizyK8inocrk8VhoLHTdrhjX7EYYw47fu4VkVigE7CnwDlTgCkAMTExpn///mWL2GaxsbFU9u9wef1hyGhY8Byem2bSctcHtPQ4aLVv1KhV6BVV+3mUnD6PPPos8quI51GWBu6irAEiRaSpiPgAt2ONL1UkEQkVEV/Heh2gF7DdZZGqiuFXE276CG6eCr41Ycf38GEv2PuL3ZEppYrgsmRhjMkCHseaByMO+NoYs01EJorIcAAR6SIiCcCtwMciktOTqzWwVkQ2AUuAScYYlyaLt3/c5crbK2dRt8Ijy6BxN0g9Yk2s9NMrkH3B7siUUpfh0jYLY8x8YH6BfX92Wl+DVT1V8LoVQHtXxlbQu4vjGT+4RUV+ZPUWGgFj5sPSN2Dp67D8bauEcctUiJsLHe6wO0KllBN3buBWVZ2nFwx4Aa7uD98+ZM3U91Efq8SRdgL8rrU7QqWUgyvbLCqNY2fO2R1C9dakBzyyHNreDBfSYe/PsGYq/ukJdkemlHKo9sliS8IZrvlHLABGxzGyj38IjJwGN34I3gGQnUnndU/Br5PhXIrd0SlV7VXLaqi3f9zFu4vjL9nf9IV8zSuMGxip7RgVSQQ63mFVQ80YhVdyPPz4krXk6DfBqrpSSlWoapksxg9ukS8J/LzjOPd/thaAvi3q8sGd0QT6VstH4x6C6oN4cKjRDTT2OgUHllv7Pbzg9AFr+PN6be2NUalqptpXQwFc06oeALUDfFi6K5Hbp/zGiVRtx7DNvGcgPIY9kQ/CffPgwZ+hzY1gLsKmmdbAhF/dAntjdQh0pSqIJgsn3z7Wk4jaNdh6OIWbP1jBnsQ0u0OqfjZMt96Kuu6NvH3hneG2z+GJ9dB1LHjXsEax/WIETOkHW76B7Cz7YlaqGtBk4aRJ7QBmP9qTjo1DSDiVwS0frmDt/pN2h1W9pCfCrZ9ZQ50XVKuplUTGb4MBL0JAXTi6CWY/AJM7wm8fwHlN8Eq5giYLh3EDIwGoHejLzIe6M6h1GKfPXuDOqatYuPWYzdFVI73/AGGtr3xOjVrQ71n4w1b43TvW7HxnDsGiF+DtNvDTXyBV/5spVZ40WTg4N3j7+3jy0V2duaPbVZzPusij09fx+Yr99gWnCuftBzH3we+ET0BYAAAeUElEQVTXwO0zoHF3OHcGlr8F77SHOb+HEzvsjlKpKkGTxWV4eXrwtxvb8ey1LTEGXp67jdcWxHHxojaouh0PD2h1PTywCB74EVrfYI0zteEr+KAbTL8N9i/XxnClykCTxRWICL8f0Jx/3NoBLw/h41/2Mv7rjZzPyrY7NHU5jbvCqK/giXUQ8wB4+VlTu352PXxyDWz9VhvDlSoFTRbFcEvncKaN6UKAjydzNh5hzLQ1pJzTEVLdWu1m8Lu3rMbwfhPAv5b1ltU398F70bBqCmSm2x2lUpWGJoti6tuiLl8/0oO6Qb78tjeZ2z76jaNnMuwOSxUloI7V43v8Nrj+HxDa1OrYt+BZeLst/PyqNWihUuqKNFmUQNuGwXz3WE+a1Q1gx7FUbv5gBTuPpdodlioOnxrQ5UGreuq2L6BRDGScsoZIf7sd/G8cJBUYAmb525B63J54lXIzmixKKDy0BrMf7UmXiFCOnjnHyI9W8NueZLvDUsXl4QltRsCDP8F9C6HldZB9HtZ9Bv/sAjPvgIMrrXPTTsCv79oarlLuQpNFKYTU8OHLB7oxrF19Us9lce+01czddMTusFRJiFhDo4+eCY+vheh7wdMHds6DadfC1MFQtzVsmqGlC6XQZFFqft6e/POOaMb0jCAz+yJPztzAJ0v36jDnlVGdSBg+GcZvhb7Pgn8oJKyG/z0BF7PhmzGQvMfuKJWylQ6tWgaeHsLLN7ShUYg/f5sfx9/mx3HkTAYvXt8GTw+xOzxVUoFhcM2L0Hu81Ufjt3/C6YNwYIX1BpWzvs/BNX+yJ06lbKAlizISER7qezWTR3fCx9ODT3/dz+Mz1nPugvbFqLR8AqDbw/DwcqjZ0GoM9wvOf86aT2D2Q7B1ttVrXKkqTksW5WR4h4bUDfRl7JdrWbD1GElpq/jknhhCavjYHZoqrYUT4OoBcOMHVo/wgyth10LYuQBO7oEtX1uLhxc06QkthlpL7WZ2R65UudOSRTnq0aw23zzSkwbBfqzZf4qRH/1GwqmzdoelSqPgUOme3tC0D1z7N3hyPTy+Doa8Ck16W8OI7FsKi/5oVVf9swv88JJVfaW9xVUVocminLWsH8S3j/WkVf0gdp9I46YPVrDtiFZTVDpXGiodoE5z6PmENTnTs7vh5qnQbqRVXZW0C1ZMhk+HwZvNtbpKVQlaDeUCDYL9+fqRHjz8xbrc3t4f3d2ZPpF17Q5NFVfvPxT/3Bq1IOpWaylWddUwaDkUal3tuviVKmdasnCRmn7efHZ/F4Z3aEh6Zjb3fbqG2esS7A5Ludol1VVrC6muegEmd9LqKlWpaMnChXy9PHlnVEcahPjx8S97efo/mziWco7H+jdDRF+trRbqRFpLzyfg7EnYvRh2LYD4n6zqqpwqK/9QaD7YKnE0H5T/7avlb0OHO+z7DkqhycLlPDyEF4a1pmGwP6/8bxtvLNrJkdMZ/GV4W7w8tWBXrZS2uipn2BG/IXZ/A1WNabKoIPf2jKBeTV/GzdrI9FUHOZ5yjvdGR+Pv42l3aMoOOdVVOVVWSfGOxLEQDv5mVVflVFnVuhpSjlA30h8yu1y+0V0pF9I/bSvQ0HYNmP5gN0JqePNT3AlGf7KS5LTzuce/i8+0MTplq5yqqnxvV90CvsFwci9knaNt3Jsw6SqYOgh+/DPsWgQZp+2OXFUTmiwqWExELb55pCfhof5sPHSaWz5cwYFkaxKeOXt0QiVFXnXVyGnw3B6493trpFyAi1mQsMaqlppxG/y9CXzUGxY8D9vnQHqSvbGrKkuroWzQPCyQbx/ryX2frmHbkRRu/mAF08Z0sTss5Y48vaFRNMzbQVyrcbS+8RlrkMP9v1pvUR1eB8e2WMuqj6xr6rS02jya9LJ+Bjey9zuoKsGlyUJEhgLvAp7AVGPMpALH+wLvAFHA7caYb5yO3Qu86Nh81RjzuStjrWhhQX78++EePDZ9PUt3JXL7lJV2h6Tc1bxnIDyG4yHX0NqvpvW2VPNB1rELGZCw1kocB36FQ6shaae1rPvUOiekSV7iaNLTagPRt/FUCbksWYiIJ/A+MBhIANaIyFxjzHan0w4CY4BnClxbC3gZiAEMsM5x7SlXxWuHQF8v/nVvDC98u4VvHH0wXv1+O48NaE6tAB1TSpE37MhDP8OKNZce9/bPaygHyMqEoxutxHFghfXG1ekD1rJphnVOUIO8xNGkl1US8dAaaXVlrixZdAV2G2P2AojILGAEkJssjDH7HccuFrj2WuBHY8xJx/EfgaHATBfGW6He/nEX7y6Ov2T/1OX7mLp8X+72uIGRjB/coiJDU+6kqGFHCvLygcZdraX3eGs+juNb80oeB1ZA6lFr+JGts61r/Gvlr7aq396aUbCgnP4eQfXK7eupykNcNVmPiIwEhhpjHnRs3w10M8Y8Xsi5nwHf51RDicgzgJ8x5lXH9ktAhjHmzQLXjQXGAtSrV6/zrFmzXPJdKsqYhem0r+PJliRrePMgHxjezIf+jb3wrobzY6SlpREYGGh3GG6jXJ6HMdQ4e4iQ09sIPrOdkNNb8c08me+ULM8anAluzZngNpwOaUtqUHOMhzfNdk8FhD3NHyhbDOVA/23kV5bnMWDAgHXGmJiizqvUDdzGmCnAFICYmBjTv39/ewMqq4Xz+N8zQ1m5N5nXF+5g/cHTTI/LJPaoJ08NjuTGTo2q1aRKsbGxVPr/puXIJc/DGDi1P6/UceBXvE7tp/bJddQ+uc46x8sfwmMgvAOs/5zGN0+EWk3LN44S0n8b+VXE83BlsjgMNHbaDnfsK+61/QtcG1suUVUC3a+uzexHe/JT3AneWLSDXcfTePo/m/h46R6eGdKSwW3q6XAhqnyIWL/4azWFTndZ+84ctjoG5iSQxB2wf5m1gDWuVf12EO6o7grvoo3m1YArk8UaIFJEmmL98r8dKO4AN4uA/xORUMf2EOCF8g/RfYkIg9vU45pWYczZeJi3ftzFruNpjP1yHZ2uCuH5oa3ofnVtu8NUVVFwI2g/0lrA6rtxYIW1xC+yOgnmvK679l/WOV7+cHU/K3E07goNo8FXq4mqEpclC2NMlog8jvWL3xOYZozZJiITgbXGmLki0gX4DggFbhCRvxhj2hpjTorIX7ESDsDEnMbuqmxEM+9L9nl6CDdHh3N9VANmrjrIez/vZsPB09w+ZSV9W9TluWtb0q5RcCF3U6qcBNSBNsOh+UDY8zP87h2rx/mh1dZruwmrrYb4XQutBUA8IKwtNO6SVwLR0kel5tI2C2PMfGB+gX1/dlpfg1XFVNi104BprozP3dwUefnXZX29PBnTqym3xjRm2vJ9fLx0L0t3JbJ0VyK/i2rA00Na0rSOjhmkXMjR34OY+6ztiN7Wz5x2j4Q11nJotVXqOO5Y1jr+N/av5Sh5OBJIo2jwDbLlq6iSq9QN3NVRgK8XTwyM5M7uTfgwdjef/3aA7zcfZcHWY4zq0phxAyOpV9PP7jBVVePc36Mg53aPqNusfZln4ciG/Akk/YRVjRW/yHGdB4S1yau6Cu8CtZtfufShw7XbRpNFJVUrwIc/Xd+G+3o1ZfLieL5ee4gZqw4ye10CY3pF8Gi/ZoTU0I59qpyUtL+HTw2I6GUtYJU+Th+AQ47kkZBT+thqLTm9zf1DraQR3tUqxTTqDH418+6rw7XbRpNFJdcwxJ9Jt0TxYJ+reevHnczfcoyPf9nLjFUHeaRfM+7rFUENH/3PrMqoJNPMFkYEQiOsJepWa1/mWau3eU7JI2ENpB2H+B+sxbrQKn007mIlkVY3wL/vwKejjqVW0fS3SBXRPCyQD+7szOaE07yxaCfL4pN4Y9FOPv11P+MGNmdUl6vw8dIhHZQb8amRN+wIOEofBwu0fWyGE9usZd1n1nmevkSvfxYuLIPgcAi5yvoZHG4NZVJY73NVZposqpio8BC+fKAbv+5O4vWFO9iUcIaX5mzjk2X7eGpwC4Z3aIhHNerYpyoREQhtYi05r+1eyICjmxwlj9VWNVbaMfyyE/Ne2813D0+o2ciRRBrnJZFgp4RSHq/0VsOhTzRZVFG9mtfhv7/vxaJtx3lj0Q72JKbzh39v5KNf9vDc0JYMaBmmHfuU+/P2h6u6WwvA+TT4uA+H/NvQuH0fOHPIsSRYS9pxOHPQWg5e5p7+oZcmkJDGEOxILgFhRQ+smNN2MvT/yvXrujNNFlWYiDC0XX0GtQ7j2w2HeefHXew4lsr9n62lS0Qozw1tRZeIWpdc9/aPu3TwQuWe5j8LV/VgT8htNO7e/9LjF85BymFH8jiU9/O0U0LJOGUtx7YU/hmePlbpxDmBFPzZaxx80N36WU1KF5osqgEvTw9ui2nM8A4N+WrlAT6I3cOa/ae49aPfuKZVGM9e25LWDfLeOHl3cbwmC+V+ihquHcDbD2o3s5bCXLwIZ5MuTSDOJZSzyXBqn7VcTkBd8PCCT4dZrwvXbQVhra2Oh56Xdq6tCjRZVCN+3p482OdqRnVpzNRl+5i6bC8/7zjBkp0nGNGhIU8NbslVtWvYHaZShSvp67uF8fCAwDBradS58HMy063xsc4czEsmp51KKSmHrVhyYop9Lf/1NerC1X2hbmsIa2W9zRUaUekb3jVZVENBft6MH9yCe3o04f0le/hq5QH+u/EI328+yuiuV9kdnlKFK+vru8XlEwB1W1hLYS5mW6WOL2+CqweAfwic2AGJcdbbXGcT8+YKyeHlZw2RUre1VQIJa22VRkKaVJqJpzRZVGO1A3358w1tuL93BO/8FM+36xP4cuUBAO6cupK+kXXpE1mX1g2CtDFcqRwenrD0HxDRB4ZPzn/sfBok7rQSx4k4a8TeE3FWaSRn8EVn3jWgbsu8UkhOMgkOL944WhXYo12TRTV3uRn7ft2dzK+7k3ltwQ4AWtUP4uF+V9O7eV3qBvlWdJhKuY8rDX3iGwjhna3F2bkzVhLJTSDbrdJI2jFrWJQjG/Kf7xNkJRHnBBLW2upH4pxEKrBHuyaLam784BaXNGZHTJjHe6M7sXRXIsvikziWco4dx1IZ/+9NALRuUJO+LerQN7IunZuE4udduetilSqR0rSd+AXnTXfrLONUXhXWCafSSHoiHF5rLc58gx3tIK2tJBLeBb4fj0+nIie6KzNNFqpQN3RoyA0dGmKMYfeJNH5xJI5V+5KJO5pC3NEUPv5lL37eHnRrWpu+LerSN7IOzcMCtcpKVW3l2XbiHwpNeliLs/TkSxPIiTjIOAmHVlmLk24rx0LPnhBUv/xiK0CThboiESGyXhCR9YJ4sM/VnLuQzboDp6zh0eOTiDuawi+7Evlll/V2SINgP/pE1qFPZF16N69DaIAOZqhUiQXUhoDeecPAgzUcStoJRxLZkffz2BY8L6TDP1rmv0e/CTCg/OaM02ShSsTP25NezevQq3kdXgBOpJ5jeXwSy+KTWBafyNEz5/h6bQJfr01ABNo3CnY0lNeh01WhJRqf6rv4THSaZaUcRKwOgEH14Or+1r7MdJgygF0hfWlx15su/XhNFuoS4wZGFvvcsCA/bo4O5+bocC5eNOw4lsrS+ESWxSeyZt8pNiecYXPCGf65ZDcBPp70aGZVWfWJrEtE7RpXrLKas+cC75bHF1KqqnJMSHUk5He4uhutJgt1idL23vbwENo0rEmbhjV5pF8zMjKzWbkvmWW7rFJH/Ik0foo7wU9xJwAID/XPbevo0awOwf5Vs+erUi5RnB7t5UiThXIZfx9PBrQMY0DLMACOnslg2a4klsYnsnx3EgmnMpix6iAzVh3EQ6Bj45DcUkeHcJ1XXKkrKo8e7SWgyUJVmAbB/tzWpTG3dWlM9kXD1sNnWBZvNZSvP3CK9QdPs/7gad75KZ4gP+uf5ls/7KR1g5q0alCTJrVq6PDqSuWoqB7tDposlC08PYQOjUPo0DiEx6+J5O8Ld/Bh7J7c46nnsgCY/PPufNfVq+nLNa3q0bpBkJVE6gcR5KfVV0q5miYL5RaeH9qK54e2yt0+dPIsfV5fwqP9mxF3NIUdR1M5lnKO4ynnmbk6/0QF4aH+tG5Qk9b1g1xSCtEh25XSZKHcVONa1ui3zgnkZHomO46lEHc0lR1HU4g7lsKu42kknMog4VQGP24/nnuuv7cnLR3JI6cU0rJ+EDVLUQrRIduV0mShKpFaAT70bFaHns3q5O7Lyr7IvqR04o6lOkogVjI5lnKOjYdOs/HQ6Xz3CA/1p1X9mrRp4JpSiFJVlSYLVal5eXrk9jAf3qFh7v5T6ZnEHbOqr+IKKYX8FOeaUohSVZUmC+W2RjQr/S/r0HIshQAs3HqMtg1rEh7qr2NfqWpJk4VyWzdFlu+4UsUphXyz7hDbj6YC5JZCAB75al2+e3UID+bO7k1o06AmkfUC8fXSkXdV1abJQlV7zqWQ+3s3BaxSyP7kdLYfTeXJmRvo16Iu24+mkJh6HoBNCWfY9M1mALw8hOZhgVbv9QZWD/a2DYIJrqHVWKrq0GShVCG8PD1oHhZE87Agnpy5gc/vt+YhOJF6jrijqWw7cobtR1LYfjSFfUnp7DiWyo5jqXzL4dx7NArxz5dA2jQoWzWWDqyo7KTJQqkSCAvyIyzIj34t6ubuO5uZxY5jqbnJY/uRFHYcS+Hw6QwOn87/Sm+Qn1e+5NGmYU0iw4KKNRqvDqyo7OTSZCEiQ4F3AU9gqjFmUoHjvsAXQGcgGRhljNkvIhFAHLDTcepKY8wjroxVqdKq4eNF9FWhRF8Vmrsv+6JhX1J6bvKwfp4hKS2TVftOsmrfydxzvT2F5mFBeVVYDWvSukFNHVhRuRWXJQsR8QTeBwYDCcAaEZlrjNnudNoDwCljTHMRuR34OzDKcWyPMaajq+JTqrhKMmR7Dk9HO0bzsMB8jeknUs/lK4HkVGPlzD44e33ePcJD/XMTSOsG1ltZx86cIzTA29YGdXfo0a5VchXPlSWLrsBuY8xeABGZBYwAnJPFCOAVx/o3wD9F30tUbqY8fzGGBfkR1tKP/o6ReAHSzzuqsZwSyI6jKblvY/3gVI3V/bXFAAT6elErwCffUrvAtrXPl1qBPgT4eJbbK7/u0KNdq+QqniuTRSPgkNN2AtDtcucYY7JE5AxQ23GsqYhsAFKAF40xy1wYq1K2CfD1onOTUDo3yavG+scPO3mvwCCKztLOZ5F2PouDJ88W6zN8vDyoVcORQAKtn6E1HAkm0PoZWiPnmC8h/t7aq13l464N3EeBq4wxySLSGfiviLQ1xqQ4nyQiY4GxAPXq1SM2NrbiIy1HaWlplf47lKfq/Dw6+8BnQ/PPUzBmYTqfDQ3AGMPZLEjNNHnLBaf1TEhzrKc4jmVmXeRYyjmOpZwr1ucLEOgDQd5CkE/eEuhtJZDnP/0Rbw/w8sDxU/Kt5xzLvw7eTud5lLGk4w7/Nr6Lzyz3/kClURH/r7gyWRwGGjtthzv2FXZOgoh4AcFAsjHGAOcBjDHrRGQP0AJY63yxMWYKMAUgJibG9K/klZixsbFU9u9QnvR5FLBwXqmfR0ZmNifPZnIyLZPk9POcTM/MtySnZ3LKaf1MxgVSM62ERLq55H7/3plZxi9jtev4eHrg4+VYPD3w9cq/Xdi6r5cHkM6W7Ea5paJaAb7UdqzX9Ku4UtGYhfN496EhFfJZV1IR/6+4MlmsASJFpClWUrgduKPAOXOBe4HfgJHAz8YYIyJ1gZPGmGwRuRqIBPa6MFalqjR/H08a+fjTKMS/WOdfyL7IqbOZTP4pnq9WHSzy/MiwQJrUDiAz+yKZWdlkZl10rDst2Rc577SefdGQcTGbjAvZpfpO//hxV6H7PT3kkjacOoG++dp2aju2awf4EFwFqtwqosHfZcnC0QbxOLAI69XZacaYbSIyEVhrjJkL/Av4UkR2AyexEgpAX2CiiFwALgKPGGNOXvopSilX8Pb0ICzIj1dvas+rN7XPdyxiwjz2T7q+TPc3xpB10eRLJJlZ+ZNJ3rHsS4796butPNa/WW5J6GR6Jslp50lOzyT1XBaJqedze9sXxdND8tpvHG06BUsreft93TK5VESDv0vbLIwx84H5Bfb92Wn9HHBrIdfNBma7MjalKpuyDKzobkQEb0/B29ODAN+SX/+n77bynNNcJ87OZ2VzKv1CbnVbclpOQrG2k9Lyqt+S0s6Tei6LpLTzJKWVJLl4U9sR+AOfraGGrxeBvp4E+Hjlrft6EejrRQ0fLwJ8PXPXA32t7QAfL7dLOlfirg3cSqkC3KEhtTLw9fKkfrAn9YP9inV+ZpZV5ZaUlteWk+zUtpOTbPacSON0xgWyLxqS0qykA7B4x4lSx+rvnZNUPPMlkhq+XgT6eOUd8/Uq5DwvAnys68Eqrbmy54EmC6VUtebj5UG9mn7Uq1my5JKclsl1k5fxyT0xpJ/PIj0zi/TzWaSdz+asYzvtfLZ1LPe487bVXpNxIZuktLJ/j/TMbAJ9XfcrXZOFUqpEStOjvbzZWSVXMLkMblOvVPe5eNGQcSHbkWCyOJuZTdr5rHzbhR3bnHCGfUnpl9yv3cuL8m2PGxhZrp0nNVkopUrE7t7bUDWq5Dw8xKpK8vUirOjTr6g8XjooStFDXSqllKr2NFkopVQpuUOVXEXRZKGUUqXkDlVyFUWThVJKVXIV0eCvyUIppSq5imjw12ShlFKqSJoslFJKFUmThVJKqSKJNXVE5SciicABu+MoozpAkt1BuBF9Hvnp88ijzyK/sjyPJsaYukWdVGWSRVUgImuNMTF2x+Eu9Hnkp88jjz6L/CrieWg1lFJKqSJpslBKKVUkTRbuZYrdAbgZfR756fPIo88iP5c/D22zUEopVSQtWSillCqSJgs3ICKNRWSJiGwXkW0iMs7umOwmIp4iskFEvrc7FruJSIiIfCMiO0QkTkR62B2TnURkvOP/k60iMlNEijfFXRUhItNE5ISIbHXaV0tEfhSReMfP0PL+XE0W7iELeNoY0wboDvxeRNrYHJPdxgFxdgfhJt4FFhpjWgEdqMbPRUQaAU8CMcaYdoAncLu9UVW4z4ChBfZNABYbYyKBxY7tcqXJwg0YY44aY9Y71lOxfhk0sjcq+4hIOHA9MNXuWOwmIsFAX+BfAMaYTGPMaXujsp0X4C8iXkAN4IjN8VQoY8xS4GSB3SOAzx3rnwM3lvfnarJwMyISAXQCVtkbia3eAZ4DLtodiBtoCiQCnzqq5aaKSIDdQdnFGHMYeBM4CBwFzhhjfrA3KrdQzxhz1LF+DCjdxOBXoMnCjYhIIDAb+IMxJsXueOwgIr8DThhj1tkdi5vwAqKBD40xnYB0XFDFUFk46uJHYCXRhkCAiNxlb1TuxVivuJb7a66aLNyEiHhjJYrpxphv7Y7HRr2A4SKyH5gFXCMiX9kbkq0SgARjTE5J8xus5FFdDQL2GWMSjTEXgG+BnjbH5A6Oi0gDAMfPE+X9AZos3ICICFaddJwx5i2747GTMeYFY0y4MSYCq+HyZ2NMtf3L0RhzDDgkIi0duwYC220MyW4Hge4iUsPx/81AqnGDv5O5wL2O9XuBOeX9AZos3EMv4G6sv6I3Opbr7A5KuY0ngOkishnoCPyfzfHYxlHC+gZYD2zB+h1WrXpzi8hM4DegpYgkiMgDwCRgsIjEY5W+JpX752oPbqWUUkXRkoVSSqkiabJQSilVJE0WSimliqTJQimlVJE0WSillCqSJguliiAi2U6vNG8UkXLrQS0iEc6jhyrlrrzsDkCpSiDDGNPR7iCUspOWLJQqJRHZLyKvi8gWEVktIs0d+yNE5GcR2Swii0XkKsf+eiLynYhsciw5w1R4isgnjjkafhARf8f5TzrmONksIrNs+ppKAZoslCoO/wLVUKOcjp0xxrQH/ok1Wi7Ae8DnxpgoYDow2bF/MvCLMaYD1vhO2xz7I4H3jTFtgdPALY79E4BOjvs84qovp1RxaA9upYogImnGmMBC9u8HrjHG7HUMBHnMGFNbRJKABsaYC479R40xdUQkEQg3xpx3ukcE8KNj0hpE5HnA2xjzqogsBNKA/wL/NcakufirKnVZWrJQqmzMZdZL4rzTejZ5bYnXA+9jlULWOCb7UcoWmiyUKptRTj9/c6yvIG+qzzuBZY71xcCjkDvHePDlbioiHkBjY8wS4HkgGLikdKNURdG/VJQqmr+IbHTaXmiMyXl9NtQxGux5YLRj3xNYM9s9izXL3X2O/eOAKY5RQrOxEsdRCucJfOVIKAJM1ulUlZ20zUKpUnK0WcQYY5LsjkUpV9NqKKWUUkXSkoVSSqkiaclCKaVUkTRZKKWUKpImC6WUUkXSZKGUUqpImiyUUkoVSZOFUkqpIv0/7R4dsmmscTIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting our loss charts\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "history_dict = history.history\n",
    "\n",
    "loss_values = history_dict['loss']\n",
    "val_loss_values = history_dict['val_loss']\n",
    "epochs = range(1, len(loss_values) + 1)\n",
    "\n",
    "line1 = plt.plot(epochs, val_loss_values, label='Validation/Test Loss')\n",
    "line2 = plt.plot(epochs, loss_values, label='Training Loss')\n",
    "plt.setp(line1, linewidth=2.0, marker = '+', markersize=10.0)\n",
    "plt.setp(line2, linewidth=2.0, marker = '4', markersize=10.0)\n",
    "plt.xlabel('Epochs') \n",
    "plt.ylabel('Loss')\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzs3Xd4lFXa+PHvnU4KhGaQmoC00AJEiiBViqiLCCqCuui6KGtbfdVF19d1WQv81rUtNiwo+yKIKOKuKE2yoihNigJCAAOEhJIESCNlMuf3xzOZTEI6mZmU+3NdufLUee45hLnnOec854gxBqWUUqo8Pt4OQCmlVO2nyUIppVSFNFkopZSqkCYLpZRSFdJkoZRSqkKaLJRSSlVIk4VSSqkKabJQSilVIU0WSimlKuTn7QBqSosWLUxkZKS3w7goWVlZhISEeDuMWkPLozgtjyJaFsVdTHls3749xRjTsqLj6k2yiIyMZNu2bd4O46LExcUxYsQIb4dRa2h5FKflUUTLoriLKQ8ROVKZ47QaSimlVIU0WSillKqQJgullFIV0mShlFJ13Ir4PLdfQ5OFUkrVcSsP5bv9GposlFJ1jie+SVfGS2sPeDsEj9FkoZSqczzxTboyXlkf7+0QPKbePGehlPKMl9Ye4KExXbwdRr2SX2AnO7eAzDwb2bk2svIKyM61kZlrIzuvgKw8G9m51u8sl/1ZeQVk59kAsNsNPj7ithg1WSilquSV9fF1PlkYY7AbKLAb7MZgDNiNcfwU7S/cVrTf+lAuXAfYeexssQ/urNwCxwe644M+1+UD3rHfeZwjCeQV2C/6PXV8YlWx9QdHd67RfydNFkopj8mz2S/4IM3OK3B8gy7xQVrKB3B2nrUNYPDz60v/cLeX+HAvJRnUpOtf++6iX8PXRwgO8CU00I/gAF9CCn8H+BES6EdIoC/BAX6EBPgSHOjYFuDYFujLbe9u4eCzV+Pn676WBU0WSqkK2QrsJJ/LIfHMeQA+23G8qEqk8APe+S26aD27xLfq/IKa+6ROPpdzUef7CPiI4COCOJZ9fYqWC/eLY7nwfVSkZ+vGDOzYvOiD3fnh7/KhH1iUCIIDfAn080Hk4qqQ3JkoQJOFUnXGivg83DkcUmaujaOp2RxNy+ZoWhZH07I54lg/fuY8Npev5H/8aGe1ruHnI0Xfikv5hhwc4EdooG+5H6whgX6MfekbNs0e5fxQF5cPdx8RxKfkB37RvsLja0Lk7C9ImHtNjbxWbafJQqk6YuWhfF65iPONMZzKyHUmgKOpjoSQls3R1GxSs6rfHXXoZc0Z3/NSl+oSx4e8S3VKcKAvgX6+F/EOimsd3qjGXktVTJOFUvVITn4BiWfOW3cGqVYiOOa4Qzh2Jpuc/LIbUgP8fGjfLLjYT4fm1k/bpsEE+Vsf9A3p23RdMbGTv9uvoclCqTrEGMOZ7HxHFVGWMxFYVUfZnEjPwZTTLNAsJKBYImjXLJgOzYLp0DyES8IC3dr1UrnPpM4Bbr+GJgulaqFz2fkcTsnk8OksDqdk8mtKFgC9n15DRq6tzPN8fYQ2TRtZCaG5lQgKl9s3CyYsyP3fQD3BE9+kK+PB0Z29HYLHaLJQqgLueggtz2bnaFoWh05n8WtKFodPW8nh15SsMtsPSiaK/u3DmdSvrVVd1CyES8OD8Hdzr5jawBPfpCujrj9vUhWaLJSqwMU8hFbYqHzIJREcPp3J4RSrCqmsPv+N/H2JahFCVMsQOjl+P/TRLrY/eRXNQgJqrDdPdTSkb9OqiCYLpWpAZq6NhJSs4kkhJZNfT2eV2TdfBNo1a0THFqF0bBlCxxYhdGwZSlSLEFo1Drqg/eChj3bRPDTQE2+nXA3p27QqoslCqUqyFdhJPHOeXwuTQkoWvzraFE6m55Z5XtNgf6IcicA1KbRvVtTDSKnaTpOFUiXYCuwkpGaxNzmDfcnpAIz+RxxH07LLfAI5wNeHyBbBdGwRSpRLQujYIoSmIbWjfl2pi+HWZCEi44FXAF/gHWPM3BL7OwDvAS2BNOBWY0yiY9//A67BGkZ9LfCgMeV1ClSq6s5l57M3OZ1fTqSzLzmduP2nOZVx4V3CodNZxdZ7t2nC9X3b0LFlCJ1ahtI6vBG+bu52Wlt6AKmGyW3JQkR8gdeAMUAisFVEPjfG7HU57AVgkTHmAxEZBTwP3CYiVwBDgN6O474FhgNx7opX1W8FdkNCahb7ktP5xXHHsC85naQyxhdqE96I7pc2JvrSMF79+iBfPDCUqBYhBAd472a8tvQAUg2TO//yBwAHjTGHAURkKTARcE0W0cDDjuUNwGeOZQMEAQGAAP7ASTfGqmqh6o6FlJ6T70wIv5xIZ29yBgdOZHA+/8KG5iB/H7pGhNH90sbOn66twmjSqOhb/KtfH6RH6yYX8U6UqvvcmSzaAMdc1hOBgSWO2QXcgFVVNQkIE5HmxpjvRWQDkIyVLOYbY/aVvICIzARmAkRERBAXF1fjb8KTMjMz6/x7qEkrD+UzqZzysBvD6WzD0Qw7x1x+Us6XXlvZLEhoF+ZDuzAf2jt+R4QIPmID0iAvjewjsOPIhefWhn8X/fsoomVRnCfKw9sN3I8A80VkBvANcBwoEJHLgO5AW8dxa0XkSmPMRteTjTELgAUAsbGxZoQ7h+T0gLi4OOr6e6hRX33hLI/MXBv7HXcJhVVI+09kkF1Kt9QAP+tuoVsr1zuGMMKDq1mN4xKHN+nfRxEti+I8UR7uTBbHgXYu620d25yMMUlYdxaISCgw2RhzVkR+D/xgjMl07PsSGAwUSxaqfjLGsP3IGQDu/tc29iVncDQtu9RjIxoHOhNCt1ZhRF/amKgWITU6tr8+hKaUe5PFVqCziERhJYmpwDTXA0SkBZBmjLEDj2P1jAI4CvxeRJ7HqoYaDrzsxlhVLXDufD4rfkzkwy1HOXAyE4DVe6ymqgBfHy67JNR5l1CYIJp5oFuqPoSmlBuThTHGJiL3Aauxus6+Z4zZIyJzgG3GmM+BEcDzImKwqqHudZy+HBgF/ITV2P2VMebf7opVeY8xht2J53j8093sTc4o87i8Ajt7k9MZEx3BXVd29GCESilwc5uFMWYVsKrEtqdclpdjJYaS5xUAd7szNuVdWbk2Vu5MYvHmI+xJSnduH3JZc6YP7MCY6Ag6//lLnTdBqfJ8+xL0mVbxcTXA2w3cqoHZm5TOh1uO8NmOJDIdI6g2Dfbnxth23DKgPVEtQrwcoVJ1SOYp+O4VCBrr9ktpslBul5NfwH92J7N48xF2HD3r3D4gshnTB7VnXI9WOkaSUlWVmwmdRsGy3xLV6iRunaAdTRbKjQ6eymDx5qN8sj2R9BzrLiIsyI/J/doybWB7ukSEeTlCpeqA7DQ4vR9S9lu/T++HlANwrugxtg7HlkPeqxDgvjtzTRaqRuXaCli95ySLfzjC5l/TnNv7tAtn+sD2XNe7NY0CKncXoWMhqQbDGMhILkoEp3+B0wesBJF1uvRzfPyh+WXQqBkc/Q6ea118//DZMPLxGgtRk4WqEUdSs/hwy1GWb0t0zvIWHODLxJg2TB/Ynp5tqj5cho6FpC7gwQbdSsURFlG18+wFcPaoy52CIzGkHIDc9NLP8Q+BFp2hZTdo2QVadLWWm0ZCQS4sGMm+bg/Sfeqci35b5dFkoaotv8DO+n2nWLz5CBvjU5zbu1/amOkD2zMxpnW9mfNZ1RIebNCtVBzjnyt9vy0P0g4Xrzo6vR9S48FW+uCVBIUXJYSW3RxJoQs0bgs+ZTxk+vn90DaWk+Gj6F4z76xMmizUBSqac/r42fN8tOUoS7cecw7nHejnw3V9WjNtYHv6tgv36rSfqh4b8iC8PoiAmMtrRRwMmAk5Z12qjhzVSGmHwW4r/dzQVtCyq/XTwpEYWnaFkJbW9ImVtWMxJP0Iv/8aNm2tmfdVDk0W6gKlzTldYDd8c+A0izcf4etfTjnnju7UMoTpAzswuV9bmgTrXYRys8AwaDeIHj8/B/IjGLtVtWO3gXH8tttdlgv32YvWXY9znleV4xz78s/Dq33KCFQgvINLUnBUHbXoDI3Ca6Yssk7Dje+7tVHblSYLVa5T6Tks23aMJVuOcfzseQD8fYVre17K9IHtGRDVTO8ilHulHoL4tRC/GhK+hYI8mgBsOuDtyErXfwaMex4Cgt17naF/dO/rl6DJQpXq2/gUPtxyhDV7TmJz3EZ0aB7MLQPaM6V/W1qEBno5QlVv2fLgyHdFCSL1YPH9fkGkNO5Ji37Xgo8fiK/128fHZdnxW3xcln0dy77Fj3M9v9hxpZ3vOC4/Fz641qqO6nebd8rJwzRZKABOpufw45Ez/HjUGu311nc3A+DrI4zv0Yrpg9ozpFMLfNw8dahqoNKTIX6N9XM4DvIyi/YFNYHLroLOY+HAavBvxM/hNzFi6AhvRQtfzYJ2AxpMogBNFg1Sns0alO/HI2dYsuUo8acyyzy2wG74as8JurYK48rOLT0YparX7AWQuM2RIFbDiZ+K77+kB3QZC53HQdvLwdfPatA9tddjDbplcm1YbkA0WTQAp9Jz+PHoGX48epYfj5zhp+PnyLXZix0TFuhHTPtw+rZvyqvr4zn03AR89S5C1aTsNDi43koQB9fB+aKHNvEPho4joPMY6w6iSdsLz/dwg26ZakscHqbJop5xvWv48egZdhw962yYdtWpZQj92jelX4em9GvflMsuCXUmh1fXx2uiUBfPGDj5s1V1FL8GErdavY0KNY2CLuOsBNFhKPgHlf96Hm7QLVNticPDNFnUIivi86o8Flhl7hpCA/2IaRdOv/bh9O3QlL7twqs/xahquCrz1HJuptXmEL/GaqDOSCra5+MPkVc6EsRYa6gK7UlXZ2iyqEVWHsrnlXL25xfY2ZuUXiw5lHXX0Ld9U8edQzidLwnTOwV18cp6ajn1kOPuYTUc2QQFeUX7QltZdw5dxlnVTIE6eGRdpcmiFjuVkcOPR86y46hVpbQ70TN3DTrntCpV4VPLg+6BlPiirq1ph10OEmg7wNE4PRZa9da7h3pCk0Ut8/53v1p3DUfPkHjmwruGjoVtDW68a9A5p1UxBTY4+RMc+d7qxvpq3+JDWQSFW11bu4yDTqMhpLn3YlVu49ZkISLjgVew5uB+xxgzt8T+DsB7QEsgDbjVGJMoIiOBl1wO7QZMNcZ85s54vcVuN/zvyp8BePrfe53bQwJ8iWkf7kwOMe3CaRqibQ3KzfJz4Ph2OLrJShDHNhd/7qGknLNW+0PvmzwXo/I4tyULEfEFXgPGAInAVhH53Biz1+WwF4BFxpgPRGQU8DxwmzFmAxDjeJ1mwEFgjbti9YaX1h7glfXx5R6TlVdAbIdm+k1fuVdOOhzbYj01ffR7K1G4tjuA1XOp7QD49b9wxQNwxb3eiVV5jTvvLAYAB40xhwFEZCkwEXBNFtHAw47lDUBpdw5TgC+NMdlujNXjHhrThT9e1Zk5/9nLwu8SCPDzIc9mJ2HuNd4OTdV3maeL7hqOfGd1b3Xt0opARE9oPxg6DIb2V0DjS2HFLLhstCaKBsqdyaINcMxlPREYWOKYXcANWFVVk4AwEWlujEl1OWYq8GJpFxCRmcBMgIiICOLi4momcg8wxrB0fx6rE2z4Cdzbx5+XtufWqffgbpmZmVoeLqpVHsYQlHOKJuf20uTcHsLP7iX4/PFih9jFl4zGXTnXJJpzTXpwrkl3bP6h1s4UIGU/rZJfp92xjWzv/wL2WvBvon8bxXmiPLzdwP0IMF9EZgDfAMeBgsKdInIp0AtYXdrJxpgFwAKA2NhYM8LNE5bXFGMM877az+qEQ/j7Cm9M789V0RG8tP0L6sp78IS4uDgtD3A+3xC3fV/F5WGMNadCYZXSkU2QXjw54B9sDaHR4QpoPxiftrE0CQih3LkMv90Joz9m2CXunmKncvRvozhPlIc7k8VxoJ3LelvHNidjTBLWnQUiEgpMNsacdTnkJmCFMSbfjXF6lDGGf6w5wJv/PYSfj/DPW/pxVXQVp2ZUDUt5s8MV2ODELkeV0iYrQbgOowFWb6XCKqUOQ+DSPuBbxblHGuhTy6qIO5PFVqCziERhJYmpQLHJc0WkBZBmjLEDj2P1jHJ1i2N7vfHK+njmbziIr4/w6i19Gd+zlXPfxE46eZAqhevscPnnrQboI5usn8StF/ZUCrvUkRyusH5adi97Wk6lKsltycIYYxOR+7CqkHyB94wxe0RkDrDNGPM5MAJ4XkQMVjWUs+VMRCKx7kz+664YPW3+1/G8vC4eH4GXb45hQq9Li+2f1Fm7xapSZKdB04703/4QbP492EvcaDfr6KhSciSHppH6IJyqcW5tszDGrAJWldj2lMvycmB5GecmYDWS1wtvxB3ihTUHEIEXb4rhuj6tvR2Sqs0yTsBPH8Puj5zDd5c63dTg+2Dcsx4NTTVM3m7gbhDe/uYw8776BRH4+5Q+XN+33uRAVZNyM+GX/8CupdbzDIXdWQMbgwjHWgyn3fR/1twczkpVgSYLN3vv2195dtU+AObd0Jsp/UsZp181XAU2+DUOdn1kJYp8x+NEPv7QdQL0vtna7uPHofCbaKeJQnmJJgs3WvR9AnP+Yz2D+NykXtx0ebvyT1ANgzGQvAt2L4Ofl0PmyaJ97QZZw2b0mATBzaxZ2ZJ3eX92ONXgabJwk8Wbj/DUyj0A/G1iD6YNbO/liJTXnT1W1A5x+pei7c06WXcQvW+CZlHFz2mgs7Kp2keThRt8tPUof15hDQz4l+uiuW1wpHcDUt6Tcw72rrTuIhI2Fm0Pbg49J1tJok3/snsv6fMNqpbQZFHDlm9PZPanVu+VJ6/pzh1Doio4Q9U7tjw4tN5qqN7/JRTkWtt9A6HbBOg91RpjqaoPxinlRZosatCKHYk8unwXxsDsq7tx15UdvR2S8hRjrIfldi2Fnz8p/hR15JXWHUT0b6z5IJSqgzRZ1JDPdyXxP8usRPHI2C7cM7yTt0NSnpB2GHY72iHSDhVtb9nNShC9boRw7dig6j5NFjVg1U/JPPTRTuwG/nhVZ+4bpdOS1guOAfwIKzF2V3Ya7PnUaoc4trloe8glVnLoc7NOJ6rqHU0WF+mrn0/wwJIdFNgN94+6TOevrk8KB/Ab/xzYcuHAV1aCOLC6aMgN/2Dofp3VkylqBPjqfylVP+lf9kVYt/ck9y/5EZvdcM/wTjw8pgui3ybrjysegNcut7qvxq+2ejYBiA90GmU1VHe7BgJDvRunUh6gyaKaNvxyij8s/pH8AsPvr4ziT+O7aqKoL/JzrEbqzW9Cbgb8tMza3qqXlSB6TYGwVuW/hlL1jCaLavjvgdPc/X/bySuwc8eQSJ6Y0F0TRX2QngRb34XtCyHbMVljYGPITbeWT/xk/az5MwyfDSPr1ej5SpVLk0UVfXcwhZmLtpFns3P74A48dW20Joq6zBirkXrzW9bDc8YxUWOr3tD/Dtj8Blw9D2Kmlf86StVzmiyq4PtDqfzug63k2uxMG9iev/6mhyaKuio/x+rRtPlNa+wlAPGF6Oth4D3QfhB89gdr+lFNFEppsqisLb+mcef7W8nJt3NzbDuemdhTE0VdlJ4E296DbQshO8Xa1qgZxN4BsXdCE8eowDsWQ9KP1gB+SilNFpWx/UgaMxZu4Xx+AVP6t+X5G3rh46OJos4wxpp+dPObVlWT3WZtj+gFg+6xxmjyb1T8HB3AT6li3JosRGQ88ArWtKrvGGPmltjfAWve7ZZAGnCrMSbRsa898A7W1KoGmOCYPc+jdhw9w2/f20p2XgGT+rZh3uTemijqClsu/PwpbHkLknZY28QXoic6qpoG6wB+SlWS25KFiPgCrwFjgERgq4h8bozZ63LYC8AiY8wHIjIKeB64zbFvEfCsMWatiIQCdnfFWpZdx85y+7tbyMy1cV2f1vx9Sm98NVHUfunJVlXT9oXWHQJYVU39Z8DlvyuqalJKVZo77ywGAAeNMYcBRGQpMBFwTRbRwMOO5Q3AZ45jowE/Y8xaAGNMphvjLNXPx89x27ubyci1cU2vS3nppj74+fp4OgxVWcZA4jZHVdNnxauaBt5tPRtRsqpJKVVp7kwWbYBjLuuJwMASx+wCbsCqqpoEhIlIc6ALcFZEPgWigHXAbGMK+zW6196kdKa/s5n0HBvjekTw8tQYTRS1lS0X9nxmJYmkH61t4gPdf2NVNXW4QsdoUqoGiDHGPS8sMgUYb4y5y7F+GzDQGHOfyzGtgflYCeEbYDLQE7gKeBfoCxwFPgJWGWPeLXGNmcBMgIiIiP5Lly6tdrwr4vOY1DmAYxl25m05T2Y+xLT05b6+gfh5qOopMzOT0FAdOqJQeeURkJtG66SvaJ30FQH51jAc+X5hJLUeS1Lrq8kNaunJUD1C/z6KaFkUdzHlMXLkyO3GmNiKjnPnncVxrMbpQm0d25yMMUlYdxY42iUmG2POikgisNOlCuszYBBWAnE9fwGwACA2NtaMGDGi2sHO+OoL7vvNYP5nwQ9k5sPIri1587b+BPr5Vvs1qyouLo6LeQ/1hmO017jt+y4sj8Kqpj0rXKqaesLAu/HvdSMd/BvRweMBe4b+fRTRsijOE+XhzmSxFegsIlFYSWIqUOzpJhFpAaQZY+zA41g9owrPDReRlsaY08AoYJsbYwXglrc3k5qVx5WdW/DGrZ5NFMpF4WivQWOt9cKqpi1vWRMMgaOq6TpHVdMQrWpSys3cliyMMTYRuQ9YjdV19j1jzB4RmQNsM8Z8DowAnhcRg1UNda/j3AIReQRYL9aTb9uBt90V668pWQCkZOYy5LLmvH17LEH+mii8ZsiD8PogQqI7w4bvrZ5NWaesfY2aQr/fWr2awtt7N06lGhC3PmdhjFkFrCqx7SmX5eXA8jLOXQv0dmd8AEdSs7hlwQ8ADOrYjHduv1wThbcFhEJYa2K3P4T1iA1wSQ9Hr6YbISDYq+Ep1RA1yCe4X1p7gFfWx1+w/YfDaXR/6ivn+oOjO/PQmC6eDE0d3QwrZsKZBIpVLJ3aA/9+wBquQ0d7VcrjGmSyeGhMl2JJ4NMfE3l42S4S5l7jxagauIJ8iJsL374Ixg6+gRxuN4WOM173dmRKKUAfHgBu6KdP9HrV6f3wzlWw8QXr4bpLukOPSRyNvMnbkSmlHCpMFiJyv4g09UQwqoGx2615JN4aBsk7rQbroQ9ZCePaF70dnVLKRWXuLCKwxnVaJiLjRcflVjUhPQkWT4YvHwNbDsRMh3u+g6AmOtqrUrVQhcnCGPMk0BnrgbgZQLyIPCcindwcm0c9OLqzt0NoOH7+FF4fDIe+tgb4u+lfcP3rENTYGu31ku7ejlApVUKlGriNMUZETgAnABvQFFguImuNMY+5M0BP0V5PHnD+rHUnsfsja/2yMTDxNQiL8G5cSqkKVZgsRORB4HYgBWt+iUeNMfki4gPEA/UiWSg3+/UbWDEL0hPBPxjGPmPNTKe1mkrVCZW5s2gG3GCMOeK60RhjF5Fr3ROWqjfyc+Drv8H3rwEG2vSHSQugxWXejkwpVQWVSRZfYs1iB4CINAa6G2M2G2P2uS0yVfed+Ak+nQmn9loz1A1/DK78H/D193ZkSqkqqkyyeAPo57KeWco2pYrYC+D7+fD1M1CQB806wQ0LoG2FoyArpWqpyiQLMS6TXjiqnxrkk9+qEs4cgc9mwZHvrPXY38HYv2lXWKXquMp86B8WkQew7iYA/gAcdl9Iqk4yBnYthVWPQl4GhFxi9XTqMtbbkSmlakBlHsq7B7gCa06KwqlRZ7ozKFXHZKXCstvhs3usRNHtWvjDD5oolKpHKryzMMacwpq4SKkLxa+DlX+AzJMQEAZXz4OYadolVql6pjLPWQQBvwN6AEGF240xd7oxLlXb5WXD2v+Fre9Y6+0Hw6Q3oWmkV8NSSrlHZaqh/gW0AsYB/8WaSzvDnUGpWu74dnjrSitR+PjDVU/DjC80UShVj1WmgfsyY8yNIjLRGPOBiHwIbHR3YKoWKrDBxn/Af+eBKYCW3a0usZe6fUJDpZSXVebOIt/x+6yI9ASaAJdU5sUdo9TuF5GDIjK7lP0dRGS9iOwWkTgRaeuyr0BEdjp+Pq/M9ZQbpR6C98ZB3HNWohh0L8yM00ShVANRmTuLBY75LJ4EPgdCgf+t6CQR8QVeA8Zg9aLaKiKfG2P2uhz2ArDIcccyCngeuM2x77wxJqbyb0W5hTGwfSGs/jPkZ0PjNtYIsR1HeDsypZQHlZssHIMFphtjzgDfAB2r8NoDgIPGmMOO11oKTARck0U08LBjeQPwWRVeX9W0b1+CPtOKRoHNOAmf3w/xq631XjfChL9DI50LS6mGptxqKGOMneqPKtsGOOaynujY5moXcINjeRIQJiLNHetBIrJNRH4QkeurGYOqisxT8N0r1vK+/8Abg61EEdQEJr8Lk9/RRKFUAyUuI3mUfoDIXKzhyT8Csgq3G2PSyjzJOm8KMN4Yc5dj/TZgoDHmPpdjWgPzgSisO5fJQE9jzFkRaWOMOS4iHYGvgdHGmEMlrjETxwOCERER/ZcuXVq5d11LZWZmEhoa6rXrB+SmcfnW+0lrGkPE6W8BOBPem1+6PUhuUAuPx+Pt8qhttDyKaFkUdzHlMXLkyO3GmAoHbqtMsvi1lM3GGFNulZSIDAaeNsaMc6w/7jjx+TKODwV+Mca0LWXf+8B/jDHLy7pebGys2bZtW3kh1XpxcXGMGDHCewHkn4dX+lgP2PkFwVV/hQEzwacy/SBqntfLo5bR8iiiZVHcxZSHiFQqWVRmWtWoUn4q03axFegsIlEiEoD1FHixXk0i0sLRLgLwOPCeY3tTEQksPAYYQvG2DlXT7HZrOPHMk9a6LQe++hPMaQpPN4ENpeZ4pVQDUZknuG8vbbsxZlF55xljbCJyH7Aa8AXeM8bsEZE5wDZjzOfACOB5ETFY1VD3Ok7vDrwlInashDa3RC8qVdPW/QX2fQ7iAyP/DMMe8XZESqlapDJdZy93WQ4CRgM/AuUmCwBjzCpgVYltT7ksLwcuqFoyxmwCelUiNlUTtr4Lm14FxOoSq4lCKVVCZQauzQShAAAgAElEQVQSvN91XUTCgbrdkqyKxK+FVY7kEBYBN/+fd+NRStVK1Wm5zMLqvaTquuTd8PEMMHaIGga3faaTFCmlSlWZNot/A4VdpnywHqRb5s6glAecOw4f3gR5mdbDdje8rcOKK6XKVJk2ixdclm3AEWNMopviUZ6Qk24lioxkaH+FNaOdJgqlVDkqkyyOAsnGmBwAEWkkIpHGmAS3Rqbco8AGy++Akz9D88tg6mLwC/R2VEqpWq4ybRYfA3aX9QLHNlXXGGM1Zh9cB8HNYfrHENzM21EppeqAyiQLP2NMXuGKYznAfSEpt9n0qjWCrG8g3LIUmlVlXEilVENWmWRxWkR+U7giIhOxxopSdcmeFbDW8YjLDW9BuwHejUcpVadUps3iHmCxiMx3rCcCpT7VrWqpY1vg07ut5av+Cj0meTcepVSdU5mH8g4BgxwD/WGMyXR7VKrmpB2GJVOhIBf63wFDHvR2REqpOqjCaigReU5Ewo0xmcaYTMcgf894Ijh1kbLTYPGNkJ0Kl10FE17QLrJKqWqpTJvF1caYs4UrjlnzJrgvJFUjbLmwdDqkHoSIXnDj++BbmVpHpZS6UGWShW/hcOFgPWcBaMf82swYWHkvHN0EYZfCtI8gMMzbUSml6rDKfNVcDKwXkYWAADOAD9wZlLpIG56Fnz6GgFCYtgyalJzNVimlqqYyDdzzRGQXcBXWGFGrgQ7uDkxV047/g2/+DuJrVT1d2tvbESml6oHKjjp7EitR3AiMAva5LSJVfYc2wL8dvZ0m/B06j/FuPEqpeqPMOwsR6QLc4vhJAT7CmrN7pIdiU1Vxci8sux3sNrjiAbj8d96OSClVj5RXDfULsBG41hhzEEBEHvJIVKpqMk5Yo8jmpkP0ROvBO6WUqkHlVUPdACQDG0TkbREZjdXAXWkiMl5E9ovIQRGZXcr+DiKyXkR2i0iciLQtsb+xiCS6PD2uSsrLgg9vhnPHoO3lMOkt8KnOnFZKKVW2Mj9VjDGfGWOmAt2ADcAfgUtE5A0RGVvRC4uIL/AacDXWhEm3iEh0icNeABYZY3oDc4DnS+z/G/BNZd9Mg2MvgE/uguSd0DQSpi4B/0bejkopVQ9V+BXUGJNljPnQGHMd0BbYAfypEq89ADhojDnsGKl2KTCxxDHRwNeO5Q2u+0WkPxABrKnEtRqm1U/A/lUQFA7Tl0NoS29HpJSqp6pUX2GMOWOMWWCMGV2Jw9sAx1zWEx3bXO3Cqu4CmASEiUhzEfEB/gE8UpX4GpQf3oTNb4JvAEz9EFp09nZESql6zNvjPzwCzBeRGVjVTcexJlf6A7DKGJMo5YxlJCIzgZkAERERxMXFuTtet8rMzKzUe2iespmePz+PAHu73MuphHxIqPi8uqay5dFQaHkU0bIoziPlYYxxyw8wGFjtsv448Hg5x4cCiY7lxVjTuSZgddtNB+aWd73+/fubum7Dhg0VH5S43ZhnWhnzl8bGxM1ze0zeVKnyaEC0PIpoWRR3MeUBbDOV+Ex3553FVqCziERh3TFMBaa5HiAiLYA0Y4zdkUzecySw6S7HzABijTEX9KZqcM4etXo+5WdDzHQY9qi3I1JKNRBu62NpjLEB92END7IPWGaM2SMic1xm3hsB7BeRA1iN2c+6K5467/xZa7jxrFMQNQyufVmHG1dKeYxb2yyMMauAVSW2PeWyvBxYXsFrvA+874bw6g5bnvV09ulfoGU3uOlf4KfToCulPEef3qrtjIH/PAS//hdCLoHpH0OjcG9HpZRqYDRZ1HbfvAA7/w/8GsG0pRDe3tsRKaUaIE0Wtdnuj2HDM4DAlHehTX9vR6SUaqA0WdRWCd/Byj9Yy+Ofh27XeDcepVSDpsmiNkqJh6XToCAPBt4Dg2Z5OyKlVAOnyaI2+PYlyDhpLWelwOIpkHMWuk6Acc95NzallEKTRe2QeQq+ewWfglxYMhXOJMClMTD5HfDx9XZ0Sinl9bGhFMCQB+H1QUQHb4bUbdCkHUz7CAJCvB2ZUkoBemdRO4S1gpbdaJG6DQIbW89ShLXydlRKKeWkdxa1QeYpSNppLeemw+uDivYNnw0jH/dOXEop5aDJojZY91ewnSczJJLQR3d5OxqllLqAVkN528m91hPaCHujda4npVTtpMnC25bfaf3u91uyQ9p5NxallCqDJgtvil8Hp/eBfyiMfqri45VSyks0WXhLgQ3W/NlaHjkbQpp7Nx6llCqHJgtv+fF9a36KppEwYKa3o1FKqXJpsvCGnHOwwTGMx5g54Bfo3XiUUqoCmiy8YeM/IDsV2g+G7r+p+HillPIytyYLERkvIvtF5KCIzC5lfwcRWS8iu0UkTkTaumz/UUR2isgeEbnHnXF61JkE+OENa3ncszqPtlKqTnBbshARX+A14GogGrhFRKJLHPYCsMgY0xuYAzzv2J4MDDbGxAADgdki0tpdsXrUuqetocd736yTGSml6gx33lkMAA4aYw4bY/KApcDEEsdEA187ljcU7jfG5Bljch3bA90cp+cc3Qx7VlhTpGpXWaVUHeLOD+E2wDGX9UTHNle7gBscy5OAMBFpDiAi7URkt+M15hljktwYq/vZ7bDaMcbTFfdDk7bejUcpparA22NDPQLMF5EZwDfAcaAAwBhzDOjtqH76TESWG2NOup4sIjOBmQARERHExcV5MPSqueTkf4k+vp3cgKZssfejoJRYMzMza/V78DQtj+K0PIpoWRTnifJwZ7I4DriOX9HWsc3JcbdwA4CIhAKTjTFnSx4jIj8DVwLLS+xbACwAiI2NNSNGjKjht1BD8s/DP+8FIHD837iy39WlHhYXF0etfQ9eoOVRnJZHES2L4jxRHu6shtoKdBaRKBEJAKYCn7seICItRKQwhseB9xzb24pII8dyU2AosN+NsbrX969BeiJE9IKYad6ORimlqsxtycIYYwPuA1YD+4Blxpg9IjJHRAofLhgB7BeRA0AE8Kxje3dgs4jsAv4LvGCM+cldsbpVxklrjm2Acc/oNKlKqTrJrW0WxphVwKoS255yWV5Oiaolx/a1QG93xuYxG56FvEzocjV0HOHtaJRSqlrqR5fU2urEz7DjX+DjB2P/5u1olFKq2jRZuIsx1qiyxg6xv4MWnb0dkVJKVZsmC3eJXwOH4yCoCYy4YKQTpZSqUzRZuENBPqx50loe/icIbubdeJRS6iJpsnCH7e9DygFo1hEu/723o1FKqYumyaKmnT9bYq6KAO/Go5RSNUCTRU3b+AKcT4MOQ6Dbtd6ORimlaoQmi5qUdhg2v2Ut61wVSql6RJNFTSqcq6LPLdC6r7ejUUqpGqPJoqYc2QR7V1pzVYz6X29Ho5RSNUqTRU2w22H1E9bykAehSclpO5RSqm7TZFETfvoYknZAaCsY8oC3o1FKqRqnyeJi5WXD+r9ay6OfgoAQ78ajlFJuoMniYn3/GqQfh1a9rYZtpZSqh7w9rWrdlnHCZa6KZ8FHc68qX35+PomJieTk5FT53CZNmrBv3z43RFX3aFkUV5nyCAoKom3btvj7+1frGposLsbXz0B+FnS9BqKGeTsaVQckJiYSFhZGZGQkUsXncDIyMggLC3NTZHWLlkVxFZWHMYbU1FQSExOJioqq1jX0q3B1nfgJdvyfNVfFmDnejkbVETk5OTRv3rzKiUKpiyEiNG/evFp3tIU0WVSHMY6ussYaKLDFZd6OSNUh1UkUL6094IZIVENysV9Q3JosRGS8iOwXkYMicsGkDiLSQUTWi8huEYkTkbaO7TEi8r2I7HHsu9mdcVbZga/g128gKByGP+btaFQD8Mr6+Bp5nZEjR7J69epi215++WVmzZpV7nmhoaEAJCUlMWXKlFKPGTFiBNu2bSv3dV5++WWys7Od6xMmTODs2bOVCb1UP/zwA1FRUcTExBATE0NoaChdu3YlJiaG22+/vUqvZbfbmTt3brnHbNu2DRFh3bp11Y65rnJbshARX+A14GogGrhFRKJLHPYCsMgY0xuYAzzv2J4N3G6M6QGMB14WkXB3xVolrnNVjJitc1WoOuWWW25h6dKlxbYtXbqUW26pXE++1q1bs3z58mpfv2SyWLVqFeHh1f+v/eWXX/LCCy+wc+dOdu7cSWxsLIsXL2bnzp0sWrSoSq9VmWSxZMkShg4dypIlS6odc2XYbDa3vn51uPPOYgBw0Bhz2BiTBywFJpY4Jhr42rG8oXC/MeaAMSbesZwEnAJaujHWytv2HqQehGadrOlSlapDpkyZwhdffEFeXh4ACQkJJCUlceWVV5KZmcno0aPp168fvXr1YuXKlRecn5CQQM+ePQE4f/48U6dOpXv37kyaNInz5887j5s1axaxsbH06NGDv/zlLwC8+uqrJCUlMXLkSEaOHAlAZGQkKSkpALz44ov07NmTnj178vLLLzuv1717d37/+9/To0cPxo4dW+w669ev56qrrirz/dpsNh5++GEGDBhA7969eeeddwA4fvw4Q4cOJSYmhp49e7Jp0yZmz55NRkZGmXcldrudTz75hA8++IAvv/zSWYYACxcupHfv3vTp04c77rgDgBMnTjBx4kTn9s2bN3Pw4EFiYmKc582dO5dnnnkGgKFDh/LQQw8RGxvL/PnzWblyJQMHDqRv376MHTuWU6dOAVZj9m9/+1t69+5N7969+eyzz1i4cCGPPPKI83XfeOMNHn300TLLpTrc2RuqDXDMZT0RGFjimF3ADcArwCQgTESaG2NSCw8QkQFAAHDIjbFWzvkzEOe4+Rn7N52rQl2UyNlfVOn4Xs9+U6njEuZeU+a+Zs2aMWDAAL788ksmTpzI0qVLuemmmxARgoKCWLFiBY0bNyYlJYVBgwbxm9/8psy67jfeeIPg4GD27dvH7t276devn3Pfs88+S7NmzSgoKGD06NHs3r2bBx54gBdffJENGzbQokWLYq+1fft2Fi5cyObNmzHGMHDgQIYPH07Tpk2Jj49nyZIlvP3229x000188sknTJw4kZSUFPz9/WnSpEmZ73fBggVccsklbNmyhdzcXAYNGsTYsWNZsmQJ1113HX/6058oKCjg/PnzDBgwgHfeeYedO3eW+lobN26ka9eudOzYkaFDhzrLcNeuXcybN49NmzbRrFkz0tLSALj33nsZM2YM9913HzabjezsbOcHflkKCgqcVXlnzpxxlv+bb77JP/7xD+bNm8fTTz9Ny5Yt2b17N8YYzp49S3Z2NsOGDWPu3Ln4+fmxcOFCPvjgg3KvVVXe7jr7CDBfRGYA3wDHgYLCnSJyKfAv4LfGGHvJk0VkJjATICIigri4OLcG2+nge7Q7f4Yz4T3ZlRwMJ2r2epmZmW5/D3VJfSyPJk2akJGR4dZrVPT6119/Pf/6178YNWoUH374IfPnzycjI4P8/Hxmz57Npk2b8PHx4fjx4xw6dIiIiAjn62ZmZmK328nIyODrr7/mnnvuISMjg6ioKHr27ElWVhYZGRksWrSI999/H5vNxokTJ9i+fTtRUVEYY8jMzCQwMBDAub5u3TomTJiA3W79N7/mmmtYu3YtEyZMoEOHDnTq1ImMjAx69uzJ/v37KSgoYOXKlQwfPrzY+y0oKHDGAFY11/79+/nwww8BSE9PZ9euXfTo0YMHH3yQc+fOce2119KrVy/nOWWV36JFi7j++uvJyMhg4sSJLFq0iFGjRrFq1Squv/56/P39ycjIcP7esGEDb7/9tvP1RKRY+QHk5uZis9nIyMigoKCA6667zrlv3759PPnkk5w8eZLc3Fwuu+wyMjIyWLNmDR9++KHzOD8/P0JCQhg8eDCffvopkZGRALRt2/aC95KTk1Pt/1PuTBbHgXYu620d25wcVUw3AIhIKDDZGHPWsd4Y+AL4szHmh9IuYIxZACwAiI2NNSNGjKjht+Ai9RB8swoQmt70GiNax1R4SlXFxcXh1vdQx9TH8ti3b5+zP3x5dwAlRc7+gp/+PKxGni2YOnUqTzzxBPHx8eTk5DBsmPWM0Pvvv8+5c+fYsWMH/v7+REZG4ufn57xmWFgYoaGh+Pj4EBYWhp+fH8HBwc79Pj4+hISEkJKSwvz589m6dStNmzZlxowZiAhhYWGICKGhoc5zCteDgoIIDAx0bg8MDCQoKIjQ0FAaNWrk3B4cHExmZia+vr7ExcXx8MMPFysTX19fQkJCnNt8fX158803GT169AXl0LdvX7744gtmzZrFY489xs033+x8nyXl5+fz73//mzVr1jBv3jzsdjtnz57Fx8eHoKAgcnJyLjhPRGjcuDF+fkUfs4XtM4XHGmOc79vX15eWLVs69z322GM88cQTTJgwgXXr1jF37lzCwsLw8fEpVoZgJbhZs2bx4osvEhkZyV133VXq+wgKCqJv3+pNn+DONoutQGcRiRKRAGAq8LnrASLSQkQKY3gceM+xPQBYgdX4Xf3WtJq07i9gz4eYaeCGRKGUp4SGhjJy5EjuvPPOYg3b586d45JLLsHf358NGzZw5MiRcl9n2LBhzm/sP//8M7t37wasb+8hISE0adKEkydP8uWXXzrPCQsLK/Wb+5VXXslnn31GdnY2WVlZrFixgiuvvLLMaxtj2L17d7H6/9KMGzeO119/3dlgvH//fs6fP8+RI0do1aoVM2fO5I477mDHjh3OD/XSGpfXrl3L5ZdfzrFjx0hISODo0aNcd911rFy5klGjRvHRRx85q58Kf48cOZI333wTsO540tPTadWqFUlJSZw5c4acnBy++KLsqshz587Rpk0bjDHFqpTGjBnDa6+95iyHM2fOADBkyBAOHTrExx9/7Ex8NcltycIYYwPuA1YD+4Blxpg9IjJHRH7jOGwEsF9EDgARwLOO7TcBw4AZIrLT8eO9T+iEb2Hfv8E/GEY96bUwlKopt9xyC7t27SqWLKZPn862bdvo1asXixYtolu3buW+xqxZs8jMzKR79+489dRT9O/fH4A+ffrQt29funXrxrRp0xgyZIjznJkzZzJ+/HhnA3ehfv36MWPGDAYMGMDAgQO56667yv0GvGPHDvr27VvhswN33303nTt3djZkz5o1C5vNxvr1651xfvrpp9x///0A/O53v6N3794XNHAvWbKESZMmFds2efJklixZQp8+fXjssccYNmwYMTExzobl+fPns3r1anr16kVsbCy//PILQUFBPPHEE8TGxjJ27Fiio0t2EC3y9NNPM2nSJC6//HJnVSDAX/7yF06ePEnPnj2JiYlh48aNzn1Tpkxh2LBh5bbjVJsxpl789O/f37hFQYExb15pzF8aG7Phefdcw2HDhg1uff26pj6Wx969e6t1Xoc//cekp6fXcDR115NPPmmWLFni7TBqjcK/jXHjxpm4uLgyjyvt7w/YZirxGevtBu7ab/dHkLwLwlrDFfd7OxrVQD04urO3Q6hVHnvsMR0bykVqair9+vUjNjaW4cOHu+UamizKk5cF6x3jPulcFcqLHhrTxe29qFTd1bx5c+Lja+Yp/7Lo2FDl2TQfMpLg0j7Qu3aNOKKUUp6kyaIs6cnwnfUUKeOe07kqlFINmn4CluXrZyA/G7pdC5FDvR2NUkp5lSaL0iTthJ2Lwcdf56pQSik0WVzIGMeosgYGzITmnbwdkWrIvn0JMk7W2MulpqY6h/Nu1aoVbdq0ca67DoxXnjvuuIP9+/eXe8xrr73G4sWLayJkAE6ePImfn59zIEDledobqqT9qyBhIzRqCsNrdtRGpaos8xR89wqMf65GXq558+bOgfKefvppQkNDi41WCkXPXvmU0U63cOHCCq9z7733XnywLpYtW8bgwYNZsmQJd911V42+tiubzVZseA5VRO8sXNnyYM3/WssjHrcShlLeNORB2PVhjd5dlObgwYNER0czffp0evToQXJyMjNnznQOMz5nTlF17NChQ9m5cyc2m43w8HBmz55Nnz59GDx4sHNU1SeffNI5zPjQoUOZPXs2AwYMoGvXrmzatAmArKwsJk+eTHR0NFOmTCE2NrbMEV+XLFnCyy+/zOHDh0lOTnZu/+KLL+jXrx99+vRh7NixQOlDeBfGWmjp0qXOpHPrrbcya9YsBgwYwBNPPMEPP/zA4MGD6du3L0OGDHF2SbXZbDz00EP07NmT3r178/rrr7NmzZpik0F9+eWX3HjjjRf971EbaQp1te1dSDsEzS+D2Du9HY2q756uwpAM/+hCpR9Be/pcdaLhl19+YdGiRcTGxgLWXAvNmjXDZrMxcuRIpkyZcsHwFOfOnWP48OHMnTuXhx9+mPfee4/Zsy+YFBNjDFu2bOHzzz9nzpw5fPXVV/zzn/+kVatWfPLJJ+zatavYEOeuEhISSEtLo3///tx4440sW7aMO++8kxMnTjBr1iw2btxIhw4dnGMylTaEd0WSk5P54Ycf8PHx4dy5c2zcuBE/Pz+++uornnzyST766CPeeOMNkpKS2LVrF76+vqSlpREeHs59991HamoqzZs3Z+HChdx5Z/387NA7i0LZaRDnmCVr7DPg6+/deJTysE6dOjkTBVjf5vv160e/fv3Yt28fe/fuveCcRo0acfXVVwPQv39/EhISSn3tG2644YJjvv32W6ZOnQpY40n16NGj1HOXLl3qHBhv6tSpzlnqvv/+e0aOHEmHDh0Aa64OgHXr1jmrwUSEpk0rriG48cYbndVuZ8+eZfLkyfTs2ZNHHnmEPXv2OF/3nnvuwdfX13k9Hx8fpk+fzocffkhaWhrbt2933uHUN3pn8e1L0Gea9UxFzlmIGgZdxns7KtUQVOYOIC8LFoyEoX8ko9N1bh3iIiSkaISC+Ph4XnnlFbZs2UJ4eDi33norOTk5F5wTEFA0AZivr2+Z04EWzl9R3jFlWbJkCSkpKc6RV5OSkvj111+r9Bo+Pj5YwyBZSr4X1/f+5z//mXHjxvGHP/yBgwcPMn58+Z8Hd955J5MnTwbg5ptvdiaT+kbvLDJPwbq/wpYFgMDYZ6GCkSyV8pgvHoG2sdbQ+B6Unp5OWFgYjRs3Jjk5mdWrV9f4NYYMGcKyZcsA+Omnn0q9c9m7dy82m43jx4+TkJBAQkICjz76KJ988glXXHFFsaHUC6uhShvC28fHxznrnt1uZ8WKFWXGVTg0OFhzfBQaM2YMb775JgUFBcWu165dO1q0aMHcuXOZMWPGxRVKLabJYsiD8NMysNug73S4tLe3I1LKsmMxJP0IE/7u8Uv369eP6OhounXrxu23315smPGacv/993P8+HGio6P561//SnR09AVDa5c1NPjy5cuJiIjgjTfeYOLEifTp04fp06cDZQ/hPW/ePMaNG8cVV1xB27Zty4zrT3/6E48++ij9+vUrdjdy991306pVK+ec2oWJDmDatGlERUXRpUuXiy6XWqsyQ9PWhZ9qD1F++L/W8ON/bWbMuaTqvUYNqY9Dcl+M+lgeVRqifONLxpwsOr6+DVGen59vzp8/b4wx5sCBAyYyMtLk5+dX6tzaVhZ33323ef/99712/cqWhw5RfjHOn4XAxpCbDi+WmOxl+GwY+bh34lJq6B+9HYFbZWZmMnr0aGw2G8YY3nrrrTr5jENMTAxNmzbl1Vdf9XYoblX3/mVq2mWjIfQSGPs36D/D29Eo1WCEh4ezfft2b4dx0cp6NqS+0TaLLx6BdgM1USilVDncmixEZLyI7BeRgyJywZM6ItJBRNaLyG4RiRORti77vhKRsyLyH7cF6MUGRNVwGZdGU6U85WL/7tyWLETEF3gNuBqIBm4RkZKzk78ALDLG9AbmAM+77Ps7cJu74gMg6zTc+L7OgKc8JigoiNTUVE0YyqOMMaSmphIUFFTt13Bnm8UA4KAx5jCAiCwFJgKunamjgYcdyxuAzwp3GGPWi8gIN8ZX7xsQVe3Ttm1bEhMTOX36dJXPzcnJuaj/7PWJlkVxlSmPoKCgcrsMV8SdyaINcMxlPREYWOKYXcANwCvAJCBMRJobY1LdGJdSXuPv709UVFS1zo2Li6Nv3741HFHdpGVRnCfKw9u9oR4B5ovIDOAb4DhQUNmTRWQmMBMgIiKCuLg4N4ToOZmZmXX+PdQkLY/itDyKaFkU54nycGeyOA60c1lv69jmZIxJwrqzQERCgcnGmIqHiCw6fwGwACA2NtaMGDHiIkP2rri4OOr6e6hJWh7FaXkU0bIozhPl4c7eUFuBziISJSIBwFTgc9cDRKSFiBTG8DjwnhvjUUopVU3izl4ZIjIBeBnwBd4zxjwrInOwHi//XESmYPWAMljVUPcaY3Id524EugGhQCrwO2NMmaOZichp4Ijb3oxntABSvB1ELaLlUZyWRxEti+Iupjw6GGNaVnSQW5OFqhoR2WaMia34yIZBy6M4LY8iWhbFeaI89AlupZRSFdJkoZRSqkKaLGqXBd4OoJbR8ihOy6OIlkVxbi8PbbNQSilVIb2zUEopVSFNFrWAiLQTkQ0isldE9ojIg96OydtExFdEdrh11OE6QkTCRWS5iPwiIvtEZLC3Y/ImEXnI8f/kZxFZIiINapAoEXlPRE6JyM8u25qJyFoRiXf8blrT19VkUTvYgP8xxkQDg4B7Sxmht6F5ENjn7SBqiVeAr4wx3YA+NOByEZE2wANArDGmJ9YzXFO9G5XHvQ+ML7FtNrDeGNMZWO9Yr1GaLGoBY0yyMeZHx3IG1odBG+9G5T2OeU2uAd7xdizeJiJNgGHAuwDGmLyqDIlTT/kBjUTEDwgGkrwcj0cZY74B0kpsngh84Fj+ALi+pq+ryaKWEZFIoC+w2buReNXLwGOA3duB1AJRwGlgoaNa7h0RabATsBhjjmPNg3MUSAbOGWPWeDeqWiHCGJPsWD4BRNT0BTRZ1CKOwRQ/Af5ojEn3djzeICLXAqeMMXV/cuaa4Qf0A94wxvQFsnBDFUNd4aiLn4iVRFsDISJyqy5ueXkAAAMgSURBVHejql2M1cW1xru5arKoJUTEHytRLDbGfOrteLxoCPAbEUkAlgKjROT/vBuSVyUCicaYwjvN5VjJo6G6CvjVGHPaGJMPfApc4eWYaoOTInIpgOP3qZq+gCaLWkBEBKtOep8x5kVvx+NNxpjHjTFtjTGRWA2XXxtjGuw3R2PMCeCYiHR1bBpN8dkmG5qjwCARCXb8vxlNA27wd/E58FvH8m+BlTV9AU0WtcMQrPnGR4nITsfPBG8HpWqN+4HFIrIbiAGe83I8XuO4w1oO/Aj8hPUZ1qCe5haRJcD3QFcRSRSR3wFzgTEiEo919zW3xq+rT3ArpZSqiN5ZKKWUqpAmC6WUUhXSZKGUUqpCmiyUUkpVSJOFUkqpCmmyUKoCIlLg0qV5p4jU2BPUIhLpOnqoUrWV3/9v7/5dm4yiMI5/H4tDQCiiIIJKB53EX+Bf4OroUMTJTQd1kvoHODlGu+gkKLjZsShVRFDQRSuu0q2CHRQCEqQ8Djm1QRve2hJfhOcDITcn4XLf6eS8N7mn7QVE/Ae+2z7Z9iIi2pTKImKLJC1JuiXpg6Q3kg5XfErSM0mLkhYkHar4PkmPJb2vx9oxFROS7lWPhieSOvX5q9XjZFHSo5YuMwJIsojYjM5vt6Gmh977ZvsYcIfBabkAt4H7to8DD4FuxbvAC9snGJzv9LHiR4BZ20eBr8C5it8ATtU8l8Z1cRGbkX9wRzSQ1LO9a4P4EnDG9qc6CPKz7T2SVoD9tn9UfNn2XklfgAO2+0NzTAFPq2kNkmaAnbZvSpoHesAcMGe7N+ZLjRgplUXE9njE+G/0h8arrO8lngVmGVQhb6vZT0Qrkiwitmd66Pl1jV+x3urzAvCyxgvAZfjVY3xy1KSSdgAHbT8HZoBJ4I/qJuJfyTeViGYdSe+GXs/bXvv57O46DbYPnK/YFQad7a4z6HJ3seLXgLt1Sugqg8SxzMYmgAeVUAR000412pQ9i4gtqj2L07ZX2l5LxLjlNlRERDRKZREREY1SWURERKMki4iIaJRkERERjZIsIiKiUZJFREQ0SrKIiIhGPwH1VchvYwbDdQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Plotting our accuracy charts\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "history_dict = history.history\n",
    "\n",
    "acc_values = history_dict['acc']\n",
    "val_acc_values = history_dict['val_acc']\n",
    "epochs = range(1, len(loss_values) + 1)\n",
    "\n",
    "line1 = plt.plot(epochs, val_acc_values, label='Validation/Test Accuracy')\n",
    "line2 = plt.plot(epochs, acc_values, label='Training Accuracy')\n",
    "plt.setp(line1, linewidth=2.0, marker = '+', markersize=10.0)\n",
    "plt.setp(line2, linewidth=2.0, marker = '4', markersize=10.0)\n",
    "plt.xlabel('Epochs') \n",
    "plt.ylabel('Accuracy')\n",
    "plt.grid(True)\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 7A - Saving our Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Saved\n"
     ]
    }
   ],
   "source": [
    "model.save(\"/home/deeplearningcv/DeepLearningCV/Trained Models/8_mnist_simple_cnn_10_Epochs.h5\")\n",
    "print(\"Model Saved\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 7B - Loading  our Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import load_model\n",
    "\n",
    "classifier = load_model('/home/deeplearningcv/DeepLearningCV/Trained Models/8_mnist_simple_cnn_10_Epochs.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 8 - Lets input some of our test data into our classifer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "import cv2\n",
    "import numpy as np\n",
    "\n",
    "\n",
    "def draw_test(name, pred, input_im):\n",
    "    BLACK = [0,0,0]\n",
    "    expanded_image = cv2.copyMakeBorder(input_im, 0, 0, 0, imageL.shape[0] ,cv2.BORDER_CONSTANT,value=BLACK)\n",
    "    expanded_image = cv2.cvtColor(expanded_image, cv2.COLOR_GRAY2BGR)\n",
    "    cv2.putText(expanded_image, str(pred), (152, 70) , cv2.FONT_HERSHEY_COMPLEX_SMALL,4, (0,255,0), 2)\n",
    "    cv2.imshow(name, expanded_image)\n",
    "\n",
    "\n",
    "for i in range(0,10):\n",
    "    rand = np.random.randint(0,len(x_test))\n",
    "    input_im = x_test[rand]\n",
    "\n",
    "    imageL = cv2.resize(input_im, None, fx=4, fy=4, interpolation = cv2.INTER_CUBIC)\n",
    "    input_im = input_im.reshape(1,28,28,1) \n",
    "    \n",
    "    ## Get Prediction\n",
    "    res = str(classifier.predict_classes(input_im, 1, verbose = 0)[0])\n",
    "\n",
    "    draw_test(\"Prediction\", res, imageL) \n",
    "    cv2.waitKey(0)\n",
    "\n",
    "cv2.destroyAllWindows()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Putting All Together!\n",
    "We don't need to run each section of code separately. Once we know it all works as it's supposed to, we can put all te pieces together and start training our model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x_train shape: (60000, 28, 28, 1)\n",
      "60000 train samples\n",
      "10000 test samples\n",
      "Number of Classes: 10\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 24, 24, 64)        18496     \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "dropout_1 (Dropout)          (None, 12, 12, 64)        0         \n",
      "_________________________________________________________________\n",
      "flatten_1 (Flatten)          (None, 9216)              0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 128)               1179776   \n",
      "_________________________________________________________________\n",
      "dropout_2 (Dropout)          (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 10)                1290      \n",
      "=================================================================\n",
      "Total params: 1,199,882\n",
      "Trainable params: 1,199,882\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n",
      "Train on 60000 samples, validate on 10000 samples\n",
      "Epoch 1/10\n",
      "60000/60000 [==============================] - 367s 6ms/step - loss: 0.2656 - acc: 0.9180 - val_loss: 0.0651 - val_acc: 0.9781\n",
      "Epoch 2/10\n",
      "60000/60000 [==============================] - 450s 8ms/step - loss: 0.0918 - acc: 0.9727 - val_loss: 0.0403 - val_acc: 0.9863\n",
      "Epoch 3/10\n",
      "60000/60000 [==============================] - 475s 8ms/step - loss: 0.0683 - acc: 0.9799 - val_loss: 0.0328 - val_acc: 0.9876\n",
      "Epoch 4/10\n",
      "60000/60000 [==============================] - 441s 7ms/step - loss: 0.0560 - acc: 0.9838 - val_loss: 0.0310 - val_acc: 0.9888\n",
      "Epoch 5/10\n",
      "60000/60000 [==============================] - 448s 7ms/step - loss: 0.0471 - acc: 0.9855 - val_loss: 0.0301 - val_acc: 0.9899\n",
      "Epoch 6/10\n",
      "60000/60000 [==============================] - 458s 8ms/step - loss: 0.0413 - acc: 0.9872 - val_loss: 0.0302 - val_acc: 0.9905\n",
      "Epoch 7/10\n",
      "60000/60000 [==============================] - 390s 6ms/step - loss: 0.0373 - acc: 0.9887 - val_loss: 0.0274 - val_acc: 0.9911\n",
      "Epoch 8/10\n",
      "60000/60000 [==============================] - 329s 5ms/step - loss: 0.0343 - acc: 0.9895 - val_loss: 0.0286 - val_acc: 0.9903\n",
      "Epoch 9/10\n",
      "60000/60000 [==============================] - 199s 3ms/step - loss: 0.0313 - acc: 0.9904 - val_loss: 0.0274 - val_acc: 0.9904\n",
      "Epoch 10/10\n",
      "60000/60000 [==============================] - 200s 3ms/step - loss: 0.0299 - acc: 0.9909 - val_loss: 0.0272 - val_acc: 0.9915\n",
      "Test loss: 0.027194885472155875\n",
      "Test accuracy: 0.9915\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist\n",
    "from keras.utils import np_utils\n",
    "import keras\n",
    "from keras.datasets import mnist\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "from keras.optimizers import SGD \n",
    "\n",
    "# Training Parameters\n",
    "batch_size = 128\n",
    "epochs = 10\n",
    "\n",
    "# loads the MNIST dataset\n",
    "(x_train, y_train), (x_test, y_test)  = mnist.load_data()\n",
    "\n",
    "# Lets store the number of rows and columns\n",
    "img_rows = x_train[0].shape[0]\n",
    "img_cols = x_train[1].shape[0]\n",
    "\n",
    "# Getting our date in the right 'shape' needed for Keras\n",
    "# We need to add a 4th dimenion to our date thereby changing our\n",
    "# Our original image shape of (60000,28,28) to (60000,28,28,1)\n",
    "x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "\n",
    "# store the shape of a single image \n",
    "input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "# change our image type to float32 data type\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "\n",
    "# Normalize our data by changing the range from (0 to 255) to (0 to 1)\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "print('x_train shape:', x_train.shape)\n",
    "print(x_train.shape[0], 'train samples')\n",
    "print(x_test.shape[0], 'test samples')\n",
    "\n",
    "# Now we one hot encode outputs\n",
    "y_train = np_utils.to_categorical(y_train)\n",
    "y_test = np_utils.to_categorical(y_test)\n",
    "\n",
    "# Let's count the number columns in our hot encoded matrix \n",
    "print (\"Number of Classes: \" + str(y_test.shape[1]))\n",
    "\n",
    "num_classes = y_test.shape[1]\n",
    "num_pixels = x_train.shape[1] * x_train.shape[2]\n",
    "\n",
    "# create model\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss = 'categorical_crossentropy',\n",
    "              optimizer = SGD(0.01),\n",
    "              metrics = ['accuracy'])\n",
    "\n",
    "print(model.summary())\n",
    "\n",
    "history = model.fit(x_train, y_train,\n",
    "          batch_size=batch_size,\n",
    "          epochs=epochs,\n",
    "          verbose=1,\n",
    "          validation_data=(x_test, y_test))\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing Our Model\n",
    "- First let's re-create our model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import keras\n",
    "from keras.models import Sequential\n",
    "from keras.utils.vis_utils import plot_model\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.image as mpimg\n",
    "import numpy as np\n",
    "from keras.utils import np_utils\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "\n",
    "input_shape = (28,28,1)\n",
    "num_classes = 10\n",
    "\n",
    "model = Sequential()\n",
    "\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "print(model.summary()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generating the diagram of the model architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Save our model diagrams to this path\n",
    "model_diagrams_path = '/home/deeplearningcv/DeeplearningCV/Trained Models/'\n",
    "\n",
    "# Generate the plot\n",
    "plot_model(model, to_file = model_diagrams_path + 'model_plot.png',\n",
    "           show_shapes = True,\n",
    "           show_layer_names = True)\n",
    "\n",
    "# Show the plot here\n",
    "img = mpimg.imread(model_diagrams_path + 'model_plot.png')\n",
    "plt.figure(figsize=(30,15))\n",
    "imgplot = plt.imshow(img) "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}