Part_3_Training_Neural_Networks_(Exercise).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "1gRj-x7h332N"
   },
   "source": [
    "# Training Neural Networks\n",
    "\n",
    "The network we built in the previous part isn't so smart, it doesn't know anything about our handwritten digits. Neural networks with non-linear activations work like universal function approximators. There is some function that maps your input to the output. For example, images of handwritten digits to class probabilities. The power of neural networks is that we can train them to approximate this function, and basically any function given enough data and compute time.\n",
    "\n",
    "<img src=\"assets/function_approx.png\" width=500px>\n",
    "\n",
    "At first the network is naive, it doesn't know the function mapping the inputs to the outputs. We train the network by showing it examples of real data, then adjusting the network parameters such that it approximates this function.\n",
    "\n",
    "To find these parameters, we need to know how poorly the network is predicting the real outputs. For this we calculate a **loss function** (also called the cost), a measure of our prediction error. For example, the mean squared loss is often used in regression and binary classification problems\n",
    "\n",
    "$$\n",
    "\\large \\ell = \\frac{1}{2n}\\sum_i^n{\\left(y_i - \\hat{y}_i\\right)^2}\n",
    "$$\n",
    "\n",
    "where $n$ is the number of training examples, $y_i$ are the true labels, and $\\hat{y}_i$ are the predicted labels.\n",
    "\n",
    "By minimizing this loss with respect to the network parameters, we can find configurations where the loss is at a minimum and the network is able to predict the correct labels with high accuracy. We find this minimum using a process called **gradient descent**. The gradient is the slope of the loss function and points in the direction of fastest change. To get to the minimum in the least amount of time, we then want to follow the gradient (downwards). You can think of this like descending a mountain by following the steepest slope to the base.\n",
    "\n",
    "<img src='assets/gradient_descent.png' width=350px>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "C-bEg-Zz4Q7z"
   },
   "source": [
    "## Backpropagation\n",
    "\n",
    "For single layer networks, gradient descent is straightforward to implement. However, it's more complicated for deeper, multilayer neural networks like the one we've built. Complicated enough that it took about 30 years before researchers figured out how to train multilayer networks.\n",
    "\n",
    "Training multilayer networks is done through **backpropagation** which is really just an application of the chain rule from calculus. It's easiest to understand if we convert a two layer network into a graph representation.\n",
    "\n",
    "<img src='assets/backprop_diagram.png' width=550px>\n",
    "\n",
    "In the forward pass through the network, our data and operations go from bottom to top here. We pass the input $x$ through a linear transformation $L_1$ with weights $W_1$ and biases $b_1$. The output then goes through the sigmoid operation $S$ and another linear transformation $L_2$. Finally we calculate the loss $\\ell$. We use the loss as a measure of how bad the network's predictions are. The goal then is to adjust the weights and biases to minimize the loss.\n",
    "\n",
    "To train the weights with gradient descent, we propagate the gradient of the loss backwards through the network. Each operation has some gradient between the inputs and outputs. As we send the gradients backwards, we multiply the incoming gradient with the gradient for the operation. Mathematically, this is really just calculating the gradient of the loss with respect to the weights using the chain rule.\n",
    "\n",
    "$$\n",
    "\\large \\frac{\\partial \\ell}{\\partial W_1} = \\frac{\\partial L_1}{\\partial W_1} \\frac{\\partial S}{\\partial L_1} \\frac{\\partial L_2}{\\partial S} \\frac{\\partial \\ell}{\\partial L_2}\n",
    "$$\n",
    "\n",
    "**Note:** I'm glossing over a few details here that require some knowledge of vector calculus, but they aren't necessary to understand what's going on.\n",
    "\n",
    "We update our weights using this gradient with some learning rate $\\alpha$. \n",
    "\n",
    "$$\n",
    "\\large W^\\prime_1 = W_1 - \\alpha \\frac{\\partial \\ell}{\\partial W_1}\n",
    "$$\n",
    "\n",
    "The learning rate $\\alpha$ is set such that the weight update steps are small enough that the iterative method settles in a minimum."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "worDfYepJH6j"
   },
   "source": [
    "## Import Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jFdhxHwr57Yn"
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "tfds.disable_progress_bar()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import logging\n",
    "logger = tf.get_logger()\n",
    "logger.setLevel(logging.ERROR)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "id": "yCtUH8paXqBQ",
    "outputId": "1a4c93cf-21a8-4574-d121-f238912d28e8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Using:\n",
      "\t• TensorFlow version: 2.3.0\n",
      "\t• tf.keras version: 2.4.0\n",
      "\t• GPU device not found. Running on CPU\n"
     ]
    }
   ],
   "source": [
    "print('Using:')\n",
    "print('\\t\\u2022 TensorFlow version:', tf.__version__)\n",
    "print('\\t\\u2022 tf.keras version:', tf.keras.__version__)\n",
    "print('\\t\\u2022 Running on GPU' if tf.test.is_gpu_available() else '\\t\\u2022 GPU device not found. Running on CPU')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "3zQV8MLaJOjN"
   },
   "source": [
    "## Load the Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 360
    },
    "colab_type": "code",
    "id": "Att74swb7Ol0",
    "outputId": "a98f6ee1-9881-4d8d-8766-b8b00a2cb4f8"
   },
   "outputs": [],
   "source": [
    "training_set, dataset_info = tfds.load('mnist', split='train', as_supervised = True, with_info = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IiSe5BPrJquE"
   },
   "source": [
    "## Create Pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9r4EMOdT9pM3"
   },
   "outputs": [],
   "source": [
    "def normalize(image, label):\n",
    "    image = tf.cast(image, tf.float32)\n",
    "    image /= 255\n",
    "    return image, label\n",
    "\n",
    "num_training_examples = dataset_info.splits['train'].num_examples\n",
    "\n",
    "batch_size = 64\n",
    "\n",
    "training_batches = training_set.cache().shuffle(num_training_examples//4).batch(batch_size).map(normalize).prefetch(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "K9SC4gnUJucy"
   },
   "source": [
    "## Build the Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "Mo2DfMVvAdbd"
   },
   "outputs": [],
   "source": [
    "model = tf.keras.Sequential([\n",
    "        tf.keras.layers.Flatten(input_shape = (28, 28, 1)),\n",
    "        tf.keras.layers.Dense(128, activation = 'relu'),\n",
    "        tf.keras.layers.Dense(64, activation = 'relu'),\n",
    "        tf.keras.layers.Dense(10, activation = 'softmax')\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5TCpaAlcKCDB"
   },
   "source": [
    "## Getting the Model Ready For Training\n",
    "\n",
    "Before we can train our model we need to set the parameters we are going to use to train it. We can configure our model for training using the `.compile` method. The main parameters we need to specify in the `.compile` method are:\n",
    "\n",
    "* **Optimizer:** The algorithm that we'll use to update the weights of our model during training. Throughout these lessons we will use the [`adam`](http://arxiv.org/abs/1412.6980) optimizer. Adam is an optimization of the stochastic gradient descent algorithm. For a full list of the optimizers available in `tf.keras` check out the [optimizers documentation](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/optimizers#classes).\n",
    "\n",
    "\n",
    "* **Loss Function:** The loss function we are going to use during training to measure the difference between the true labels of the images in your dataset and the predictions made by your model. In this lesson we will use the `sparse_categorical_crossentropy` loss function. We use the `sparse_categorical_crossentropy` loss function when our dataset has labels that are integers, and the `categorical_crossentropy` loss function when our dataset has one-hot encoded labels. For a full list of the loss functions available in `tf.keras` check out the [losses documentation](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/losses#classes).\n",
    "\n",
    "\n",
    "* **Metrics:** A list of metrics to be evaluated by the model during training. Throughout these lessons we will measure the `accuracy` of our model. The `accuracy` calculates how often our model's predictions match the true labels of the images in our dataset. For a full list of the metrics available in `tf.keras` check out the [metrics documentation](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/metrics#classes).\n",
    "\n",
    "These are the main parameters we are going to set throught these lesson. You can check out all the other configuration parameters in the [TensorFlow documentation](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/Model#compile)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "jYv3pv5-InR1"
   },
   "outputs": [],
   "source": [
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Y5CjYa8ES3OI"
   },
   "source": [
    "## Taking a Look at the Loss and Accuracy Before Training\n",
    "\n",
    "Before we train our model, let's take a look at how our model performs when it is just using random weights. Let's take a look at the `loss` and `accuracy` values when we pass a single batch of images to our un-trained model. To do this, we will use the `.evaluate(data, true_labels)` method. The `.evaluate(data, true_labels)` method compares the predicted output of our model on the given `data` with the given `true_labels` and returns the `loss` and `accuracy` values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "u_7aijzvJQZ7",
    "outputId": "f66f355e-d030-4c30-e50c-7bba125a20cf"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2/2 [==============================] - 0s 1ms/step - loss: 2.3984 - accuracy: 0.1562\n",
      "\n",
      "Loss before training: 2.398\n",
      "Accuracy before training: 15.625%\n"
     ]
    }
   ],
   "source": [
    "for image_batch, label_batch in training_batches.take(1):\n",
    "    loss, accuracy = model.evaluate(image_batch, label_batch)\n",
    "\n",
    "print('\\nLoss before training: {:,.3f}'.format(loss))\n",
    "print('Accuracy before training: {:.3%}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zvsfbLEMZjZ5"
   },
   "source": [
    "## Training the Model\n",
    "\n",
    "Now let's train our model by using all the images in our training set. Some nomenclature, one pass through the entire dataset is called an *epoch*. To train our model for a given number of epochs we use the `.fit` method, as seen below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "id": "Z-CgmnKBZDjq",
    "outputId": "38ab455c-767a-4705-c172-9d7cc926c239"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "938/938 [==============================] - 3s 3ms/step - loss: 0.2732 - accuracy: 0.9206\n",
      "Epoch 2/5\n",
      "938/938 [==============================] - 1s 1ms/step - loss: 0.1150 - accuracy: 0.9651\n",
      "Epoch 3/5\n",
      "938/938 [==============================] - 1s 1ms/step - loss: 0.0791 - accuracy: 0.9754\n",
      "Epoch 4/5\n",
      "938/938 [==============================] - 1s 1ms/step - loss: 0.0593 - accuracy: 0.9819\n",
      "Epoch 5/5\n",
      "938/938 [==============================] - 1s 1ms/step - loss: 0.0453 - accuracy: 0.9860\n"
     ]
    }
   ],
   "source": [
    "EPOCHS = 5\n",
    "\n",
    "history = model.fit(training_batches, epochs = EPOCHS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IFgG_WfUjCic"
   },
   "source": [
    "The `.fit` method returns a `History` object which contains a record of training accuracy and loss values at successive epochs, as well as validation accuracy and loss values when applicable. We will discuss the history object in a later lesson. \n",
    "\n",
    "With our model trained, we can check out it's predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 243
    },
    "colab_type": "code",
    "id": "ghr7z-SnctRw",
    "outputId": "8e946c9a-56b5-45f4-e79f-c6451ff8b7d5"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1EAAAHXCAYAAABd89BGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de7htZVk3/u+NG5AzIiImJuorQoEHKFQOCppmmYog5Y8k0erNU6aplUfQ0FfLDKnXyBQV9S2VAlEUCcU8ZeBGNBREU0RUREDlIKDC8/tjjCXL5Vp77zH3XGvOxfx8rmteY80xxjPGPcdee+313c94nlGttQAAALBhNpl0AQAAAKuJEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUArFpV1frXrpOuZVZM6ppvzHmr6q1922M29LhVdVS//qOjVcxtmRAFAExcVW1ZVU+vqvdV1aVV9cOqur6qvlZVJ1fVk6pqi0nXuVKq6pJ5v9zPvW6uqquq6uNV9dyq2nLSdc6qPmAdU1X3n3QtTMaaSRcAAMy2qnpMkjcm2Xne6uuT3JJk1/51WJLXVNWRrbWPrHSNE3R9kuv6rzdLskOSA/rXH1TVwa21KyZV3Cry7SRfSnLlgDY/6Ntcusi2o5I8NMklSc7fyNpYhfREAQATU1VHJTk1XYD6UpIjk+zYWtu6tbZtku2TPCHJR5P8QpKHTKbSiXlta23n/rVDkh2TvDJJS/JL6cIn69Fae2FrbffW2t8PaHNK3+b3lrM2VichCgCYiKq6b5IT0v0+8oEkD2itvaO1dtXcPq21H7TW/rW1dnCS30ly7WSqnQ6ttataay9J8pZ+1eOq6hcmWRPMIiEKAJiUVybZPMk3kxzRWrthXTu31t6d5HUbcuCqul1VHVxVr6+qtVX1nar6UVV9q6pOqaqHraPtJv2Yl7P7MUg/rqrvVtUXqurEqnrUIm3uUVX/UFUXV9UN/Ziur1fVR6vqhVW144bUPcA/z/t673l1/HQCharavKpeXFWfr6pr+/XbL6j74Kr6t6q6vL8+l6/v+ixov2dV/Uvf7saquqiqXlpVmy+x/9ZVdXhVvbOqLqiq7/fX6ytV9caquvcynXfJiSXWcY6fm1hibl26W/mS5C0Lxq1d0u93Yv/+5PWc4+X9fp/a0LqYDsZEAQArrqrumuTR/dvjW2s/2JB2rbW2gafYI8n8sVM3JflRkrskOSTJIVX14tbaqxZp+/YkR8x7/4Mk26a7le6X+tcZcxurau90txtu06/6cbqxTL/Yvx6a5LPz24zBN+d9ve0i22+f5GNJ9u3r+eHCHarq2CQv7t+2dJ9zp9x6fV7dWnvhOmrYL93thFsluSZJJblPklck+c2qekRr7boFbY5K8nfz3l+b7j/179W/jqiqQ1prZ435vONyQ5LvpBubtml//vnh/7v98k1JnpLkMVV1x/m9q3OqqpI8uX974jLVyzLREwUATMJB6X75TZLTluH4P0ryniSPSTfeaovW2tZJ7pzkpUluTnJsVT1wfqOqeki6AHVLkucm2ba1tn26UPIL6ULAJxac67XpAtR/Jdm7tbZZa+0O6X7J/9Ukx6ULKOP0i/O+/v4i25+ZZLckT0yydf8Zdk0X7lJVT8ytAervk+zU13yn3Bpy/qKqnrSOGt6Q5ItJ7tta2y7dNXhKulDxoCzea3hVf/z9kmzfj3u7fbrQ+8501+z/VdVWYz7vWLTW3tVa2znJXM/Rn8wbs7Zza+1X+/0+1de4WZLfXeJwD09y93R/Ju9arppZHkIUADAJe/TLm9JNKDFWrbWLW2u/3Vp7f2vtO3M9WK21K1prxyZ5eboQ97QFTR/UL89srR3XWru2b9daa99urb2ttfb8Jdr8SWvts/Nq+GFr7TOttee21v5zzB/xD+dOk+TcRbZvneR3+l/6f9TX8/XW2o/7HpC/7Pf7l9baH7fWruz3uaq19uzcervgsVW11O+LNyV5VGvtv/u2P2qtvTXJM/rtv19Vd5/foLX2z621Z7fW/nOu97G/thelm1TkrHRB7gnr+OyDzzshb+qXT1li+1P75clz32esHkIUADAJd+yX3xtwi944va9f7r9g/TX9cqd1hIeF5trcZaOrWoeq2qyqfqmq3pRuyvekC0HfXWT3z7fWzlziUPdP8r/6r49dYp+X98u7p7slcDEntNauXmT9SUkuS/d75uOXaPtz+u+D0/u3C/9clu28y+ikdD2i96+qB8zfUFXb5dYa3cq3CglRAMBtUlVt0T+U9qNVdUU/QUTrJwaY6zFaOLPdWel+8d07yUere8jv+ma/+0C/PKmqXl1VD6qqTcf0MY6eV/NNSb6Q5Pf7bZ/Orb0vC62r52tuIorvtta+sNgOrbUv5dZxV3svtk+6cWCLtb0lyceXaltVu1TVa/oJP75f3UOE5z7j3/a7reuaj3TeldaPgzq1f7uwN+qIdLcxfrm19rEVLYyxEKIAgEmYG2h/h/72srGqqrukewjq69JN7HCndCHku+kmBph76OrPjL1prX0lydPTja85MN0kE9+sqq/1s+/9TI9C7wXpxshsk+TP0wWYa6rqI1X19KraYiM+yvV9vd9J8q0kFyb5t3S3vh3YWltsPFRy6wQHi7lTv/zmOvZJul6d+fsvtK72c9t+pm1VPTTdZ/izdEFnu3STS8x9xrlevXWNiRp83gmau6XviKrabN76uVv53hJWJSEKAJiEC/vl5ulmVhu349JNrPDVdLe+7dA/wHenfmKABy3VsLV2YpJ7JHlOkvemC3y7phs/tbaqXrRg/6uSHJDkEUmOT9fLtVmSg9NNgnBBVe0y4ueY/7Ddu7bWfqm1dlj/PK2frKPdzRtw7EWnAx+TnwvGfe/cO9KN1zor3YOTt2itbT/3GZP86VLtRz3vhJ2V5Gvpbl99bJJU1S8n+ZV0f0Zvm1xpbAwhCgCYhP9INylC0v9yOS79//g/rn/7u621f2utfW/Bbnde1zH6yShe31o7JF2vxr5JTkn3S/pfVveg4Pn7t9baWa21P2mt7Z1uOvQ/SnJ1knvm1tvUpsFcL9UvrnOvZC74LdWrta5b7ubGh81v++D+mFcneVxr7eOttRsXtFvnn8uI552YfpzX3JinuVv65m7H/FBr7VsrXxXjIEQBACuutXZZbh1L9MdVtdizjn7OBt76t2Nu7WX57BL7/NqGnC/5aUA6N8nhuXXiggPW0+Z7rbU3JpnrtXrouvZfYef1y62qatFJI6pqtyR3XbD/Qot+pv7P6MBF2s6Fsotbaz/33Krehvy5DD3vcrhl7rQbsO9b0vU6/Xo/a+DctPEmlFjFhCgAYFJekm6c0i7png10+3XtXFW/nVtv91qXa3JrL9deixznLkn+eIlzbLbY+iRprd2c7sG1SR/SqmqTqlqzjlpumL//lDg/yVf6r1+0xD7H9MtLkpyzxD5Pr6rtF1n/pCR3Sxc0/m3e+rlnZd17sT/rqnpkulsg12foeZfD3Nitxer4Ga21byb5YJLbpXsW1p3S9ZQtx/PRWCFCFAAwEa2189M9FLYleXSSz/az4e0wt09VbVdVh1bV2ekeSLrNBhz3unQz1yXJiVV1//5Ym1TVw9PdSrhUD8KrqurkqjpkQR13rqrj042Vakn+vd+0bZKvVNWLq2qvqrrdgnO9st/vQ+u/Iiujv8XsJf3bx1XV31XVHZOkqu7Yf87/r9/+kn7Wu8XcPskZVbVn33bTqnpykhP67W9urV06b/9PJvlhuvFBJ/Vhdm4Wxacm+dfcOuHIugw973KYm9Xw0H668vWZm2Bibur2d7TWfrzUzky/df3PCQDAsmqtvbmqrkryj0l2TzcbXqrqunRhZX5o+nqSj2zgoZ+b5Ox0PVGfrarr0/3n8RbpxuQ8NbdOPz3fmnQTURzW13FNusA1v46XtNYumPf+7umet3Rskh9X1bXpZp27Xb/9q9mwHrQV01p7V1XtleTFSZ6V5BlV9YN0dc/9J/urW2vvXMdhnpHkn5L8d992i3QTaiRdiP2Zz9xa+35VvTDJ69PdGnl4326rdNf9/HS3uB2/nvIHnXeZvD3J89Pd1nllVV2RrpfystbaYrd6np7k27l1zJZb+VY5PVEAwES11k5NN/nCM9ONk7os3S/Va9LdTnZyuufq3GdDn6nTWvuvdBMZnJrke0k2TXJFurB2/ySfW6Lp3yZ5drpZ+S5OF6A2T/KNdD1hD2mtvWre/tck+a10swGek+42rW3STU1+brqQcv9+DNhUaa29JMnD033WK9PNmndVutvMfq219sL1HOJTSR6Y5N3pbstsSb6U5GVJDup7BBee8/gkh+bWXqk1SS5KcnSS/dJNd74+g887bq21i9LNxnhGutsUd04XphedhbGfSXHuAc/nLgjhrEI1mYeEAwDA7Kiqi5PcO8nTW2snrG9/ppsQBQAAy6gfH3dWuh7KX2itXbOeJkw5t/MBAMAyqaodk/x1//ZEAeq2QU8UAACMWVW9NslvpxsvtWm6cWe/3Fq7YqKFMRZ6ogAAYPx2TPfcqhuSnJnkYQLUbYeeKAAAgAH0RAEAAAwgRAEAAAywZiPaug8QgJp0AQCw0vREAQAADCBEAQAADLAxt/MBwKpVVV9Lsm2SSyZcCgCTsWuSa1pr9xjaUIgCYFZtu8UWW+ywxx577DDpQgBYeRdeeGFuuOGGkdoKUQDMqkv22GOPHdauXTvpOgCYgH322SfnnXfeJaO0NSYKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKgKlUnadW1aer6tqq+mFVfbaqnl1Vt5t0fQDMLiEKgGn1tiRvTnKPJO9K8k9JNkvy+iTvqqqaYG0AzLA1ky4AABaqqkOSHJnka0n2ba1d2a/fNMm7kxyW5MlJ3jqpGgGYXXqiAJhGh/bLv5kLUEnSWvtxkpf2b/94xasCgAhRAEynnfvlVxfZNrdu76rafoXqAYCfcjsfANNorvfpHotsu+e8r3dP8ul1Haiq1i6xafcR6gIAPVEATKX398s/raod5lZW1ZokL5+33x1WtCoAiJ4oAKbTvyR5UpLfSPLFqjotyQ+T/FqSeyX5cpJ7J7l5fQdqre2z2Pq+h2rvcRUMwOzQEwXA1Gmt3ZLksUmen+TydDP1PTXJZUkOSHJVv+sVEykQgJmmJwqAqdRa+0mSv+lfP1VVWyS5f5IbknxhAqUBMOP0RAGw2hyZ5PZJ3t1PeQ4AK0pPFNzGvf3tbx+p3Uc/+tGR2p122mmD2xx99NEjnetZz3rWSO1YHapq29baNQvW/WqSVye5LskrJlIYADNPiAJgWv17Vd2Q5IIk1yb55SS/meSmJIe21hZ7hhQALDshCoBpdXKSJ6abpW+LJN9K8qYkr26tXTLBugCYcUIUAFOptfbXSf560nUAwEImlgAAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhgzaQLgFlz0003jdTuhBNOGKndq171qpHaXXHFFSO1G8WHP/zhkdo961nPGnMlAADrpycKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKgKlWVY+uqjOr6rKquqGqvlpV76mqB0+6NgBmkxAFwNSqqtckeX+SvZOckeT1Sc5L8rgkn6yqJ02wPABm1JpJFwAAi6mqnZM8P8l3kty3tXbFvG0HJ/lIklckecdkKgRgVumJAmBa3T3dv1P/NT9AJUlr7ewk1ya50yQKA2C26YmCEV177bUjtXvHO0b7T/PnPOc5I7VbDT7zmc+M1G7UP4NtttlmpHasuC8n+VGSfatqx9balXMbquohSbZJcuqkigNgdglRAEyl1trVVfXnSV6X5ItVdWqSq5LcK8ljk/x7kj9a33Gqau0Sm3YfV60AzBYhCoCp1Vo7rqouSXJikj+ct+krSd668DY/AFgJxkQBMLWq6s+SnJzkrel6oLZKsk+SryZ5Z1X91fqO0VrbZ7FXkouWsXQAbsOEKACmUlUdlOQ1SU5rrf1pa+2rrbUfttbOS/L4JN9M8ryquuck6wRg9ghRAEyr3+qXZy/c0Fr7YZJz0v079oCVLAoAhCgAptXm/XKpaczn1v9oBWoBgJ8SogCYVh/vl/+7qu46f0NV/UaS/ZPcmORTK10YALPN7HwATKuTk5yV5NeSXFhVpyS5PMke6W71qyR/0Vq7anIlAjCLhCgAplJr7Zaq+s0kz0zyxHSTSWyZ5OokH0hyfGvtzAmWCMCMEqIAmFqttR8nOa5/AcBUMCYKAABgACEKAABgACEKAABgAGOiIMmNN944uM2ee+450rkuvfTSkdqttE033XSkdgcddNDgNocddthI59pyyy1HagcAsDH0RAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAywZtIFwDhdd911I7V76UtfOrjNpZdeOtK5VtojHvGIkdodc8wxI7Xbb7/9RmoHALBa6IkCAAAYQIgCAAAYQIgCYCpV1VFV1dbzunnSdQIwe4yJAmBanZ/k5UtsOzDJw5J8cOXKAYCOEAXAVGqtnZ8uSP2cqvrP/ss3rlxFANBxOx8Aq0pV7ZnkQUm+meT0CZcDwAwSogBYbf6oX765tWZMFAArTogCYNWoqi2SPCnJLUneNOFyAJhRxkQBsJr8dpLtk5zeWvvGhjSoqrVLbNp9bFUBMFP0RAGwmvzvfvmPE60CgJmmJwqAVaGqfinJfkkuS/KBDW3XWttnieOtTbL3eKoDYJboiQJgtTChBABTQYgCYOpV1e2THJluQok3T7gcAGac2/mYShdccMFI7R7xiEeM1O7yyy8fqd0ott1225Have997xup3X777TdSuzVr/Hhgqhye5A5J3r+hE0oAwHLREwXAajA3ocQbJ1oFAESIAmDKVdUeSQ7IwAklAGC5uF8HgKnWWrswSU26DgCYoycKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABggDWTLgAWc/rpp4/U7vLLLx9zJUvbfvvtR2p32mmnjdTuwAMPHKkdAADjpScKAABgACEKAABgACEKAABgAGOiAJhZF3zzB9n1L0YbgwnAaC559aMnXcJG0xMFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFwNSrqgOr6l+r6ttVdVO/PLOqfnPStQEwezwnCoCpVlUvSfKXSa5M8v4k306yY5IHJDkoyQcmVhwAM0mIAmBqVdXh6QLUWUkOba1du2D7phMpDICZJkSxrI499tiR2h199NFjrmT8zjjjjJHaPfCBDxxzJXDbVFWbJHlNkh8mOWJhgEqS1tqPV7wwAGaeEAXAtNovyT2SnJzke1X16CR7JrkxyTmttf+cZHEAzC4hCoBp9av98jtJzkuy1/yNVfWxJE9orX13pQsDYLYJUQBMq5365dOSfC3JryX5ryR3T/I3SX49yXvSTS6xpKpau8Sm3cdSJQAzxxTnAEyr2/XLStfj9OHW2nWttS8keXySy5I8tKoePLEKAZhJeqIAmFbf65dfba19bv6G1toNVfWhJL+fZN8kS46Paq3ts9j6vodq7zHVCsAM0RMFwLT6Ur/8/hLb50LWFitQCwD8lBAFwLT6WJKfJLl3VW22yPY9++UlK1YRAESIAmBKtdauTPKuJNsledn8bVX1iHQTS/wgyWgPbQOAERkTBcA0+9MkD0zy4qp6SJJz0s3O9/gkNyf5w9baUrf7AcCyEKIAmFqttSuq6oFJXpIuOD0oybVJTk/yf1prn55kfQDMJiEKgKnWWrs6XY/Un066FgBIjIkCAAAYRIgCAAAYwO18LKuTTz55pHa33HLLmCtZtze84Q2D2+y7777LUAkAANNOTxQAAMAAQhQAAMAAQhQAAMAAxkQBMLP2vOt2WfvqR0+6DABWGT1RAAAAAwhRAAAAAwhRAAAAAwhRAAAAAwhRAAAAAwhRAAAAAwhRAAAAAwhRAAAAAwhRAAAAA6yZdAGsDi960YtGave5z31uzJWs233uc5+R2j3taU8b3KaqRjoXAACrm54oAACAAYQoAACAAYQoAACAAYQoAACAAYQoAKZWVV1SVW2J1+WTrg+A2WR2PgCm3Q+SHLfI+utWuhAASIQoAKbf91trx0y6CACY43Y+AACAAfREATDtNq+qJyX5xSTXJ/l8ko+11m6ebFkAzCohCoBpt3OSty9Y97Wqekpr7T/W17iq1i6xafeNrgyAmeR2PgCm2VuSPDxdkNoqyV5J/jHJrkk+WFX3m1xpAMwqPVEATK3W2ssXrLogydOq6rokz0tyTJLHr+cY+yy2vu+h2nsMZQIwY/REAbAandAvHzLRKgCYSXqi2CBf/OIXJ13CBnnoQx86Urt/+Id/GNzmE5/4xEjnuvDCC0dqN6r73ve+I7U77LDDRmr3mMc8ZnCbqhrpXMy0K/rlVhOtAoCZpCcKgNXowf3yqxOtAoCZJEQBMJWq6peraodF1t89yd/3b9+xslUBgNv5AJhehyf5i6o6O8nXklyb5F5JHp3k9kk+kOS1kysPgFklRAEwrc5Ocp8kD0h3+95WSb6f5BPpnhv19tZam1x5AMwqIQqAqdQ/SHe9D9MFgJVmTBQAAMAAQhQAAMAAQhQAAMAAQhQAAMAAQhQAAMAAQhQAAMAAQhQAAMAAnhM1g6699trBbc4777xlqGT83vjGN066hKlz/vnnj9TupJNOGqndX/3VXw1u84IXvGCkcwEATIKeKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKABWjao6sqpa//qDSdcDwGxaM+kCWHk33njj4Dbf+MY3lqESbote97rXDW7zghe8YBkq4bamqu6W5O+SXJdk6wmXA8AM0xMFwNSrqkryliRXJTlhwuUAMOOEKABWg2cneViSpyS5fsK1ADDjhCgAplpV7ZHk1Ule31r72KTrAQBjogCYWlW1Jsnbk1ya5EUjHmPtEpt2H7UuAGabEAXANHtZkgckOaC1dsOkiwGARIgCYEpV1b7pep/+prX2n6Mep7W2zxLHX5tk71GPC8DsMiYKgKkz7za+i5O8dMLlAMDPEKIAmEZbJ9ktyR5Jbpz3gN2W5Oh+n3/q1x03sSoBmElu5wNgGt2U5M1LbNs73TipTyT5UpKRb/UDgFEIUQBMnX4SiT9YbFtVHZMuRL2ttfamlawLABK38wEAAAwiRAEAAAwgRAGwqrTWjmmtlVv5AJgUY6Jghd3tbncbqd3Tnva0kdqdeuqpI7U799xzR2oHAHBbpycKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABgACEKAABggDWTLoCVt8kmw7Pz5ptvPtK5brrpppHajWqUz5YkBxxwwOA2J5100kjn2mabbUZqt8MOO4zU7oILLhip3bnnnjtSOwCA2zo9UQAAAAMIUQAAAAMIUQAAAAMIUQAAAAMIUQAAAAMIUQBMrap6TVV9uKq+UVU3VNXVVfXZqjq6qu446foAmE1CFADT7LlJtkry70len+SdSX6S5Jgkn6+qu02uNABmledEATDNtm2t3bhwZVW9MsmLkrwwyTNWvCoAZpqeKACm1mIBqvfufnnvlaoFAOYIUQCsRo/pl5+faBUAzCS38wEw9arq+Um2TrJdkl9JckC6APXqDWi7dolNu4+tQABmihAFwGrw/CR3nvf+jCRHtda+O6F6AJhhQhQAU6+1tnOSVNWdk+yXrgfqs1X1W62189bTdp/F1vc9VHuPu1YAbvuEqBm09dZbD25zv/vdb6RznXPOOSO1G9VrX/vakdo997nPHXMl4/f+979/pHannXbamCuByWmtfSfJKVV1XpKLk5yUZM/JVgXArDGxBACrTmvt60m+mOSXq2rHSdcDwGwRogBYrX6hX9480SoAmDlCFABTqap2r6qdF1m/Sf+w3Z2SfKq19r2Vrw6AWWZMFADT6lFJ/rqqPpbkf5JclW6GvocmuWeSy5P84eTKA2BWCVEATKuzkrwxyf5J7pdk+yTXp5tQ4u1Jjm+tXT258gCYVUIUAFOptXZBkmdOug4AWMiYKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAE8J2oGXXPNNYPbnHPOOctQyfgdfPDBI7X74Ac/OLjN9ttvP9K5TjnllJHaHX/88SO1u+mmm0ZqN6p99913Rc8HALDS9EQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMsGbSBbDytttuu8Ft9t9//5HO9clPfnKkdqN6wAMesKLnuy3beeedR2p37LHHjrkSZlVV3THJ45M8OsleSe6a5EdJ/jvJW5K8pbV2y+QqBGBWCVEATKvDk/xDkm8nOTvJpUnunOTQJG9K8htVdXhrrU2uRABmkRAFwLS6OMljk5w+v8epql6U5Jwkh6ULVP86mfIAmFXGRAEwlVprH2mtvW/hLXuttcuTnNC/PWjFCwNg5glRAKxGP+6XP5loFQDMJLfzAbCqVNWaJL/Xvz1jA/Zfu8Sm3cdWFAAzRU8UAKvNq5PsmeQDraGFwmwAAA6BSURBVLUPTboYAGaPnigAVo2qenaS5yW5KMmRG9KmtbbPEsdam2Tv8VUHwKzQEwXAqlBVz0zy+iRfTHJwa+3qCZcEwIwSogCYelX1nCR/n+SCdAHq8gmXBMAME6IAmGpV9edJ/jbJ+ekC1BUTLgmAGSdEATC1quql6SaSWJvk4a21KydcEgCYWAKA6VRVT07yiiQ3J/l4kmdX1cLdLmmtvXWFSwNgxglRAEyre/TL2yV5zhL7/EeSt65INQDQE6Jm0GabbTa4zRFHHDHSub785S+P1O6KKwx5WGiXXXYZqd2BBx44UruXvexlI7XbfXfPL2U8WmvHJDlmwmUAwM8xJgoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGCANZMugNXhGc94xkjtHvvYx47U7sUvfvFI7U466aSR2o1i//33H6ndoYceOlK7I488cqR2d7rTnUZqBwDA4vREAQAADCBEAQAADCBEAQAADCBEAQAADCBEAQAADCBEAQAADCBEATCVquoJVfV3VfXxqrqmqlpVvWPSdQGA50QBMK1ekuR+Sa5LclmS3SdbDgB09EQBMK2em2S3JNsmefqEawGAn9ITBcBUaq2dPfd1VU2yFAD4GXqiAAAABtATBcBtWlWtXWKTMVYAjERPFAAAwAB6olhWu+yyy0jt3va2t61oO+C2q7W2z2Lr+x6qvVe4HABuA/REAQAADCBEAQAADCBEAQAADCBEAQAADGBiCQCmUlUdkuSQ/u3O/fLBVfXW/usrW2vPX/HCAJh5QhQA0+r+SZ68YN09+1eSfD2JEAXAinM7HwBTqbV2TGut1vHaddI1AjCbhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAAIABhCgAplZV7VJVJ1bVt6rqpqq6pKqOq6o7TLo2AGbXmkkXAACLqap7JflUkp2SvDfJRUn2TfInSR5VVfu31q6aYIkAzCg9UQBMqzekC1DPbq0d0lr7i9baw5L8bZL7JHnlRKsDYGYJUQBMnaq6Z5JHJrkkyf9dsPnoJNcnObKqtlrh0gBAiAJgKj2sX57ZWrtl/obW2rVJPplkyyQPWunCAMCYKACm0X365cVLbP9yup6q3ZJ8eF0Hqqq1S2zafbTSAJh1eqIAmEbb9csfLLF9bv32K1ALAPwMPVEArEbVL9v6dmyt7bPoAboeqr3HWRQAs0FPFADTaK6nabsltm+7YD8AWDFCFADT6Ev9crcltt+7Xy41ZgoAlo0QBcA0OrtfPrKqfubfqqraJsn+SW5I8umVLgwAhCgApk5r7X+SnJlk1yTPXLD55Um2SnJSa+36FS4NAEwsAcDUekaSTyU5vqoenuTCJA9McnC62/hePMHaAJhheqIAmEp9b9SvJHlruvD0vCT3SnJ8kge31q6aXHUAzDI9UQBMrdbaN5I8ZdJ1AMB8eqIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGqNbaqG1HbgjAbUZNuoBRVdVVW2yxxQ577LHHpEsBYAIuvPDC3HDDDVe31u44tK0QBcDGWM0h6qYkt0vyuUnXMmV275cXTbSK6eO6LM21WZzrsrhpui67JrmmtXaPoQ3XjL8WAFgVLkiS1to+ky5kmlTV2sR1Wch1WZprszjXZXG3letiTBQAAMAAG9MTtWpv4QAAABiVnigAAIABhCgAAIABhCgAAIABNmaKcwAAgJmjJwoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQqA24Sq2qWqTqyqb1XVTVV1SVUdV1V3GHicHfp2l/TH+VZ/3F2Wq/bltrHXpqq2qqrfrar/V1UXVdX1VXVtVX2mqp5XVZst92dYDuP6nllwzIdU1c1V1arq2HHWu1LGeV2qaq+qOqmqvtEf64qq+o+q+r3lqH05jfFnzAFV9d6+/Y1VdWlVfaCqHrVctS+XqnpCVf1dVX28qq7pv+/fMeKxxv73cTl52C4Aq15V3SvJp5LslOS9SS5Ksm+Sg5N8Kcn+rbWrNuA4d+yPs1uSjyQ5N8nuSR6X5IokD26tfXU5PsNyGce16X+5+2CSq5OcneQrSXZI8pgkO/fHf3hr7cZl+hhjN67vmQXH3CbJ55PsmGTrJK9srb1knHUvt3Fel6o6KsmbkvwwyfuTXJJk+yR7JvlWa+2JYy5/2YzxZ8zTk7whyfVJTklyWZJdkhyaZMskL2mtvXI5PsNyqKrzk9wvyXXpPsvuSd7ZWnvSwOOM/e/jsmuteXl5eXl5repXkg8laUn+eMH61/XrT9jA4/xjv//rFqx/dr/+jEl/1klcmyT3T/K7STZbsH6bJGv74zxv0p91Et8zC9qemC5ovqg/xrGT/pyTui5JHpTkJ0nOT7LzIts3nfRnXenrkmTTJN9PckOS+yzYtkeSG9MFzs0n/XkHXJeDk9w7SSU5qL8W75jU991KvvREAbCqVdU9k/xPuv/lvldr7ZZ527ZJ8u10/8Dv1Fq7fh3H2SrJd5PckuQurbVr523bpD/Hrv05VkVv1LiuzXrOcUSSdyZ5f2vtMRtd9ApYjutSVY9LcmqSI5OsSfKWrLKeqHFel6r6WJIDk+zVWrtg2YpeAWP8GXPnJJcn+Xxr7X6LbP98kr2S7NimrddlA1TVQel6qgf1RK3Ez6nlYEwUAKvdw/rlmfP/8U2SPgh9Mt1tMg9az3EenGSLJJ+cH6D649yS5Mz+7cEbXfHKGde1WZcf98ufbMQxVtpYr0tV7ZTkn5Kc2lobaTzIlBjLdenHDx6Y5DNJvlBVB1fV8/vxcw/v/1NiNRnX98sV6f6jZrequvf8DVW1W7oenfNXY4DaSCvxc2rsVts3MQAsdJ9+efES27/cL3dboeNMk5X4TE/tl2dsxDFW2rivyxvT/U71tI0pagqM67r86rz9P9K//jrJa5OcleT8qvpfG1HnShvLdWnd7V/PTPe9sraq3lZV/6eqTkp3W+wXkhw+hnpXm1X5s3fNpAsAgI20Xb/8wRLb59Zvv0LHmSbL+pmq6llJHpVu3MuJoxxjQsZ2XarqqekmHvmd1tp3xlDbJI3ruuzUL387yZXpJk34cJI7JTk63S2Pp1fVXq21H41e7ooZ2/dLa+09VfWtJP+cZP4Mhd9JdwvoqrhVeMxW5c9ePVEA3NZVv9zYQcDjOs40GfkzVdWhSY5LN8bjsNbaj9fTZDXZoOtSVbumuwbvaa29e5lrmgYb+v1yu3nLP2itndJau6a19j9JnpzuNr/dkhy2PGWuuA3+e1RVT0rXG/fxdJNJbNkvP5zk75P8yzLVuJpN5c9eIQqA1W7ufym3W2L7tgv2W+7jTJNl+UxVdUi6X/auSHLQaploY55xXZcT08209oxxFDUFxnVdvtcvb0rygfkb+lva3tu/3XdogRMyluvSj3s6Md1te0e21i5qrd3QWrsoXe/c2iSH9xM0zJJV+bNXiAJgtftSv1zqfvm5AdxL3W8/7uNMk7F/pqo6PMl70t1+9NDW2pfW02Qajeu67J3u1rXv9g8ZbVXV0t2WlSQv7tedunHlrphx/126duFEAb25kLXFgNomaVzX5ZHppjn/j0UmULglycf6t/uMUuQqtip/9hoTBcBqd3a/fGRVbbLI9Lj7p+st+PR6jvPpfr/9q2qbRaY4f+SC860G47o2c22OSHJSkm8mOXgV9kDNGdd1OSnd7VgL3TvJQ9KNFVub5LMbXfHKGNd1+Xy6sVA7VtWdFxkrtme/vGTjS14R47oum/fLOy2xfW79ahgnNk5j/Tm1UvREAbCq9eMszkz3DKdnLtj88iRbJTlp/vNFqmr3qtp9wXGuS/L2fv9jFhznWf3xP7SagsO4rk2//snprs+lSR6ymq7DQmP8nnl2a+0PFr5ya0/U6f26/7tsH2aMxnhdfpLuwdVJ8lfzpzSvqr2SHJVuSvyTx/wRlsUY/x59vF8+oaruO39DVd0/yRPSjfv5yPiqnx5VtWl/Xe41f/0o13caeNguAKte/4/yp9LdWvXeJBcmeWC6ZzpdnGS/+c9e6W+5SmutFhznjv1xdkv3i8w56QZ9Py7d+J/9+n/wV41xXJuqOjjdYPhN0o3p+MYip/p+a+24ZfoYYzeu75kljn1UVuHDdpOx/l3aMt1kCQ9K1xP30XQ9LYelu43vea211y3zxxmbMV6XE5M8JV1v0ylJvp4uPBySZLMkx7XWnrvMH2ds+vGRh/Rvd07y6+lmGJwLjFe21p7f77trkq8l+XprbdcFxxl0faeBEAXAbUJV3S3JK9JNuX3HdE+5PzXJy1trVy/Yd8lfiKtqh3TTMB+S5C5JrkrywSQva61dtpyfYbls7LWZFwrW5ed+MZp24/qeWeS4R2WVhqhkrH+XtkzyZ0memOQeSW5Mcm6Sv2mtfXA5P8NyGMd1qapKN0PhUUnul2SbJNekC5r/1FpbVbPzVdUx6X5eLuWnPxfWFaL67Rt8faeBEAUAADCAMVEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAADCFEAAAAD/P8yK53YFdON7wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x648 with 2 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 235,
       "width": 424
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "for image_batch, label_batch in training_batches.take(1):\n",
    "    ps = model.predict(image_batch)\n",
    "    first_image = image_batch.numpy().squeeze()[0]\n",
    "  \n",
    "  \n",
    "fig, (ax1, ax2) = plt.subplots(figsize=(6,9), ncols=2)\n",
    "ax1.imshow(first_image, cmap = plt.cm.binary)\n",
    "ax1.axis('off')\n",
    "ax2.barh(np.arange(10), ps[0])\n",
    "ax2.set_aspect(0.1)\n",
    "ax2.set_yticks(np.arange(10))\n",
    "ax2.set_yticklabels(np.arange(10))\n",
    "ax2.set_title('Class Probability')\n",
    "ax2.set_xlim(0, 1.1)\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n4WcPdR9jKMB"
   },
   "source": [
    "WOW!! Now our network is brilliant. It can accurately predict the digits in our images. Let's take a look again at the loss and accuracy values for a single batch of images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "rFZKBfTgfPVy",
    "outputId": "b4d7816a-bbfa-4bb8-c453-82506029aeb8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2/2 [==============================] - 0s 932us/step - loss: 0.0196 - accuracy: 1.0000\n",
      "\n",
      "Loss after training: 0.020\n",
      "Accuracy after training: 100.000%\n"
     ]
    }
   ],
   "source": [
    "for image_batch, label_batch in training_batches.take(1):\n",
    "    loss, accuracy = model.evaluate(image_batch, label_batch)\n",
    "\n",
    "print('\\nLoss after training: {:,.3f}'.format(loss))\n",
    "print('Accuracy after training: {:.3%}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Wa5_vwtotNeg"
   },
   "source": [
    "> **Exercise:** Create a network with 784 input units, a hidden layer with 128 units, then a hidden layer with 64 units, then a hidden layer with 32 units and finally an output layer with 10 units. Use a ReLu activation function for all the hidden layers and a softmax activation function for the output layer. Then compile the model using an `adam` optimizer, a `sparse_categorical_crossentropy` loss function, and the `accuracy` metric. Finally, print the loss and accuracy of your un-trained model for a single batch of images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 105
    },
    "colab_type": "code",
    "id": "txuSaeuirvgc",
    "outputId": "33af7e60-e292-4788-f168-940351b7e6b4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2/2 [==============================] - 0s 912us/step - loss: 2.3565 - accuracy: 0.0938\n",
      "\n",
      "Loss before training: 2.357\n",
      "Accuracy before training: 9.375%\n"
     ]
    }
   ],
   "source": [
    "## Solution\n",
    "\n",
    "model = tf.keras.Sequential([\n",
    "        tf.keras.layers.Flatten(input_shape = (28, 28, 1)),\n",
    "        tf.keras.layers.Dense(128, activation = 'relu'),\n",
    "        tf.keras.layers.Dense(64, activation = 'relu'),\n",
    "        tf.keras.layers.Dense(64, activation = 'relu'),\n",
    "        tf.keras.layers.Dense(10, activation = 'softmax')\n",
    "])\n",
    "\n",
    "model.compile(optimizer='adam',\n",
    "              loss='sparse_categorical_crossentropy',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "for image_batch, label_batch in training_batches.take(1):\n",
    "    loss, accuracy = model.evaluate(image_batch, label_batch)\n",
    "\n",
    "print('\\nLoss before training: {:,.3f}'.format(loss))\n",
    "print('Accuracy before training: {:.3%}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bgdaQEVUumxo"
   },
   "source": [
    "> **Exercise:** Train the model you created above for 5 epochs and then print the loss and accuracy of your trained model for a single batch of images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 275
    },
    "colab_type": "code",
    "id": "HzxZtgBDt3Ak",
    "outputId": "e487178a-e5dd-411b-e5dc-566983d4aa27"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "938/938 [==============================] - 1s 2ms/step - loss: 0.2711 - accuracy: 0.9190: 0s - loss: 0.2712 - accuracy: 0.91\n",
      "Epoch 2/5\n",
      "938/938 [==============================] - 1s 2ms/step - loss: 0.1103 - accuracy: 0.9662\n",
      "Epoch 3/5\n",
      "938/938 [==============================] - 2s 2ms/step - loss: 0.0797 - accuracy: 0.9762\n",
      "Epoch 4/5\n",
      "938/938 [==============================] - 1s 2ms/step - loss: 0.0606 - accuracy: 0.9807\n",
      "Epoch 5/5\n",
      "938/938 [==============================] - 2s 2ms/step - loss: 0.0474 - accuracy: 0.9851: 0s - l\n",
      "2/2 [==============================] - 0s 1ms/step - loss: 0.1226 - accuracy: 0.9688\n",
      "\n",
      "Loss after training: 0.123\n",
      "Accuracy after training: 96.875%\n"
     ]
    }
   ],
   "source": [
    "## Solution\n",
    "\n",
    "EPOCHS = 5\n",
    "history = model.fit(training_batches, epochs = EPOCHS)\n",
    "\n",
    "for image_batch, label_batch in training_batches.take(1):\n",
    "    loss, accuracy = model.evaluate(image_batch, label_batch)\n",
    "\n",
    "print('\\nLoss after training: {:,.3f}'.format(loss))\n",
    "print('Accuracy after training: {:.3%}'.format(accuracy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "CfBqrMikvVCY"
   },
   "source": [
    "> **Exercise:** Plot the prediction of the model you created and trained above on a single image from the training set. Also plot the probability predicted by your model for each digit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 243
    },
    "colab_type": "code",
    "id": "NOwMUqYzvKtK",
    "outputId": "5f653945-0fe4-4699-e2cc-98e67e050dbb"
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1EAAAHXCAYAAABd89BGAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZwlZXkv8N8ju6xBUFyiuCEYjAjGFVE0Me7igvEajFtyXWNi9BpcoojLxWtiFKPXuIDrjQuJGkFRiRhJiFEHCUFZYgAVUNlkHxbhvX9UtbRt98zUmdN9TnO+38/nfGr6VD1Vz6npmenfvFVvVWstAAAAbJhbTLoBAACA1USIAgAAGECIAgAAGECIAgAAGECIAgAAGECIAgAAGECIAgAAGECIAgAAGECIAgAAGECIAgAAGECIAgAAGECIAgAAGECIAgAAGECIAgBWrapq/WvXSfcyKyZ1zjfmuFX1ob72kA3db1U9u3//a6N1zM2ZEAUATFxV3bKqXlhVn6+qH1bV1VV1VVWdXVVHVdVBVbXVpPtcKVV1zrwf7udeN1TVxVV1QlW9rKpuOek+Z1UfsA6pqr0m3QuTsemkGwAAZltVPT7J+5LsMu/tq5LcmGTX/vWUJG+tqme21r660j1O0FVJrux/vXmSHZPs27/+sKr2b61dMKnmVpEfJzkjyUUDai7ra364yLpnJ3loknOSnLyRvbEKGYkCACamqp6d5LPpAtQZSZ6ZZKfW2jatte2S7JDkqUm+luR2SfabTKcT85ettV36145Jdkry5iQtyT3ThU/Wo7X2qtba7q21vxlQ85m+5g+WszdWJyEKAJiIqvrNJO9N9/PIF5Lcp7X2sdbaxXPbtNYua639fWtt/yS/l+SKyXQ7HVprF7fWXpvkyP6tJ1bV7SbZE8wiIQoAmJQ3J9kiyXlJntFaW7uujVtrn0ry9g3ZcVVtUlX7V9U7q2pNVf20qq6rqvOr6jNV9fB11N6iv+fl+P4epOur6sKq+m5VHVFVj1qk5s5V9X+r6syqWtvf0/WDqvpaVb2qqnbakL4H+Lt5v957Xh+/mEChqraoqtdU1SlVdUX//g4L+t6/qv6hqn7Sn5+frO/8LKjfs6o+0dddU1WnV9VfVNUWS2y/TVUdWFUfr6pTq+rS/nx9v6reV1V3X6bjLjmxxDqO8SsTS8y9l+5SviQ5csF9a+f02x3Rf33Ueo7xhn67Eze0L6aDe6IAgBVXVbdP8tj+y8Nba5dtSF1rrW3gIfZIMv/eqWuTXJfktkkOSHJAVb2mtfaWRWo/muQZ876+LMl26S6lu2f/OnZuZVXtne5yw237t65Pdy/THfvXQ5N8Z37NGJw379fbLbJ+yyRfT3K/vp+rF25QVW9K8pr+y5buc946N52fw1prr1pHDw9Kdznh1kkuT1JJ7pHk0CSPqarfaa1duaDm2UneNe/rK9L9p/5d+9czquqA1tpxYz7uuKxN8tN096Zt1h9/fvi/sF9+IMlzkjy+qm41f3R1TlVVkmf1Xx6xTP2yTIxEAQCT8LB0P/wmyT8uw/6vS/LpJI9Pd7/VVq21bZLcJslfJLkhyZuq6v7zi6pqv3QB6sYkL0uyXWtth3Sh5HbpQsC/LDjWX6YLUP+eZO/W2uattV9L90P+byV5R7qAMk53nPfrSxdZ/+IkuyV5epJt+s+wa7pwl6p6em4KUH+T5NZ9zzvnppBzcFUdtI4e3pPke0l+s7W2fbpz8Jx0oeIBWXzU8OJ+/w9KskN/39uW6ULvx9Ods/9XVVuP+bhj0Vr7ZGttlyRzI0d/Mu+etV1aa7/Vb3di3+PmSX5/id09Ismd0v2efHK5emZ5CFEAwCTs0S+vTTehxFi11s5srT2ttXZ0a+2ncyNYrbULWmtvSvKGdCHuBQtKH9Avv9xae0dr7Yq+rrXWftxa+3Br7RVL1PxJa+0783q4urX27dbay1pr/zbmj/hHc4dJ8q1F1m+T5Pf6H/qv6/v5QWvt+n4E5I39dp9orf1xa+2ifpuLW2svzU2XC76pqpb6efHaJI9qrf1nX3tda+1DSV7Ur39eVd1pfkFr7e9aay9trf3b3Ohjf25PTzepyHHpgtxT1/HZBx93Qj7QL5+zxPrn9suj5r7PWD2EKABgEm7VL3824BK9cfp8v3zwgvcv75e3Xkd4WGiu5rYb3dU6VNXmVXXPqvpAuinfky4EXbjI5qe01r68xK72SnK3/tdvWmKbN/TLO6W7JHAx722tXbLI+x9Jcm66nzOftETtr+i/D47pv1z4+7Jsx11GH0k3IrpXVd1n/oqq2j439ehSvlVIiAIAbpaqaqv+obRfq6oL+gkiWj8xwNyI0cKZ7Y5L94Pv3km+Vt1Dftc3+90X+uVHquqwqnpAVW02po/x+nk9X5vku0me16/7Rm4afVloXSNfcxNRXNha++5iG7TWzshN913tvdg26e4DW6z2xiQnLFVbVXeoqrf2E35cWt1DhOc+41/3m63rnI903JXW3wf12f7LhaNRz0h3GeN/tda+vqKNMRZCFAAwCXM32v9af3nZWFXVbdM9BPXt6SZ22DldCLkw3cQAcw9d/aV7b1pr30/ywnT31zwk3SQT51XV2f3se780otD7X+nukdk2yZ+nCzCXV9VXq+qFVbXVRnyUq/p+f5rk/CSnJfmHdJe+PaS1ttj9UMlNExwsZud+ed46tkm6UZ352y+0rvq5db9UW1UPTfcZXpku6GyfbnKJuc84N6q3rnuiBh93guYu6XtGVW0+7/25S/mODKuSEAUATMJp/XKLdDOrjds70k2scFa6S9927B/ge+t+YoAHLFXYWjsiyZ2T/GmSz6ULfLumu39qTVW9esH2FyfZN8nvJDk83SjX5kn2TzcJwqlVdYcRP8f8h+3evrV2z9baU/rnaf18HXU3bMC+F50OfEx+JRj3o3MfS3e/1nHpHpy8VWtth7nPmOTPlqof9bgTdlySs9NdvvqEJKmq30hy33S/Rx+eXGtsDCEKAJiEf043KULS/3A5Lv3/+D+x//L3W2v/0Fr72YLNbrOuffSTUbyztXZAulGN+yX5TLof0t9Y3YOC52/fWmvHtdb+pLW2d7rp0J+f5JIkd8lNl6lNg7lRqjuuc6tkLvgtNaq1rkvu5u4Pm1/7wH6flyR5YmvthNbaNQvq1vn7MuJxJ6a/z2vunqe5S/rmLsf8Umvt/JXvinEQogCAFddaOzc33Uv0x1W12LOOfsUGXvq3U24aZfnOEtv89oYcL/lFQPpWkgNz08QF+66n5mettfclmRu1eui6tl9hJ/XLratq0Ukjqmq3JLdfsP1Ci36m/vfoIYvUzoWyM1trv/Lcqt6G/L4MPe5yuHHusBuw7ZHpRp1+t581cG7aeBNKrGJCFAAwKa9Nd5/SHdI9G2jLdW1cVU/LTZd7rcvluWmU616L7Oe2Sf54iWNsvtj7SdJauyHdg2uTPqRV1S2qatN19LJ2/vZT4uQk3+9//eoltjmkX56T5JtLbPPCqtphkfcPSvLr6YLGP8x7f+5ZWXdf7Pe6qh6Z7hLI9Rl63OUwd+/WYn38ktbaeUm+mGSTdM/C2jndSNlyPB+NFSJEAQAT0Vo7Od1DYVuSxyb5Tj8b3o5z21TV9lX15Ko6Pt0DSbfdgP1emW7muiQ5oqr26vd1i6p6RLpLCZcaQXhLVR1VVQcs6OM2VXV4unulWpKv9Ku2S/L9qnpNVd2rqjZZcKw399t9af1nZGX0l5i9tv/yiVX1rqq6VZJU1a36z/k/+vWv7We9W8yWSY6tqj372s2q6llJ3tuv/2Br7Yfztv/XJFenuz/oI32YnZtF8blJ/j43TTiyLkOPuxzmZjV8cj9d+frMTTAxN3X7x1pr1y+1MdNvXf9zAgCwrFprH6yqi5P8bZLd082Gl6q6Ml1YmR+afpDkqxu465clOT7dSNR3quqqdP95vFW6e3Kem5umn55v03QTUTyl7+PydIFrfh+vba2dOu/rO6V73tKbklxfVVekm3Vuk379WdmwEbQV01r7ZFXdK8lrkrwkyYuq6rJ0fc/9J/thrbWPr2M3L0ry/iT/2ddulW5CjaQLsb/0mVtrl1bVq5K8M92lkQf2dVunO+8np7vE7fD1tD/ouMvko0leke6yzouq6oJ0o5TnttYWu9TzmCQ/zk33bLmUb5UzEgUATFRr7bPpJl94cbr7pM5N90P1pukuJzsq3XN17rGhz9Rprf17uokMPpvkZ0k2S3JBurC2V5L/WKL0r5O8NN2sfGemC1BbJPlRupGw/Vprb5m3/eVJHpduNsBvprtMa9t0U5N/K11I2au/B2yqtNZem+QR6T7rRelmzbs43WVmv91ae9V6dnFikvsn+VS6yzJbkjOSvC7Jw/oRwYXHPDzJk3PTqNSmSU5P8vokD0o33fn6DD7uuLXWTk83G+Ox6S5T3CVdmF50FsZ+JsW5Bzx/a0EIZxWqyTwkHAAAZkdVnZnk7kle2Fp77/q2Z7oJUQAAsIz6++OOSzdCebvW2uXrKWHKuZwPAACWSVXtlORt/ZdHCFA3D0aiAABgzKrqL5M8Ld39Upulu+/sN1prF0y0McbCSBQAAIzfTumeW7U2yZeTPFyAuvkwEgUAADCAkSgAAIABhCgAAIABNt2IWtcBAlCTbgAAVpqRKAAAgAGEKAAAgAE25nI+AFi1qursJNslOWfCrQAwGbsmuby1duehhUIUALNqu6222mrHPfbYY8dJNwLAyjvttNOydu3akWqFKABm1Tl77LHHjmvWrJl0HwBMwD777JOTTjrpnFFq3RMFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFwFSqznOr6htVdUVVXV1V36mql1bVJpPuD4DZJUQBMK0+nOSDSe6c5JNJ3p9k8yTvTPLJqqoJ9gbADNt00g0AwEJVdUCSZyY5O8n9WmsX9e9vluRTSZ6S5FlJPjSpHgGYXUaiAJhGT+6XfzUXoJKktXZ9kr/ov/zjFe8KACJEATCddumXZy2ybu69vatqhxXqBwB+weV8AEyjudGnOy+y7i7zfr17km+sa0dVtWaJVbuP0BcAGIkCYCod3S//rKp2nHuzqjZN8oZ52/3ainYFADESBcB0+kSSg5I8Osn3quofk1yd5LeT3DXJfyW5e5Ib1rej1to+i73fj1DtPa6GAZgdRqIAmDqttRuTPCHJK5L8JN1Mfc9Ncm6SfZNc3G96wUQaBGCmGYkCYCq11n6e5K/61y9U1VZJ9kqyNsl3J9AaADPOSBQAq80zk2yZ5FP9lOcAsKKEKACmUlVtt8h7v5XksCRXJjl0xZsCgLicD4Dp9ZWqWpvk1CRXJPmNJI9Jcm2SJ7fWFnuGFAAsOyEKgGl1VJKnp5ulb6sk5yf5QJLDWmvnTLAvAGacEAXAVGqtvS3J2ybdBwAs5J4oAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAGbWqeddll0PPia7HnzMpFsBYBURogAAAAYQogAAAAYQogAAAAYQogAAAAYQogAAAAYQogAAAAYQogAAAAYQogCYalX12Kr6clWdW1Vrq+qsqvp0VT1w0r0BMJuEKACmVlW9NcnRSfZOcmySdyY5KckTk/xrVR00wfYAmFGbTroBAFhMVe2S5BVJfprkN1trF8xbt3+SryY5NMnHJtMhALPKSBQA0+pO6f6d+vf5ASpJWmvHJ7kiyc6TaAyA2WYkCkb0xje+caS6173udWPuZHnsv//+I9Udeuihg2v23XffkY7Fzd5/Jbkuyf2qaqfW2kVzK6pqvyTbJvnspJoDYHYJUQBMpdbaJVX150nenuR7VfXZJBcnuWuSJyT5SpLnr28/VbVmiVW7j6tXAGaLEAXA1GqtvaOqzklyRJI/mrfq+0k+tPAyPwBYCe6JAmBqVdUrkxyV5EPpRqC2TrJPkrOSfLyq/s/69tFa22exV5LTl7F1AG7GhCgAplJVPSzJW5P8Y2vtz1prZ7XWrm6tnZTkSUnOS/LyqrrLJPsEYPYIUQBMq8f1y+MXrmitXZ3km+n+HbvPSjYFAEIUANNqi3651DTmc+9ftwK9AMAvCFEATKsT+uX/rKrbz19RVY9O8uAk1yQ5caUbA2C2mZ0PgGl1VJLjkvx2ktOq6jNJfpJkj3SX+lWSg1trF0+uRQBmkRAFwFRqrd1YVY9J8uIkT083mcQtk1yS5AtJDm+tfXmCLQIwo4QoAKZWa+36JO/oXwAwFdwTBQAAMIAQBQAAMIAQBQAAMIB7oiDJRRddNLjmda973UjHqqqR6lba1772tZHqHvOYxwyuOemkk0Y61t3udreR6gAANoaRKAAAgAGEKAAAgAFczgfAzNrz9ttnzWGPnXQbAKwyRqIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAG2HTSDcA4XX311SPVPf/5zx9zJ7PriiuuGFxz7LHHjnSsl7zkJSPVAQBsDCNRAAAAAwhRAAAAAwhRAEylqnp2VbX1vG6YdJ8AzB73RAEwrU5O8oYl1j0kycOTfHHl2gGAjhAFwFRqrZ2cLkj9iqr6t/6X71u5jgCg43I+AFaVqtozyQOSnJfkmAm3A8AMEqIAWG3mnknwwdaae6IAWHFCFACrRlVtleSgJDcm+cCE2wFgRrknCoDV5GlJdkhyTGvtRxtSUFVrlli1+9i6AmCmGIkCYDX5n/3ybyfaBQAzzUgUAKtCVd0zyYOSnJvkCxta11rbZ4n9rUmy93i6A2CWGIkCYLUwoQQAU0GIAmDqVdWWSZ6ZbkKJD064HQBmnMv5mEo33DDafzKfeOKJI9V95jOfGaluNdh+++1Hqlu7du1Idddee+3gmmOOGe1RPy984QtHqttkk01GqmOiDkzya0mO3tAJJQBguRiJAmA1mJtQ4n0T7QIAIkQBMOWqao8k+2bghBIAsFxczgfAVGutnZakJt0HAMwxEgUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADDAppNuABZz1llnjVT3yEc+csydTI+DDjpopLrDDz98pLr3v//9I9W98pWvHFzzpS99aaRjnX322SPV3e1udxupDgAgMRIFAAAwiBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFwNSrqodU1d9X1Y+r6tp++eWqesykewNg9mw66QYAYF2q6rVJ3pjkoiRHJ/lxkp2S3CfJw5J8YWLNATCThCgAplZVHZguQB2X5MmttSsWrN9sIo0BMNOEKJbV97///ZHqDjzwwJHqWmsj1a2k7bfffqS6ww47bKS6HXbYYaS6lTyXox7r/PPPH6nubne720h1rKyqukWStya5OskzFgaoJGmtXb/ijQEw84QoAKbVg5LcOclRSX5WVY9NsmeSa5J8s7X2b5NsDoDZJUQBMK1+q1/+NMlJSe41f2VVfT3JU1trF650YwDMNiEKgGl16375giRnJ/ntJP+e5E5J/irJ7yb5dLrJJZZUVWuWWLX7WLoEYOaY4hyAabVJv6x0I07/1Fq7srX23SRPSnJukodW1QMn1iEAM8lIFADT6mf98qzW2n/MX9FaW1tVX0ryvCT3S7Lk/VGttX0We78fodp7TL0CMEOMRAEwrc7ol5cusX4uZG21Ar0AwC8IUQBMq68n+XmSu1fV5ous37NfnrNiHQFAhCgAplRr7aIkn0yyfZLXzV9XVb+TbmKJy5Icu/LdATDL3BMFwDT7syT3T/KaqtovyTfTzc73pCQ3JPmj1tpSl/sBwLIQogCYWq21C6rq/klemy44PSDJFUmOSfK/W2vfmGR/AMwmIQqAqdZauyTdiNSfTboXAEjcEwUAADCIEAUAADCAy/lYVm95y1tGqjvllFNGqquqkepG0Vobqe7qq68eqe68884bqe52t7vdSHU//OEPR6obxai/b8cff/xIdfvtt99IdQAAiZEoAACAQYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAAYQoAACAATaddAOsDkcfffRIdR/72MfG3Mny2G677QbXXHbZZSMd6/rrrx+p7jGPecxIdTvuuONIdeeff/5IdStp2223nXQLAMAMMhIFAAAwgBAFAAAwgBAFAAAwgBAFAAAwgBAFwNSqqnOqqi3x+smk+wNgNpmdD4Bpd1mSdyzy/pUr3QgAJEIUANPv0tbaIZNuAgDmuJwPAABgACNRAEy7LarqoCR3THJVklOSfL21dsNk2wJgVglRAEy7XZJ8dMF7Z1fVc1pr/7y+4qpas8Sq3Te6MwBmksv5AJhmRyZ5RLogtXWSeyX52yS7JvliVd17cq0BMKuMRAEwtVprb1jw1qlJXlBVVyZ5eZJDkjxpPfvYZ7H3+xGqvcfQJgAzxkgUAKvRe/vlfhPtAoCZZCRqBp1++umDaw499NCRjvXzn/98pLpRbbnlliPVff7znx9cc/HFF490rJe85CUj1V177bUj1V100UUj1V111VUj1a2kJzzhCZNugcm5oF9uPdEuAJhJRqIAWI0e2C/PmmgXAMwkIQqAqVRVv1FVOy7y/p2S/E3/5cdWtisAcDkfANPrwCQHV9XxSc5OckWSuyZ5bJItk3whyV9Orj0AZpUQBcC0Oj7JPZLcJ93le1snuTTJv6R7btRHW2ttcu0BMKuEKACmUv8g3fU+TBcAVpp7ogAAAAYQogAAAAYQogAAAAYQogAAAAYQogAAAAYQogAAAAYQogAAAAbwnKgZ9O1vf3twzbe+9a2RjlVVI9WN6vGPf/xIdQ95yEPG3MnSDjjggBU7VpIcddRRI9U97WlPG3MnS9t5551Hqtt2223H3AkAwPoZiQIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAIAABhAiAJg1aiqZ1ZV619/OOl+AJhNm066AVbeiSeeuGLHaq2NVLfLLruMVPfud797pLqbs3vc4x4j1Y36ezeKu9zlLiPV3eY2txlzJ0yzqvr1JO9KcmWSbSbcDgAzzEgUAFOvqirJkUkuTvLeCbcDwIwTogBYDV6a5OFJnpPkqgn3AsCME6IAmGpVtUeSw5K8s7X29Un3AwDuiQJgalXVpkk+muSHSV494j7WLLFq91H7AmC2CVEATLPXJblPkn1ba2sn3QwAJEIUAFOqqu6XbvTpr1pr/zbqflpr+yyx/zVJ9h51vwDMLvdEATB15l3Gd2aSv5hwOwDwS4QoAKbRNkl2S7JHkmvmPWC3JXl9v837+/feMbEuAZhJLucDYBpdm+SDS6zbO919Uv+S5IwkI1/qBwCjEKIAmDr9JBJ/uNi6qjokXYj6cGvtAyvZFwAkLucDAAAYRIgCAAAYQIgCYFVprR3SWiuX8gEwKe6JWsW+8pWvjFR3xBFHjLmTpVXVSHXvec97RqrbaaedRqpbDS688MKR6t7+9rePVDfq711rbXDNU5/61JGOBQAwCUaiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABth00g0wutNPP32kuuuuu27MnSzt3ve+90h1j3vc48bcyep3wgknjFT34Q9/eMydrNs222wzuOb+97//MnQCALA8jEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBAAAMIEQBMLWq6q1V9U9V9aOqWltVl1TVd6rq9VV1q0n3B8BsEqIAmGYvS7J1kq8keWeSjyf5eZJDkpxSVb8+udYAmFWeEwXANNuutXbNwjer6s1JXp3kVUletOJdATDTjEQBMLUWC1C9T/XLu69ULwAwR4gCYDV6fL88ZaJdADCTXM4HwNSrqlck2SbJ9knum2TfdAHqsA2oXbPEqt3H1iAAM0WIAmA1eEWS28z7+tgkz26tXTihfgCYYUIUAFOvtbZLklTVbZI8KN0I1Heq6nGttZPWU7vPYu/3I1R7j7tXAG7+hKhV7Nprr510C+u11157jVS32WabjbmT1e9tb3vbpFvYIKP0ue+++y5DJ9wctdZ+muQzVXVSkjOTfCTJnpPtCoBZY2IJAFad1toPknwvyW9U1U6T7geA2SJEAbBa3a5f3jDRLgCYOUIUAFOpqnavql0Wef8W/cN2b53kxNbaz1a+OwBmmXuiAJhWj0rytqr6epL/TnJxuhn6HprkLkl+kuSPJtceALNKiAJgWh2X5H1JHpzk3kl2SHJVugklPprk8NbaJZNrD4BZJUQBMJVaa6cmefGk+wCAhdwTBQAAMIAQBQAAMIAQBQAAMIAQBQAAMIAQBQAAMIAQBQAAMIAQBQAAMIDnRE2Byy+/fKS6973vfWPuZPxe/vKXT7qFqXPssceOVHfqqaeOuZN1e97znjdS3Qte8IIxdwIAMF2MRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAwgRAEAAAyw6aQbIDnuuONGqrv00ktHqmutDa559KMfPdKx9txzz5HqVtp55503uObggw8e6VhHHXXUSHXXXHPNSHVbbLHFSHXPf/7zR6qDcamqWyV5UpLHJrlXktsnuS7JfyY5MsmRrbUbJ9chALNKiAJgWh2Y5P8m+XGS45P8MMltkjw5yQeSPLqqDmyj/M8QAH/1q7kAAA+fSURBVGwEIQqAaXVmkickOWb+iFNVvTrJN5M8JV2g+vvJtAfArHJPFABTqbX21dba5xdestda+0mS9/ZfPmzFGwNg5glRAKxG1/fLn0+0CwBmksv5AFhVqmrTJH/Qf3nsBmy/ZolVu4+tKQBmipEoAFabw5LsmeQLrbUvTboZAGaPkSgAVo2qemmSlyc5PckzN6SmtbbPEvtak2Tv8XUHwKwwEgXAqlBVL07yziTfS7J/a+2SCbcEwIwSogCYelX1p0n+Jsmp6QLUTybcEgAzTIgCYKpV1Z8n+eskJ6cLUBdMuCUAZpwQBcDUqqq/SDeRxJokj2itXTThlgDAxBIATKeqelaSQ5PckOSEJC+tqoWbndNa+9AKtwbAjBOiAJhWd+6XmyT50yW2+eckH1qRbgCgJ0RNgVNOOWWkuosuGu2qlkX+J3e9dt5555GOtdKuuuqqkeoe8YhHDK4544wzRjrWKOc/SbbaaquR6o488siR6u573/uOVAfj0lo7JMkhE24DAH6Fe6IAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAGEKIAAAAG2HTSDbA63Pa2t13R451zzjkj1R188MEj1Z155pkj1Y3inve850h1n/jEJ0aq23PPPUeqAwBgcUaiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAAAABhCiAJhKVfXUqnpXVZ1QVZdXVauqj026LwDwnCgAptVrk9w7yZVJzk2y+2TbAYCOkSgAptXLkuyWZLskL5xwLwDwC0aiAJhKrbXj535dVZNsBQB+iZEoAACAAYxEAXCzVlVrlljlHisARmIkCgAAYAAjUVPgnve850h1m2222Uh1119//eCao48+eqRjbbnlliPVHXHEESPVnXvuuSPVjeIJT3jCSHXvete7Rqq74x3vOFIdzLrW2j6Lvd+PUO29wu0AcDNgJAoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAE0sAMJWq6oAkB/Rf7tIvH1hVH+p/fVFr7RUr3hgAM0+IAmBa7ZXkWQveu0v/SpIfJBGiAFhxLucDYCq11g5prdU6XrtOukcAZpMQBQAAMIAQBQAAMIAQBQAAMIAQBQAAMIAQBQAAMIAQBQAAMIDnRE2Bpz3taSPVnXzyySPVHXbYYYNrvvvd7450rO9973sj1Y1qiy22GKnu3e9+9+Ca5z73uSMdCwCA1c1IFAAAwABCFAAAwABCFAAAwABCFAAAwABCFAAAwABm5wNgZp163mXZ9eBjJt0GwM3eOYc9dtItjJWRKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAGEKAAAgAE8J2oVe8tb3rKidQArrarukOTQJI9KcqskP07y2SRvaK39bJK9ATC7hCgAplJV3TXJiUluneRzSU5Pcr8kf5LkUVX14NbaxRNsEYAZ5XI+AKbVe9IFqJe21g5orR3cWnt4kr9Oco8kb55odwDMLCEKgKlTVXdJ8sgk5yR594LVr09yVZJnVtXWK9waAAhRAEylh/fLL7fWbpy/orV2RZJ/TXLLJA9Y6cYAwD1RAEyje/TLM5dY/1/pRqp2S/JP69pRVa1ZYtXuo7UGwKwzEgXANNq+X162xPq593dYgV4A4JcYiQJgNap+2da3YWttn0V30I1Q7T3OpgCYDUaiAJhGcyNN2y+xfrsF2wHAihGiAJhGZ/TL3ZZYf/d+udQ9UwCwbIQoAKbR8f3ykVX1S/9WVdW2SR6cZG2Sb6x0YwAgRAEwdVpr/53ky0l2TfLiBavfkGTrJB9prV21wq0BgIklAJhaL0pyYpLDq+oRSU5Lcv8k+6e7jO81E+wNgBlmJAqAqdSPRt03yYfShaeXJ7lrksOTPLC1dvHkugNglhmJAmBqtdZ+lOQ5k+4DAOYzEgUAADCAEAUAADCAEAUAADCAEAUAADCAEAUAADCA2fkAmFl73n77rDnssZNuA4BVxkgUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAAEIUAADAANVaG7V25EIAbjZq0g2Mqqou3mqrrXbcY489Jt0KABNw2mmnZe3atZe01m41tFaIAmBjrOYQdW2STZL8x6R7mTK798vTJ9rF9HFelubcLM55Wdw0nZddk1zeWrvz0MJNx98LAKwKpyZJa22fSTcyTapqTeK8LOS8LM25WZzzsriby3lxTxQAAMAAGzMStWov4QAAABiVkSgAAIABhCgAAIABhCgAAIABNmaKcwAAgJljJAoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQoAAGAAIQqAm4WqukNVHVFV51fVtVV1TlW9o6p+beB+duzrzun3c36/3zssV+/LbWPPTVVtXVW/X1X/r6pOr6qrquqKqvp2Vb28qjZf7s+wHMb1PbNgn/tV1Q1V1arqTePsd6WM87xU1b2q6iNV9aN+XxdU1T9X1R8sR+/LaYx/x+xbVZ/r66+pqh9W1Req6lHL1ftyqaqnVtW7quqEqrq8/77/2Ij7Gvufx+XkYbsArHpVddckJya5dZLPJTk9yf2S7J/kjCQPbq1dvAH7uVW/n92SfDXJt5LsnuSJSS5I8sDW2lnL8RmWyzjOTf/D3ReTXJLk+CTfT7Jjkscn2aXf/yNaa9cs08cYu3F9zyzY57ZJTkmyU5Jtkry5tfbacfa93MZ5Xqrq2Uk+kOTqJEcnOSfJDkn2THJ+a+3pY25/2Yzx75gXJnlPkquSfCbJuUnukOTJSW6Z5LWttTcvx2dYDlV1cpJ7J7ky3WfZPcnHW2sHDdzP2P88LrvWmpeXl5eX16p+JflSkpbkjxe8//b+/fdu4H7+tt/+7Qvef2n//rGT/qyTODdJ9kry+0k2X/D+tknW9Pt5+aQ/6yS+ZxbUHpEuaL6638ebJv05J3Vekjwgyc+TnJxkl0XWbzbpz7rS5yXJZkkuTbI2yT0WrNsjyTXpAucWk/68A87L/knunqSSPKw/Fx+b1PfdSr6MRAGwqlXVXZL8d7r/5b5ra+3Geeu2TfLjdP/A37q1dtU69rN1kguT3Jjktq21K+atu0V/jF37Y6yK0ahxnZv1HOMZST6e5OjW2uM3uukVsBznpaqemOSzSZ6ZZNMkR2aVjUSN87xU1deTPCTJvVprpy5b0ytgjH/H3CbJT5Kc0lq79yLrT0lyryQ7tWkbddkAVfWwdCPVg0aiVuLvqeXgnigAVruH98svz//HN0n6IPSv6S6TecB69vPAJFsl+df5Aarfz41Jvtx/uf9Gd7xyxnVu1uX6fvnzjdjHShvreamqWyd5f5LPttZGuh9kSozlvPT3Dz4kybeTfLeq9q+qV/T3zz2i/0+J1WRc3y8XpPuPmt2q6u7zV1TVbulGdE5ejQFqI63E31Njt9q+iQFgoXv0yzOXWP9f/XK3FdrPNFmJz/TcfnnsRuxjpY37vLwv3c9UL9iYpqbAuM7Lb83b/qv9621J/jLJcUlOrqq7bUSfK20s56V1l3+9ON33ypqq+nBV/e+q+ki6y2K/m+TAMfS72qzKv3s3nXQDALCRtu+Xly2xfu79HVZoP9NkWT9TVb0kyaPS3fdyxCj7mJCxnZeqem66iUd+r7X20zH0NknjOi+37pdPS3JRukkT/inJzklen+6Sx2Oq6l6ttetGb3fFjO37pbX26ao6P8nfJZk/Q+FP010CuiouFR6zVfl3r5EoAG7uql9u7E3A49rPNBn5M1XVk5O8I909Hk9prV2/npLVZIPOS1Xtmu4cfLq19qll7mkabOj3yybzln/YWvtMa+3y1tp/J3lWusv8dkvylOVpc8Vt8J+jqjoo3WjcCekmk7hlv/ynJH+T5BPL1ONqNpV/9wpRAKx2c/9Luf0S67dbsN1y72eaLMtnqqoD0v2wd0GSh62WiTbmGdd5OSLdTGsvGkdTU2Bc5+Vn/fLaJF+Yv6K/pO1z/Zf3G9rghIzlvPT3PR2R7rK9Z7bWTm+trW2tnZ5udG5NkgP7CRpmyar8u1eIAmC1O6NfLnW9/NwN3Etdbz/u/UyTsX+mqjowyafTXX700NbaGespmUbjOi97p7t07cL+IaOtqlq6y7KS5DX9e5/duHZXzLj/LF2xcKKA3lzI2mpAb5M0rvPyyHTTnP/zIhMo3Jjk6/2X+4zS5Cq2Kv/udU8UAKvd8f3ykVV1i0Wmx31wutGCb6xnP9/ot3twVW27yBTnj1xwvNVgXOdmruYZST6S5Lwk+6/CEag54zovH0l3OdZCd0+yX7p7xdYk+c5Gd7wyxnVeTkl3L9ROVXWbRe4V27NfnrPxLa+IcZ2XLfrlzkusn3t/NdwnNk5j/XtqpRiJAmBV6++z+HK6Zzi9eMHqNyTZOslH5j9fpKp2r6rdF+znyiQf7bc/ZMF+XtLv/0urKTiM69z07z8r3fn5YZL9VtN5WGiM3zMvba394cJXbhqJOqZ/793L9mHGaIzn5efpHlydJP9n/pTmVXWvJM9ONyX+UWP+CMtijH+OTuiXT62q35y/oqr2SvLUdPf9fHV83U+PqtqsPy93nf/+KOd3GnjYLgCrXv+P8onpLq36XJLTktw/3TOdzkzyoPnPXukvuUprrRbs51b9fnZL94PMN9Pd9P3EdPf/PKj/B3/VGMe5qar9090Mf4t093T8aJFDXdpae8cyfYyxG9f3zBL7fnZW4cN2k7H+WbpluskSHpBuJO5r6UZanpLuMr6Xt9bevswfZ2zGeF6OSPKcdKNNn0nyg3Th4YAkmyd5R2vtZcv8ccamvz/ygP7LXZL8broZBucC40WttVf02+6a5OwkP2it7bpgP4PO7zQQogC4WaiqX09yaLopt2+V7in3n03yhtbaJQu2XfIH4qraMd00zAckuW2Si5N8McnrWmvnLudnWC4be27mhYJ1+ZUfjKbduL5nFtnvs7NKQ1Qy1j9Lt0zyyiRPT3LnJNck+VaSv2qtfXE5P8NyGMd5qapKN0Phs5PcO8m2SS5PFzTf31pbVbPzVdUh6f6+XMov/l5YV4jq12/w+Z0GQhQAAMAA7okCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAYQIgCAAAY4P8DNIz0vXYHor4AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x648 with 2 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 235,
       "width": 424
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "## Solution\n",
    "\n",
    "for image_batch, label_batch in training_batches.take(1):\n",
    "    ps = model.predict(image_batch)\n",
    "    first_image = image_batch.numpy().squeeze()[0]\n",
    "  \n",
    "  \n",
    "fig, (ax1, ax2) = plt.subplots(figsize=(6,9), ncols=2)\n",
    "ax1.imshow(first_image, cmap = plt.cm.binary)\n",
    "ax1.axis('off')\n",
    "ax2.barh(np.arange(10), ps[0])\n",
    "ax2.set_aspect(0.1)\n",
    "ax2.set_yticks(np.arange(10))\n",
    "ax2.set_yticklabels(np.arange(10))\n",
    "ax2.set_title('Class Probability')\n",
    "ax2.set_xlim(0, 1.1)\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "dqREWxKKVwql"
   },
   "source": [
    "## Automatic Differentiation\n",
    "\n",
    "Let's now take a minute to see how TensorFlow calculates and keeps track of the gradients needed for backpropagation. TensorFlow provides a class that records automatic differentiation operations, called `tf.GradientTape`. Automatic differentiation, also known as algorithmic differentiation or simply “autodiff”, is a family of techniques used by computers for efficiently and accurately evaluating derivatives of numeric functions.\n",
    "\n",
    "`tf.GradientTape` works by keeping track of operations performed on tensors that are being \"watched\". By default `tf.GradientTape` will automatically \"watch\" any trainable variables, such as the weights in our model. Trainable variables are those that have `trainable=True`. When we create a model with `tf.keras`, all of the parameters are initialized with `trainable = True`. Any tensor can also be manually \"watched\" by invoking the watch method.\n",
    "\n",
    "\n",
    "Let's see a simple example. Let's take the following equation:\n",
    "\n",
    "$$\n",
    "y = x^2\n",
    "$$\n",
    "\n",
    "The derivative of `y` with respect to `x` is given by:\n",
    "\n",
    "$$\n",
    "\\frac{d y}{d x} = 2x\n",
    "$$\n",
    "\n",
    "Now, let's use `tf.GradientTape` to calculate the derivative of a tensor `y` with respect to a tensor `x`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "id": "2-ktpx5dVU3O",
    "outputId": "d4a54fba-61eb-4419-e9d9-8162785ef09d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Gradient calculated by tf.GradientTape:\n",
      " tf.Tensor(\n",
      "[[1.1966898  0.12552415]\n",
      " [0.29263484 0.9696375 ]], shape=(2, 2), dtype=float32)\n",
      "\n",
      "True Gradient:\n",
      " tf.Tensor(\n",
      "[[1.1966898  0.12552415]\n",
      " [0.29263484 0.9696375 ]], shape=(2, 2), dtype=float32)\n",
      "\n",
      "Maximum Difference: 0.0\n"
     ]
    }
   ],
   "source": [
    "# Set the random seed so things are reproducible\n",
    "tf.random.set_seed(7)\n",
    "\n",
    "# Create a random tensor\n",
    "x = tf.random.normal((2,2))\n",
    "\n",
    "# Calculate gradient\n",
    "with tf.GradientTape() as g:\n",
    "    g.watch(x)\n",
    "    y = x ** 2\n",
    "    \n",
    "dy_dx = g.gradient(y, x)\n",
    "\n",
    "# Calculate the actual gradient of y = x^2\n",
    "true_grad = 2 * x\n",
    "\n",
    "# Print the gradient calculated by tf.GradientTape\n",
    "print('Gradient calculated by tf.GradientTape:\\n', dy_dx)\n",
    "\n",
    "# Print the actual gradient of y = x^2\n",
    "print('\\nTrue Gradient:\\n', true_grad)\n",
    "\n",
    "# Print the maximum difference between true and calculated gradient\n",
    "print('\\nMaximum Difference:', np.abs(true_grad - dy_dx).max())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QgLCJaooV5Un"
   },
   "source": [
    "The `tf.GradientTape` class keeps track of these operations and knows how to calculate the gradient for each one. In this way, it's able to calculate the gradients for a chain of operations, with respect to any one tensor.\n",
    "\n",
    "To know more about `tf.GradientTape` and trainable variables check the following links\n",
    "\n",
    "* [Gradient Tape](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/GradientTape)\n",
    "\n",
    "* [TensorFlow Variables](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/Variable)\n",
    "\n",
    "Next up you'll write the code for training a neural network on a more complex dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "am0SvU9KWAD3"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Part 3 - Training Neural Networks (Solution).ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}