HW1/Practical/Q7a.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CHDXBfEmWAJT"
   },
   "source": [
    "# CE-40959: Deep Learning\n",
    "## HW1\n",
    "#### Name: Pouya Khani\n",
    "#### Student No.: 99210283"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "I90qRc4S1vgm"
   },
   "source": [
    "Please read the codes and instructions given to you carefully."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "y8GOy2dW19v7"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import matplotlib.pyplot as plt\n",
    "from model import *\n",
    "from linear import *\n",
    "from module import *\n",
    "from optimizer import *\n",
    "from relu import *\n",
    "from sgd import *\n",
    "from sigmoid import *\n",
    "from softmax_crossentropy import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oSbUOPo11-q1"
   },
   "source": [
    "## 1. Modules\n",
    "In this problem you see some classes for typical modules used in neural networks. These modules include:\n",
    "\n",
    "1. **`ReLU`**: ReLU activation function\n",
    "\n",
    "2. **`Sigmoid`**: Sigmoid activation function\n",
    "\n",
    "3. **`SoftmaxCrossEntropy`**: A module which represents the softmax activation function followed by a cross entropy loss function.\n",
    "\n",
    "4. **`Linear`**: Fully connected layer which multiplies the input by a weight matrix and adds a bias term to it\n",
    "\n",
    "In this problem you have to implement `forward()` and `backward()` functions of the modules in the above list. Then your implemented codes will be tested here in this notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vKHSlGoN4aZi"
   },
   "source": [
    "### 1.1. Activation Functions\n",
    "Open files `relu.py` and `sigmoid.py`. There you can see **`ReLU`** and **`Sigmoid`** classes. \n",
    "Implement `forward()` and `backward()` functions of theses classes.\n",
    "\n",
    "Here are some tips that might help you:<br/>\n",
    "**1)** `forward()` function of the layers takes an input called `x` which is a numpy 2d-array with shape `(N, D)`. Implement the forward pass corresponding to considered function. Store the results of forward pass in `out`.<br/>\n",
    "**2)** `backward()` function of the layers takes a parameter called `dout`. `dout` is the gradient of loss w.r.t. the ouput of layer in the forward pass. Provided `dout`, you have to compute the gradient w.r.t. the inputs (i.e. `x`)  to the layer. Store the results in `dx`. <br/>\n",
    "**3)** For implementing backward pass, you may need some of the variables computed during forward pass; Save these variables in the `self.cache` attribute of the layer during forward pass and use them in the backward pass.<br/>\n",
    "**4)** Test your implementation with the following cells. You should see a small value as error between your funnctions outputs and `correct output`, `correct_dx` for each activation functions. We get error of orders 1e-8 and less."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rP-EErtUyC9R",
    "outputId": "db3ae854-9b88-4495-a22b-0373ffaab927"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Relu Test Cell:\n",
      "Relative error forward pass: 1.2327591637534371e-08\n",
      "Relative error backward pass: 0.0\n"
     ]
    }
   ],
   "source": [
    "###########################################################################\n",
    "#                             Relu Test                                   #\n",
    "###########################################################################\n",
    "np.random.seed(22)\n",
    "N = 5\n",
    "d = 6\n",
    "x = np.random.randn(N,d)\n",
    "relu = ReLU('test')\n",
    "print('Relu Test Cell:')\n",
    "output = relu.forward(x)\n",
    "correct_output = [[0.,         0. ,        1.08179168, 0.,         0.  ,       0.   ,     ],\n",
    "                  [0.9188215 , 0.  ,       0.62649346, 0. ,        0.02885512, 0.,        ],\n",
    "                  [0.58775221, 0.75231758, 0.   ,      1.05597241 ,0.74775027, 1.06467659],\n",
    "                  [1.52012959, 0.  ,       1.85998989, 0.  ,       0. ,        0.337325  ],\n",
    "                  [1.04672873 ,0.62914334, 0.36305909, 0.5557497,  0.,         0.02369477]]\n",
    "\n",
    "print('Relative error forward pass:', np.linalg.norm(output - correct_output))\n",
    "    \n",
    "dx = relu.backward(np.ones((N,d), dtype=np.float32))\n",
    "correct_dx = [[0., 0. ,1. ,0. ,0., 0.],\n",
    "              [1., 0. ,1., 0. ,1., 0.],\n",
    "              [1. ,1. ,0. ,1., 1., 1.],\n",
    "              [1. ,0. ,1., 0., 0. ,1.],\n",
    "              [1. ,1. ,1. ,1. ,0., 1.]]\n",
    "print('Relative error backward pass:', np.linalg.norm(dx - correct_dx))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8bxGWZGDanc4",
    "outputId": "0278bcc2-b93e-494b-fc69-d3ace17ef0ba"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sigmoid Test Cell:\n",
      "Relative error forward pass: 1.3652868830496266e-08\n",
      "Relative error backward pass: 1.515199259019241e-08\n"
     ]
    }
   ],
   "source": [
    "###########################################################################\n",
    "#                             sigmoid Test                                #\n",
    "###########################################################################\n",
    "np.random.seed(22)\n",
    "N=5\n",
    "d=6\n",
    "x = np.random.randn(N,d)\n",
    "sigmoid = Sigmoid('test')\n",
    "print('Sigmoid Test Cell:')    \n",
    "output = sigmoid.forward(x)\n",
    "correct_output = [[0.4770287,  0.18795539, 0.74683289, 0.44045266, 0.37962761, 0.26849495],\n",
    "                  [0.71480192, 0.24905997, 0.65169394, 0.36319727, 0.50721328, 0.44256287],\n",
    "                  [0.64284923, 0.67968348, 0.25759572, 0.74192012, 0.6786883,  0.74358323],\n",
    "                  [0.82055756, 0.18413151, 0.86529577, 0.16817555, 0.34387488, 0.58354059],\n",
    "                  [0.74014623, 0.65229519, 0.58978075, 0.63546853, 0.25189151, 0.50592342]]\n",
    "    \n",
    "print('Relative error forward pass:', np.linalg.norm(output - correct_output))\n",
    "    \n",
    "dx = sigmoid.backward(np.ones((N,d), dtype=np.float32))\n",
    "correct_dx = [[0.24947232, 0.15262816, 0.18907352, 0.24645411, 0.23551049, 0.19640541],\n",
    "              [0.20386014, 0.1870291,  0.22698895, 0.23128501, 0.24994797, 0.24670098],\n",
    "              [0.2295941,  0.21771385, 0.19124017, 0.19147466, 0.21807049, 0.19066721],\n",
    "              [0.14724285, 0.1502271,  0.116559,   0.13989254, 0.22562495, 0.24302097],\n",
    "              [0.19232979, 0.22680617, 0.24193942, 0.23164828, 0.18844218, 0.24996491]]\n",
    "    \n",
    "print('Relative error backward pass:', np.linalg.norm(dx - correct_dx))\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VhU008dXEGab"
   },
   "source": [
    "### 1.2. Softmax activation with Cross Entropy loss\n",
    "\n",
    "You have to implement a numerically stable version of softmax in this problem.\n",
    "\n",
    "Open file `softmax_crossentropy.py`. There you see a class for Softmax activation with Cross Entropy loss used in neural networks.\n",
    " \n",
    "\n",
    "Implement `forward()` and `backward()` function of SoftmaxCrossentropy class corresponding to forward and backward pass of softmax activation followed by cross entropy loss. Test your implementation with the following functions. The order of outputs should be 1e-8 or smaller."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gnm4K_L-EbJh",
    "outputId": "2b075f0c-183f-44ec-a9bf-36935c6ae280"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Softmax with Cross Entropy Test Cell:\n",
      "Loss relative error: 9.134629874729683e-08\n",
      "Gradient relative error: 1.695804576567757e-08\n"
     ]
    }
   ],
   "source": [
    "###########################################################################\n",
    "#                  Softmax with Cross Entropy Test                        #\n",
    "###########################################################################\n",
    "np.random.seed(22)\n",
    "N=5\n",
    "d=6\n",
    "  \n",
    "x = np.linspace(1000, 1015, num=N*d).reshape(N,d)\n",
    "y = np.random.randint(0, d, (N, ))\n",
    "    \n",
    "softmax_ce = SoftmaxCrossentropy('test')\n",
    "print('Softmax with Cross Entropy Test Cell:')    \n",
    "loss, _ = softmax_ce.forward(x, y=y)\n",
    "dx = softmax_ce.backward()\n",
    "    \n",
    "correct_loss = 1.6883967462546619\n",
    "print('Loss relative error:', np.abs(loss - correct_loss))\n",
    "    \n",
    "correct_dx = [[ 0.00636809,  0.0106818,   0.01791759,  0.03005485,  0.05041383, -0.11543615],\n",
    "              [ 0.00636809,  0.0106818,   0.01791759,  0.03005485, -0.14958617,  0.08456385],\n",
    "              [ 0.00636809,  0.0106818,   0.01791759,  0.03005485, -0.14958617,  0.08456385],\n",
    "              [-0.19363191,  0.0106818,   0.01791759,  0.03005485,  0.05041383,  0.08456385],\n",
    "              [ 0.00636809,  0.0106818,   0.01791759,  0.03005485, -0.14958617,  0.08456385]]\n",
    "print('Gradient relative error:', np.linalg.norm(dx - correct_dx))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GY2kXfpbMwPO"
   },
   "source": [
    "### 1.3. Linear\n",
    "Open file `linear.py`. There you see a class for linear (or fully connected) layer used in neural networks.<br/> \n",
    "\n",
    "Implement `forward()` and `backward()` functions corresponding to forward and backward pass of a linear layer.<br/>\n",
    " \n",
    "We have implemented `__init__()` constructor for you. You can see `W`, `b`, `dW`, `db` variables in this class.\n",
    "\n",
    "In the forward pass, You have to output $y=XW+b$ in which $W$ and $b$ are parameters of the layer. In the backward pass you get $\\large{\\frac{\\partial{loss}}{\\partial{y}}}$ as `dout`; You have to compute $\\large{\\frac{\\partial{loss}}{\\partial{X}}}$, $\\large{\\frac{\\partial{loss}}{\\partial{W}}}$ and $\\large{\\frac{\\partial{loss}}{\\partial{b}}}$ and save them in `dx`, `dW`, and `db`. The output of the `backward()` function will be `dx`. \n",
    "\n",
    "Note that a linear layer with the shape of weights $(D, K)$ represents a layer in a MLP with $K$ neurons (or units). In other words, it takes a $D$-dimensional input and gives back a $K$-dimensional ouput.\n",
    "\n",
    "Test your implementation with the following functions. The order of ouputs should be 1e-8 or smaller."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "y0FyQoJ8NiDi",
    "outputId": "d63fdf05-7c98-429a-9665-2b197fea9eac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Relative error forward pass: 1.223759443671177e-08\n",
      "Relative error dx: 1.2514524575828446e-08\n",
      "Relative error dw: 7.449439033436333e-09\n",
      "Relative error db: 3.529100840673046e-09\n"
     ]
    }
   ],
   "source": [
    "###########################################################################\n",
    "#                         Linear Test                            #\n",
    "###########################################################################\n",
    "np.random.seed(22)\n",
    "D = 4\n",
    "K = 3\n",
    "N = 5\n",
    "x = np.random.randn(N,D)\n",
    "linear = Linear('test', D, K, l2_coef=1.)\n",
    "output = linear.forward(x)\n",
    "    \n",
    "correct_output = [[-0.51242952, -1.47921276, -2.32943713],\n",
    "                  [-1.17901283, -2.60908172,  0.54809823],\n",
    "                  [ 0.74600461, -2.24752841, -1.1013558 ],\n",
    "                  [ 0.75284837,  1.80111973, -2.27011589],\n",
    "                  [ 2.03171234, -3.05396933,  1.35213333]]\n",
    "    \n",
    "print('Relative error forward pass:', np.linalg.norm(output - correct_output))\n",
    "    \n",
    "dout = np.random.randn(N, K)\n",
    "dx = linear.backward(dout)\n",
    "    \n",
    "correct_dx = [[-0.25519113, -0.09724317,  0.280189,    0.87644613],\n",
    "              [ 1.20379991, -0.78816259, -1.27930227, -4.1952743 ],\n",
    "              [-0.77808532, -0.05005675, -3.14028536, -8.02818572],\n",
    "              [ 0.95446653, -1.90375857,  1.62080372,  3.57597736],\n",
    "              [ 2.86716776, -1.39892213,  0.31786772, -0.88234943]]\n",
    "print('Relative error dx:', np.linalg.norm(dx - correct_dx))\n",
    "    \n",
    "correct_dW = [[ 3.33629487, -4.43357113, -1.89100503],\n",
    "              [ 1.31103323,  2.17687036, -2.33906146],\n",
    "              [ 1.69538051, -0.89256682, -0.86018824],\n",
    "              [-0.87944724,  7.48073741, -7.0605863 ]]\n",
    "print('Relative error dw:', np.linalg.norm(linear.dW - correct_dW))\n",
    "    \n",
    "correct_db = [-1.02223284, -3.61915576, -0.16696389]\n",
    "print('Relative error db:', np.linalg.norm(linear.db - correct_db))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DbIbV_oku_fq"
   },
   "source": [
    "## 2. Optimization Algorithm\n",
    "\n",
    "Open file `sgd.py`.<br/> \n",
    "you have to implement `SGD` class.\n",
    "\n",
    "Test your implementation with the following cells. The order of errors should be 1e-8 or smaller."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "AMwdZUmqRMQO",
    "outputId": "5495c3f1-9465-4bff-c7a5-a2888ec0865c"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W Relative error: 1.4426196588173724e-08\n",
      "b Relative error:  5.771526066321419e-09\n"
     ]
    }
   ],
   "source": [
    "###########################################################################\n",
    "#                           SGD Test                             #\n",
    "###########################################################################\n",
    "N, D = 5, 4\n",
    "np.random.seed(22)\n",
    "linear = Linear('test', N, D, l2_coef=1.)\n",
    "linear.dW = np.random.randn(N, D)\n",
    "linear.db = np.random.randn(D,)\n",
    "\n",
    "sgd = SGD(1e-2)\n",
    "\n",
    "sgd.update(linear)\n",
    "\n",
    "correct_W = [[-0.10241721, -1.46964209,  1.07816109, -0.24488267],\n",
    "             [-0.48024364, -1.00250896,  0.89382974, -1.0787318 ],\n",
    "             [ 0.62884208, -0.56053815,  0.03772041, -0.22940006],\n",
    "             [ 0.58673243,  0.75482687, -1.05771443,  1.06682406],\n",
    "             [ 0.74180098,  1.07106567,  1.53120796, -1.50966439]]\n",
    "correct_b = [ 1.86566377, -1.59381353, -0.62684131,  0.33332912]\n",
    "\n",
    "print('W Relative error:', np.linalg.norm(correct_W - linear.W))\n",
    "print('b Relative error: ', np.linalg.norm(correct_b - linear.b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "B78R9RWgKWFD"
   },
   "source": [
    "## 3. Test Network on CIFAR10\n",
    "CIFAR10 is a dataset of 50,000 32x32x3 color training images and 10,000 test images, labeled over 10 categories (labels are 0 to 9)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "bedhflJ8RE_n"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import cifar10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YzulAL2CTz5Y"
   },
   "source": [
    "### 3.1. Load Data\n",
    "download the dataset using cifar10.load_data() and store the results into (X_train, y_train), (X_val, y_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_lx0mp8vOfZf",
    "outputId": "7083f27f-d4ac-48b6-db90-7445f042673c",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
      "170500096/170498071 [==============================] - 229s 1us/step\n",
      "(50000, 32, 32, 3) (50000, 1)\n",
      "(10000, 32, 32, 3) (10000, 1)\n"
     ]
    }
   ],
   "source": [
    "(X_train, y_train), (X_val, y_val)= cifar10.load_data()\n",
    "print(X_train.shape, y_train.shape)\n",
    "print(X_val.shape, y_val.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3po6F_CYOfZf"
   },
   "source": [
    "flatten the images (reshape the images to single dimension vectors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cOFG4QMjOfZf",
    "outputId": "0c1785e6-6469-4cff-dc59-3562c74a4c8e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((50000, 3072), (10000, 3072))"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train = X_train.reshape(50000,3072)\n",
    "X_val = X_val.reshape(10000,3072)\n",
    "X_train.shape, X_val.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "bJh3JJGjrQ-h",
    "outputId": "eab19538-f257-4f12-a04b-f7d0417ba748"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6, 9, 9, ..., 9, 1, 1], dtype=uint8)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train\n",
    "y_train.squeeze()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Fm6WcPL8OfZg"
   },
   "source": [
    "convert the labels to scalers (using `np.squeeze()` method)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "jp-q_Uw2OfZg",
    "outputId": "dad90b8f-2075-4bda-b28b-1ee1c829a340"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((50000,), (10000,))"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train = y_train.squeeze()\n",
    "y_val = y_val.squeeze()\n",
    "y_train.shape, y_val.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z0oq5JkMOfZg"
   },
   "source": [
    "### 3.2. Normalization\n",
    " Run the cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "EsctzHrpOfZg",
    "outputId": "d45d538c-c09a-4f31-fd33-187fcbbe5cde"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "255 0\n",
      "255 0\n"
     ]
    }
   ],
   "source": [
    "print(X_train.max(), X_train.min())\n",
    "print(X_val.max(), X_val.min())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JeiQ7zurOfZh"
   },
   "source": [
    "rgb values range from 0 to 255. normalize the data by dividing every value by 255."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "OGMC2nHbOfZh",
    "outputId": "d2d9166c-08bd-4425-8d1b-2e124838ae5a",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0 0.0\n",
      "1.0 0.0\n"
     ]
    }
   ],
   "source": [
    "X_train = X_train / 255\n",
    "X_val = X_val / 255\n",
    "print(X_train.max(), X_train.min())\n",
    "print(X_val.max(), X_val.min())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NJNdvzNHOfZh"
   },
   "source": [
    "### 3.3. Plot Function\n",
    "This function get a history of training (loss, acc, val_loss, val_acc) as input and plots it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "4Q35JDeqOfZh",
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "def plot_history(history):\n",
    "    losses, accs, val_losses, val_accs = history\n",
    "    x = [i for i in range(len(losses))]\n",
    "    \n",
    "    # plot for losses\n",
    "    plt.plot(x, losses, '-g', label='train')\n",
    "    plt.plot(x, val_losses, '-r', label='validation')\n",
    "    plt.xlabel('epoch')\n",
    "    plt.ylabel('loss')\n",
    "    plt.legend(loc='upper right')\n",
    "    plt.show()\n",
    "    \n",
    "    plt.plot(x, accs, '-g', label='train')\n",
    "    plt.plot(x, val_accs, '-r', label='validation')\n",
    "    plt.xlabel('epoch')\n",
    "    plt.ylabel('accuracy')\n",
    "    plt.legend(loc='upper left')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q3XG5NOyOfZi"
   },
   "source": [
    "# 2. Run the model(5 points)\n",
    "\n",
    "model specification:\n",
    "\n",
    "\n",
    "1.   1 hidden layer with 100 neurons (use `Linear` module)\n",
    "2.   one of the activation functions (`ReLU` or `sigmoid`)\n",
    "3.   a 10 neurons output layer (use `Linear` module)\n",
    "4.   softmax corss entropy loss (use `SoftmaxCrossentropy` module)\n",
    "\n",
    "\n",
    "\n",
    "We train the model for 10 epochs and `batch_size = 1024`. We also plot the history of training.\n",
    "\n",
    "You can use `model.add(Module)` to add a module to the model. The modules are added in order.\n",
    "It's also recommended to read the `model.py` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "t_1Mw7xYOfZj",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1: loss = 14.9747, acc = 0.1501, val_loss = 14.2288, val_acc = 0.1772\n",
      "Epoch 2: loss = 13.8589, acc = 0.1839, val_loss = 13.6353, val_acc = 0.1851\n",
      "Epoch 3: loss = 13.1980, acc = 0.1951, val_loss = 13.0029, val_acc = 0.1936\n",
      "Epoch 4: loss = 12.6340, acc = 0.1952, val_loss = 12.3533, val_acc = 0.1984\n",
      "Epoch 5: loss = 12.0181, acc = 0.1973, val_loss = 11.4516, val_acc = 0.2033\n",
      "Epoch 6: loss = 11.2625, acc = 0.2004, val_loss = 10.9509, val_acc = 0.2077\n",
      "Epoch 7: loss = 10.2942, acc = 0.1994, val_loss = 9.9636, val_acc = 0.1812\n",
      "Epoch 8: loss = 9.2665, acc = 0.1962, val_loss = 7.8199, val_acc = 0.2156\n",
      "Epoch 9: loss = 7.9736, acc = 0.1917, val_loss = 7.2783, val_acc = 0.1947\n",
      "Epoch 10: loss = 6.6432, acc = 0.1942, val_loss = 6.0691, val_acc = 0.2084\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEGCAYAAABiq/5QAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA1g0lEQVR4nO3dd1xV9f/A8dcHxD3DPRLUnCAiKGriSs2VG3fixFVa9i311+5blk3NiSs1t6ippWnujaKJC/2auciFew/k8/vjQ05U0HvvAe77+XjwCC5nvM9N3ufcz/mc91tprRFCCOE8XKwOQAghhGNJ4hdCCCcjiV8IIZyMJH4hhHAykviFEMLJpLE6gMTImTOn9vDwsDoMIYRIUbZt23ZGa53r4ddTROL38PAgIiLC6jCEECJFUUodSeh1GeoRQggnI4lfCCGcjCR+IYRwMilijF8IkXrcvn2b6Ohobty4YXUoqUb69OkpWLAgbm5uiVpeEr8QwqGio6PJkiULHh4eKKWsDifF01pz9uxZoqOj8fT0TNQ6MtQjhHCoGzdu4O7uLknfRpRSuLu7J+kTlCR+IYTDSdK3raS+n6k68a8/up7vN31PnI6zOhQhhEg2UnXin7FrBu8se4dG0xsRczXG6nCEEMnEhQsXGDVqVJLXa9CgARcuXLB9QA6WqhP/iAYjGFF/BCsPrcRnjA8rD620OiQhRDLwuMQfGxv7xPUWL15M9uzZ7RSV46TqxK+Uok/FPoR3CydruqzUnlKbD1Z+QGzck//nCiFSt4EDB3Lw4EHKlStHhQoVCAwMpHHjxpQuXRqApk2b4ufnR5kyZRg7duzd9Tw8PDhz5gyHDx+mVKlSdO/enTJlylC3bl2uX79u1eEkmVNM5/TJ68O2kG30XdKXL9Z9warDq5jefDqFsxe2OjQhnNpbv7/FjpM7bLrNcnnLMbTe0Ccu89VXX7F792527NjB6tWradiwIbt37747HXLixIm88MILXL9+nQoVKtCiRQvc3d0f2MaBAweYMWMG48aNo1WrVsydO5cOHTrY9FjsJVVf8d8vU9pMTGgygenNp7Pr1C7KhZZjXtQ8q8MSQiQDFStWfGAO/I8//oiPjw+VKlXi2LFjHDhw4JF1PD09KVeuHAB+fn4cPnzYQdE+P6e44r9fW++2VCxQkTZz29Bidgt6+ffiu7rfkcEtg9WhCeF0nnZl7iiZMmW6+/3q1atZvnw5mzZtImPGjNSoUSPBOfLp0qW7+72rq2uKGupxmiv++xV9oSgbumzgP5X/w+iI0QSMD2BvzF6rwxJCOEiWLFm4fPlygr+7ePEiOXLkIGPGjOzbt4/Nmzc7ODr7c8rED5DWNS3f1P2Gxe0Wc/LKSfzH+jN++3i01laHJoSwM3d3d15++WW8vLx49913H/hdvXr1iI2NpVSpUgwcOJBKlSpZFKX9qJSQ6Pz9/bU9G7GcuHyC1+e/zopDK2hdpjWhjULJlj6b3fYnhDOLioqiVKlSVoeR6iT0viqltmmt/R9e1mmv+O+XL0s+lnZYyuBagwnbG4ZvqC/h0eFWhyWEEHYhiT+eq4srgwIHsbbzWuJ0HFV/qsrXG76Wcg9CiFRHEv9DqhSqwp89/qRJiSYMWD6A+tPqc+rKKavDEkIIm5HEn4AcGXIwJ2gOYxqOYe2RtfiM8eGPg39YHZYQQtiEJP7HUErRw78HW7tvxT2jO3Wn1mXg8oHcvnPb6tCEEOK5SOJ/Cq/cXmztvpXu5bszZMMQAn8K5ND5Q1aHJYQQzyx1J/5r1yDu+W/OZnTLyNjXxjKr5SyizkRRLrQcs/fMtkGAQoiUIHPmzAAcP36cli1bJrhMjRo1eNq086FDh3Lt2rW7P1tV5jl1J/7PP4eiReG//4Vjx557c63KtGJHjx2UylmK1mGtCVkUwrXb156+ohAiVcifPz9hYWHPvP7Did+qMs+pO/FXrmwS/0cfgYcHNGgA8+bBrVvPvEnPHJ6s67yOgS8PZNz2cVQYV4Hdp3fbLmYhhN0NHDiQkSNH3v35k08+4fPPP+eVV16hfPnyeHt7s2DBgkfWO3z4MF5eXgBcv36dNm3aUKpUKZo1a/ZArZ5evXrh7+9PmTJl+PjjjwFT+O348ePUrFmTmjVrAvfKPAN8//33eHl54eXlxdChQ+/uzy7ln7XWyf7Lz89PP5eDB7X+4AOt8+fXGrTOnVvr//xH66io59rssr+W6Tzf5NHpP0+vx2wdo+Pi4p4vTiGcwN69e+/90K+f1tWr2/arX7+nxrB9+3ZdrVq1uz+XKlVKHz16VF+8eFFrrXVMTIwuWrTo3b/pTJkyaa21PnTokC5TpozWWuvvvvtOd+7cWWutdWRkpHZ1ddVbt27VWmt99uxZrbXWsbGxunr16joyMlJrrXXhwoV1TEzM3f3++3NERIT28vLSV65c0ZcvX9alS5fW27dv14cOHdKurq76zz//1FprHRQUpH/++eenv6/xgAidQE612xW/UmqiUuq0UuqRy2Gl1DtKKa2Uymmv/T+gSBEz3HPkCPz6K1SpAkOHQqlSEBgIkyfD1atJ3mydonWI7BlJtcLV6PlbT4LmBHH++nnbxy+EsClfX19Onz7N8ePHiYyMJEeOHOTNm5f/+7//o2zZstSuXZt//vmHU6ce/wzP2rVr79bfL1u2LGXLlr37u9mzZ1O+fHl8fX3Zs2cPe/c+uQjk+vXradasGZkyZSJz5sw0b96cdevWAfYp/2zPssyTgBHAlPtfVEoVAuoCR+2474SlSQMNG5qvkydhyhQYPx46dYI334R27aBbN/Dzg0R2rc+TOQ9L2i/h243f8v7K99l6fCszWsygSqEq9j0WIVKD+CENKwQFBREWFsbJkydp3bo106ZNIyYmhm3btuHm5oaHh0eC5Zif5tChQ3z77bds3bqVHDly0KlTp2fazr/sUf7Zblf8Wuu1wLkEfvUD8B5gbXW4vHnhvfdg/35YuxaaNTMnggoVwNcXhg+HcwmF/ygX5cJ7L7/H+s7rcVEuVPupGoPXDeZO3B07H4QQ4lm1bt2amTNnEhYWRlBQEBcvXiR37ty4ubmxatUqjhw58sT1q1WrxvTp0wHYvXs3O3fuBODSpUtkypSJbNmycerUKZYsWXJ3nceVgw4MDOSXX37h2rVrXL16lfnz5xMYGGjDo32QQ2/uKqWaAP9orSMTsWyIUipCKRURExNjz6DuDfecOAGjR5tPBn37Qv780L49rFqVqGmhAQUD2NFjBy1Lt+T9le/z6tRXOXH5hP1iF0I8szJlynD58mUKFChAvnz5aN++PREREXh7ezNlyhRKliz5xPV79erFlStXKFWqFB999BF+fn4A+Pj44OvrS8mSJWnXrh0vv/zy3XVCQkKoV6/e3Zu7/ypfvjydOnWiYsWKBAQE0K1bN3x9fW1/0PHsWpZZKeUB/Kq19lJKZQRWAXW11heVUocBf631madtx95lmRP0558wYQJMmwYXLpj7BF27mmGh/PmfuKrWmol/TuTNJW+SOW1mJjedTP2X6jskbCGSOynLbB/JtSxzUcATiIxP+gWB7UqpvA6MIfF8fWHECDh+HKZOhRdfhPffh0KF4LXXYMECuJ1w+QalFF3LdyUiJII8mfPQYHoDui3sRlRMlIMPQgghHuWwxK+13qW1zq219tBaewDRQHmt9UlHxfBMMmS4N9xz4AAMGAAREdC0qTkZDBpkXk9A6Vyl2dJtC/0C+jF151RKjypNvan1WHJgiZR7FkJYxp7TOWcAm4ASSqlopVRXe+3LYYoVg8GDzVPACxZAxYrwzTdQvDjUqGE+GTx0xz2DWwaG1hvK0beP8t+a/yXyVCQNpjeg1MhSjNwykiu3rlhzLEJYyJ5DzM4oqe+ntF58XidOmBvD48fDwYOQLZv5hNCtmxkuesitO7eYs2cOw8KHsfX4VrKly0ZX3668UfENPHN4WnAAQjjWoUOHyJIlC+7u7qhETpsWj6e15uzZs1y+fBlPzwdzyOPG+CXx20pcnJkWOn48zJ0LN25A+fLmBNC2LTxUj0NrzebozQwLH0bY3jA0msYlGtMvoB/VC1eXPwiRat2+fZvo6OjnmtsuHpQ+fXoKFiyIm5vbA69L4nek8+dh+nQYNw4iIyF9emjdGnr3Ns8JPJTUoy9FM2rrKMZuG8vZ62fxyeND34C+tPNuR/o06S06CCFESieJ3wpaw/bt5lPA1Klw5Yp5Krh3b2jTBjJmfGDx67evM23XNIaFD2P36d3kzJiTHn496F2hN/mzPHkKqRBCPEwSv9UuXzbJf+RI2LMHcuSAzp2hZ0946aUHFtVas+rwKoaFD2PR/kW4urgSVDqIfgH9CCgYYNEBCCFSGkn8yYXWsG4djBpl7gXExkLdutCnj6kh5Or6wOIHzx1k+JbhTPxzIpdvXSagQAD9AvrRsnRL3FzdHrMTIYSQxJ88nTxp7gOEhsI//5jnAnr0ME8I58nzwKKXbl5i0o5JDN8ynL/O/UX+LPnpU6EPIX4h5MzomCKnQoiURRJ/chYbC4sWmWGgFSvAzQ2Cgsy9gCpVHrgZHKfjWHJgCUPDh7L87+WkT5Oe9t7t6RfQD+883hYehBAiuZHEn1Ls2wdjxsCkSXDxIpQta04A7dtDfN/Pf+05vYcfw3/k550/cz32OjU9atIvoB+NijfC1cU14e0LIZyGJP6U5upVMyV05EgzJTRrVggOhl69TAOZ+5y7fo5x28YxcutIjl06RpEcRXijwht08e1CtvTZLDoAIYTVJPGnVFrD5s3mZvDs2aZfcK1a5lNA48ZmWChebFws86PmMyx8GBuObSBz2sx08unEmwFvUty9uIUHIYSwgiT+1OD0aZg40QwFHTliykOHhJivfPkeWHTb8W0MCx/GzN0zuR13mwYvNaBfQD/qFKkjTwUL4SQk8acmd+7A4sXmU8Dvv5vGMc2amU8B1as/cDP45JWTjIkYw+iI0Zy+epoiOYrQsWxHOvp0lNpAQqRykvhTq7/+MtNBJ040rSJLlzYngNdfN/cF4t2MvcnsPbP5acdPrDq8CoBqhasR7BNMy9ItyZou6+P2IIRIoSTxp3bXr8OsWeZTwNatkCmTSf69e4P3g9M8j1w4wtSdU5kcOZkD5w6QIU0GmpVqRrBPMK94viIzgoRIJSTxO5OtW80JYOZMUyU0MNCcAJo3h7Rp7y6mtSb8n3Am75jMzD0zuXDjAvmz5KeDdweCywVTOldpCw9CCPG8JPE7o7NnzfMAo0ebXgF58kD37qY+UIECDyx6I/YGi/YvYnLkZH7/63fu6Dv45/cn2CeYtl5tcc/obs0xCCGemSR+ZxYXB8uWmWcCfvvN1ANq0QLefPORJ4MBTl05xfRd05kcOZnIU5G4ubjRsHhDgn2CafBSA9K6pn3MjoQQyYkkfmH8/bc5AUyYYJ4M9vU1J4C2bU3fgIdEnoxkSuQUpu2axqmrp3DP4E5br7YElwvGL5+fTA0VIhmTxC8edPWqKRM9fLgpE+3ubp4H6NULChV6ZPHYuFiW/rWUyZGTWbB/Abfu3KJ0rtIE+wTToWwH6RcgRDIkiV8kTGtYvdqcABYsMMM+TZuaTwHVqj0yDARw/vp5Zu+ZzeTIyWyK3oSLcqF2kdoE+wTTtGRTMrplfGQdIYTjSeIXT3fkiJkNNH68eSbA29ucANq3f6Rb2L8OnD3AlMgpTNk5haMXj5IlbRZalWlFR5+OBL4YKENBQlhIEr9IvGvXYMYM8ykgMtJ0C+va1TSL8fBIcJU4Hceaw2uYsnMKc/bM4ertq3hm96Sjj3lKuEiOIo49BiGEJH7xDLSG9evhxx9h/nwzO6hxY/MpoFatBIeBAK7eusq8qHlMjpzMykMr0WgCXwyko09HgkoHScVQIRxEEr94PseOmeJwY8fCmTOmNMQbb5ingx/qE/DAaheP3X1KeP/Z/aRPk55mJZvRrXw3anrUlKEgIexIEr+wjRs3TGmI4cNh2zbIls00je/TB4oVe+xqWmu2/LOFKZFTmLF7BudvnKe4e3FCyofQqVwneUBMCDuQxC9s698+AT/+CGFhpmJo/frQty/UqQMuLo9d9frt64TtDSN0Wygbjm0gnWs6WpZuSQ+/HlR9sap8ChDCRiTxC/s5ftxUCA0NhVOnoHhxMwwUHPxAhdCE7D69m9CIUKbsnMKlm5conas0Pfx60NGnI9nTZ3dM/EKkUpL4hf3dvGmu/ocPh/BwM/bfqZM5CZQo8cRVr966yqw9sxgTMYatx7eSIU0G2ni1oYdfDyoWqCifAoR4BpL4hWNt2WJOALNmwe3bULeumQ3UoMETh4EAtp/YTmhEKNN2TePq7auUy1uOHn49aO/dnizpsjjoAIRI+STxC2ucOmVmAo0eDSdOQJEi5hNASIjpGfAEl25eYvqu6YyJGEPkqUgyp81MO6929PTviW8+XwcdgBAplyR+Ya3bt2HePPMpYMMGUw/ou++gZcvHPg/wr39nBIVuC2Xm7plcj71OhfwV6OHXgzZebciU9sknECGclSR+kXysW2eGfSIjzYNgP/4IZcokatULNy7wc+TPjNk2hr0xe8maLiuvl32dHn498M7j/fQNCOFEJPGL5OXOHTML6IMP4NIlMw3044/NcwGJoLVmw7ENjIkYQ9jeMG7eucnLhV6mh18PWpZuSQa3DHY+ACGSP0n8Ink6cwbefx/GjYPcuWHIEPM08FNuAD+wiWtnmLxjMqHbQjlw7gAvZHiBYJ9gevj1oETOJ88mEiI1k8Qvkrdt28xN382boVIlGDEC/PyStAmtNasOryJ0WyjzouYRGxdLDY8a9PTrSbNSzaRzmHA6j0v8ib+sSvoOJyqlTiuldt/32jdKqX1KqZ1KqflKqez22r9IYfz8zE3fSZNMl7AKFaBHD/OJIJGUUtTyrMWslrOIfjuaL1/5kiMXjtBmbhsKfl+QgcsHcvDcQfsdgxAphN2u+JVS1YArwBSttVf8a3WBlVrrWKXUEACt9YCnbUuu+J3MxYvw6afmpm/WrPD55+Yk4Oqa5E3F6Tj+OPgHodtCWbh/IXf0HeoUqUNP/568Vvw13Fzd7HAAQiQPDr/i11qvBc499NoyrXVs/I+bgYL22r9IwbJlg++/N7N+fH1NATh/f1MiOolclAuvFnuVea3nceStI3xa41OizkTRYnYLPIZ58P2m77l2+5odDkKI5MtuiT8RugBLHvdLpVSIUipCKRURExPjwLBEslGmDCxfDrNnw9mzEBhobvyeOPFMmyuQtQAfVf+IQ/0OsbDNQkrmLMk7y97Bc5gn3278lqu3rtr4AIRInixJ/Eqp94FYYNrjltFaj9Va+2ut/XPlyuW44ETyohQEBUFUlJn9M3u2KQL37bdw69YzbTKNSxpeK/EaKzquYF3ndZTNU5Z3/3gXz2GefL3ha67cumLjgxAieXF44ldKdQIaAe11SphSJJKHTJnMWP+ePVCjBrz7LpQtC8uWPddmq75YlT9e/4MNXTbgm8+XAcsH4DnMk6/Wf8Xlm5dtE7sQyYxDE79Sqh7wHtBYay0DqyLpihWDRYvg11/NQ2CvvgrNm8Phw8+12SqFqrC0w1I2dd2Ef35/Bq0YhMcwDwavG8ylm5dsE7sQyYQ9p3POADYBJZRS0UqprsAIIAvwh1Jqh1JqjL32L1K5hg1h924YPBiWLoVSpcxMoOvXn2uzlQpWYkn7JYR3C6dSwUq8v/J9PIZ68Pnaz7l446KNghfCWvIAl0j5jh0zQz+zZoGHB/zwAzRp8tTib4mx9Z+tfLb2M379369kT5+dtyu9Td+AvtIkRqQIDp/OKYTDFCoEM2fCypXmXkCzZqYN5P79z73pCgUqsKjtIiK6R1CtcDU+Xv0xHkM9+GT1J5y/ft4GwQvheJL4RepRsyb8+ScMHQqbNoG3NwwYAJef/yatX34/FrRZwPaQ7dTyrMWnaz7FY5gHH636iHPXzz19A0IkI5L4Reri5gb9+sH//gcdOsDXX5u2j9Onmwbxz8k3ny/zWs9jR48d1ClSh/+u/S8eQz14f8X7nL121gYHIIT9SeIXqVOePDBxornyL1AA2reH6tXN08A24JPXh7BWYezsuZN6xerx5fov8RjmwaDlgzhzLfH1hYSwgiR+kbpVqmQav48bZx4CK1/eVAE9Z5vhGe883swOms2uXrto+FJDhmwYgsdQDwb8MYCYq/LEuUieJPGL1M/FBbp1M8M/vXub/r/Fi5tGMFdtU6ahTO4yzGw5k929d9O4RGO+2fgNHsM8eHfZu5y+etom+xDCVmQ6p3A+kZGm9eO6dZAxIzRoYHr/NmwImTPbZBf7zuzj87WfM2P3DNK5pqOXfy/effld8mbOa5PtC5EY0ohFiPtpDWvWQFgYzJ0LJ09C+vRQr545CTRqlOg2kE/yv7P/4/O1nzNt1zTSuqalp19P3nv5PfJlyWeDgxDiySTxC/E4d+7Axo33TgL//ANp00LduuYk0Lgx5MjxXLs4cPYAX6z7gqk7p+Lm6kZI+RAGVB1A/iz5bXQQQjxKEr8QiREXZ24Gh4WZr6NHzRTR2rXNSaBJE3B3f+bNHzx3kC/WfcGUyCmkcUlD9/LdGVB1AAWzSmsKYXuS+IVIKq0hIgLmzDEngUOHTBewWrXMSaBpU9Mg/hn8ff5vBq8bzOTIybi5uPFJjU/oX7k/aVzS2PYYhFOTxC/E89DaPBUcFmZOBH/9ZWYLVa9uTgLNm0PepN+4PXT+EP2X9eeXfb9QLm85xr02Dv/8j/ydCvFMJPELYStaw65d904C+/aZgnCBgfdOAgUKJGmT86Lm8cbiNzh19RT9AvrxWc3PyJzWNjOMhPOSxC+EvezZc++ewO7d5rUqVcxJoEULePHFRG3mwo0LDFw+kNBtoRTOVpjRDUdT/6X6dgxcpHaS+IVwhH37zMygsDDYscO8VrGiOQm0bAmenk/dxPqj6+m+qDv7zuyjrVdbhtYbSu5Mz3YvQTg3SfxCONpff907Cfz779fP795JoFixx656M/YmX63/ii/WfUGWdFn4ru53BPsEo2zQY0A4D0n8Qljp0KF7J4HwcPOaj485AbRuDS+9lOBqUTFRdF/UnQ3HNlDLsxahjUIp9sLjTxhC3E8asQhhJU9P+M9/YPNmOHLEdAnLnBk+/NC0jRw4MMG2kaVylWJt57WMaTiGiOMReI/25qv1X3H7zm0LDkKkFpL4hXC0F1+Et96C9eshOho6d4YhQ8wngLVrH1ncRbnQw78HUX2iaPBSAwatGIT/OH+2/LPF8bGLVEESvxBWKlDAlIxevtyUjqhe3VQQvXTpkUXzZ8nP3FZzmd96PmevnaXyhMq89ftbXLl1xYLARUomiV+I5OCVV2DnTujf35SLLlMGfvstwUWblmzK3j576enXkx/Df6TMqDIsPrDYwQGLlCxRiV8p1U8plVUZE5RS25VSde0dnBBOJVMm+O470zUse3ZTIbR9e4h5tKFL1nRZGdlwJOu7rCdz2sw0nN6QNmFtOHXllOPjFilOYq/4u2itLwF1gRzA68BXdotKCGdWsSJs2waffmqeDC5dGmbMSLBncJVCVfizx598VuMz5u+bT8mRJZmwfQIpYbaesE5iE/+/k4cbAD9rrffc95oQwtbSpoWPPjL1gYoWhXbtTHno6OhHF3VNy4fVPySyZyTeub3ptqgbtabU4sDZAxYELlKCxCb+bUqpZZjEv1QplQWIs19YQgjAjPVv2GCmf65caa7+x4wx5aMfUjJnSVZ3Ws3YRmP588SfeI/2ZvC6wdy6c8uCwEVyltjE3xUYCFTQWl8D3IDOdotKCHGPq6uZ/rl7NwQEQK9eULMmHHj0it5FudDdrztRfaJoXKIx7698H7+xfoRHhzs+bpFsJTbxVwb2a60vKKU6AB8AF+0XlhDiEZ6esGwZTJxoZgCVLQtffw2xsY8smi9LPmYHzWZhm4VcuHGByhMq03dJXy7fvGxB4CK5SWziHw1cU0r5AO8AB4EpdotKCJEwpcwDX3v3Qv36MGCA+RQQGZng4q+VeI09vffQp0IfRmwZQelRpVm0f5GDgxbJTWITf6w20wSaACO01iOBLPYLSwjxRPnywbx5pvbPP/+Avz988AHcuPHIolnTZWV4g+Fs7LqR7Omz03hmY1rNacXJKyctCFwkB4lN/JeVUoMw0zh/U0q5YMb5hRBWatHCXP136ABffAG+vuZmcAIqFazEtpBtfFHrCxbuX0ipkaUYt20ccVrmaTibxCb+1sBNzHz+k0BB4Bu7RSWESLwXXoCffoKlS02ht8BA6NsXrjxayiGta1r+L/D/2NlrJ+XyliPk1xBqTq7J/jP7LQhcWCVRiT8+2U8DsimlGgE3tNYyxi9EclK3rpn58+abMGKEmQq6dGmCixZ3L87KjiuZ0HgCu07touyYsny+9nOp+ukkEluyoRWwBQgCWgHhSqmW9gxMCPEMMmeGYcNM5c+MGaFePQgOhrNnH1lUKUUX3y5E9YmiacmmfLjqQypPqMye03ssCFw4UmKHet7HzOEP1lp3BCoCH9ovLCHEc6lSxbR+/PBDmD7dPPg1Z06CZR/yZM7DrJazCAsK48jFI5QfW54h64dwJ+6O4+MWDpHYxO+itT59389nk7CuEMIK6dLBZ5+Zuj+FCkGrVtC8ORw/nuDiLUq3YE/vPTQq3oiBKwZS9aeqMvafSiU2ef+ulFqqlOqklOoE/AZIHVghUoKyZU3nr2++gd9/N1f/EyYkePWfO1NuwoLCmN58OvvP7KdcaDl+2PSDzPxJZRJ7c/ddYCxQNv5rrNZ6wJPWUUpNVEqdVkrtvu+1F5RSfyilDsT/N8fzBC+ESKQ0aUzrx127oFw56NYNateGv/9+ZFGlFG2927Kn9x7qFKlD/2X9qTGpBgfPHXR83MIuEj1co7Weq7XuH/81PxGrTALqPfTaQGCF1volYEX8z0IIRylWzBR7Cw2FiAjw8jIF4O48Op6fL0s+FrRZwKQmk9h5aidlx5Rl1NZRcvWfCjwx8SulLiulLiXwdVkp9WhvuPtordcC5x56uQkwOf77yUDTZw1cCPGMXFwgJAT27DGdv/r3NzeD9zw6m0cpRXC5YHb33k3gi4H0WdyHOj/X4fCFw46PW9jMExO/1jqL1jprAl9ZtNZZn2F/ebTWJ+K/PwnkedyCSqkQpVSEUioiJoEOREKI51SwICxcaJq8/P236ff7mBu/BbMWZEn7JYxtNJYt/2zBe7Q347aNk4YvKZRlM3Pia/889l+N1nqs1tpfa+2fK1cuB0YmhBNRCtq0gXXr4No1UwAugVr/ZlFFd7/u7Oq1iwr5KxDyawj1p9Un+tKjzWFE8uboxH9KKZUPIP6/p5+yvBDCEUqWNGP9y5bBjz8+cVGP7B4s77icEfVHsO7oOrxGeTF5x2S5+k9BHJ34FwLB8d8HAwscvH8hxOOEhJj2jgMGmHr/T+CiXOhTsY9p95jHm04LOtFkZhNOXD7xxPVE8mC3xK+UmgFsAkoopaKVUl0xDdrrKKUOALWRhu1CJB9Kwfjxpuhbu3am4NtTFHuhGKuDV/N93e/54+8/8BrtxYxdM+TqP5mzW+LXWrfVWufTWrtprQtqrSdorc9qrV/RWr+kta6ttX541o8Qwkq5csGkSWaGz4AnPqpzl6uLK29XfpsdPXbw0gsv0W5eO4LmBHH6qozkJldSdkEI8aBXXzU9focPh8WJf0C/RM4SrO+ynq9e+YpF/1uE1ygv5u6da784xTOTxC+EeNSXX4K3t5nlczrxV+5pXNIwoOoAtoVso1C2QrSc05J2c9tx9tqj1UGFdSTxCyEelT69qep58SJ06ZJgXZ8n8crtxeaum/msxmfM2TsHr9Fe0us3GZHEL4RImJeXKez2228wenSSV3dzdePD6h+ytftWcmfKTeOZjen0Sycu3Lhg+1hFkkjiF0I83htvmGYu77xjevs+g3J5y7G1+1Y+CPyAqTun4jXKi9//+t3GgYqkkMQvhHg8pUw/3yxZoH17uHnzmTaT1jUt/631XzZ13UTWdFmpP60+IYtCuHTziSW/hJ1I4hdCPFnevDBxouno9cEHz7WpCgUqsL3Hdt6r8h4T/pyA92hvVvy9wjZxikSTxC+EeLpGjaB3b/j2W1i+/Lk2lT5NeobUGcL6zutJ55qO2j/Xps9vfbhy64qNghVPI4lfCJE433xjavo8pnl7UlUuVJkdPXfwVsBbjI4Yjc8YH9YdWWeDQMXTSOIXQiROxoxmimdMDHTvnuQpnglu0i0jP9T7gdWdVqO1pvqk6ry/4n1p9mJnkviFEInn6wuDB8P8+Wbc30aqFa7Gzl476VyuM4PXD6bVnFZcu33NZtsXD5LEL4RImv79oVYt6NsX/vc/m202c9rMjG88nu/qfse8qHlUn1Rdqn3aiSR+IUTSuLjAlCmQLp2Z4nn7ts02rZSif+X+/NLmF6Jioqg4viKRJyNttn1hSOIXQiRdgQIwbpxp2P7JJzbffOMSjVnfZT1aa6r+VJVf//erzffhzCTxCyGeTYsW0LWrKei2dq3NN18ubzm2dN9CCfcSNJnZhKGbh0qdfxuRxC+EeHZDh0LRotChA1y4YPPN58+SnzWd1tCkRBPeXvo2fRb3ITYu1ub7cTaS+IUQzy5zZjPF88QJ6NnTJlM8H5YpbSbCWoXxXpX3GB0xmobTG3LxxkWb78eZSOIXQjyfChXg009h1iyYOtUuu3BRLgypM4Txr41n5aGVVJlYhUPnD9llX85AEr8Q4vkNGACBgdCnD/z9t91207V8V5Z1WMaJyycIGB/AxmMb7bav1EwSvxDi+bm6ws8/m6meHTpArP3G4Wt61rxb5bPW5FrM2DXDbvtKrSTxCyFso3BhGDMGNm2CL76w665K5CxBeLdwAgoG0G5eOz5d/anM+EkCSfxCCNtp0wZefx0++8ycAOzIPaM7yzosI9gnmE/WfEKH+R24EXvDrvtMLSTxCyFsa8QIc/Xfvj1csm+jlXRp0vFTk58YXGsw03dN55UprxBzNcau+0wNJPELIWwra1Yzu+fIEVPPx86UUgwKHMTslrPZfmI7AeMD2BvzbG0inYUkfiGE7VWpAh9+CJMnm2meDhBUJog1ndZw7fY1Kk+ozB8H/3DIflMiSfxCCPv44AOoVMk82HX0qEN2WbFARbZ030LhbIWpP60+YyLGOGS/KY0kfiGEfaRJA9OmmamdHTvCnTsO2e2L2V5kfZf1vFrsVXr91ou3f3+bO3GO2XdKIYlfCGE/RYqYm71r1pjWjQ6SNV1WFrRZQN+KfRkaPpSms5pKT9/7SOIXQthXx47QqpUZ84+IcNhu07ikYVj9YYyoP4LFBxZTdWJVjl085rD9J2eS+IUQ9qWUebArb15o1w6uXnXo7vtU7MNv7X7j7/N/EzA+gIjjjjv5JFeS+IUQ9pcjhynp8Ndf8PbbDt99vWL12Nh1I2ld01Ltp2rMi5rn8BiSE0n8QgjHqFHDFHMbN840a3cwr9xehHcLxyevDy1mt2DI+iFOW+ZBEr8QwnE+/RT8/KBbNzh+3OG7z5M5Dys7rqR1mdYMXDGQbgu7cevOLYfHYTVJ/EIIx0mb1kzxvHEDgoMhLs7hIWRwy8D0FtP5sNqHTNwxkVenvsq56+ccHoeVJPELIRyrRAnTsnH5cvNfC7goFz6r+Rk/N/uZjcc2UnlCZQ6cPWBJLFawJPErpd5WSu1RSu1WSs1QSqW3Ig4hhEW6dYOmTWHQINixw7IwOpTtwIqOKzh77SyVJlRizeE1lsXiSA5P/EqpAkBfwF9r7QW4Am0cHYcQwkJKmZu87u5miue1a5aFUvXFqoR3Cyd3ptzU+bkOk3ZMsiwWR7FqqCcNkEEplQbICDj+Lo8Qwlo5c5oiblFR8N57loZS9IWibOq6iWqFq9F5QWf+b8X/Eacdf//BURye+LXW/wDfAkeBE8BFrfWyh5dTSoUopSKUUhExMVJfW4hUqU4d6N8fRo6ExYstDSV7+uwsab+EkPIhfLn+S9qEtUm1jV2sGOrJATQBPIH8QCalVIeHl9Naj9Va+2ut/XPlyuXoMIUQjjJ4MJQtC507w6lTlobi5urGmEZj+Lr218zZO4faU2pz9tpZS2OyByuGemoDh7TWMVrr28A8oIoFcQghkoN06WD6dNOtq0sXsPihKqUU7778LrNaziLieARVJlbh4LmDlsZka1Yk/qNAJaVURqWUAl4BoiyIQwiRXJQpA99+a4Z7evWC3butjohWZVqxvONyzlw7Q+UJlQmPDrc6JJuxYow/HAgDtgO74mMY6+g4hBDJTO/e0L07jB8P3t7g4wNffw3HrKuoWfXFqmzsspHMaTNTc3JNftn3i2Wx2JIls3q01h9rrUtqrb201q9rrW9aEYcQIhlRCsaONaUchg+HjBlNbZ/ChU2dn3Hj4Px5h4dVImcJNnfbjHceb5rPas6P4T86PAZbkyd3hRDJS+7c8MYbsGmTqeb56adw8iSEhJjSzs2aQViYKfvgqJAy5WZV8CqalGxCv9/70X9p/xQ93VMSvxAi+Spa1DRwiYoyTVz69IHNmyEoCPLkMTeDV6xwSFvHjG4ZCQsKo2/Fvvyw+QeC5gRx/fZ1u+/XHlRKKEvq7++vIxzYuUcIkYzduQOrVplib3PnwuXLkC8ftG0L7duDr68ZNrKjoZuH0n9pfyoVrMSCNgvIlSl5TjlXSm3TWvs/8rokfiFEinX9Ovz6qzkJLF4Mt29DyZKmDET79qbnr53M3TuXDvM7UDBrQZa0X0KxF4rZbV/P6nGJX4Z6hBApV4YMZtjnl1/MfYDQUHOP4KOPzDBR5cqm2bsdnv5vUboFKzuu5Pz181QaX4mNxzbafB/2IolfCJE6vPCCuQG8Zg0cOQJffWX6+775phkKatDAfDKwYc/fyoUqs6nrJnJkyEGtybWYu3euzbZtT5L4hRCpz4svmqmgO3ear//8xzwU1qGD+UTQvv29oaHn9JL7S2zquony+coTNCeIHzb9kOxbOkriF0Kkbt7e5ur/8GHzaaBDB1iyBBo2hAIF7k0dfY5knTNjTlZ0XEHzUs3pv6w//X7vx504+880elaS+IUQzsHFBapVM/cBTpww9wVq1IAJE6BKFShWzEwd3bfvmTafwS0Ds4Nm079Sf4ZvGU6L2S24dtu6PgNPIolfCOF80qWDJk1g9mxTEfSnn8wMoMGDoVQp0xD+jz+SvFkX5cJ3r37HsHrDWLh/ITUn1+T01dN2OIDnI4lfCOHcsmaFTp1Moo+Ohu+/N88G1KsHQ4Y80xBQ34C+zGs9j12ndlF5QmX2n9lv+7ifgyR+IYT4V7588PbbsH07tGwJAwdCq1bmRJBETUs2ZVXwKi7fvEyViVVYf3S9HQJ+NpL4hRDiYZkzw8yZ8M03MG8eVKoEBw4keTMBBQPY3G0zOTPmpPaU2szeM9sOwSadJH4hhEiIUmYa6NKl5j6Av795SjiJiuQowsYuG6lQoAKtw1rzzYZvLJ/uKYlfCCGepHZtUyCuaFF47TVTLTQuaZU53TO688frf9CqTCveW/4ebyx+g9i4WDsF/HSS+IUQ4mk8PGDDBujYET75BJo2hYsXk7SJ9GnSM6PFDN6t8i6jIkbRbFYzrt6y3VPESSGJXwghEiNDBpg0yTSJWbIEKlSAvXuTtAkX5cLXdb5mZIORLD6wmBqTa3Dyykm7hPvEOBy+RyGESKmUMk/6rlxpmsNXrGiawiRR7wq9WdBmAXtj9lJpfCWiYhzbdlwSvxBCJFVgIGzbBl5epjrooEFJbgbTqHgj1nRaw43YG1SZWIU1h9fYKdhHSeIXQohnUaCAqf0TEmJqATVoAGfPJmkT/vn92dxtM3kz56Xu1LpM3zXdTsE+SBK/EEI8q3TpTO2fceNg9Woz5XPHjiRtwiO7Bxu7bKRSwUq0n9eer9Z/ZffpnpL4hRDieXXrBmvXmjLPVaqYuv9JkCNDDpZ1WEZbr7YMWjGInr/2tOt0T0n8QghhCwEBZty/QgVT+vmtt5JU7z9dmnRMbT6VQVUHMXb7WBrPaMzlm0kvFZEYkviFEMJW8uSB5cuhXz8YNgzq1IHTia/O6aJcGPzKYEIbhbLs4DKqT6rO8cvHbR6mJH4hhLAlNzcYOhR+/hnCw02J5y1bkrSJEL8QFrVdxOELh+1S2VMSvxBC2EOHDrBxI6RJY6Z/TpiQpNXrv1Sfw28dpqZnTZuHJolfCCHsxdfX1PmpVs3cAO7ZE27eTPTqWdNltUtYkviFEMKe3N3h999N8/fQUKhZE47bftw+KSTxCyGEvbm6moe8Zs+GnTvNuP966xqzSOIXQghHCQqCzZtNo5eaNWHUqGdq7fi8JPELIYQjeXnB1q3w6qvQpw906QLXrzs0BEn8QgjhaNmzw8KF8PHHptRzYCAcPeqw3UviF0IIK7i4mKYuCxeafr5+fqbcsyN27ZC9CCGESNhrr5mhn1y5zJO+339v93F/SfxCCGG14sXNU77NmsE770C7dnDVfm0ZLUn8SqnsSqkwpdQ+pVSUUqqyFXEIIUSykSULzJkDX34Js2ZB5cpw8KBddmXVFf8w4HetdUnAB3Bs3zEhhEiOlIKBA80DX9HRpr7/2rU2343DE79SKhtQDZgAoLW+pbW+4Og4hBAi2apb15R4rlgRPD1tvnkrrvg9gRjgJ6XUn0qp8UqpTA8vpJQKUUpFKKUiYmJiHB+lEEJYydMTli6FQoVsvmkrEn8aoDwwWmvtC1wFBj68kNZ6rNbaX2vtnytXLkfHKIQQqZYViT8aiNZah8f/HIY5EQghhHAAhyd+rfVJ4JhSqkT8S68Aex0dhxBCOKs0Fu33TWCaUiot8DfQ2aI4hBDC6ViS+LXWOwB/K/YthBDOTp7cFUIIJyOJXwghnIwkfiGEcDJKW9D9JamUUjHAkWdcPSdwxobhpHTyftwj78WD5P14UGp4PwprrR95ECpFJP7noZSK0FrLjeR48n7cI+/Fg+T9eFBqfj9kqEcIIZyMJH4hhHAyzpD4x1odQDIj78c98l48SN6PB6Xa9yPVj/ELIYR4kDNc8QshhLiPJH4hhHAyqTrxK6XqKaX2K6X+Uko9UvPfWSilCimlViml9iql9iil+lkdU3KglHKNbwb0q9WxWE36YN+jlHo7/u9kt1JqhlIqvdUx2VqqTfxKKVdgJFAfKA20VUqVtjYqy8QC72itSwOVgD5O/F7crx/S7/lf0gcbUEoVAPoC/lprL8AVaGNtVLaXahM/UBH4S2v9t9b6FjATaGJxTJbQWp/QWm+P//4y5o+6gLVRWUspVRBoCIy3OharSR/sR6QBMiil0gAZgeMWx2NzqTnxFwCO3fdzNE6e7ACUUh6ALxD+lEVTu6HAe0CcxXEkB4nqg+0MtNb/AN8CR4ETwEWt9TJro7K91Jz4xUOUUpmBucBbWutLVsdjFaVUI+C01nqb1bEkE4nqg+0MlFI5MCMDnkB+IJNSqoO1Udleak78/wD3t6cvGP+aU1JKuWGS/jSt9Tyr47HYy0BjpdRhzBBgLaXUVGtDspT0wb6nNnBIax2jtb4NzAOqWByTzaXmxL8VeEkp5Rnf4rENsNDimCyhlFKY8dsorfX3VsdjNa31IK11Qa21B+bfxUqtdaq7qkss6YP9gKNAJaVUxvi/m1dIhTe6req5a3da61il1BvAUsyd+Yla6z0Wh2WVl4HXgV1KqR3xr/2f1nqxdSGJZEb6YANa63ClVBiwHTMb7k9SYekGKdkghBBOJjUP9QghhEiAJH4hhHAykviFEMLJSOIXQggnI4lfCCGcjCR+IexMKVVDKoCK5EQSvxBCOBlJ/ELEU0p1UEptUUrtUEqFxtfrv6KU+iG+PvsKpVSu+GXLKaU2K6V2KqXmx9d4QSlVTCm1XCkVqZTarpQqGr/5zPfVu58W/1SoEJaQxC8EoJQqBbQGXtZalwPuAO2BTECE1roMsAb4OH6VKcAArXVZYNd9r08DRmqtfTA1Xk7Ev+4LvIXpDVEE8zS1EJZItSUbhEiiVwA/YGv8xXgG4DSmbPOs+GWmAvPi69dn11qviX99MjBHKZUFKKC1ng+gtb4BEL+9LVrr6PifdwAewHq7H5UQCZDEL4ShgMla60EPvKjUhw8t96w1Tm7e9/0d5G9PWEiGeoQwVgAtlVK5AZRSLyilCmP+RlrGL9MOWK+1vgicV0oFxr/+OrAmvrtZtFKqafw20imlMjryIIRIDLnqEALQWu9VSn0ALFNKuQC3gT6YpiQV4393GnMfACAYGBOf2O+vZvk6EKqU+ix+G0EOPAwhEkWqcwrxBEqpK1rrzFbHIYQtyVCPEEI4GbniF0IIJyNX/EII4WQk8QshhJORxC+EEE5GEr8QQjgZSfxCCOFk/h8cp8OcmNV6AAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA7LklEQVR4nO3dd3iUVfbA8e8h9B56CU1ApRMSEESQJk1ERBBEXcFdsYG67rqyu666/tS1rbq66oIdpUgSQVQ6BAQBJUFACAKhhw7SQwhJ7u+POwkBApkk8+admZzP8+QhM/OWwyhz5rZzxRiDUkopdbFibgeglFLKP2mCUEoplSNNEEoppXKkCUIppVSONEEopZTKUXG3A/CVatWqmYYNG7odhlJKBZT4+PjDxpjqOb0WNAmiYcOGxMXFuR2GUkoFFBHZebnXtItJKaVUjjRBKKWUypEmCKWUUjkKmjGInJw7d46kpCRSUlLcDiVolC5dmrCwMEqUKOF2KEophwV1gkhKSqJChQo0bNgQEXE7nIBnjOHIkSMkJSXRqFEjt8NRSjksqLuYUlJSqFq1qiYHHxERqlatqi0ypYqIoE4QgCYHH9P3U6miI+gThFIqyBw/Dh98AOnpbkcS9DRBOOzYsWO89957eT6vf//+HDt2zPcBKRXo/vtfGD0aPvvM7UiCniYIh10uQaSlpV3xvFmzZlG5cmWHolIqgMXE2D+feQbOnHE3liCnCcJh48aNY+vWrbRt25b27dvTpUsXBg4cSPPmzQEYNGgQERERtGjRggkTJmSd17BhQw4fPsyOHTto1qwZ999/Py1atKB3796c0X8Uqqjatg1+/hluvRX27LGtCeWYoJ7mmt3jcx5nzf41Pr1m21pteavvW1c85uWXX2b9+vWsWbOGxYsXc/PNN7N+/fqsaaIff/wxVapU4cyZM7Rv357bb7+dqlWrXnCNLVu2MGXKFD744APuuOMOYmJiuPvuu336d1EqIGS2Ht56C1JS4F//gvvvB21tO0JbEIWsQ4cOF6whePvtt2nTpg0dO3Zk9+7dbNmy5ZJzGjVqRNu2bQGIiIhgx44dhRStUn4mJgYiIqBhQ5scjh6FV15xO6qgVWRaELl90y8s5cqVy/p98eLFLFiwgBUrVlC2bFm6deuW4xqDUqVKZf0eEhKiXUyqaNq9G3780SYGgPBwuPNO+M9/YOxYqFPH3fiCkLYgHFahQgVOnjyZ42vHjx8nNDSUsmXL8uuvv7Jy5cpCjk6pAPLVV/bP228//9z//R+cOwf//Kc7MQU5TRAOq1q1Kp07d6Zly5Y8+eSTF7zWt29f0tLSaNasGePGjaNjx44uRalUAIiOhtatoWnT8881bgwPPggffQSbNrkXW5ASY4zbMfhEZGSkuXjDoI0bN9KsWTOXIgpe+r6qQrdvH9StC889Z6e3ZnfggE0U/fpBVJQr4QUyEYk3xkTm9Jq2IJRS/m/6dDAGhgy59LWaNeFPf7ItjFWrCj+2IKYJQinl/2Ji4NprwbN+6BJ/+hNUqwbjxtlEonxCE4RSyr8dOgSLF+fceshUsSI8/TQsWgTz5xdaaMFOE4RSyr99/TVkZFw4eyknDz5o10eMG2ePVwWmCUIp5d9iYuwgdJs2Vz6uVCl4/nlbimPatMKJLcg5miBEpK+IbBKRRBEZl8PrT4hIgoisE5GFItIg22tzROSYiHzrZIxKKT929CgsWGBbD97sRTJiBLRqZbubUlOdjy/IOZYgRCQEeBfoBzQH7hSRi0eYfgYijTGtgWjg1WyvvQbc41R8/qp8+fIA7N27lyGX6XPt1q0bF0/pvdhbb71FcnJy1mMtH64C0jffQFpa7t1LmUJC7ErrrVvhww+dja0IcLIF0QFINMZsM8akAlOBW7MfYIyJNcZkfoqtBMKyvbYQyHkJchFQp04doqOj833+xQlCy4ergBQdDfXqQfv23p/Tvz906WJXV5865VxsRYCTCaIusDvb4yTPc5fze2B2Xm4gIqNFJE5E4g4dOpSPEJ03btw43n333azHzz33HC+88AI9e/akXbt2tGrViq+//vqS83bs2EHLli0BOHPmDMOHD6dZs2bcdtttF9Rieuihh4iMjKRFixY8++yzgC0AuHfvXrp370737t2B8+XDAd544w1atmxJy5Yteeutt7Lup2XFlV85cQLmzfO+eymTiC3gd/AgvPmmc/EVAX5RrE9E7gYigRvzcp4xZgIwAexK6ise/PjjsGZN/gK8nLZtbdnhKxg2bBiPP/44jzzyCADTpk1j7ty5PProo1SsWJHDhw/TsWNHBg4ceNn9nt9//33Kli3Lxo0bWbduHe3atct67cUXX6RKlSqkp6fTs2dP1q1bx6OPPsobb7xBbGws1apVu+Ba8fHxfPLJJ/z4448YY7juuuu48cYbCQ0N1bLiyr989x2cPXvl6a2X06mT3TPitdfs7Kbq1X0fXxHgZAtiD1Av2+Mwz3MXEJFewN+BgcaYsw7G44rw8HAOHjzI3r17Wbt2LaGhodSqVYu//e1vtG7dml69erFnzx4OHDhw2Wt8//33WR/UrVu3pnXr1lmvTZs2jXbt2hEeHs6GDRtISEi4YjzLli3jtttuo1y5cpQvX57BgwezdOlSQMuKKz8TEwO1a9sP+/x46SU4fdr+qfLFyRbEKqCpiDTCJobhwIjsB4hIODAe6GuMOehgLLl+03fS0KFDiY6OZv/+/QwbNoxJkyZx6NAh4uPjKVGiBA0bNsyxzHdutm/fzuuvv86qVasIDQ1l5MiR+bpOJi0rrvzG6dMwaxbcdx8Uy+f32ObN4d574b33bA9Cgwa5nhKQzpyBEiWguO8/zh1rQRhj0oAxwFxgIzDNGLNBRJ4XkYGew14DygNRIrJGRGZmni8iS4EooKeIJIlIH6diddqwYcOYOnUq0dHRDB06lOPHj1OjRg1KlChBbGwsO3fuvOL5Xbt2ZfLkyQCsX7+edevWAXDixAnKlStHpUqVOHDgALNnnx/CuVyZ8S5dujBjxgySk5M5ffo006dPp0uXLj782yrlA3Pm2A8+b2cvXc4//2nHJC4u8BcsjIFRo2DAAEcWBzo6BmGMmQXMuui5Z7L93usK5wbNp1aLFi04efIkdevWpXbt2tx1113ccssttGrVisjISK699tornv/QQw8xatQomjVrRrNmzYiIiACgTZs2hIeHc+2111KvXj06d+6cdc7o0aPp27cvderUITY2Nuv5du3aMXLkSDp06ADAH/7wB8LDw7U7SfmX6Gg7blDQLy/16tnNhP79b/jzn+0aiWDy8svw5Zd2UD6/La0r0HLfKs/0fS1ka9bYMtbDhwffB1xOUlJscrjzTpgwoeDX++03uOoqm2y++abg1/MX334LAwfa/y8mTcrbTK9stNy3UoHm2DHbdx4RYbfWfOklW7G0KJg/365fKGj3UqYqVeCpp+wH6rJlvrmm2zZutKvGw8PtgsB8JofcaIJQyl9kZEBsLNx9t52988gjkJ4Ob78NTz5pPzh//dXtKJ0XHQ2hodCjh++u+dhj9j196qnALwd+9KidwlumDMyYAWXLOnaroE8QwdKF5i/0/XTAnj3w4ot2K80ePew33VGjIC7OFp4bO9b2n5csCdkWXQal1FSYOdN2nZQo4bvrli0Lzz4Ly5cHdjdTerrtetuxw04Drlcv11MKIqgTROnSpTly5Ih+qPmIMYYjR45QunRpt0MJfOfO2V3SBgyA+vVtcbl69eDzz2Hv3vPdS5ldBzVqwLBh8OmndoVxsFq0yHav5WdxXG7uu88m4b/9zX7QBqK//hXmzrVfFG64wfHb+cVKaqeEhYWRlJSEv5bhCESlS5cmLCws9wNVzn79FT76CCZOtKUgMrs97rsPmjS58rljxtgEMnGi/T0YxcRAhQpw002+v3aJEraldscd9n0cOdL393DSF1/YleGPPAL3318otwzqWUxK+YVTp+z+BB99ZLs4ihe3LYff/x769s3bAqfrroPjx+0gpUMDk65JS4NataBPHzsrxwnGQIcOcOAAbN4MgdIajouzLYaOHe1YlA+733QWk1KFzRhYudJ+06td2yaDI0fg1VchKel891JeV7+OGQObNsHChc7E7abvv7fvka9mL+VExK4d2L3bduMFgv37YdAgmzyjonw7NpMLTRBK+dKhQ/DGG9Cypa0hNHmy7U9futR+63/ySahZM//Xv+MOu0bgnXd8F7O/iI62g8l9+zp7n549bRfWiy/a1pg/O3sWBg+2M5e+/rrQiw5qglCqoNLTYfZsmwjq1rXrFSpUsIu89u2DTz6x3QO+6BIqVQpGj7YzcYJp9Xt6um1V9e/v6LTNLC+/bBfQvfpq7se6xRg73rBihZ2ckNuWqw7QBKFUfm3fDv/4BzRsaD/YliyxXUDr15/vXqpY0ff3ffBBW1YhULpIvLF8ue1KcbJ7Kbt27eyssDfftEncH737rh23evppGDrUlRA0QSiVFykpttuoZ09bvuHFF213UlSUXc/wxhvQooWzMYSFwW232RW02XYNDGgxMbZ1dPPNhXfPF16w042ff77w7umt2FhbgXbgQFtw0CWaIJTyxpo1dsFanTpw112wbZv9YNm583z3UsmShRfPmDG2X3rq1MK7p1MyMmyC6NPHds0VliZNbCvvgw9gy5bCu29utm+3LYZrrrHTcR0owuctTRBKXU5ysh1HyKyHNGGC/RCbPx+2brXdSw6vZL2srl1t4b533gn80hGrVtmZXU4sjsvNM8/YlsvTTxf+vXNy6pQto5GebgelneiizANNEEpdbP9+++Ffvz488ICdn//227avesoU6NXL1W91gB3wHjPGtmyWL3c3loKKibFTN2+5pfDvXasW/PGPdp1KfHzh3z+7jAy7eG/DBlvCO7eFk4VAE4RSmdatszWQGjSwYwtduti5+ZndS1WquB3hhe66CypXDuwpr8bY6a29etm/ixuefBKqVoVx49y5f6YXXrDJ8rXXoHdvd2Px0AShiraMDLu1Za9edhrhtGl2GunmzXbaZZcu/rtiuVw5W6IjJsbWbwpEa9bYPvfCmr2Uk0qV4O9/hwUL7I8bZsywxQR/9zvbovETmiBU0XTmjB1TaNHCzpzZuPH8Ctt33vGL5r1XHn7Y9lf7YmMdN0RHQ0iI7Xd300MP2S7FceMc2brzitavh3vugfbtYfx4v/pCoglCFS3799uByczxhTJlbBG07dtt0Tx/60bKTePGdg3G+PG2VHYgyexe6tYNqlVzN5bSpe2stPh4G1NhOXLEJsfy5W2L1c9qQ2mCUEXDL7/Y7pgGDWxf7/XXw+LF9gPhrrsKd4qqr40ZYxNfTIzbkeTNhg22K8+N2Us5uftuu6bl73+36yOclpZmF+tl1uaqW9f5e+aRJggVvIyBOXPsgF/r1nbNwB/+YEtuf/013HijXzXn8613b7vPQaANVsfE2Pd/0CC3I7FCQuzWromJdgWz05580hZd/N//bJVWP+RoghCRviKySUQSReSSKQIi8oSIJIjIOhFZKCINsr12r4hs8fzc62ScKsikpNhVxi1bQr9+to/3pZfs+MK778LVV7sdoW8VK3a+Zo/bUzXzIjra1qiqVcvtSM4bMAA6d7arl0+fdu4+n34Kb71lt0IdNcq5+xSQY/tBiEgIsBm4CUgCVgF3GmMSsh3THfjRGJMsIg8B3Ywxw0SkChAHRAIGiAcijDFHL3c/3Q9CcfCgrU/03nu2qmrbtvDEE7YZH6BdSOkZ6azet5pF2xdhMNQsV5Ma5WpQo1wNapa3v5cuXtpWJa1b167A/eQTt8PO3ebNdqXwf/4Djz7qdjQX+uEHm7hefNHuPudrK1fa1muXLraFm9eS7z52pf0gnIysA5BojNnmCWIqcCuQlSCMMbHZjl8J3O35vQ8w3xjzm+fc+UBfYIqD8apAtWGD/Tb2+ee2PPKAATYxdOsWkF1Ie07sYd7WeczdOpcF2xZw5MyRKx5foWQFapavySvtyzNg0kTG9cqgfJ0G5xNJtqQSWiaUYuIHPcuZ4yWDB7sbR046d7aL9l55xU5kqFrVd9feu9f+ncPC7GI4l5NDbpyMri6wO9vjJOC6Kxz/e2D2Fc69ZARHREYDowHq169fkFhVoDHGlrx48037LaxMGdtUf/xx+800gJw5d4bvd37P3K1zmbd1HhsObQCgdvnaDLh6AH0a96HXVb0oW6IsB08f5ODpgxw4fSDr98zH3960jcGLDxA6OYZnOyRjuLR3oHix4lQvW/3CVkjZC1sk2X9KF3doVk10tO1399fta196yY5bvfQS/PvfvrlmSootsnjiBMyb59vE4xC/SF8icje2O+nGvJxnjJkATADbxeRAaMrfZFZTffNNO7ZQq5adlfTAA+5PlfSSMYb1B9dntRK+3/k9Z9PPUiqkFF0adGFk25H0adyHljVaIhe1gBqVbESj0EaXv/jCnvxjfSJ/m36EI+eO2+Rx6tJkkvn7lt+2cODUAc6kncnxchVLVbygFVKrfC1uqH8DvRv3plrZfL7f27fD6tV2xbC/atnSLlr773/tOEFBv4AaY/8f/ekn+Oore/0A4GSC2ANkr2QW5nnuAiLSC/g7cKMx5my2c7tddO5iR6JUgeHQIXj/fTvIfPCg/Xb36acwfLgttubnDicfZv7W+czbNo95W+ex96Rd+dy8enMebv8wvRv3pmuDrpQtUcDNcsaMgcGDCZk1mxqDBlGjXA1a1sj9w+h06ulLkkdWYkm2jzcf2czC7Qt5P+59BKF93fb0a9KPvk360r5Oe0KKhXgXY2b3kpurp73xz3/a2lvPPlvwcZ233oKJE+G552wrIkA4OUhdHDtI3RP7gb8KGGGM2ZDtmHAgGuhrjNmS7fkq2IHpdp6nVmMHqX+73P10kDpIbdxoWwuff25bD/372/GFHj38enwhNT2VFbtXZLUSVu9bjcEQWjqUmxrfRJ/GfbjpqpuoV8nH1WDT0uw+FU2bOrJvdXpGOvH74pmTOIfZibP5ac9PZJgMqpSpQu/GvenXpB99GvehZvkrbKvaqZNd1BcIM66eeMIOpK9bl/99PubPt9uoDhpk9w1xu9DjRa40SO1YgvDcuD/wFhACfGyMeVFEngfijDEzRWQB0ArI3NJplzFmoOfc+4DMKQQvGmOumMI1QQSRzPGFt96yey2ULm2b+48/Ds2auR3dZSX+lsjcxLnM2zaPRdsXcSr1FCESQqd6neh9VW/6NOlDRO0I779p59fLL8Nf/2oH75s3d/RWR5KPMH/bfOYkzmFO4hwOnD4AQLva7ejbuC/9mvajY1hHihfzdFYkJdkS6S+9ZGP0d4cP29Xq3brZtTN5lZgIHTrYGWYrVtgV037GtQRRmDRBBIGTJ+Gzz2y/76ZNULOmnd//4IOFvlm7N46nHCd2R2xWUth2dBsAjSo3ok/jPvRu3JsejXpQqXSlwg3s8GE7+Pv739suuUKSYTJYu39tVuti+e7lpJt0KpWqxE2Nb6Jv477cvmAPlZ961v73DZT1KC+8YMu/L1tmZzh568QJ21rav9/ueXHVVc7FWACaIJR/27zZJoVPP7VJokMHW1576FC/Gl/I7F7JTAgrdq8g3aRTvmR5ejTqkdVKaBza+JLB5UI3atT5bVArFXKC8jiecpwF2xZkJYw9J/ew5GOonVaKDz59lH5N+tG5fmdKhvj5GpXTp20romlTW/7dm/+2GRl2rOG77+yMpR49nI8znzRBKP+TkWG7j955B+bOtRvGDBtmE0OHDm5HB9hvxNuPbmfJziVZaxJ+O2OHwSJqR2S1EjrV6+R/H3Lx8RAZ6TcL0YwxbNqwhGta9+CzWxsyul0S5zLOUb5keXo26knfJn3p16QfDSo3yP1ibnj/fVs595tv7Dqb3DzzDPzf/9mNpsaOdT6+AtAEofzHsWN2Rsi779ptO2vXtqWWR4+2XUouOZ16ml8O/sLa/WtZe2At6w6sY92BdZxMPQnYNQm9G/fOWpNQvZz/dXldolMnWy3011/9Y2A080P2l1842bQBsTtis1oXO47tAODaatdmzYzq2qCrc+sw8urcOTueU7q03cMi5ArjSFFRcMcdtjjkhx/69WQK0ASh/EFCgm0tfP65bbJ37my/WQ0ebFsPhcQYw+4Tu7MSwdoDa1m7fy2JvyVmLSyrWKoirWu2pk3NNrSu2ZpOYZ1yXJPg9yZPtpVq58yxe2m7rVcvO0i9ceMFH5rGGDYf2czsxNnMSZzD4h2LOZt+ljLFy9C9UfeshNGkist7dHz5pZ1W/dlndtJETtautZWC27SB2Fi/6iK9HE0Qyh3p6bZJ/s47sGiR/ccyYoSdq9+uXe7nF9CZc2fYcGjDBa2CtQfWcizlWNYxV4VeRZuabexPLftnw8oNAy8Z5CQ11S7wat/e/ndw0+HDdlHjU0/ZGkdXkHwumSU7lmQljC2/2RnwTao0yZoZ1a1ht4KvGcmrjAz7Xh45YgfZL/7wP3zYduulpdlB6dq1Cze+fNIEoQrXkSO2XPJ778HOnXZa40MPwf33O7La2RjDvlP7LmkVbDqyiQxjdwcrV6IcrWq2ymoVtKnZhlY1W1GxVEWfx+NXnn3W9oUnJro7i+ajj2yp9dWrITw8T6du/W1rVldU7I5Yks8lU6Z4Ge5ocQcPRDxAx7COhZfQ58+35dXffNNOu8507px9fsUKWLrUJpIAoQlCFY61a21rYdIku6jtxhttN9Ktt/qsKFlqeioJhxIuaRUcTj6cdUz9SvUvaRU0rtLYP4rUFba9e+0mSY89Bq+/7l4c/fvbsZCtWwvUJ5+SlsKyXcuITohm0i+TOJV6ilY1WjE6YjR3t76byqUr+y7my+nVy/6/vnUrVPR8wRgzxo6rff653XgogGiCUM45d85uuP7OO/abU5kydn/dRx6x5TAK4ODpg5e0CjYe3khaRhoApYuXpmWNlhe0ClrXbE1omVAf/MWCyPDhdqZYUhKUK1f49z92DGrUsN+4X33VZ5c9lXqKKb9MYXz8eOL3xVOmeBmGtRzGAxEPcF3d65xrVcTF2RbCP/5htyn94AM7yeLPf/bv+lKXoQlCeW3e1nm88sMrnE07S4bJIMNkYDBZv2f+VD6RyuBlRxi29Ci1jqWxu2pxvrihEl9eV45jZeSK52a9Zq7w2kWVSOtUqHNJq6Bp1abnV+iqy1u2zO49MGGC7eYrbJ9/bgd1V66E665U0Dn/4vfGMyF+ApPXT+ZU6ila12zN6Ha2VeHIQsU77oBZs2x9peHDoXt3+/hKs5v8lCYI5ZVZW2Zx25e3UadCHRqH2i6ZzB8RoZgU4+ptJxgwbzs3LN9DibQM1rWqydx+TVkTURcpXvyCc4pJMQS55Lns18vxNc85lUtXti2DWm3yXzlU2dIl4eF2kHXt2sKfdnnrrfDzz3Y8yuF7nzx7kinrbati9b7VlC1RluEthjM6YjQd6nbwXati82Y77TU9HZo0sVVaQwOz5XqlBIExJih+IiIijMq/2Vtmm5L/V9JEjI8wvyX/duGLZ88aM2mSMR07GgPGlCtnzMMPG5OQ4E6wKu8+/ND+t1uypHDve+KEMaVKGfPYY4V7X2PMqj2rzP0z7zflXixneA7T5v025r2f3jPHU4775gaPPmpM5crGbNjgm+vlU0ZGhjl59mS+z8fWxsvxc1VbEIq5iXO5deqttKjRggX3LDjfh79vH4wfb3/277fflMaMgZEjXSvfoPIpOdnWZ+rVC6ZNK7z7Tp0Kd95pS1R06VJ4983mxNkTTP5lMuPjx7Nm/xrKlijLnS3v5IGIB4isE5n/VoUx9n0t5HGdlLQUVu9bzYrdK1iRZH+uqXoNi+5dlK/raReTuqzM5NC8enMW/G4BVUqH2r7id96xK0LT0qBfPzsbqU8f/1iRq/LnL3+BN96AHTsKbye3oUPtGEhSkuv988YY4vbGMT5+PFPWTyH5XDJta7XlgYgHGNFqhN9OeU46kXRBMli9bzWp6amAXcfTKawTPRr14L7w+/J1fU0QKkfzts5j4JSBNKvejIW/W0iVc8XtrldTp9rpe6NG2dlITZu6Haryhe3bbdG5v//dro1wWnKyrcJ77712TYwfOXH2BJPWTWJ8/HjWHlhLuRLlbKsi0rYq3HI27Sw/7//5goSQdCIJsLP22tdpT6ewTnSq14lOYZ2uvO+GlzRBqEss2LaAW6bcwjVVr2Hh7xZSdUuS/ba3datdXPXEE35Zu14V0K232hbirl3Ol4H46iu7a9zChX5bzdQYw6q9qxgfN56pG6aSfC6ZdrXbMbrdaEa0GkGFUhUcvf/ek3svSAbxe+M5m2431mxQqQGd6nXi+rDr6VSvE21qtqFEiO/L0miCUBfITA5XV72ahfcsoNrkGbbiZ2iobT107ep2iMopmSuBC2NB11132fUX+/f7bKGkk46nHGfSL7ZVse7AOsqXLM+IliMYHTGaiDoRBb5+anoqa/evZfnu5VkJYdfxXQCUCilFZJ3IC1oHtSsUTqkOTRAqy8JtCxkwZYBNDoO/ptqf/gFffGEHLydNsguaVPDKyLDTMytXti0Jp5w9a7uXhg2zC8kCiDGGH/f8yIT4CUxdP5UzaWeIqB3BAxEPcGerOylf0ruW9f5T+y9oHcTtjSMlLQWAehXrZSWCTmGdCK8d7lrJeE0QCoBF2xcxYPIAmlRpwpL27xF6z/226Nhzz9l+6QBc5KPy4b//tZMOfvrJuZpB334Lt9xi9/zo29eZexSCYynH+GLdF4yPH8/6g+spX7I8d7W6iwciHiC89vmaUufSz7HuwLqsZLB89/KsEuYlQ0rSrna7rK6iTmGdqFuxrkt/o0tpglDEbo/l5sk307hKY34oNpqKf3wKKlSwJaF79nQ7PFWYTpyweyQPHmxLVzth5Ei7h/OBA1DSzzZTygdjDCuTVjI+fjxfbviSlLQU2tdpzw31byB+Xzyr9qziTNoZAOpWqHtB66Bd7XaUKu6/Zb81QRRxi3cspv+k/jQv15Cl8W0o88VUuwn75MkBU5JY+djYsbb0xu7dvu9WTE21mz8NHOhcAnLR0TNHs1oVm49sJrx2eFYyuL7e9dSrVM/tEPNEE0QRtmTHEvpP7k/3lNrMiClB8Y2b4Omn7Uwl7VIqun79FZo1s3sz/O1vvr323Lm2W+nrr22SCFLGU0sspFhg/zu6UoJwdNWTiPQVkU0ikigi43J4vauIrBaRNBEZctFrr4jIes/PMCfjDFbf7/ye/pP788jmUGa+sY/iBw/b3cWef16TQ1F37bV2YsL779vFkL4UE2OnSPfu7dvr+hkRCfjkkBvHEoSIhADvAv2A5sCdItL8osN2ASOByRedezPQDmgLXAf8WUT8c5mjn1q6cymDP+3HJ7NK8upneygW3s7upRvk/2hVHowda1c4f/21766ZlgbTp8OAAXb/ZhXQnGxBdAASjTHbjDGpwFTg1uwHGGN2GGPWARkXndsc+N4Yk2aMOQ2sAwJ3KkQhW7ZrGWP+04elH6Rzxw/H7DaPsbF2YFKpTDffDA0b2llNvrJ0qd16c8iQ3I9Vfs/JBFEX2J3tcZLnOW+sBfqKSFkRqQZ0By4Z+RGR0SISJyJxhw4dKnDAweCHXT8w4ale/PD+Wa5JLmunG778ckAsVFKFLCQEHn4YFi+GX37xzTWjo+2mUQE8tVWd55eV14wx84BZwHJgCrACSM/huAnGmEhjTGT16tULOUr/s3xLLBvu6M7EqWcp2TqcYmvW2G+JSl3OfffZrqB33y34tTIybPdS//7u7FynfM6rBCEiX4nIzSJ52tR3Dxd+6w/zPOcVY8yLxpi2xpibAAE25+HeRU78sijKdOvF6B/PcWrsA5RctgLq13c7LOXvqla1JTE+/xyOHi3YtVassCXib7/dN7Ep13n7gf8eMALYIiIvi8g1XpyzCmgqIo1EpCQwHJjpzc1EJEREqnp+bw20BuZ5GWuRs+mDl2l80x1cdRR+m/Ix5d/+H5TwfVEvFaQeecRWXv3004JdJzraFgDUVmvQ8CpBGGMWGGPuws4s2gEsEJHlIjJKRHL8JDLGpAFjgLnARmCaMWaDiDwvIgMBRKS9iCQBQ4HxIrLBc3oJYKmIJAATgLs911PZpaay7w/DuGb0X9lZoxRnVi6jyvBRbkelAk14OHTubLuZMi6eL+IlY+z01t69bal4FRS8Hrn0fKO/G7gH+BmYBNwA3At0y+kcY8ws7FhC9ueeyfb7KmzX08XnpWBnMqnL2bmTU4P6U3tNAp91rUTP6HhqVW/sdlQqUI0dC8OH23Uy/fvn/fxVq+yq7MLYZ0IVGm/HIKYDS4GywC3GmIHGmC+NMWMB3TSgsH3zDWltWpO+MYExo2rS85v1hGlyUAUxeLAtu5LfKa8xMXamXBCvnC6KvB2DeNsY09wY8y9jzL7sL1xuibZywLlz8OSTMHAgG8qdYtBf6vHUm6sIq1hI20eq4FWiBDz4oK2+umVL3s41xo4/9Oxp9xRRQcPbBNFcRCpnPhCRUBF52JmQVI5277YF9l5/nQ+vK8mwP9bjsz8tC7jCYMqPjR5tE0Vetwdduxa2bdPFcUHI2wRxvzHmWOYDY8xR4H5HIlKXmj0bwsNJX7uG+4aX4aU76zLv/u+pX0mnsSofqlXLfsh/8gmcOuX9edHRUKyY3c5UBRVvE0SIiEjmA0+dpcAv8u7v0tJspc3+/UmuEUrHB0OI7ViT2HtjNTkoZ4wdC8eP210GvZHZvdStm91BTgUVbxPEHOBLEekpIj2xq5vnOBeWYu9e26f7r39xeMQgmow4zKGwKsTeG0uDyg3cjk4Fq44doV07O1jtzVYACQl2V0JdHBeUvE0QTwGxwEOen4XAX5wKqshbsADatoW4OHa8/TxXt15CifIVWTxyMQ0rN3Q7OhXMRGwrYsMGW6MpNzEx9pzbbnM8NFX4vF0ol2GMed8YM8TzM94Yc0ltJFVA6el2f+jevaF6dTbOmki75DepUKoCi+/V5KAKybBhtgSHN1NeY2LsIjvdmTAoebsOoqmIRItIgohsy/xxOrgiZd8+6NMH/vlP+N3vWPfNh9zw42jKlyxP7L2xNApt5HaEqqgoUwbuvx9mzIBduy5/3JYtsG6dzl4KYt52MX0CvA+kYUtvTwS8HMVSuZoxA1q1guXL4eOPWfvyH+kePYCyJcqyeORirgq9yu0IVVHz4IP2z//97/LHxMTYPwcPdj4e5QpvE0QZY8xC7B7WO40xzwFakaugTp2y39Ruuw0aNIDVq1nXP4KeE3va5HCvJgflkgYN7KroDz6AlJScj4mOhg4doJ6uxQlW3iaIs55S31tEZIyI3IaW2CiYn36yRdI++gjGjYMVK/gl9Bw9J/akdPHSxN4bS+MqWj5DuWjsWLs73JdfXvrajh0QH6/dS0HO2wTxGLYO06NABLZo371OBRXU0tJsQbPrr4fUVLsV6L/+RVLKQXpM7EHJkJLE3htLkypN3I5UFXXdu0Pz5vDOO5dOec3sXtLprUEt1wThWRQ3zBhzyhiTZIwZZYy53RizshDiCy7bt9sFRc88Y2eKrF0LN94IwMS1EzmcfJh5d8+jadWm7sapFNjpq2PG2JbCjz9e+FpMjG0BX6VdoMEs1wThmc56QyHEEryMgYkToU0bu/fvF1/ApElQuXLWIVEJUXQM60iLGi3ci1Opi91zj93fIfuU16Qku3ucth6CnrddTD+LyEwRuUdEBmf+OBpZsPjtN1tn/9577eK3devsFo/ZJP6WyJr9axjafKg7MSp1OeXLw8iRMG0aHDhgn5s+3f6p4w9Bz9sEURo4AvQAbvH8DHAqqKCxaBG0bg1ffQUvvWTHGxpcWiYjakMUAEOa6z845YceecSWmp8wwT6OjoYWLeAab3YeVoHMqx3ljDG6j2VenD0LTz8N//43NG1qm+ORl982IyohiuvqXqcF+JR/uvpqu4jzf/+D++6DpUvtOJoKel4lCBH5BLikcpcx5j6fRxToEhJsF9KaNXax0euvQ7lylz18629b+Xn/z7x+0+uFF6NSeTV2LAwYYLubjNHxhyLC2z2pv832e2ngNmCv78MJYMbYTd+ffBIqVICZM+GWW3I9LSpBu5dUAOjb185YWrDAtopbtnQ7IlUIvO1iisn+WESmAMsciSgQ7d8Po0bZDd/79YOPP7abr3ghKiGKDnU7aAlv5d9CQuxYxJ/+ZAenz28Po4KYt4PUF2sK1MjtIBHpKyKbRCRRRMbl8HpXEVktImkiMuSi114VkQ0islFE3s6+YZFfmTnT1lFavNhOBfzuO6+Tw7aj21i9b7XOXlKB4fe/h9/9Dh54wO1IVCHxdgziJBeOQezH7hFxpXNCgHeBm4AkYJWIzDTGJGQ7bBcwEvjzRedeD3QGWnueWgbcCCz2Jt5Ccfo0PPGEndnRtq1d19C8eZ4ukTl76fZm2p+rAkClSvDZZ25HoQqRt11MFfJx7Q5AojFmG4CITAVuBbIShDFmh+e1jItviR3rKAkIUAI4kI8YnBEXZweit2yBv/wFnn8eSpXK82WiEqKIrBOppbyVUn7J2/0gbhORStkeVxaRQbmcVhfYne1xkue5XBljVmB3sNvn+ZlrjNnozbmOSk+HF1+ETp0gORkWLoRXXslXcth2dBvx++K1e0kp5be8HYN41hhzPPOBMeYY8KwjEQEi0gRoBoRhk0oPEemSw3GjRSROROIOHTrkVDjWjh22jtLTT9spfuvW2WJm+RSdEA2gCUIp5be8TRA5HZdb99QeIHuh+DDPc964DVjpKRB4CpgNdLr4IGPMBGNMpDEmsnr16l5eOo+MsbWT2rSxxfUmToQpUyA0tECXjUqIIqJ2hHYvKaX8lrcJIk5E3hCRxp6fN4D4XM5ZBTQVkUYiUhIYDsz08n67gBtFpLiIlMAOUBd+F9OxYzBihC1Y1qqVTRD33FPgKX7bj24nbm+cth6UUn7N2wQxFkgFvgSmAinAI1c6wRiTBowB5mI/3KcZYzaIyPMiMhBARNqLSBIwFBgvIhs8p0cDW4FfgLXAWmPMN3n6mxXU4sW2jlJUlN2/YfFiaOSbb/sxG+2ykqEtNEEopfyXmIs3AglQkZGRJi4uruAXSk2Ff/wDXnsNmjSx3UsdOhT8utlc9+F1pGWkET86t0aYUko5S0TijTE5FovzdhbTfBGpnO1xqIjM9VF8/mPjRujYEV59Ff7wB1i92ufJYeexnfy05yftXlJK+T1vu5iqeWYuAWCMOYoXK6kDhjHw3nsQEQG7dtl69xMm2Fr4Pqazl5RSgcLbBJEhIlm1qEWkITlUdw1Ihw7ZonqPPAJdu9od3wYNcux2UQlRhNcKp3GVxo7dQymlfMHbaq5/B5aJyBLsyuYuwGjHoipMxYrZrqX//Mfuv1ssv+Wpcrfr+C5+3PMjL/V4ybF7KKWUr3hbamOOiERik8LPwAzgjINxFZ6qVW2CKFnS8VtldS/p7CWlVADwtljfH4DHsIvd1gAdgRXYLUgDXyEkB7DdS21rtaVJlSaFcj+llCoIb/tTHgPaAzuNMd2BcOCYU0EFo93Hd7MyaaUOTiulAoa3CSLFGJMCICKljDG/ArpjeR7o7CWlVKDxdpA6ybMOYgYwX0SOAjudCioYRSVE0aZmG5pWbep2KEop5RVvB6lv8/z6nIjEApWAOY5FFWR2H9/NiqQVvND9BbdDUUopr3nbgshijFniRCDBLLP20pDmQ3I5Uiml/Idzk/5VlqiEKFrVaMU11XTYRikVODRBOCzpRBLLdy/XwWmlVMDRBOGwmAQt7a2UCkyaIBwWvTGaljVacm21a90ORSml8kQThIP2ntzLD7t+0O4lpVRA0gThoJiEGAxGE4RSKiBpgnBQVEIULaq3oFn1Zm6HopRSeaYJwiH7Tu5j2a5l2npQSgUsTRAOidno6V7S2UtKqQClCcIhUQlRNK/enObVm7sdilJK5YsmCAfsO7mPpTuXaveSUiqgOZogRKSviGwSkUQRGZfD611FZLWIpInIkGzPdxeRNdl+UkRkkJOx+tJXG7/S2UtKqYCX52J93hKREOBd4CYgCVglIjONMQnZDtsFjAT+nP1cY0ws0NZznSpAIjDPqVh9LSohimbVmtGiRgu3Q1FKqXxzsgXRAUg0xmwzxqQCU4Fbsx9gjNlhjFkHZFzhOkOA2caYZOdC9Z39p/bz/c7vtfWglAp4TiaIusDubI+TPM/l1XBgSk4viMhoEYkTkbhDhw7l49K+l9W9pLOXlFIBzq8HqUWkNtAKmJvT68aYCcaYSGNMZPXq1Qs3uMuISoji2mrX0qK6di8ppQKbkwliD1Av2+Mwz3N5cQcw3RhzzmdROejAqQN8v/N7hjQbgoi4HY5SShWIkwliFdBURBqJSElsV9HMPF7jTi7TveSPvtr4FRkmQ7uXlFJBwbEEYYxJA8Zgu4c2AtOMMRtE5HkRGQggIu1FJAkYCowXkQ2Z54tIQ2wLJGC2OI1KiOLqqlfTqkYrt0NRSqkCc2yaK4AxZhYw66Lnnsn2+yps11NO5+4gf4Parjh4+iBLdi7hrzf8VbuXlFJBwa8HqQPJ9I3TbfeSTm9VSgUJTRA+EpUQRdMqTWlds7XboSillE9ogvCBQ6cPEbsjlqHNh2r3klIqaGiC8IHpv07X2UtKqaCjCcIHohKiaFKlCW1qtnE7FKWU8hlNEAV0OPkwsdu1e0kpFXw0QRTQ9I3TSTfpOntJKRV0NEEUUFRCFI1DG9O2Vlu3Q1FKKZ/SBFEAh5MPs2j7Iu1eUkoFJU0QBTDj1xm2e0lnLymlgpAmiAKISojiqtCrCK8V7nYoSinlc5og8ulI8hEWbluo3UtKqaClCSKfsrqXdPaSUipIaYLIp6iEKBpVbkS72u3cDkUppRyhCSIffjvzGwu3L2RIc905TikVvDRB5MOMX2eQlpGm3UtKqaCmCSIfohKiaFi5IZF1It0ORSmlHKMJIo+OnjnKgm0LGNJMu5eUUsFNE0Qefb3pa9u9pIvjlFJBThNEHkUlRNGgUgPa12nvdihKKeUoTRB5cCzlGPO3ztfZS0qpIsHRBCEifUVkk4gkisi4HF7vKiKrRSRNRIZc9Fp9EZknIhtFJEFEGjoZqze+/vVrzmWc09lLSqkiwbEEISIhwLtAP6A5cKeINL/osF3ASGByDpeYCLxmjGkGdAAOOhWrt6ISoqhfqT4d6nZwOxSllHKcky2IDkCiMWabMSYVmArcmv0AY8wOY8w6ICP7855EUtwYM99z3CljTLKDsebqWMox5m2dp7OXlFJFhpMJoi6wO9vjJM9z3rgaOCYiX4nIzyLymqdFcgERGS0icSISd+jQIR+EfHkzN8203Us6e0kpVUT46yB1caAL8GegPXAVtivqAsaYCcaYSGNMZPXq1R0NKCohinoV63Fd3escvY9SSvkLJxPEHqBetsdhnue8kQSs8XRPpQEzANeq4h1POW67l3T2klKqCHEyQawCmopIIxEpCQwHZubh3Moiktks6AEkOBCjV2ZumklqeqrOXlJKFSmOJQjPN/8xwFxgIzDNGLNBRJ4XkYEAItJeRJKAocB4EdngOTcd2720UER+AQT4wKlYcxOVEEVYxTCuC9PuJaVU0VHcyYsbY2YBsy567plsv6/Cdj3ldO58oLWT8XnjeMpx5m6dy8ORD1NM/HXIRimlfE8/8XLxzeZvbPeSzl5SShUxmiByEZUQRd0KdekY1tHtUJRSqlBpgriCE2dPMDdxLrc3u127l5RSRY5+6l3BN5u+4Wz6We1eUkoVSZogriB6YzR1KtTh+nrXux2KUkoVOk0Ql3Hy7Elmb5mt3UtKqSJLP/ku49vN39ruJV0cp5QqojRBXEZUQhS1y9emc/3OboeilFKu0ASRg1Opp5idqN1LSqmiTT/9cvDt5m9JSUvR2UtKqSJNE0QOohKiqFW+Fp3rafeSUqro0gRxkVOpp5i1ZRa3N7udkGKX7FGklFJFhiaIi3y3+TvbvaSzl5RSRZwmiItEJURRs1xNbqh/g9uhKKWUqzRBZHM69bR2LymllIcmiGy+2/IdZ9LO6OwlpZRCE8QFMruXutTv4nYoSinlOk0QHqdTT/Pd5u8Y3Gywdi8ppRSaILLM2jLLdi/p7CWllAI0QWSJSoiietnqdGmg3UtKKQWaIABIPpfMd1ts91LxYsXdDkcppfyCowlCRPqKyCYRSRSRcTm83lVEVotImogMuei1dBFZ4/mZ6WScs7fMJvlcsnYvKaVUNo59XRaREOBd4CYgCVglIjONMQnZDtsFjAT+nMMlzhhj2joVX3ZRCVFUK1uNGxveWBi3U0qpgOBkC6IDkGiM2WaMSQWmArdmP8AYs8MYsw7IcDCOKzpz7gzfbv6Wwddq95JSSmXnZIKoC+zO9jjJ85y3SotInIisFJFBOR0gIqM9x8QdOnQoX0EeSznGwGsGMqLViHydr5RSwcqfvzI3MMbsEZGrgEUi8osxZmv2A4wxE4AJAJGRkSY/N6ldoTaTb59c8GiVUirIONmC2APUy/Y4zPOcV4wxezx/bgMWA+G+DE4ppdSVOZkgVgFNRaSRiJQEhgNezUYSkVARKeX5vRrQGUi48llKKaV8ybEEYYxJA8YAc4GNwDRjzAYReV5EBgKISHsRSQKGAuNFZIPn9GZAnIisBWKBly+a/aSUUsphYky+uu79TmRkpImLi3M7DKWUCigiEm+MiczpNV1JrZRSKkeaIJRSSuVIE4RSSqkcaYJQSimVo6AZpBaRQ8DOAlyiGnDYR+EEOn0vLqTvx4X0/TgvGN6LBsaY6jm9EDQJoqBEJO5yI/lFjb4XF9L340L6fpwX7O+FdjEppZTKkSYIpZRSOdIEcd4EtwPwI/peXEjfjwvp+3FeUL8XOgahlFIqR9qCUEoplSNNEEoppXJU5BOEiPQVkU0ikigi49yOx00iUk9EYkUkQUQ2iMhjbsfkNhEJEZGfReRbt2Nxm4hUFpFoEflVRDaKSCe3Y3KTiPzR8+9kvYhMEZHSbsfka0U6QYhICPAu0A9oDtwpIs3djcpVacCfjDHNgY7AI0X8/QB4DFuuXsF/gDnGmGuBNhTh90VE6gKPApHGmJZACHbPm6BSpBME0AFINMZsM8akAlOBW12OyTXGmH3GmNWe309iPwDyso94UBGRMOBm4EO3Y3GbiFQCugIfARhjUo0xx1wNyn3FgTIiUhwoC+x1OR6fK+oJoi6wO9vjJIrwB2J2ItIQu83rjy6H4qa3gL8AGS7H4Q8aAYeATzxdbh+KSDm3g3KLZ0vk14FdwD7guDFmnrtR+V5RTxAqByJSHogBHjfGnHA7HjeIyADgoDEm3u1Y/ERxoB3wvjEmHDgNFNkxOxEJxfY2NALqAOVE5G53o/K9op4g9gD1sj0O8zxXZIlICWxymGSM+crteFzUGRgoIjuwXY89ROQLd0NyVRKQZIzJbFFGYxNGUdUL2G6MOWSMOQd8BVzvckw+V9QTxCqgqYg0EpGS2EGmmS7H5BoREWwf80ZjzBtux+MmY8xfjTFhxpiG2P8vFhljgu4boreMMfuB3SJyjeepnkBR3id+F9BRRMp6/t30JAgH7Yu7HYCbjDFpIjIGmIudhfCxMWaDy2G5qTNwD/CLiKzxPPc3Y8ws90JSfmQsMMnzZWobMMrleFxjjPlRRKKB1djZfz8ThGU3tNSGUkqpHBX1LiallFKXoQlCKaVUjjRBKKWUypEmCKWUUjnSBKGUUipHmiCU8gMi0k0rxip/owlCKaVUjjRBKJUHInK3iPwkImtEZLxnv4hTIvKmZ2+AhSJS3XNsWxFZKSLrRGS6p34PItJERBaIyFoRWS0ijT2XL59tv4VJnhW6SrlGE4RSXhKRZsAwoLMxpi2QDtwFlAPijDEtgCXAs55TJgJPGWNaA79ke34S8K4xpg22fs8+z/PhwOPYvUmuwq5sV8o1RbrUhlJ51BOIAFZ5vtyXAQ5iy4F/6TnmC+Arz/4JlY0xSzzPfwZEiUgFoK4xZjqAMSYFwHO9n4wxSZ7Ha4CGwDLH/1ZKXYYmCKW8J8Bnxpi/XvCkyD8uOi6/9WvOZvs9Hf33qVymXUxKeW8hMEREagCISBURaYD9dzTEc8wIYJkx5jhwVES6eJ6/B1ji2akvSUQGea5RSkTKFuZfQilv6TcUpbxkjEkQkaeBeSJSDDgHPILdPKeD57WD2HEKgHuB/3kSQPbqp/cA40Xkec81hhbiX0Mpr2k1V6UKSEROGWPKux2HUr6mXUxKKaVypC0IpZRSOdIWhFJKqRxpglBKKZUjTRBKKaVypAlCKaVUjjRBKKWUytH/A8sGwZasUVkaAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "lr=0.01\n",
    "lin1cof = 0.001\n",
    "lin2cof =0.01\n",
    "model = Model(SGD(learning_rate=lr))\n",
    "model.add(Linear('l1', 3072, 100, l2_coef=lin1cof))\n",
    "model.add(ReLU('l1_relu'))\n",
    "model.add(Linear('l2', 100, 10, l2_coef=lin2cof))\n",
    "model.add(SoftmaxCrossentropy('loss'))\n",
    "history = model.fit(X_train, y_train, X_val, y_val, batch_size=1024, epochs=10)\n",
    "plot_history(history)\n",
    "losses, accs, val_losses, val_accs = history\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| lr  |lin1cof|lin2cof|trn acc|val acc|\n",
    "|-----|-------|-------|-------|-------|\n",
    "|1.000|0.01000|0.01000|0.10307|0.10720|\n",
    "|1.000|0.01000|0.00100|0.09963|0.09896|\n",
    "|1.000|0.01000|0.00010|0.10000|0.09972|\n",
    "|1.000|0.01000|0.00001|0.10160|0.10145|\n",
    "|1.000|0.00100|0.01000|0.10166|0.10731|\n",
    "|1.000|0.00100|0.00100|0.10004|0.09961|\n",
    "|1.000|0.00100|0.00010|0.10032|0.09549|\n",
    "|1.000|0.00100|0.00001|0.10010|0.10156|\n",
    "|1.000|0.00010|0.01000|0.11597|0.13379|\n",
    "|1.000|0.00010|0.00100|0.10069|0.10037|\n",
    "|1.000|0.00010|0.00010|0.09965|0.10113|\n",
    "|1.000|0.00010|0.00001|0.10006|0.09776|\n",
    "|1.000|0.00001|0.01000|0.11617|0.11599|\n",
    "|1.000|0.00001|0.00100|0.10024|0.09679|\n",
    "|1.000|0.00001|0.00010|0.09955|0.10547|\n",
    "|1.000|0.00001|0.00001|0.09896|0.10362|\n",
    "|0.100|0.01000|0.01000|0.11214|0.10080|\n",
    "|0.100|0.01000|0.00100|0.10162|0.10341|\n",
    "|0.100|0.01000|0.00010|0.10238|0.10710|\n",
    "|0.100|0.01000|0.00001|0.10425|0.10449|\n",
    "|0.100|0.00100|0.01000|0.10417|0.10851|\n",
    "|0.100|0.00100|0.00100|0.10313|0.11176|\n",
    "|0.100|0.00100|0.00010|0.10720|0.11209|\n",
    "|0.100|0.00100|0.00001|0.10331|0.11024|\n",
    "|0.100|0.00010|0.01000|0.10907|0.10395|\n",
    "|0.100|0.00010|0.00100|0.10089|0.10493|\n",
    "|0.100|0.00010|0.00010|0.10319|0.10634|\n",
    "|0.100|0.00010|0.00001|0.10079|0.10080|\n",
    "|0.100|0.00001|0.01000|0.10500|0.10015|\n",
    "|0.100|0.00001|0.00100|0.10472|0.10297|\n",
    "|0.100|0.00001|0.00010|0.10114|0.09863|\n",
    "|0.100|0.00001|0.00001|0.10478|0.10493|\n",
    "|0.010|0.01000|0.01000|0.18300|0.19770|\n",
    "|0.010|0.01000|0.00100|0.19883|0.19629|\n",
    "|0.010|0.01000|0.00010|0.19269|0.18511|\n",
    "|0.010|0.01000|0.00001|0.19141|0.18511|\n",
    "|0.010|0.00100|0.01000|0.21134|0.22526|\n",
    "|0.010|0.00100|0.00100|0.19930|0.21159|\n",
    "|0.010|0.00100|0.00010|0.20850|0.20497|\n",
    "|0.010|0.00100|0.00001|0.19952|0.19900|\n",
    "|0.010|0.00010|0.01000|0.20174|0.19878|\n",
    "|0.010|0.00010|0.00100|0.19307|0.19119|\n",
    "|0.010|0.00010|0.00010|0.19651|0.18859|\n",
    "|0.010|0.00010|0.00001|0.19881|0.19857|\n",
    "|0.010|0.00001|0.01000|0.19682|0.17003|\n",
    "|0.010|0.00001|0.00100|0.19562|0.19792|\n",
    "|0.010|0.00001|0.00010|0.20296|0.20475|\n",
    "|0.010|0.00001|0.00001|0.21279|0.21322|\n",
    "|0.001|0.01000|0.01000|0.16498|0.17611|\n",
    "|0.001|0.01000|0.00100|0.16526|0.16634|\n",
    "|0.001|0.01000|0.00010|0.17126|0.18023|\n",
    "|0.001|0.01000|0.00001|0.14921|0.13770|\n",
    "|0.001|0.00100|0.01000|0.16817|0.16667|\n",
    "|0.001|0.00100|0.00100|0.16005|0.15679|\n",
    "|0.001|0.00100|0.00010|0.16144|0.15668|\n",
    "|0.001|0.00100|0.00001|0.16870|0.16580|\n",
    "|0.001|0.00010|0.01000|0.16840|0.17491|\n",
    "|0.001|0.00010|0.00100|0.16396|0.17676|\n",
    "|0.001|0.00010|0.00010|0.16919|0.16612|\n",
    "|0.001|0.00010|0.00001|0.17613|0.17535|\n",
    "|0.001|0.00001|0.01000|0.16368|0.15148|\n",
    "|0.001|0.00001|0.00100|0.16876|0.17144|\n",
    "|0.001|0.00001|0.00010|0.16052|0.16243|\n",
    "|0.001|0.00001|0.00001|0.16848|0.16515|"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Q7a.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.7"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}