Ex3_One_vs_All_Logistic_regression.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Logistic-regression-on-the-MNIST-dataset\" data-toc-modified-id=\"Logistic-regression-on-the-MNIST-dataset-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Logistic regression on the MNIST dataset</a></span><ul class=\"toc-item\"><li><span><a href=\"#Dataset\" data-toc-modified-id=\"Dataset-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Dataset</a></span></li><li><span><a href=\"#Displaying-samples-of-the-MNIST-dataset\" data-toc-modified-id=\"Displaying-samples-of-the-MNIST-dataset-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Displaying samples of the MNIST dataset</a></span></li></ul></li><li><span><a href=\"#Multiclass-Classification\" data-toc-modified-id=\"Multiclass-Classification-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Multiclass Classification</a></span><ul class=\"toc-item\"><li><span><a href=\"#One-vs-All-classification\" data-toc-modified-id=\"One-vs-All-classification-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>One vs All classification</a></span></li><li><span><a href=\"#One-vs-all-Prediction\" data-toc-modified-id=\"One-vs-all-Prediction-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>One-vs-all Prediction</a></span></li></ul></li><li><span><a href=\"#Implementation-using-SKlearn\" data-toc-modified-id=\"Implementation-using-SKlearn-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Implementation using SKlearn</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic regression on the MNIST dataset\n",
    "This notebook explores a Python implementation of logistic regression on the MNIST dataset \n",
    "\n",
    "These are Python implementations of exercise 3) from Andrew Ng's course: Machine Learning on coursera.\n",
    "\n",
    "Some similar notebooks and sources: \n",
    "[1][1] \n",
    "[2][2] \n",
    "[3][3]\n",
    "\n",
    "### Libraries\n",
    "\n",
    "[1]: http://nbviewer.jupyter.org/github/JWarmenhoven/Machine-Learning/blob/master/notebooks/Programming%20Exercise%203%20-%20Multi-class%20Classification%20and%20Neural%20Networks.ipynb\n",
    "\n",
    "[2]: https://github.com/kaleko/CourseraML/blob/a815ac95ba3d863b7531926b1edcdb4f5dd0eb6b/ex3/ex3.ipynb\n",
    "\n",
    "[3]: https://github.com/nex3z/machine-learning-exercise/blob/master/coursera-machine-learning-python/ex3/ex3.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import math as m\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import scipy.io as sio\n",
    "import scipy.optimize as opt\n",
    "from sklearn import linear_model\n",
    "%matplotlib inline\n",
    "sns.set(style='white')\n",
    "sns.set_context('notebook')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "The initial dataset is a subset of the MNIST handwritten dataset provided by Andrew Ng's Coursera ML course. It contains 5000 training examples of handwritten digits, where each training example is a 20 pixel by 20 pixel grayscale image of the digit. Each pixel is represented by a floating point number indicating the grayscale intensity at that location. The 20 by 20 grid of pixels is “unrolled” into a 400-dimensional vector. Each of these training examples becomes a single row in our data matrix X. This gives us a 5000 by 400 matrix X where every row is a training example for a handwritten digit image.\n",
    "\n",
    "The second part of the training set is a 5000-dimensional vector y that contains labels for the training set.\n",
    "\n",
    "### Load the data from matlab format into numpy format \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5000, 400)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "(5000, 1)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mat_data = sio.loadmat('/Users/User/Desktop/Computer_Science/stanford_ml/machine-learning-ex3/ex3/ex3data1.mat')\n",
    "sio.whosmat('/Users/User/Desktop/Computer_Science/stanford_ml/machine-learning-ex3/ex3/ex3data1.mat')\n",
    "X = mat_data['X']\n",
    "y =  mat_data['y']\n",
    "#np.place(y,y==10,0) #Replace value 10 with value 0 (due to Octave non zero indexing..)\n",
    "display(X.shape)\n",
    "display(y.shape)\n",
    "m , n = X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Displaying samples of the MNIST dataset\n",
    "We will explore in this section various ways of displaying the MNIST dataset. Note that each row corresponds to a digit, and to display a digit we must reshape the row into a 20 x 20 matrix and transpose the result\n",
    "\n",
    "### Using Matplotlib .imshow() and subplots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaIAAAGoCAYAAAAEkOEJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXe4XFXZt+/QSSGETmihZQHSkRJ6kSoo5aWKQQRM4AUURNoH0hQRQhFEBUGp8lJDKEqRKiAgTQhl0UKv6Y2efH/M/GbtmZwkp8w5e8+c331duU7OnjV7Zj9n7b2evnpMnz4dY4wxJi/myPsLGGOM6d54ITLGGJMrXoiMMcbkihciY4wxueKFyBhjTK54ITLGGJMrc3XWiUMIA4CRMcbebXzfdGDRGOPoNrznivJnDWvTl6w+xxHA4cBnwMvA/8YYx7b3fPWg0WRYPk8P4ArgBZ0rhDAncC6wA6U5NyzG+KeOfE5HaUTZls+1K3B1jLFPR89VTxpNniGEc4E9Ad3jMca4d3vP1xk0oEzb/Qy1RQSEELYCjgO2iTGuDfwduDTfb9V4hBBWBe4D/qfmpSHAQGB1YH3gZyGEDbr46zU8IYSVgWFAj7y/SxOwMbBPjHHt8r9CLUKNRkefoZ1mEc2KEMJA4GKgD7Ak8Bywd4zx8/KQX4cQ1qe0UJ4UY7yj/L6DgMPKx8cAh8cYX5nF51wIbF5z+IsY44Y1x9YD/hljfK/8+y3AZSGEeWKMX7b3OjuTAsoQ4H+By4B3ao7vBlwaY/waGBdC+D9gf+DJ1l5vV1JE2YYQegLXAEcDf+vA5XU5RZNnCGFeYB3g2BDCisCrwFExxtp5W1iKJlM6+AzNZSECDgGujDFeE0KYG3ga+C5wc/n1N2OMQ0IIqwMPhRBWAVYDDgA2izFODSFsBwwHVp3Zh8QYj2zl93kCODKEsFyM8W3gQGAeYGHgw3ZcX1dQNBkSYzwcoHzeLMsA72Z+fw9Ys7XnzYHCyRa4pPzv+bZfTu4UTZ79gfuBk4AXgWOAESGEdWOMjdJqpmgy7dAzNK+F6Dhg2xDCsZRcNv2BrB/0TwAxxpEhhJeAQcCmwErAYyEEjesXQlhoZh/S2tU8xvivEMJpwPAQwjTgL5R8x4W0hsoUSoazYQ4ge4P3AL5pw/u7mkLJNoRwGPB1jPEv5bhBo1EoecYYRwE7Zd43DDgZGACMauvF5UTRZNqhZ2heC9F15c++AbgTWJZqv3f2ITUH8BUwJ6Ug7XEAIYQ5KAl/3Mw+pLWreQihD/BQjPHy8u9LAWeQAplFpFAynA3vlD9H9KdkFRWVosn2R0DPEMJzlLTM+cv/3ynG+EErz5EnhZJnCGFNYK0Y49WZwz3Kn9soFE2mHXqG5pWssD1weozx+vLvG1ISkvgRQAhhXUor+BPA3cC+IYQly2OGUgqM14P+wIMhhAXKv/8/4LqCm+lFk+GsGAH8OIQwVwhhQWAf4NYu+Nz2UijZxhg3iDGuXg4C7wR8Vg6wN8IiBAWTJzANuDCEsHz590OB5zPxjUagaDLt0DO0sy2iXiGEyTXHBgEnUjLhpgATgIcoCUusEEJ4lpI7Z59yCuA9IYTfAveWTb+JwO4xxukZM7NdxBhjCOEs4ImylvAIpTTEItAQMpwNfwRWBP5LSaO/JMb4UGd+YCtpBtkWiYaQZ9lddQRweyiVFrwH7Nuhk3YejSLTDj1De3gbCGOMMXniOiJjjDG54oXIGGNMrnghMsYYkyteiIwxxuRKl9QR9enTp6EyIiZNmlToXl69e/duKHlOnjy50PIEz9F6s8giizSUPEePHl1oeTb7PW+LyBhjTK54ITJMnz4dp/EbY/LCC5ExxphcyavXnMmJr75K7bRkBc0zzzwAfPHFFwDMMUdJP5lrrjQ9evQotAvdGNMKdM9//fXXAHz5ZXVPUj0LAOaee+4u+162iIwxxuSKFyJjjDG5UljXnEzIrCtJ5qTcRLUupDnnzDafNVmmTZsGwAorrFA5tuuuuwKw1FJLAfDiiy8C8NRTTwEQY6yMlduuK831ZqA2CWRWLk7N9VqXaXfmm2++qfqpe9z3euvRvQ/p/h04cCAAm222GZCepQ89lHoRv/zyy0C1i76zsEVkjDEmVwpnEWn1nnfeeQEYOnRo5bVddtkFgAkTJgDw6quvAnDrraWtbZ544onK2Pnmm6/zv2wDICtyxRVXBOCSSy6pvLbpppsCKWApuY4ePRpIGhHAZZddBiSNSee1hdQy0uB79uxZdVyWZRbJcocddgBgpZVK3fz/+te/VsZ8/vnnQNJcmxFZgtkA+uqrrw7AOuusA8B//vMfAF577bWq90B6dsysFCEru+5gbUoOWStcz9NDDjkEgD59+gAwceJEAL73ve9Vxg4ePBiAsWNLe9t15txr3lltjDGmISiMRVSrxfzkJz8B4NBDD60cu+eeewB49tlnAVh//fUBGDZsWNVPgLvuugvovmnH0sj79u0LwAknnADAJptsUhkzfPhwAK644goA3nzzTQDWXXddALbaaqvK2EsvvRSAe++9F4AzzzwTgHfeeQewZSQU51lmmWUA+Mtf/gLAzTffDMAf/vCHyljN+W233RaAiy66qOrnpEmTKmObWYPPxjAADj744Mr/pbkPGDAAgLvvvhuAAw88EIDll1++MnbJJUsbj+qel1dFv3/44YeVsS+99BLQ3PNWlvZBBx1UOab79ve//z0AF1xwAZBibscdd1xl7K9+9SsAjjnmGACmTp0KdI5lZIvIGGNMrnTJDq2taSgpv/Caa64JwJVXXgnA4Yen3WYfeOABIGmH0vrlP95nn30qY8866ywg+T7bolEWvaFkaxogSp77778/kDTxq666qjLm1FNPBeDTTz8FUnaM5JrVfHbffXcAzj33XADeeOMNIGmssqZgRi2z2ZueSvMEWHrppQE477zzgBnn5mOPPVYZqwxGxYJ69eoFwAEHHACkOAjMKNOiz9HWND2VJSSrXXNpyJAhlTG1WXKffPIJAP/+97+BFE/L/l/zdv755wfSvM7GPGUlvP766wCMGzeu0PJsS9NTyXWxxRYD4I477qi8Jqtm7733BuC9994DknwXWmihylh5nCTryZNLO5a3xiJy01NjjDENRWFiRPLjSvNWTcvjjz9eGSONsTbuM3LkSCD5PyH5NVdeeWUg+Z1ra5GajdrY0A9+8AMAHnnkESBZQQBjxowBki9dSDvKWsuKcbz99ttAiisdffTRVT8haWTNKmPRUmbcaaedBqQMuNNPPx2ARx99FICFF164MlY+eGWG/ehHPwKStdnM8QtI8bRVVlkFSN4PyTWLsjU1VpZ+NgtR55Pc9LuyQfv161cZqzho1kpqFpRhqTmo+BrAkUceCaTYrqxG3bPKmAW48847gfR8cNacMcaYpsULkTHGmFzJ3TUn948KULfbbjsArr32WqDa9K4tDqylf//+lf8r4Pv8888D3aMQEFJShtIw5fb54Q9/CMDHH39cGSuzfGZkXWtydyjYftJJJwFwzTXXANUBUaXYdkVrkDyR3Pfdd9/Ksb322gtIrlDJRy4luUohuaF//etfAyk1vpnnahbNKbnWlYykdHZILiPNO5UR7LbbbkB1cF0uPbmPlbCj848bN64y9pVXXgGaKy1e81GuNM25Dz74oDJGLuJa96XclnLpQ3IRdwXdY8YbY4wpLLmrrNJ4pLmvt956AJx//vlA6wLe0gRURAgp6Pazn/0MSAG8ZmyWmG0MqyDsfvvtB8CTTz4JJMuwNjGhrej9aoz6zDPPALDjjjtWxqiYuFmRpr3NNtsA1UWA0spVaKkyBGmaSp4B+Ne//gWklPqWWrI0M7L81ELmpptuAtIzAGCBBRYAUvKRitp/+9vfAvDRRx9VxkrLl9Vf2xQ5a2nqOdBM1qeeA2ussQYAe+yxBwBvvfVWZYxKAnr37g2k5qeSr8oNILUDu/zyy6vO74JWY4wxTUfuFpGoTdmcXQPDlt4rnztU+zqz52t2i0ja5KKLLgqkYklpnR31iUu7fPfdd4GUXr/99ttXxnQXjV6ylGwhzS+lDEs7lWav1jIAv/zlL4Hkw2+meEVbUKxS6cKjRo2qvKZ7es8996z6ffHFFweqywZUVK1Gnt2N2gJhlQpkm0EvscQSAPzud78D0jYQmq9Za+fss88G0j2v1lOdYbnbIjLGGJMruVtEWoFVwKrMoc033xyAESNGVMbWFqNKg1xrrbWAtNEbpBVfm7vJJ9rsSJ7ykytLRpp6th2NjrVFs5H1KXkqLpfVpLqibVSeaN5prmazi6RxKgtJrX7Gjx8PVGvw2tKgo3G7RkVzqbYAWvFMSPfvP//5TwCOPfZYAAYNGgTAKaecUhmrzNDuju4/3evZtlJqfqxiV1k5ylLOZiFef/31ABx22GEA3HLLLQC8//77QH29S7aIjDHG5EruFpG0IGUinXPOOUDK2FATSUgapJCfWK1/snUrTz/9NND9tE1pl5Kn6rCkoWebyKotvjSf1jQ1VDxKtTPKlstu4tZd0NxVTQqkWjdp6pqjJ554IgAPPvhgZWx3m5tCc3TZZZcFknYu2agRKaRs13/84x9AqjeURaQYHKSmp4oxNXsd28zQvJRltPPOO1deW2211YBkqf/mN78BknWa9WYoc1nPZNVmqjWVLSJjjDFNgxciY4wxuVIY21XFaGqNohYea6+9dmWMTE6Z6yoW3GCDDYCU4ACpq24zpmvXkr1GddVVQF2FwquuuiqQuhZD2pNE+xHJ7SHzPLtzplx92qNEXaX1HhVndickH6XKQ2p9tOmmmwLJpXTbbbcB1X+r7pLmXovuX7U2kntXSQoKnEPaC0eB96WWWgpILuJFFlmkMlZF3HI3dbcCYV2n3PGS0ZZbblkZo+SEiy++uOq9chNnW6o9/PDDQNqzSMlg2nU4u+NtR92gtoiMMcbkSmEsIiGN8dVXXwWqCwBrkUZ61FFHVf0OqfVHdwhYZgsh1dLn//7v/wAYNmwYkNKHlXoJae8cvefGG28EklUl7RNSO5vBgwcDKU1Z+8So1Q80V9uUWSGNM9uSRoW9CpiffPLJVb931wSFLLWau6ztEAIAP/3pTytjtUdTbXNkvSfbuFfJChojy6u7WETyKil5RgkJauMDyVrUztW1z8fsHlias2oLpPY/3/rWt4BU1N7SedpK93hiGGOMKSyFNRdq96rPIu1IP9XuPKuJZ62jZid73Wo3o51YN9lkEyC1+8hamEopHjJkCJA0UWmbWe1IBazSMuVrVppyS4WyzYpSXZV+LKsH0lYDkv8LL7wA2BLKIllIO1cDXZUE6H6GlP6u+VV7X2fnvlrZSNuXZdRdkCw+++wzAC644AKg+n7U3J2ZBdPSs+SGG24AkuWvEoV6Fq7bIjLGGJMrhbWI2oJ8wNI+uzO18RnFcFrKIDrwwAMB2HrrrQHYYostgJY1SWmi0l6VDSZNtTvEhSQDaZOyIFUkCKmRqbTIrFVpSmiuqEmpNHe18dloo40qYwcMGACk+JHmsbR1Nd2FlJnY3a1Pybel7Ne23Kea58pAnjp1KpDicvW855v/6WGMMabQNKRFVFtPJC1dtTKQtKJmb8A5O2aVzSLZaCO7++67D5i1pqP4UUubjTU7ypKT5XjwwQcDaeMwgD//+c9Akkt3ydhqD5KRWnQpUzbr2VA8onZrB2XcTZo0qXJM93x3mpOzoqNzT7ElbbUuy0jtlbRtByRrqb2y91/MGGNMrnghMsYYkysN6ZoTKuS8+uqrgdTiA5Ipr50H7SKZEcmk1o05K3dmd91FFJLbQSna2vNKwXZIyRvdoZC63iixI5vgobmYdcFlyRa0mvqi54PSwdWKSfdB1g3XUXeoLSJjjDG50qMrgvl9+vTp1A9RkVa2YZ80pfZYQpMmTSq0+dS7d++GysCYPHlyoeUJbZujsno03xRQh66zvIs+RxdZZJGGmqOjR48utDyb/Z63RWSMMSZXusQiMsYYY2aGLSJjjDG54oXIGGNMrnghMsYYkyteiIwxxuSKFyJjjDG54oXIGGNMrnghMsYYkyteiIwxxuSKFyJjjDG54oXIGGNMrnghMsYYkyteiIwxxuSKFyJjjDG54oXIGGNMrnghMsYYkyteiIwxxuSKFyJjjDG54oXIGGNMrnghMsYYkyteiIwxxuSKFyJjjDG54oXIGGNMrnghMsYYkyteiIwxxuSKFyJjjDG54oXIGGNMrszVWScOIQwARsYYe7fxfdOBRWOMo9vwnivKnzWsTV+y5XPtClwdY+zT0XPVE8uzvjSaPEMI5wJ7AmPLh2KMce/2nq8zaCSZhhAGA0dnDvUFlgaWjjF+3J5z1pvuJM9OW4gakRDCysAwoEfe36UZsDzrysbAPjHGx/L+Is1AjPEq4CqAEMLcwMPAWUVZhBqNjsozl4UohDAQuBjoAywJPAfsHWP8vDzk1yGE9Sm5Dk+KMd5Rft9BwGHl42OAw2OMr8zicy4ENq85/EWMccMWxvYErqG0qv+tA5fX5Vie9aVo8gwhzAusAxwbQlgReBU4Ksb4TseutOsomkxrOA74JMZ4SduvLB+aTZ55WUSHAFfGGK8pr55PA98Fbi6//maMcUgIYXXgoRDCKsBqwAHAZjHGqSGE7YDhwKoz+5AY45Ft+E6XlP893/bLyR3Ls74UTZ79gfuBk4AXgWOAESGEdWOM09txfXlQNJkCEEJYBPg5sF6bryhfmkqeeS1ExwHbhhCOBQZSutGyftA/AcQYR4YQXgIGAZsCKwGPhRA0rl8IYaGZfUgbNM7DgK9jjH8p+2UbDcuzvhRKnjHGUcBOmfcNA04GBgCj2npxOVEomWb4CTAixvhmG68nb5pKnnktRNeVP/sG4E5gWarjCN9k/j8H8BUwJ6Wg93EAIYQ5KAl/3Mw+pA2r+Y+AniGE54B5gPnL/98pxvhBK8+RJ5ZnfSmUPEMIawJrxRivzhzuUf7cRqFQMs2wN9DW9xSBppJnXunb2wOnxxivL/++ISUhiR8BhBDWpbSCPwHcDewbQliyPGYocF89vkyMcYMY4+oxxrUpaZ6fxRjXbpCHJlie9aZQ8gSmAReGEJYv/34o8HyM8b06nb8rKJpMCSH0K39WIyaANJU8O9si6hVCmFxzbBBwIjA8hDAFmAA8ROkCxAohhGeB6ZQyhcYC94QQfgvcG0KYBkwEdo8xTs+Ymc2O5VlfGkKeZffKEcDtIYQ5gfeAfTt00s6jIWRaZiXgwxhjkS3LbiHPHtOnN0qs0xhjTDPizgrGGGNyxQuRMcaYXPFCZIwxJle8EBljjMkVL0TGGGNypUsKWnv37t1QqXmTJ08udJPOnj17NpQ8p06dWmh5AvTp06ehZDpp0qRCy7RXr14NJc8pU6YUWp7Nfs+7+7ZpE998UyrYVtr/nHOWauh69Cj0fWyMKTB2zRljjMkVW0SmVcgS+v73vw/AnnvuCcCpp54KwGuvvVYZO9dczT2tJIvPP/98NiNnJFtALjnNN9989flixjQotoiMMcbkihciY4wxuVJ4H0rWlfHll19WHWuLS0Pv+frrr2c47zzzzNPh79mMZGU0//zzA3DwwQcDsN56pX2vLr30UgBeeSVt8tisrjm55JZcstS8+Dvf+Q4APXv2rIxR0obG1qLkDoDXX38dgAcffBCAadOmVZ3DmCJQ24+0M+anLSJjjDG5UljVVRplVrv+xS9+AcASSywBwMknnwwkK6ellVqvSaPfaafSRpcLLrhgZcyNN94IwFdfFbkbfMeQViOtG6q185bIju3Xrx8Aiy22GADvv/8+AG+99VarztXISHaai2eccQYAe+21FwAfffRRZaysm+233x6AeeedF0iyzFrf//3vf4GU6CFZNqtFWQ8kR/1N5pgj6dK197/G6FmSnaPNbHXWllhkrzV7T2df08/s6/q/5rCeofJMzczqbw+2iIwxxuRK4VQvreK9evUC4Mgj066zhx12GAB//OMfgWTBtKTd6DVZPt/97ncBOOmkkwB48cUXK2NvuOGG+l1AQZl77rmB6njGuHGlHYJnZs1kLcQtttgCgIEDBwLpbzBq1CigueNssqpl5ey8884AvPDCCwD87Gc/q4z9z3/+A6T5OyvNW6998cUXgC2hWSHtXHNV83j06NGVMZrjkqusJXlQxo8fXxk7q2dHoyPvRUvzqXfv3lWvTZgwAUjPXVk9AP379wdgv/32A9L8/8lPfgLAffelzV1lNbUXW0TGGGNypXAq2GeffQbAHnvsAcDRRx9deW3y5NKOuQ888ACQNFVp41kNfqWVSrvmnnbaaUDK8pImcM8991TGqjBRGlUzIW170003BZI2A3DKKacAKXtL2qa0z6z1tM022wBJRk888QTQnBplLbrGqVOnAmkO/e1vfwPg4YcfroyVRjlx4kQgaeW1vviZHTPV1MaKNWdXXHFFAN54443K2FtvvRWAt99+G0gxvG233RZIHhWADz74oOq8jUxtFvHZZ58NwIABA6peh+Qh0jPz008/BdI936dPn8rYZZZZBkhxYT0zlSFbT9nZIjLGGJMrhVEHlImx6qqrAsnvntXKZQm9+uqrwIyxDZ0DYJNNNgFSbEia6hVXXAHATTfdVBnbjBlftVbN4MGDAVhzzTUrY2Qt1SINSn8LgNVXXx1IfnbFR5pRdrXoGp966ikgzTP9zMpggQUWAGDhhRcGYMyYMUCyurNtgdwwthrNu+x9LAtcnhE9FxTfXHrppStj5QXRPT5o0CAgWfPZusPa2phmQPe8PEWKFb388suVMYrlKDYki33KlClA8joBrLXWWkB63h511FFAsibrGRe2RWSMMSZXvBAZY4zJldxdc7WBNrWQkVtI6cEAp59+OgCffPIJMGOSwpZbblkZKxNepr3cKueeey5QnfbZ0dTDIqIg749//GMgpV4OGTKkMkYFlLUmtkx8yQpg7bXXBuCWW24B4N133wW6l1tJc0muo1133RVIgXOAFVZYAYDNN98cgMcffxxIBcC33357ZayKX+UizRZndgd079f+XGONNSpjDjzwQAD2339/IM3NJ598EoATTjihMlaJTh9//DEAP/zhD6vGZtO3m0nWugf1HDzkkEMAWHbZZQEYO3ZsZWxtMWpt6zMlOgDsuOOOQErKkYuuM0o1muevYYwxpiHJ3SKShrPKKqsAKagujUXaDMBzzz0HpCCvNKBFF10UgOOPP74yNoQAwDvvvAPAMcccA6TAXbPuASNNRxaMZCI5SEOH2ScaZNsgSWsfPnx41e/NmPI+O6RVrr/++gBsttlmldc+/PBDAK699logpbpqHydp9pDm/GWXXQak9PBmtzI1R2sTamTBZIvYV155ZSBp9XfccQeQEhKyng21Wtpoo42AVLJx8cUXAynBAZrLIqpFHh7NxZZKBpR6LStq3XXXBZIFCvD8888DcM0113TyN7ZFZIwxJmdyt4ikHangUimHOr7ddttVxl533XVAKqZUIaE0H2lCkAoKTzzxRAAeeeQRoDktoWwqqjSeww8/HICFFloISPG1rAY5u4K0rCYlv7taIzVj+uvskAV+2223Aak4+KGHHqqMkQWqRqZ6z5VXXglUy1R/I2n3++67L5DmfjNZRtkGmboHFWM79NBDgVRakI1BqEhV243Iotf9nUXtpxTb1JxVC6+W7pNmZlZWn2JCej6ceeaZQGoBBHDOOecAKU7fmbF0W0TGGGNyJXeLSHEKWSxqSqrMGRWmQmr7s/vuuwOpsEp+TjWaBLj66qsB+Mc//gE0d1PObOt2teOQdqkmnMrWyo6dWRv32gaTkDJm5IfvDoWsonazu6uuuqrq9euvv77yf/nVFTvTvJMGqp+QfO/LL788kP5myvBshvYzLbWL0vYtyu7Sa/fee2/VT0jWojLeJE/Nv2yMUs+O1VZbDYChQ4cCqY1Nd5qzM6N2Oxi1PVJW7XnnnVcZe/PNNwNdk1Vsi8gYY0yu5K5ySUsZOXIkkDYLU/O9bAsPWUdqPbHPPvsAyRJSXQzA5ZdfDiQffTNnd2W1bLUzkZatOIRiQ4svvnhlbK0PWVpSS9sSSI6z2oSw2dFcVXsj1bC0Ztt5ySs7D1WLpRiTNHrFnqTJZz+70dCcUusjgB122AFI13fRRRcBqe2Wsr0gXbe0clmlijP97//+b2Wstuf46U9/CqSaN83j7jhna5H3SDVCP//5z4FU06a4EHRtHNgWkTHGmFzxQmSMMSZXcnfNCbks9FPuIaXBQipGVSLDbrvtBiTXXDaIrDTPZkzXriVrQms3ShWjbrzxxkBq97HTTjtVxipILJeFCjUl+6xbVEH47pi2XYtcmi25Kdvi/pHbSW5oFWHrZ0suqkZD8sjOGxX7jhgxAkj3tVxoWfdlbaKIZC+3qFxLkNxKSvmu3Ququ5Ldp0339BlnnAGke/7UU08FUjISVO/W2tnYIjLGGJMrhbGIapE2kw2oSwNVQ0lp/Up3VXM+aO7khFqySQV33nknkPYP2nPPPYG030h2R0vtjaM2NNqpVdqr9neBVGisYLwto6RpZ1Piay0AafK1u99C0kY1j7Wjpv522TTmRm3Mq+vOJl789re/BdK9Patrqy3uVTPjvffeG0iF2gAXXnghkCz6RrUi64XmYPb5cMQRRwBpjqmQNW8Pki0iY4wxuVJYi0hkiy7l39Suq1q9peVnta7u5BfOajyKqamYb8kll6wam5WRZFtbbKmf/fv3r4xVOri0ze4k31okAzUy3XDDDSuvycqUr13tqLR1SbaFilKaVciqpqcq3mzWxpzyVsxsDmWtxr59+wJwwAEHACk1W5bRjTfeWBmr+6C7W0JC83S//farHFND2bvuuguAYcOGVb0nr/u6OWe6McaYhqHwFlE240Ob5SlrbtKkSQA88MADQNIooXF96h1FWrSsHRVNipY0ntpiS2mWaq8CSZ7a0747a52S8YQJE4C0VQEkq12Fw4oDqXHkwgsvXBkri0htmZ5++mkAHnvsMaC52lK1RdPOFmirkekGG2wAJA1e7WeyMurOVnoWyW+55ZYDUuNnSJmYp512GpCemXnPNVtExhhNeQzHAAAgAElEQVRjcqXwFlEWaTyqu1BbIGWKNUOTyHohWbVHS9R7shl2ysBx1lySgdqiyJKBVHuhrUmklaqlSlbzlEU/ZswYAH75y18CyYrtrvM5m/GqdkraKkIavNv2zIhia4pDqtZKTWAhzbFnn30WKE52sS0iY4wxueKFyBhjTK706AoXS+/evdv9IdnApYLC2tterjmlcNYrgD558uRC2/s9e/ZsKL/Y1KlTCy1PgD59+rRbptl0Y91PtT9bg5IgWjOPJ02aVGiZ9urVqy5ztHb/nM5q2zNlypRCy7M197wSY5TMpZCF9hKDtGNBZ5cItPWet0VkjDEmVwpvEWVRSrKKBqU51rsthS2i+tLsFlEedBeLqKtoBotIz3KVWsgyyhaxq4ygs8svbBEZY4xpKLrEIjLGGGNmhi0iY4wxueKFyBhjTK54ITLGGJMrXoiMMcbkihciY4wxueKFyBhjTK54ITLGGJMrXoiMMcbkihciY4wxueKFyBhjTK54ITLGGJMrXoiMMcbkihciY4wxueKFyBhjTK54ITLGGJMrXoiMMcbkihciY4wxueKFyBhjTK54ITLGGJMrXoiMMcbkihciY4wxueKFyBhjTK54ITLGGJMrXoiMMcbkihciY4wxueKFyBhjTK54ITLGGJMrc3XWiUMIA4CRMcbebXzfdGDRGOPoNrznivJnDWvTl6w+x7nAnsDY8qEYY9y7veerN40kzxDCYODozKG+wNLA0jHGj9tzznrTSPIsn6Nw87PRZFg+Tw/gCuAFnSuEMCdwLrADpWfisBjjnzryOe38bgPopvLstIWoAdkY2CfG+FjeX6TRiTFeBVwFEEKYG3gYOKsoi1CD4vnZQUIIqwIXAxsCL2ReGgIMBFYH+gD/DiE8E2N8suu/ZeNQT3nmshCFEAZSuoA+wJLAc8DeMcbPy0N+HUJYn5Lr8KQY4x3l9x0EHFY+PgY4PMb4yiw+50Jg85rDX8QYN6wZNy+wDnBsCGFF4FXgqBjjOx270q6haPKs4TjgkxjjJW2/snwomjwbcX4WTYZl/he4DKiV227ApTHGr4FxIYT/A/YHCrMQNbs887KIDgGujDFeU9aYnwa+C9xcfv3NGOOQEMLqwEMhhFWA1YADgM1ijFNDCNsBw4FVZ/YhMcYjW/l9+gP3AycBLwLHACNCCOvGGKe34/q6mqLJE4AQwiLAz4H12nxF+VI0eTbi/CyaDIkxHg5QPm+WZYB3M7+/B6zZ2vN2EU0tz7wWouOAbUMIx1Iy4foDWb/onwBijCNDCC8Bg4BNgZWAx0IIGtcvhLDQzD6ktat7jHEUsFPmfcOAk4EBwKi2XlwOFEqeGX4CjIgxvtnG68mbQsmzQednoWQ4G+YAsgt6D+CbNry/K2hqeea1EF1X/uwbgDuBZSl9WZH90nMAXwFzAlfHGI8DCCHMQemPMW5mH9La1T2EsCawVozx6szhHuXPbQQKJc8MewNtfU8RKJQ8G3R+FkqGs+Gd8ueI/pS0+CLR1PLMK317e+D0GOP15d83pCQ08SOAEMK6lFb0J4C7gX1DCEuWxwwF7qvT95kGXBhCWL78+6HA8zHGok3GmVE0eRJC6Ff+rEYMrhdNno04P4smw1kxAvhxCGGuEMKCwD7ArV3wuW2hqeXZ2RZRrxDC5Jpjg4ATgeEhhCnABOAhSsITK4QQnqVk3u0TYxwL3BNC+C1wbwhhGjAR2D3GOD1jdraLsjl7BHB7OfXwPWDfDp20c2gIeZZZCfgwxlhkrb0h5Fnw+dkQMpwNfwRWBP4LzANcEmN8qDM/cBZ0S3n2mD69qLFOY4wx3QF3VjDGGJMrXoiMMcbkihciY4wxueKFyBhjTK50SR1R7969GyojYvLkyT1mPyo/LM/6069fv4aS6bhx4wot0549ezaUPKdOnVpoeS644IINJc/x48e3SZ5uemqMaViyWb/Tpk2rOjbHHNUOn9rfuzO12dKSXY8eaf3Q/7PHOgv/ZYwxxuSKLSJjTMPxzTeljjZZK2exxRYDoGfPngCMHVvaukna/ueff14Z253qJyUrgK+//hqA+eefH0jy69evHwBffvllZeyUKVOqjnWmRWmLyBhjTK54ITLGGJMrDeWakzld+1MmY1cE1ZqJWvdEa+TXnvd0JySftsiyVqYtHdcc7+4B96++KrUuXGSRRQA45JBDKq9973vfA2DxxRcH4NFHHwVgvvnmA+Coo46qjP3ggw8AmHPObN/Q5kDzRi45yQpg4403BmCjjTYCYIEFFgBggw02AGDMmDGVsa+//joAN9xwAwAPPVRqFzf33HMD9b33u/esNsYYkztd0vS0tu4lu5Jq1a79HtJ8tPqWz1P1c4kllgDg3XdLmwFmV/OOaDpFr3vpSB1RVs76O0hjnDy51PQ3G9ysZa655qr62VLaZy1Flye0r46opXtH81UBXv3U2HnmmacyVjJcaKGFqsauuOKKQNJWAV599VUgafITJkwotEzrVUdUq91vvnlpz7bDDz8cgO23374yVvL873//C6Q5ud56pQ2C11hjjcrYV14p7Zatv1cz1BFJRko8+PnPfw7AoEGDKmOWX355nQ9IMps4cSJQfR/36tULgDfeeAOAU089FYAHHngAgKlTp1bG1t7/ba0jskVkjDEmV3KJEUmLhrR6L7vssgB89NFHAKy0UmmrjfXXX78ydqedSrslay8NafKXXnopAGeffXZlrLR7xzBKSLOUpQmw//77A0nTOeusswB49tlngaQtQYpNyLe84YalnYMvueQSAO6///7K2Hnnnbfu379IKAVW83jdddetvPaLX/wCgDPOOAOARRddFEiyzI791re+VXVMY2UZZS2iYcOGAelv1cxknw/6v+bqr371KyBZkdn7+9577wXgpptuAuCUU04B4JNPPgGqU5Ob6bmge1tzTHNvr732Aqq9Q5Ln008/DcAjjzwCwN133w2k5ybANttsAyTrU3PvtddeA+DFF1+sjM1a+u3BFpExxphcycUiymoma665JgB77LEHkIrODj744Bnep9V69OjRAPz9738H4KmnngKqM4qktUob6Mp2FUVE8pAWDjB06FAAllpqKSBZlF988cUM75f/uW/fvkCyrP72t78B1VpssyK5yM8uH/yqq65aGdOnTx8AdtxxRwBOOOEEIBUQZuNv0mTHjRsHpKykF154oepckDTXjmqeRUayyV6jtPHDDjsMSJaQ5vMf//jHylhZ5/KyyKJUYWazzlHNI805PUslT8kK0hzTvBw5ciQwY9wd4OOPPwbSc7d///4AHHDAAQCcdtpplbF6prf3+WqLyBhjTK7kYhFlV83a1fbxxx8HUlZLNk5xxRVXADBw4EAgrfjSfLLxpE8//RRIGR/K8NB7mrF+oCWkDUlLzGoxa6+9NpA08r/+9a9AyorJZizKQt1ll10AuOCCC4DkW1a8rhnRnFH21Q9/+EMAtt12WwBuvvnmytjLL78cSPUainm+8847QKptAfj3v/8NwHvvvQckDVRegWxWnr5D9m/SLGiO6l7fc889K6/9v//3/4A0v5Q9qOPyikDSyjfddFMgyUytf7Kxy0Zv8ZP1WmgeKsYr5CG67bbbKsdUSyXPxumnnw6k569iv5BiQYoZy+L6n//5H6Ba9g8//DBQ/bxuC7aIjDHG5IoXImOMMbmSi2suGzxbeeWVgeTKUOdcFVMpkA5w7LHHAqlwVe62fffdd6afJVNeZuS1114LpAIuaM4EBgVmFfiV7GReQ3JPyqSXu03vzZrp66yzDpCCm1dffTWQkhbaa5I3ArrGwYMHA6mVjALpWdeH5CB355NPPgnA22+/DVQn6gjNv1m1qmpGV3JtCxolf2Tb9siFpiLV4447DoD77rsPqHYJS36at7WF2tnnTqOTvRaVuqgERtf70ksvAXDHHXdUxirpQ89MJd589tlnQHUJzHnnnQfAfvvtB6TiYRXDDhgwoDJWrrn2YovIGGNMruSixmbTrMePHw+kFVnFqtKAbr311spYBdEV1JVFtPDCCwPVmqRSapXYoIK4rbfeGqhugKhAcjNpnbJqdt55ZyAlG7z//vuVMSoOHD58OJCuX1aptE9If58jjjgCSJZmM6cTC83X2qDtaqutBsBbb71VGRtjBNK8VtKMtPNmTDZoK7XNipUOfOihhwLV96GeA0cffTSQkpmUDp9Fc16WgcYo9V37E2U/uxmQPGvT02W5nHjiiZVjko3m42OPPQbA+eefD1RbNjNL7JpZs96O0Dx/DWOMMQ1JLhZRNo1S2or87ordKO016wuVr762OFUp2ln+85//AGnVVoqtit6yaczSyJrBIpJWpAK13/3ud0CyHk866aTK2Ouuuw5IVo2u/8c//jGQ0mCz73vwwQeBFMPrDkguzzzzDJDiZGohky2+vv7664FktauViopUZVlC97WOlJ6uuKWsHcXXJF9Icbk333wTmFFm2ZibyhE222wzIMlaXpWsRdTMlryeASpOzT7XZAk+99xzABx55JEAvPzyy0D1s1ljuyLV3RaRMcaYXMm9oFXtN6QF1Vo72bHt0WKkMSk2IktLvtLaz2h0dC3y71555ZVA8gXfeeedlbHyE0tGW2yxBZDaf+g9kDR9xY+6E9Io1eRRsTMVXau5JCTtXn8HZXjKksxapM28OVst2Wa7ajO1++67A6kgXbHarIy0OVvtva/5nY2LKKtLbcMUr1PhcDPd51lqsy6FLBlZoAB33XUXkDLiFOttqSC9NpbXmdgiMsYYkyu5F39oNZdWKM1Jq3BbtMWsdiQtX9qXMkdUr6QMvGZD8pQcTz75ZGDGVvGQtMqll14aSFsNSIPSeyHFmLprXAPSnJQWqRiR4o6QtsnYbrvtgLQZ22677QZUa55qFdTM1MYsAY4//nggxXSEGnLKeoQ0X2s3yFNG2JAhQypjVdel9jeXXXYZkCyjZqp1y16LWvHIolRjWJHN6lSmrI7NKuYmGe+6665VY2sbSdcDW0TGGGNyxQuRMcaYXCmcrSrXmdK4lTILM5qRMvtltqvLLqRO3DLX5SqR+XrhhRfW/bsXkdq2MVn3pQLA2tFR7T6U2p5Ni28mt0ZbqS2UVGsTFf9l5aQ0Y3WKVwBdbiK56iC1ttEOos1UZCkkOxWdQyoL0H2rIlW1l8kG12sD8eoSrd1DlQIOKfVYLZdUnlB7rmYgm2atMIP2ZlJncrnQsnKQG0/PUrniVCajlmuQWi1tsMEGQJKfXJ3ZHVo7Ktvmm/nGGGMaisKouVpRtQp/+9vfBqpTiFV0JaTB66eCa9lj0qSUrKB97bPnamZtX3KV9pktEFYhr/YXuemmmwD4/e9/D1TLpRm19dYi2S266KJASn1VMbaKpyG19llllVUA2HLLLavee//991fGqv1PM8tWsssmaajQUnNRDWFVGrDJJptUxqqBqRp7ah8dWU2SISTLQAF5pc438/0N6R5Xc1NZibKalllmmcpYWaNKUFJig5oaa95Carem82hnbO1hlJ33HS0Qbt47wBhjTEPQoyvaN/Tu3Xu2HyLtSA0llUqcTUWUFiQNQFqWtCIVvwGMGjUKSP5RNaxUamfWx1rL5MmTC+1Mbo08a5F8l1tuucoxbYkhv7C0d7WPr1eqdtHlCdCvX7+ZyrS2sE9p7bLes9q+5Kyxiv8oXfaYY46pjJWPvT0a+7hx4wot0549e06HlG6dvY9VHC3LR6UBujez8Qa1kpKMFI9TAfyZZ55ZGSt56u/VltKPqVOnFlqeCy644GzveclIBdd77LEHkKwfSFtESDaap7Jo9PfKomel0uqHDh0KwIcffjjDZ4vx48e3SZ62iIwxxuRKYSyiWlZffXUAFl988coxZeBoFVebd2VxZC0iaVm1DT1bQ9E1+I5YRGr3DkmzUYPOrbbaCkjae71azxRdnjBri0joXlFxpjIxs81hpVFKO5fmrnZAkyZNqoztSGyoUSyilgqpVfSrLUokP7XmybYDevTRR4GUCacMO2V/qUVY9jPak8HVDBaRUCactrxR5nD2WEuWD1Tf83oOKE5/+eWXA6lh6qyeD7aIjDHGNBSFtYikFWVX7toMsNoag6zW1ZG89qJr8G2Rp/zuu+yyC5DqWSBZi4rHyVqS9VSvuouiyxNaZxEJWeaSU0sNI2WRS4aKt9UrQ65RLKKWUKxXHg3VGKn5afaZpMy32lor3ev1kmczWURC81RyhZS92Ro0h/U30LOkNZ4SW0TGGGMaCi9ExhhjcqWwlV5yZXTnbs/1QO4jtTzKmtVqBfLnP/+56j3NXGBZD1qT8tqSu86UkGzkOlKyTDYdWEjWlmfbkezULg1SwXVrqA19dOa+WX7iGGOMyZXCWkSmPiggrCSF7A6tKlwVtoTaRzM10+xKavci6w471eZB9r4u6j1ezG9ljDGm29Al6dvGGGPMzLBFZIwxJle8EBljjMkVL0TGGGNyxQuRMcaYXPFCZIwxJle8EBljjMkVL0TGGGNyxQuRMcaYXPFCZIwxJle8EBljjMkVL0TGGGNyxQuRMcaYXPFCZIwxJle8EBljjMkVL0TGGGNyxQuRMcaYXPFCZIwxJle8EBljjMkVL0TGGGNyxQuRMcaYXPFCZIwxJle8EBljjMkVL0TGGGNyxQuRMcaYXPFCZIwxJle8EBljjMkVL0TGGGNyZa7OOnEIYQAwMsbYu43vmw4sGmMc3Yb3XFH+rGFt+pLV5zgX2BMYWz4UY4x7t/d89aYB5bk/8AtgOjAVODLG+FR7z1dvLM/60oDyLPT9Dg0p0zWAi4C+wDfAkBjj0615b6ctRA3IxsA+McbH8v4ijU4IIQDnAOvGGD8MIewE3AIsm+83a0wsz07B93sdCSH0BO4BDoox/j2E8H3gWmCV1rw/l4UohDAQuBjoAywJPAfsHWP8vDzk1yGE9Sm5Dk+KMd5Rft9BwGHl42OAw2OMr8zicy4ENq85/EWMccOacfMC6wDHhhBWBF4FjooxvtOxK+0aiiZP4Avg4Bjjh+XfnwKWCCHME2P8sr3X2VVYnvWlaPJs9PsdiidTYDvgjRjj38u/3waMau315GURHQJcGWO8JoQwN/A08F3g5vLrb8YYh4QQVgceCiGsAqwGHABsFmOcGkLYDhgOrDqzD4kxHtnK79MfuB84CXgROAYYEUJYN8Y4vR3X19UUSp4xxreAtwBCCD2A84DbGuGhWcbyrC+FkieNf79D8WQ6EPgohHA5sBYwHji2tReT10J0HLBtCOFYShfQH8j6Qf8EEGMcGUJ4CRgEbAqsBDxW8lQA0C+EsNDMPqS1q3mMcRSwU+Z9w4CTgQG0YVXPkULJMzO+F3AFsAywQxuvKU8sz/pSKHk2wf0OBZMpMDclmW4VY3yi7Jr7ewhhuRjjF7O7mLwWouvKn30DcCclX3ePzOvfZP4/B/AVMCdwdYzxOIAQwhyUhD9uZh/S2tU8hLAmsFaM8erM4R7lz20ECiXP8vmWBW4HXqY0OT9r7XsLgOVZXwolzya436FgMgU+AF6OMT5Rft+IEMJlwAqU5uwsySt9e3vg9Bjj9eXfN6QkJPEjgBDCupRW8CeAu4F9QwhLlscMBe6r0/eZBlwYQli+/PuhwPMxxvfqdP7OplDyDCH0AR4Ebokx7tNgD02wPOtNoeRJ49/vUDyZ/gNYPoSwXvlzN6eU4dkqC7OzLaJeIYTJNccGAScCw0MIU4AJwEOUhCVWCCE8S+lC9okxjgXuCSH8Frg3hDANmAjsHmOcnjEz20XZfD0CuD2EMCfwHrBvh07aOTSEPIHDgeWA3UIIu2WObxNjHNPRk9cRy7O+NIQ8G+h+h8aR6UchhF2BP5RdyF+Uz/35bN4KQI/p0xslNmeMMaYZcWcFY4wxueKFyBhjTK54ITLGGJMrXoiMMcbkSpfUEfXr16+hMiLGjRvXY/aj8mOBBRZoKHlOnDix0PIE6Nu3b0PJdMKECYWWqedofenVq1enyvPLL0tNOr7++uvKsfnnnx+AHj3aLpopU6a06U22iIwxxuRKQ3bfVsp57c8sc8455wzHjDHGJL76qtRMYptttgFgrbXWqrz25z//GYDPPy+VArXHMmottoiMMcbkSmEtolkV2n7zTamN0rzzzgvAfPPNN8OYyZNLxciduYo3Ilm5yh+sn9OmTZvp++aee+6qn5brrNEclbwlL1vq9UVyhjR/peXPMUdJz9ZzwnM28cUXpT6kq65aarx9wQUXADB27NjKmD/+8Y9d9n1sERljjMkVL0TGGGNypXCuObkyZHJnzWkdW3nllQE4/PDDAVh22dKOyW+//XZl7NFHHw3AXHMV7hK7FMlT7rd55pmn8poaHa6ySmk33yWWWKLqvUrpBHjhhRcAGDlyZNX5urO7o1a22WOLL744AP369QNg0qRJAHzyySeVsd1Zdm1F7jY9A3Rf694HWHDBBYEUcJesH3ustBv41KlTK2O7q+zlvlxggQUA+MUvfgHAwIEDATj//PMrY5WkIHd8Z2KLyBhjTK4UxlyQprPQQqXNAo855hggBdMgBdiWWmopANZYYw0APv30UwBuvPHGytju3lVcWrqK0jbbbDMABg8eXBmz+uqrA0mr7NOnD5C0xWwg+K233gLg97//PZBSO7ujZim5SLZ77bVX5bUtt9wSSBqmkCZ62mmnVY49/PDDgK321rDvvqVdGtZZZx0All56aQBWWGGFyhglJ+i1119/HUha/ogRIypju2vSiJ4Le+yxB5Dk+v777wNwyy23VMZ25b1ti8gYY0yuFEYVk5a55JKlzQO33XZboFrjEePHjweSdi5L6Pnnn6+M6a5apuI60hxlWW611VZAillAsiSfeeYZIPnSJbvsZllbb701AD/5yU8AuOaaa4CuKXYrCpqjsiR/9atfAbD55ptXxshqv+GGGwC47LLLANhtt9J+dtJAAf71r3918jduTCRDSM+BX//61wD07NkTSBa65i7A/fffD8BLL70EJC1ff7fuagUpvgYwYMAAAIYOHQokC0nzNCvPrpSXLSJjjDG5UhizQVr4G2+8AcCwYcMAOOWUUypjFl10UQDuvfdeAE488cSqc2RX8NoYUTNr7NlYjjLfJDdplE8//TQAt912W2Xsc889ByQtaMyYMVXnkxUFKfYhP3x35vvf/z4A3/nOdwB46KGHKq9dfvnlANx+++1Ast433XRTIGUompmTnWMff/wxkLIONY+POOIIAD766KOZvl8/m/nenxUtxcl33HFHIHk7YowAXHvttUD1s0TZcl3xLPVTxRhjTK4UxiKS9lLrY99nn30qYxZZZBEANtxwQyDlwkvrzMaFtGorZqLXmlGjz9ax7LLLLkCyYBQ3+/nPfw7Av//978pYyUK1RWqFIg1o5513rozVa6rF6I5ZiZKXtMdHH30USLVVkDT42pZICy+8cNXr0D1l2Bqyng1ZPKoJuu+++4AUI1Kmp5kRPRcULwY47LDDgHQ//+Mf/wDgnXfeAaplr2dnrWWpn/V8ljbfU9kYY0xD4YXIGGNMrhTGNSdk7skszKYeit69ewPJ7bHddtsByR0FycR85JFHAPjnP/8JtNw6qNHJXotcmyqgvP7664HkklP6a0soFXuLLbYAqgs1ZearkFUuuu6UJi85q5XUm2++CVS7KOTmlNutV69eACy33HIAvPbaa13zZRuYrKtZyR1qmSS5Kg35ww8/rIzNtq/qztR2fD/11FMrr6k9mp4Hv/vd76rGZl2dyyyzDAAbbLABkAq4r7rqKgCmTJlSGdtRN50tImOMMblSWHW2NjAGaaWXJaQiLLVTUeuf7Nhdd90VgAMPPBCAJ554AmguTT7blFDp2dLWVdTX0p5NQhqoUr+PPPJIABZbbLHKmHvuuQeAO++8s15fu2GpTQ9uCVnekqG0S6XLwsyTFbIptKI7FWNm5aJ5K238l7/8JZCScs4999zK2AcffBBornu7LdTuWP2zn/0MgI033rgyRtb86aefDqSSDc3lTTbZpDL2nHPOAdJzQfNy1KhRQHUpyKyeL63BFpExxphcKbzq0JJ2o2NqraI4UrbFvuJIKoL97ne/C6SU22bSmrJWo3amVfsYadItXW+tL1lbZ6hQ87///W9lrIqH1RaomeTXXmo10Cyak5KtNMZXXnmlMkYxOVmksmy1nUE25pHdObPZyc4tyUvNYtdcc00gFQifffbZlbFqo6S4UTOWaswKWSyDBg0C4OCDDwaq5aBGAQ888ACQYsnf/va3ATjjjDMqYzV3Vfi+3nrrVf1U6jfM+CxpK93rL2WMMaZwFFat1er+5JNPVo5ppZfmKK1z+PDhM4z9zW9+A6QVv7toR7pOFazNCmniyo7bb7/9APjss8+A5EeGFNvozpaQ5CUZy3LJylpzUs171157bSBZ6GqYCilDSa+p6FVZShMnTqyMPeCAA4DuIf/sNaqgtbbJ7g477ACkrC9IGXbvvfce0H3ueVkjkpssoeWXXx5I8WJI7dE0l9UkQIWuimVCsp70vJUlpKLsbHZjRzMWu8dfyhhjTGEpjHpV62dXbEdbVENq5CnNUa1V5KtUjjzMmGUk7ai7t1XJajHaEO/QQw8FksylfSoLCZLvV3VK+r2ZWycJWedqeaSGm3379gWqtUFZ4LKWVLcleSmTKYvm5Lhx44A0r1UDln1/dyB7j+q6pZVrvqn5qWKikDbV1N+gu1BbsyYrXPf63//+98pYxc9Uh3XQQQcBqZFvVvaHHHIIkOa3ttu56aabgJZbqrWX5n16GGOMaQi8EBljjMmVwrrmjj32WAB+8IMfVI6dd955AFx88cVAKtbUe5XS2dJ5ldrdzC6kWSH3ktyaABdccAGQgpBq+6HjEyZMqIyVG0rtf9TeQ117ldYNzSHj7C6hm222GZASYBQEVtf37NjRo0cDqZWPkhdWXXVVAJi4cOIAABQoSURBVF599dXKWO0oKveS5J/daVg0c5JCW1J/NY/lRs7KXrLtTsW/WVT0q1IBuSiV8g5w/vnnA6n0RckJkms28UayVYugq6++Gkhzup5ybvwnhjHGmIamMGqWtCGt4gq09evXrzLmuOOOA2DrrbcG0p4k0qi0ymePffDBBwDccccdQHNrlrNC8lD7Hkia/rvvvgskzUeWpoKVAN/73veAVPimlj9KIGm2JJDsPFGA96STTgJSCrEsGTXohWQl6TVpj2rI+9JLL1XGqlBQFqQ+s7vMUWnUSkSQlZ1NqBGaX9L6lWCjnVshWZLdRX5Cz05Z42rDNWTIEKC6GbSenWparOet5HjLLbdUxt51111A2oG4MywhYYvIGGNMrvToCk22X79+s/2Q2nYp8gErVgSw/vrrA7DGGmsAya/ZUqsVaamyAJRyKO1rVv7ocePGFTpXdoEFFmj1H01ajNqfZAsApV2qjYo0UhUGKi4EKQZ05ZVXAnDRRRcBSbOalTwnTpxYaHkC9O3bt0qm2euRhl7btqelsbXNemc1tiOxtAkTJhRaprOao9LCFZ9Q7GHEiBEAvPzyyzO8R/et5rF2ad57770rYzpiERV9jvbq1Wu297yehypS3XPPPYHk+YA0l5955pmq96pIVc9JSIXtkn1b5uuUKVPaJE9bRMYYY3KlMM7UWg1SGvhPf/rTypiVVloJSP7hwYMHA2nFln8e4JJLLgFScz/5NbtTYWCW/v37Ay23a19ttdWAJBtZlll/sbbcePzxx4HWbYXQyGSta82d7pqNVW80Z9S+R7EIFfuqMBWSZ0PZXIplaDuIbMF7d4sN1aL5qa0d5LVQFmxLY0Vt4TC0rk1YvWjOp4gxxpiGoTAxopmR/X7yLWvVlo9Zq3k2e6m2xqgtmnszxYhqs42OP/74ymvaUFC+dVmUah6bzfBSA872NDcsuv8dZowRFZ1GjhEJzU1p52o7k81+3XbbbYEUr9BmbMoMy2bYdcTbUfQ52poYUZFwjMgYY0xD4YXIGGNMrhTeNZelNk1b6YotUZuc0BazvZlcc0JuzWxn4tn97bMBzY4kJRTd7QF2zdWb9riPZ3U/11Lv5KOiz1G75owxxphOpKHyHWutm2ZNHe4Mmj3d2jQutXtbme6Hn0rGGGNypUtiRMYYY8zMsEVkjDEmV7wQGWOMyRUvRMYYY3LFC5Exxphc8UJkjDEmV7wQGWOMyRUvRMYYY3LFC5Exxphc8UJkjDEmV7wQGWOMyRUvRMYYY3LFC5Exxphc8UJkjDEmV7wQGWOMyRUvRMYYY3LFC5Exxphc8UJkjDEmV7wQGWOMyRUvRMYYY3LFC5Exxphc8UJkjDEmV7wQGWOMyRUvRMYYY3LFC5Exxphc8UJkjDEmV7wQGWOMyZW5OuvEIYQBwMgYY+82vm86sGiMcXQb3nNF+bOGtelLVp9jf+AXwHRgKnBkjPGp9p6v3lie9aUB5bkbcBowDRgLHBJjfKO95+sMLNP60oDybPc9b4sICCEE4Bxghxjj2sCvgFvy/VaNi+VZX0II8wPXALuX5Xk7cGG+36qxsUzrS0fv+U6ziGZFCGEgcDHQB1gSeA7YO8b4eXnIr0MI61NaKE+KMd5Rft9BwGHl42OAw2OMr8zicy4ENq85/EWMccPaY8DBMcYPy78/BSwRQpgnxvhle6+zq7A860sB5Tkn0APoW/69N/A5DYRlWl8KKM8O3fO5LETAIcCVMcZrQghzA08D3wVuLr/+ZoxxSAhhdeChEMIqwGrAAcBmMcapIYTtgOHAqjP7kBjjka35MjHGt4C3AEIIPYDzgNsa4aFZxvKsL0WT5+QQwlDgsRDCGEoP0U3ae3E5YZnWl6LJ8y06cM/ntRAdB2wbQjgWGAj0p6SRiD8BxBhHhhBeAgYBmwIrUZo4GtcvhLDQzD6kDau5xvcCrgCWAXZo4zXlieVZXwolzxDCGsAvgdVijG+EEI4Ebg4hrB1jnN7ei+xiLNP6Uih5Zsa3657PayG6rvzZNwB3AstSMpPFN5n/zwF8RUljuTrGeBxACGEOSsIfN7MPae1qXj7fspT8xC8DW8UYP2vtewuA5VlfiibP7YFHM4H0i4HzgYWBVgekc8YyrS9Fk2eH7vm8khW2B06PMV5f/n1DSkISPwIIIaxLaQV/Argb2DeEsGR5zFDgvnp8mRBCH+BB4JYY4z4N9tAEy7PeFEqewDPAFiGExcu/7wqMaktWVAGwTOtLoeTZ0Xu+sy2iXiGEyTXHBgEnAsNDCFOACcBDlIQlVgghPEspDXCfGONY4J4Qwm+Be0MI04CJlDJepmfMzPZyOLAcsFsopXSKbWKMYzp68jpiedaXhpBnjPH+EMI5wIMhhC8ppRp/v0Mn7Tws0/rSEPKkg/d8j+nTG8EdaowxpllxHZExxphc8UJkjDEmV7wQGWOMyRUvRMYYY3KlS+qIFlxwwYbKiBg/fnyP2Y/Kjz59+jSUPCdNmlRoeQIstNBCDSXTsWPHFlqmPXv2bCh5Tp06tdDy7Nu3b0PJc8KECW2Spy0iY4wxuZJXZwVjugXTpk2r+tmjx4yK4pxzzjnDMdMyKjf55ptvqn7PMsccJf3acp09mpeSp2SX/X9Lc7be2CIyxhiTK4WziLT6fv3110BasSFpOLUrtYtyu44vvyw109XfZZ555gGqtU//PZKGudpqqwEwZMgQAJZZZpnKmPPPPx+ARx99FOgazbNR0bybd955Aejfvz8A888//wxjx48fD8DYsWO76Ns1HpqfvXr1AmCxxRYDYOLEiZUxY8aUGiJoXmatpXpji8gYY0yueCEyxhiTK4VzzclkHDBgAAALLLBA5bV33nkHgAkTJlSNnXvuuYFq14bdQ21HMpPbTe5RSDJef/31geQa+de//gWkvwnAXHPNVXW+7oSuWXNx2WWXBZLLQ/ICOOOMMwA49NBDAXjxxReBJGsDX331FQDrrbceAD/4wQ8A+P73S/1JJV+AceNKuxmcfvrpAPzpT38C0t/Crs/0zFx00UWBNAd33XVXIM1BgEsuuQSAW2+9FUjPg86Qoy0iY4wxuVIYi0gr9RJLLAHAb37zGwBWXTXtYvvhh6Xt0F966SUA7rjjDgAee+wxIGlPkAJr3VErby3ScCSrnj17ArD00ksD8J3vfKcydtNNNwVgnXXWAWChhUqbOt50000AHH/88ZWxkyZNqjpvs5O1HJW8IavyzjvvBGDEiBFACg4DrLnmmgCstdZaAIwcObLzv2wDoGcBJMv7rLPOAmDjjTcG4MEHHwTgD3/4Q2XsM888A8Arr7zSFV+zYcgmfPXp0weAo48+GoA99tgDSPfst771rcpYyfzdd98F4JFHHgHSHK8n3eNJYYwxprAUxiISsmA+//xzoHr11Wq90UYbAbDzzjsDcOyxxwIwfPjwylj72WfPwIEDAdhzzz2B5CeW313pmwCvvvoqkDT8HXfcEYDtt98eSBYspHhRd7GI9tprr8r/ZUW++eabANx9990APPvss0BKQ86OGTRoEAA33nhj53/ZBiDr2dA9rzT4Cy64AIAzzzwTgKlTp1bGZi0pSLGMztDgG4msxb7SSqW983Svy3o87LDDAFhqqaUqY1VeMHjwYACefPJJYMY4aD3oHk8KY4wxhaUwFpEyrT755BMATjjhBKC6AFBxiaFDhwKw5ZZbAvDtb38bgNtvv70ytjNW7UZG2uLCCy9cOXbRRRcBsMEGGwCpsPIvf/kLkDQggOeffx5I2tWGG24IJG0zq+k3u8wlS1mUJ554YuU1WUCvvfYakAouJRNZ+gCffvopkGJyZkZkVUvmo0aNAlIWYraQWrJed911gWSt///2ziXExjeO45//TqIZUrIhJBZEyRJho5SFjUvKLTbEAlm4TsnGfWGDBYXYWCJSspJL2RJFIUouuWX3X/z7vr/nvPOOMf+ZM8855/1+Nkcz7xwzz3ne9/l9f1fFlJX9BfV6PqQxIilMxeJv374NwIMHDwCYMGFCce3bt2+B2OfyMqX3+lBhRWSMMSYrLaOIZKHIwnn//j0QpzLAr1+/AJgxYwYQiqjsGza9kWWZ+t8vX74MRN2FsraqWqPIcty5cycQltX+/fsB+PDhQ3Ftpzeb1BoqVpnGKQ4ePAiExa6WNFqTdG1GjBgBwOvXr4HYx52+fv2RxhaVKausLmXRyoOiZwFEJtiiRYuAyKp98uQJ0KgM6hC/1DM1jZFNmTKl4RrVDWk90qw5ZXMqM/H379+A64iMMcZ0ID6IjDHGZKVlXHNCclLSO009lCtE6YRy3ynIXtWSxvyH1jV1u509exboneaq13Q9le65e/duAO7fvw/AlStXmvlrtyRaL7nStFchioIV0C0H28eNG1dcu3jxYiBa0Wi97Zrr7ZrTvl2+fDkQbk21+oFwh54+fRqIFjV1dXlWTSdQYbXWVckKKrRet25dca1cznfu3AFif7qg1RhjTMfRcoqoPI9IFiZEc0gVZV24cAGAGzdu9Pk+5VdZRZ3a+qdsrWsdlYKpQlSA8ePHA/Dx40cgCoLV0kPJIABHjx4FoghTykg/m1pJnbq2QmpbluKmTZuK723fvh2AQ4cOAb0ntEpZAkydOhWAV69eAfVIJf4TVSnVCxYsACKtWHNzVIR98eLF4lqVHTx79gyoThCpI6nCfPr0KQCPHj0C4jmxcuVKAJYtW1Zce+nSJSBSu1PlP+S/Y9Pe2RhjjPkLsiuish9Tfkn5gDdv3lxcK2tSqZwqDlQjxDQupHTiHz9+APD9+3cgigjTa2XNt2tKZ2pBSgHJ5yvLcc+ePQBMmjSp17X6eU0RldUkix0ixXjbtm1AFAnqc+p0FZQiC1t+9r179xbfO3PmDBDxinPnzgFhcWr9IGIYsuDrHtdUWvzWrVuLr2lMgfaZCt6lyKvaIlWNhakzaaxXaI26u7sBWL16NdDYlFeqSc/kqmm4Q0V7PnmNMcZ0DNkVUbmQVYOvlFGUxinKikXZc3pNC1vld9f8eikhxT/UMh5ijITiHe1G+nfLwlGR6saNG4Fouik/OsDLly+BKHLbtWsXEMozbeWxb98+IJSQVGSdlJAoFwoqgxCiNZWawC5ZsgSArq4uICx8iAymOrWbqUKFkgsXLgRC7UB4NlR4qeeBxhmkRapSTXWnPNhy+vTpxfe0x968eQPA+vXrgRjvorg7xBBHFWU3EysiY4wxWcmuiGR1Kz5x4MABIEaEK7YDYRUpRiTGjBkDNDb01NfU1lzWkrLwUr/piRMnGv7vdqFccwWwY8cOAFasWAHEiIyrV68CoQwhWutrOJY+C1nqacNZWamq4zh27BgQTT6bmVHTqmgPKZMLYgSGXufPnw/AtWvXgFhbCMu1XWOTg6U8tloteqQeIRTmrVu3AHj8+DEQVn5VtmZdlaX+/lGjRgHRHFoeo/QaxTAnT54MxPqmz0BdMxyxy3reAcYYY1oGH0TGGGOykt2fInmuIKRcapLgaQsZpWqmk0MhXCPpLA1JUM281ywjue/ULghCgrabpJdrR64NCJecitGOHz8ORHBX3bIBtmzZAoQbScWYmtuSuuZWrVoFROBSn08diwXlktMcrHQ6rVyUKg5WcohcpnPmzCmu1RrW1TWndO2ZM2cCkdghFzyEK1MJDVozPSdSt5Her93u48GiZ52eB2vXrgUakz6ErtGzUq5N7ds0+Wg47+163gHGGGNahuyKSCexLB+lUJ8/fx5oTLOWNV5OJ/z8+XPDz0JYRZosqpO+alKp3q/d0j/LwUmAadOmAVGUumHDBgCWLl0KhGICuHfvHhCp2ZrbonVQiifAkSNHgN4NUutkfWrvTJw4EYgUeZUIQASIVT4wd+5cIFLjU5Up676uaO9I7ahAPd1TssqlGlVw+fz584afgXomzKTonlQavNYxnUGmNdKaa+KtptkqcQkilXs4FLsVkTHGmKxkNyF0il+/fr3hVX74tK1EVVvz9OtVFlHZz6lr1fon/Xe7xTtkqaTTUaUk1dpHTU6lGqVsICYvyqIvK8LUEmo3tdgM5F9XI85Zs2YB0NPTU1yjCZdKc1+zZg0QbXwePnxYXFvX2JBQfEdeC6UQq90MRMskKUwp/lOnTgGNz4I6qfMqyl4fPTvTFPd3794BcPLkSQDu3r0LxHNDih7g5s2bQHiamrlf630nGGOMyc4/w9Gipbu7u636wHz58qWlTavRo0c3rGfa4kfFgCqkVAHvixcvgMaYmyzIZlvm3759a+n1BBg7dmy/e1SKSI1j1YYqtcR1jaxQxTBU+KthjjA4Bf7p06eWXtORI0f+9T0v78e8efOAxqan2seKrR0+fBiIsSSKdcDg9vHPnz9bej27urr6XU89B9SuR0MD0xiR2vakbakgmvPOnj27+JriRWqLNpD9+vXr1wGtpxWRMcaYrFgRVdBuiii1yGUVqV1PualsM8b89kenKKLy0MaqLK++UGZmGscczL3XSYpIyHJPRxEou1UxSlnn5VHsg6UTFJHQM6Bqf0mxl0dlpG2qhGJDaWPZv8WKyBhjTFvhg8gYY0xWsqdvm8GTSnC5KpxuPfSU3ZypC2mg72F6I3dRWqQqV1x57eue+v4ntEZVe62vol9NG65q8TMcafH+NI0xxmTFisgY01KkasfK5/8zECWTu5jfn7IxxpisDEv6tjHGGNMXVkTGGGOy4oPIGGNMVnwQGWOMyYoPImOMMVnxQWSMMSYrPoiMMcZkxQeRMcaYrPggMsYYkxUfRMYYY7Lig8gYY0xWfBAZY4zJig8iY4wxWfFBZIwxJis+iIwxxmTFB5Exxpis+CAyxhiTFR9ExhhjsuKDyBhjTFZ8EBljjMmKDyJjjDFZ8UFkjDEmKz6IjDHGZOVfCRTfS0y+zoMAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10acf5518>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ROW = 5\n",
    "COLUMN = 5\n",
    "plt.figure(figsize = [6,6])\n",
    "for i in range(ROW * COLUMN):\n",
    "    idx = np.random.randint(0, m)\n",
    "    image = X[idx ,:].reshape([20,20]).T\n",
    "    plt.subplot(ROW, COLUMN, i+1)          \n",
    "    plt.imshow(image, cmap='gray')  \n",
    "    plt.title(\"Label = {}\".format(y[idx][0]))\n",
    "    plt.axis('off')  \n",
    "plt.tight_layout()   \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining images into a single row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAAuCAYAAADTNll1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAGsZJREFUeJztnHl4TFn+/9+36taWqkpFoojYojSJ2JuxJY1MWyY8GJ0mjH2IpoklaDJE65nB4JGW0RJkEEswaKKJToamJZhIiG0GMQkhiWwlyJ6qe+/5/eFX90enNj3d7ffN97yepx6SOuee5Z7zPp/zOZ8ThhACCoVCoTQuJO+6AhQKhUL56aHiTqFQKI0QKu4UCoXSCKHiTqFQKI0QKu4UCoXSCKHiTqFQKI0QKu4UCoXSCKHiTqFQKI0Q9pcoRKfT0ZtSjQSO42AymaBWq/FzXIBjGAY8z6Ourg4KhQJSqdSpfIQQmEwmKJXKn6Ve7wKGYdC1a1fcunULDMO86+pQ3gEvX7780S/+nVnuDMNAKpVCrVajVatWEATB7qTkOA41NTXiz47S/zcQQlBXV/ejJ5S7uzvatWv3o8u2/Ovu7u5U+traWlRXV8NkMtmtM8dxYBgG3t7e8Pb2RuvWrd+qDzmOQ0pKCjIyMmA2m+2mZRhGLE+v18Pd3R0ajQaCINito9lsxqBBg1BWVoasrCz4+/tDLpfbLUsQBKxcuRJ37txBfX290+15Wzp27IhJkyZh6dKlGDhwoMM++CEmkwkuLi5o166dU+92wIAB2Ldv349q09vmMZvNqKmpQW1t7VuNCYZhQAhBfX39fzVnHCEIAjiOw5dffom9e/eC4zin8plMJjH/27RrzJgxmDhx4ltrDCEEPM/D3d39nS/Iv4jl/kMIIZg3bx48PT3RqlUr9OrVC+vWrcPRo0fB87zVPL6+vujduzcOHz4MANBqtTCbzeLL+ylxcXHBiBEjcObMGXERqq+vB8s67i5CCOLj4/H+++/D09PTqTyv55VIJKivr0ezZs0wYcIE/PWvf7X5DJ7noVAo8NFHH0Gr1SI7OxtXrlyxae127NgRM2bMwLRp08AwDOrq6mAwGCCRSBwORI7j0LNnT7z33nv44osvHLbLZDKhffv2mDZtGoYMGQKO42A0GjFjxgxUV1fbLE8qleL+/fuIiIiAr68vtm/fjpiYGMTGxlpNTwiBQqHA7NmzcePGjbfqb0t+C/b6gGEYHDhwAHq9HhKJBDU1NRgxYgRyc3MhkTi2kQgh8PX1RVhYGMaNG4dbt25h+PDhkEqlNsv19vaGUql06vk/5Ne//jUyMjIcLviWtvn5+aFPnz548eIFvvvuO9TV1YEQ4jAvz/OQyWTw9/eHQqHA+fPnHRoYPM+DYRhIJBKn35dKpUJwcDCCg4NhMpkgCILDPBzHoXfv3qioqIC/vz8SEhKcKosQgvnz5+Px48c4fvy40wulIAhQqVQICwvDsGHDMHbsWFRVVTXoD1sLxk+9GLwTy53neaxYsQLTp09HfX09kpKSEBkZiWXLllltICEEZ8+eFb9TqVS4efMmQkJCnF5ZmzZtin79+mHEiBE2FxALv/vd7zBgwACxvDVr1mDmzJlOlcNxHHx8fFBXV+dUegBQKBSYOXMm0tLS8Pnnn+PZs2f485//jOjoaKuDn+d5dOvWDenp6bh37x7Wr1+P5cuXY9iwYVYXO0EQ4OrqivPnz8Pd3R0GgwHe3t7o0aMHYmNjIZPJ7NZPEARs3rwZiYmJ6N+/P/bu3Ws3PcMwOHfuHPbv34+vv/4agwcPxq9+9SssXrwYXbt2tWvxMgyDsrIyHDx4EJGRkYiKisKyZctsWmqCIMDNzQ3FxcVYvny5QyEkhECpVGLVqlVYsWIFUlNTkZqaijt37iA4ONiuRdivXz+0adMG3t7e2LJlC86dO4fQ0FC75QGAXC5HZGQkEhIScO7cORQUFKBjx45IS0uDl5eX1TpqNBqsX78eJSUlTrumLHh7e2Pz5s3w8PCwK4KCIMDf3x+PHz9GTEwMiouL0adPHyxZsgSff/45Ll++jKCgIKvzxSL8Z8+eRXZ2NgIDAxEZGWm1PIZh0KRJE6xatQqPHj1CeXk5nj59iuzsbAQFBb2xI7eW12QyITs7G3PmzMHOnTsxf/58KBQKu33g5+eH69evY8qUKRg5ciTc3NwQHx/vcO7X19fDx8cH//rXvzB79myn5zHHcdi3bx9yc3Mxd+5c+Pr6on379m+URwiBSqXCsWPHkJmZiUuXLuHKlStITExEamoqAgMDf9Kd5zux3KVSKZYuXQqWZXH48GG8fPkSbdq0QadOncDz/BsT1GKZJScn4+jRozCbzWjTpg2uXLmCv/3tbw5fMvCq43fv3o26ujrodDpcvHgRlZWVNoVAr9ejurpaLP/999/HvXv3nGqbxfp2dhUmhCAkJARLly7FyJEj0bp1a3h6eiIyMtKm6E6ePBnh4eFwc3MTJ5NEIoFGo4FOp7Mq8DzP46uvvkJsbCzMZjPMZjM8PDzQuXNnqFQqVFZW2qxjTU0NJk+ejKtXr8JoNDq0tnieh1qtxvbt23H79m0IgoBp06bh5MmTePbsmUM3iwWGYXD9+nUAAMuyVi1JQRDQtWtX/Pvf/8aDBw8cWpscx8HT0xNz5syByWQCIUTcSk+cOBFnz55FRUVFg3yWdFKpVBxPERER6N+/P+Li4uwuKhKJBC1btsQ//vEPpKamonnz5uA4DtOmTUNpaWmD9IIgoEWLFhAEAeXl5W9tuXt4eMDd3d3hrrZ58+ZYu3Yttm/fjri4OOTn5+PChQuIjY1FUFAQMjIy4OrqarU/eZ7HoEGDYDQasW3bNhw6dAjBwcEQBKHBYiQIAv7whz9g3LhxuHDhAq5evYoBAwZApVLhs88+AwB8++23VuvIcRw6duyIuro6bNmyBd9//71D1xEhBJMmTULbtm2xaNEiuLm5QaFQYOjQoTbzCYIAQRAwfvx4zJ8/H1OmTIHZbHZqYeU4Ds2aNcPgwYPx+PFjVFdXIysrC7Nnz8b8+fPfSGsymfD8+XPo9Xrs3r1bdGt5eXkhMDAQ3333ncPynOWdiLtEIsGePXsAvLLCtVotSkpK0KtXLwiC8MZgtkze1NRUVFRUQCaTQSKRWB1EtpBKpbh48SL279+P0NBQ6HQ6vHz50uqkEQQB3t7e+Oabb8TfqdVqcdFxtB20TARndxQ8z2PevHk4deoUpFIpvvjiC3z00UcoLS1tIO4Wq3P16tVgWVY83LQscL/5zW/g5eWFSZMmvdE2ixth06ZNol/Uz88PERERSE9PR1VVld36vffee6ipqUFCQgI4joNCobDbPolEAo7j4OfnB4Zh0KNHD3Tv3h1HjhwRt+TO0rt3b3ECWEOj0WDq1Kk4ceIEampq4OrqCpPJZFMQLWcBFuF7/vw5WJaFSqVCp06d4OnpaVXcX4dlWdTX1yMnJwd9+vRpYJBYQxAE+Pn5ISYmBiaTCQcPHkRubq7VBZzjOBgMBvA8jydPnoDjOHAcB5ZlHe6yLEilUptj3IJGo0HLli2Rl5eHmpoaGAwGrFixAkOGDMG6detw4MABVFVV2XyGwWBAVFQUUlNT4erqitu3b2PYsGE4e/bsGwaAyWTC6NGjUVhYiE8++QSVlZU4cOAACCFIS0vD2rVrcfr0aavz2Ww2o3v37rhy5QqSkpJQW1sLhmHsjiHLHC4qKgIhBFVVVXjx4gVSUlKg0WjEZ7xOSEgIgoKC4OPjg+TkZOTl5UEulzfQox/CMAx0Oh02b96MmzdvYseOHXjx4gXu3r2Lw4cPo2nTpjAajWJanucRFhYGV1dX5OXlib9r164dFi5c+JO6Zt6JuAuCIAqSxXry8fFBaWmp1cbV1tZCJpM1mOA8z4uWsi2htxzEVFZWIjk5GWfOnEFubi68vLywYMECrF69+o28giCgXbt2CA4ORkFBAYYMGQJPT0+EhobizJkzqKiosPkCBEFAs2bNwLKsw+2fBZZlIZVK8ejRIyxatAijR48WF7EfQgiBWq2GTCYTD5eqq6uxZs0a8DyPFy9eID4+3upg5DgO165dg1qthkajQXV1NU6fPo1ly5bZtcR79uyJ3bt3Izg4WBy0LMti27ZtiI6OtmmFz5o1C9HR0SgsLBT71NqOhmEYccHkOE70QQuCALlcjoiICDx48AA8z1ut5+TJkxEQEIC+ffti06ZN0Ol0qK2txcCBA5GXl9egL1iWRUFBAby9vUUrPDo6Gh9//DEiIyNx//59h0Jtqef+/fvxpz/9yW7/EULg6ekJNzc39OrVC2lpafD29oZcLrcp1J06dcLSpUtRU1ODBw8e4OLFi2jbti1KSkpw8OBBbNu2zaEIVFRUoLq6Gkql0maanJwcdOvWDVFRUYiIiBAjoGbOnInvv//e7vNZlkV8fDzOnj0rzpOcnBwEBgY26A+ZTIakpCSMHDkSYWFhuHz5MjIzMyGXy9G7d29069YNer0eRqOxQd/7+/tjw4YN8PHxgUQiQa9evVBZWYlHjx6hrq7Oah/yPA9fX1/cunVLHJ88z6OoqAjjxo3Drl27GozbvXv3oqKiAvn5+WjZsiUSExPBsiwUCgUmT54Mo9EIuVzeQINMJhNycnKQl5eHgQMHinPTy8sLUqnU6liqr69HWVkZ1Go1gFf6Nnr0aPGMxBlvhDP8oj53y/ZXJpOhefPm0Ov1CAgIQIcOHWA2mxEfH99gYEgkEjx69AheXl6Qy+Wi1cUwDLRaLTp27IgWLVpYLc8SGte5c2esXLlSXE0//fRT7Nu3T9zGvw7LsoiJiYGXlxdiY2PRt29fGI1GTJ8+3a6wA//P4nK02r8Oz/MoKSnBqlWrcPz4cZSVldntv7Zt24quqsTERMTHx+PZs2dgGAYXLlzAqVOnbOYvKSmBQqFAbW0tpFIp4uPj7VqChBCMHTsWLVq0QFZWFn7/+9/j0qVLGD9+PPR6vU1rmhACDw8PSCQScacwZ84cqFSqBmktIv7ZZ5/B398fWq0WMpkMCoUCoaGhqKurQ2xsrN1DZQDIyspCUlISEhISkJycjOXLl9sVXcsuzOKTNZvNOHDggM30wKtJWV9fj+bNm8NsNjuM/LG4cXbs2IH+/fsDeCXc9g5RAUCpVEKr1UIulyMsLAwGgwGEELRu3RqTJk1yGCliMBiccn2xLIvq6mqEh4cjOTlZXEBHjx7tcOdpsThLSkowdOhQbNu2DcHBwTbzJScno6amBgsWLMCePXuQkJCAmzdvIisrC7t27UJQUJBV/7ZWq4VEIoHBYMBXX32FuLg4JCYmYv369ejVq5fN+gmCgH/+85/iz2q1Gn379sXTp0+tzk2VSoWUlBSsX78emzZtQlRUFA4ePIijR48iLi4OU6dObbBrJ4RAr9eD4zjExMSIc4lhGIwYMQKVlZVO+ezd3d3h7++Py5cvO70zcwqLL/Hn/Li6upImTZqQsLAwkpaWRk6cOEECAwPJtWvXSGlpKSktLSWzZs0iGo2GuLq6NvhotVqSm5tLnjx5QnJyckhubi7Jy8sjCQkJJCQkhHh7e7+RXq1Wkx07dpD8/Hzy5MkTsnPnTjJ27FgydepUMmTIEOLj40N0Op3Vsl5/hkwmI3q9nly8eJG4uLjYTe/q6kpYliUTJ04kpaWlZPHixQ7Tq9Vq0rp1a7J161bSpEkTkpGRQdLS0mymnz9/PqmsrCQnT54knp6exNXVlchkMpKfn09KSkrIvHnziFKpbJBPo9EQNzc34uLiQhQKBZFKpWTChAkkPz+fqFQqotVqrZanUqnIjRs3SHp6OvnLX/5CysrKiK+vLzl69Cjp3Lmz1ffl4uJCBg8eTJKTk0lAQABRqVSkWbNm5Nq1a+TBgwcN8rRt25akpqaS0NBQsmnTJvL8+XNSXl5OKioqSEVFBenevbvdvl+4cCF5/vw5CQgIIDKZjKjVajJ58mQydOhQm+PJ8lEoFMRoNJJnz56RqKgou2l1Oh0pLCwk5eXlpLKykhiNRlJSUkKKi4vJ9OnTiZubW4NnX7x4keTl5ZGJEycSuVxO+vXrR5KTk0l6ejrp0qWLzbJ8fX1JamoqKSoqIkVFRaS4uJgUFRWR0tJS8vz5c6LVam2+M7VaTdLT00laWprD9lvGxtGjR4nRaCQMw5CRI0eSwsJC4uHhYbMMSzm9e/cm+fn5JCUlhXh7exO1Wm01rcFgEPvEYDCQQYMGkS5duhAPDw9SVlZGysvLbZYzaNAgkpeXR4xGI9m9ezeRyWRELpeTXbt2kYqKCtKmTZsG5SqVSlJeXk4mTJhAdDod0Wg0JCIiglRUVDjsk8DAQKLT6UR9sPz//PnzohZZ0rq7u5MjR46QiIgIolKpxH5Zu3YtefnyJXFzc7Pbh5bP+PHjSX5+vtVx/t/o7i9qubv+Xz9Tamoq/Pz8oFAoUFxcDJlMBr1eb9efPWfOHBw+fBixsbHYuHEj7ty5g9DQUJw6dUr0aVmoq6vD7NmzIZVKMXfuXCxcuBApKSk4duwYrly5goKCAoe+c6lUCoVCAYZhcOPGDafap1Kp0LFjRwCwa4G/nj4sLAxffvklamtr0bRpU+Tm5tpMTwiBIAi4ceMGysrKUFNTg65du0KpVKKqqgp79+61arF169YNoaGhYptcXFxw7NgxHDlyBAEBATYtQUvUBsuy6N+/PwRBQHx8PLKysqy6PIBXkSELFizAJ598gszMTLAsi8rKSmzdutWqtcqyLHQ6HYxGI65cuQKTySRudQ8dOoSysjK7u6Dr16/jP//5DxYsWIAxY8Zg1apVWLduHTIyMuxaxzzPIzAwEBKJBHfv3sWuXbtsprVY4EqlEgUFBTh37hwqKyvFcL5Vq1ahZ8+eYnpBEODu7o4ePXrg+PHj+Prrr8GyLK5du4ZDhw5BrVbbdZcUFBRgz549YFlWdNvJZDJUVVXh6tWrdg+NJRIJCgoKnD7zYRgGH374Ib799lu4uLjgwoULyMvLQ4sWLey6FgkhGDVqFM6fP4+EhAR4eHjYTGuZCwqFAiUlJcjMzER2djb0ej1kMhlSU1NtlvX48WPcuXMHwKs5aflcvHgR9fX1Vi1dhmHw5MkT+Pj4iIein376KcrKyhzuejIzM9GkSRMxLt5sNuO3v/0t2rdvL97bsODu7o4WLVrg6dOnYli2Wq3GnDlznA5DBYCJEyfizp07Pyrk1R7ML3GbT6fTEYlEgubNm2P37t1o0qQJVCoVpk2bhszMTADA9OnTcfLkSbuHexZqa2uRlZWFMWPGNBB2AGIMN8MwTkdmWEMQBKjVasTExGDy5MkOt0wDBw7EunXrcO/ePYSHh9uNQCGE4PTp02jXrh2qqqpACMEHH3yA+vp6m+cHSqUSy5Ytw4wZM8QDZUIINm7ciJ07d9pcsCZNmoTPP/8cu3fvRkVFBYYPH45mzZrB09MTw4YNw/37922W2bdvX0yfPh1yuRwSiQSLFy+2G8EhlUoRFRWFBw8eoLy8HBMmTICnpycePnyIDRs24ObNm2+k5zgObdq0werVq6FSqbBt2zYYjUZkZ2eDZVmHk8Ny5rJ161ZotVocO3YMt27dwpMnT+zmGz16NDZs2ICTJ09i5cqVMJvNdicXz/OIi4vD8OHDoVQqkZOTg0mTJiE/Px9SqRRms1msqyX09ObNm+J3wKv7E4QQrF69GocPH0Ztba3ddikUijcOa6urq0Vxs5dvx44daNWqFUaMGGG3TWazGR06dMClS5eQlpaGv//971iwYAEMBoNVMXsdjuMQHh4OPz8/ZGZmoqCgAElJSU7fBUlISEBAQACGDBmCe/fu2Z1bPM8jJiYGXbp0gZeXF6qqqnD79m0cPXoUp0+fttrGoKAgbNy4ESaTCXq9HnFxcdiyZYs412zh6emJuLg4PH36VHQPrlmzBvfv3xcjtl6v16hRo7B9+3ZUVlZCq9XCZDJh8ODBePjwoVPaIwgCUlNTMWvWLOTk5DT4/r+5ofqLiTvLsggJCUF4eDjS09ORlJSE8+fPiz4ps9nstL+prq4O2dnZmDVrlrg4/BxY+iY9PR2BgYEO/Wfh4eFYvHgxPvzwQ9y7d8/hJIyJicEHH3yAgwcP4uTJk7h7967DKACdTofExEQYDAZUVFTgm2++werVq+1GbKjVasydOxeLFy8Wt2yPHj1CSkoK/vjHP9qtJ8/zb4TUKZVKhxZx9+7dceLECbi4uKC4uBjnzp3Dhg0bUFxc3GDAW3y3lvheuVwuWsRvg0UoLZauo/zR0dEYNWoUPv74Y2RkZDglSm5ubhg8eDDUajVu3LiB27dv2wzRBIApU6bA39//je/u3r2LjRs3OnVo9sO56YwVKAgCEhMT4eLigmHDhjkV979jxw707dsXer0eAHDixAnMmzfP4WLn7e2N9PR0sCyL7t27o7Cw0KkINpPJhKKiIty+fRvDhw93SgQJIejQoQMMBgOMRiNu3LgBs9ls871pNBosWrQIs2fPRkZGBsaNG+d0VJOnpye6du2K58+f48GDB3j58qXNdpnNZhw5cgQ+Pj44deqUuOg4M54EQYBGo8HBgwcxYcIEq/ryP0LcLYdsbm5uqKqqQkFBAWQy2Y8K/SGEIDo6GqdOnfpJ40KtYTKZcPjwYSxZssShq8XFxQXNmjXDw4cPnRInpVIJd3d3FBYWguM4pxY3QRDg6+sLLy8vVFZWilfuHS0KUqkUbdu2FQ97a2pqUFZW9tbXq53BcvhnCRksKysTb/j+EuPNmfqlpKTAx8cH3bp1c3hQbkEQBNTX14MQAplMZjV64nU4joNGo3nj2VVVVW99KeltCQkJQefOnREZGelUu1iWRUBAAEJDQ5GQkIC0tDSHFi7wqn2nT5+GSqVCYGCg0yItl8vx6NEjLF++3Olbo5byOI6DRCJxqB2CIEAmk6F169YoLy/Hy5cvneoLS6isZdfCsqzDuazRaKDValFUVCRGuzgzzi1hnkuWLMHChQutei3+R4j7z14IhfIWKBQKdO7cGdevX3/nfwPkfxOWMNe6ujqHu8D/DUgkEnTp0gX379+3euns/3txp1AoFMovC/177hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojRAq7hQKhdIIoeJOoVAojZD/A6pE6bgQFdX0AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x10b69e2b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sample = np.random.choice(m, 25)\n",
    "plt.imshow(X[sample].reshape(-1,20).T, cmap = 'gray')\n",
    "plt.axis('off')  \n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining images into a single large array and then displaying "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-0.5, 99.5, 99.5, -0.5)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWYAAAFmCAYAAABeJjAWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJztnXnczXX6/6/mO43sbktuW3bKkrKVkBFJk0STNk2ylLRNkymTtEml1MhE0kPfxiBKUZaIopWEyE5kyx43sqRm5vv7q+v3uq45nzO321ne5/Z6/vW6nffnnM923s71+lzX9T7t//7v/4QQQkg4/CrdO0AIIcTCiZkQQgKDEzMhhAQGJ2ZCCAkMTsyEEBIYnJgJISQwODETQkhgcGImhJDA+HUqPiQrK4tVLIQQ4sjJyTkt1r/zFzMhhARGSn4xk8wCy/T/9a9/qf6f//kfM+6002L+Zx8c8doO/Pvf/87VNvj3r3/9/782mXIOyH/nn//8p+p0X2/+YiaEkMDgxEwIIYHBiZkQQgKDHvMpivdWf/75Z9Wnn3666goVKqjev39/5DahgX4hHutvfvMbM65EiRKq0T8sWrSoGYc+486dO1UfP3485vYh4H3yqHOCzxHi+fH4jKFAgQLmtdCOPQo8Pn+s2dnZqgsXLqx69+7dqo8dO2a2SdZx8xczIYQEBidmQggJjIywMpK5ykqmhGB5Bc8dhrLFihUz4+rXr6+6WbNmqq+55hrVXbt2Ndts2rRJtU+lSwXx7ovGjRurbtGiherSpUubceecc45qtCtq1aplxuH5uvbaa1V//vnnMbdPJWhFoD7jjDPMuNq1a6suVaqU6sqVK6suWLCg2QbP8ZYtW1R/8sknZhzeWyF/p/D8XHjhhea1YcOGqcbz0717d9Uff/yx2cZbY4mCv5gJISQwODETQkhgZISV4UMj/PtXv/pVTI2ZBSL26TmGMz6zIBOruuJlWGBo2rZtW9W33HKL2aZatWqq8en0+++/r3rXrl1mGzzf6QCvo4i1KR5++GHVzZs3V+2v9+HDh1VjOO7B0H3Dhg2q03UO0GJAK+Kyyy5T3bJlS7NN9erVVWMIXqRIEdWYcSIiUq5cOdV4rBj2+799xka6wXOF3+lLLrnEjKtatarqPXv2JH/H4sBfzIQQEhicmAkhJDCCsjIw5MAE/379+plxJUuWVI2h1ubNm1VnZWWZbTBM/fHHH1VPmzbNjPvggw9UY6gcmq2B9oV/kt6mTRvVvXv3Vl2lShXV/ukynhMM2wcNGqTaJ9enOxPDf37fvn1VN2nSRPW6detUP/vss2abZcuWqcbw1WdYoAXy008/qU6VlYGfKWKzaEaPHq0avxtr1qwx27zzzjuqFy9erHrjxo2qvV116623qu7fv7/qGjVqmHFRzaBCA22b1q1bm9dwjsDz/cMPP6hO1fXmL2ZCCAkMTsyEEBIYnJgJISQwgvKYo0D/U8Q2GNm+fbvq+fPnq16xYoXZBqvZ0FvynmOfPn1Uow+brAqfEwH9VdyfgQMHmnFXXnml6hkzZqh++umnVV9wwQVmG0wnfP7551UfPHhQdboq2xD0/i6++GLzWqdOnVTj8wH0xn0aJf6N5+Do0aNmHB57OlLkvIdbpkwZ1Zje+NBDD6mePHmy2ebIkSOq0Z/H98amTiI2xRLPydtvv23G+fMaEugdYxUopoiK2FRBTBNdvXq16lQ9V+EvZkIICQxOzIQQEhjpj00BTEnDFJWnnnrKjMPKIgxZMcz1FV7Lly9XPXHiRNXjx48347Bpz5dffhnz/dJV7YUhZ9myZVX71EBMG8OQ7Oabb445RkRk1KhRqkM4VgT3B/tD//GPfzTjfA/lX6hbt65qX7G2b98+1UuXLlX95JNPmnHffPON6nSkCXqiUjn37t2rGqsaRaz9hdugRdatWzezTaNGjVSPHTtW9cKFC824EM7JL/jmVnisl19+uWp/vwwePFj13//+d9X4veOaf4QQcorCiZkQQgIjKCsDwZDBP5H2T8xjbRMviwIzDXyzkkOHDqlOR4VXPDBcxGwUH35iuP/cc8+pxqyF9evXm2169uypGjMQnnjiCdWpbE6D1xybEz366KOqsTmRiLU80LpCWwyvr4gN1du3b6/aV7Zhtg5WC6YqW8dnxKC1gsc6cuRI1d4CHDdunGr8DnXo0EH1vffea7aZM2eOagz1fSViCBk7v+CbW1WsWFE19t/24H2CGqsFU0X6ZxtCCCEGTsyEEBIY4cQfcYjXjzm3YJI5hq9+eZkRI0aoxoKDQoUKnfBnJhMM9YsXL25ew8ZDderUUX333XerXrJkidkGMxzwnKAt4hvcJDN8xXAUi2HQjsnJyTHbYBg/adIk1Rh2Y5GFiO1lfMcdd6i+4YYbzLgBAwbEHIcrhyfT7vLnGoshcN/Q6sGe1CK2KGXRokWq0a7CEF7E2hd4vkMouIrC99XGJbXOOuss1di8ScQ2fUr38fEXMyGEBAYnZkIICQxOzIQQEhgZ4TEnAkylQl/Re2rYJD7dPlM8sLqpWLFi5jWsZrvttttUY5qXr4766quvVKMXmY4m4f6zvvvuO9UzZ85UPWHCBLPN7Nmz/+t7+ecT2KAGmzxhip6IbXyFDbGmTp0a83OSDTYNwuuK/jd6xyK26X2PHj1UYxrmgw8+aLYJyXfNLb7qFxcPwCrZWbNmmXFff/216nSn//EXMyGEBAYnZkIICYx8a2X4akFsWIJLvGPTF5H0N6tBi8HbDRgq47756sW//OUvMd/jjDPOUO3DvfLly6vGVCyslEtlKIvHt3LlStVYoejB8DO3KZV4TrCact68eWZcq1atYu5bCKCtgffCqlWrzDjsrYznB6+x3ya0tS6jwBS5SpUqmdcuvfRS1Zg62bhxYzMO1xZ95ZVXVGOlMNf8I4SQUxROzIQQEhj51srAqj0RkZYtW6rGJ+yYtSBiQ510LJcTZVeI5L6pUrwshKh/b9iwoerXX389V9ukCjwetGaSuT/etkHLAxscecssHeB9gf3EMUNDxPZTxiwcrO7zPa79e4QEVodiRR8ej4j97uO5KleunBmHlbG4TB1matHKIISQUxROzIQQEhicmAkhJDDylceMnpP3jzAVZvHixTG1SHpSoTB17ZJLLlHduXNnMw7XocNOb/GqlNCTxc/xDcPRk8fqulQ2x0dwv31T9l/wfh/+HeUF+hREfO8SJUqoxg6EIvbcbdq06b9+TrLBex3Tw+666y7V2EBfRGTo0KGqMUUOPfNevXqZbXDxAKxyTNd9geC1wxTYNm3aRI7DtDrfXa5evXqq013lyF/MhBASGJyYCSEkMDLeysB0pYIFC6ru27evGYeVbQMHDlT9448/mnHpsDLwGKpWrar6xhtvNOMwfe/5559XvW3btlx9DjZwwSY2IiLTp0+PuT+pCtV92hnaCth8B8dhRaCIDU2xig/xISqGvR07dlSNVXIi1r7AZjepOj/egkErA5v6o4WH97mIbeqPIf1bb72lunv37mabJk2aqJ48ebLqEKwMvJZYqYnrXIqIXHHFFaqrVKmiGtcCFBE5duyYarR60pEmyl/MhBASGJyYCSEkMDLeysAnrljh06VLFzMOw7AFCxaoTnffVRFrUeB+ov0iItKnTx/VF110kWofth84cEB1dnZ2zM9B60LEPnEPoUkPWhZY1YV2g7cRMBRFiwptAH9suDQ9Vvd99NFHZhxmxGCjq3RUh4rY8BrD8x07dqj+4osvzDYY+teqVUs1fle8RYH2UAj3BYL7s2HDBtVo84nYfsy4/h9anyIi7777rmrs052OzBv+YiaEkMDgxEwIIYGR/jj+JMGQDkN/32f5pZdeUp2bJj+pBPcHnwYPGTIkchyG9BieiYisXbtWNTZj+fzzz1WjnSNin/KnI2T14SKehwEDBqhGiwGtKxGRatWqqa5Tp45qDM997+otW7ao/uyzz1SPGjXKjMPt0l184Fm3bp1qLLR48cUXzTi8xueee65qDPWxD7GIyJtvvqk6hEyMKNCS9AVJuIRUhw4dVPtl5f7xj3/EfC0d15u/mAkhJDA4MRNCSGBwYiaEkMA4zVcUJYOsrKyEfgh6ZRUqVFD92muvqfZL2WM1UGgecxT+2qCPVrp0adU+ZQv9MUydw0ZF3jdLVzOe3ICpc9hMqHDhwmYcpr6hRs/cL6CAaXV4rvAe8+8RAnhvFCpUSDU2yse17kTsvb506VLVWDW3bNkysw2eh5DvESTeWplly5ZVjdWPIiLff/99zPdI5hyRk5MT880z40wTQsgpBCdmQggJjIy0MjAEwbSoP//5z6off/xxsw2mRYVQ7ZcXMKSPt9Ychl6ZYtvkBX8O8F6O0v4c5IdzhefBWzC5AY/VWzaZdB6iwOsf7/zgsafquGllEEJIhsCJmRBCAiMjrQzcZ7QlsIfvwYMHzTb4ND8/hGeEkMyHVgYhhGQInJgJISQwMjI9Aa0IfMq6b9++XG1DyKkOfm9Qh9ag6VSFv5gJISQwODETQkhgcGImhJDAyEiPmSSXqOqo0Br5kBMDG8jjogK40MLcuXPNNlhVGPJzmpCr+/ICfzETQkhgcGImhJDAOOWtDF/5iA2SsFoQx+Ey9yL5I8THkBXXgMNj279/f0r3KdVgqO/DYbzmIYfAiO83fOaZZ6p+7LHHVB85ckT14sWLzTZ4zUO7z/GexQpg7NHux23fvj3y/UI6Pv5iJoSQwODETAghgXFKWhnxGjfVrFlT9fnnn68alyrCpXhERDZv3qw6U3o9+17GxYoVUz1y5EjVGN736NHDbIPhfqaE9x48hoYNG6quVKmSGYfX/OjRo6pDW24J7YusrCzz2jPPPKMal1Hq37+/an9fhBTei9jrhUuL9enTR/Xtt99utsHlw0aPHq16zJgxZhwuLZbu6xrWXUUIIYQTMyGEhEZmxN0JBsO1rl27mtd69eqlukaNGqpx5em1a9eabTZu3Kg6U6wM/8T+d7/7neqWLVuqnjVrVsr2KRV4Gwv/vvPOO1U3aNDAjFuzZo3qb775RnW6Q14Rez9jeD9w4EAzDm2JwYMHq8bso9CsC2+t4PfrjjvuiKnHjRtntsHMkm7duqn2Vg9mqqRqlewo0n9XEUIIMXBiJoSQwODETAghgZEZhmgCQB+tadOmqh944AEzrlSpUqrRWxo2bJjqhQsXmm1Otrl4qvysqLUSRUTq16+v+vTTT1f94Ycfqj5+/LjZBseFDB639yyvvvpq1c2bN1eN1XAi9hqnYp3MEwFTyG677TbV3ifH13bv3q26QIECSdy7EwfPr/fwr7vuOtV4PJgK+Morr5htMF1u7969qgcMGGDGYUrk7NmzVRcsWDDX+54o+IuZEEICgxMzIYQERr61MuKlRbVo0UJ1iRIlzDhMI5s+fbrqN998M/KzTtZ+wLQdrD5KNGjnXH755eY1DOkxHeyLL75QHVoIn1vwmlatWtW8dsstt6jGRkXjx48347Zt26Y6hJRIvJbnnXee6muvvVb1U089ZbbBlL8o+8Jf46h+zMlME8TrhRabiK3qQ3sRK/r8MeCxvvvuu6o7duxoxnXu3Fn1/PnzVeO5TlV6JH8xE0JIYHBiJoSQwEh/TJYkMPwQEbngggtUX3/99ap9pdOhQ4dU/+///q9qfIqd6Ke0PgMgkWBYiBknWOEoYu2UF154QTXaGpmSheHBELxDhw7mtQsvvFD1kiVLVL/88stmHFZ+psPK8OE57sONN96oGjMLpk2bFrkNgnaFD9Wxh/PBgwdVY6ZDvPfOC3is2DxMxF6XN954QzVmpsSzG9AqnDlzpnntoYceUv23v/1N9YYNG3L13omEv5gJISQwODETQkhgcGImhJDAyFceM3pl6KeK2O5T2AQdvUMR23Vr/fr1qpNZHYXN6BNd+YfeG3aQa9WqlRn32WefqcaUIvTUMqkZPt4L2dnZqq+66iozDs/PokWLVPv1DdOdIuefmeD1u+iii1R3795dtfel8fpFdaTD74mITSlbtmyZ6iFDhphxW7ZsUX2yHeqwynLp0qXmNbxG8SoEo8BzcOzYMfMaLhaB3nY60kT5i5kQQgKDEzMhhARGxlsZGGZgk53777/fjGvXrl3M7X3DE0yRw/SwZKbJJNoiwLD3nHPOUX3PPfeo9mHcP/7xD9VY5RZag5vcgqE6psjVrl3bjMP0KWzYFEKVY7ymU3g/f/nll6pzu/4k2mdY8YaVkCK2mQ++X+vWrc241157LfKzTga0mkTsdyW33xs8j/h+mAoqYhscpRv+YiaEkMDgxEwIIYGRkVYGhqn4BBefKPsGJQhWePkQDJ8oh7CeW26I17QFe9ZWr15d9cSJE802GLJmaoUfnge8Lxo1aqTaV22+//77qhcvXqw6tGvv9wezjqIsD7QrRGwYj42P+vTpoxqtPBHb57hWrVqqsUpOROStt95SjZWs6TiP3v7AfShevLjqPXv2mHHffvut6nR/B8K6+wghhHBiJoSQ0MhIKwMb82D/2bvvvlt1oUKFzDZoX+ByUqEVEuQFv+QT9lrGJ+74FBobFYmIHD58WPXJLpWVLqKyUTBs9813Vq1aFfO10IppvCVQtGhR1StXrlSN2Tb+XsZjwmKjTZs2qcbmPSL2nNatW1e17xuOf6cjkyeqWZeISLdu3VSjreWtnrPOOks1fqdYYEIIIYQTMyGEhAYnZkIICYyMMFS9F3T++eer7tu3r2pMhfIpM5gWh54crvOWSeDxYZMeEVvhh+l/UQ2aRDLTV/beH6Y4XXPNNaqrVKmiesGCBWabCRMmxHy/0NLlPLh/6CXjMfjzg82KGjRooBq/G/v27TPb4EICXbt2Vf3qq69G7k+qwLRZXOhh6NChZly9evVUr1ixQnXbtm3NOJw/MN0W10r0c1Gy0urCvvsIIeQUhBMzIYQERrBWBobq1apVM6/deuutqkuUKKEaw6np06ebbbCyLRPDdg+GcT169DCvYfj5wQcfqJ4xY4bqdFc2JQI8ByIiVatWVd2sWTPVGH7iORAR2b59u+qQbS1vzW3dulU19mPGlK+NGzeabUqWLKka+w1j4yNMPxURueuuu1SPHTtW9axZs8y4dNxPmNJ2ww03qPYWxXPPPacaG3TVr1/fjMOe5GiFlS9fXrVfCxJ7VCdyXuEvZkIICQxOzIQQEhhBWRlRFTYYpoiI/P73v1eNYeqOHTtU+5Dj0KFDqjO1xzCGsxUqVFDdpUsXMw7DtRdffFE1VkdlYoXjfwOzBrBhEy5H5K2MkG0trNTzVsbIkSNVDxw4MOa/T5061WyDVa5YOeizGBDM2Jg8ebLqk10+KhGgdYn3PC5zJWKtHlxWbsyYMWbc8OHDVWNl5IABA1SPGjXKbIPNnObOnasaz09eqkj5i5kQQgKDEzMhhATGaalo0JGVlZWrD4lKGB83bpwZh09TcZXre++9V/XMmTPNNiGHrFH4a4Ph7BNPPKEaey6L2B662KwoL8vyhAbaMXXq1DGvvf7666ox6wALbqZNm2a2yVRbK6ppD2ZVNG3a1GyDoT/2TMae1PPmzTPboAV4suF5MsG5Axt3iYjUqFFD9fz582Nq/x54fnE5sscff9xsc+6556ru16+faswKizf35OTkxDyR/MVMCCGBwYmZEEICgxMzIYQERlA5U+jxYBWXr/xDrwsbjGBaVH5IB/MpUg0bNlTdokUL1Z9++qkZN378eNWZ1JgnN+A9Uq5cOfMarueGCyNgdVZ+uC9E7HHk5OSoHjFihGrf9D437+XPT6acL7y3cf1BEXvP4NzhU/6iXsMFJu68806zDT7X+vOf/6x69erVqn36Xm5SDTP/m0oIIfkMTsyEEBIYQaXLRS0/j+lOIrZBDVbeYPgQQmXSyeKvTbFixVRjOuHBgwfNOFx/LbS0ppMlXnOrIUOGqMYUy0mTJqnO1PQ4EgaYRidi7yesKty1a5dqXIdRxH4nmS5HCCEZAidmQggJjKCsDAT3y2cnRGUa5Af7Ih74dBm1tyvy+3n4Bd+PGe+TU+m+IOkj6juZ2ypJWhmEEJIhcGImhJDA4MRMCCGBEWxZD/oymVJ9lGzQN80PVXwniz8HPCck1STrO8k7mRBCAoMTMyGEBAY9gnwOphb++OOPqn0K4hlnnKGa1lH+JyodFdfQFLEVuKfSfXH8+PGY/56qylH+YiaEkMDgxEwIIYFx6sQmpxBYgVSwYEHVt956q2psuCIiMnbsWNXr1q1TnR+q5jBsz22lq68qxO0wYyjkdfA8eAynn366alwT73e/+53ZZs6cOapXrlypOj/aGnh+cO1EtHreeecds02yzgN/MRNCSGBwYiaEkMDI+HgEQ058kupDUQTDD7+0eOjhaCz8sWKY2rp1a9W4vDoueS8icvToUdWDBg1SnalWBoafGKIWLVrUjMPrjeMKFy5sxhUpUkQ1Zrfs2LEj5vYi4RW8/Pzzz6obNGigeuTIkarxPhARef/995O/Y2nCf29Kly6t+k9/+pPqadOmqfbZTLQyCCHkFIETMyGEBAYnZkIICYyM9JjRK0PPsH379qorVKhgtkFvCJcjX7hwoRmHa3qF5hEi6Gfi+RCxKU+4Dh5WLa1atcpsM2/ePNWZ4rN7vw/PQ3Z2tuq2bduqvvzyy802UR5z9erVzbiaNWuq3rNnj2r0Z1977TWzDVbRpeOces8bn6f07NlTNXqrvXv3Nttgipx/HpPp+Oq+Dh06qN64caPql19+WXWqzkG4Mw8hhJyicGImhJDACNbKwDDMp7VcdNFFqu+44w7VF198sWqseBOxYW9OTo7qCRMmmHEY+mNoHFp4j/vWtGlT89r999+vGtO+MBVq4sSJZpsFCxaoDjlFDu+Fc845x7x2yy23qK5bt67qs88+W7VPl0O7AbW3hw4cOKC6WLFiqjEF8b333jPbbN68WXU6KuV8Q6Lbb79d9VVXXaUa7Ri0tERs6mV+AK3KihUrmtdatmypetSoUarx2qfqfPAXMyGEBAYnZkIICYxgrQwMWbt27Wpee/jhh1XjU1Ks0Jk+fbrZBp/AYjMf/xR6165dqjGcCeGJNIbXGJ7feeedZhxmEGzfvl31Aw88oNpno2AGSsi2DR4bhuAiIvXq1VN97Ngx1VidN2XKFLPN0qVLVX/++eeqDx48aMb16tVL9T333KMan95jRaBIes4j3ucNGzY0r911112qv/rqK9WYdZCqyrZ0gfPKTTfdZF7bt2+f6i+//FJ1Os4BfzETQkhgcGImhJDACCpOieoXe9lll5lxGI4MGDBA9bhx42K+l4gNMzFjo3nz5mZc7dq1Y+5DVD/eZOKzUQoVKqQan7D7ogkMR1evXq36k08+UR16wx0E9w2fkM+fP9+MwzD+o48+Uj116lTVeD5E7FN6zEbx575q1aqqcRmu5cuXq/b2RzruE2y2hPaLiEjJkiVV33fffaoxhM9v1oWIvcb169dXjQVpItYixYyWdNiY4X4bCSHkFIUTMyGEBAYnZkIICYygDCX05NA3w0o2EeuDYSoUeoS+cgurfLDJj/dW165dqxor5Xzj9FTgfU6sWqtcuXLkdlu3blWNKWXoPXv/E/3ZqHXx/LlKlfeG1xX90IEDB5pxeI0OHTqkGp8v+MotPAa8Z5o0aWLGNW7cOOb+rFixQrX3mPGZQDJBD7VRo0aqr7nmGjNuxIgRqj/88EPVIVd65gV//+K93rFjR9UbNmww4z799FPV6U6P5S9mQggJDE7MhBASGEFZGVHs3r3b/B21nh+GMBdccIF5DdODWrVqpfq7774z47BSrnz58qr379+vOlWhnw/JSpQoEVN7iwH7BX/77beqMaSvUqWK2QYrCTH0Q43nQETk66+/jrkPyUwTw3Pv++n69epibePBewmtkB49ephxaIWhnbJt2zbVqaz0w3sD0/dwv/Hai9jGVZgOhufHHwPahiGnVCLexsQKSLQynnrqKTMOm6OhzYG2WKqucWacaUIIOYXgxEwIIYGREVaGD6Ew5KxRo4ZqXBrm6quvNtucddZZqjHkzcrKMuMGDRqkGm2O/v37q16zZk3c/UsU3rKpVq2aarQefEiPvZXxPZ577jnV+PRexC7FheEaHhuGdCIizzzzjGqsrvMkMvzDDARPuXLlVFeqVEn1kSNHVPvw/vDhw6qxvzP25hWx1sErr7yiGvsXo6WQbKKq2dq1a6fa99xG2wW/H9i72lt7H3zwgeq9e/eqDs3WiFeZ27p1a9Xr169X7a8XVg53795dNVaYpqoyMqyzSwghhBMzIYSERrBWBj5Z9WE3FhZg+IkJ/ZhdISLy7rvvqsbGM/5J/nnnnae6c+fOqp988knVvjkMPqVPZsYGvjeGVL4PMIamWFSADZt8oQWGf1iIgtcBm+CI2EKd2bNnR+7PyVoZGLajJeWbN11//fWqcZXrH374QTX2IRaxPZgxkwezXvx2uBo22kjJDHN9hg5aVFdeeaVqDM9xNXgRkWeffVY1Fp/gfvuMBjxW/A74ZavSbW3g+fHLh+F3Gq0ZzM4SsVlHmKHhm2WlAv5iJoSQwODETAghgcGJmRBCAiNYjxmpU6eO+Rs9Z/SSR48erdovJY++MnqWfo2zMmXKqK5Vq5ZqTKsrUKCA2Saq6U8yQY/Re7i//e1vY26Dx71o0SLz2meffaYaKx6xmbj3mHPT+CivoNeJzxEw5Q8XNRCxFY/YkAZTC72v2KZNG9V4Tn0K4tChQ1Vj46xUNSryqZNly5ZVjfcpPjNBn1TEfm8w9W3VqlWq/TqBTZs2VX3mmWeqxkZZImF5zP6aYHotevA+9fKNN95Qjb50OtZu5C9mQggJDE7MhBASGMFaGZjCM2vWLPPaunXrVO/cuVM1hpg+TMH0MNQ+BMPUI1znDT/TpxQlK9Tx7/v999/H1MWKFTPj8Njx+DDMnTRpktkGU/Hq1aunGsNAXxWGdhFW1+Wll62/Xtg06Omnn1aNISb+u4jI+PHjVR87dkx1ly5dVD/++ONmG7QI8Bz4+wLfA3t2o5XmUxATid8fvP4rV65UjVZEqVKlzDbFixdX3a9fP9VTpkxR7c9P27ZtVeNamel8mAYoAAAb1ElEQVSw7+KB++O/D9nZ2arx3l68eHHk++G9kI5j5S9mQggJDE7MhBASGMFaGfH6AOMTZRyH4YcPK6OWqf/9739vxj344IOqMWPjhRdeUI1P/0WSV/EVL3zFp+INGjQw4zA8R43ZJA899JDZBi0L/Fw8NmyOJGKXJ8rLOcAQ0V+vm266STVmEzz66KOqx4wZY7bBii+sCuzZs6dq37gGLQ/swYuZHCIinTp1Ul26dGnVuOzZ5s2bzTaJvC/iLQWGFhVeO289YTUkWkKYkYPZHiL2u5aTkxPzc0IA7yWfNYVWBmZ4TZ8+3YzDjB3M4koHYZ1dQgghnJgJISQ0ODETQkhgnJaKVJCsrKyEfgh6v1GN033KFlYwYRNsXyWHHiSmY2GXtnQtbY7HjV4ZVsOJiJx77rmq0QvEa+0r29CLxuNbunSp6iFDhpht0HOO16g8Crx2lStXNq+hf43pXOgrYzc5EetFo5+KlWC+4xqm2H3yySeqsXOeiO00iJV2f//731U/9thjZhtMq0y0J4vnDvdn5MiRqr1PHvVd37Jli2rvz2L6HHqyqVr3Mrfg/evT5YYPH64aKz39MwF8ztGtWzfVWBmZ6OdJOTk5Mb8s/MVMCCGBwYmZEEICIyOsDN/ABZvsYNiOYRiGLCI2NMXUMB/aDhs2TPVbb72lGkPRdDQ18WCYjBWKIiLXXXedakyLwuq8uXPnmm0OHDigGpvR41p+uGacyH+GvScKHkPNmjXNa9icHCvtsHrNV7ZhZRqmdk2YMEH14MGDzTaYahbvu4ANk3AhgtWrV6vG9eREUpdSFmVrXHvttWYcrhmJ1+7gwYOqJ0+ebLbBxvJoX4TwHYjCNyZDWwvTKPHfRUTmzJmj+v3331edF5sut9DKIISQDIETMyGEBEZGWBl+DTnMqnj++edV4xNTDM1FRJYsWaJ62bJlqt9++20zDhvCYLiXSaFbVL/pqCpJ/xpaR/EaPp0suG/Y61fE9sZFGwHvVwzBRWyzK9Tz5s1T7RtQ5Ta7APcVzw+ekxAyFaLWaxSx5y4qPPcVmMlcxzBVRGVx+bkvXuVwsqCVQQghGQInZkIICYyMsDLiPWXFJX/27dun+tVXXzXbYFMSzE7w4V6qQhgSH8y48MvR/4K/drt27VJ9+PBh1Vg0FLIlRU49aGUQQkiGwImZEEICgxMzIYQERkZ4zB5MV/L+8y94LxH/Dq2Kj/wnUelpiL92eF1Da+ROSCzoMRNCSIbAiZkQQgIjI8t6GLLmf7AKK4SKOkJSCWc1QggJDE7MhBASGJyYCSEkMDgxE0JIYHBiJoSQwMjIrAxCSHLwBWfYvzi03tP5Gf5iJoSQwODETAghgcGJmRBCAuOU8ZjRO4vXBAnH4XpnIXtq/hhwjURsEh/yMSSan376SbVvqB8Feqh43kTyd7MrvOf9QhFVq1ZVjeto+jU18xt4TvD75L9Dv/nNb5Ly+fzFTAghgcGJmRBCAiPfWhlRfZpFRMqUKaMaQzURG6ps3LhR9ffff5+rz8Ltkxn+4meWLVvWvHbllVeqnjlzpurt27erzq2tEbXkvUj6G0j5/cG/mzZtqrpevXqq0Z7y4PnB8+aJ9x6ZCKbEXX/99ea1F198UfVf//pX1U899ZQZh/d6pto+eP/gMXTq1Ek1risqIrJ48eKk7At/MRNCSGBwYiaEkMDI+JgMww8M76tUqWLG3XLLLaqbN2+uGm0NERumTps2TfXDDz+s2odqLVq0UL1q1SrV+/fvV52IsD8qY+S+++4z4/r06aN6w4YNqjdv3qw6npWBn1OoUCHVhQsXNuP27NmjOh22hrcyrrrqKtWPPPKI6uzsbNWHDx822xQpUkQ1Pn3/+OOPzbgRI0aoXrRoker8YGtgllKTJk3Ma8ePH1d95plnqvb3T9TyX5kEZu9069ZN9RNPPKF6+fLlZpvu3burzsnJUX2y3wf+YiaEkMDgxEwIIYHBiZkQQgIjIw2yqCq+tm3bqvbpPOglr1ixQvWECRPMuFKlSqnGNCKsJDv33HPNNug/jh07VjV6U+jV5hXcH0wBa9++vRm3a9cu1eip5jaNCT/n0ksvVd2xY0czrm/fvqrRT09mhSH6gM2aNTOvDRw4UHXRokVVDx48WPWrr75qtrngggtU4/FcdtllZly1atVUd+7cWXUifcVUgt8b/G6gj+z59NNPVeP3QeQ/KwYzAZ9Si8feq1cv1fgc4e233zbbJPo5kr5Xwt6JEEJIQuDETAghgZGRVkZUGPboo4+q9qHVAw88oHrOnDmqMewXESlYsKDq6tWrq8a0GKys85+1evVq1Scb0vvmO3is/fr1i/nvIiLjxo1TvWbNmpj7GQ88v7Vr11Zds2ZNM6506dKqsTIy0VZGlHXVsmVLMw5D0UmTJql+5ZVXVB87dsxsM2vWLNW7d+9WPX78eDOuXLlyqjMxbBeJruLE9FGsmBQRWbdunWpMFcsP1X3+Pu3atatqrAjGCmC0c0Ts/ZjI+56/mAkhJDA4MRNCSGBkvJWBmRi1atVSfdddd5ltMLzHMAzDcRGbeYD2RY0aNVSvX7/ebIOf9eGHH6ouUKBAnKOIDT4pLlGihHnt9ttvV92qVSvVU6dONeOeeeYZ1UeOHFGdl8ZFuD/pqu7CfahQoYLqiy++2IzDJkTDhw9XjfaFr9TDc7Jy5UrVo0ePNuP+9Kc/qf7tb3+reuLEiapDz8pAa6xixYqqe/bsqdpXd86YMUM1VrXm5d4OAcwmwYwcEVvth3YVngOspPXjEknYdxIhhJyCcGImhJDAyEgrA8FQNN7SQPh348aNVWPDHxFbULFlyxbVjz/+uOopU6aYbTDJPC9LzaB1gFkh1157rRnXu3dv1ZhB4ItpMNMEQ87cWhEYzmJmCn6m/zuZRSVoZdSvX191w4YNzbjPPvtMdW7DbrS18PzMnj3bjLvxxhtVo6WEn/ndd9+ZbdLd4Mg3ecL9ue6661Rjdsvnn39utsECLAzbMykrA68rWpeYqSViM28wAwVtLX+fJ+s88BczIYQEBidmQggJDE7MhBASGBnpMaOXjFV86AtiQxoRkXbt2qnGajbvw+G6Zu+++65qTJHzPvLJpg5hg/bLL79c9T333GPGoVeGqU/oS4vYCjhMFUPv2PvN2PS+UaNGqjElza+Dh9skoklTbsBnBf68Hzx4UHVeUtdwG2wQL2KvUVZWlmr0Jbdu3XrCn5lMfOUoPlu57bbbVGMjJqySFLHPWTI1RQ7vdfwO4DwgInLgwAHV6CtjVWteniHlBf5iJoSQwODETAghgZGRVgamqGCqGjYbweosEZsShJVx2D9ZxFbKYQjk0+8SCX5O2bJlVVeqVMmMO3TokGq0K3z6HvLVV1+pxj7SGJqLiLz55puqS5YsGXPf3njjDbNNMsM6tJjwcy688ELVvp+ubzCTSPCei2oGFAJ4TtC6EBF5+umnVeNah88995xqb1elKnRPJPHSBLEBma+sxbUc586dqzqZqaBR8BczIYQEBidmQggJjIywMnwGAYaV2IMZ7Qq/hNANN9ygGivj8ImriH1qm6oQBsPFBQsWqH799dfNOKzCwyfKuIyS55JLLlF99OhR1f7Y7rvvPtUYCq5du1b1smXLzDapOj+YLYEZEf6+SKSt4N8L/8b7L4QKODwPGLbjPS9i7xnMYEL7wmdyZKKV4Y8BK0Rvvvlm1fh9ELFzBs4L6ei/zV/MhBASGJyYCSEkMIK1MvDpcrFixcxr2HwEQ/V7771XNRYbiNgQBp9WT5s2zYxLx1N2DJWw+Q7aCyL2KXJ2drbqeJYChtqYieHtD+zJi0UuWGgRQr9hPB6flbFv376Tem+89j58xeIKDIHxM1N5fnBfcVVzbHR19dVXm22wlzAuTYbZPpm6bFa8DKoePXqorlKliup33nnHjMOGVOnIxEDS/00jhBBi4MRMCCGBwYmZEEICIyiPOaqiCr1jEZHOnTurxmYsuN5e+/btzTbonWFjH+9TpttHxf3E9clEbGP6nTt3nvB7oz/rm/Sg746+Pabs+WrBVPlweC/EW8Pw7LPPVv3ee+/l6r3Rm0QfGe8rEZHKlSurHjNmjOpvv/02cn8SiX/2gfvdpk0b1bjfhw8fNtsMGzZM9d69e1Vjil0I6X95AX12fEYiYu9nfCaAFY8i9nzRYyaEEGLgxEwIIYERlJWB4VmtWrVUX3HFFWYcNiGaP3++6k6dOqnG5eZFbLi2bt061T4898u3p5N4tkpeQi0M9+rWrWtewyqxJUuWqM6tJZBoMKRGSwf3DdOgRESaNWumOiqdzFsC+Bqug4daxPYlHjVqlGq8Rsm0Abytdf7556sePHiw6uLFi6vGdSpFRKZOnao6P9gXOF+UL19etf/ulypVSvWIESNUow0lkn4bEwlnTwghhIgIJ2ZCCAmOoKwMDCu7dOmiGrMRRKz9MGnSJNW4JJJfOv6ll15SPWvWLNV+Wab8BobuGPpdf/31ZhxmNCxevFg1PqlOV8iLn/v111+rXrFihRmH4f2DDz6oGvtI4z3mt+nfv79q3+AGK+U2bdqkOplNfnBfMVQXsSF5xYoVVQ8aNEj1hAkTzDZof0VdS39+MGsJv1PpylrA/UGL4uGHH1Zdp04ds82MGTNU43nzhGTp8BczIYQEBidmQggJDE7MhBASGEF5zOiBlilTRjWmQfm/f/jhB9XojU6ePNlsg39j6lFIKTLJAM8prhN43nnnmXGYQoiVbdh0PF3nCqshV65cqRr9VBG7ph1Wi7Zr1061T5dDb33Hjh0x30vE3lv++UUiQQ8Vr9fdd99txtWoUUP1N998o3rhwoWqfYpd1H7jPdKkSRPzGq47ietHbt261YxL1b2B+4rrFtarV0/19u3bzTYjR45UjQ3wsdIzNPL3rEQIIRkIJ2ZCCAmM01LRGD4rKytXH4JhCjaC91VqmKqDTUkwpMvJyYncJr/bF0hUetGTTz5pxmEY/8gjj6gOLdzD+9Vfx7Jly6rG1Em0MnzqJVojaANs27bNjEM7JZlpVXh82PC9dOnSkfuDqX143/u173Lzmbimot8HXHzCpxOmKtUM0/kwTRDTGT/66COzzZQpU1SH9t3PycmJeeLC2ktCCCGcmAkhJDSCsjIQDMH9MvVI1FLyPmQJqaonXeC19n2okWRmHSQSf+/iMeXlvsZ7JIRzEFW16f/G/c5NdV88/OdEWUchfJ9yez+f7DlJJrQyCCEkQ+DETAghgZH+eC0CDD/SvcxLfiG0UP1k8WFpfjgmJMqiiPV3oggtayEe+e1+RjLnKhBCyCkCJ2ZCCAkMTsyEEBIY+cuYIYSQOBw/fly1XxQAFz3Aysp0wF/MhBASGJyYCSEkME5JK8OHMEh+S7shlniVbah9OlpoFWMkGn+NMQWwU6dOqnG9RxGR2bNnq8b+23hfpOo+4C9mQggJDE7MhBASGME2MTpZfDiDfWWfeOIJ1WeddZYZ16VLF9W0NfIH2Je4UKFC5rXChQvHfG3nzp1mXFTTIBIG8Rqd9e3bV3WPHj1Ur1q1yozD7/udd96pGnuVJ7rikk2MCCEkQ+DETAghgZFvY3Xfn7VKlSqqL7zwQtU+kTwV1k4iwP30WSb4Nz6RxlAtvzeGwnNwzjnnqL799tvNuNq1a6uuXr26arS7REQmTJigOlVP6VN1L+YHawav92WXXWZe++Mf/6h67Nixqt977z0zrk2bNqqLFi2a6F08IfiLmRBCAoMTMyGEBAYnZkIICYx85TFjygymQYmI3H333apr1Kih+sMPP0z+jiUIPD70iNu3b2/GYXXT1q1bVaNPunnzZrNNfkgN/PHHH1U3btxY9dChQ1WXL1/ebIPndPfu3aorV65sxkX59on2Z6N8ZUz39OD+xFsHD48VUwjjrZeH2+A9EkJDfbwm9evXVz1w4EAz7ttvv1U9cuRI1Zs2bTLjPv74Y9Xpfh6T/rNLCCHEwImZEEICI/PjVwDDs549e5rXrrjiCtUYuk2ZMiX5O5ZHfFiLqX333nuvaqxS8tsVLFhQddu2bVX37t3bbPPNN9/E/JyQ8SF4hQoVVN9///2qMfXphhtuMNtg5SemWdWpU8eM+8Mf/qD69ddfz+Me/yf+GuPfXbt2VX3fffeZcXiv4/Gh9fDdd9+ZbdCqWb9+vervv/8+cv/WrVuneuHChaqxr7FIelLu8Pq3a9dOtber7rjjDtVoXxQoUMCMS0ezoij4i5kQQgKDEzMhhARGxlsZGNLVrVtXtQ9Z8cnq4cOHVW/cuNGMS3cIg/iKvqpVq6rGzIulS5eaccOHD1ednZ2tGkO6J5980mxz6623qsbzE8LTdyReuIn2xUUXXaQabYBFixaZbbDv7qRJk1T36dPHjHvwwQdVo0Uwd+5c1XmxgHzznWLFiqnu2LGj6ooVK5pxeN9HZXKULFky8nN9Js8veHvohx9+UD1s2DDVmN3gSeZ3CM8XWjiYlbFixQqzDWZe4fJRnpC++2F96wghhHBiJoSQ0MhIKwPDraysLNWDBg1SjQ1pRGzxwdGjR1Xv2bMnGbuYEHyIiuEaFtD079/fjMPwHMMzLCrxTXrKlCmj+uDBg6pDszIwG6Bz587mNeylPWbMGNWYeeNDWTw/GCaPGzfOjGvSpIlqzILBc33kyBGzTW7OnQ+f0aLALAhvS2BWBV7XEiVKqPa9p3F/sLETWgKlSpUy2+B7dO/eXfXbb79txuH3KJmWAH730d7B48ElokTs9z2elRESYX3rCCGEcGImhJDQ4MRMCCGBkZEeM3pYqLF6bfr06WYbXOsLvUD0nv37pQP0lX3zlFq1aqnGFDls0iLynw2cfgH9UDxXIvb8YBMYTNlL17nBfUBf0ae0bdmyRfWrr74a873iHQN6sP6+GDJkiGr0r9F7/uCDD8w2ufEzvQ+NHvpf//pX1S+++KIZ99NPP6mOt94dgseO+4bPaV544QWzTcuWLVUXL15ctb/HUtXUHz8HGw1hFd9XX31ltjnZ+zbesSXrO8FfzIQQEhicmAkhJDAy0srA8A9Tu7A6y6fL9evXTzVW+4VmZSA+zMWUNqxuwko9ERumYnoRplz5SrJmzZqpfu2111TjuUrXOoGYQnb22Werxp7LIiLPPvusarR3fLOa3OCPdfv27aqLFCmiukWLFqo/+ugjs01emuJE2XS+ChT3Ly/XBe+LHTt2RL4X/r1t2zbVvvFRqr43eE7Lli2runTp0qorVap0wu8lYs9xPEsRwfN4stcE4S9mQggJDE7MhBASGBlpZSAY7mNY4Zcfx/AzJydHNYbJIRAvfMWQE22NeM1z8Jxgn1pcXkvEZqqgFZKqp+0e/FwMU3GJMMzCEBF55513VCfadsH7DDUu+eStp0Seu0RbBXhvNWjQQHXNmjXNODyP7733nupDhw6ZcamqEMXzgBksx44dU12tWjWzDV4HzGbxy6mh/dmoUSPVuMyYP87ly5erxqynAwcOxDmK/w5/MRNCSGBwYiaEkMDgxEwIIYGR8R4zgv6Pr4bC1zAFzKea4Rp56cY3LccuYthJDdetExHZsGGDajwP2GULtYhNKcO0utxWlSUa/NwqVaqoRu9vxowZZhtMaTtZz9P7w7g/mKKJHqN/XhHa2ol4THh+cC1I310OPVmsNvXPP/KSkpgX0PPG9QhXr16tumHDhmYbfGaCaaEdOnQw43BxBXwGg892/MIa1113neovvvhC9UMPPaQ6L348fzETQkhgcGImhJDAyFdWBob+fvl5DDNDbo6P+JSvL7/8UvXOnTtV41psInbBAFyy/uabb1btQ9b9+/erxpSpefPmnehuJwS8lljthylOmJ4kYtenwzS23IKhvg/Ve/bsqRptDb8PIYPn9Mwzz1R98cUXq/bN9bHp/Ndff63ap5qlCvxO7Nu3T/XYsWNV+/Us0fLCNFMP2iF/+9vfVM+ZMydyG2w01apVK9XY8MmnztHKIISQDIQTMyGEBEZGWBnxmo0gGGL6UB23wapA/+Q8L41nkoUPFzHr4LHHHlONYZeIXeMOw3s8Jz6jAXvYYmOeSZMmqfaZHMms9oqqgIxnN+D1j7qO/l6Ker+mTZuacb1791a9YMEC1WgVpavJU27BY8Xjw8o/v27hzJkzVWN2grc80gGeb7yfb7rpJjMOLQZs/jVgwAAzDtdYxH7TaO1dc801ZhvMaJk2bZpqbPKUl+8JfzETQkhgcGImhJDACNbKwLDLL2PTunVr1ZgUjoUW2LBHxD6lv//++1VjqC9iQ/d0PXmOAkM3DMl69eplxmESPdoSmMkxfPhwsw2GazfeeKNqPI+Y0C+SXCsD3xuXwcLMgvPPP99sg4UxGJLjNv6aYvHBFVdcodqHudgk55lnnlGN909o94u3bfD+wUwMtPb8NcasnFQVkeQWtKiwr/ro0aPNOCxQKlGihGosAhGxxWZ4L2VnZ6vGTBAR2wP8rbfeUo22X14sLv5iJoSQwODETAghgcGJmRBCAiMoUww9MUx9wqorEZFHHnlENfpM6OX4BkBbt25Vjf7smjVrzLh0p8jlFjxW9GD93xMnTlSN5xSbjItYrxV13bp1Va9du/Yk9vjEQI8ZK7Jeeukl1dg0X0Skdu3aqrGhDJ4PrCIUEWnevLlqbJS+atUqM+7RRx9VjfcSnqvQ8B4zVvu1bNlSNaYJ4nkTyZxjxbTXWbNmmdeWLFmiuly5cqqvuuoqM65evXqqcTENTI98+eWXzTa4DiLes1zzjxBC8hmcmAkhJDCCsjKilm73TUQwrcVbFrG2FxFZv369amw840P60Ku3YhGv729UP2UflmLohqEfNoPCNfVEklslie+H13jkyJExP19EpFOnTqrvuece1Rhi+nSnRYsWqZ4wYYJqXN9OxFb4hRzSI74y8tJLL1WNKWB4j2Avb5HMsfYQn8a5d+9e1djADO9zkei5BN/PX/tk9dzmL2ZCCAkMTsyEEBIYQVkZCIYPmEUh8p8hSG7AqiwMP1K17Hq6yG0oimEvVs0VKVLkhN8r0aC9hDbW0KFDzTjsyYv7jeBSSSK2Vy4uGeWPNbSqvtzgrR6sZsPjwe9AJloX/42o73jItmX+npUIISQD4cRMCCGBkRHxmX/yGdrqw5mID1kxxB8/fnzkuHjvkQow/PShOmZOYMMmxO8zhrkY3ueHkN6H8NjPG6832ljeJswP5yET4S9mQggJDE7MhBASGJyYCSEkME7zPl0yyMrKSv6HkITx888/x/x3evuZhf9u4zp92JwKPebly5ebbbAajn5z4snJyYl5UvmLmRBCAoMTMyGEBEZKrAxCCCG5h7+YCSEkMDgxE0JIYHBiJoSQwODETAghgcGJmRBCAoMTMyGEBAYnZkIICQxOzIQQEhicmAkhJDA4MRNCSGBwYiaEkMDgxEwIIYHBiZkQQgKDEzMhhAQGJ2ZCCAkMTsyEEBIYnJgJISQwODETQkhgcGImhJDA4MRMCCGBwYmZEEICgxMzIYQExv8DLFN0QAwezswAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1a1a149a58>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "row = 5\n",
    "column = 5\n",
    "max_row = row * 20\n",
    "max_col = column * 20\n",
    "M = np.zeros((max_row, max_col))\n",
    "plt.figure(figsize = [6,6])\n",
    "\n",
    "for i in range(row):\n",
    "    for j in range(column):\n",
    "        sample = np.random.choice(m, 1)\n",
    "        sliceRow, sliceCol = i * 20, j * 20\n",
    "        M[sliceRow:sliceRow + 20, sliceCol:sliceCol + 20] = X[sample].reshape(-1,20).T\n",
    "                    \n",
    "plt.imshow(M, cmap='gray')\n",
    "plt.axis('off')             \n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multiclass Classification\n",
    "\n",
    "#### Logistic regression hypothesis\n",
    "#### $$ h_{\\theta}(x) = g(\\theta^{T}x)$$\n",
    "#### $$ g(z)=\\frac{1}{1+e^{−z}} $$\n",
    "\n",
    "#### Regularized Cost Function \n",
    "#### $$ J(\\theta) = \\frac{1}{m}\\sum_{i=1}^{m}\\big[-y^{(i)}\\, log\\,( h_\\theta\\,(x^{(i)}))-(1-y^{(i)})\\,log\\,(1-h_\\theta(x^{(i)}))\\big] + \\frac{\\lambda}{2m}\\sum_{j=1}^{n}\\theta_{j}^{2}$$\n",
    "#### Vectorized Cost Function\n",
    "#### $$ J(\\theta) = \\frac{1}{m}\\big((\\,log\\,(g(X\\theta))^Ty+(\\,log\\,(1-g(X\\theta))^T(1-y)\\big) + \\frac{\\lambda}{2m}\\sum_{j=1}^{n}\\theta_{j}^{2}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "def costFunctionReg(theta,X,y,lamda):\n",
    "    '''Compute cost for logistic regression with regularization  '''\n",
    "    \n",
    "    #Initializations\n",
    "    theta = theta[:,np.newaxis]\n",
    "    m = len(y) \n",
    "    J = 0\n",
    "    \n",
    "    #Computations\n",
    "    z = X @ theta\n",
    "    h = sigmoid(z)\n",
    "    J_reg = (lamda / (2*m)) * np.sum(np.square(theta[1:]))\n",
    "    J = (1/m) * ( (-y.T @ np.log(h)) - (1 - y).T @ np.log(1-h)) + J_reg;\n",
    "    \n",
    "    if np.isnan(J[0]):\n",
    "        return(np.inf)\n",
    "    \n",
    "    return(J[0])  \n",
    "\n",
    "\n",
    "def GradientReg(theta,X,y,lamda):\n",
    "    '''Compute gradient for logistic regression with regularization  '''\n",
    "    \n",
    "    #Initializations\n",
    "    theta = theta[:,np.newaxis]\n",
    "    m = len(y)\n",
    "    grad = np.zeros(theta.shape)\n",
    "    \n",
    "    #Computations\n",
    "    z = X @ theta\n",
    "    h = sigmoid(z)\n",
    "    grad = (1/m)*(X.T @ ( h - y)) + (lamda/m)*np.r_[[[0]],theta[1:].reshape(-1,1)]\n",
    "    \n",
    "    return(grad.flatten())\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Testing the implementation using dummy data\n",
    "Note how much more complicated it is to create the dummy X_t array in Numpy... in Matlab the following code is equivalent\n",
    "\n",
    "    X_t = [ones(5,1) reshape(1:15,5,3)/10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Cost: %f\n",
      " [2.5348194]\n",
      "Expected cost: 2.534819\n",
      "\n",
      "Gradients:\n",
      "\n",
      " %f \n",
      " [ 0.14656137 -0.54855841  0.72472227  1.39800296]\n",
      "Expected gradients:\n",
      "\n",
      " 0.146561\n",
      " -0.548558\n",
      " 0.724722\n",
      " 1.398003\n",
      "\n"
     ]
    }
   ],
   "source": [
    "theta_t = np.array([-2, -1, 1, 2]);\n",
    "X_t = np.concatenate([np.ones(5)[:,np.newaxis], np.arange(1,16).reshape(3,5).T/10],1)\n",
    "y_t = np.array([1,0,1,0,1])[:,np.newaxis]\n",
    "lambda_t = 3;\n",
    "J = costFunctionReg(theta_t,X_t,y_t,lambda_t)\n",
    "grad = GradientReg(theta_t,X_t,y_t,lambda_t)\n",
    "\n",
    "print('\\nCost: %f\\n', J);\n",
    "print('Expected cost: 2.534819\\n');\n",
    "print('Gradients:\\n');\n",
    "print(' %f \\n', grad);\n",
    "print('Expected gradients:\\n');\n",
    "print(' 0.146561\\n -0.548558\\n 0.724722\\n 1.398003\\n');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One vs All classification\n",
    "One-vs-all classification works by training multiple regularized logistic regression classifiers, one for each of the K classes in our dataset. In the handwritten digits dataset, $K = 10$.\n",
    "\n",
    "** Dimensions of the parameter matrix**\n",
    "\n",
    "The classifier parameters is a matrix $\\Theta \\in \\mathbb{R}^{K \\times(N+1)}$ , where each row of $\\Theta$ corresponds to the learned logistic regression parameters for one class. We can do this with a “for”-loop from 1 to K, training each classifier independently.\n",
    "\n",
    "Note that the y argument to this function is a vector of labels from 0 to 9. When training the classifier for class $k \\in \\{1,...,K\\}$, we will want a $m$-dimensional vector of labels $y$, where $y_j \\in \\{0,1\\}$ indicates whether the $j$-th training instance belongs to class $k (y_j = 1)$, or if it belongs to a different class $(y_j = 0)$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def OnevsAll(X,y,K,lamda):\n",
    "    #Initialize variables\n",
    "    m,n = X.shape\n",
    "    all_theta = np.zeros((K,n+1))\n",
    "    initial_theta = np.zeros((n + 1, 1))\n",
    "    \n",
    "    #Add column of 1s for y-intercept\n",
    "    X = np.concatenate([np.ones(m)[:,np.newaxis], X],1)\n",
    "    \n",
    "    for k in range(1,K+1):\n",
    "        model = opt.minimize(fun = costFunctionReg, x0 = initial_theta, args = (X, (y==k)*1, lamda), \n",
    "                             method = None, jac = GradientReg, options={'maxiter':100})\n",
    "        all_theta[k-1] = model.x\n",
    "    \n",
    "    return all_theta\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_theta = OnevsAll(X,y,10,.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## One-vs-all Prediction\n",
    "After training the one-vs-all classifier, we can now use it to predict the digit contained in a given image. For each input, we compute the “probability” that it belongs to each class using the trained logistic regression classifiers. \n",
    "\n",
    "The one-vs-all prediction function will pick the class for which the corresponding logistic regression classifier outputs the highest probability and return the class label (1, 2,..., or K) as the prediction for the input example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predictOneVsAll(all_theta, X): \n",
    "    m = X.shape[0]\n",
    "    num_labels = all_theta.shape[1]\n",
    "    p = np.zeros((m, 1));\n",
    "    X = np.concatenate([np.ones(m)[:,np.newaxis], X],1)\n",
    "    z = X @ all_theta.T\n",
    "    h = sigmoid(z)\n",
    "    \n",
    "    return (np.argmax(h, axis=1)+1)\n",
    "\n",
    "def Accuracy(y_pred, y):\n",
    "    return np.mean(y_pred == y.ravel()) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = predictOneVsAll(all_theta, X)\n",
    "print('Training set accuracy: {} %'.format(np.mean(y_pred == y.ravel())*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Implementation using SKlearn\n",
    "Of course, training accuracy depends greatly on the regularization value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_SK = linear_model.LogisticRegression(C=.1, penalty='l2', solver='liblinear')\n",
    "# Scikit-learn fits intercept automatically, so we exclude first column with 'ones' from X when fitting.\n",
    "model_SK.fit(X,y.ravel())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred2 = model_SK.predict(X)\n",
    "print('Training set accuracy: {} %'.format(np.mean(pred2 == y.ravel())*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}