HandWritten-Digit-Recognition-Neural-Network.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "attempted-recommendation",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "subjective-virtue",
   "metadata": {},
   "source": [
    "# **Reading Data**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "future-philippines",
   "metadata": {},
   "source": [
    "Here we read our data from MNIST data set. You should download the data set and place it in a directory called \"/MNIST Dataset \" next to your code in order to run the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "elegant-smoke",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAM20lEQVR4nO3dXahc9bnH8d/vpCmI6UXiS9ik0bTBC8tBEo1BSCxbQktOvIjFIM1FyYHi7kWUFkuo2It4WaQv1JvALkrTkmMJpGoQscmJxVDU4o5Es2NIjCGaxLxYIjQRJMY+vdjLso0za8ZZa2ZN8nw/sJmZ9cya9bDMz7VmvczfESEAV77/aroBAINB2IEkCDuQBGEHkiDsQBJfGeTCbHPoH+iziHCr6ZW27LZX2j5o+7Dth6t8FoD+cq/n2W3PkHRI0nckHZf0mqS1EfFWyTxs2YE+68eWfamkwxFxJCIuSPqTpNUVPg9AH1UJ+zxJx6a9Pl5M+xzbY7YnbE9UWBaAivp+gC4ixiWNS+zGA02qsmU/IWn+tNdfL6YBGEJVwv6apJtsf8P2VyV9X9L2etoCULeed+Mj4qLtByT9RdIMSU9GxP7aOgNQq55PvfW0ML6zA33Xl4tqAFw+CDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJ9Dw+uyTZPirpnKRPJV2MiCV1NAWgfpXCXrgrIv5Rw+cA6CN244EkqoY9JO2wvcf2WKs32B6zPWF7ouKyAFTgiOh9ZnteRJywfb2knZIejIjdJe/vfWEAuhIRbjW90pY9Ik4Uj2ckPS1paZXPA9A/PYfd9tW2v/bZc0nflTRZV2MA6lXlaPxcSU/b/uxz/i8iXqilKwC1q/Sd/UsvjO/sQN/15Ts7gMsHYQeSIOxAEoQdSIKwA0nUcSNMCmvWrGlbu//++0vnff/990vrH3/8cWl9y5YtpfVTp061rR0+fLh0XuTBlh1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkuCuty4dOXKkbW3BggWDa6SFc+fOta3t379/gJ0Ml+PHj7etPfbYY6XzTkxcvr+ixl1vQHKEHUiCsANJEHYgCcIOJEHYgSQIO5AE97N3qeye9VtuuaV03gMHDpTWb7755tL6rbfeWlofHR1tW7vjjjtK5z127Fhpff78+aX1Ki5evFha/+CDD0rrIyMjPS/7vffeK61fzufZ22HLDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJcD/7FWD27Nlta4sWLSqdd8+ePaX122+/vZeWutLp9/IPHTpUWu90/cKcOXPa1tavX18676ZNm0rrw6zn+9ltP2n7jO3JadPm2N5p++3isf2/NgBDoZvd+N9LWnnJtIcl7YqImyTtKl4DGGIdwx4RuyWdvWTyakmbi+ebJd1Tb1sA6tbrtfFzI+Jk8fyUpLnt3mh7TNJYj8sBUJPKN8JERJQdeIuIcUnjEgfogCb1eurttO0RSSoez9TXEoB+6DXs2yWtK56vk/RsPe0A6JeO59ltPyVpVNK1kk5L2ijpGUlbJd0g6V1J90XEpQfxWn0Wu/Ho2r333lta37p1a2l9cnKybe2uu+4qnffs2Y7/nIdWu/PsHb+zR8TaNqUVlToCMFBcLgskQdiBJAg7kARhB5Ig7EAS3OKKxlx//fWl9X379lWaf82aNW1r27ZtK533csaQzUByhB1IgrADSRB2IAnCDiRB2IEkCDuQBEM2ozGdfs75uuuuK61/+OGHpfWDBw9+6Z6uZGzZgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJ7mdHXy1btqxt7cUXXyydd+bMmaX10dHR0vru3btL61cq7mcHkiPsQBKEHUiCsANJEHYgCcIOJEHYgSS4nx19tWrVqra1TufRd+3aVVp/5ZVXeuopq45bdttP2j5je3LatEdtn7C9t/hr/18UwFDoZjf+95JWtpj+m4hYVPw9X29bAOrWMewRsVvS2QH0AqCPqhyge8D2m8Vu/ux2b7I9ZnvC9kSFZQGoqNewb5K0UNIiSScl/ardGyNiPCKWRMSSHpcFoAY9hT0iTkfEpxHxL0m/k7S03rYA1K2nsNsemfbye5Im270XwHDoeJ7d9lOSRiVda/u4pI2SRm0vkhSSjkr6Uf9axDC76qqrSusrV7Y6kTPlwoULpfNu3LixtP7JJ5+U1vF5HcMeEWtbTH6iD70A6CMulwWSIOxAEoQdSIKwA0kQdiAJbnFFJRs2bCitL168uG3thRdeKJ335Zdf7qkntMaWHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSYMhmlLr77rtL688880xp/aOPPmpbK7v9VZJeffXV0jpaY8hmIDnCDiRB2IEkCDuQBGEHkiDsQBKEHUiC+9mTu+aaa0rrjz/+eGl9xowZpfXnn28/5ifn0QeLLTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJMH97Fe4TufBO53rvu2220rr77zzTmm97J71TvOiNz3fz257vu2/2n7L9n7bPy6mz7G90/bbxePsupsGUJ9uduMvSvppRHxL0h2S1tv+lqSHJe2KiJsk7SpeAxhSHcMeEScj4vXi+TlJByTNk7Ra0ubibZsl3dOnHgHU4EtdG297gaTFkv4uaW5EnCxKpyTNbTPPmKSxCj0CqEHXR+Ntz5K0TdJPIuKf02sxdZSv5cG3iBiPiCURsaRSpwAq6SrstmdqKuhbIuLPxeTTtkeK+oikM/1pEUAdOu7G27akJyQdiIhfTyttl7RO0i+Kx2f70iEqWbhwYWm906m1Th566KHSOqfXhkc339mXSfqBpH229xbTHtFUyLfa/qGkdyXd15cOAdSiY9gj4m+SWp6kl7Si3nYA9AuXywJJEHYgCcIOJEHYgSQIO5AEPyV9Bbjxxhvb1nbs2FHpszds2FBaf+655yp9PgaHLTuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJMF59ivA2Fj7X/264YYbKn32Sy+9VFof5E+Roxq27EAShB1IgrADSRB2IAnCDiRB2IEkCDuQBOfZLwPLly8vrT/44IMD6gSXM7bsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5BEN+Ozz5f0B0lzJYWk8Yj4re1HJd0v6YPirY9ExPP9ajSzO++8s7Q+a9asnj+70/jp58+f7/mzMVy6uajmoqSfRsTrtr8maY/tnUXtNxHxy/61B6Au3YzPflLSyeL5OdsHJM3rd2MA6vWlvrPbXiBpsaS/F5MesP2m7Sdtz24zz5jtCdsT1VoFUEXXYbc9S9I2ST+JiH9K2iRpoaRFmtry/6rVfBExHhFLImJJ9XYB9KqrsNueqamgb4mIP0tSRJyOiE8j4l+Sfidpaf/aBFBVx7DbtqQnJB2IiF9Pmz4y7W3fkzRZf3sA6tLN0fhlkn4gaZ/tvcW0RySttb1IU6fjjkr6UR/6Q0VvvPFGaX3FihWl9bNnz9bZDhrUzdH4v0lyixLn1IHLCFfQAUkQdiAJwg4kQdiBJAg7kARhB5LwIIfctc34vkCfRUSrU+Vs2YEsCDuQBGEHkiDsQBKEHUiCsANJEHYgiUEP2fwPSe9Oe31tMW0YDWtvw9qXRG+9qrO3G9sVBnpRzRcWbk8M62/TDWtvw9qXRG+9GlRv7MYDSRB2IImmwz7e8PLLDGtvw9qXRG+9GkhvjX5nBzA4TW/ZAQwIYQeSaCTstlfaPmj7sO2Hm+ihHdtHbe+zvbfp8emKMfTO2J6cNm2O7Z223y4eW46x11Bvj9o+Uay7vbZXNdTbfNt/tf2W7f22f1xMb3TdlfQ1kPU28O/stmdIOiTpO5KOS3pN0tqIeGugjbRh+6ikJRHR+AUYtr8t6bykP0TEfxfTHpN0NiJ+UfyPcnZE/GxIentU0vmmh/EuRisamT7MuKR7JP2vGlx3JX3dpwGstya27EslHY6IIxFxQdKfJK1uoI+hFxG7JV06JMtqSZuL55s19Y9l4Nr0NhQi4mREvF48Pyfps2HGG113JX0NRBNhnyfp2LTXxzVc472HpB2299gea7qZFuZGxMni+SlJc5tspoWOw3gP0iXDjA/Nuutl+POqOED3Rcsj4lZJ/yNpfbG7OpRi6jvYMJ077WoY70FpMcz4fzS57nod/ryqJsJ+QtL8aa+/XkwbChFxong8I+lpDd9Q1Kc/G0G3eDzTcD//MUzDeLcaZlxDsO6aHP68ibC/Jukm29+w/VVJ35e0vYE+vsD21cWBE9m+WtJ3NXxDUW+XtK54vk7Ssw328jnDMox3u2HG1fC6a3z484gY+J+kVZo6Iv+OpJ830UObvr4p6Y3ib3/TvUl6SlO7dZ9o6tjGDyVdI2mXpLcl/b+kOUPU2x8l7ZP0pqaCNdJQb8s1tYv+pqS9xd+qptddSV8DWW9cLgskwQE6IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUji3y9hG/l2EQpSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAM2UlEQVR4nO3db6hc9Z3H8c9nsymI6YPEP+GSRpMNeWBZJNEYhMTlSmjJ5kksBmkelCwUbx9EabGEFfdBfFhk27KPArcoTZdoCaSuQYqbbCyGolZvJJobQ/4YoknMH0uEJoLE2O8+uMdyG2fOjHPmzJnr9/2Cy8yc75xzvhz95Jw5Z878HBEC8PX3D003AGAwCDuQBGEHkiDsQBKEHUjiHwe5Mtuc+gdqFhFuNb3Snt32WttHbZ+w/XiVZQGol3u9zm57lqRjkr4j6YykNyVtjIh3S+Zhzw7UrI49+0pJJyLiZERclfRbSesrLA9AjaqEfYGk09Nenymm/R3bY7YnbE9UWBeAimo/QRcR45LGJQ7jgSZV2bOflbRw2utvFdMADKEqYX9T0lLbi21/Q9L3Je3uT1sA+q3nw/iIuGb7EUn/K2mWpGci4nDfOgPQVz1feutpZXxmB2pXy5dqAMwchB1IgrADSRB2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSQIO5AEYQeSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgrADSRB2IAnCDiRB2IEkeh6fXZJsn5J0WdLnkq5FxIp+NAWg/yqFvXB/RPy5D8sBUCMO44EkqoY9JO2xfcD2WKs32B6zPWF7ouK6AFTgiOh9ZntBRJy1faukvZIejYj9Je/vfWUAuhIRbjW90p49Is4WjxclPS9pZZXlAahPz2G3faPtb37xXNJ3JU32qzEA/VXlbPx8Sc/b/mI5z0bES33pCkDfVfrM/pVXxmd2oHa1fGYHMHMQdiAJwg4kQdiBJAg7kEQ/boRBzTZs2FBaf/jhh9vWPvzww9J5P/3009L6jh07Suvnz58vrZ84caK0jsFhzw4kQdiBJAg7kARhB5Ig7EAShB1IgrADSXDX2wxw8uTJ0vqiRYsG00gLly9fLq0fPnx4QJ0MlzNnzpTWn3rqqba1iYlqv+DGXW9AcoQdSIKwA0kQdiAJwg4kQdiBJAg7kAT3s88AZferS9Kdd97ZtnbkyJHSee+4447S+l133VVaHx0dLa3fe++9bWunT58unXfhwoWl9SquXbtWWv/oo49K6yMjI5XW/8EHH7StVb3O3g57diAJwg4kQdiBJAg7kARhB5Ig7EAShB1IgvvZUcncuXNL68uWLWtbO3DgQOm899xzTy8tdaXT7+UfO3astN7p+wvz5s0rrW/evLltbdu2baXzdtLz/ey2n7F90fbktGnzbO+1fbx4LP8vDqBx3RzG/1rS2uumPS5pX0QslbSveA1giHUMe0Tsl3TpusnrJW0vnm+X9EB/2wLQb71+N35+RJwrnp+XNL/dG22PSRrrcT0A+qTyjTAREWUn3iJiXNK4xAk6oEm9Xnq7YHtEkorHi/1rCUAdeg37bkmbiuebJL3Qn3YA1KXjdXbbz0kalXSzpAuStkr6H0k7Jd0m6X1JD0XE9SfxWi2Lw3jMCA8++GBpfefOnaX1ycnJ0vr999/ftnbpUscolWp3nb3jZ/aI2NimtKZSRwAGiq/LAkkQdiAJwg4kQdiBJAg7kAS3uCKlW2+9tbR+6NChSvNv2LChtL5r167SehUM2QwkR9iBJAg7kARhB5Ig7EAShB1IgrADSTBkM1Iq+ylnSbrllltK6x9//HFp/ejRo1+5p7qxZweSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJLifHV9bq1atalt7+eWXS+edPXt2aX10dLS0vn///tJ6nbifHUiOsANJEHYgCcIOJEHYgSQIO5AEYQeS4H52fG2tW7euba3TdfR9+/aV1l977bWeempSxz277WdsX7Q9OW3ak7bP2j5Y/LXfqgCGQjeH8b+WtLbF9F9GxLLi7/f9bQtAv3UMe0Tsl3RpAL0AqFGVE3SP2H6nOMyf2+5NtsdsT9ieqLAuABX1GvZtkpZIWibpnKSft3tjRIxHxIqIWNHjugD0QU9hj4gLEfF5RPxV0q8krexvWwD6raew2x6Z9vJ7kibbvRfAcOh4nd32c5JGJd1s+4ykrZJGbS+TFJJOSfpRfS0Crd1www2l9bVrW11EmnL16tXSebdu3Vpa/+yzz0rrw6hj2CNiY4vJT9fQC4Aa8XVZIAnCDiRB2IEkCDuQBGEHkuAWV8xYW7ZsKa0vX768be2ll14qnffVV1/tqadhxp4dSIKwA0kQdiAJwg4kQdiBJAg7kARhB5JgyGbMWJ1uM/3kk0/a1spuf5Wk119/vaeehgFDNgPJEXYgCcIOJEHYgSQIO5AEYQeSIOxAEtzPjsbcdNNNpfU33nijtN5p2OVnn322bW0mX0fvFXt2IAnCDiRB2IEkCDuQBGEHkiDsQBKEHUiC6+yo1axZs9rWOv12++LFi0vrx48fL60vXbq0tJ5Nxz277YW2/2D7XduHbf+4mD7P9l7bx4vHufW3C6BX3RzGX5P004j4tqR7JW22/W1Jj0vaFxFLJe0rXgMYUh3DHhHnIuKt4vllSUckLZC0XtL24m3bJT1QU48A+uArfWa3vUjSckl/kjQ/Is4VpfOS5reZZ0zSWIUeAfRB12fjbc+RtEvSTyLiL9NrMfWrlS1/TDIixiNiRUSsqNQpgEq6Crvt2ZoK+o6I+F0x+YLtkaI+IuliPS0C6IeOh/G2LelpSUci4hfTSrslbZL0s+LxhVo6xIy2ZMmStrW777670rIfe+yxSvNn081n9lWSfiDpkO2DxbQnNBXynbZ/KOl9SQ/V0iGAvugY9oj4o6SWPzovaU1/2wFQF74uCyRB2IEkCDuQBGEHkiDsQBLc4opKbr/99tL6nj17el72li1bSusvvvhiz8vOiD07kARhB5Ig7EAShB1IgrADSRB2IAnCDiTBdXZUMjZW/otjt912W8/LfuWVV0rrUz+QhG6xZweSIOxAEoQdSIKwA0kQdiAJwg4kQdiBJLjOjlKrV68urT/66KMD6gRVsWcHkiDsQBKEHUiCsANJEHYgCcIOJEHYgSS6GZ99oaTfSJovKSSNR8R/2X5S0sOSPire+kRE/L6uRtGM++67r7Q+Z86cnpf93nvvldavXLnS87LxZd18qeaapJ9GxFu2vynpgO29Re2XEfGf9bUHoF+6GZ/9nKRzxfPLto9IWlB3YwD66yt9Zre9SNJySX8qJj1i+x3bz9ie22aeMdsTtieqtQqgiq7DbnuOpF2SfhIRf5G0TdISScs0tef/eav5ImI8IlZExIrq7QLoVVdhtz1bU0HfERG/k6SIuBARn0fEXyX9StLK+toEUFXHsNu2pKclHYmIX0ybPjLtbd+TNNn/9gD0Szdn41dJ+oGkQ7YPFtOekLTR9jJNXY47JelHNfSHGe7tt99uW1uzZk3pvJcuXep3O6l1czb+j5LcosQ1dWAG4Rt0QBKEHUiCsANJEHYgCcIOJEHYgSQ8yGFvbTPGLlCziGh1qZw9O5AFYQeSIOxAEoQdSIKwA0kQdiAJwg4kMeghm/8s6f1pr28upg2jYe1tWPuS6K1X/ezt9naFgX6p5ksrtyeG9bfphrW3Ye1LordeDao3DuOBJAg7kETTYR9veP1lhrW3Ye1LordeDaS3Rj+zAxicpvfsAAaEsANJNBJ222ttH7V9wvbjTfTQju1Ttg/ZPtj0+HTFGHoXbU9OmzbP9l7bx4vHlmPsNdTbk7bPFtvuoO11DfW20PYfbL9r+7DtHxfTG912JX0NZLsN/DO77VmSjkn6jqQzkt6UtDEi3h1oI23YPiVpRUQ0/gUM2/8i6Yqk30TEPxfTnpJ0KSJ+VvxDOTci/n1IentS0pWmh/EuRisamT7MuKQHJP2bGtx2JX09pAFstyb27CslnYiIkxFxVdJvJa1voI+hFxH7JV0/LMp6SduL59s19T/LwLXpbShExLmIeKt4flnSF8OMN7rtSvoaiCbCvkDS6Wmvz2i4xnsPSXtsH7A91nQzLcyPiHPF8/OS5jfZTAsdh/EepOuGGR+abdfL8OdVcYLuy1ZHxF2S/lXS5uJwdSjF1GewYbp22tUw3oPSYpjxv2ly2/U6/HlVTYT9rKSF015/q5g2FCLibPF4UdLzGr6hqC98MYJu8Xix4X7+ZpiG8W41zLiGYNs1Ofx5E2F/U9JS24ttf0PS9yXtbqCPL7F9Y3HiRLZvlPRdDd9Q1LslbSqeb5L0QoO9/J1hGca73TDjanjbNT78eUQM/E/SOk2dkX9P0n800UObvv5J0tvF3+Gme5P0nKYO6z7T1LmNH0q6SdI+Sccl/Z+keUPU239LOiTpHU0Fa6Sh3lZr6hD9HUkHi791TW+7kr4Gst34uiyQBCfogCQIO5AEYQeSIOxAEoQdSIKwA0kQdiCJ/wcZ+A96t/oPvwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "\n",
    "# A function to plot images\n",
    "def show_image(img):\n",
    "    image = img.reshape((28, 28))\n",
    "    plt.imshow(image, 'gray')\n",
    "\n",
    "\n",
    "# Reading The Train Set\n",
    "train_images_file = open('MNIST Dataset/train-images.idx3-ubyte', 'rb')\n",
    "train_images_file.seek(4)\n",
    "num_of_train_images = int.from_bytes(train_images_file.read(4), 'big')\n",
    "train_images_file.seek(16)\n",
    "\n",
    "train_labels_file = open('MNIST Dataset/train-labels.idx1-ubyte', 'rb')\n",
    "train_labels_file.seek(8)\n",
    "\n",
    "train_set = []\n",
    "train_img = []\n",
    "train_lable = []\n",
    "for n in range(num_of_train_images):\n",
    "    image = np.zeros((784, 1))\n",
    "    for i in range(784):\n",
    "        image[i, 0] = int.from_bytes(train_images_file.read(1), 'big') / 256\n",
    "\n",
    "    label_value = int.from_bytes(train_labels_file.read(1), 'big')\n",
    "    label = np.zeros((10, 1))\n",
    "    label[label_value, 0] = 1\n",
    "\n",
    "    train_set.append((image, label))\n",
    "    train_img.append(image.reshape(784))\n",
    "    train_lable.append(label.reshape(10))\n",
    "\n",
    "\n",
    "# Reading The Test Set\n",
    "test_images_file = open('MNIST Dataset/t10k-images.idx3-ubyte', 'rb')\n",
    "test_images_file.seek(4)\n",
    "\n",
    "test_labels_file = open('MNIST Dataset/t10k-labels.idx1-ubyte', 'rb')\n",
    "test_labels_file.seek(8)\n",
    "\n",
    "num_of_test_images = int.from_bytes(test_images_file.read(4), 'big')\n",
    "test_images_file.seek(16)\n",
    "\n",
    "test_set = []\n",
    "test_img = []\n",
    "shifted_test_img = []\n",
    "test_lable = []\n",
    "for n in range(num_of_test_images):\n",
    "    image = np.zeros((784, 1))\n",
    "    for i in range(784):\n",
    "        image[i] = int.from_bytes(test_images_file.read(1), 'big') / 256\n",
    "\n",
    "    label_value = int.from_bytes(test_labels_file.read(1), 'big')\n",
    "    label = np.zeros((10, 1))\n",
    "    label[label_value, 0] = 1\n",
    "\n",
    "    test_set.append((image, label))\n",
    "    test_img.append(image.reshape(784))\n",
    "    test_lable.append(label.reshape(10))\n",
    "    image = np.roll(image.reshape(28, 28), 4)\n",
    "    for i in range(28):\n",
    "        for j in range(28):\n",
    "            if j<4:\n",
    "                image[i][i]=0\n",
    "                \n",
    "    shifted_test_img.append(image.reshape(784))\n",
    "\n",
    "\n",
    "# Plotting an image\n",
    "show_image(test_img[0])\n",
    "plt.show()\n",
    "show_image(shifted_test_img[0])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "manual-recipe",
   "metadata": {},
   "source": [
    "# **Usefull Functions** "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "previous-transmission",
   "metadata": {},
   "source": [
    "I defined some usefull functions that we need throughout this notebook. There are some activation functions and some other functions defined in the below cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "checked-transfer",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "def derivative_sigmoid(x):\n",
    "    return  sigmoid(x) * (1 - sigmoid(x))\n",
    "\n",
    "def calculate_layers(inputs, w, b, activation=\"sigmoid\"):\n",
    " \n",
    "    if activation == \"sigmoid\":\n",
    "            acv_fun = sigmoid\n",
    "            dacv_fun = derivative_sigmoid\n",
    "    if activation ==  \"relu\":\n",
    "            acv_fun = rectified\n",
    "            dacv_fun = derivative_rectified\n",
    "    if activation ==  \"tanh\":\n",
    "            acv_fun = tanh\n",
    "            dacv_fun = derivative_tanh\n",
    "            \n",
    "            \n",
    "    current = [inputs]\n",
    "    i = 0\n",
    "    for (weight, bias) in zip(w, b):\n",
    "        if i == 2:\n",
    "            current.append(sigmoid(np.dot(weight, current[-1]) + bias))\n",
    "            break\n",
    "        current.append(acv_fun(np.dot(weight, current[-1]) + bias))\n",
    "        i += 1\n",
    "    return current\n",
    "\n",
    "def calculate_z(inputs, w, b, activation=\"sigmoid\"):\n",
    "  \n",
    "    if activation == \"sigmoid\":\n",
    "            acv_fun = sigmoid\n",
    "            dacv_fun = derivative_sigmoid\n",
    "    if activation ==  \"relu\":\n",
    "            acv_fun = rectified\n",
    "            dacv_fun = derivative_rectified\n",
    "    if activation ==  \"tanh\":\n",
    "            acv_fun = tanh\n",
    "            dacv_fun = derivative_tanh\n",
    "            \n",
    "    current = [inputs]\n",
    "    res = []\n",
    "    \n",
    "    for (weight, bias) in zip(w, b):           \n",
    "        res.append(np.dot(weight, current[-1]) + bias)\n",
    "        current.append(acv_fun(np.dot(weight, current[-1]) + bias))\n",
    "    return res\n",
    "\n",
    "def get_accuracy(output, labels):\n",
    "    max_index = 0\n",
    "    true_label = 0\n",
    "    count = 0.0\n",
    "    for i in range(np.shape(output)[0]):\n",
    "        maximum = -1\n",
    "        for j in range(np.shape(output)[1]):\n",
    "            if output[i][j] > maximum:\n",
    "                maximum = output[i][j]\n",
    "                max_index = j\n",
    "            if labels[i][j] == 1:\n",
    "                true_label = j\n",
    "\n",
    "        if max_index == true_label:\n",
    "            count += 1\n",
    "#         else:\n",
    "#             print(f\"{true_label} {max_index}\")\n",
    "    return count/np.shape(output)[0]\n",
    "\n",
    "\n",
    "# rectified linear function\n",
    "def rectified(x):\n",
    "    return np.maximum(0,x)\n",
    "\n",
    "def derivative_rectified(x):\n",
    "    x[x<=0] = 0\n",
    "    x[x>0] = 1\n",
    "    return x\n",
    "\n",
    "def tanh(x):\n",
    "    return np.tanh(x)\n",
    "\n",
    "def derivative_tanh(x):\n",
    "    return 1.0 - (tanh(x)**2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "controversial-shift",
   "metadata": {},
   "source": [
    "# **Before Training** "
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "comparative-boost",
   "metadata": {},
   "source": [
    "Before I start the training process I need to define some vectores and matrices for weights and biases of my neural network. My network in going to have 2 hidden layers, each containing 16 neurons and the last layer contains 10 neurons which is the number of digit classes we have.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "internal-waterproof",
   "metadata": {},
   "outputs": [],
   "source": [
    "def before_training(activation, size):\n",
    "\n",
    "    W01 = np.random.normal(0, .5, size=(16, 784))\n",
    "    W12 = np.random.normal(0, .5, size=(16, 16))\n",
    "    W23 = np.random.normal(0, .5, size=(10, 16))\n",
    "    b1 = np.zeros((16, 1))\n",
    "    b2 = np.zeros((16, 1))\n",
    "    b3 = np.zeros((10, 1))\n",
    "    weights = [W01, W12, W23]\n",
    "    biases = [b1, b2, b3]\n",
    "\n",
    "\n",
    "    imgs = np.array(test_img[:size]).transpose()\n",
    "    lables = np.array(test_lable[:size]).transpose()\n",
    "\n",
    "    output = calculate_layers(imgs, weights, biases, activation)[-1]\n",
    "    output = np.array(output).transpose()\n",
    "    lables = lables.transpose()\n",
    "\n",
    "\n",
    "    print(f\"Accuracy Before Training: {get_accuracy(output, lables)}\")\n",
    "    return W01, W12, W23, b1, b2, b3\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stuffed-edgar",
   "metadata": {},
   "source": [
    "# **Training**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quarterly-sellers",
   "metadata": {},
   "source": [
    "For training the network I used the Stochastic Gradient Decent for the cost function but the activation function comes in a variery of choices:\n",
    "1. Sigmoid Function\n",
    "2. Rectified Linear Unit (ReLU)\n",
    "3. Tanh\n",
    "\n",
    "Note: The purpose of using different types of activation functions was to compare their accuracy anf time and as the first method of trainig is by iteration and is very slow on it self these functions are just implemented on the vectorized training function and the iterative function uses the sigmoid function."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pretty-peoples",
   "metadata": {},
   "source": [
    "### **Training by iteration**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "retained-shadow",
   "metadata": {},
   "source": [
    "The goal in this part is to train the network by iterating through all indices in the weighs and biases matrices."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "motivated-attribute",
   "metadata": {},
   "outputs": [],
   "source": [
    "def learning_iterative_sgd(learning_rate, number_of_epochs, batch_size, w_b, number_of_samples):\n",
    "    (W01, W12, W23, b1, b2, b3) = w_b\n",
    "    cost_by_epochs = [[0]*10]*number_of_epochs\n",
    "    samples = train_set[:number_of_samples]\n",
    "    for n in range(number_of_epochs):\n",
    "        np.random.shuffle(samples)\n",
    "        batches = [samples[x:x+batch_size] for x in range(0, number_of_samples, batch_size)]\n",
    "\n",
    "        for batch in batches:\n",
    "            grad_W01 = np.zeros((16, 784))\n",
    "            grad_W12 = np.zeros((16, 16))\n",
    "            grad_W23 = np.zeros((10, 16))\n",
    "\n",
    "            grad_b1 = np.zeros((16, 1))\n",
    "            grad_b2 = np.zeros((16, 1))\n",
    "            grad_b3 = np.zeros((10, 1))\n",
    "\n",
    "            grdad_w = [grad_W01, grad_W12, grad_W23]\n",
    "            grad_b = [grad_b1, grad_b2, grad_b3]\n",
    "\n",
    "\n",
    "            for data in batch:\n",
    "                image = data[0]\n",
    "                label = data[1]\n",
    "\n",
    "                grad_a0 = np.zeros((784, 1))\n",
    "                grad_a1 = np.zeros((16, 1))\n",
    "                grad_a2 = np.zeros((16, 1))\n",
    "\n",
    "                weights = [W01, W12, W23]\n",
    "                biases = [b1, b2, b3]\n",
    "\n",
    "                layers = calculate_layers(np.array(image), weights, biases)\n",
    "                z = calculate_z(np.array(image), weights, biases)\n",
    "\n",
    "                cost = [0.0] * 10\n",
    "                for i in range(10):\n",
    "                    cost[i] += (1.0/number_of_samples) *( (layers[3][i] - label[i]) ** 2 )\n",
    "                cost_by_epochs[n] = [x + y for (x, y) in zip(cost, cost_by_epochs[n])]\n",
    "\n",
    "\n",
    "                # layer number 3\n",
    "                for i in range(np.shape(W23)[0]):\n",
    "                    eq =  2*(layers[3][i]-label[i]) * derivative_sigmoid(z[2][i])\n",
    "                    grad_b3[i] += eq\n",
    "                    for j in range(np.shape(W23)[1]):\n",
    "                        grad_W23[i][j] += eq * layers[2][j]\n",
    "\n",
    "\n",
    "\n",
    "                # layer number 2\n",
    "                # dc/da\n",
    "                for i in range(16):\n",
    "                    for x in range(10):\n",
    "                        grad_a2[i] += 2 * (layers[3][x]-label[x]) * derivative_sigmoid(z[2][x]) * W23[x][i]\n",
    "\n",
    "                 # dc/db , dc/dw\n",
    "                for i in range(np.shape(W12)[0]):\n",
    "                    eq = 2* grad_a2[i] * derivative_sigmoid(z[1][i])\n",
    "                    grad_b2[i] += eq\n",
    "                    for j in range(np.shape(W12)[1]):\n",
    "                        grad_W12[i][j] += eq * layers[1][j]\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "                # layer number 1\n",
    "                # dc/da\n",
    "                for i in range(16):\n",
    "                    for x in range(16):\n",
    "                        grad_a1[i] += 2 * grad_a2[x] * derivative_sigmoid(z[1][x]) * W12[x][i]\n",
    "                     \n",
    "\n",
    "                 # dc/db , dc/dw\n",
    "                for i in range(np.shape(W01)[0]):\n",
    "                    eq = 2* grad_a2[i] * derivative_sigmoid(z[0][i])\n",
    "                    grad_b1[i] += eq\n",
    "                    for j in range(np.shape(W01)[1]):\n",
    "                        grad_W01[i][j] += eq * layers[0][j]\n",
    "\n",
    "\n",
    "            W01 -= learning_rate * (grad_W01 / batch_size)\n",
    "            W12 -= learning_rate * (grad_W12 / batch_size)\n",
    "            W23 -= learning_rate * (grad_W23 / batch_size)\n",
    "\n",
    "            b1 -= learning_rate * (grad_b1 / batch_size)\n",
    "            b2 -= learning_rate * (grad_b2 / batch_size)\n",
    "            b3 -= learning_rate * (grad_b3 / batch_size)\n",
    "\n",
    "    return cost_by_epochs, [W01, W12, W23], [b1, b2, b3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "impressed-indiana",
   "metadata": {},
   "source": [
    "### **Training in vectorized form**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "distinct-gallery",
   "metadata": {},
   "source": [
    "This second approach to training the my neural network is fatser because it doeas all the calculations by matrix operations and it consumes much less amount of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "pediatric-relief",
   "metadata": {},
   "outputs": [],
   "source": [
    "def learning_vectorized_sgd(learning_rate, number_of_epochs, batch_size, w_b, number_of_samples, activation=\"sigmoid\"):\n",
    "    \n",
    " \n",
    "    if activation == \"sigmoid\":\n",
    "            acv_fun = sigmoid\n",
    "            dacv_fun = derivative_sigmoid\n",
    "    if activation ==  \"relu\":\n",
    "            acv_fun = rectified\n",
    "            dacv_fun = derivative_rectified\n",
    "    if activation ==  \"tanh\":\n",
    "            acv_fun = tanh\n",
    "            dacv_fun = derivative_tanh\n",
    "    \n",
    "    (W01, W12, W23, b1, b2, b3) = w_b\n",
    "    cost_by_epochs = [[0]*10]*number_of_epochs\n",
    "    samples = train_set[:number_of_samples]\n",
    "    \n",
    "    for n in range(number_of_epochs):\n",
    "        np.random.shuffle(samples)\n",
    "        batches = [samples[x:x+batch_size] for x in range(0, number_of_samples, batch_size)]\n",
    "\n",
    "        for batch in batches:\n",
    "            grad_W01 = np.zeros((16, 784))\n",
    "            grad_W12 = np.zeros((16, 16))\n",
    "            grad_W23 = np.zeros((10, 16))\n",
    "\n",
    "            grad_b1 = np.zeros((16, 1))\n",
    "            grad_b2 = np.zeros((16, 1))\n",
    "            grad_b3 = np.zeros((10, 1))\n",
    "\n",
    "            grdad_w = [grad_W01, grad_W12, grad_W23]\n",
    "            grad_b = [grad_b1, grad_b2, grad_b3]\n",
    "\n",
    "\n",
    "            for data in batch:\n",
    "                image = data[0]\n",
    "                label = data[1]\n",
    "\n",
    "                grad_a0 = np.zeros((784, 1))\n",
    "                grad_a1 = np.zeros((16, 1))\n",
    "                grad_a2 = np.zeros((16, 1))\n",
    "\n",
    "                weights = [W01, W12, W23]\n",
    "                biases = [b1, b2, b3]\n",
    "\n",
    "                layers = calculate_layers(np.array(image), weights, biases, activation)\n",
    "                z = calculate_z(np.array(image), weights, biases, activation)\n",
    "\n",
    "\n",
    "                cost = [0.0] * 10\n",
    "                for i in range(10):\n",
    "                    cost[i] += (1.0/number_of_samples) *( (layers[3][i] - label[i]) ** 2 )\n",
    "                cost_by_epochs[n] = [x + y for (x, y) in zip(cost, cost_by_epochs[n])]\n",
    "\n",
    " \n",
    "\n",
    "                # layer number 3\n",
    "                grad_b3 += (2*(layers[3]-label) * derivative_sigmoid(z[2]))\n",
    "                grad_W23 +=  2*((layers[3]-label) * derivative_sigmoid(z[2])) @ np.transpose(layers[2])\n",
    "                \n",
    "\n",
    "\n",
    "                # layer number 2\n",
    "                # dc/da                                                      \n",
    "                grad_a2 = np.transpose(W23) @ (2* derivative_sigmoid(z[2]) * (layers[3]-label))\n",
    "\n",
    "                # dc/db , dc/dw\n",
    "                grad_b2 += 2* grad_a2 * dacv_fun(z[1])\n",
    "                grad_W12 += 2* grad_a2 * dacv_fun(z[1])  @ np.transpose(layers[1])\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "                # layer number 1\n",
    "                # dc/da\n",
    "                grad_a1 = np.transpose(W12) @ (2 * dacv_fun(z[1]) * grad_a2)\n",
    "\n",
    "\n",
    "                 # dc/db , dc/dw\n",
    "                grad_b1 +=  2* grad_a2 * dacv_fun(z[1])        \n",
    "                grad_W01 +=  2* grad_a1 * dacv_fun(z[0]) @ np.transpose(layers[0])\n",
    "\n",
    "\n",
    "            W01 -= learning_rate * (grad_W01 / batch_size)\n",
    "            W12 -= learning_rate * (grad_W12 / batch_size)\n",
    "            W23 -= learning_rate * (grad_W23 / batch_size)\n",
    "\n",
    "            b1 -= learning_rate * (grad_b1 / batch_size)\n",
    "            b2 -= learning_rate * (grad_b2 / batch_size)\n",
    "            b3 -= learning_rate * (grad_b3 / batch_size)\n",
    "            \n",
    "\n",
    "\n",
    "    return cost_by_epochs, [W01, W12, W23], [b1, b2, b3]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "irish-integer",
   "metadata": {},
   "source": [
    "# Defining Some function to show results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "electronic-trigger",
   "metadata": {},
   "source": [
    "The first function shows the result of feeding a number of images from the test set and printing the accuracy of our network to the result of classifying these images. and this function also plots the cost changes during the trainig phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "impossible-slave",
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_results_test_data(wghs, bias, size, activation):\n",
    "\n",
    "    imgs = np.array(test_img[:size]).transpose()\n",
    "    lables = np.array(test_lable[:size]).transpose()\n",
    "\n",
    "    output = calculate_layers(imgs, wghs, bias, activation)[-1]\n",
    "    output = np.array(output).transpose()\n",
    "    lables = lables.transpose()\n",
    "\n",
    "    print(f\"Accuracy After Training: {get_accuracy(output, lables)}\")\n",
    "    plt.plot([np.linalg.norm(x)**2 for x in cost_by_epochs])\n",
    "    plt.show"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "asian-montgomery",
   "metadata": {},
   "source": [
    "The second function shows the result of feeding a number of shifted images from the test set and printing the accuracy of our network to the result of classifying these images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "cathedral-senator",
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_results_4pixel_shifed(wghs, bias, size, activation):\n",
    "\n",
    "    imgs = np.array(shifted_test_img[:size]).transpose()\n",
    "    lables = np.array(test_lable[:size]).transpose()\n",
    "\n",
    "    output = calculate_layers(imgs, wghs, bias, activation)[-1]\n",
    "    output = np.array(output).transpose()\n",
    "    lables = lables.transpose()\n",
    "#     print(output)\n",
    "\n",
    "    print(f\"Accuracy After Training: {get_accuracy(output, lables)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "enclosed-filename",
   "metadata": {},
   "source": [
    "The last function shows the result of feeding a number of images from the train set and printing the accuracy of our network to the result of classifying these images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "second-picture",
   "metadata": {},
   "outputs": [],
   "source": [
    "def show_results_training_data(wghs, bias, size, activation):\n",
    "\n",
    "    imgs = np.array(train_img[:size]).transpose()\n",
    "    lables = np.array(train_lable[:size]).transpose()\n",
    "\n",
    "    output = calculate_layers(imgs, wghs, bias, activation)[-1]\n",
    "    output = np.array(output).transpose()\n",
    "    lables = lables.transpose()\n",
    "#     print(output)\n",
    "\n",
    "    print(f\"Accuracy After Training: {get_accuracy(output, lables)}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "operating-munich",
   "metadata": {},
   "source": [
    "# Testing our NN"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "tracked-subscription",
   "metadata": {},
   "source": [
    "### Testing our iterating function for trainig the network"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "declared-receipt",
   "metadata": {},
   "source": [
    "First thing we do is to randomly create our wights and biases matrices and to calculate the accuracy before doing ant trainings on the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "incoming-witness",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Before Training: 0.11\n"
     ]
    }
   ],
   "source": [
    "W01, W12, W23, b1, b2, b3 = before_training(\"sigmoid\", 100)\n",
    "weights_biases = (W01, W12, W23, b1, b2, b3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "lovely-footage",
   "metadata": {},
   "source": [
    "Next we train our network by feeding the first 100 images into the neural network from our train set and then training it by backpropagation. \n",
    "Here are some measurements we used to train our network:\n",
    "\n",
    "Learning Rate: 1 <br>\n",
    "Number of Epochs: 20 <br>\n",
    "Batch Size: 10 <br>\n",
    "Number of samples: 100 <br>\n",
    "Activation function: Sigmoid <br>\n",
    "_________________________\n",
    "\n",
    "Accuracy before training: 0.15 <br>\n",
    "Wall time of running the training function: 1min 11s <br>\n",
    "Accuracy after training: 0.65 <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "foreign-pastor",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1min 6s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "cost_by_epochs, weights1, biases1 = learning_iterative_sgd(learning_rate=1, number_of_epochs=20, batch_size=10, w_b=weights_biases, number_of_samples=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "expanded-garden",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy After Training: 0.61\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAk9ElEQVR4nO3deXRV9b338fc3AwmQgUBCSEggINEEFREiWgdEUYvWiq2oaLvUDo/XWjvdevt4b5/b6/Wu+9zaxw7e1tpyHTparVgrrSgOiDgiIUwCARkCSSAQICRMIdP3+eOcsE7TEA5mOMk5n9daWZyz9++c/c3m5JOd3/7t3zZ3R0REoldcpAsQEZHepaAXEYlyCnoRkSinoBcRiXIKehGRKJcQ6QI6yszM9IKCgkiXISIyoKxYsWKvu2d1tq7fBX1BQQGlpaWRLkNEZEAxs+0nWqeuGxGRKKegFxGJcgp6EZEoF1bQm9ksM9toZpvN7L5O1k83szIzazGzOZ2sTzOzKjP7WU8ULSIi4Ttp0JtZPPAIcDUwEbjFzCZ2aLYDuAN46gRv8x/A0o9fpoiIfFzhHNFPAza7+1Z3bwKeBmaHNnD3CndfA7R1fLGZTQWygVd6oF4RETlF4QT9aKAy5HlVcNlJmVkc8EPg3pO0u9PMSs2stLa2Npy3FhGRMPX2ydi7gYXuXtVVI3ef5+4l7l6SldXpeP+Tqj/SzMOvfcTaqvqP9XoRkWgVzgVT1UB+yPO84LJwfAK4xMzuBlKAQWZ2yN3/7oRud8XFwY9f20RCvHF2XnpPv72IyIAVTtAvBwrNbByBgJ8L3BrOm7v759ofm9kdQElvhDxAanIi+cMHs2FXQ2+8vYjIgHXSrht3bwHuARYBG4A/uvs6M3vAzK4DMLPzzKwKuBH4pZmt682iT6R4VJqCXkSkg7DmunH3hcDCDsu+F/J4OYEuna7e41fAr065wlNQlJPGaxt209jcSnJifG9uSkRkwIiqK2Mn5qTS5rBp98FIlyIi0m9EVdAX56QBqPtGRCREVAV9fsYQhg6KZ8MuHdGLiLSLqqCPizPOGJWqI3oRkRBRFfQQ6L7ZsKsBd490KSIi/UJUBn1DYws76xsjXYqISL8QhUGfCsCGneq+ERGBKAz6M0YFRt6U1yjoRUQgCoM+JSmBsSOGaOSNiEhQ1AU9QJFG3oiIHBeVQV+ck8a2fYc52tQa6VJERCIuaoPeHTZqKgQRkSgN+lGaCkFEpF1UBn1exmBSkhIU9CIiRGnQx8UZRaNSKdfIGxGR6Ax6CE6FUKOpEEREojboi3JSOdjYQlXd0UiXIiISUVEb9O1z05fXqPtGRGJb1Ab9GdmpmGnkjYhI1Ab90KQExg4foqAXkZgXtUEPge4bdd2ISKyL+qCv2HeYI00tkS5FRCRiojroi0al4q4TsiIS26I66NtH3qifXkRiWVhBb2azzGyjmW02s/s6WT/dzMrMrMXM5oQsn2xm75nZOjNbY2Y392TxJ5OXMZjUpARdISsiMe2kQW9m8cAjwNXAROAWM5vYodkO4A7gqQ7LjwC3ufuZwCzgJ2Y2rJs1h83MKMrR3PQiEtvCOaKfBmx2963u3gQ8DcwObeDuFe6+BmjrsHyTu38UfLwT2ANk9UjlYWofedPWpqkQRCQ2hRP0o4HKkOdVwWWnxMymAYOALZ2su9PMSs2stLa29lTfukvFOWkcOtZC9QFNhSAisalPTsaaWQ7wW+AL7t7Wcb27z3P3Encvycrq2QP+9hOy69V9IyIxKpygrwbyQ57nBZeFxczSgBeB77r7+6dWXvednp2iqRBEJKaFE/TLgUIzG2dmg4C5wIJw3jzY/nngN+4+/+OX+fENGZTAuBFDNfJGRGLWSYPe3VuAe4BFwAbgj+6+zsweMLPrAMzsPDOrAm4Efmlm64IvvwmYDtxhZquCX5N74xvpSvvc9CIisSghnEbuvhBY2GHZ90IeLyfQpdPxdb8DftfNGrutaFQqL67dxaFjLaQkhfUti4hEjai+MrZd+wnZjTqqF5EYFBtBn9s+FYL66UUk9sRE0OemJ5OWnKCRNyISk2Ii6ANTIaQp6EUkJsVE0ANMzEljo6ZCEJEYFDNBX5yTyuGmVirrjkS6FBGRPhUzQV80SnPTi0hsipmgP2NUKnGmkTciEntiJuiTE+MZlzlUR/QiEnNiJuiBwMgbXTQlIjEmpoJ+Yk4alfuPcrCxOdKliIj0mZgK+uKcVAA21qifXkRiR0wFvUbeiEgsiqmgz0lPJn1wIus18kZEYkhMBb2ZUZyTSrlOyIpIDImpoIdA942mQhCRWBJzQT8xJ40jTa1s36+pEEQkNsRc0LffhEQnZEUkVsRc0BdmpxBnUK6gF5EYEXNBn5wYz/isFI28EZGYEXNBD4HuG3XdiEisiNGgT6X6wFEaNBWCiMSA2Az64BWy5eq+EZEYEJtBr5E3IhJDwgp6M5tlZhvNbLOZ3dfJ+ulmVmZmLWY2p8O6283so+DX7T1VeHdkpyWRMSRRV8iKSEw4adCbWTzwCHA1MBG4xcwmdmi2A7gDeKrDa4cD/wacD0wD/s3MMrpfdveYGUWj0jTyRkRiQjhH9NOAze6+1d2bgKeB2aEN3L3C3dcAbR1e+0ngVXff7+51wKvArB6ou9uKc9LYWNNAq6ZCEJEoF07QjwYqQ55XBZeFI6zXmtmdZlZqZqW1tbVhvnX3FOek0tjcRsW+w32yPRGRSOkXJ2PdfZ67l7h7SVZWVp9ss/2ErEbeiEi0Cyfoq4H8kOd5wWXh6M5re9WEkSnEx5lG3ohI1Asn6JcDhWY2zswGAXOBBWG+/yLgKjPLCJ6EvSq4LOKSE+M5LWuogl5Eot5Jg97dW4B7CAT0BuCP7r7OzB4ws+sAzOw8M6sCbgR+aWbrgq/dD/wHgV8Wy4EHgsv6heKcNMp1/1gRiXIJ4TRy94XAwg7LvhfyeDmBbpnOXvsE8EQ3auw1RaPSeGHVTuqPNJM+JDHS5YiI9Ip+cTI2UopzUgHYoAunRCSKxXTQTzw+8kZBLyLRK6aDPis1ieFDB7FBQyxFJIrFdNCbGcU5qeq6EZGoFtNBD4EpizfWHKSltePsDSIi0UFBn5PGsZY2KvYdiXQpIiK9IuaDvqh95I1OyIpIlIr5oJ8wMoUETYUgIlEs5oM+KSGeCSNTdIWsiEStmA96gKJRqTqiF5GopaAncEJ2V30jB440RboUEZEep6An9Gbh6r4RkeijoEcjb0QkuinogZGpyWSmDFLQi0hUUtAHFeekaSoEEYlKCvqgolGpbNp9SFMhiEjUUdAHFeek0dTSxra9hyNdiohIj1LQB7WPvPnJax/xp7IqymsaaNbRvYhEgbBuJRgLCkemMP30LF7bsJsX1+4CYFBCHGdkpzIxJ42JuWmcmZtGUU4aKUnabSIycJi7R7qGv1FSUuKlpaUR235La6D7Zv2uBtbtbGD9zgbW7ayn7kjz8TYFI4ZwZm46E3PTjv8SGJmahJlFrG4RiW1mtsLdSzpbp0PTDhLi4yjMTqUwO5XZk0cD4O7UNDSyPhj863c1sLa6/viRP0BmyiCKc9KYnD+MKWMzmJKfoRuOi0i/oKAPg5mRkz6YnPTBzCzOPr68obGZ8l0HWb+znnU7G/hwZwM/X7KF1rbAX0mFI1OYMiaDqWMzmDI2g/GZQ4mL01G/iPQtBX03pCUnMm3ccKaNG3582eFjLayuOkDZ9jrKdhzg5XU1PFNaCUD64ESmjBkWCP4xGZyTP4yh6u8XkV4WVsqY2SzgYSAeeMzdv99hfRLwG2AqsA+42d0rzCwReAyYEtzWb9z9v3qw/n5naFICF56WyYWnZQLQ1uZs3XuYsh11lG2vY8X2Ot7YWAtAnAVG+7Qf9U8dm0H+8CGRLF9EotBJT8aaWTywCbgSqAKWA7e4+/qQNncDk9z9LjObC3zG3W82s1uB69x9rpkNAdYDM9y94kTbi/TJ2L5Qf6SZlZXB4N9Rx6odBzjc1ArAxRMy+bdPT6QwOzXCVYrIQNLdk7HTgM3uvjX4Zk8DswmEdrvZwP3Bx/OBn1lgCIoDQ80sARgMNAExP89A+pBEZpwxkhlnjASgtc3ZWHOQNzfV8uiSzcx6+C1u/0QB37iikPTBOqErIt0TzgVTo4HKkOdVwWWdtnH3FqAeGEEg9A8Du4AdwEPuvr+bNUed+DhjYm4aX5lxGm/cO4ObSvJ58t1tXP7QEp7+YMfxk7siIh9Hb18ZOw1oBXKBccC3zWx8x0ZmdqeZlZpZaW1tbS+X1L+NSEnivz57Nn+552LGZQ7lvj+t5fpH3mHFdv1+FJGPJ5ygrwbyQ57nBZd12ibYTZNO4KTsrcDL7t7s7nuAd4C/60Ny93nuXuLuJVlZWaf+XUShs0an8+xdn+DhuZOpPXiMGx59j289s4rdDY2RLk1EBphwgn45UGhm48xsEDAXWNChzQLg9uDjOcBiD5zl3QFcDmBmQ4ELgPKeKDwWmBmzJ4/m9W9fylcvO40X1+zisoeW8OiSLRxraY10eSIyQJw06IN97vcAi4ANwB/dfZ2ZPWBm1wWbPQ6MMLPNwD8C9wWXPwKkmNk6Ar8wnnT3NT39TUS7oUkJ/NMni3j1H6dz0YRMHny5nE/+eCmvb9hNf5vCQkT6H811MwAt3VTLv/9lHVtqDzPjjCz+9dqJnJaVEumyRCSCuhpeqWmKB6Dpp2fx8jen838+VcyKijo++eOl/N+FGzjY2HzyF4tIzFHQD1CJ8XF8+ZLxLL53BjdMyeN/3trKZQ+9yVPLdmgefRH5Gwr6AS4rNYkH50ziha9exNgRQ/iX59dyxY/e5IVV1bRp/L2IoKCPGpPyhjH/rk/wxB0lDBmUwDeeXsXVD7/FK+tqdMJWJMYp6KOImXF5UTYvfu1ifnrLuTS1tnHnb1dw/c/f5Z3NeyNdnohEiII+CsXFGZ8+J5dXvzWdH9wwidqGRj732DJumfc+K7bXRbo8EeljGl4ZA461tPLUsh088sZm9h5q4orikXz7qjOO3xBdRAa+roZXKuhjyOFjLfzq3Qp++eYWGhpb+PQ5uXzrikLGawy+yICnoJe/UX+kmXlvbeGJtytoam3jxql5fH1mIbnDBke6NBH5mBT00qnag8f4+ZLN/P79HQB87oIx3HPZBEakJEW4MhE5VQp66VL1gaP892sfMb+siiGJ8Xz18gnccWEByYnxkS5NRMKkKRCkS6OHDebBOZNY9M3pnD9+ON9/qZwrfvQmf12zU2PwRaKAgl6OmzAyhcduP4/ff/l8UpMTueepldzw6Lus3KEhmSIDmYJe/s5FEzL569cu5gc3TKKy7iif+fm7fP0PK6mqOxLp0kTkY1DQS6fi44ybzstnyb0z+PrlE3hlfQ2X//BNfvByuWbJFBlgFPTSpaFJCfzjVWew+NszuPbsHH6+ZAuXPbSEp5btoEWzZIoMCAp6CUvusMH86ObJLLjnIsZnpvAvz6/lU//9Nks3xfbN3EUGAgW9nJJJecN45h8u4Befn0JjSyu3PfEBtz/xAZt2H4x0aSJyAgp6OWVmxqyzcnjlW4G7XJXtqGPWT5by3efXsvfQsUiXJyIdKOjlY0tKiOfLl4znzX+6jNs+UcDTyyu57KElPP72Nt3lSqQfUdBLtw0fOoj7rzuTRd+czrljMviPv67n6offUv+9SD+hoJceM2FkCr/+wnk8dlsJza1t3PbEB3z516Vs33c40qWJxDQFvfQoM+OKidm88q3p/O9ZRby3ZS9X/mgpD75czuFjLZEuTyQmKeilVyQlxPOVGaex+N4ZXHtODo8Gx98/v7JK8+eI9LGwgt7MZpnZRjPbbGb3dbI+ycyeCa5fZmYFIesmmdl7ZrbOzNaaWXIP1i/9XHZaMj+6aTJ/uvtCctKT+dYzq7nh0XdZU3Ug0qWJxIyTBr2ZxQOPAFcDE4FbzGxih2ZfAurcfQLwY+DB4GsTgN8Bd7n7mcAMQNfPx6ApYzJ4/u6L+MGcSezYf5TZj7zDd+avpvaghmOK9LZwjuinAZvdfau7NwFPA7M7tJkN/Dr4eD4w08wMuApY4+6rAdx9n7u39kzpMtDExRk3leTzxr2X8r8uGc/zK6u5/KEl/M/SrTS1aDimSG8JJ+hHA5Uhz6uCyzpt4+4tQD0wAjgdcDNbZGZlZvadzjZgZneaWamZldbWakhetEtNTuRfrilm0TenU1KQwX8u3MCsh5fyxsY9kS5NJCr19snYBOBi4HPBfz9jZjM7NnL3ee5e4u4lWVlZvVyS9Bfjs1J48gvTePKO83CHLzy5nFvmvc/STbU6YSvSg8IJ+mogP+R5XnBZp22C/fLpwD4CR/9L3X2vux8BFgJTulu0RJfLikay6JvT+ddrJ7J17yFue+IDrv3p2/xl9U5a2xT4It0VTtAvBwrNbJyZDQLmAgs6tFkA3B58PAdY7IFDskXA2WY2JPgL4FJgfc+ULtFkUEIcX7p4HEu/cxk/uGESR5tb+dofVnL5D5fw+2XbaWzWqR2Rjyusm4Ob2TXAT4B44Al3/08zewAodfcFwSGTvwXOBfYDc919a/C1nwf+GXBgobt32k/fTjcHF4C2NueV9bt5dMlmVlfVk5mSxJcuHsfnLhhDWnJipMsT6Xe6ujl4WEHflxT0EsrdeW/rPh5dsoW3PtpLalICn7tgLF+8qICRabokQ6Sdgl6iwofV9fzizS0sXLuLhLg4bpiaxz9MH09B5tBIlyYScQp6iSoVew8z762tzF9RRUtrG1efncNXLj2Ns0anR7o0kYhR0EtU2nOwkSffqeB3723n4LEWLinM5KuXTeCC8SMiXZpIn1PQS1RraGzmqWU7ePztbdQePMa0ccP5xsxCLjxtBIELtEWin4JeYkJjcytPf7CDR9/cwu6GY5SMzeDrMwu5pDBTgS9RT0EvMaWxuZVnSyv5+ZIt7KpvZHL+ML5xRSEzTs9S4EvUUtBLTDrW0spzK6p55I3NVB84yqS8dL5+eSEzi0cq8CXqKOglpjW1tPH8yip+9sZmKvcf5czcNL4+s5Ari7OJi1PgS3RQ0IsAza1tvLBqJz9b/BEV+45QNCqVr88sZNaZoxT4MuAp6EVCtLS28Zc1O/np4s1srT3M6dkpfO3yQq45O4d4Bb4MUAp6kU60tjl/DQb+5j2HmDAyhS9cVMCnzs5h2JBBkS5P5JQo6EW60NbmvPRhDT9d/BHlNQdJjDcuPX0k15+byxXF2SQnxke6RJGT6iroE/q6GJH+Ji7O+NSkHK45exTrdjbwwqpqFqzeyWsbdpOSlMAnzxzF9efmcuFpmerakQFJR/QinWhtc5Zt3cefV1Xz0toaDh5rISs1iU9PyuX6c3M5e3S6hmhKv6KuG5FuaGxu5Y3yPfx5VTVvlNfS1NrG+MyhzJ48muvPzWXsCM2eKZGnoBfpIfVHmln44S7+vLKaZdv2AzA5fxjXT87l2nNyyUxJinCFEqsU9CK9YOeBoyxYvZM/r6ymvOYg8XHG9MJM5kzNZ2bxSJ3ElT6loBfpZRtrDvL8ymr+vLKamoZG0pIT+PQ5ucyZmsfk/GHqz5dep6AX6SOtbc67W/Yyf0UVL39Yw7GWNsZnDWXO1Dw+e24eo9J1+0PpHQp6kQhoaGxm4ZpdPFdWxfKKOszg4gmZzJmax1UTRzF4kLp2pOco6EUirGLvYf5UVsVzZdVUHzhKalICn5qUw5ypeUwdm6GuHek2Bb1IP9HW5ry/bR/zV1Tx0toajja3UjBiCJ+dksdnp4wmL2NIpEuUAUpBL9IPHTrWwktrA107728NDNW88LQR3FSSz6yzRmnUjpwSBb1IP1e5/wjPlVUxf0UVVXVHSU1OYPbkXG4qyddVuBKWbge9mc0CHgbigcfc/fsd1icBvwGmAvuAm929ImT9GGA9cL+7P9TVthT0Esva2pz3t+7jj6WVvBQctVM0KpUbS/K5fnIuI3RBlpxAt4LezOKBTcCVQBWwHLjF3deHtLkbmOTud5nZXOAz7n5zyPr5gAPLFPQi4ak/2sxfVu/k2dJKVlfVkxhvXFGczU0l+VxSmElCfFykS5R+pLuzV04DNrv71uCbPQ3MJnCE3m42cH/w8XzgZ2Zm7u5mdj2wDTj88coXiU3pgxP5/AVj+fwFYymvaeDZ0iqeX1nNSx/WkJ2WxA1T8ripJJ+CTM21I10L55BgNFAZ8rwquKzTNu7eAtQDI8wsBfjfwL93tQEzu9PMSs2stLa2NtzaRWJG0ag0/vXaibz/zzP5xeencGZuOr94cwszHlrCTb98j/krqjjS1BLpMqWf6u356O8Hfuzuh7o6meTu84B5EOi66eWaRAasQQlxzDorh1ln5bC7oZHnyqp4trSKe59dzb+98CFXTsxmZnE200/PIn1wYqTLlX4inKCvBvJDnucFl3XWpsrMEoB0AidlzwfmmNkPgGFAm5k1uvvPulu4SKzLTkvm7hkT+Mqlp1G6vY4/Lq/ktQ27+fOqncTHGecVZDCzKJvLi0cyPnOoRu7EsHBOxiYQOBk7k0CgLwdudfd1IW2+CpwdcjL2s+5+U4f3uR84pJOxIr2ntc1ZVVnH6xv2sLh8D+U1BwEoGDGEy4pGMrMom2njhjMoQSdyo023Tsa6e4uZ3QMsIjC88gl3X2dmDwCl7r4AeBz4rZltBvYDc3uufBEJV3ycMXXscKaOHc53ZhVRfeAoi8v3sHjDbn6/bAdPvlNBSlIClxRmcnnRSC4rGqk59GOALpgSiRFHmlp4d/M+Xi/fw+Ly3exuOIYZnJM3jJlFI7m8eCQTc9LUxTNA6cpYEfkb7s66nQ0sLt/D6+V7WF15AIDxWUP54kXjuGFKnmbXHGAU9CLSpT0HG1m8YQ+/W7adD6sbGDYkkVunjeH2CwvITtMc+gOBgl5EwuLufLBtP4+/vY1XN+wmIc64dlIuX7p4HGeNTo90edKF7l4ZKyIxwsw4f/wIzh8/gu37DvPkOxU8W1rJ8yurmTZuOF+6eBxXFGcTH6d+/IFER/Qi0qX6o838cXklv3q3guoDRxk7Ygh3XFjAjSX5pCTpWLG/UNeNiHRbS2sbi9bt5vG3t1K24wCpyQnMPS+f2y8s0A1T+gEFvYj0qLIddTz+9jZe/rAGgFlnjuKLF49j6tiMCFcWu9RHLyI9asqYDKbcmkH1gaP8+t0K/vDBDl5cu4uzR6dzY0ke152Ty7AhgyJdpgTpiF5Euu3QsRaeW1HFHz7YQXnNQQbFx3HlmYG58y+ekKmTt31AXTci0ifaL8R6trSSF1bv5MCRZnLSk/nslNHMmZrPOM2d32sU9CLS5461tPLa+j08u6KSpZtqaXM4ryCDG6fmc82kHI3Y6WEKehGJqJr6Rv60sor5pVVs3XuYIYPiufqsHG4syeP8ccM1v04PUNCLSL/g7pTtqOPZ0ir+umYXh461MGb4EOZMzeOGqXmMHjY40iUOWAp6Eel3jjS18PKHNTxbWsV7W/dhBpPzh3FewXCmjs2gZGwGIzSFctgU9CLSr1XuP8JzZVW8/dFe1lTV09TaBsD4zKFMHZsRCP+CDN0pqwsKehEZMBqbW/mwup7S7XWUVuxnxfY66o40AzB86KDjR/slBRmcNTqdpARNpwy6YEpEBpDkxHhKCoZTUjAcLj0Nd2dL7WFKK/ZTur2OFdvreHX9biBws/Rz8tKZOnY45xVkUFIwXDdF74SO6EVkwKk9eIwV2+tYsX0/yyvqWLeznuZWP35T9CuKs5lZnB1T4/bVdSMiUa2xuZVVlQdYuqmW1zfsYePuwE3Rx2cN5YribC4vGknJ2AwS4qP3pugKehGJKZX7j7C4fA+vbdjN+1v30dzqpA9OZMYZWcwszubSwizSh0RXF4+CXkRi1qFjLby1qZbXy/fwRvke9h1uisouHgW9iAjQ2uasqjzA6xt2s7h8D+U1wS6ezKHMLB7JjDNGMmVMxoC8MbqCXkSkE6FdPMu27qeptY3EeGNS3jCmjRvO+eMCo38Gwrw8CnoRkZM4dKyF0or9LNu2n2Vb97Gmqp6WtsBInrNy0wL30g0Gf38cwtntoDezWcDDQDzwmLt/v8P6JOA3wFRgH3Czu1eY2ZXA94FBQBPwT+6+uKttKehFpD840tRC2fYDLNu2j2Vb97Oq8gBNrW2YwcScNM4fN+L4UX/G0MjfZKVbQW9m8cAm4EqgClgO3OLu60Pa3A1Mcve7zGwu8Bl3v9nMzgV2u/tOMzsLWOTuo7vanoJeRPqj9iGcy7buZ9m2fZTtqKOxOTBVwxnZqZw/fvjxeXpyIzA5W3eD/hPA/e7+yeDzfwZw9/8KabMo2OY9M0sAaoAsD3lzC0xQsQ/IcfdjJ9qegl5EBoKmljbWVB1g2bb9vL91Hyu213GkqRWAUWnJTB2bwbljhjFlbAZn5qb1+lQN3Z0CYTRQGfK8Cjj/RG3cvcXM6oERwN6QNjcAZZ2FvJndCdwJMGbMmDBKEhGJrEEJccenavjqZRNobm2jfNdBynYEpmko21HHi2t3HW979uh0po7NYMqYYUwZk8HItOQ+q7VPTiWb2ZnAg8BVna1393nAPAgc0fdFTSIiPSkxPo6z89I5Oy+d2y8sAGBPQ2NI8B/gV+9UMG9poLsnL2MwU8ZkBMM/g6KcVBJ76crdcIK+GsgPeZ4XXNZZm6pg1006gW4azCwPeB64zd23dLtiEZEBYmRaMrPOymHWWTlA4PaK63Y2UBY84l+2bR8LVu8EIDkxjiuKs/nZrVN6vI5wgn45UGhm4wgE+lzg1g5tFgC3A+8Bc4DF7u5mNgx4EbjP3d/psapFRAagpIR4powJHMFD4I5bO+sbKQvOyjk0qXf68U8a9ME+93uARQSGVz7h7uvM7AGg1N0XAI8DvzWzzcB+Ar8MAO4BJgDfM7PvBZdd5e57evobEREZaMyM0cMGM3rYYD59Tm7vbUcXTImIDHxdjbqJ3jk7RUQEUNCLiEQ9Bb2ISJRT0IuIRDkFvYhIlFPQi4hEOQW9iEiU63fj6M2sFtjejbfI5G8nU+tvVF/3qL7uUX3d05/rG+vuWZ2t6HdB311mVnqiiwb6A9XXPaqve1Rf9/T3+k5EXTciIlFOQS8iEuWiMejnRbqAk1B93aP6ukf1dU9/r69TUddHLyIifysaj+hFRCSEgl5EJMoNyKA3s1lmttHMNpvZfZ2sTzKzZ4Lrl5lZQR/Wlm9mb5jZejNbZ2bf6KTNDDOrN7NVwa/vdfZevVxnhZmtDW7/724AYAH/HdyHa8ys5+9vduLazgjZN6vMrMHMvtmhTZ/uQzN7wsz2mNmHIcuGm9mrZvZR8N+ME7z29mCbj8zs9j6s7/+ZWXnw/+/54B3fOnttl5+FXqzvfjOrDvk/vOYEr+3y570X63smpLYKM1t1gtf2+v7rNncfUF8E7nK1BRgPDAJWAxM7tLkb+EXw8VzgmT6sLweYEnycCmzqpL4ZwF8jvB8rgMwu1l8DvAQYcAGwLIL/3zUELgaJ2D4EpgNTgA9Dlv2AwG0yAe4DHuzkdcOBrcF/M4KPM/qovquAhODjBzurL5zPQi/Wdz9wbxj//13+vPdWfR3W/xD4XqT2X3e/BuIR/TRgs7tvdfcm4Glgdoc2s4FfBx/PB2aamfVFce6+y93Lgo8PAhuA0X2x7R42G/iNB7wPDDOznAjUMRPY4u7duVq629x9KYHbZIYK/Zz9Gri+k5d+EnjV3fe7ex3wKjCrL+pz91fcvSX49H0gr6e3G64T7L9whPPz3m1d1RfMjpuAP/T0dvvKQAz60UBlyPMq/j5Ij7cJftDrgRF9Ul2IYJfRucCyTlZ/wsxWm9lLZnZm31YGgAOvmNkKM7uzk/Xh7Oe+MJcT/4BFeh9mu/uu4OMaILuTNv1lP36RwF9onTnZZ6E33RPsWnriBF1f/WH/XQLsdvePTrA+kvsvLAMx6AcEM0sBngO+6e4NHVaXEeiKOAf4KfDnPi4P4GJ3nwJcDXzVzKZHoIYumdkg4Drg2U5W94d9eJwH/obvl2OVzey7QAvw+xM0idRn4VHgNGAysItA90h/dAtdH833+5+lgRj01UB+yPO84LJO25hZApAO7OuT6gLbTCQQ8r939z91XO/uDe5+KPh4IZBoZpl9VV9wu9XBf/cAzxP4EzlUOPu5t10NlLn77o4r+sM+BHa3d2cF/93TSZuI7kczuwO4Fvhc8JfR3wnjs9Ar3H23u7e6exvwPyfYbqT3XwLwWeCZE7WJ1P47FQMx6JcDhWY2LnjENxdY0KHNAqB9dMMcYPGJPuQ9Ldif9ziwwd1/dII2o9rPGZjZNAL/D335i2iomaW2PyZw0u7DDs0WALcFR99cANSHdFP0lRMeSUV6HwaFfs5uB17opM0i4Cozywh2TVwVXNbrzGwW8B3gOnc/coI24XwWequ+0HM+nznBdsP5ee9NVwDl7l7V2cpI7r9TEumzwR/ni8CIkE0EzsZ/N7jsAQIfaIBkAn/ubwY+AMb3YW0XE/gTfg2wKvh1DXAXcFewzT3AOgIjCN4HLuzj/Tc+uO3VwTra92FojQY8EtzHa4GSPq5xKIHgTg9ZFrF9SOAXzi6gmUA/8ZcInPd5HfgIeA0YHmxbAjwW8tovBj+Lm4Ev9GF9mwn0b7d/DttHouUCC7v6LPRRfb8NfrbWEAjvnI71BZ//3c97X9QXXP6r9s9cSNs+33/d/dIUCCIiUW4gdt2IiMgpUNCLiEQ5Bb2ISJRT0IuIRDkFvYhIlFPQi4hEOQW9iEiU+/8UOPGQYOvDvgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_results_test_data(weights1, biases1, 100, \"sigmoid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dense-rachel",
   "metadata": {},
   "source": [
    "### Testing our vectorized function for trainig the network "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "postal-gabriel",
   "metadata": {},
   "source": [
    "Now in order to compare time consumed by each or the iterative and vectorized functions for network trainig, we train the network with the same measurements by our vectorized function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "residential-creature",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 958 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "cost_by_epochs, weights21, biases21 = learning_vectorized_sgd(1, 20, 10, weights_biases, number_of_samples=100, activation=\"sigmoid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "focal-above",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy After Training: 0.64\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAoFklEQVR4nO3deXxU9b3/8dcnCUkIWcgOJAQQEiUoSEgBq+KCC65UxRbFrfXqbau21vq719baeq29vXZRa7ULalttVbS0KnWpgqhIVSQgsgVCQJYgJAFCIED27++POdIQExjJciaZ9/PxyCNnvud7Zj4zmZl3zvmexZxziIhI+InwuwAREfGHAkBEJEwpAEREwpQCQEQkTCkARETCVJTfBXweaWlpbujQoX6XISLSoyxZsmSHcy69dXuPCoChQ4dSVFTkdxkiIj2KmW1qq12bgEREwpQCQEQkTCkARETClAJARCRMKQBERMKUAkBEJEwpAEREwlSvD4DmZsfTizbz8vJtfpciIhJSetSBYEcjIsJ4dvFmDjQ0cf4JAzAzv0sSEQkJvX4NAGDGhCGUlNeweGOV36WIiISMsAiAi8YMIiE2iqcWtXk0tIhIWAqLAOgbHcllBdm8umI7O2vq/C5HRCQkhEUAAMyYkEN9UzN/XVLmdykiIiEhbAIgNzOB8cNSeHrRZpqbnd/liIj4LmwCAOCqiUPYvGs/C0t3+F2KiIjvwioAzh2VSWq/aA0Gi4gQZgEQExXJ5YWDmVdcwfbqWr/LERHxVVgFAMCV43Nodo5Zizf7XYqIiK+CCgAzm2Jma82s1MzuaGN+jJk9681fZGZDvfZUM3vTzGrM7OFWy0Sb2UwzKzGzNWZ2Wac8oyPISY1jUm46sz7YQmNTc3c8pIhISDpiAJhZJPAIcB6QD1xhZvmtul0PVDnnRgAPAPd57bXAXcDtbdz1nUCFcy7Pu9+3j+oZHIUZE3LYvqeWN9ZUdNdDioiEnGDWAMYDpc65Dc65emAWMLVVn6nAE970bGCymZlzbp9zbiGBIGjta8BPAZxzzc65bts158zjMhiQGMtTi7QZSETCVzABkAVsaXG7zGtrs49zrhGoBlLbu0Mz6+9N/tjMlprZX80ss52+N5pZkZkVVVZWBlHukUVFRjB9/GAWlFSyeef+TrlPEZGexq9B4CggG3jXOVcAvAf8oq2OzrmZzrlC51xhenp6pxUw/Qs5REYYT3+gtQARCU/BBMBWYHCL29leW5t9zCwKSAJ2HuY+dwL7gb97t/8KFARRS6cZkBTLWSMzeK5oC3WNTd350CIiISGYAFgM5JrZMDOLBqYDc1r1mQNc601PA+Y759o934I37x/A6V7TZGD156i7U8yYMIRd++r558rt3f3QIiK+O2IAeNv0bwZeA4qB55xzq8zsHjO72Ov2OJBqZqXAbcDBXUXNbCNwP3CdmZW12IPov4G7zWw5cDXw3U56TkE7ZUQaQ1LjNBgsImEpqCuCOedeAV5p1fbDFtO1wOXtLDu0nfZNwKRgC+0KERHGleNz+Omraygp30teZoKf5YiIdKuwOxK4tWnjsomOjOBprQWISJgJ+wBIjY/hvBMG8LelZeyvb/S7HBGRbhP2AQCB00TvrW3kHx994ncpIiLdRgEAFA5JJi8zXoPBIhJWFACAmTFjwhCWl1WzvGy33+WIiHQLBYDnkoIs+vaJ1GCwiIQNBYAnMbYPU08cxIvLPmFPbYPf5YiIdDkFQAszJgzhQEMTzy9tfaYLEZHeRwHQwgnZSYzOTuKpRZs4zJksRER6BQVAKzMm5FBSXsPijVV+lyIi0qUUAK1cNGYQCbFRPLVok9+liIh0KQVAK3HRUVxWkM2rK7azs6bO73JERLqMAqANV07Iob6pmdlLyvwuRUSkyygA2pCXmcD4oSk8/cFmmps1GCwivZMCoB0zJuawaed+FpZ227XqRUS6lQKgHVOOH0BKv2gNBotIr6UAaEdMVCSXF2Yzr7iC7dW1fpcjItLpFACHMWP8EJqaHbMW6/xAItL7KAAOIyc1jkl56cz6YAuNTc1+lyMi0qmCCgAzm2Jma82s1MzuaGN+jJk9681fZGZDvfZUM3vTzGrM7OF27nuOma3s0LPoQjMm5LB9Ty3z11T4XYqISKc6YgCYWSTwCHAekA9cYWb5rbpdD1Q550YADwD3ee21wF3A7e3c96VAzdGV3j0mH5fBgMRYfvv2eu0SKiK9SjBrAOOBUufcBudcPTALmNqqz1TgCW96NjDZzMw5t885t5BAEBzCzOKB24B7j7r6bhAVGcHt5x7Lh5t3M2vxFr/LERHpNMEEQBbQ8puvzGtrs49zrhGoBlKPcL8/Bn4J7D9cJzO70cyKzKyosrIyiHI732UFWUw8JoX/e7WYyr06PYSI9A6+DAKb2YnAcOfc80fq65yb6ZwrdM4Vpqend31xbTAz7v3SCdQ2NHPvy6t9qUFEpLMFEwBbgcEtbmd7bW32MbMoIAnYeZj7PAkoNLONwEIgz8zeCq5kf4zIiOfrpw/nxWWf8M46f9ZEREQ6UzABsBjINbNhZhYNTAfmtOozB7jWm54GzHeHuaKKc+63zrlBzrmhwClAiXPu9M9bfHf75unDGZbWj7teWEltQ5Pf5YiIdMgRA8Dbpn8z8BpQDDznnFtlZveY2cVet8eBVDMrJTCwe3BXUe+//PuB68ysrI09iHqM2D6R3Pul49m4cz+PvFnqdzkiIh1iPenSh4WFha6oqMjvMvjOs8t4afknvPrtUxmRkeB3OSIih2VmS5xzha3bdSTwUbjzgpHERUfx/edX6trBItJjKQCOQlp8DN877zg++HgXf9VFY0Skh1IAHKUvFw6mcEgyP32lmF376v0uR0Tkc1MAHKWICON/Lz2BvbWN/OTlYr/LERH53BQAHZCXmcCNk47hb0vLeG/94Q57EBEJPQqADrrlzFwGp/TlzhdWUNeoYwNEpOdQAHRQ3+hIfjz1eDZU7uN3b23wuxwRkaApADrB6cdmcOHogTzyVikf79jndzkiIkFRAHSSH16YT0xUBD94YYWODRCRHkEB0EkyEmP5rynH8a/SnbywrPW58kREQo8CoBPNGJ/DiYP7c+9Lxezer2MDRCS0KQA6UUSE8b+XnMDuAw3c9881fpcjInJYCoBOlj8oketPGcYzH2xh8cZdfpcjItIuBUAXuPWsXLL69+XO51dQ39jsdzkiIm1SAHSBuOgo7pk6ipLyGh5bqGMDRCQ0KQC6yOSRmUwZNYBfzVvH5p2Hve69iIgvFABd6O6LR9EnMoK7XtR1A0Qk9CgAutCApFi+e04eb5dU8tLybX6XIyJyCAVAF7vmpKGMyU7i+8+vYENljd/liIgcFFQAmNkUM1trZqVmdkcb82PM7Flv/iIzG+q1p5rZm2ZWY2YPt+gfZ2Yvm9kaM1tlZv/Xac8oxERGGI/MKKBPZAQ3PFnEntoGv0sSEQGCCAAziwQeAc4D8oErzCy/VbfrgSrn3AjgAeA+r70WuAu4vY27/oVz7jhgLHCymZ13dE8h9GUnx/GbGQVs2rmfW2cto6lZ4wEi4r9g1gDGA6XOuQ3OuXpgFjC1VZ+pwBPe9GxgspmZc26fc24hgSA4yDm33zn3pjddDywFsjvwPELexGNS+dHFo5i/poJfvL7W73JERIIKgCxgS4vbZV5bm32cc41ANZAaTAFm1h+4CHijnfk3mlmRmRVVVlYGc5ch6+qJQ7hyQg6/fWs9L+qEcSLiM18Hgc0sCngGeMg51+YRU865mc65QudcYXp6evcW2AXuvmgU44em8N9/W87KrdV+lyMiYSyYANgKDG5xO9tra7OP96WeBARzkdyZwDrn3INB9O0VoqMi+M1VBaTERXPDk0VU7q3zuyQRCVPBBMBiINfMhplZNDAdmNOqzxzgWm96GjDfHeHIJzO7l0BQ3Pq5Ku4F0uJjmHlNIVX76/nGX5bofEEi4osjBoC3Tf9m4DWgGHjOObfKzO4xs4u9bo8DqWZWCtwGHNxV1Mw2AvcD15lZmZnlm1k2cCeBvYqWmtkyM/uPznxioe74rCR+Pm0MRZuq+NEcHSksIt0vKphOzrlXgFdatf2wxXQtcHk7yw5t524tuBJ7r4vGDKJ42x5+89Z68gcmcvVJQ/0uSUTCiI4E9tnt5xzL5OMy+J9/rOa99cEMm4iIdA4FgM8iIowHp5/I0LR+fPOpJWzZpTOHikj3UACEgITYPjx6TSFNzY4bnixiX12j3yWJSBhQAISIYWn9+PWVBZSU7+X2v35Es04XISJdTAEQQk7LS+d7543k1ZXbefjNUr/LEZFeLqi9gKT7/Mepwyjetof755Zw7IAEzh01wO+SRKSX0hpAiDEz/vfSExiTncRtzy5j7fa9fpckIr2UAiAExfaJ5PdXFxIXE8UNTxZRta/e75JEpBdSAISoAUmx/P7qcWyvruXmZ5bS2KTTRYhI51IAhLCCnGR+csnx/Kt0J/e+XOx3OSLSy2gQOMRdXjiYNdv38vjCjxmREc9VE4f4XZKI9BJaA+gBvn/+SM44Np0fzVnFwnU7/C5HRHoJBUAPEBlhPHTFWEakx/ONp5ZQWlHjd0ki0gsoAHqIhNg+PHZtIdGREVz/xGLtGSQiHaYA6EEGp8Qx85pxbKuu5T91IRkR6SAFQA8zbkgKP7tsNB98vIsfvLBCF5IRkaOmvYB6oC+NzWJDZQ0PzS9lREY8N04a7ndJItIDKQB6qFvPymN95T5++uoahqXFc3Z+pt8liUgPo01APVREhPGLy8cwOiuJb8/6kFWfVPtdkoj0MEEFgJlNMbO1ZlZqZne0MT/GzJ715i8ys6Fee6qZvWlmNWb2cKtlxpnZCm+Zh8ws7K8R/Hn1jY7k0WsKSerbh/94ooiKPbV+lyQiPcgRA8DMIoFHgPOAfOAKM8tv1e16oMo5NwJ4ALjPa68F7gJub+OufwvcAOR6P1OO5gmEu4zEWB67tpDqAw3c8GQRtQ1NfpckIj1EMGsA44FS59wG51w9MAuY2qrPVOAJb3o2MNnMzDm3zzm3kEAQHGRmA4FE59z7LrAby5PAlzrwPMLaqEFJPPiVE1m+tZrv6mpiIhKkYAIgC9jS4naZ19ZmH+dcI1ANpB7hPsuOcJ8AmNmNZlZkZkWVlZVBlBuezhk1gDumHMfLy7fx4Bvr/C5HRHqAkB8Eds7NdM4VOucK09PT/S4npN046RguH5fNQ2+s48VlW/0uR0RCXDABsBUY3OJ2ttfWZh8ziwKSgJ1HuM/sI9ynfE5mxk8uOYHxw1L4f7OXs2RTld8liUgICyYAFgO5ZjbMzKKB6cCcVn3mANd609OA+e4wh6g657YBe8xsorf3zzXAi5+7evmM6KgIfn/VOAYmxfKffy6irGq/3yWJSIg6YgB42/RvBl4DioHnnHOrzOweM7vY6/Y4kGpmpcBtwMFdRc1sI3A/cJ2ZlbXYg+ibwGNAKbAeeLVznpIk94vm8Wu/QF1jM9f/qYiauka/SxKREGQ96VwyhYWFrqioyO8yeoyF63Zw7R8/4LS8dB69ppDICB1qIRKOzGyJc66wdXvIDwLL0TslN43/uXgU89dU8OOXVuvEcSJyCJ0LqJe7auIQPt6xj8cXfkxKv2i+NTnX75JEJEQoAMLAneePZPf+Bu6fW0JibBTXnTzM75JEJAQoAMJARIRx32UnsLe2gbv/sZrEvn24tCD7yAuKSK+mMYAwERUZwUNXjOWLw1P5f7OX8/qq7X6XJCI+UwCEkdg+kcy8ppDjs5K4+ZkPeXf9Dr9LEhEfKQDCTHxMFE989QsMTY3jhieKWLZlt98liYhPFABhqH9cNH++fgKp8TFc98cPKCnf63dJIuIDBUCYykyM5S/XTyA6MoKrH1/Ell06ZYRIuFEAhLGc1Dj+fP0EahuamfHYIl1RTCTMKADC3LEDEvjTV7/Ajpo6rn78A3bvr/e7JBHpJgoAYWxOMo9eU8jHO/Zx3R8Xs08njxMJCwoAAeDkEWk8dMVYlpft5j//vIS6Rl1bWKS3UwDIQVOOH8DPpo1hYekOvv3MMhqbmv0uSUS6kAJADjFtXDY/vDCff67azvf+vkIXmBfpxXQuIPmMr50yjOoDDfzqjXUk9u3DDy4YSeDCbSLSmygApE23npVL9YEGHl/4MUl9++g00iK9kAJA2mRm/PDCfPbWNnL/3BIiI4yvnzZcVxUT6UU0BiDt+vQ00heMHsjPX1vLlx75FyvKqv0uS0Q6SVABYGZTzGytmZWa2R1tzI8xs2e9+YvMbGiLed/z2tea2bkt2r9jZqvMbKWZPWNmsZ3yjKRTRUVG8PAVY3noirFs31PL1EcWcvecVeypbfC7NBHpoCMGgJlFAo8A5wH5wBVmlt+q2/VAlXNuBPAAcJ+3bD4wHRgFTAF+Y2aRZpYFfAsodM4dD0R6/SQEmRkXjxnEvNtO46qJQ3jivY2c9cu3eWn5J7rOsEgPFswawHig1Dm3wTlXD8wCprbqMxV4wpueDUy2wG4jU4FZzrk659zHQKl3fxAYf+hrZlFAHPBJx56KdLWkvn24Z+rxvPDNk8lIjOHmpz/k2j8uZtPOfX6XJiJHIZgAyAK2tLhd5rW12cc51whUA6ntLeuc2wr8AtgMbAOqnXOvt/XgZnajmRWZWVFlZWUQ5UpXGzO4Py/edAp3X5TP0k1VnP3AAn79xjodPSzSw/gyCGxmyQTWDoYBg4B+ZnZVW32dczOdc4XOucL09PTuLFMOIzLCuO7kYbzx3dM4Oz+TX84t4bxfvaOrjIn0IMEEwFZgcIvb2V5bm328TTpJwM7DLHsW8LFzrtI51wD8Hfji0TwB8VdmYiyPXFnAn776BRqbHFc+uojbnl3Gjpo6v0sTkSMIJgAWA7lmNszMogkM1s5p1WcOcK03PQ2Y7wKjg3OA6d5eQsOAXOADApt+JppZnDdWMBko7vjTEb+cfmwGr39nErecOYJ/LP+EM3/xFk8v2qxTSYiEsCMGgLdN/2bgNQJf0s8551aZ2T1mdrHX7XEg1cxKgduAO7xlVwHPAauBfwI3OeeanHOLCAwWLwVWeHXM7NRnJt0utk8k3z3nWF799iTyByXy/edXcNnv3mX1J3v8Lk1E2mA9aTe+wsJCV1RU5HcZEgTnHM9/uJWfvFzM7gMN3HDqMXz3nDz6ROrYQ5HuZmZLnHOFrdv1aZQuYWZcWpDNG989jWkF2fzu7fV85ffv8cnuA36XJiIeBYB0qf5x0dw3bTQPXTGWtdv3csFD7/B2iXbnFQkFCgDpFhePGcScW04hIyGW6/74Ab98fS1NGiAW8ZUCQLrN8PR4XrjpZKYVZPPr+aVc9dgiKvbW+l2WSNhSAEi36hsdyc8vH8PPpo3mwy1VXPDQQt5bv9PvskTCkgJAfPHlwsG8cNPJJMREMeOx93nkzVIdMyDSzRQA4pvjBiQy55ZTOP+EwPUGvvbEYqr21ftdlkjYUACIr+Jjovj1FWP58dRRvFu6kwseeoelm6v8LkskLCgAxHdmxtUnDWX2N04iIsL48u/e4/GFH+taAyJdTAEgIWN0dn9evuVUzjgugx+/tJpv/GWprjwm0oUUABJSkuL6MPPqcdx5/kjmFpdz4UMLWblV1yEW6QoKAAk5ZsYNk47h2RsnUt/YzKW/fZfH3tmgC86IdDIFgISswqEpvPytU/ji8FTufbmYM37+Fn95f5OCQKSTKAAkpKXGx/DH677AX66fwMD+ffnBCys54+dv8dSiTdQ3NvtdnkiPptNBS4/hnOOddTt4YF4JH27eTVb/vtx85gguK8gmOkr/y4i0p73TQSsApMdxzrFg3Q4emFvCsi27yU7uy81njOCycdm63oBIGxQA0us453irpJIH55bwUVk1g1P6cssZuVxSkKUgEGlBASC9lnOOt9ZW8sC8EpaXVZOTEsctZ47gkrFZRCkIRBQA0vs555i/poIH5pWwcusehqTGccuZuXzpxEEKAglrHbokpJlNMbO1ZlZqZne0MT/GzJ715i8ys6Et5n3Pa19rZue2aO9vZrPNbI2ZFZvZSUf53ESAwPEDk0dm8o+bT+HRawqJj4ni9r9+xFn3v82jCzawcN0OyvfU6hQTIp6oI3Uws0jgEeBsoAxYbGZznHOrW3S7Hqhyzo0ws+nAfcBXzCwfmA6MAgYB88wszznXBPwK+KdzbpqZRQNxnfrMJGyZGWfnZ3LWyAzmri7nV2+s4yevFB+cnxgbRW5mArkZ8S1+xzMgMRYz87Fyke51xAAAxgOlzrkNAGY2C5gKtAyAqcDd3vRs4GELfJKmArOcc3XAx2ZWCow3s9XAJOA6AOdcPaDzAEunMjPOGTWAs/Mz2VFTz7qKvawrrzn4+/XV5cxavOVg/4SYKEZkxgcCISOB3MxAQAxKUjBI7xRMAGQBW1rcLgMmtNfHOddoZtVAqtf+fqtls4ADQCXwRzMbAywBvu2c29f6wc3sRuBGgJycnCDKFTmUmZGeEEN6QgxfHJ52yLydNXWsq6gJ/JQHgmH+mkqeKyo72KdfdCQnj0jjhxflk52sFVXpPYIJgK563ALgFufcIjP7FXAHcFfrjs65mcBMCAwCd2uV0uulxseQGh/DxGNSD2mv2lfvBcNe1m7fy+wlZZz7wALuOH8kM8bnEBGhNQLp+YIZBN4KDG5xO9tra7OPmUUBScDOwyxbBpQ55xZ57bMJBIJISEjuF834YSnMmDCEe6Yez2u3TqJgSDJ3vbCS6Y++z8c7PrOyKtLjBBMAi4FcMxvmDdZOB+a06jMHuNabngbMd4FdLeYA0729hIYBucAHzrntwBYzO9ZbZjKHjimIhJTBKXE8+bXx/GzaaIq37WHKgwt4dMEGmnQdY+nBjrgJyNumfzPwGhAJ/ME5t8rM7gGKnHNzgMeBP3uDvLsIhARev+cIfLk3Ajd5ewAB3AI85YXKBuCrnfzcRDqVmfHlwsGclpfOD15YyU9eKealFdv4+bTR5GUm+F2eyOemA8FEjoJzjpeWb+NHc1axt7aBW87M5RunD9cpKCQkdehAMBE5lJlx0ZhBzP3OJKYcP5D755Zw8cP/0tXLpEdRAIh0QGp8DL++Yiwzrx7Hzpo6pj7yL+775xpqG3TRGgl9CgCRTnDOqAHM/c5pXFaQxW/fWs/5D73Dkk27/C5L5LAUACKdJCmuDz+bNoYnvzaeuoZmpv3uPe6es4r99Y1+lybSJgWASCeblJfOa9+ZxNUTh/Cndzdy7oMLmLu6nIYmXcJSQov2AhLpQos27OS//7acjTv3kxzXhynHD+CCEwYx8ZgUnaJauo2uByDik7rGJt5eW8nLK7Yxb3U5++qbSO0XHQiD0QOZMCyVSJ1aQrqQAkAkBNQ2NPHW2gpeWr6NN4orONDQRFp8DOd5YfCFoSkKA+l0CgCREHOgvok311bw8vJtvLGmnNqGZjISYjj/hIFcMHog43KSddI56RQKAJEQtq+ukflrAmHw5toK6hqbGZAYezAMxg7urzCQo6YAEOkhauoaeaO4nJeWb+PttZXUNzWTFh9DQU5/CoYkU5CTzOjsJGL7RPpdqvQQ7QWAX9cDEJF2xMdEMfXELKaemMWe2gbmrS5n4bodLN1cxeurywGIijDyByVSkJPM2Jz+FOQkk53cV1cuk89FawAiPcjOmjo+3LybpZurWLq5io+2VHPAO+1EeoK3lpCTTMGQZE7I0lqCBGgNQKQXSI2P4az8TM7KzwSgsamZNdv38uHmKpZu3s2STVW8tiqwltAn0sgfmMjYnGQmDEvh1Lx04mP0kZd/0xqASC+zo6aOpZsCgbB0cxXLy3ZT29BMdGQEE4encvbIDCaPzGRQ/75+lyrdRIPAImGqoamZpZuqmFdcztzV5WzcuR+AUYMSOWtkJmfnZzJqUKLGD3oxBYCI4JxjfeU+5hWXM291OUs2V+EcDEyKZfLIDM4amclJw1OJidLYQW+iABCRz9hZU8f8NRXMKy5nQckODjQ00S86kkl56Zw1MpMzjssgpV+032VKBykAROSwahuaeG/9TuYWl/NGcTnle+qIMCgcksJpx6Yzbkjg+IO4aA0k9zQdCgAzmwL8isBF4R9zzv1fq/kxwJPAOGAn8BXn3EZv3veA64Em4FvOuddaLBcJFAFbnXMXHqkOBYBI92hudqz8pJp5q8uZW1xB8bY9AERGGMdmJhw89mBsTn+GpfXT+EGIO+oA8L6kS4CzgTJgMXCFc251iz7fBEY7575uZtOBS5xzXzGzfOAZYDwwCJgH5DnnmrzlbgMKgUQFgEjo2rWvno+2BPYq+nDzbpZt2U1NXeBCN/3j+nDi4P6MHZxMwZD+jBncn8TYPj5XLC115DiA8UCpc26Dd0ezgKnA6hZ9pgJ3e9OzgYct8C/BVGCWc64O+NjMSr37e8/MsoELgJ8Atx3VsxKRbpHSL5ozjsvgjOMyAGhqdqyvrGHppkAgfLilirdLKnEOzGBEenyLtYRkRmTE6yynISiYAMgCtrS4XQZMaK+Pc67RzKqBVK/9/VbLZnnTDwL/BSQc7sHN7EbgRoCcnJwgyhWRrhYZYeRlJpCXmcD08YHP5Z7aBpZvqfbWEgKnrXiuqAyAuOhI8jITGDkwkZEDEzhuQCLHDkggqa/WFPzky2iOmV0IVDjnlpjZ6Yfr65ybCcyEwCagrq9ORI5GYmwfTslN45TcNCCwy+nGnfv5cHMVy8uqWbN9D6+s2MYzH2w+uExW/74cNyAQDMd5wTA0NU5XS+smwQTAVmBwi9vZXltbfcrMLApIIjAY3N6yFwMXm9n5QCyQaGZ/cc5ddVTPQkRCjpkxLK0fw9L6cWlBNhAIhfI9dRRv30Pxtj2s2baXNdv38FZJJU3Ngf/vYqIiyMtMOCQYRg1MIilOawudLZhB4CgCg8CTCXx5LwaudM6tatHnJuCEFoPAlzrnvmxmo4Cn+fcg8BtA7qeDwN6ypwO3axBYJHzVNTZRWlFzMBDWbN9L8bY97KipBwLjCqOzkjg1N51JeemMzelPH60lBO2oB4G9bfo3A68R2A30D865VWZ2D1DknJsDPA782Rvk3QVM95ZdZWbPERgwbgRuavnlLyICEBMVyahBSYwalHRIe+XeOtZs38OSTVW8s24Hv3mrlIffLCUhJoqThqcyKS+dSbnp5KTG+VR5z6YDwUSkx6g+0MC7pTtYsK6SBSU72Lr7AABDU+MOhsFJw1Ppp7OeHkJHAotIr+KcY8OOfSwoqWRBSSXvb9jFgYYm+kQaBTnJTMpL57S8dPIHJob95TQVACLSq9U1NrFkYxVve2sHnx69nBYfTeGQFAan9GVQ/8BPVv++DEyKJaVfdFgcxawAEJGwUrG3loXrdrCgpJKPyqr5ZPcB6hqbD+kTExVBVv9PgyE28Dvp0Nu94apqCgARCWvOOXbtq+eT3bVs3X2AT3YfYFv1gUNuV+yt+8xyqf2iGZrWj5NHpDEpN40xg3veHkgKABGRI6hvbKZ8z78D4ZPdB9i6u5bibXtYXrabZsfBPZBOzUtnUm4aQ1L7+V32EemawCIiRxAdFcHglDgGp3x2t9Lq/Q28u34HC7zNSq+vDlx7OScljlNz05iUF9gDqSedCE9rACIin9Onp7l4xxtwfm/9DvbVNxEZYYwd3J9Tc9M5NS+N0VlJIXFaC20CEhHpIg1NzXy4eTcLSip5Z10ly7dW4xwkxkZx8og0ThqeSl5mArkZ8aTGx3R7fQoAEZFuUrWvnn+t38E7JYGD1rZV1x6clxzXh9yMBIZnxJObEU9uZjwjMuIZkBjbZbukKgBERHzgnGNbdS2lFTWsq6ihtKKG0oq9rKuoYff+hoP94mOi/h0KGYFQyM1IICu5b4evpaBBYBERH5jZwQPQJuWlH2x3zrFzXz3rymsorayhtDwQCgtKKpm9pOxgv5ioCIanx/PMjRM7/foJCgARER+YGWnxMaTFx3DS8NRD5lUfaDi4plBaUcOmnftJjO38r2sFgIhIiEnq24dxQ5IZNyS5Sx/H//2TRETEFwoAEZEwpQAQEQlTCgARkTClABARCVMKABGRMKUAEBEJUwoAEZEw1aPOBWRmlcCmo1w8DdjRieV0NtXXMaqvY1Rfx4R6fUOcc+mtG3tUAHSEmRW1dTKkUKH6Okb1dYzq65hQr6892gQkIhKmFAAiImEqnAJgpt8FHIHq6xjV1zGqr2NCvb42hc0YgIiIHCqc1gBERKQFBYCISJjqdQFgZlPMbK2ZlZrZHW3MjzGzZ735i8xsaDfWNtjM3jSz1Wa2ysy+3Uaf082s2syWeT8/7K76vMffaGYrvMf+zAWYLeAh7/VbbmYF3VjbsS1el2VmtsfMbm3Vp1tfPzP7g5lVmNnKFm0pZjbXzNZ5v9u8qoeZXev1WWdm13ZjfT83szXe3+95M+vfzrKHfS90YX13m9nWFn/D89tZ9rCf9S6s79kWtW00s2XtLNvlr1+HOed6zQ8QCawHjgGigY+A/FZ9vgn8zpueDjzbjfUNBAq86QSgpI36Tgde8vE13AikHWb++cCrgAETgUU+/q23EzjAxbfXD5gEFAArW7T9DLjDm74DuK+N5VKADd7vZG86uZvqOweI8qbva6u+YN4LXVjf3cDtQfz9D/tZ76r6Ws3/JfBDv16/jv70tjWA8UCpc26Dc64emAVMbdVnKvCENz0bmGxm1h3FOee2OeeWetN7gWIgqzseuxNNBZ50Ae8D/c1soA91TAbWO+eO9sjwTuGcWwDsatXc8j32BPClNhY9F5jrnNvlnKsC5gJTuqM+59zrzrlG7+b7QHZnP26w2nn9ghHMZ73DDlef973xZeCZzn7c7tLbAiAL2NLidhmf/YI92Mf7EFQDqXQzb9PTWGBRG7NPMrOPzOxVMxvVvZXhgNfNbImZ3djG/GBe4+4wnfY/eH6+fgCZzrlt3vR2ILONPqHyOn6NwBpdW470XuhKN3ubqP7Qzia0UHj9TgXKnXPr2pnv5+sXlN4WAD2CmcUDfwNudc7taTV7KYHNGmOAXwMvdHN5pzjnCoDzgJvMbFI3P/4RmVk0cDHw1zZm+/36HcIFtgWE5L7WZnYn0Ag81U4Xv94LvwWGAycC2whsZglFV3D4//5D/rPU2wJgKzC4xe1sr63NPmYWBSQBO7ulusBj9iHw5f+Uc+7vrec75/Y452q86VeAPmaW1l31Oee2er8rgOcJrGq3FMxr3NXOA5Y658pbz/D79fOUf7pZzPtd0UYfX19HM7sOuBCY4YXUZwTxXugSzrly51yTc64ZeLSdx/X79YsCLgWeba+PX6/f59HbAmAxkGtmw7z/EqcDc1r1mQN8usfFNGB+ex+AzuZtM3wcKHbO3d9OnwGfjkmY2XgCf6NuCSgz62dmCZ9OExgsXNmq2xzgGm9voIlAdYvNHd2l3f+8/Hz9Wmj5HrsWeLGNPq8B55hZsreJ4xyvrcuZ2RTgv4CLnXP72+kTzHuhq+prOaZ0STuPG8xnvSudBaxxzpW1NdPP1+9z8XsUurN/COylUkJgD4E7vbZ7CLzZAWIJbDooBT4AjunG2k4hsDlgObDM+zkf+Drwda/PzcAqAns1vA98sRvrO8Z73I+8Gj59/VrWZ8Aj3uu7Aijs5r9vPwJf6Ekt2nx7/QgE0TaggcB26OsJjCm9AawD5gEpXt9C4LEWy37Nex+WAl/txvpKCWw///Q9+OlecYOAVw73Xuim+v7svbeWE/hSH9i6Pu/2Zz7r3VGf1/6nT99zLfp2++vX0R+dCkJEJEz1tk1AIiISJAWAiEiYUgCIiIQpBYCISJhSAIiIhCkFgIhImFIAiIiEqf8P9PB/uznGQj0AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_results_test_data(weights21, biases21, 100, activation=\"sigmoid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "backed-hawaii",
   "metadata": {},
   "source": [
    "######  Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "analyzed-halloween",
   "metadata": {},
   "source": [
    "The vectorized functions consumes mush less amount of time compared to the iterative function: <br>\n",
    "Vectorized Wall Time of Training: 749ms <br>\n",
    "Iterative Wall Time of Training: 1min 11s\n",
    "_________________________"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "terminal-magnet",
   "metadata": {},
   "source": [
    "If we increase the epoch size of the vectorized function by 200 we will get the following results:<br>\n",
    "earning Rate: 1 <br>\n",
    "Number of Epochs: 200<br>\n",
    "Batch Size: 10<br>\n",
    "Number of samples: 1000<br>\n",
    "Activation function: Sigmoid<br>\n",
    "\n",
    "Accuracy before training: 0.14<br>\n",
    "Wall time of running the training function: 8.15s<br>\n",
    "Accuracy after training: 0.61<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "confident-gauge",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Before Training: 0.15\n"
     ]
    }
   ],
   "source": [
    "W01, W12, W23, b1, b2, b3 = before_training(\"sigmoid\", 100)\n",
    "weights_biases = (W01, W12, W23, b1, b2, b3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "second-youth",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 7.63 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "cost_by_epochs, weights21, biases21 = learning_vectorized_sgd(1, 200, 10, weights_biases, number_of_samples=100, activation=\"sigmoid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "athletic-healing",
   "metadata": {},
   "source": [
    "### Sigmoid VS Tanh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "talented-brake",
   "metadata": {},
   "source": [
    "##### 1. Sigmoid Activation Function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fatal-improvement",
   "metadata": {},
   "source": [
    "First thing we do is to randomly create our wights and biases matrices and to calculate the accuracy before doing ant trainings on the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "golden-billy",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Before Training: 0.106\n"
     ]
    }
   ],
   "source": [
    "W01, W12, W23, b1, b2, b3 = before_training(\"sigmoid\", 500)\n",
    "weights_biases = (W01, W12, W23, b1, b2, b3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "intimate-dispatch",
   "metadata": {},
   "source": [
    "\n",
    "Learning Rate: 1 <br>\n",
    "Number of Epochs: 200 <br>\n",
    "Batch Size: 10<br>\n",
    "Number of samples: 500<br>\n",
    "Activation function: Sigmoid<br>\n",
    "\n",
    "_________________________\n",
    "\n",
    "Accuracy before training: 0.08<br>\n",
    "Wall time of running the training function: 39.3s<br>\n",
    "Accuracy after training: 0.789<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "french-reservoir",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 37.8 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "cost_by_epochs, weights21, biases21 = learning_vectorized_sgd(1, 200, 10, weights_biases, number_of_samples=500, activation=\"sigmoid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "genuine-textbook",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy After Training: 0.78\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAW1klEQVR4nO3dfYxc1X3G8e+zMzu73l3bYLPYgE3sBJPUKFFCLCdSkjYNSQNRgpMGUtO0pSoViVqqVmmVEqVCFFWV6AuoVVFTKmgpTQMpbRRLcUJeaBupCg5rQgKGOFkM4aXGXmwwft+3X/+4d9bj8ax31t6ZWZ/7fCRr75x7ZuY3d8fPnD33zr2KCMzMLF1dnS7AzMxay0FvZpY4B72ZWeIc9GZmiXPQm5klrtzpAuqdc845sWrVqk6XYWZ2Rtm6devLETHYaN28C/pVq1YxNDTU6TLMzM4okn423TpP3ZiZJc5Bb2aWOAe9mVniHPRmZolz0JuZJc5Bb2aWOAe9mVnikgn6nfsOc9s3t7Nj5ECnSzEzm1eSCfqR/Uf524eG2TFysNOlmJnNK8kEfXcpeynjk5MdrsTMbH5JLuhHJ3zFLDOzWskEfSUP+rFxj+jNzGolE/TdZQEwNuGgNzOrlU7QV0f0Dnozs+MkF/SeozczO14yQV/xiN7MrKFkgr67lM/Re2esmdlxkgn6UpeQPKI3M6uXTNBLorury3P0ZmZ1kgl6yKZvPKI3MzteWkFf7mLcQW9mdpy0gr7kqRszs3pJBX2l1OWpGzOzOkkFvefozcxOlFjQe0RvZlavqaCXdLmk7ZKGJd3YYH2PpPvz9VskrcrbuyXdI+lxSU9J+twc13+c7lIXo+OeozczqzVj0EsqAXcAVwBrgWskra3rdh3wSkRcBNwO3Jq3Xw30RMSbgbcDn6p+CLRCd9kjejOzes2M6NcDwxGxIyJGgfuADXV9NgD35MsPAJdJEhBAv6QysAAYBV6bk8obqHiO3szsBM0E/QXA8zW3X8jbGvaJiHFgH7CULPQPAjuB54C/ioi9p1nztDxHb2Z2olbvjF0PTADnA6uBP5T0+vpOkq6XNCRpaGRk5JSfzMfRm5mdqJmgfxFYWXN7Rd7WsE8+TbMY2AP8KvCNiBiLiN3A/wLr6p8gIu6MiHURsW5wcHD2ryLXXZK/GWtmVqeZoH8EWCNptaQKsBHYVNdnE3BtvnwV8FBEBNl0zfsAJPUD7wR+PBeFN+KpGzOzE80Y9Pmc+w3Ag8BTwJcjYpukWyRdmXe7C1gqaRj4DFA9BPMOYEDSNrIPjH+KiB/N9YuoyoLeUzdmZrXKzXSKiM3A5rq2m2qWj5AdSll/vwON2lslO47eI3ozs1pJfTO2UvbhlWZm9ZIKes/Rm5mdKMGg9xy9mVmt5IJ+1CN6M7PjJBX01VMgZEd2mpkZJBb03aUuImBi0kFvZlaVVtCXs5fjeXozs2PSCvpSHvSTnqc3M6tKKugrJQEw5i9NmZlNSSroyyVP3ZiZ1Usq6KembnyIpZnZlMSCPpu68bH0ZmbHJBX0FY/ozcxOkFTQT03djHuO3sysKq2gz4+j99SNmdkxaQV99fBKB72Z2ZSkgt5z9GZmJ0oq6Ktz9OM+jt7MbEqSQe85ejOzY5IK+krZc/RmZvWSCnp/M9bM7ERJBX3Zx9GbmZ0gqaD3KRDMzE6UVND78EozsxMlFfSeozczO1GiQe85ejOzqsSCPp+j9xWmzMymJBX0kuguyVM3ZmY1kgp6yKZvxic9dWNmVpVk0HvqxszsmCSD3lM3ZmbHJBf0Fc/Rm5kdJ7mg7y53+fBKM7MayQV9uUs+BYKZWY3kgr671MWYd8aamU1JLugrZe+MNTOr1VTQS7pc0nZJw5JubLC+R9L9+fotklbVrHuLpO9J2ibpcUm9c1j/CbKjbjxHb2ZWNWPQSyoBdwBXAGuBayStret2HfBKRFwE3A7cmt+3DPwr8OmIuAR4LzA2Z9U30F3yHL2ZWa1mRvTrgeGI2BERo8B9wIa6PhuAe/LlB4DLJAn4JeBHEfFDgIjYExETc1N6Y92lLsYd9GZmU5oJ+guA52tuv5C3NewTEePAPmApcDEQkh6U9KikzzZ6AknXSxqSNDQyMjLb13CciqduzMyO0+qdsWXg3cAn858fk3RZfaeIuDMi1kXEusHBwdN6wkq5i6PjLf2jwczsjNJM0L8IrKy5vSJva9gnn5dfDOwhG/1/NyJejohDwGbg0tMt+mT6KmUOHnXQm5lVNRP0jwBrJK2WVAE2Apvq+mwCrs2XrwIeiogAHgTeLKkv/wD4BeDJuSm9sf6eEodGx1v5FGZmZ5TyTB0iYlzSDWShXQLujohtkm4BhiJiE3AXcK+kYWAv2YcBEfGKpNvIPiwC2BwRX2vRawHyEf2oR/RmZlUzBj1ARGwmm3apbbupZvkIcPU09/1XskMs22Kgp8To+CRjE5NTlxY0Myuy5JKwr5J9dh3yPL2ZGZBg0Pf3lAA46Hl6MzMgyaDPRvQHjzrozcwgxaDPp268Q9bMLJNc0PdV8qkbj+jNzIAEg95TN2Zmx0s26A956sbMDEgx6POpmwMe0ZuZASkG/dSI3kFvZgYJBv2C7urOWE/dmJlBgkHf1SX6KiXvjDUzyyUX9JBN3/g4ejOzTJpB7xG9mdmUJIO+r1L2zlgzs1ySQT/Q46tMmZlVJRn0fT0ln73SzCyXZND3V8qeozczy6UZ9D0lnwLBzCyXZND3Vco+BYKZWS7JoK+O6COi06WYmXVcokFfZmIyODo+2elSzMw6Ls2gr/ic9GZmVUkGffUqU94ha2aWaNAP5Kcq9g5ZM7NEg77P56Q3M5uSZND3V3xOejOzqjSD3hcINzObkmbQVzxHb2ZWlWTQD/R6RG9mVpVk0Pf3ZHP0HtGbmSUa9D3lEpVSFwe8M9bMLM2gh2xUf+DoWKfLMDPruGSDfqDXV5kyM4OEg76/Umb/Ec/Rm5klG/QLe32VKTMzaDLoJV0uabukYUk3NljfI+n+fP0WSavq1l8o6YCkP5qjumfU3+OLj5iZQRNBL6kE3AFcAawFrpG0tq7bdcArEXERcDtwa93624Cvn365zRvo8YjezAyaG9GvB4YjYkdEjAL3ARvq+mwA7smXHwAukyQASR8FngG2zUnFTVrYW2a/g97MrKmgvwB4vub2C3lbwz4RMQ7sA5ZKGgD+GPjTkz2BpOslDUkaGhkZabb2k+qveERvZgat3xl7M3B7RBw4WaeIuDMi1kXEusHBwTl54oHeModGJ5iY9HVjzazYyk30eRFYWXN7Rd7WqM8LksrAYmAP8A7gKkl/AZwFTEo6EhF/d7qFz6R68ZGDo+Ms6u1u9dOZmc1bzQT9I8AaSavJAn0j8Kt1fTYB1wLfA64CHoqIAN5T7SDpZuBAO0Ieaq4ydcRBb2bFNmPQR8S4pBuAB4EScHdEbJN0CzAUEZuAu4B7JQ0De8k+DDrK56Q3M8s0M6InIjYDm+vabqpZPgJcPcNj3HwK9Z2y6qmKfeSNmRVdst+MHfCI3swMKEDQH/D5bsys4NIPeo/ozazgHPRmZolLNuh91I2ZWSbZoK+Uu6iUu3zUjZkVXrJBDz6DpZkZFCDofdSNmRVd0kGfXXzE1401s2JLOugX9pQ5cHSs02WYmXVU0kHf31PioEf0ZlZwSQf9QG+3j6M3s8JLOugX9pbZf8RTN2ZWbEkH/eIF3ew7PEZ2anwzs2JKPujHJoLDY56nN7PiSj7oAfYd9vSNmRWXg97MLHHFCPpDDnozK65iBL1H9GZWYA56M7PEJR30ixz0ZmZpB/3CnjISvOagN7MCSzrou7rEot5uj+jNrNCSDno49u1YM7OictCbmSWuEEH/qoPezAqsEEHvEb2ZFVnyQb9oQbePujGzQks+6H2qYjMrukIEvU9VbGZFVoigB3871syKy0FvZpa44gS9T1VsZgVVnKD3iN7MCspBb2aWuKaCXtLlkrZLGpZ0Y4P1PZLuz9dvkbQqb/+ApK2SHs9/vm+O65/R2f1Z0O85ONrupzYzmxdmDHpJJeAO4ApgLXCNpLV13a4DXomIi4DbgVvz9peBj0TEm4FrgXvnqvBmLeztZqCnzEv7jrT7qc3M5oVmRvTrgeGI2BERo8B9wIa6PhuAe/LlB4DLJCkifhAR/5e3bwMWSOqZi8JnY/niXge9mRVWM0F/AfB8ze0X8raGfSJiHNgHLK3r83Hg0Yg4emqlnrrli3p56TUHvZkVU1t2xkq6hGw651PTrL9e0pCkoZGRkTl/fo/ozazImgn6F4GVNbdX5G0N+0gqA4uBPfntFcBXgN+IiKcbPUFE3BkR6yJi3eDg4OxeQROWL+pl5MBRxicm5/yxzczmu2aC/hFgjaTVkirARmBTXZ9NZDtbAa4CHoqIkHQW8DXgxoj43zmqedaWL+5lYjJ4+YCPvDGz4pkx6PM59xuAB4GngC9HxDZJt0i6Mu92F7BU0jDwGaB6COYNwEXATZIey/+dO+evYgbLF/UCeJ7ezAqp3EyniNgMbK5ru6lm+QhwdYP7/RnwZ6dZ42lbvjgP+n2HYeVZnS3GzKzNkv9mLMB5edDv9A5ZMyugQgT9kv4KlVKXp27MrJAKEfSSWLa4x4dYmlkhFSLoIf/SlIPezAqoOEG/eIGnbsyskAoT9Oef1cvOV4/4S1NmVjiFCfqLBgcYnZjkub2HOl2KmVlbFSboL162EICf7DrQ4UrMzNqrMEH/hnMHABjevb/DlZiZtVdhgn6gp8wFZy3gp7s9ojezYilM0AOsWTbgqRszK5xiBf25Azw9coCJyeh0KWZmbVOwoF/I6Pgkz/vIGzMrkGIF/bJsh6zn6c2sSAoV9BflR948tfO1DldiZtY+hQr6hb3dvHHZQoZ+9kqnSzEza5tCBT3A+tVL2PrsXp8KwcwKo5BBf3B0gic9fWNmBVHIoAf4/jN7O1yJmVl7FC7oly3q5XVL+9jioDezgihc0AOsX7WER57d6y9OmVkhFDLof/FN5/LqoTEe3rGn06WYmbVcIYP+fW86l4GeMl997MVOl2Jm1nKFDPre7hIfvGQ5X3/iJY6MTXS6HDOzlipk0ANc+dbz2X9knP/evrvTpZiZtVRhg/5db1jKkv4Kmx9/qdOlmJm1VGGDvlzq4v0/dy7/9ePdjI77W7Jmlq7CBj3ABy9Zzv6j43zPR9+YWcIKHfTvuugc+iolvrnN0zdmlq5CB31vd4n3vnGQbz25y1+eMrNkFTroAT7ylvPZvf8o33pyV6dLMTNricIH/QfWLuPCJX38w3efJsKjejNLT+GDvlzq4rffs5ofPPeqL0hiZkkqfNADXP32lSztr/D5rzzO/iNjnS7HzGxOOeiBBZUSf7PxbTw9cpDf+9IPODru0yKYWToc9Ll3rzmHWzZcwn9vH+ET//Aw//fq4U6XZGY2J5oKekmXS9ouaVjSjQ3W90i6P1+/RdKqmnWfy9u3S/rgHNY+5z75jtfxhV+7lOFd+3n/bf/D7d/6Cdtf2s+kD700szOYZjrSRFIJ+AnwAeAF4BHgmoh4sqbP7wBviYhPS9oIfCwifkXSWuBLwHrgfODbwMURMe3cyLp162JoaOg0X9bpeW7PIf5881N8I/8i1dl93axbtYQ3LV/IirMXsHhBN4sWdGc/e7vpKXdR6hLlri7KJeXL2U9JHX0tZlYMkrZGxLpG68pN3H89MBwRO/IHuw/YADxZ02cDcHO+/ADwd8oSbgNwX0QcBZ6RNJw/3vdO5YW0y4VL+/jCr7+d5/ceYssze9myYw9bn3uF7zy1i9kO7rsEkhAggRDk2V/bJlVva2odNW2N1ivvdGxd9XEb9J/l582pfD6J2d1p1jXNrvusP2Rn/ZLnWf125nvvxYP8yYfXzvnjNhP0FwDP19x+AXjHdH0iYlzSPmBp3v5w3X0vqH8CSdcD1wNceOGFzdbeciuX9LFySR9XvX0FAEfHJxjZf5R9h8fYd3iM1w6P8drhccYmJxmfCMYng4nJyezn1O0gCCIggOofUEHWkLUdW0/ep3ofquvr7nv849X2PXF9MMtPp1OYqZrtXWb7nYXZP/4s+8/68Vtb/6n8DuzMd95ZC1ryuM0EfctFxJ3AnZBN3XS4nGn1lEusOLuPFWd3uhIzs+Y1szP2RWBlze0VeVvDPpLKwGJgT5P3NTOzFmom6B8B1khaLakCbAQ21fXZBFybL18FPBTZ37abgI35UTmrgTXA9+emdDMza8aMUzf5nPsNwINACbg7IrZJugUYiohNwF3AvfnO1r1kHwbk/b5MtuN2HPjdkx1xY2Zmc2/GwyvbbT4cXmlmdqY52eGV/masmVniHPRmZolz0JuZJc5Bb2aWuHm3M1bSCPCz03iIc4CX56icueS6Zsd1zd58rc11zc6p1vW6iBhstGLeBf3pkjQ03Z7nTnJds+O6Zm++1ua6ZqcVdXnqxswscQ56M7PEpRj0d3a6gGm4rtlxXbM3X2tzXbMz53UlN0dvZmbHS3FEb2ZmNRz0ZmaJSyboZ7qAeRvrWCnpvyQ9KWmbpN/P22+W9KKkx/J/H+pQfc9KejyvYShvWyLpW5J+mv9s66VVJL2xZrs8Juk1SX/QiW0m6W5JuyU9UdPWcPso87f5e+5Hki5tc11/KenH+XN/RdJZefsqSYdrttsXWlXXSWqb9ncn6XP5Ntsu6YNtruv+mpqelfRY3t62bXaSjGjd+yy7jN2Z/Y/s9MlPA68HKsAPgbUdquU84NJ8eSHZhdXXkl1T94/mwbZ6Fjinru0vgBvz5RuBWzv8u3wJeF0nthnw88ClwBMzbR/gQ8DXyS4H+05gS5vr+iWgnC/fWlPXqtp+HdpmDX93+f+FHwI9wOr8/22pXXXVrf9r4KZ2b7OTZETL3mepjOinLmAeEaNA9QLmbRcROyPi0Xx5P/AUDa6TO89sAO7Jl+8BPtq5UrgMeDoiTufb0acsIr5Ldk2FWtNtnw3Av0TmYeAsSee1q66I+GZEjOc3Hya7glvbTbPNprMBuC8ijkbEM8Aw2f/fttYlScAngC+14rlP5iQZ0bL3WSpB3+gC5h0PV0mrgLcBW/KmG/I/ve5u9/RIjQC+KWmrsouyAyyLiJ358kvAss6UBmQXran9zzcfttl022c+ve9+i2zUV7Va0g8k/Y+k93Sopka/u/myzd4D7IqIn9a0tX2b1WVEy95nqQT9vCNpAPgP4A8i4jXg74E3AG8FdpL92dgJ746IS4ErgN+V9PO1KyP7W7Ejx9wqu1TllcC/503zZZtN6eT2mY6kz5Ndwe2LedNO4MKIeBvwGeDfJC1qc1nz7ndX5xqOH1C0fZs1yIgpc/0+SyXo59VFyCV1k/0CvxgR/wkQEbsiYiIiJoF/pEV/rs4kIl7Mf+4GvpLXsav6p2D+c3cnaiP78Hk0InblNc6Lbcb026fj7ztJvwl8GPhkHg7k0yJ78uWtZPPgF7ezrpP87ubDNisDvwzcX21r9zZrlBG08H2WStA3cwHztsjn/u4CnoqI22raa+fUPgY8UX/fNtTWL2lhdZlsZ94THH9x92uBr7a7ttxxo6z5sM1y022fTcBv5EdFvBPYV/Ond8tJuhz4LHBlRByqaR+UVMqXXw+sAXa0q678eaf73W0CNkrqkbQ6r+377awNeD/w44h4odrQzm02XUbQyvdZO/Yyt+Mf2Z7pn5B9En++g3W8m+xPrh8Bj+X/PgTcCzyet28CzutAba8nO+Lhh8C26nYClgLfAX4KfBtY0oHa+oE9wOKatrZvM7IPmp3AGNlc6HXTbR+yoyDuyN9zjwPr2lzXMNncbfV99oW878fz3+9jwKPARzqwzab93QGfz7fZduCKdtaVt/8z8Om6vm3bZifJiJa9z3wKBDOzxKUydWNmZtNw0JuZJc5Bb2aWOAe9mVniHPRmZolz0JuZJc5Bb2aWuP8HYuraFCHZf6QAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_results_test_data(weights21, biases21, 1000, activation=\"sigmoid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acceptable-correction",
   "metadata": {},
   "source": [
    "##### 2. Tanh Activation Function"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "seven-nigeria",
   "metadata": {},
   "source": [
    "First thing we do is to randomly create our wights and biases matrices and to calculate the accuracy before doing ant trainings on the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "objective-advocate",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Before Training: 0.11\n"
     ]
    }
   ],
   "source": [
    "W01, W12, W23, b1, b2, b3 = before_training(\"tanh\", 500)\n",
    "weights_biases = (W01, W12, W23, b1, b2, b3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "primary-depression",
   "metadata": {},
   "source": [
    "\n",
    "Learning Rate: 1 <br>\n",
    "Number of Epochs: 200<br>\n",
    "Batch Size: 10<br>\n",
    "Number of samples: 500<br>\n",
    "Activation function: Tanh<br>\n",
    "\n",
    "______________________\n",
    "\n",
    "Accuracy before training: 0.082<br>\n",
    "Wall time of running the training function: 35.3s<br>\n",
    "Accuracy after training: 0.635<br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fantastic-lighting",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 33.2 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "cost_by_epochs, weights22, biases22 = learning_vectorized_sgd(1, 200, 10, weights_biases, number_of_samples=500, activation=\"tanh\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "sized-mustang",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy After Training: 0.655\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD4CAYAAADiry33AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAvvElEQVR4nO3dd3xc1Zn/8c8zVb1XyzZykTsGG2MI2AQwxVSHALuQ7IYkJGTZsJtfOixZQkiyG9JgE0iygLMhJCwQIBsnmGYMBmMwlo2bXGW5SZZt9T4ajeb8/pgrWV0jq4y487xfL7+YuXNndHQ1fOfMc889R4wxKKWUsi9HpBuglFJqdGnQK6WUzWnQK6WUzWnQK6WUzWnQK6WUzbki3YCeMjIyTH5+fqSboZRSHymbN2+uNMZk9vVYWEEvIsuB/wKcwBPGmB/1ePwi4GFgPnCLMeb5Lo/dBnzHuvsDY8yTA/2s/Px8CgsLw2mWUkopi4gc7u+xQUs3IuIEHgWuAuYAt4rInB67HQE+Czzd47lpwHeB84DFwHdFJHUojVdKKTU84dToFwPFxpgSY4wfeAZY0XUHY8whY8x2INjjuVcCrxtjqo0xNcDrwPIRaLdSSqkwhRP0ecDRLvdLrW3hGM5zlVJKjYBxMepGRO4QkUIRKayoqIh0c5RSylbCCfoyYFKX+xOtbeEI67nGmMeMMYuMMYsyM/s8aayUUuo0hRP0m4ACEZkiIh7gFmBVmK//KnCFiKRaJ2GvsLYppZQaI4MGvTEmANxFKKB3A88ZY4pE5AERuR5ARM4VkVLgZuC/RaTIem418H1CHxabgAesbUoppcaIjLdpihctWmROZxx9eV0L/7vxCJ9YkMfUzIRRaJlSSo1fIrLZGLOor8fGxcnYkVDR0Mov1hZTUtEU6aYopdS4Ypug97hCv4q/vedQfqWUim72CXqnFfQBDXqllOrKNkHvdmqPXiml+mKboPe6tEevlFJ9sU3QezTolVKqT/YLei3dKKVUN/YJej0Zq5RSfbJN0LucDhyiQa+UUj3ZJughNPKmTUs3SinVja2C3uNy0Ko9eqWU6sZWQe91OfRkrFJK9WCroPc4HVqjV0qpHuwV9C4NeqWU6kmDXimlbM5WQa+jbpRSqjdbBb1HT8YqpVQv9gp6pw6vVEqpnuwV9FqjV0qpXmwV9F4NeqWU6sVWQa81eqWU6s1eQa+jbpRSqhdbBb1br4xVSqlebBX0ejJWKaV606BXSimbs13Qt2qNXimlurFV0HutGr0xJtJNUUqpccNWQd+xQHggqEGvlFIdbBX0bl0gXCmlerFV0Hf06DXolVLqFHsGvZ6QVUqpTvYKei3dKKVUL/YKeqtHr1MVK6XUKWEFvYgsF5G9IlIsInf38bhXRJ61Ht8oIvnWdreIPCkiO0Rkt4jcM8Lt78ZrBb3Od6OUUqcMGvQi4gQeBa4C5gC3isicHrvdDtQYY6YDDwEPWttvBrzGmDOBc4AvdXwIjAYddaOUUr2F06NfDBQbY0qMMX7gGWBFj31WAE9at58HlomIAAaIFxEXEAv4gfoRaXkf9GSsUkr1Fk7Q5wFHu9wvtbb1uY8xJgDUAemEQr8JKAeOAD81xlT3/AEicoeIFIpIYUVFxZB/iQ56MlYppXob7ZOxi4F2YAIwBfi6iEztuZMx5jFjzCJjzKLMzMzT/mE6jl4ppXoLJ+jLgEld7k+0tvW5j1WmSQaqgE8Brxhj2owxJ4F3gUXDbXR/tHSjlFK9hRP0m4ACEZkiIh7gFmBVj31WAbdZt28C1prQzGJHgEsBRCQeOB/YMxIN74tXe/RKKdXLoEFv1dzvAl4FdgPPGWOKROQBEbne2m0lkC4ixcDXgI4hmI8CCSJSROgD43+MMdtH+pfo4HE6AQ16pZTqyhXOTsaY1cDqHtvu63LbR2goZc/nNfa1fbS4XQJo6UYppbqy15WxOupGKaV6sVfQa41eKaV6sWfQa+lGKaU62SvotXSjlFK92CroRQSP06E9eqWU6sJWQQ/gdor26JVSqgvbBb3H5dCgV0qpLjTolVLK5mwZ9LrwiFJKnWK/oHc6aNWgV0qpTvYLepdTSzdKKdWF/YJeR90opVQ39gt6PRmrlFLd2DPotUavlFKd7Bf0Th11o5RSXdku6GPcTlr87ZFuhlJKjRu2C/qUODe1LW2RboZSSo0btgv6tHgPNU1+QkvWKqWUsl3Qp8Z5CAQN9b5ApJuilFLjgu2CPi3eA0B1kz/CLVFKqfFBg14ppWxOg14ppWzOtkFfo0GvlFKAjYO+SoNeKaUAGwZ9nMdFjNtBTbMGvVJKgQ2DHiAtzkNVowa9UkqBXYM+waM9eqWUstgy6FPjPDrqRimlLLYM+vR4DXqllOpgy6BPtea7UUopZdOgT4/30NAaoDWg0xUrpZQtgz7VGktf26zTFSullC2DPr3joikdYqmUUuEFvYgsF5G9IlIsInf38bhXRJ61Ht8oIvldHpsvIu+JSJGI7BCRmBFsf59S46xpEHSIpVJKDR70IuIEHgWuAuYAt4rInB673Q7UGGOmAw8BD1rPdQF/AP7JGDMXuBgY9XqKTmymlFKnhNOjXwwUG2NKjDF+4BlgRY99VgBPWrefB5aJiABXANuNMdsAjDFVxphRP0OaEOMCoLFVFx9RSqlwgj4PONrlfqm1rc99jDEBoA5IB2YARkReFZEtIvKtvn6AiNwhIoUiUlhRUTHU36GXOE8o6Js06JVSatRPxrqAJcCnrf/eICLLeu5kjHnMGLPIGLMoMzNz2D803uMEoNmvwyuVUiqcoC8DJnW5P9Ha1uc+Vl0+Gagi1Pt/2xhTaYxpBlYDC4fb6MG4nA68LgdNfu3RK6VUOEG/CSgQkSki4gFuAVb12GcVcJt1+yZgrTHGAK8CZ4pInPUB8HFg18g0fWDxXhfNrdqjV0op12A7GGMCInIXodB2Ar81xhSJyANAoTFmFbASeEpEioFqQh8GGGNqROTnhD4sDLDaGPPSKP0u3cR5nNqjV0opwgh6AGPMakJll67b7uty2wfc3M9z/0BoiOWYivdoj14ppcCmV8YCxHm1R6+UUmDjoI/3uHR4pVJKYeeg9zp1eKVSSmHnoPe4tHSjlFLYOOjjvE49GauUUtg46LVHr5RSIbYN+jiPC19bkPagiXRTlFIqomwb9PHejvlutFevlIputg36UzNYap1eKRXdbBv0HT16rdMrpaKdfYPe6tHryBulVLSzbdDHaY9eKaUAGwd9Z49eg14pFeXsG/QdPXot3Silopxtgz5Oe/RKKQXYOOg7SjeN2qNXSkU52wZ9x8nYZp2qWCkV5Wwb9G6nA4/LQZNOVayUinK2DXqAeI9Ta/RKqahn66CP87h01I1SKurZOuhDq0xpj14pFd1sHfRxHpfW6JVSUc/WQR/vdeoC4UqpqGfroA/V6DXolVLRzdZBn+DV5QSVUsrWQZ8c66a2uS3SzVBKqYiyddCnxXto8AXwB4KRbopSSkWM7YMeoKbZH+GWKKVU5Ng66DMSQkFf1ahBr5SKXrYO+rR4LwBVTa0RbolSSkWOzYM+1KOvbtIevVIqetk66LV0o5RSYQa9iCwXkb0iUiwid/fxuFdEnrUe3ygi+T0enywijSLyjRFqd1iSYtw4HaKlG6VUVBs06EXECTwKXAXMAW4VkTk9drsdqDHGTAceAh7s8fjPgZeH39yhcTiE1DiPlm6UUlEtnB79YqDYGFNijPEDzwAreuyzAnjSuv08sExEBEBEPgEcBIpGpMVDlB7v0dKNUiqqhRP0ecDRLvdLrW197mOMCQB1QLqIJADfBr430A8QkTtEpFBECisqKsJte1jSE7RHr5SKbqN9MvZ+4CFjTONAOxljHjPGLDLGLMrMzBzRBqTFe6jSoFdKRTFXGPuUAZO63J9obetrn1IRcQHJQBVwHnCTiPwYSAGCIuIzxjwy3IaHK1S60ZOxSqnoFU7QbwIKRGQKoUC/BfhUj31WAbcB7wE3AWuNMQZY2rGDiNwPNI5lyAOkJ3ipt+a78bhsPZpUKaX6NGjyWTX3u4BXgd3Ac8aYIhF5QESut3ZbSagmXwx8Deg1BDNSdL4bpVS0C6dHjzFmNbC6x7b7utz2ATcP8hr3n0b7hi09/tRFU9lJMZFoglJKRZTtaxk6DYJSKtrZPujTE3RiM6VUdLN90Ocmx+B2CluP1ka6KUopFRG2D/p4r4sr5uTwfx+W0Rpoj3RzlFJqzNk+6AH+7txJ1DS38fquE5FuilJKjbmoCPol0zPIS4nl2U1HB99ZKaVsJiqC3ukQrp2fy3sHqrR8o5SKOlER9ABzJiQRCBpKKpoi3RSllBpTURP0s3KSANh7vCHCLVFKqbEVNUE/NTMet1PYe0KDXikVXaIm6N1OB9MyE7RHr5SKOlET9AAzcxI16JVSUSeqgn5GdiJltS00+Noi3RSllBozURX0s3ISAdindXqlVBSJqqCfaQX9Hi3fKKWiSFQF/YTkWBwC5bW+SDdFKaXGTFQFvcMhpMZ5dLUppVRUiaqgB0iJc2vQK6WiStQFfVq8h5omHXWjlIoeURf0KVq6UUpFmagL+rQuQf8PT2zkL1vLItwipZQaXVEX9Cnxbmqa2mjwtbG+uJLXinQxEqWUvUVd0KfFefC3BzlgTVe853h9hFuklFKjK+qCPjXOA8Du8lDAH6pqxtemi5Eopewr+oI+vnvQtwcNByoaI9kkpZQaVdEX9HFu4FTQg859o5Syt+gLeqtHv6e8geRYNx6nI2rnvimpaKT4ZHT+7kpFE1ekGzDWOmr0Da0B5uQmETSGfVEa9N9dVURNs5+//cvSSDdFKTWKoi7ok2PdiIAxkJscQ2KMiw8OVke6WRFR0dBK8clGWgPteF3OSDdHKTVKoq5043QIybGhOn12cgyzcpM4VuejqrE1wi0bew2+AIGgYd9xPRmtlJ1FXdBDaCw9QG5SDB+bmg7A+uLKQZ/3ys5yVu8oH9W2jaW6ltCcP0XH6iLcEqXUaIrKoE+JO9WjPzMvmbR4D+v2Vgz6vJ+8updH3ywe7eaNiUB7kMbWAAA7RzDofW3trN5RTqA9OGKvqZQanqgM+jRr5E1ucgwOh7C0IIO391cQDJp+n9Pga6OksomqRntMiFbvC3Te3lk2clcHP7K2mH/+4xZ+8treEXtNpdTwhBX0IrJcRPaKSLGI3N3H414RedZ6fKOI5FvbLxeRzSKyw/rvpSPc/tOSYpVucpJiAPj4jEwqG/3sKu8/8IqO1WMMVDW1Ykz/Hwh9eX5z6bgbxthRtkmP97DneP2I9MBrmvz8bsMhErwu/ntdCWt26TxCSo0Hgwa9iDiBR4GrgDnArSIyp8dutwM1xpjpwEPAg9b2SuA6Y8yZwG3AUyPV8OHo6NHnJIeCfmlBJgDr9vVfvtleWgtAW7vp1hsejK+tnW8+v407nto8rqZa6Aj686el42sLUlLZNOzXXLn+II2tAZ7+4nnMnZDE3S9up6bJHt+AlPooC6dHvxgoNsaUGGP8wDPAih77rACetG4/DywTETHGfGiMOWZtLwJiRcQ7Eg0fjmvOzOXOi6eRGBOq1WcmepmXlzRgnX576ak69lBG6BypbsYYKKlo4pdr959+o0dYR9BfMC10MnrXseGXb/6yrYxLZ2Uxf2IKP735LGqb2/j+33YN+3WVUsMTTtDnAUe73C+1tvW5jzEmANQB6T32uRHYYozplZIicoeIFIpIYUXF4CdFh+usSSl8e/msbts+PiOTzUdqqPf1vfrU9tK6zmGZlUOo0x+yesrz8pL473UlnQEbaR3tWDApFbdThn11cHvQUF7rY1ZOIgCzc5O48+JpvPhhWbfpJpRSY29MTsaKyFxC5Zwv9fW4MeYxY8wiY8yizMzMsWhSLxfPzKI9aNjQxzDL2mY/R6qb+fiMUNuG0qM/VBUK+i9dNI1A0IyboYwdQZ+R4GFaZgJ7hzldc2VjK4GgITcltnPbNfNzgdC3GaVU5IQT9GXApC73J1rb+txHRFxAMlBl3Z8I/Bn4jDHmwHAbPFoWTEohMcbFW32UbzrKGhfPDAV95RDqzgcrm0mNc3P+1JErkYyEeivok2LdzMxJZO8we/THalsAmGCd94BTJ7vL61qG9dpKqeEJJ+g3AQUiMkVEPMAtwKoe+6widLIV4CZgrTHGiEgK8BJwtzHm3RFq86hwOR0smZ7Bun0VvUbVHK1pBmDh5FRgaD36w1VN5GfEk5noJSvRO26Cvq6lDa/LQYzbycycRI7V+YZVViqv8wGQm3yqR58c6ybG7eBEvW/Y7VVKnb5Bg96qud8FvArsBp4zxhSJyAMicr2120ogXUSKga8BHUMw7wKmA/eJyFbrX9aI/xYjZGlBJuV1Pg5XNXfbXlbrwyGQlxpLapx70LH0xhgef7uEkopGDlU2kZ8eD8DcCUkDDuEcS3XNbZ3nHDrq6sOZrrmzR59yqkcvIuQkxXR+CAyk8FA1v35r3H7hU+ojLaxJzYwxq4HVPbbd1+W2D7i5j+f9APjBMNs4Zs7MSwZCc9XnZ8R3bi+raSE7KQa300F6gpfKQXr0Rcfq+eHq3Ww4UMmxOl+XoE/m7f2V+NraiXEPPImYr62dn722l7suLegM5JFU7zsV9DNzkgDYc7yBc/PTTuv1yut8xLqdvdqakxwTVo9+5fqDvFp0nC8unYLLGZXX8Sk1avT/qC4KshNwCOzuUa8uq20mzzrJmB7vGbRH/9ftoRGlb1r1/vyMOADmTEiiPWjC6jkXHqrh8XcO8s7+0RmFVNdyKugnWLN4DueEbHldC7kpMYhIt+0D9eif3niEe/+8A4CtR2sJGqiIwsnllBptGvRdxLidTMmI7zUcsKy2hQlW0GckeKls6j+MjDH8bVt5ZzkE6Fa6gfBOyHacFxitKRfqWtpIsoJeRJiZnciOYUyFcKzWx4Qu9fkOOcmxnKxv7TW9RF1zG/+5ejd/3HiEnWV1nR8G4ZR5lFJDo0Hfw+zcJPZ06dm2Bw3H63zkpVo9+oSBe/QfHq2lrLaFLy6dytKCDOBU0E9KjSPB66IonKCvDgX9YGWi09W1Rw9w5dwcth2t5a29J0/r9Y7VtpDbZcRNh5wkL/72INXNft47UEW7Ffgr3z1IgzWp2sNrTl1IdlyDXqkRp0Hfw+zcJI5Wt7B2zwmW/ngtO8vqaGs3naWbjAQvdS1t+AN9zw3zWtEJ3E7h8rnZ3HPVbL555UySrdkyHQ5hRnZCWKWb0prQyc2hXJw1FD2D/rYL8pmaEc8Df93V7+/WH38gSEVja7cx9B1yrF7+/31Yxq2Pv88T75RQ1djK/6w/yJVzs8lO8rJm9wk6Kj7ao1dq5GnQ99BRcvnGn7ZztLqF3793GOBUjT4hNE9OdZMffyDIz1/f163XXXyykSkZ8STFuJkzIYkvXzK92+sXZCVSfHLwhT46Sjej0aNvDxoafIHO0g2Ax+Xg3mtmU1LZxGu7jg/p9U7U+zCm+xj6Dh29/Kc/OALAo28W840/baM1EOQbV8zkkpmhQVjzJiQT43ZQXqtj7pUaaRr0PczODdXRq62Lol7aETqx2lm6iQ9N1VPZ2Mpbe0/yizf288Lm0s7nH6lu4oz0ePpTkJ1AVZN/0LH4p3r04Qd9RUNrWL3xBmuah54jZC6cHio1HRriBGedY+j77NGHgr6koon89DgaWgO8ubeCr14+g4LsxM6L0M6elEJucizlOuZeqRGnQd9DbnIMybFucpNj+OTCPHxtoeDs6NFnJoZ69Eeqm3mlKNTzLTxcA0AwaDhc1Ux+ely/rz8ju2PMeu9e/f4TDSz98Vr2HK+noiEU8OGejA0GDVc8tI6H1+wbdN+OC6N6Bn2M20lGgrfzQyZcfV0V2yEjwYvTEarL3Lp4Mnd+fBpXzs3mjoumArCkIJO5E5JYPi+HnKQYrdErNQqibnHwwYgID6yYS1ZiDDXNfl7cUkZKnJt4b+hQzctLJicphpXrD7LfqrVvOVyDMYYTDT5aA8FBe/QAxScb+Ni07vO+/WXrMY5Wt/DYuhIgNJQz3B59eb2PmuY2XtpRzjevnNlrmGNX/QU9wMTU2CEH/e7yejxOR5+/t9MhZCV6Ka/zcdGMzM5vTB0SvC5e+telALywuZSNUbpQu1KjSXv0fVhxdh4fm5bOhdMycAjdhg16XU6+9PGpbD5cQ70vwLJZWVQ1+TlY2cShylBdPX+AoM9JiiHR6+qzR79md2ihjr9Z69IumJxCs7+dZv/g8993lFsOVzWzf5BzALXNgwV9c6/tA9lWWsvs3EQ8rr7fTjnJMWQlersNOe1LbkoMx+t9nSNzlFIjQ4N+AMlxbi6emcWCySndtt9y7mQyEjzEeZx89fIZQKh8c9iaqfKMAUo3IkJBHyNvympb2HO8gYwET2ed/exJoZ8bTvnmYJe6+uuDrOzUURbKTOy9NMDE1DjKalsGXFaxq2DQsLOsnvkTU/rd5yvLCvjBJ+YN+C0DQiN02oOGJzcc4h9XbtTAV2qEaNAPYuVti/jhDWd22xbrcfKTm8/ihzfMY05uEsmxbjYfquFQVTNup/Q5nryrGdmJvXrda/eExq9/68rQPPkel6OzzBFO+eZQZRNel4P5E5MHD3rr9bL6DPpY2toNJxvCKxmVVDbS2Bpg/sTkfve5eGYWV8zNGfS1cq3ZLn/08h7e2V/J1qM1nY8VHqqmTEfkKHVaNOgH0V8v9JKZWdywYCIOh3DOGalsKKmkpKKRSalxg87VUpCdSHWTn2c+ONI5U+bru05wRnocn1yYR3Ksm4kpsZ097r7G0pfXtfDegarO+4eqQpOnXTY7m61Ha6lr7n8mypP1rcR5nJ3nHbqaaI0uCrd8s+1oaH79s6xvH8PRMULHb61f+8bu0Idfi7+df1i5kf94afewf4ZS0UiDfgTcfM5Ejla38MaekwOWbTrcuDCPxVPSuPvFHXzvr7soPtnA2/sqWHF2Hi6ngy99fCo3njOR9IRTQzl7+o/Ve7jtfz7oLPMcrGwiPyOOeXmhbwH7B1iM/GSDr8/ePIRKN0DYJ2S3l9YS53EyLTMhrP0H0vFN6JwzUjlvSlrnt5x3iyvxtQV590Bl2CWljxpfWzvPby617e+nIkuDfgQsn5fDgskptAfNgCNuOqTEeXjmi+fzmY+dwe82HOJrz20jxu3gsxfkA/DPF0/ny5dMJ91axLznmHt/IMhbe07iDwTZd6KB9qDhaHUL+RnxFGSFTngOdFHWyYZWshL7Li8NuUdfWse8vOTOIZTDkRbv6aznXzY7mz3HGyitaeaNPaFSVG1z27iZ5nmkvbLzON/40zbe2nd6U1AoNRAN+hEgItx79Wzg1PDJwTgcwj1XzeaM9Di2l9Zxy7mTSbOCvUOM20mi19VZujHGEAwa3i+p6pwnZkdZHcdqW/C3B5mSHk9eSiwxbseAI28qG1rJTOq7Rx/jdpKZGN5Y+rqWNnaW1XUuyDJcIsJXL5/B7Nwkls0OXTH7/OZS3th9knPzQz9jfR9LPY6VYNDwxu4To9LrLqkI/b0GO7+i1OnQoB8hi/LTePkrS7npnIlhPyfW4+SnN5/FOWekdl5A1FNG4qn57+9fVcRlD63jjxsPE2t9COwsq+tclzY/Ix6HQ5ielTBg0J9saCUzoe+gh/DH0q/dc4JA0HDl3OxB9x2qqZkJXDY7i4fX7OdkQyu3nDuZmdmJvBvBoH9jz0luf7JwyFNEhKPEGjX1+q4TOtpIjTgN+hE0OzcJr2vgBUV6Ojc/jRfuvKBzGuSeMhJCF01tO1rLk+8dpqSiiVeLTrC0IIN5ecmhoLdComP8/vTMBIq7DN/84Uu7eN6apqHZH6CxNUBWPz16CM2yubu8ftBpGl7ecZycpBjOGmBo5XA8+umFXDM/l8QYF5fMyuKC6el8cLCaptbBrysIR1Vj65Dm+y88HLqY67Wike91H6pqwuNyUNno7zbaSKmRoEE/zk3LTGDjwWpuf7KQjAQvf7j9PPJSYrl18WTOnJjM7uMNPL+5lNzkGLKt8C7IDq0B29gaYENxJY+/c5An3gldbXuyvmNoZf9DQD97YT6NrQE+97tN/YZqsz/Aun0VXDk3G8cI1Of74nU5efRTC9n4b8tIi/dw7fwJtAaCPPjKHoJBM+xFxx998wD/uPKDzoXRe64V3NOHR2qBUM++rX1oM3wOxBjDwYomrps/AbdTeHWIHyTtQcPtv9vEv/7vh2yI4DeejxpjDA/8dRfbjtZGuimjToN+nLv3mtl8/sIp1LX4+c41s1lSkMG7d1/KJbOymDshCX8gyLbSOr562YzOoaDTszqmWWjkR6/sAULLBJ5s8HWOoe/rYqkOCyen8uinFrKzrI4H/roLgECXYPMHgvzqzQO0BoIsn5c7Kr93V3Ge0DDQc85I5fYlU/j9e4e59Gdv8bH/XMvL1lXEp+O9ktDw1MfeLmHl+oNc/tDbnRO+9RRoD7K9tJbJaXHUtbSx6VD3qRqG8y2jorGVJn878ycm8/EZWfyp8GhYV0N32FlWxxt7TvJq0XH+YeXGzrUM1MBKa1r47bsHw5of6qNOg36cS4xx8+/XzmHXA8v5xIK8bo91rHE7PSuBTy489ViBFfTfXVXE9tK6ztE8G4qruvTo+w96gMvmZPPFi6bybOFRfvjSLs5+4HV+s+4ALf52rn9kPY+8WcyyWVmdJ0nHyjevnMn8icnEelzMyknkWy9sDyvY/IEgX39uGzvLQuP+65rb2HO8nsQYF/+3tYwfvLSL4pONrNvXvZRTfLKBL/6+kLf2VuBrC/LlS6bhdTl4deepOn1JRSMLvv86r+w8vQ+dgxWnzrHcefE0aprbeHrjkbCfv8G6nuKFOy9ARHjq/cOn1Y5o0zGC6+39lZxssPdkehr0HxHuPi7Cyk+P59bFk/jPT57Z7SKtyWlxeJwOth2t5dbFk7n3mtmkxLlZX3zqDT1Y0AN89bIZ5KfH8fg7BzHG8PCafdy/qog9xxt45FMLeOK2RWO+kHeM28mqu5bw8leW8vhnFgFw46838OymIwOWUwoPVfPCllJ+aF109cGhaoyBB1bMRYA5uUmkx3t6lU1+/vo+Xt91gq8+uxUITeV8xdwcnnr/cGcYP7vpKP5AkOc3l53W79RxMn1qRjznnJHKx6am89jbJfja2sN6/oYDlczMTmReXjLL5+bwzAdHhvSNIFp1rPTWHjSs2noswq0ZXTp75UeYwyH85yfn99rucjr44Q3zSI51d049cMG0dN4triQz0YvLIaTGeXo9r6cYt5MnblvE5sM1nD81ncsfeptnC49yw4I8rp0/YcR/n6GalBbH0184n/tW7eTbL+zgZ6/t46uXz+DWxZN77bvOOun6XkkVhYeq2VhShcfl4Kp5uczMTiIvJZb/WL2bl3aU42tr71xY5uWdx5maEU9JZROZiV7yUmJ58MYzafS18W9/3oHB8MKWUkTg7X0V1PvaSIrpPVncQEoqm/A4HZ0n5P91WQG3Pv4+j75ZzNevmNm5X3vQ8FzhUfafaCTW4+DSWVnMnZDMpkPV3HJu6He+7YJ8XtpRzjW/WE+C18XUzHhuXzJlwLmIotWuY/VMz0og3uPkT4WlfO7CKSNyPch4pD16m7p50aRu88ssmZ5JeZ2PV3ceJyPBG/YJ1OlZifz9uZM5Iz2erywrIDvJyz1XzxqtZg/ZmROTefHOC/jtZxeRnxHPPS/u4Kn3DvXab93eCs6alEJavIfvv7SbN/acZMGkFGLcTuZMSCI5zs2V87JpbA1wxUNvc8GP1rLi0XdxOx08/cXzWZyfxrJZWYgIcR4Xj31mEUsLMrj3zzupbPRz1yXT8bcHWXMa4+APVjQxOT2uM2Q+Ni2dGxdO5FdvHWB7aW3nfs8VHuWeF3fw7KYj/Pe6Em789Xvc9JsN+NqCXGBNeX1ufiqfuzC0LGRqvId1+yr4wpOF1DaPzpKUH2W7y+uZk5vE5y6cwt4TDXzr+e3sKK3jvQNVg56Y/6jRoI8Sn1gwgTPzkimpbBpwaOVAvnzJdN799qUDjtiJBBHh0lnZ/PEL53HZ7Cz+/S9F3PeXnZ3hdrLex57jDVw5N5u7r5rF7vJ6DlY2scRaUavDBdMySPS6qGnyc+fF08hO8vKFJVPISY7hmTvO50c3nvr25HY6+OWtC5icFkdeSiz/uqyAvJRYXtgytGkMgkHD7uP1TMnofkX1fdfNISPBwz88sZFfvrGfg5VN/PTVvZybn8rO713Jlvsu556rZrGnvAGHwHlT0zuPxXevm8vKz57L7z+/mD/cfh7VTX7uX1Vku/DqqvhkI/e8uJ0rHlrHgYrBl+qsbfZTVtvCnAlJfGJBHl+7fAYvbCnlukfWc+vj7/ONP20Pu3T2USDj7Y+/aNEiU1hYGOlm2FJVYyt//9j7nD0phZ/efFakmzMqfG3t/Mfq3fzh/cMkxbr56mUzqGn28/Ca/fztX5YwLy8ZX1s7+040MCM7kRh39+se9p1oIDnWTXZSeB9mdS1ttPjbyUmO4VdvFfPjV/aytCCDH904v3NVsleLjlN4qJpvLZ/V61zLa0XHueOpzTz092dxw4LuF9sVn2zkRy/vZo01uZsIrPryEs7sMlPoliM1lNW0cN1Z/ZfSHl6zj4fX7OdT503me9fP7fN8z0dZe9Bw0Y/fpLrJT7sxXD0vh4dvWTDgczYUV/KpJzby+88v5qIZmRhjeG3XCVoDQfafaOCXa4tJi/ewbFYWiTFuLpiWzmVzRv7CwJEkIpuNMYv6fEyDPrr4A0GcDrFtLbLD7vJ6vvfXIt4vCQ2DzEmKYcPdl47amH8Ijct+ZtNRvruqiGDQcNGMTJwO6ZzW4PYlU/j3a+fQ1BrA19ZOUqybm37zHtVNrbz59Yv7PbG970QDf9laRkaCl89dOGXI7QoGDT95bS+/fusA55yRyn/dcnbn5HV2sHbPCT7/u0J+9emFfHikhpXrD/LWNy5hco8JBuua22hobWNiahz/tWY/D63ZR+F3LiOjj6vENxRX8scPjrChuJKWtnZ8bUGumJPN91bM5UhVMyvXH2TxlDQunplFRoKHlDDOeY02DXoVlYwxbD1aS4u/nfyM+H6vPh5px2pbeOKdg7yzv4LaljZuWJCHr62d37932LrSOVRScjmEQNDwwxvm8enzzhj1dq3adox/e3EHAlx9Zi4JMS72nWggOymGKRnxTMmIZ+6EJCanxQ26SMxoqG32kxzrHvLP/sKTm9hWWseGuy+lpsnPkgff5Nqzcvn5350NhDo3j6zdz8r1B2nyt7M4P40PDlWz6IxUnr/zgkFfv609yG/XH+ShNftwiNAaCBLncdLgOzWy6ayJySwtyCQ51k1JZSMVDa0keF0kxrhJjHER73VxvM5Ha6Cdq+blkhTrpqLBh68tSILXRWail7b2IEmx7s51pYdKg16pCGtrD/KdP++kLRikICuRWLeD4/Wt+NrauefqWUOeOuN0Haps4qE1+3hjd+jq3ulZCVQ0tHZbaCYlzs38iSksnZ7BjJxEqptCj03NSBiRdQd6Msbwuw2H+MFLu1k+N4ef/d1ZvUpq/dl1rJ5rf/kOd148jW9ai/b8+JU9/OqtA/zb1bO4Yk4O335hOxsPVnPNmblMTo/jD+8f5rqzJnDftXPC/jkAR6qa+f5Lu0iKcXP/9XMor/NRdKyOY7U+Xt5ZTtGxeowJLdE5ISWWxtY2GnwBGnwB2oOGRK8LhG4fED1dOz+XRz61MOw2daVBr5TqpuOag456fbM/QElFEzvK6th2tJYtR2r6XNf4nqtmccdFU0esx2+M4YG/7eJ/3j3E/InJbC+tY1ZOIlfOzSEt3kOC18X509JJjXMTsMLyRH0rGw9WsX5/JX/+sIx4r4vVX1naeU6kPWj4l//dwuodoYvaPE4HP75pfucFh8aYUfnG0h40NPoCJMW6ur2+MQZfW5AYtwN/e5ANxVUYDNlJMcS6ndS1tFHZ6MfrcpCTHKM9eqXU2CmrbaGspoX0hFD9+eev7+Ol7eUUZCWwKD+VWLcLt1NwOx24nEJKrJuspBgyErwUHatjy5FaZmQlUJCdSEaCh7R4D+kJXpJiQkHoDwR55M1ifvHGfj5/4RS+c81sXik6zq/eKu7sHffkcTk6F9tJ8Lq47qwJfP2KGb3q7L62dl7YUopThEX5qUzPOr3w/CjRoFdKDVswaHj6gyO8svM4u8rraQsE8bcHCQRNn1MrZyV6+1x72O0U0uO91PvaaPa3c+PCifzkpvndTpR3nLCuaGzl/QNV+AJBHBJaVjMr0cv5U9OZnZtk+0EFQ6FBr5QaVcGgoabZz0mr3p+XEsv0rATqmts4WtNMVZOfqsZWqpv8VDaGbsd5nFw4PYNls7M1sEfAQEGvUyAopYbN4RDSE7ykJ3iZ3WVC0+Q4N8lxyf0/UY2JsK6cEJHlIrJXRIpF5O4+HveKyLPW4xtFJL/LY/dY2/eKyJUj2HallFJhGDToRcQJPApcBcwBbhWROT12ux2oMcZMBx4CHrSeOwe4BZgLLAd+Zb2eUkqpMRJOj34xUGyMKTHG+IFngBU99lkBPGndfh5YJqHxRSuAZ4wxrcaYg0Cx9XpKKaXGSDhBnwcc7XK/1NrW5z7GmABQB6SH+VxE5A4RKRSRwoqK8NfwVEopNbhxMbuRMeYxY8wiY8yizMzMSDdHKaVsJZygLwMmdbk/0drW5z4i4gKSgaown6uUUmoUhRP0m4ACEZkiIh5CJ1dX9dhnFXCbdfsmYK0JDdBfBdxijcqZAhQAH4xM05VSSoVj0HH0xpiAiNwFvAo4gd8aY4pE5AGg0BizClgJPCUixUA1oQ8DrP2eA3YBAeDLxhj7zOavlFIfAePuylgRqQCGs4x9BlA5Qs0ZSdquodF2Dd14bZu2a2hOt11nGGP6PMk57oJ+uESksL/LgCNJ2zU02q6hG69t03YNzWi0a1yMulFKKTV6NOiVUsrm7Bj0j0W6Af3Qdg2NtmvoxmvbtF1DM+Ltsl2NXimlVHd27NErpZTqQoNeKaVszjZBP9ic+WPYjkki8qaI7BKRIhH5irX9fhEpE5Gt1r+rI9S+QyKyw2pDobUtTUReF5H91n9Tx7hNM7scl60iUi8i/y8Sx0xEfisiJ0VkZ5dtfR4fCfmF9Z7bLiILx7hdPxGRPdbP/rOIpFjb80Wkpctx+81otWuAtvX7txurNSr6adezXdp0SES2WtvH7JgNkBGj9z4zxnzk/xG6YvcAMBXwANuAORFqSy6w0LqdCOwjNI///cA3xsGxOgRk9Nj2Y+Bu6/bdwIMR/lseB86IxDEDLgIWAjsHOz7A1cDLgADnAxvHuF1XAC7r9oNd2pXfdb8IHbM+/3bW/wvbAC8wxfr/1jlW7erx+M+A+8b6mA2QEaP2PrNLjz6cOfPHhDGm3BizxbrdAOymj6mZx5mu6wk8CXwick1hGXDAGDOcq6NPmzHmbULTeHTV3/FZAfzehLwPpIhILqOgr3YZY14zoWnBAd4nNGngmOvnmPVnzNaoGKhdIiLA3wH/Oxo/eyADZMSovc/sEvRhzXs/1iS0pOICYKO16S7rq9dvx7o80oUBXhORzSJyh7Ut2xhTbt0+DmRHpmlAaJ6krv/zjYdj1t/xGU/vu88T6vV1mCIiH4rIOhFZGqE29fW3Gy/HbClwwhizv8u2MT9mPTJi1N5ndgn6cUdEEoAXgP9njKkHfg1MA84Gygl9bYyEJcaYhYSWhvyyiFzU9UET+q4YkTG3Epod9XrgT9am8XLMOkXy+PRHRO4lNGngH61N5cBkY8wC4GvA0yKSNMbNGnd/ux5upXuHYsyPWR8Z0Wmk32d2CfpxNe+9iLgJ/QH/aIx5EcAYc8IY026MCQKPE6ElFY0xZdZ/TwJ/ttpxouOroPXfk5FoG6EPny3GmBNWG8fFMaP/4xPx952IfBa4Fvi0FQ5YZZEq6/ZmQnXwGWPZrgH+duPhmLmATwLPdmwb62PWV0Ywiu8zuwR9OHPmjwmr9rcS2G2M+XmX7V1rajcAO3s+dwzaFi8iiR23CZ3M20n39QRuA/4y1m2zdOtljYdjZunv+KwCPmONijgfqOvy1XvUichy4FvA9caY5i7bM0XEad2eSmgdiJKxapf1c/v7242HNSouA/YYY0o7NozlMesvIxjN99lYnGUei3+EzkzvI/RJfG8E27GE0Feu7cBW69/VwFPADmv7KiA3Am2bSmjEwzagqOM4EVrf9w1gP7AGSItA2+IJrUqW3GXbmB8zQh805UAboVro7f0dH0KjIB613nM7gEVj3K5iQrXbjvfZb6x9b7T+vluBLcB1EThm/f7tgHutY7YXuGos22Vt/x3wTz32HbNjNkBGjNr7TKdAUEopm7NL6UYppVQ/NOiVUsrmNOiVUsrmNOiVUsrmNOiVUsrmNOiVUsrmNOiVUsrm/j9/dYCx7PB3jwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_results_test_data(weights22, biases22, 1000, activation=\"tanh\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fabulous-lindsay",
   "metadata": {},
   "source": [
    "###### Comparing Sigmoid and Tanh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "otherwise-satisfaction",
   "metadata": {},
   "source": [
    "By compiling the cells above for a number ot times and comparing the results we can find out that the tanh is slightly faster than the sigmoid function but its accuracy after trainig will be lower."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "attempted-milan",
   "metadata": {},
   "source": [
    "### Testing the entire dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "happy-alexandria",
   "metadata": {},
   "source": [
    "First thing we do is to randomly create our wights and biases matrices and to calculate the accuracy before doing ant trainings on the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "posted-pursuit",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy Before Training: 0.0905\n"
     ]
    }
   ],
   "source": [
    "W01, W12, W23, b1, b2, b3 = before_training(\"sigmoid\", 60000)\n",
    "weights_biases = (W01, W12, W23, b1, b2, b3)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "transsexual-rachel",
   "metadata": {},
   "source": [
    "Next we train our network by feeding the entire train images into the neural network and then training it by backpropagation. Here are some measurements we used to train our network:\n",
    "\n",
    "Learning Rate: 1 <br>\n",
    "Number of Epochs: 5 <br>\n",
    "Batch Size: 50 <br>\n",
    "Number of samples: 60000 <br>\n",
    "Activation function: Sigmoid <br>\n",
    "\n",
    "_________________________\n",
    "\n",
    "Accuracy before training: 0.11 <br>\n",
    "Wall time of running the training function: 1min 55s <br>\n",
    "Accuracy of test set after training: 0.92 <br>\n",
    "Accuracy of test set with 4 pixel shif after training: 0.29 <br>\n",
    "Accuracy of train set after training: 0.93 <br>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "private-proposal",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 1min 58s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "cost_by_epochs, weights3, biases3 = learning_vectorized_sgd(1, 5, 50, weights_biases, number_of_samples=60000, activation = \"sigmoid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "injured-bhutan",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy After Training: 0.9324\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD4CAYAAADlwTGnAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjQuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8+yak3AAAACXBIWXMAAAsTAAALEwEAmpwYAAAggklEQVR4nO3dfZAU933n8fd3Z/b5CbQsCPZpEGBL6FlaI1jih1ixjWxHJLEcI1vWrkt3qlysS3K+VEpOKrk71VUlvkr5IWcnOZVkg2RbSJYdG9uS5QfJTxICFj0jhLxCCyxCsDzDwrJP3/tjmmUYFnYWdqdnpj+vKoqe7l/PfLulmQ/d/Z0ec3dERCR6isIuQEREwqEAEBGJKAWAiEhEKQBERCJKASAiElHxsAuYiBkzZngikQi7DBGRvLJx48a97l6fPj+vAiCRSNDZ2Rl2GSIiecXMto01X6eAREQiKqMAMLNlZrbFzLrM7O4xlpea2cPB8nVmlgjm15nZU2Z21My+mjK+wsx+bGavmdkmM/unSdsiERHJyLgBYGYx4GvATcBC4FYzW5g27A7ggLvPB74EfCGY3w/8PfDXYzz1P7v7pcC1wFIzu+n8NkFERM5HJkcAi4Aud9/q7gPAamB52pjlwKpg+lHgRjMzd+9z99+SDIJR7n7M3Z8KpgeA54DGC9gOERGZoEwCoAHYkfK4J5g35hh3HwIOAXWZFGBm04A/BH6RyXgREZkcoV4ENrM48BDwL+6+9Sxj7jSzTjPr7O3tzW6BIiIFLJMA2Ak0pTxuDOaNOSb4UK8F9mXw3PcCv3P3L59tgLvf6+6t7t5aX39GG6uIiJynTAJgA7DAzOaaWQmwAliTNmYN0B5M3wI86ePcZ9rM/jfJoPirCVU8QSMjzrfXbefHL+2aypcREck7434RzN2HzOwu4AkgBnzd3TeZ2T1Ap7uvAe4HHjSzLmA/yZAAwMy6gRqgxMz+CPggcBj4O+A14DkzA/iqu983idsWvD48vGE7R08M8eErLyZ4LRGRyMvom8Du/hjwWNq8f0iZ7gc+fpZ1E2d52qx8EpsZ7W0JPvfIi/y2ay/vXqDTSCIiEJFvAn/kqtnMqCph5dPdYZciIpIzIhEApfEYn1zUzJNb9rBtX1/Y5YiI5IRIBADApxa3EDPjgbVj3hNJRCRyIhMAs2rKuOnK2TzSuYO+E0NhlyMiErrIBABAR1uCI/1DfO/59K8xiIhET6QC4LrmaVzZUMuqZ7oZ52sKIiIFL1IBYGZ0tCXo2nOUp7sy+aKyiEjhilQAAHz06tnUVZaw8pnusEsREQlV5AKgNB7jkzc084vXdrN937GwyxERCU3kAgDgUzckW0IffLY77FJEREITyQC4uLaMZVdczMMbdnBsQC2hIhJNkQwASLaEHu4f4j/UEioiERXZALi+ZTpXNNSoJVREIiuyAWBmtC9J8Pruo6x9Qy2hIhI9kQ0AgD+8eg4XVZbwDbWEikgERToAyopj3LqoiV9s3s2O/WoJFZFoiXQAANy2uAUz48FndZdQEYmWyAfA7Npyll1+MavXb1dLqIhESuQDAKBjabIl9PvPvxV2KSIiWaMAAFpbprNwtlpCRSRaFAAEdwldmmDL7iOs3aqWUBGJBgVA4Oar5zC9ophVagkVkYhQAASSLaHN/OzV3fQcUEuoiBQ+BUAKtYSKSJQoAFLMmVbOhy6fxer1Ozg+MBx2OSIiU0oBkKZ9SYJDxwf5wQu6S6iIFDYFQJpFcy/istk1rFRLqIgUOAVAmuQPx7fw2ttHWPfm/rDLERGZMgqAMSy/poFpFcWsfLo77FJERKZMRgFgZsvMbIuZdZnZ3WMsLzWzh4Pl68wsEcyvM7OnzOyomX01bZ3rzezlYJ1/MTOblC2aBGXFMVa8q5mfvvo2Ow8eD7scEZEpMW4AmFkM+BpwE7AQuNXMFqYNuwM44O7zgS8BXwjm9wN/D/z1GE/9b8B/BhYEf5adzwZMlU8vaQHgwbVqCRWRwpTJEcAioMvdt7r7ALAaWJ42ZjmwKph+FLjRzMzd+9z9tySDYJSZzQZq3P1ZT15pfQD4owvYjknXMK2cDy68mNUbttM/qJZQESk8mQRAA7Aj5XFPMG/MMe4+BBwC6sZ5zp5xnhMAM7vTzDrNrLO3tzeDcidPx9IEB4+pJVREClPOXwR293vdvdXdW+vr67P62jfMvYhLL65m5TPb1BIqIgUnkwDYCTSlPG4M5o05xsziQC1wrttq7gye51zPGbpkS2iCzbsOs14toSJSYDIJgA3AAjOba2YlwApgTdqYNUB7MH0L8KSf45/M7r4LOGxmi4Pun9uBH0y4+ixYfk0DteXFrFrbHXYpIiKTatwACM7p3wU8AWwGHnH3TWZ2j5ndHAy7H6gzsy7gc8Boq6iZdQNfBDrMrCelg+jPgfuALuAN4PHJ2aTJVV4SY8WiJp7YtJu31BIqIgXE8uncdmtrq3d2dmb9dXsOHOM9/+cp/uy98/ibZZdm/fVFRC6EmW1099b0+Tl/ETgXNE6v4AMLZ/HQerWEikjhUABkqL0twYFjg6x5UT8cLyKFQQGQoSWX1PHOWdWsfFp3CRWRwqAAyJCZ0d6W4NVdh+ncdiDsckRELpgCYAL+6No51JbrLqEiUhgUABNQURLnE+9q4ieb3mbXIbWEikh+UwBM0KcXt+DufFM/HC8ieU4BMEFNF1Vw42WzeGj9DrWEikheUwCch8+0JdjfN8AP1RIqInlMAXAelsyr4x2zqvTD8SKS1xQA5+FkS+imtw6zUS2hIpKnFADn6Y+vbaCmLM7KZ7rDLkVE5LwoAM7TyZbQx195m7cP9Y+/gohIjlEAXIBPL04w4s631qklVETyjwLgAjTXVXDjpbP49jrdJVRE8o8C4AJ1tCXY1zfAj1/aFXYpIiITogC4QEvn1zF/plpCRST/KAAu0MmW0Jd3HuK57QfDLkdEJGMKgEnwJ9c2UK2WUBHJMwqASVBZGudPW5t4/OVd7D6sllARyQ8KgEly+5IWht35lu4SKiJ5QgEwSVrqKnn/O2fy7fXbOTGkllARyX0KgEnUsTTB3qNqCRWR/KAAmES/N38G8+or1RIqInlBATCJzIyOtgQv9Rzi+R0Hwy5HROScFACT7E+ua6S6NM4qtYSKSI5TAEyyytI4H29t4scv7WKPWkJFJIcpAKbAaEvouu1hlyIiclYZBYCZLTOzLWbWZWZ3j7G81MweDpavM7NEyrLPB/O3mNmHUub/NzPbZGavmNlDZlY2KVuUAxIzKvn9d87kW+u2MzA0EnY5IiJjGjcAzCwGfA24CVgI3GpmC9OG3QEccPf5wJeALwTrLgRWAJcDy4B/NbOYmTUAfwG0uvsVQCwYVzDa2xLsPXqCx15WS6iI5KZMjgAWAV3uvtXdB4DVwPK0McuBVcH0o8CNZmbB/NXufsLd3wS6gucDiAPlZhYHKoC3LmxTcsu758/gkvpKvqGLwSKSozIJgAZgR8rjnmDemGPcfQg4BNSdbV133wn8M7Ad2AUccvefjvXiZnanmXWaWWdvb28G5eaGoiKjfUmCF3cc5Pnt+uF4Eck9oVwENrPpJI8O5gJzgEozu22sse5+r7u3untrfX19Nsu8YB+7vpEqtYSKSI7KJAB2Ak0pjxuDeWOOCU7p1AL7zrHuHwBvunuvuw8C3wPazmcDcllVaZxbrm/kxy/vYs8RtYSKSG7JJAA2AAvMbK6ZlZC8WLsmbcwaoD2YvgV40pP3QlgDrAi6hOYCC4D1JE/9LDaziuBawY3A5gvfnNzT3pZgcNj5tlpCRSTHjBsAwTn9u4AnSH5IP+Lum8zsHjO7ORh2P1BnZl3A54C7g3U3AY8ArwI/AT7r7sPuvo7kxeLngJeDOu6d1C3LEXNnVPK+d9arJVREco7l003LWltbvbOzM+wyJuyXW/bQ8Y0NfGXFNSy/Jv36uYjI1DKzje7emj5f3wTOgvcsqGfujEr9ZKSI5BQFQBYkW0JbeH77QV7UXUJFJEcoALLkY9c3UlkSU0uoiOQMBUCWVJcV8/HWJn740lv0HjkRdjkiIgqAbLp9SQuDw85D69USKiLhUwBk0SX1Vbz3HfV889ltagkVkdApALKsoy3BniMn+Mmmt8MuRUQiTgGQZe99Rz2JugpWPv1m2KWISMQpALKsqMi4fUmC57Yf5KWeg2GXIyIRpgAIwS2tyZZQfTFMRMKkAAhBTVkxH7u+kR+9uIu9R9USKiLhUACE5PYlCQaGR3hIdwkVkZAoAEIyf2YV714wg2+u28bgsFpCRST7FAAh+szSBLsPn+Anr6glVESyTwEQove9YyYtdRW6P5CIhEIBEKKTLaGd2w7wys5DYZcjIhGjAAjZx1sbqVBLqIiEQAEQspqyYj52XSNrXnyLfWoJFZEsUgDkgPa2FgaGRli9YUfYpYhIhCgAcsD8mdW8e8EMHlyrllARyR4FQI5oX5Lg7cP9/HTT7rBLEZGIUADkiN+/dCbNF1Ww8hndJVREskMBkCNiRcbtS1rY0K2WUBHJDgVADvl4axPlxfrheBHJDgVADqktL+ZPrmvgBy++xf6+gbDLEZECpwDIMR1tCQaGRvTD8SIy5RQAOWbBrGqWzq/jm89uY0gtoSIyhRQAOaijbS67DvXz01fVEioiUyejADCzZWa2xcy6zOzuMZaXmtnDwfJ1ZpZIWfb5YP4WM/tQyvxpZvaomb1mZpvNbMmkbFEBeP+lM2mcXq77A4nIlBo3AMwsBnwNuAlYCNxqZgvTht0BHHD3+cCXgC8E6y4EVgCXA8uAfw2eD+ArwE/c/VLgamDzhW9OYYgVGe1LEqx/cz+vvnU47HJEpEBlcgSwCOhy963uPgCsBpanjVkOrAqmHwVuNDML5q929xPu/ibQBSwys1rgPcD9AO4+4O4HL3hrCsifqiVURKZYJgHQAKTepawnmDfmGHcfAg4BdedYdy7QC3zDzJ43s/vMrHKsFzezO82s08w6e3t7Myi3MNRWFPPH1zXw/Rd2ckAtoSIyBcK6CBwHrgP+zd2vBfqAM64tALj7ve7e6u6t9fX12awxdO1LEpzQXUJFZIpkEgA7gaaUx43BvDHHmFkcqAX2nWPdHqDH3dcF8x8lGQiS4p0XV9M2r44H13arJVREJl0mAbABWGBmc82shORF3TVpY9YA7cH0LcCT7u7B/BVBl9BcYAGw3t3fBnaY2TuDdW4EXr3AbSlI7W0J3jrUz883qyVURCZXfLwB7j5kZncBTwAx4OvuvsnM7gE63X0NyYu5D5pZF7CfZEgQjHuE5If7EPBZdx8Onvq/At8KQmUr8JlJ3raC8AeXzaJhWjnfeLqbZVfMDrscESkglvyHen5obW31zs7OsMvIuv/3qzf4x8df4/G/fDeXza4JuxwRyTNmttHdW9Pn65vAeeAT72qirLhILaEiMqkUAHlgWkUJf3ytWkJFZHIpAPJEe1uC/sERHu5US6iITA4FQJ649OIaFl9yEQ+u1V1CRWRyKADySEfbXHYePM7PN+8JuxQRKQAKgDzyB5fNpGFauS4Gi8ikUADkkXisiE8vaWHt1n289rbuEioiF0YBkGc+0dpEabyIVc9sC7sUEclzCoA8M70y2RL6H8/3cPCYWkJF5PwpAPLQyZbQR9QSKiIXQAGQhy6bXcMNcy/igbXbGB7Jn1t5iEhuUQDkqY62BD0HjvML3SVURM6TAiBPfWDhLObUlumH40XkvCkA8lQ8VsRtS1p45o19vL77SNjliEgeUgDksRXvaqY0XqSjABE5LwqAPHZRZQnLr5nDfzy3k0PHBsMuR0TyjAIgz7W3JTg+OKyWUBGZMAVAnrt8Ti2LEhfxwLPdagkVkQlRABSAjqUJduw/zpOv6S6hIpI5BUAB+ODCWcyuLdNdQkVkQhQABSAeK+K2xS38tmsvv1NLqIhkSAFQIG5d1ExJvIhVa7vDLkVE8oQCoEBcVFnC8qvn8N2NOzl0XC2hIjI+BUABOdkS+h21hIpIBhQABeSKhlrelZiuu4SKSEYUAAWmvS3B9v3H+OUWtYSKyLkpAArMhy6/mItrdJdQERmfAqDAFMeKuG1xM7/53V669qglVETOLqMAMLNlZrbFzLrM7O4xlpea2cPB8nVmlkhZ9vlg/hYz+1DaejEze97MfnTBWyKjRltC9cPxInIO4waAmcWArwE3AQuBW81sYdqwO4AD7j4f+BLwhWDdhcAK4HJgGfCvwfOd9JfA5gvdCDldXVUpf3jVHL77XA+H+9USKiJjy+QIYBHQ5e5b3X0AWA0sTxuzHFgVTD8K3GhmFsxf7e4n3P1NoCt4PsysEfgIcN+Fb4ak62hLcGxgmO909oRdiojkqEwCoAFIbSzvCeaNOcbdh4BDQN04634Z+BtgZKJFy/iubKzl+pbpPLC2mxG1hIrIGEK5CGxmHwX2uPvGDMbeaWadZtbZ29ubheoKR0dbgm37jvHL19USKiJnyiQAdgJNKY8bg3ljjjGzOFAL7DvHukuBm82sm+Qppfeb2TfHenF3v9fdW929tb6+PoNy5aRlV1zMrJpSVupisIiMIZMA2AAsMLO5ZlZC8qLumrQxa4D2YPoW4El392D+iqBLaC6wAFjv7p9390Z3TwTP96S73zYJ2yMpimNF3HZDC79+vZc3eo+GXY6I5JhxAyA4p38X8ATJjp1H3H2Tmd1jZjcHw+4H6sysC/gccHew7ibgEeBV4CfAZ919ePI3Q87m1huaKYkV8YC+GCYiaSz5D/X80Nra6p2dnWGXkXc+98gLPPHK2zz7tzdSXVYcdjkikmVmttHdW9Pn65vAEdDRlqBvYJhHN6olVEROUQBEwFWN07iueRqrnlFLqIicogCIiPa2BN37jvGr36mVVkSSFAARcdMVs5lZXcrKp7vDLkVEcoQCICJK4kV86oYWfvV6L1vVEioiKAAi5ZM3NFMcMx5Yqy+GiYgCIFLqq0v56FVzeHRjD0d0l1CRyFMARExHW4KjJ4b4rlpCRSJPARAxVzdN45qmaTywdptaQkUiTgEQQZ9ZmmDr3j5+rZZQkUhTAETQTVfMpr66lFW6P5BIpCkAIijZEtrMU1t6eXNvX9jliEhIFAARdaoltDvsUkQkJAqAiJpZXcZHrpzNdzp7OHpiKOxyRCQECoAIaw9aQr/3nFpCRaJIARBh1zZP5+qmaazUXUJFIkkBEHEdbS1s7e3jt117wy5FRLJMARBxH75yNjOqSlmpllCRyFEARFxpPMYnb2jmqS176FZLqEikKACE225oJma6S6hI1CgAhJk1ZXz4ytl8p3MHfWoJFYkMBYAA0LE0wRG1hIpEigJAALi2aRpXNday8plu3NUSKhIFCgABwMzoaEvwhlpCRSJDASCjPnLVbGZUleguoSIRoQCQUaXxGJ9c1MwvXtvD9n3Hwi5HRKaYAkBO86nFLcTM+Lvvv8zq9dvp7N7PwWMDYZclIlMgHnYBkltm1ZTxZ++dx72/2cpvfnfqWkBdZQnzZlYxf2YV8+qTf8+fWcXsmjKKiizEikXkfFk+dXy0trZ6Z2dn2GVEwvCIs/PAcbp6j9C15yhv7Omjq/coXXuOcuj44Oi48uIY82ZWJkMhCIZ5M6tI1FVSEtcBpkguMLON7t6aPj+jIwAzWwZ8BYgB97n7P6UtLwUeAK4H9gGfcPfuYNnngTuAYeAv3P0JM2sKxs8CHLjX3b9yntsmUyBWZDTXVdBcV8H7L501Ot/d2dc3kAyFIBC69hyls/sAP3jhrdPWb7mogktSjhbm1Vcyb2YVNWXFYWySiKQZNwDMLAZ8DfgA0ANsMLM17v5qyrA7gAPuPt/MVgBfAD5hZguBFcDlwBzg52b2DmAI+O/u/pyZVQMbzexnac8pOcjMmFFVyoyqUhZfUnfasr4TQ2zt7RsNhpN//+r1PQwOnzrSnFldmhIKpwJiZnUpZjqdJJItmRwBLAK63H0rgJmtBpYDqR/Wy4H/GUw/CnzVku/k5cBqdz8BvGlmXcAid18L7AJw9yNmthloSHtOyTOVpXGubKzlysba0+YPDo+wY/+x5NFC76nTSd97budpv0ZWXRrnkpnJU0nzZlYGf1fRclEF8ZhOJ4lMtkwCoAHYkfK4B7jhbGPcfcjMDgF1wfxn09ZtSF3RzBLAtcC6sV7czO4E7gRobm7OoFzJNcWxIi6pr+KS+io+mDLf3dlz5MQZp5N+29XLd1NuSVEcMxJ1lacdLcwLQqKiRH0MIucr1HePmVUB3wX+yt0PjzXG3e8F7oXkReAslidTzMyYVVPGrJoyls6fcdqyw/2DvLHnKG/09o0GxOu7j/CzzbsZTvn1sjm1ZWN2J9VVluh0ksg4MgmAnUBTyuPGYN5YY3rMLA7UkrwYfNZ1zayY5If/t9z9e+dVvRSsmrJirm2ezrXN00+bf2JomO37jo0eLbzRmzyttHr9Do4PDo+Oqy0vTobBydNJM6uYX19Nw/RyYmpbFQEyC4ANwAIzm0vyw3sF8Mm0MWuAdmAtcAvwpLu7ma0Bvm1mXyR5EXgBsD64PnA/sNndvzg5myJRUBqPsWBWNQtmVZ82f2TE2XW4P2hZPTrasvqL13bzcOdAyvpFzJ2R7EYabVutr+KS+krKimPZ3hyRUI0bAME5/buAJ0i2gX7d3TeZ2T1Ap7uvIflh/mBwkXc/yZAgGPcIyYu7Q8Bn3X3YzH4P+DTwspm9ELzU37r7Y5O8fRIRRUVGw7RyGqaV89531J+27EDfAG/0nn6d4eWeQzz28i5Ofg3GDJqmVzCvvvKM00nTKkpC2CKRqacvgklk9Q8O8+bevtNPJ+05yta9fQwMjYyO07egJd9d0BfBRApRWXGMy2bXcNnsmtPmn+1b0D9+addZvwXdNL2CmvI4NWXF1JQXB3+felxdFqdYraySYxQAImkm+i3oDW/u54cvvsXIOAfTFSUxasqSYZAMifiYYXG2Mbq1hkw2BYBIhs71LeiREadvYIgj/UMc7h/k8PEhDh8fDKYHOdx/6vHJMXuPDrB1b9/o8uFxEqSsuCglJOLBkcW5giR+WqDoIrekUwCITIKiIqO6LPmBPIfyCa/v7hwbGD4VEKPhMXTWEDnQN8C2fcc4fHyQQ8cHGRonQEriRecMiZry+BmBUpsytjRepO9WFBgFgEgOMDMqS+NUlsaZXTv++HTuTv/gSEpYpAbG2IFy6PggPQeOjR6tDAyPnPM1SmJFo+FRncFRx6lASU6XF8cUIDlGASBSAMyM8pIY5SUxZtWUnddz9A8Oj3nUceQcRyJvHTw+OqZ/8NwBEi+y0ZCoKksGQllxjPLiZN2jj4Pp8uIYZSnT5SVFZ4xPHaOL7BOnABARINkVVVYcY2b1+GPHcmJoOOX01ZlHHUdOmx6if3CYoyeG6D1ygv7BYY4PDnN8YJj+wZFxj0bGEi+yM0IjOV00GhqjAVIco6IkPWDGD6TSeFFBtf4qAERkUpTGY5RWxZhRVXrBzzU0PEL/0EgQCKfC4Xgw3Z8yffqYkeTywWGODQxxfHCE/oFh9h4dOHPs4DDn8zWospOBcsYRyukBc3qgFGVwhHNqfHHMsnK6TAEgIjknHiuiKlZEVenUfUS5OyeGRs4ImP6UIDl32KSMHxym78QQe48OBOufWif1S4WZip08mkkJjx989vcoL5ncTi4FgIhEkpmNnvaaNoWvMzziaae4xg6Q1KOX9KOdYwPDU/I9EAWAiMgUihWd6vDKNbpsLiISUQoAEZGIUgCIiESUAkBEJKIUACIiEaUAEBGJKAWAiEhEKQBERCIqr34T2Mx6gW3nufoMYO8kljNZVNfEqK6JUV0TU6h1tbh7ffrMvAqAC2FmnWP9KHLYVNfEqK6JUV0TE7W6dApIRCSiFAAiIhEVpQC4N+wCzkJ1TYzqmhjVNTGRqisy1wBEROR0UToCEBGRFAoAEZGIKrgAMLNlZrbFzLrM7O4xlpea2cPB8nVmlsiRujrMrNfMXgj+/Kcs1PR1M9tjZq+cZbmZ2b8ENb9kZtdNdU0Z1vU+MzuUsq/+IUt1NZnZU2b2qpltMrO/HGNM1vdZhnVlfZ+ZWZmZrTezF4O6/tcYY7L+fsywrqy/H1NeO2Zmz5vZj8ZYNrn7y90L5g8QA94ALgFKgBeBhWlj/hz492B6BfBwjtTVAXw1y/vrPcB1wCtnWf5h4HHAgMXAuhyp633Aj0L4/2s2cF0wXQ28PsZ/x6zvswzryvo+C/ZBVTBdDKwDFqeNCeP9mEldWX8/prz254Bvj/Xfa7L3V6EdASwCutx9q7sPAKuB5WljlgOrgulHgRvNzHKgrqxz918D+88xZDnwgCc9C0wzs9k5UFco3H2Xuz8XTB8BNgMNacOyvs8yrCvrgn1wNHhYHPxJ7zrJ+vsxw7pCYWaNwEeA+84yZFL3V6EFQAOwI+VxD2e+EUbHuPsQcAioy4G6AD4WnDZ41MyaprimTGRadxiWBIfwj5vZ5dl+8eDQ+1qS/3pMFeo+O0ddEMI+C05nvADsAX7m7mfdX1l8P2ZSF4Tzfvwy8DfAyFmWT+r+KrQAyGc/BBLufhXwM06lvJzpOZL3Nrka+L/A97P54mZWBXwX+Ct3P5zN1z6XceoKZZ+5+7C7XwM0AovM7IpsvO54Mqgr6+9HM/sosMfdN071a51UaAGwE0hN6sZg3phjzCwO1AL7wq7L3fe5+4ng4X3A9VNcUyYy2Z9Z5+6HTx7Cu/tjQLGZzcjGa5tZMckP2W+5+/fGGBLKPhuvrjD3WfCaB4GngGVpi8J4P45bV0jvx6XAzWbWTfI08fvN7JtpYyZ1fxVaAGwAFpjZXDMrIXmRZE3amDVAezB9C/CkB1dUwqwr7TzxzSTP44ZtDXB70NmyGDjk7rvCLsrMLj553tPMFpH8/3jKPzSC17wf2OzuXzzLsKzvs0zqCmOfmVm9mU0LpsuBDwCvpQ3L+vsxk7rCeD+6++fdvdHdEyQ/I55099vShk3q/oqf74q5yN2HzOwu4AmSnTdfd/dNZnYP0Onua0i+UR40sy6SFxpX5Ehdf2FmNwNDQV0dU12XmT1Esjtkhpn1AP+D5AUx3P3fgcdIdrV0AceAz0x1TRnWdQvwX8xsCDgOrMhCiEPyX2ifBl4Ozh8D/C3QnFJbGPssk7rC2GezgVVmFiMZOI+4+4/Cfj9mWFfW349nM5X7S7eCEBGJqEI7BSQiIhlSAIiIRJQCQEQkohQAIiIRpQAQEYkoBYCISEQpAEREIur/A/LNZ5GW6x4RAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "show_results_test_data(weights3, biases3, 60000, \"sigmoid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "opponent-brook",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy After Training: 0.3227\n"
     ]
    }
   ],
   "source": [
    "show_results_4pixel_shifed(weights3, biases3, 60000, \"sigmoid\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "particular-tiffany",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy After Training: 0.9415\n"
     ]
    }
   ],
   "source": [
    "show_results_training_data(weights3, biases3, 60000, \"sigmoid\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "standing-appreciation",
   "metadata": {},
   "source": [
    "###### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cooked-invitation",
   "metadata": {},
   "source": [
    "The reason of the decrease in the accuracy of the 4pixel shifted test set is that the network is treained in the ways to set the weights ans biases for some exact input neurons, meaning that by changing those neurons the network will still apply the same amount of importantness to those despite the fact that our numbers have shifted by 4pixels and those neurons do not represent the numbered pixels of the image."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}