Planet_amazon_resnet34.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Kaggle Planet Competition: How to land in top 4%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "%reload_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os, sys\n",
    "sys.path.append('../')\n",
    "from fastai.conv_learner import *\n",
    "from fastai.plots import *\n",
    "import planet\n",
    "from planet import f2, opt_th"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = '../data/planet/' #path to folder containing data\n",
    "sz=64   #image size\n",
    "bs=64    #batch size "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluation Criteria and Initial Model \n",
    "If you read the evaluation criteria of competition, you will know it is based on f2 score. we define metrics for model accordingly. You can find the further information about F2 score [here](https://clusteval.sdu.dk/1/clustering_quality_measures/5). \n",
    "\n",
    "For our initial model, we will be using pre-trained implementation of deep residual model renet34 which was [made public by Microsoft](https://medium.com/r/?url=https%3A%2F%2Farxiv.org%2Fpdf%2F1512.03385.pdf) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics=[f2]\n",
    "f_model = resnet34"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [],
   "source": [
    "#  Prepare data Validation set\n",
    "label_csv = f'{path}train_v2.csv'\n",
    "n = len(list(open(label_csv)))-1 #total number of images\n",
    "val_idxs = get_cv_idxs(n) #it will return 20% indexes from training data set to used for val data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_data(sz, bs):\n",
    "    \"\"\"Returns data generator\"\"\"\n",
    "    tfms =  tfms_from_model(f_model, sz, aug_tfms=transforms_top_down, max_zoom=1.05)\n",
    "    return ImageClassifierData.from_csv(path, 'train-jpg', f'{path}train_v2.csv', bs, tfms, suffix='.jpg', val_idxs=val_idxs, test_name='test-jpg')\n",
    "data = get_data(bs, sz)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = ConvLearner.pretrained(f_model, data, metrics=metrics) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding the Learning Rate\n",
    "Learning rate(LR) is one of the most important hyper parameter of your model. It determines how fast or slow your model will learn.If LR is too high, model will try to learn too fast and loss function will not converge. If LR is very too low you model will take too long to converge.\n",
    "\n",
    "Finding a good learning rate using fastai library is very easy, just use the  following two commands. They will plot a graph of LR against loss function, a good value for LR will be where the slop of the loss function is highest. As we can see slope is highest between *0.1 to 1*, you can use any value in between this range. It would be a good idea to experiment with a few values in this range to find the optimal value.After experimenting with three values, 0.2 seemed to work best for me."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 228,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f79e1615d95c4db79a40babf532f5c3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=1), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  0%|          | 1/506 [00:00<06:47,  1.24it/s, loss=0.853]  \n",
      "  1%|          | 3/506 [00:00<02:47,  3.01it/s, loss=0.839]"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.22186  0.31918  0.80651]                        \n",
      "\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEOCAYAAABmVAtTAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XmcFPWd//HXp3vuYQ5gBgSGWxBQFHFASYJiPKI5JB4xksRoEoMmazySzcZcu/lp7uwmmzUmkZhsjNHglRhUEjWJZ1RkQBABQQSE4Rxu5r4+vz+6nR3HYegZpqjunvfz8egHVdXfrv58p4Z+T1X1t8rcHREREYBI2AWIiEjyUCiIiEgbhYKIiLRRKIiISBuFgoiItFEoiIhIG4WCiIi0USiIiEgbhYKIiLRRKIiISJuMsAvorpKSEh81alTYZYiIpJQlS5bscvfSw7VLuVAYNWoUFRUVYZchIpJSzOzNRNrp8JGIiLRRKIiISBuFgoiItFEoiIhIG4WCiIi0USiIiEibPhMK1Q3NLN64h427asIuRUQkaaXcOIWe+s1zG/jxE2sBeN/xg3nPuFLOP+EYSvpl09rqvLh+N9UNzWzaU8vG3TWs2HIAd+fEsiIG5GczZ/pwhhTlHnL97k5Dcys5mdGj1SURkV5n7h52Dd1SXl7uPRm8tmFXDet2VvP31Tt45JVtVDc0k5cVZdKQQnYcrGfznrq3tZ82qj+tDqu3HaC+qYWczCjTRw/AHcxg0+5aGppbaXWnpdWpa2zhYEMzJf2ymTikgOYWp7qhmf11TWw/UM+IAXmMHJBHTlaUU0cPICcjSlV1A9GIkZ8VpV9OBiMG5FHSL5sRA/Iws976kYmIYGZL3L38sO36Sii05+6s2naAn/1jHfvrmsjNjPKBE4cwcmA+RbkZ7K9r5pSR/dvab9pdy0/+tpb1VdW0OjS3OqNL8sjNzCBiEI0YOZlRBuRn8ebuWtbuOEhuZpT87CgFOZkMKshmXVU1VQcb2FfbxJZ9dV1UBwPys5h1XCljS/uxu7qRQYXZ1De14A5b9tXxj9d2ApCTEcGBWceVUpSbRUm/rFhANbUwsF82A/OzKMzJpHJvLblZUTbvqeWNqhrMoHJvHZlRIzczg5ED8xiQn8UxhTkMKsxmy946GltaiUaMSUMKycqI0NjcSl1jC5kZEbIzIhxTmENJv2wiEYWXSCpQKCQpd2fj7loiBoMLc2h1p6ahhYP1TazceoCD9c28sH43L7yxm13VDW97rRkMzM9myvAiBhfmUNfYQl1TC/94bScNza0Jvf/gwmxaHUYNzKOl1altbGHj7hrqmxJ7fXtZGRHKinMpG5BH/7xMCnIyGFqcyzGFOQA0tzhDinMYVJDDsP659MvuM0crRZJOoqEQ6P9SMzsP+CkQBe5w9+93eH4EcCdQHG9zk7svDLKmsJkZo0vy37YsLyuD0oJsxpT2A+Bjp44AYH9dE9kZERqaWsnJihAxIzP6zu8G1DQ0YwaNza1EIkZuZpS9tY3srm5kb00jQ4pzaWltJSMSYVSH9wZobmmlobmVyr11HKhvYkB+FjmZUVpbnZVbD9DqTm5WlOxohAP1sffaeaCeyr11bN5bS+XeOjbuquFAfRP7apsO2feC7AxKCrI5eUQxeVlRBuRl0T8/q20vpXzUAKLa8xAJVWB7CmYWBdYC5wCVwGJgjruvatdmHvCyu//CzCYBC919VFfrTfU9hXS3t6aRPbWNuEOrO/tqm9hxoJ4t++rYvr+eyr21vFK5n6aWVvbVNdH+129YcS5ThhczrH8uZf1zGVacy7D+uWRnRMnPilJakK1zLSI9lAx7CtOBde6+Pl7QfGA2sKpdGwcK49NFwNYA65GjoH9+7K//RLS0Ogfqmthd08hr2w/w0MtbWL3tAE+s3kFjJ4fDinJjh6gmDyvinEmDObGsiNEl/bR3IdKLggyFYcDmdvOVwKkd2nwLeNzMvgDkA2cHWI8kmWjE2kLk2EH9+OCJQwFobXV21TSwZW8dlXvr2s6drN1xkAP1zTy1Zid/eXU7ABkRY+a4EuZMH8F7Jwwio5PDayKSuCBDobM/3zoeq5oD/Nbd/8vMZgB3mdkJ7v62PxPNbC4wF2DEiBGBFCvJIxIxBhXETlCfPKL/O55vaXXW7axmeeU+Xtt2kEde2crcu5YwuDCbi6aWMWPMQE4aXkxRbmYI1YuktiDPKcwAvuXu74vPfxXA3b/Xrs1K4Dx33xyfXw+c5u47D7VenVOQjppbWvnHazv5w0ubeHptFa3xX+mzJgziqpljGFuaT3FeFlkZ2ouQvisZziksBsaZ2WhgC3AZ8LEObTYBZwG/NbOJQA5QFWBNkoYyohHOPf4Yzj3+GA7UN/HK5v28sH4Xdz7/Jn+Pj+nol53BGeNLOff4wbzv+GM08lzkEAIdp2Bm7wf+m9jXTX/j7t8xs5uBCndfEP/G0a+AfsQOLf2buz/e1Tq1pyCJqm1s5uk1VeyqaWTV1v08sWonu6obKM7L5IzxpfzLmccyfnBB2GWKHBUavCbSQWur8+KG3dy3eDN/f20ntY0tzJk+nLkzxzJiYF7Y5YkEKhkOH4kklUjEeNfYEt41toQ9NY385Im1zF+8iXsWbeITp43ki+eMpzgvsa/TiqQr7SlIn7Z9fz23PbmOuxe9SXFeFjedN4FLTinTNZ0k7SS6p6CvY0ifdkxRDrd8+AQevW4mY0ry+bcHX+Ejt7/Aisr9YZcmEgqFgggwcUgh9109g//8yEls3FXDh372HDfMf5ndHS5KKJLuFAoicZGIcckpZTz15Vlce+axPLpiG2f/+Gn+Gh89LdIXKBREOijIyeRf33ccj143kxED8rjm90uY98wbpNr5N5GeUCiIHML4wQXce/UMPnDiEL678DU++7slOpwkaU+hINKFnMwot152MjedP4FnX6/igp/9k9e2Hwi7LJHAKBREDiMSMa45Yyz3XT2DppZWLrztee54dj3NLd2/W51IslMoiCTopOHFPPyF9/CusQP59qOrueBn/2T1Nu01SHpRKIh0w+DCHO64opyff3wqu6ob+MgvX+D5dbvCLkuk1ygURLrJzHj/5CH8+dp3M7Q4hyv+9yUeXq6bBkp6UCiI9NCQolzuv/pdnDy8P9fNf5m7Xnwz7JJEjphCQeQIFOVl8rvPTOfsiYP55kOvcs+iTWGXJHJEFAoiRygnM8ptH5vKeycM4mt/WsF9izcf/kUiSUqhINILsjIi/PzjUzl9fClf+eMr/OnlyrBLEukRhYJIL8nJjDLv8lOYMWYgX7pvOQt08llSkEJBpBflZEa544pyykcN4MZ7l/GXFdvCLkmkWxQKIr0sLyuD31w5jSnDi/nCH17miVU7wi5JJGEKBZEA9MvO4H8/NY3jhxXx+buX8MzaqrBLEkmIQkEkIIU5mfzu09MZW9qPf7l7KWu2Hwy7JJHDUiiIBKgoN5PfXDmNnKwoH/vVi6zbWR12SSJdUiiIBGxocS7z556GGVx9VwXVDc1hlyRySIGGgpmdZ2ZrzGydmd3UyfM/MbNl8cdaM9sXZD0iYRlb2o9b50xl4+5avnz/ct3FTZJWYKFgZlHgNuB8YBIwx8wmtW/j7je6+xR3nwLcCvwxqHpEwjZj7EC+ct5x/OXV7fzq2fVhlyPSqSD3FKYD69x9vbs3AvOB2V20nwP8IcB6REL32ZljOP+EY/jhX9fw6pb9YZcj8g5BhsIwoP1FYCrjy97BzEYCo4F/BFiPSOjMjO9eOJkB+Vl86b7lNDS3hF2SyNsEGQrWybJDHUi9DHjA3Tv9H2Jmc82swswqqqr0fW9Jbf3zs/jBJSeyZsdB/vtvr4ddjsjbBBkKlcDwdvNlwKEuBnMZXRw6cvd57l7u7uWlpaW9WKJIOM48bhCXlpfxq2fWs6JSh5EkeQQZCouBcWY22syyiH3wL+jYyMyOA/oDLwRYi0jSuen8iQwqyOazv6tg54H6sMsRAQIMBXdvBq4FHgNWA/e5+0ozu9nMLmjXdA4w3/UdPeljBuRnMe+T5eyra+Rrf1oRdjkiAFiqfRaXl5d7RUVF2GWI9Jp5z7zBdxe+xncuPIGPnzoy7HIkTZnZEncvP1w7jWgWCdln3jOG08eXcssjq9i8pzbscqSPUyiIhCwaMb5/0WSiZnztTys02llCpVAQSQJDi3P5t/Mm8Ozru3hUN+aRECkURJLEJ04byQnDCrn54VUcrG8KuxzpoxQKIkkiGjG+/eHJVFU3aFCbhEahIJJEpgwvZs70Efz2+Y2s26mb8sjRp1AQSTJfOmc8eVlRbnlkddilSB+kUBBJMgP7ZXP9WeN4em0VT762M+xypI9RKIgkoU/OGMWYknxueXQVjc2tYZcjfYhCQSQJZWVE+MYHJ7K+qobfvbAx7HKkD1EoiCSpM48bxOnjS/np319nd3VD2OVIH6FQEElSZsY3PzCR2sYWfvzE2rDLkT5CoSCSxMYNLuDy00byh5c2sXaHvqIqwVMoiCS5688aR05mlNueXBd2KdIHKBREklz//CwuP20kDy/fyvqq6rDLkTSnUBBJAVfNHENWRoTbnnwj7FIkzSkURFJAaUE2nzh1JA8t28LGXTVhlyNpTKEgkiLmnjGGjIhx6z90bkGCo1AQSRGDCnL4xGnaW5BgKRREUsjV2luQgCkURFLIoIIcPh4/t6D7OUsQFAoiKeaqmaMx4NfPbQi7FElDCgWRFDO0OJfZU4Zx7+LN7K1pDLscSTOBhoKZnWdma8xsnZnddIg2l5rZKjNbaWb3BFmPSLq4+owx1DW1cOcLG8MuRdJMYKFgZlHgNuB8YBIwx8wmdWgzDvgq8G53Px64Iah6RNLJ+MEFvHfCIH7/4pu634L0qiD3FKYD69x9vbs3AvOB2R3afBa4zd33Ari7bjMlkqDLTxvJrupGnli1I+xSJI0EGQrDgM3t5ivjy9obD4w3s3+a2Ytmdl6A9YikldPHlzKsOJc/vLQp7FIkjQQZCtbJMu8wnwGMA2YBc4A7zKz4HSsym2tmFWZWUVVV1euFiqSiaMT46LThPLdulwazSa8JMhQqgeHt5suArZ20+bO7N7n7BmANsZB4G3ef5+7l7l5eWloaWMEiqebS8uFEI8b8xZsP31gkAUGGwmJgnJmNNrMs4DJgQYc2DwFnAphZCbHDSesDrEkkrRxTlMN7JwzigSWbdcJZekVgoeDuzcC1wGPAauA+d19pZjeb2QXxZo8Bu81sFfAk8GV33x1UTSLp6GOnjmBXdSOPr9oedimSBsy942H+5FZeXu4VFRVhlyGSNFpandN/+CSjSvK4+6rTwi5HkpSZLXH38sO104hmkRQXjRiXTRvOP9ft1glnOWIKBZE0cOm02AnnPyzW11PlyCgURNLA4MIczpowiAcqKnXCWY6IQkEkTXzs1BHsrtEJZzkyCgWRNDFzXGyE8z2LdAhJek6hIJImohFjzvThPP/GbjbohLP0kEJBJI20jXDW9ZCkhxQKImlkUGEOZ08cxP1LKmlobgm7HElBCgWRNPOxU0eyp6aRx1fqktrSfQoFkTQz89gSyvrrhLP0jEJBJM1EIsac6SN4Yf1u1ldVh12OpBiFgkgauuSUMiIGf3p5S9ilSIpRKIikocGFObz72BL+uHQLra2pddFLCZdCQSRNXTy1jC376li8cU/YpUgKUSiIpKlzjx9MXlaUPy7VISRJXEKhYGbXm1mhxfzazJaa2blBFyciPZeXlcH5Jwzh0RXbqG/SmAVJTKJ7Cp929wPAuUAp8Cng+4FVJSK94qKpw6huaObxVRqzIIlJNBQs/u/7gf919+XtlolIkpoxZiBDi3J4cEll2KVIikg0FJaY2ePEQuExMysAdNF2kSQXiRgXTh3Gs69XseNAfdjlSApINBQ+A9wETHP3WiCT2CEkEUlyF00to9XhIY1ZkAQkGgozgDXuvs/MPgF8A9gfXFki0lvGlvbj5BHFPLi0EneNWZCuJRoKvwBqzewk4N+AN4HfBVaViPSqi6eWsXZHNa9uORB2KZLkEg2FZo/9iTEb+Km7/xQoCK4sEelNHzpxKFkZER5cqhPO0rVEQ+GgmX0VuBx41MyixM4rdMnMzjOzNWa2zsxu6uT5K82sysyWxR9Xda98EUlEUV4m50wczJ+XbaGxWd8RkUNLNBQ+CjQQG6+wHRgG/KirF8SD4zbgfGASMMfMJnXS9F53nxJ/3JF46SLSHRefMoy9tU08uWZn2KVIEksoFOJBcDdQZGYfBOrd/XDnFKYD69x9vbs3AvOJHX4SkRCcPq6Ukn7ZGrMgXUr0MheXAi8BHwEuBRaZ2SWHedkwYHO7+cr4so4uNrNXzOwBMxt+iPefa2YVZlZRVVWVSMki0kFGNMKHpwzlyTU72VPTGHY5kqQSPXz0dWJjFK5w908S2wv45mFe09mI547fh3sYGOXuJwJ/A+7sbEXuPs/dy929vLS0NMGSRaSji08po6nFWbBMYxakc4mGQsTd2x+I3J3AayuB9n/5lwFb2zdw993u3hCf/RVwSoL1iEgPTBxSyKQhhTyoK6fKISQaCn81s8fi3xa6EngUWHiY1ywGxpnZaDPLAi4DFrRvYGZD2s1eAKxOsB4R6aGLTyljxZb9rN1xMOxSJAkleqL5y8A84ETgJGCeu3/lMK9pBq4FHiP2YX+fu680s5vN7IJ4s+vMbKWZLQeuA67sWTdEJFGzpwwlI2I64SydslQb9l5eXu4VFRVhlyGS0q66czErtuzn+ZvOIhrRBY/7AjNb4u7lh2vX5Z6CmR00swOdPA6amcbLi6Soi6eWseNAA8+t2xV2KZJkugwFdy9w98JOHgXuXni0ihSR3vXeiYMoys3UISR5B92jWaQPys6IcsFJQ3ls5XYO1jeFXY4kEYWCSB910dRhNDS3snDFtrBLkSSiUBDpo6YML2ZMaT4PLtGYBfk/CgWRPsrMuHhqGS9t3MOm3bVhlyNJQqEg0oddNHUYZug+C9JGoSDShw0pyuXdY0v448uVtLam1pglCYZCQaSPu+SUMjbvqWPRhj1hlyJJQKEg0se97/hjKMjO4P6KzYdvLGlPoSDSx+VmRfnQlKEsfHWbxiyIQkFEYoeQ6pta+cuK7WGXIiFTKIgIJw8vZnRJvr6FJAoFEXlrzMIwFm3Yw+Y9GrPQlykURASAi6aWETG456VNYZciIVIoiAgAQ4tzOWfSYOa/tIn6ppawy5GQKBREpM0VM0axt7aJR17RRfL6KoWCiLSZMXYgxw7qx53PbyTV7soovUOhICJtzIwrZoxkxZb9LNu8L+xyJAQKBRF5mwunltEvO4PfvfBm2KVICBQKIvI2/bIzuOSUMh59ZRtVBxvCLkeOMoWCiLzD5TNG0tjSyr2L9fXUvibQUDCz88xsjZmtM7Obumh3iZm5mZUHWY+IJGZsaT9mjivh9y9uormlNexy5CgKLBTMLArcBpwPTALmmNmkTtoVANcBi4KqRUS674oZo9h+oJ4nVu0IuxQ5ioLcU5gOrHP39e7eCMwHZnfS7hbgh0B9gLWISDedOWEQZf1z+e3zG8MuRY6iIENhGND+Au2V8WVtzOxkYLi7PxJgHSLSA9GIcflpI1m0YQ+vbT8QdjlylAQZCtbJsrbRMGYWAX4CfOmwKzKba2YVZlZRVVXViyWKSFcuLR9OdkZEX0/tQ4IMhUpgeLv5MmBru/kC4ATgKTPbCJwGLOjsZLO7z3P3cncvLy0tDbBkEWmvf34Ws6cM5U9Lt7C/Tjfg6QuCDIXFwDgzG21mWcBlwIK3nnT3/e5e4u6j3H0U8CJwgbtXBFiTiHTTJ2eMoq6phfsW63adfUFgoeDuzcC1wGPAauA+d19pZjeb2QVBva+I9K4ThhXxrrEDuf2Z9dQ16uqp6S7QcQruvtDdx7v7WHf/TnzZv7v7gk7aztJegkhyuvGc8eyqbuD3L+rcQrrTiGYROaxpowbwnmNL+OXTb1Db2Bx2ORIghYKIJOTGc8axu6aRu/RNpLSmUBCRhJwycgAzx5Xo3EKaUyiISMK+8N5x7Klp5MGllWGXIgFRKIhIwqaN6s9Jw4v59XMbaG3VndnSkUJBRBJmZsydOYYNu2p4dIXu45yOFAoi0i3nnXAME44p4L8eX0OTLquddhQKItIt0Yjxr+cex8bdtfzhJd2EJ90oFESk286aOIgZYwbyo8fWsPOgrnqfThQKItJtZsa3LzyBhqZWvv3I6rDLkV6kUBCRHhlb2o9rZo1lwfKtPLNWl7RPFwoFEemxz88ay+iSfL7551epb9KAtnSgUBCRHsvJjHLL7BN4c3ctP39yXdjlSC9QKIjIEXnPuBJmTxnKL55+g3U7q8MuR46QQkFEjtg3PjCJ3MwoX//TCtw10jmVKRRE5IiVFmTzlfMnsGjDHv64dEvY5cgRUCiISK+YM20EU0cU852Fq9lb0xh2OdJDCgUR6RWRiPGdCyezv66J7y7U2IVUpVAQkV4zcUghn505hvuXVPLs6xq7kIoUCiLSq244exxjSvO56cEV1DTo1p2pRqEgIr0qJzPKDy8+ka376/jhX18LuxzpJoWCiPS68lEDuGLGKO584U1e2rAn7HKkGxQKIhKIL7/vOMr65/KVB1/RJTBSSKChYGbnmdkaM1tnZjd18vw1ZrbCzJaZ2XNmNinIekTk6MnPzuAHF5/Ihl013PLIqrDLkQQFFgpmFgVuA84HJgFzOvnQv8fdJ7v7FOCHwI+DqkdEjr53H1vC3NPHcPeiTTy1ZmfY5UgCgtxTmA6sc/f17t4IzAdmt2/g7gfazeYDGh8vkma+dO54xpbmc8O9y9i4qybscuQwggyFYcDmdvOV8WVvY2b/YmZvENtTuC7AekQkBNkZUX5z5TTc4fr5L+u+zkkuyFCwTpa9Y0/A3W9z97HAV4BvdLois7lmVmFmFVVVGhAjkmpGDsznexdNZnnlfv77b2vDLke6EGQoVALD282XAVu7aD8f+HBnT7j7PHcvd/fy0tLSXixRRI6W908ewqXlZfz8qTdYtH532OXIIQQZCouBcWY22syygMuABe0bmNm4drMfAF4PsB4RCdl/fOh4Rg7I48Z7l7G/tinscqQTgYWCuzcD1wKPAauB+9x9pZndbGYXxJtda2YrzWwZ8EXgiqDqEZHw5Wdn8NPLTmbnwQa+pnsvJKWMIFfu7guBhR2W/Xu76euDfH8RST4nDS/mi+eO54d/XcMp/+zPp98zOuySpB2NaBaRo+6a08dyzqTBfGfhaio26jIYyUShICJHXSRi/PjSkxhanMOnf7uYXz2zXpfCSBIKBREJRUFOJnd+ajpTRvTnOwtXc9m8F6lt1KW2w6ZQEJHQjCntx+8+PZ1b55zMK5X7uPgXL7CvVrfyDJNCQURC96GThvLrK6fxxs5qPnNnBdW6OU9oFAoikhTOPG4QP71sCss27+PyXy/SOYaQKBREJGmcP3kIt845mZc37ePae15WMIRAoSAiSeX9k4dw8+zj+dvqHXz+7qW0tGqA29GkUBCRpPPJGaO4Zfbx/OO1nfzosTUa+XwUBTqiWUSkpz5x2khWbTvIL59+g+qGJv7jQ8eTGdXfsUFTKIhIUjIzvvPhEyjMzeD2p9ezaU8dv/j4VPKz9bEVJMWuiCStSMT46vkT+cHFk3nu9Spm3/ZP/vrq9rDLSmsKBRFJeh+dNoJ5l5fT0NzC5+5ewl0vbNR5hoAoFEQkJZw9aTBP3HgGZ4wv5Zt/Xsnnfr9Ug9wCoFAQkZSRkxnl11dM46vnT+CJ1TuY9aOn+Oe6XWGXlVYUCiKSUqIR4+ozxjJ/7mkU5mRwzV1LeGylzjP0FoWCiKSkaaMG8PurTmV0aT5X37WE8/77GRau2BZ2WSlPoSAiKWtocS73XzODb31oEgCfv3spNz34CnWNujxGT+kLvyKS0rIzolz57tF8/LSR/PiJtfzy6TeoeHMvV7xrFOcdfwylBdlhl5hSLNW+1lVeXu4VFRVhlyEiSerZ16u48d7l7KpuIC8ryufOGMupYwYyeVgRuVnRsMsLjZktcffyw7ZTKIhIumlsbuWNqmp+8sRaHl+1A4C8rCjnTBrMBScNZea4UrIy+tbRc4WCiAjw+o6DvFFVw9Nrd7JwxXb21zVRlJvJ+Sccw4yxAzl+aCFjS/thZmGXGiiFgohIB43NrTy3roqHl2/j8ZXbqYmfkB45MI/rzxrHzHGlOE5BdmZSHWraX9fEr59dzyWnDGfEwLwerSPRUAj0RLOZnQf8FIgCd7j79zs8/0XgKqAZqAI+7e5vBlmTiPRdWRkR3jthMO+dMJi6xhY27Kph6aa93LNoE1+8b3lbu2jEGFuaT25WBkOLcjhr4mDys6K0uDNxSCFjSvKpb2rFDLbsq2PEgDwamlt5bdsBtu6vJycjwunjS8mKRohEer4H4u7c8ewGbv3H6xyob6akIJtPzhjVCz+JQwtsT8HMosBa4BygElgMzHH3Ve3anAkscvdaM/scMMvdP9rVerWnICK9raG5hRfX72FDVTWNLa0crG9m1dYDNLa0smb7QXYebHhb++K8TA7UNfHW/X/eCoz6pta2NjmZERqbWzltzEBuOHs8Q4pyyMqIUNPQzKiB+Tix8AGob2qhoamVwtwM6ppaMIxIBL7+p1d5YEkls44r5V/PPY4ThhX1uI/JsKcwHVjn7uvjBc0HZgNtoeDuT7Zr/yLwiQDrERHpVHZGlDPGl3LG+NJ3PNfU0srGXTU0tzqt7qyo3M+iDXswoGxAHsOKc1i59QARM2aOK2H4gDy27qvjb6t3kBGJ8OdlW7j09hfesd7+eZmcWFbM7poGNlTVUNPYQlY0QmNLK5lRo192Bntrm7j+rHHccPa4o3bOI8hQGAZsbjdfCZzaRfvPAH8JsB4RkW7LjEYYN7igbf74oUVcNn1El68ZP7iAWccNAuCGs8fxwhu7OVjfTENLK7iz82ADSzftZdv+OoYU5XJSWTGjBuazq6aBwpxMDtQ3sWVvHR8pH95pUAUpyFDoLNY6PVZlZp8AyoEzDvH8XGAuwIgRXW8MEZFkUpyXxfmTh4RdRsKC/KLEBsa6AAAJnUlEQVRuJTC83XwZsLVjIzM7G/g6cIG7N3R8HsDd57l7ubuXl5Ye3dQUEelLggyFxcA4MxttZlnAZcCC9g3M7GTgdmKBsDPAWkREJAGBhYK7NwPXAo8Bq4H73H2lmd1sZhfEm/0I6Afcb2bLzGzBIVYnIiJHQaDjFNx9IbCww7J/bzd9dpDvLyIi3dO3Lv4hIiJdUiiIiEgbhYKIiLRRKIiISJuUu0qqmVUB+4D98UVFnUyXALuO4G3ar7MnbTp7ruOyzuo+1HSq9afj/FvT7ZeF2adE+tNxmbZR9yTSn67a9VZ/2k+n2jbq7f6MdPfDD/Ry95R7APO6mgYqemv9PWnT2XMdlx2uD6ncny760X5ZaH1KpD/aRkfWp0T6c6TbqLv/p1JtGwXdn0M9UvXw0cMJTPfW+nvSprPnOi5LpA+p2p+O8w8fos2ROJI+JdKfjsu0jbon0fUcyTbqyf+pI5FunwudSrnDR4kwswpP4BKxqSLd+gPp16d06w+kX5/Un8Sk6p7C4cwLu4Belm79gfTrU7r1B9KvT+pPAtJyT0FERHomXfcURESkBxQKIiLSRqEgIiJt+lwomNksM3vWzH5pZrPCrqc3mFm+mS0xsw+GXUtvMLOJ8e3zgJl9Lux6jpSZfdjMfmVmfzazc8Ou50iZ2Rgz+7WZPRB2LUci/v/mzvi2+XjY9Ryp3touKRUKZvYbM9tpZq92WH6ema0xs3VmdtNhVuNANZBD7O5woeml/gB8BbgvmCq7pzf65O6r3f0a4FJit2kNTS/15yF3/yxwJfDRAMs9rF7qz3p3/0ywlfZMN/t3EfBAfNtc8I6VJYHu9KfXtksQI+KCegCnA1OBV9stiwJvAGOALGA5MAmYDDzS4TEIiMRfNxi4Ow36czaxu9pdCXwwHbZR/DUXAM8DH0uH/sRf91/A1DTqzwNh/74dYf++CkyJt7kn7NqPtD+9tV0CvclOb3P3Z8xsVIfF04F17r4ewMzmA7Pd/XtAV4dT9gLZQdSZqN7oj5mdCeQT+yWvM7OF7t4aaOFd6K1t5O4LgAVm9ihwT3AVd62XtpEB3wf+4u5Lg624a738fyjpdKd/xI4UlAHLSNKjJt3sz6reeM+k/EF00zBgc7v5yviyTpnZRWZ2O3AX8LOAa+uJbvXH3b/u7jcQ++D8VZiB0IXubqNZZvY/8e208FDtQtSt/gBfILZHd4mZXRNkYT3U3e0z0Mx+CZxsZl8NurhecKj+/RG42Mx+QcCXjuhlnfant7ZLSu0pHIJ1suyQI/Lc/Y/EfhmSVbf609bA/be9X0qv6e42egp4KqhiekF3+/M/wP8EV84R625/dgPJGG6H0mn/3L0G+NTRLqYXHKo/vbJd0mFPoRIY3m6+DNgaUi29Id36A+nXJ/UntaRb/wLtTzqEwmJgnJmNNrMsYiddF4Rc05FIt/5A+vVJ/Ukt6da/YPsT9tn1bp6J/wOwDWgilpafiS9/P7CW2Bn5r4ddZ1/tTzr2Sf1JrUe69S+M/uiCeCIi0iYdDh+JiEgvUSiIiEgbhYKIiLRRKIiISBuFgoiItFEoiIhIG4WCBM7Mqo/Ce1yQ4GXGe/M9Z5nZu3rwupPN7I749JVmlhTX4DKzUR0v0dxJm1Iz++vRqkmOPoWCpAwzix7qOXdf4O7fD+A9u7o+2Cyg26EAfA24tUcFhczdq4BtZvbusGuRYCgU5Kgysy+b2WIze8XM/l+75Q9Z7O5xK81sbrvl1WZ2s5ktAmaY2UYz+39mttTMVpjZhHi7tr+4zey38ausPm9m683skvjyiJn9PP4ej5jZwree61DjU2b2XTN7GrjezD5kZovM7GUz+5uZDY5fzvga4EYzW2ZmM+N/RT8Y79/izj44zawAONHdl3fy3Egz+3v8Z/N3MxsRXz7WzF6Mr/Pmzva8LHYXsUfNbLmZvWpmH40vnxb/OSw3s5fMrCC+R/Bs/Ge4tLO9HTOLmtmP2m2rq9s9/RCQ8ncqk0MIexi3Hun/AKrj/54LzCN2lccIsZu2nB5/bkD831zgVWBgfN6BS9utayPwhfj054E74tNXAj+LT/8WuD/+HpOIXXse4BJil+KOAMcQu6fGJZ3U+xTw83bz/aFt9P9VwH/Fp78F/Gu7dvcA74lPjwBWd7LuM4EH2823r/th4Ir49KeBh+LTjwBz4tPXvPXz7LDei4ldOv2t+SJiN2BZD0yLLyskdmXkPCAnvmwcUBGfHkX8Zi7AXOAb8elsoAIYHZ8fBqwI+/dKj2Ae6XDpbEkd58YfL8fn+xH7UHoGuM7MLowvHx5fvhtoAR7ssJ63Ln2+hNgtFTvzkMfuLbHKzAbHl70HuD++fLuZPdlFrfe2my4D7jWzIcQ+aDcc4jVnA5PM2q5sXGhmBe5+sF2bIUDVIV4/o11/7gJ+2G75h+PT9wD/2clrVwD/aWY/AB5x92fNbDKwzd0XA7j7AYjtVQA/M7MpxH6+4ztZ37nAie32pIqIbZMNwE5g6CH6IClOoSBHkwHfc/fb37bQbBaxD9QZ7l5rZk8Ru4c2QL27t3RYT0P83xYO/Tvc0G7aOvybiJp207cCP3b3BfFav3WI10SI9aGui/XW8X99O5yEL0zm7mvN7BRiF0r7npk9TuwwT2fruBHYAZwUr7m+kzZGbI/ssU6eyyHWD0lDOqcgR9NjwKfNrB+AmQ0zs0HE/grdGw+ECcBpAb3/c8TutBWJ7z3MSvB1RcCW+PQV7ZYfBArazT8OXPvWTPwv8Y5WA8ce4n2eJ3YZZIgds38uPv0iscNDtHv+bcxsKFDr7r8nticxFXgNGGpm0+JtCuInzouI7UG0ApcTu+dvR48BnzOzzPhrx8f3MCC2Z9Hlt5QkdSkU5Khx98eJHf54wcxWAA8Q+1D9K5BhZq8AtxD7EAzCg8QuP/wqcDuwCNifwOu+BdxvZs8Cu9otfxi48K0TzcB1QHn8xOwqOrkLlru/BhTFTzh3dB3wqfjP4XLg+vjyG4AvmtlLxA4/dVbzZOAlM1sGfB34trs3Ah8FbjWz5cATxP7K/zlwhZm9SOwDvqaT9d1B7J6/S+NfU72d/9srOxN4tJPXSBrQpbOlTzGzfu5ebWYDgZeAd7v79qNcw43AQXe/I8H2eUCdu7uZXUbspPPsQIvsup5ngNnuvjesGiQ4Oqcgfc0jZlZM7ITxLUc7EOJ+AXykG+1PIXZi2IB9xL6ZFAozKyV2fkWBkKa0pyAiIm10TkFERNooFEREpI1CQURE2igURESkjUJBRETaKBRERKTN/wdrq+BraqzW8gAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f15f5cf59b0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "lrf = mdl.lr_find()\n",
    "mdl.sched.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training the model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Total size of chips in competition is 256X256, we start training our model with 64x64 and will gradually increase the size of image as training progress. This is a very good technique to avoid our fitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "08fbd8b8e0594688ad2d11deb6320daa",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.1415   0.13351  0.88145]                        \n",
      "[ 1.       0.1429   0.1284   0.88529]                        \n",
      "[ 2.       0.13353  0.12606  0.89027]                        \n",
      "[ 3.       0.13395  0.12541  0.89018]                        \n",
      "[ 4.       0.1304   0.12389  0.8899 ]                        \n",
      "[ 5.       0.12874  0.12111  0.89422]                        \n",
      "[ 6.       0.12952  0.12238  0.89179]                        \n",
      "\n"
     ]
    }
   ],
   "source": [
    "lr=0.2\n",
    "data = get_data(64, 64) #data generator for batch size=64, image size=64x64\n",
    "learn = ConvLearner.pretrained(f_model, data, metrics=metrics)\n",
    "learn.fit(lr, 3, cycle_len=1, cycle_mult=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When training the model, fastai implements a technique called **stochastic gradient descent with restarts (SGDR)**, which trains model in cycles, where each cycle consists of one or more epochs. For each cycle, it starts with LR original value and will exponentially decrease the LR([Exponential learning rate schedule](https://towardsdatascience.com/learning-rate-schedules-and-adaptive-learning-rate-methods-for-deep-learning-2c8f433990d1)) as the training progress. Second parameter in fit denotes the total number of cycles. Total number of epochs in a cycle are controlled by 2 parameter *cycle_len* and *cycle_mult* as follows.\n",
    "\n",
    "`number of epochs in first cycle = cycle_len \n",
    " number of epochs in second cycle = number of epochs in previous(first) cycle x cycle_mult\n",
    " number of epochs in third cycle =  number of epochs in previous(second) cycle x cycle_mult`\n",
    " \n",
    " \n",
    "Here is the graph the show changes in LR for each cycle in above training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZIAAAEKCAYAAAA4t9PUAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3Xl8VPXV+PHPmaxk3yYREiCBQELYJeAuKFawWsCKW61ia2trtbb10apPW7Uu/VV9WrU+ttXHre5rFVTUurBYFyTIvoc97PsqhMD5/TE3dowJmZDc3FnO+/W6r0zuNmduJnPmfldRVYwxxpij5fM6AGOMMZHNEokxxphWsURijDGmVSyRGGOMaRVLJMYYY1rFEokxxphWsURijDGmVSyRGGOMaRVLJMYYY1ol3usA2kNeXp4WFxd7HYYxxkSUGTNmbFFVf3P7xUQiKS4upqqqyuswjDEmoojIqlD2s6ItY4wxrWKJxBhjTKtYIjHGGNMqlkiMMca0iiUSY4wxreJqIhGRkSKyWESqReSmRrZfJyILRGSOiHwgIl2Dto0TkaXOMi5o/SARmeuc8y8iIm6+BmOMMUfmWiIRkTjgIeAsoAK4WEQqGuw2E6hU1X7AK8A9zrE5wK3AccAQ4FYRyXaO+RtwJdDDWUa69RqMMcY0z81+JEOAalVdDiAiLwCjgQX1O6jqpKD9PwO+7zweAbynqtucY98DRorIZCBDVT911j8FjAHedvF1RISDhw7zj09WEucTygrSObZrNskJcV6HFRVq6wLX9uDhw6QkxJGZkkDn7BQ656SQn56E3RSbWOdmIikE1gT9XkPgDqMpV/CfhNDYsYXOUtPI+m8QkSsJ3LnQpUuXlsT9DXWHDrN1by0FGcmtOo+bFqzbxZ1vLfzq9/SkeM6v7MxVw7rjT0/yMLLIN2/dTu6auLDRbZkdEhjQOYuBXbI4pYefAZ2ziPNZYjGxxc1E0th/kza6o8j3gUpgaDPHhnxOVX0EeASgsrKy0X1C9cxnq7jtjQW8etUJDOqa05pTueaQBl7in87vT3ZqAuNnreOpT1fy8ow1/O7sCi4Y3NnbACPY4cOBa/voZZUc2zWb7ftqWbNtH6u37WPBul3MWrODBz5Yyv3vLyU3NZEzehVwfmURg7pm292KiQluJpIaIPjTqwhY13AnETkD+A0wVFUPBB07rMGxk531Rc2ds62t37kfgGufn8Wk64eRGB++jd1y0xIZVpbP6eUFXDu8B795bS6/fnUOM1Zt544xfcI69nCXlOAjJzWRnNREuvvTvrZt576DTF6yiQ8WbuLNOet4sWoN3fypXHJcVy4e0pmUxJgYjcjEKDc/VaYDPUSkREQSgYuACcE7iMhA4GFglKpuCtr0LnCmiGQ7lexnAu+q6npgt4gc77TWugwY7+JrACCjQwIAa3d8yXPTQhp6Jix096fx7I+O55rTSnmxag0/e/YLDtQd8jqsqJSZksDoAYX85eKBfP6bM7hnbD+yUxK5480FnPTHD3ng/aXs/PKg12Ea4wrXEomq1gHXEEgKC4GXVHW+iNwuIqOc3e4F0oCXRWSWiExwjt0G3EEgGU0Hbq+veAeuAh4FqoFltGNF++DibP53UjX7D0bOh3GcT7h+RBl3jO7N+ws3cvWzMzl0uFUlfaYZqUnxXFDZmVevOpFXrzqRQV2zue/9JQy7dxJPfryCg4cOex2iMW3K1XIOVZ2oqj1Vtbuq3uWsu0VV6xPGGapaoKoDnGVU0LGPq2qpszwRtL5KVfs457xGVdvtU/Fnw0rZsqeWt+asb6+nbDOXnlDMbd+p4P2FG7nrrcYrjk3bG9Q1m0fHDebNn59Mr44Z3PbGAkbcN5V/L93idWjGtBkrMG+BE0tzKc1P46lPV3odylG5/KQSfnBSMY9/vIKXqtY0f4BpM30KM3n2R8fx2LhKFPj+Y9P49Suz2bnPirtM5LNE0gKC8L0hXZhds5PqTbu9Dueo/PbsCk7snsut4+dTvWmP1+HEFBFheK8C3v7FKVw1rDuvfrGWM+6bwuTFm5o/2JgwZomkhc7p3xGfwIRZrjcWc0WcT7jvwgF0SIzj58/PpLbOyuvbW3JCHDeOLGf81SeRk5LI5U9M5/+9vdDqTkzEskTSQvnpyZzQPZfxs9fRjtUzzWpJKAUZydx9Xj8Wrt/Fw1OWuRdUlHDrr9ynMJPx15zEJcd14eEpyxn7909Zu+NLl57NGPdYIglBw4RxTr9OrNq6j0Ubwq94K9QOcN+qKODsfh15cFI1yzdbEVcopNH+sK2TnBDHXef25a+XHMvyTXsY/b//Zsaqbc0faEwYsUTSAvWf0aeX5wPw4aLILtu+9TsVJMX7+M1r88Lq7ioWfbtvR167+kTSkuK5+JFpvGyNIUwEsURyFAoykundKYNJEZ5I8tOTuXFkOZ8u38q/Fmz0OpyYV5qfzutXn8SQkhxueGUO9767yBK8iQiWSI7S6eX5fLF6O9v31nodSqtcNLgzpflp/PHtRVbZGwayUhJ58geDuXhIZx6atIz/fm0udfZ3MWHOEslROq08n8MKH1VHdsey+Dgf//3tclZs2ctz01Z7HY4h8Df5w7l9uea0Up7/fA1XP/dFRI2mYGKPJZKj1K8wk/SkeD5bvtXrUFrttLJ8TuyeywMfLGXvgTqvwzEEGk1cP6KMW79TwbvzN3LFP6bzZa0lExOeLJEcpfg4H4NLcqIikdR/aG3bW8szn0XOoJSx4AcnlfCn8/vzybKt/PipKrszMWHJEkkrHN8th+Wb97Jp136vQ2m1Y7tkc0qPPB6Zuty++YaZ8wYV8T9j+/Pxsi386B+WTEz4sUTSCsd3ywXgsxXh0O6/9a17fjG8B1v31vJsBA2V3x7CoeHUeYOKuNdJJj9+qsqmAzBhxRJJCJr6IKnomBF29SSt6TJXWZzDid1z+fuU5fattxFeT3Y4dlARd5/Xj4+WbuFXL86y6QBM2LBE0gINP0fi43wc2zWbGSu3exKPG645rZQtew4wftZar0MxjbigsjO/PbsXE+du4HfjrSOpCQ+uJhIRGSkii0WkWkRuamT7qSLyhYjUicjYoPWnORNd1S/7RWSMs+1JEVkRtG2Am6+hOQO7ZLFk025274+O4cBP6J5Lr44ZPPbvFfYhFaZ+dEo3fjq0O89NW8197y3xOhxj3EskIhIHPAScBVQAF4tIRYPdVgOXA88Fr1TVSfWTXQGnA/uAfwXtckPQZFiz3HoNoRjYJRtVmFuz08sw2oyI8KOTS1iycQ8f2eRLYevGkWVcUFnEXz6s5ulPV3odjolxbt6RDAGqVXW5qtYCLwCjg3dQ1ZWqOgc4UtfdscDbqrrPvVCP3oCiLABmrtnhcSRt5zv9O+FPT+LRf6/wOhTTBBHhD+f2ZXh5PrdOmG9zmhhPuZlICoHgkedqnHUtdRHwfIN1d4nIHBG5T0SSjjbAtpCZkkA3fyozV0dPPUlivI9xJ3Rl6pLNLNkYfiMcm4D4OB9/uXggZcdk8PPnZtrfynjGzUTSWBuXFhW6i0hHoC/wbtDqm4FyYDCQA9zYxLFXikiViFRt3ry5JU/bYgM7ZzNz9Y6oqlP43nFdSYz38fSn1hQ4nKUmxfPYuEqSE+P44ZPT2bLngNchmRjkZiKpAToH/V4EtHRawQuA11T1q5psVV2vAQeAJwgUoX2Dqj6iqpWqWun3+1v4tC0zoEsWW/fWUrM9eiYlyklN5Oy+HXl95lr21dqwKeGsU1YHHr2ski17DnCl9X43HnAzkUwHeohIiYgkEiiimtDCc1xMg2It5y4FCczgNAaY1waxHlFz9xkDOwfqSWZ5WE/ixs3QxUO6sPtAHW/MjsxphdtKJNxp9u+cxZ8vGMAXq3fw+zfmex2OiTGuJRJVrQOuIVAstRB4SVXni8jtIjIKQEQGi0gNcD7wsIh89R8gIsUE7mimNDj1syIyF5gL5AF3uvUaQtWjII14nzB/3S6vQ2nTTnODi7MpzU+zUYEdHvdHbNa3+3bk6tO68/zna3jhc/ubmfYT7+bJVXUiMLHBuluCHk8nUOTV2LEraaRyXlVPb9soQ9fUNLZJ8XH0KEhn/rroaAJcT0T43pAu3P7mAuat3UmfwkyvQzLNuO5bZcyp2ckt4+dT3jGDAc7dsjFusp7tbaR3pwwWrNsVEcUgLXHesUUkxft4zr7hRoQ4n/CXiwbiT0/iZ8/MYKtVvpt2YImkjfTulMHWvbVs3BVd/7iZKQmc3a8jE2ats1GBI0R2aiIPXzqIrXtr+fnzM22GReM6SyRtpHenQLFPtBVvQWCwwD0H6vjXgg1eh2JC1KcwkzvH9OGTZVv5y4fVXodjopwlkjbSq2M6QFhUuLe140tyKczqwD+/sIEcI8n5lZ0579giHvxwKZ8ss+FujHsskbSR9OQEinNTovKOxOcTzh1YyEdLN0fFJF6x5PbRvSnJTeVXL86y+hLjGkskbah3p8yovCMBOPfYQg4rvG7Dy0eU1KR4HvzeQLbvO8j1L8+OusYgJjxYIglBqP97FZ0yqNn+JTu/bP8h5d3+eOjuT2NA5yxenbE25j6MIv3V9u6UyW/P7sWkxZt5zAbiNC6wRNICzXVIq68nWerh4HniYre5844tZPHG3SxYH513Xc0K9x6JR3Dp8V05s6KAu99ZFDVTHpjwYYmkDfUsCCSSxVE6Cut3+nciIU54zSrdI46IcM/YfuSmJvHLF2faeFymTVkiaUOFWR1ITYxjyYboTCRZKYkM7ennzTnrOWzzhUecrJRE7j2/H8s27+XudxZ5HY6JIpZI2pCI0KMgnSUb93gdimvO6deJDbv2MyOK5l+JJaf08HP5icU88fFKPq62JsGmbVgiaWNlBelRPcHQGRUFJMX7eDPGRwSOZDeOLKebP5XrX57tScMQE30skbSxnseks3VvbdROMJSWFM/p5fm8NXcDh6x4KyJ1SIzjvgsGsGn3AW4d7/osDCYGWCJpYz0L0gCi+q7knH6d2LLnANOWb/U6FHOU+nfO4uenl/L6rHW8OcfuLk3rWCJpY2VOy61orXAHOL08n5TEON6Ys97rUNpFtHabufq0UvoXZfK71+dF7R20aR+WSEKgLeiS5k9PIislgcXtXOHenh92HRLjGN6rgHfmreegjSwbsRLifNx7fn/2HjjErRNsVkVz9FxNJCIyUkQWi0i1iNzUyPZTReQLEakTkbENth0SkVnOMiFofYmITBORpSLyojONb7sIZfZBEaFnQbpnnRLbcobEI/lOv45s33eQT5bFTvGWm509vdKzIJ1rh5fy1pz1vDPPRnc2R8e1RCIiccBDwFlABXCxiFQ02G01cDnwXCOn+FJVBzjLqKD1dwP3qWoPYDtwRZsH30o9C9JYvHF3VA8lMrTMT3pSPG9Z+XrE+8nQ7lR0zOB34+exc5+14jIt5+YdyRCgWlWXq2ot8AIwOngHVV2pqnOAkMpHJDDX7enAK86qfwBj2i7ktlFWkM7u/XVsiOKRcpPi4zitPJ/3F26yiZMiXEKcj3vG9mPb3lrueGuB1+GYCORmIikE1gT9XkMjc7AfQbKIVInIZyJSnyxygR2qWneU52wX3fMDLbeWbdrrcSTuGtnnGLbtraVqlXVOjHR9CjO5amh3XplRw+TFm7wOx0QYNxNJYwXKLSnr6aKqlcD3gPtFpHtLzikiVzqJqGrz5s0teNrWK/U7iWRz9PZwBxja009ivM/K1qPEz4eXUpqfxn//cy6791sRlwmdm4mkBugc9HsREHKBuqquc34uByYDA4EtQJaIxDd3TlV9RFUrVbXS7/e3PPpW8KcnkZ4UH/WJJDUpnlN75PHego1RXR8UK5Li47hnbD/W79rPPe8s9jocE0HcTCTTgR5OK6tE4CJgQjPHACAi2SKS5DzOA04CFmjg02oSUN/Caxwwvs0jbyURoVt+WtQnEoAzex/D2h1fMm9tjA4tH2WO7ZLNuBOKeWbaKmat2eF1OCZCuJZInHqMa4B3gYXAS6o6X0RuF5FRACIyWERqgPOBh0WkvjF7L6BKRGYTSBx/VNX6WsAbgetEpJpAncljbr2G1ujuT436OhKAM3oVEOcT3p0fvcVbLelHFA3+68yeFKQnc/M/51pDChMSV/uRqOpEVe2pqt1V9S5n3S2qOsF5PF1Vi1Q1VVVzVbW3s/4TVe2rqv2dn48FnXO5qg5R1VJVPV9VXe+SezSlNt39aWzYtZ89B+qa37kNeFW0lJOayJDiHN6J4kQSa9KTE7htVAUL1+/iiY9Xeh2OiQDWs70FpAW9/br7UwFYsbl970q86DI3oncB1Zv2RH1RXnt19gwHI3ofwxm98vnze0uo2b7P63BMmLNE4pLuMdJyCwL1JEBUF2/FGhHhtlG9Abh1/HxrTGGOyBKJS7rkphDnk5hIJJ2yOtC/KJN3rRlwVCnKTuG6b/Xkg0Wb7EuCOSJLJC5Jio+jS05KTCQSCNyVzK7ZyaYo7s0fi35wUjG9OmZw24QF1rfENMkSiYtipeUWwPBe+QB8uMh6RUeT+Dgffzi3Dxt37+dP/1ridTgmTFkicVF3fxortu6NiZkEywrSKczqwPsLLZFEm4Fdsvn+cV156tOVzF+30+twTBiyROKibv5UausOs3b7l16H4joRYXivfD6u3sL+g4e8Dse0sevPLCM7JdEq3k2jLJG4KJZabgEM71XAlwcP8Wm0zVFin5tkpiRw48hyqlZt559frPU6HBNmLJGE4Gg/R9ozkYTDZ91xJTmkJMbxwaKNXodiXDB2UBEDOmfx/95exC6reDdBLJG4KDs1keyUBJa1Z6dEDzvNJSfEcUqPPD5cuCkqiz9iqD9io3w+4fbRvdm69wD3v7fU63BMGLFE4rLivFRWbomNllsAw8sLWLdzPwvW2yCO0ahfURYXDe7CPz5dyeIN3kwnbcKPJRKXleSmsnJr7CSS08rzEYEPrfVW1Pr1iDLSk+O5Zfy8qLzzNC1nicRlxXmprN+5ny9rY6Mlkz89if5FWbxv/UmiVnZqIjeMKGPaim1MmB3yFEMmioWcSEQk1c1AolVxXuCyrdoWO3clw8vzmb1mB5t3uz4ws/HIRYO70Kcwgz9MXNhuI1yb8NVsIhGRE0VkAYE5RRCR/iLyV9cjixIluYFEElP1JL0KAJhkdyVRK84n3D66Dxt3HeDBD6ziPdaFckdyHzAC2AqgqrOBU90MKpoU56UAsGJL7AzF3atjOp0yk3l/oTUDjmbHdsnm/EFFPPbvFTHTV8o0LqSiLVVd02BVSAX+IjJSRBaLSLWI3NTI9lNF5AsRqRORsUHrB4jIpyIyX0TmiMiFQdueFJEVIjLLWQaEEkurtKJCMT05gby0xJi6IxERTu+Vz7+rt3CgLvLrhqw6uWm/HllOckIcd7210OtQjIdCSSRrROREQEUkUUSuxynmOhIRiQMeAs4CKoCLRaSiwW6rgcuB5xqs3wdc5syYOBK4X0SygrbfoKoDnGVWCK/BU8W5qaxwueVWuDWeGdYzn321h6haud3rUIyL/OlJ/Pz0Uj5ctIkpSzZ7HY7xSCiJ5KfA1UAhUAMMAH4WwnFDgGpnatxa4AVgdPAOqrpSVecAhxusX6KqS53H64BNgD+E53RNa2bHa8++JBIm3eZOLM0lMc7H5MXRU0/SkhkyY8nlJxXTNTeFO95cwEGb4z0mhZJIylT1ElUtUNV8Vf0+0CuE4wqB4CKxGmddi4jIECARWBa0+i6nyOs+EUlq4rgrRaRKRKo2b/b2m1JJXiqbdh9gbwy1bklJjOe4bjlMXmzfUqNdUnwcv/l2L6o37eGZz1Z5HY7xQCiJ5MEQ1zXU2Ne3FhXAiEhH4GngB6pa/1XnZqAcGAzkADc2dqyqPqKqlapa6fd7ejNDcX3LrRjqmAgwtKefpZv22JzfMeBbFQWcXJrH/e8vZfveWq/DMe2syUQiIieIyH8BfhG5Lmi5DYgL4dw1QOeg34uAkHsviUgG8BbwW1X9rH69qq7XgAPAEwSK0MJafcutlTHUcgtgWFlgsiu7K4l+IsLvzqlg9/6D3Pe+TYAVa450R5IIpAHxQHrQsgsYe4Tj6k0HeohIiYgkAhcBE0IJytn/NeApVX25wbaOzk8BxgDzQjmnl2L1jqS7P5Wi7A6WSGJE2THpXHJcV575bJWNwxVj4pvaoKpTgCki8qSqtrjgU1XrROQa4F0CdzCPq+p8EbkdqFLVCSIymEDCyAa+IyK/d1pqXUCgr0quiFzunPJyp4XWsyLiJ1B0NotAY4CwlpoUT356EitiqAkwBL6lDivz888v1nKg7hBJ8aHcyJpIdt23ejJh9jrueHMBT18xxBooxIgmE0mQfSJyL9AbSK5fqaqnN3egqk4EJjZYd0vQ4+kEirwaHvcM8EwT52z2ecNRrI0CXG9Yz3ye+Ww1VSu3c1JpntfhGJdlpybyyzN68Ps3FvD+wk18q6LA65BMOwilsv1ZYBFQAvweWEmg2CpmtEUXjVgbBbheNDQDDrc+OuHu+8d3pTQ/jbveWhAVHVJN80JJJLmq+hhwUFWnqOoPgeNdjivqFOelsmVPLbtdmllOw7T/tTUDjj0JcT5+e3YvVm7dx5Mfr/Q6HNMOQkkk9Z9860XkbBEZSCPFUdGutSW9Je3Ucisci6SjpRlwOF7bcDWsLJ/Ty/N58MNqGwU6BoSSSO4UkUzgv4DrgUeBX7kaVRSqH07e7aFSwpE1A45Nvz27F/sPHuLP7y32OhTjsiMmEme8rB6qulNV56nqaao6SFVDasZr/qNrTuwNJ1/PmgHHpm7+NC47oZgXp69h0QabejmaHTGRqOohYFQ7xRLVOiTG0SkzOSYTSX0z4E+WRcdowCZ01w4vJT05gbveWmjT8kaxUIq2PhGR/xWRU0Tk2PrF9ciiUHGe+6MAhysbDTg2ZaUkcu3wHny0dAuTbXTgqBVKIjmRQB+S24E/Ocv/uBlUtIrVviQQHc2AzdG59PiuFOemcNdbC6mz0YGjUrOJxKkXabhEZKdArxXnprB930F27nOnCXA4s2bAsSsx3sfNzujAz09vOEeeiQYhzZAY69qqaDdWx9yqF6nNgMO1j04kObOigONKcrjvvSXscqkvlfGOJZIQtcWYQSV5LiaSCPisq28GbDPpxZ760YG376vloUnVXodj2pglknbUOScFEVwdvDGc+8x196dSmBW5zYDD+dpGgj6FmXx3YBFP/Hsla7ZF1l2pObJmE4mIfLeRZbiI5LdHgNEkOSGOTpkdWLU1Nv+JvmoGXL2F2jqrdI1FN4wow+eDP76zyOtQTBsK5Y7kCgK92S9xlv8DrgM+FpFLXYwtKhXnpcTccPLBhpXls7f2EFWrtnkdivHAMZnJ/OTU7rw1Zz0zVllT8GgRSiI5DPRS1fNU9TygAjgAHEcT09yapnWN0VGA653YPZeEOGFKhBZvmdb7ydBu5KcnccebC6yTYpQIJZEUq+rGoN83AT1VdRv/GdCxUSIyUkQWi0i1iNzUyPZTReQLEakTkbENto0TkaXOMi5o/SARmeuc8y8SYTPnlOSmsmPfQXbsi815rVOT4hlcbM2AY1lKYjzXjyhj1podvDFnvdfhmDYQSiL5SETedD7YxwHjgakikgrsaOogZ5yuh4CzCNzFXCwiFQ12Ww1cDjzX4Ngc4FYCdz1DgFtFJNvZ/DfgSqCHs4wM4TWEjeKvWm7FZj0JwLAyP4s37mbdji+9DsV45Lxji6jomMHdby9i/0EbNifShZJIrgaeBAYAA4GngKtVda+qnnaE44YA1aq6XFVrgReA0cE7qOpKVZ1DoPgs2AjgPVXdpqrbgfeAkc587Rmq+qkG7omfIjBve8Qozq0fTj52i7fqmwFPtWbAMSvOJ/z27F6s3fElj3+8wutwTCuF0rNdVfUVVf2Vqv7SeRxKwWYhENyNtcZZF4qmji10Hh/NOY9aW3ZIq28CHMv1JD3y0+iUmRwxxVtWjO+OE0vzOKNXAX+dtMzmLIlwoTb/XSoiO0Vkl4jsFpFQxoRurO4i1H/Jpo4N+ZwicqWIVIlI1ebNrf/AaquKmPomwG19RxJJn3UiwtAyPx9Xb+Ggjb0U027+djn7Dx7ivveXeB2KaYVQirbuAUapaqaqZqhquqpmhHBcDdA56PciYF2IcTV1bA1fn52xyXOq6iOqWqmqlX6/P8SnbR/FeSmscKmOJFLaHgztmc/uA3UR1QQ0Qi5tROnuT+P7x3flhc9Xs3jDbq/DMUcplESyUVUXHsW5pwM9RKRERBKBi4BQJ8R6FzhTRLKdSvYzgXdVdT2wW0SOd1prXUag8j+iFOemsiqGi7YATirNJd4nNlyK4RfDe5CWFM9dE4/mY8aEg1ASSZWIvCgiFwf3bm/uIFWtA64hkBQWAi+p6nwRuV1ERgGIyGARqQHOBx4WkfnOsduAOwgko+nA7c46gKsIdJCsBpYBb7fkBYeDkrzYbgIMkJ6cwKCu2RFTT2Lck50amLNk6pLNNs1AhIoPYZ8MYB+Bu4J6CvyzuQNVdSIwscG6W4IeT+frRVXB+z0OPN7I+iqgTwhxh62uzijAK7bsZWCXRI+j8c6wsnzufmcRG3ftpyAj2etwjIcuO6GYpz9bxR8mLuTk0jzi42wYwEgSSqutHzSy/LA9gotWJXmBJsCxOuZWvWFlgbor6+VuEuN93HxWOUs27uHFKpuzJNI0eUciIr9W1XtE5EEaaRSkqte6GlkUa49RgCNB+THpFGQkMWXJZi4Y3Ln5A0xUG9H7GIYU5/Dnfy1hVP9OpCcneB2SCdGR7kjqa76qgBmNLOYoJcU7TYBjvMJdRBja089HSzfbFKwGEeG35/Ri695a/jp5mdfhmBZo8o5EVd9wfv6j/cIJT250SCvJS43pYVLqDSvL56WqGmau2cHg4hyvw2lUJPXRiXT9irL47sBCHvv3Cr43pAudc1K8DsmEIJQOiT1F5BER+ZeIfFi/tEdw4aSt+xAU56W0aafESO19fVJpHnE+sdY65is3jCzDJzZnSSQJpWnEy8BM4LfADUGLaYXi3FR2ftn2TYAjrdNcZocEBnXJjpD+JBF2cSNBuIHfAAAgAElEQVRUx8wO/HRoYM6S6Stt3ppIEEoiqVPVv6nq56o6o35xPbIoVxzUBDjWDS3zM2/tLjbt3u91KCZM/OTU7nTMTOb2NxZw+HCE3m7HkFASyRsi8jMR6SgiOfWL65FFuf8MJ2+JZGjPQDPgqUu2eByJCRcdEuO4cWQ5c9fu5J8z13odjmlGKIlkHIGirE/4T4utKjeDigWdczrgE1i5xSrce3fKwJ+eFCHFW6a9jOrfif6ds7jnnUXsPVDndTjmCI6YSETEB3xfVUsaLN3aKb6olRQfR6csawIMX28GfMiKMYzD5xNuOaeCTbsP8Pcp1hw4nB0xkajqYeB/2imWmFOSlxrTE1wFG9rTz459B5m1pslJN00MGtQ1m1H9O/HI1OWstRk1w1YoRVv/EpHzIm1u9Lbk1nfkrrkp1pfEcUqPPHwCU6wZsGngxrPKAbj7bWsOHK5CSSTXEWgCfKCFE1uZZtQ3Ad6+N3ZHAa6XlZLIwDBtBhzahKDGLYVZHfjJqd2YMHsdM1ZZc+BwFMqgjemq6lPVxBZObBVVxIU+BCVOy60VbVBP0pbTAXtlaE8/c9buZOsem3bVfN1PhnanICOJ299caM2Bw1BIYzU7E0wNEZFT6xe3A4sF9cPJt+UkV5Fc/jiszI8qTF0afnclEHmdPaNJalI8vx5Rzuw1Oxg/25oDh5tQhkj5ETCVwARVv3d+3uZuWLGhS04KPoEV1gQYgD6dMslLS7TJrkyjzh1YSL+iTO5+ezH7aq05cDgJ5Y7kF8BgYJWqngYMBEL6TxeRkSKyWESqReSmRrYnObMvVovINBEpdtZfIiKzgpbDIjLA2TbZOWf9tvwQX2vYSYz3UZjdwVpuOXw+4dQefqYusWbA5pvqmwNv2LWfh6cs9zocEySURLJfVfdD4INfVRcBZc0dJCJxwEPAWUAFcLGIVDTY7Qpgu6qWAvcBdwOo6rOqOkBVBwCXAitVdVbQcZfUb1fViG7mY/O3f93QMj/b9x1k7tqdXodiwlBlcQ7n9OvIw1OXsc6aA4eNUBJJjYhkAa8D74nIeGBdCMcNAapVdbmq1gIvAKMb7DMaqB+m/hVgeCPNjC8Gng/h+SJScW4qK7bstZZBjlN6+BHBRgM2TbrprHIOK9xjowOHjVBabZ2rqjtU9Tbgd8BjwJgQzl0IBM+ZWeOsa3QfVa0DdgK5Dfa5kG8mkiecYq3fRXr/luK8VHbtr2P7voNehxIWclIT6V+UZfUkpklF2Sn8+JQSXp+1jpmrt3sdjiH0Vlsni8gPVHUK8CnfTAiNHtbIuoZfu4+4j4gcB+xT1XlB2y9R1b7AKc5yaRMxXykiVSJStXlz6z6U3LxZKM4NTNxjQ6X8x7AyP7Nrdlj/GtOkq4aV4k9P4vc2OnBYCKXV1q3AjcDNzqoE4JkQzl0DBE/EXcQ3i8S+2kdE4oFMILjH0UU0uBtR1bXOz93AcwSK0L5BVR9R1UpVrfT7/SGE642vRgG2CvevDO0ZXs2A7WMq/KQlxfPrEWXMWrPDRgcOA6HckZwLjAL2AqjqOiA9hOOmAz1EpEREEgkkhQkN9plAYHRhgLHAh+pUFjgDRp5PoG4FZ128iOQ5jxOAc4B5tAeXCtA6Z6c4owC3LpFEUxVLv6IsslMSmGLFW+YIzju2iIFdsvjj2wvZtd+Khr0USiKpdT7c6z/gU0M5sVPncQ2BficLgZdUdb6I3C4io5zdHgNyRaSawFAswU2ETwVqVDW4nV8S8K6IzAFmAWuB/wslnnD1VRPgNhpzK7JrjALifMKpPf1MWbI5rIotouDSRhWfT7h9VB+27q3lgfeXeh1OTIsPYZ+XRORhIEtEfgz8kBA/vFV1IjCxwbpbgh7vJ3DX0dixk4HjG6zbCwwK5bkjSXFuqtWRNDCszM/4WeuYv24XfYsyvQ7HhKm+RZlcNLgLT36ykgsHd6ZnQSiFJaathdJq638INM19lUD/kVtU9UG3A4slJXnWBLihU3oE6rWsGbBpzg0jykhLiue2CfPtf8gjIbXaUtX3VPUGVb1eVd9zO6hY0zU3ld3WBPhr8tKS6FeUySRLJKYZOamJXH9mTz5ZtpW3523wOpyY1GQiqR8uvpHFhpFvYyV5gSbAK6zl1tecXp7PzDU7bDRg06zvHdeVXh0zuPPNBTYOlweaTCT1w8U3ssTkMPJuKs61JsCNOaNXAarw4SK7KzFHFucTbh/dm3U79/O3yTYtb3sLqWgr1rk910eR0wTYxtz6ut6dMuiYmcz7Czd6HYqJAIOLcxgzoBMPT1lu/0vtzBJJGEiM91GUncIKm3b3a0SEM3oVMHXJFvYfPORdIFZ/GzFu/nYvEuKEO95c4HUoMcUSSYjc7kNQnJfaqqKtaP2sG94rny8PHuLTZVu9DsVEgIKMZK4d3oP3F25ikhWJthtLJGGiODeFlVvboglwdHWbO6F7LqmJcbwXBsVbET4+aMz4wUkldPOncuuE+d7eycYQSyRhothpArzNBir8mqT4OE7t6eeDhRutj4AJSWK8jzvH9GH1tn08NKna63BigiWSMFHiDN5oTYC/6YxeBWzcdYB5a63VuQnNid3z+O7AQv4+ZRnVm/Z4HU7Us0QSJrr70wBYttne9A2dVp6PTwiL4i0TOf777F6kJMbzm9fm2t2syyyRhInC7A4kxfvs21MjclITqeyaw/sLLJGY0OWlJXHTWeVMW7GNf35hQ827yRJJmIjzCd38aZZImnBGRT4L1u9irc3TbVrgwsrOHNsli7smLmTHPqt/dIslklC0011xaX4a1Va01agzehUA8IEVb5kW8PmEu87ty84vD/LHt22Od7dYIglRe7T8LPWnUbP9S2uy2Ihu/jS6+VN5z4PiLbdHNjDu6tUxgx+dXMIL09dQtXJb8weYFrNEEka656eienQV7rFQmfitigI+XbaVnTZKsmmhX5zRg8KsDvzmtXkcPHTY63CijquJRERGishiEakWkZsa2Z4kIi8626eJSLGzvlhEvhSRWc7y96BjBonIXOeYv0gU9RIrzQ+03GpNPUn0XI1vOqtPR+oOq2djb0XxpY16KYnx/H5UbxZv3M0jU5c3f4BpEdcSiYjEAQ8BZwEVwMUiUtFgtyuA7apaCtwH3B20bZmqDnCWnwat/xtwJdDDWUa69RraW0leKj6BZVbh3qj+RZl0yky2OSfMUTmjooCz+3bkgQ+WWqOWNubmHckQoFpVl6tqLfACMLrBPqOBfziPXwGGH+kOQ0Q6Ahmq+qkzj/xTwJi2D90bSfFxdMlJsQr3JogII/t0ZOrSzew5YHNOmJa7bVRvUhLjuPHVORw6HP3Fwe3FzURSCKwJ+r3GWdfoPqpaB+wEcp1tJSIyU0SmiMgpQfvXNHNOAETkShGpEpGqzZs3t+6VtKPSfGsCfCRn9T2G2rrDNkeJOSr+9CRuOaeCGau289SnK70OJ2q4mUgau7No+BWgqX3WA11UdSBwHfCciGSEeM7AStVHVLVSVSv9fn8LwvZW9/w0VmzZS51VCDZqUJds/OlJvD13vdehmAh17sBChpX5ueedxazZZlM3tAU3E0kN0Dno9yJgXVP7iEg8kAlsU9UDqroVQFVnAMuAns7+Rc2cM6KV+tM4eEhZbW/wRvl8wsjexzB58Wa+rLVm0qblRIQ/nNuXOJ9w8z9t+JS24GYimQ70EJESEUkELgImNNhnAjDOeTwW+FBVVUT8TmU9ItKNQKX6clVdD+wWkeOdupTLgPEuvgagfef6aIuWW9HurD7H8OXBQ0xZYsVb5uh0yurATWeV8+/qLbxcVdP8AeaIXEskTp3HNcC7wELgJVWdLyK3i8goZ7fHgFwRqSZQhFXfRPhUYI6IzCZQCf9TVa3vSXQV8ChQTeBO5W23XkMwaafGn93z6wdvbNkowLH0nWpISQ7ZKQlMnNs+rbfsC2t0+t6QLgwpyeGOtxawcdd+r8OJaPFunlxVJwITG6y7JejxfuD8Ro57FXi1iXNWAX3aNtLwkZGcQH560lHfkcRCX4f4OB9nVhzDW3PXc6DuEEnxce3yvNHcRycW+XzCPef1Y+QDU7nx1Tk8cflgm7zsKFnP9jDUsyCdJRt3ex1GWDur7zHsOVDH1CVbvA7FRLDivFRuPqsXkxdv5vnP1zR/gGmUJZIwVH5MIJFYO/emnVSaR3ZKAuNn2fDgpnUuPb4rJ5fmcedbC1i11SaWOxqWSMJQ2THpHKg7zEp7UzcpIc7H2f068v7CjdY50bSKzyfce34/4n3CdS/Nti9wR8ESSRjq1TEDgMUbrHjrSMYMKGT/wcO8t8CGTDGt0zGzA3eM6cOMVdt5eOoyr8OJOJZIwlBpfho+gUXrbY7yIzm2SzaFWR0YPyuquhIZj4zq34mz+3bkvveWMH/dTq/DiSiWSELQ3h2WkhPiKMlLZZHdkRyRzyeMGtCJj5ZuYcueA16HYyKciHDnmD5kpSRy3YuzbV6gFrBEEqbKO2ZYIgnB6AGdOHRYmWhDppg2kJ2ayL1j+7F4427+MHGh1+FEDEskIWrv5uXlBems3rYv9IrkGK0fLD8mg7KCdFeLt6xDYmwZVpbPj08p4alPV/HOPPuCEgpLJGGq3Klwb2l/kljsUDV6YCdmrNru+gB87TW6gfHeDSPK6VeUya9fmUPNdhv3rjmWSMJU+THpACxab8VbzRnVvxMi8OoXNmaSaRuJ8T4evHgghxWufX6mTc/bDEskYaowqwNpSfEs3mAtt5pTlJ3CSd3zeLmqhsPWB8C0ka65qfzhu335YvUO7n9/idfhhDVLJGHK5xPKjklnod2RhOSCwZ1Zu+NLPlm21etQTBQZ1b8TF1Z25q+TlzFlSeRMkNfeLJGEsT6dMpi/bqd9yw7BmRUFZCTH81KVjZdk2tatoyromZ/OL16YaRNhNcESSRjrU5jJ3tpDLN9iQ6U0JzkhjjEDC3ln/gZ27jvodTgmiqQkxvP3Swdx6LDy02dmWP+SRlgiCYFXzT/7FWUBMHftDm8CiDAXVHamtu4w42fbQI6mbZXkpXL/hQOYv24Xv3ltns2q2IAlkjDW3Z9Kh4Q45tTYcA2h6FOYSUXHDF6cbsVbpu0N71XAtcN78OoXNTwzbbXX4YQVVxOJiIwUkcUiUi0iNzWyPUlEXnS2TxORYmf9t0RkhojMdX6eHnTMZOecs5wl383X8NXztseTNBAf56OiUwbz1jafSDRWeyQ2cOHgzsxft4tZa9ruLs6urKn3y+E9GFbm5/Y35lO1clvzB8QI1xKJM+f6Q8BZQAVwsYhUNNjtCmC7qpYC9wF3O+u3AN9R1b4E5nR/usFxl6jqAGeJ6om7+xZmMm/trpCHto71LnPfPbaQ1MQ4nvpkZZufOwb7epoGfD7hgQsHUpjVgZ88PcMq3x1u3pEMAapVdbmq1gIvAKMb7DMa+Ifz+BVguIiIqs5U1foxL+YDySKS5GKsYatfUSZfHjzEss1HN/VurElPTuC8QUW8OWe9DeRoXJGZksBjlw/m4KHD/PDJ6ezab4073EwkhUBwYXWNs67RfVS1DtgJ5DbY5zxgpqoGfyo84RRr/U6aGBNERK4UkSoRqdq8OXLbf/crygRgrtWThOyyE4qpPXSYFz63cmzjju7+NP5+6SBWbNnL1c9+EfM9391MJI19wDcsnzniPiLSm0Bx10+Ctl/iFHmd4iyXNvbkqvqIqlaqaqXf729R4OGkJC+N1MQ4ZtdYy61QleancXJpHs98tjrm/8GNe07snscfzu3LR0u3cNuE+THdksvNRFIDdA76vQhoOETrV/uISDyQCWxzfi8CXgMuU9WvpixT1bXOz93AcwSK0KJWnE8Y0CWLqpXbvQ4loow7sZgNu/bzr/kbvQ7FRLELBnfmJ0O78ey01fx1cuzOrOhmIpkO9BCREhFJBC4CJjTYZwKBynSAscCHqqoikgW8Bdysqh/X7ywi8SKS5zxOAM4B5rn4GsJCZdccFm3YxW4riw3Z6eX5dMlJ4ZGPlsf0N0XjvhtHlDN6QCfufXcxz8docapricSp87gGeBdYCLykqvNF5HYRGeXs9hiQKyLVwHVAfRPha4BS4HcNmvkmAe+KyBxgFrAW+D+3XsNXr8XtJ2jG4OIcDivMXG3FW6GK8wlXntqN2Wt28KmNv2Vc5PMJ947tz7AyP795bW5MTrIW7+bJVXUiMLHBuluCHu8Hzm/kuDuBO5s47aC2jDESDOiSRZxPqFq5jVN7Rm59T3sbO6iIBz5Yyl8nL+PE0jyvwzFRLDHex98uGcT3H5vGL1+YRUZyAif3iJ33nPVsD5GXE0alJcVT0TGD6UeoJ7HSm29KTojjipNL+Hf1Fua0orGCFY2ZUHRIjOPxcYPp5k/lx09VxdSdsCWSCFFZnM3MNdubbYVknea+7pLjupCRHM9Dk6q9DsXEgMyUBJ6+4jiKsjvwgyc/55PqLV6H1C4skUSIwcU57D942MbdaqH05AQuP6mEd+dvtL44pl3405N4/srj6ZKTwg//MZ2PYyCZWCKJECd0y0UEPloauZ0rvfLjU0rITkngnncXeR2KiRF5aUk8/+PjKc5N5YdPTmfSoqgeyckSSaTITk2kX1EWU22WthZLT07g6tNK+Wjplpj4dmjCQ25aEs/+6Dh6FKTxo6eqeDmKJ12zRBJBTu2Rx6w1O2zipqPw/eO70ikzmbvfWWQzTpp2k5uWxAtXnsAJ3XK54ZU5PDSpOiobb1giiSCn9vRzWOHjZfatuqWSE+K4fkQZc2p28sqMGq/DMTEkLSmexy8fzKj+gU6Lv319XtQN3WOJJATh8gViQOcs0pPimbLYireOxrkDCxlcnM3/e3sh2/fWeh2OiSGJ8T7uv3DAV8OpXPLotKgandoSSQRJiPMxrDyf9xZupC7KvtG0BxHhjjF92LW/zireTbvz+YSbz+rF/RcOYPaaHYx68N9R05LQEkmIwqV7xtl9O7Jtby2fLf/67GzhctcU7sqPyeCHJxXz/OdrQm4BZ5fWtKUxAwt59aoTATjvb5/w6EfLI77ezhJJhBlW5ic1MY635jYcSDlAwiblha/rvlVGaX4a1788u0VFXNbZ07SVPoWZvHntKZza08+dby1k3BOfs2nXfq/DOmqWSCJMckIcw3sV8Pa8DRyoO+R1OBGpQ2Ic9184gG17a7npn3OishWNCX85qYn832WDuHNMHz5fsY0z75/Ky1VrIvL9aIkkAp1fWcSOfQd5e+4Gr0OJWH0KM/n1iHLenb+RBz+04VOMN0SE7x/flbeuPZnu/jRueGUOFz3yGdWbImtqbUskEeik7nl0y0vl6c9WeR1KRPvRKSV8d2Ahf35vCW/Mbryo0Jj2UJqfzss/OYE/nNuXhet3MfL+qfz29bls2h0ZxV2WSCKQzxf4FjNj1XY+X7Gt+QNMo0SEP3y3L4OLs/nVi7N4OwbnkTDhw+cTvndcFz74r2FcNKQzL3y+hqH3TOaPby9iY5jXn7iaSERkpIgsFpFqEbmpke1JIvKis32aiBQHbbvZWb9YREaEes5YcfGQLuSnJ3H3O4siskw1XCQnxPH45YPp3zmLa56fydOfrbLraTzlT0/izjF9ef+6oZxRUcDDU5dx8t0fct2Ls5i5entYvj9dSyQiEgc8BJwFVAAXi0hFg92uALarailwH3C3c2wFgal5ewMjgb+KSFyI52wzt46fx7cf+IjXZ6116ymOWofEOH71rZ7MWLWdZ6fF5vSebSU9OYEnfzCYU3rk8bvX53HdS7PZZh0WjceK81J58OKBTL5+GJcc15V35m/g3L9+wul/msL97y9h/rqdYdNs2M0ZEocA1aq6HEBEXgBGAwuC9hkN3OY8fgX4XwnMIDUaeEFVDwArnKl4hzj7NXfONpOTmkSnrA50yupAv6JMN56iVS6s7Mzb8zZw+5sLGDOgk9fhRLT05AQeGzc4MKPipGo+XLSJS4/vypiBhdZHx3iqa24qt43qzXVn9uSduRt4beZaHvhgKfe/v5S8tERO6J5H38IMenXMoLs/DX96Eglx7Vtr4WYiKQSCh7usAY5rah9VrRORnUCus/6zBscWOo+bO2eb+cUZPdw6dZvw+YQHLhzAZY9/zktVNn5Ua8X5hOu+1ZNz+nXknncW89Dkav53UjVJ8VaVaLyXkZzABYM7c8HgzmzatZ+Plm7ho6WbmbZi2zcai+SmJpKaFE9SvI9Hx1XSNTfV1djcTCSNdd9q+N2uqX2aWt/Yf3Sj3xdF5ErgSoAuXbo0HWWEy05N5OWfnsALn69m+Za99DwmzeuQIl7PgnQeHVdJzfZ9fLR0C/PX7UQVSvPt2prwkJ+RzHmDijhvUBEA2/fWsnD9LlZt28fGXfvZtPsAX9Ye4kDdITokxLkej5uJpAboHPR7EdCwjWX9PjUiEg9kAtuaOba5cwKgqo8AjwBUVlZGdeFEckIcl59U4nUYUacoO4WLh0TvlxATPbJTEzmxNI8TPXp+N+/ZpwM9RKRERBIJVJ5PaLDPBGCc83gs8KEGmiRMAC5yWnWVAD2Az0M8pzHGmHbk2h2JU+dxDfAuEAc8rqrzReR2oEpVJwCPAU87lenbCCQGnP1eIlCJXgdcraqHABo7p1uvwRhjTPMkHNskt7XKykqtqqryOgxjjIkoIjJDVSub28+aoxhjjGkVSyTGGGNaxRKJMcaYVrFEYowxplUskRhjjGmVmGi1JSKbgaOdvCMP2NKG4bjJYnWHxeoOi9UdbRlrV1X1N7dTTCSS1hCRqlCav4UDi9UdFqs7LFZ3eBGrFW0ZY4xpFUskxhhjWsUSSfMe8TqAFrBY3WGxusNidUe7x2p1JMYYY1rF7kiMMca0iiWSJojISBFZLCLVInKT1/EAiMhKEZkrIrNEpMpZlyMi74nIUudntrNeROQvTvxzRORYl2N7XEQ2ici8oHUtjk1Exjn7LxWRcY09l0ux3iYia51rO0tEvh207WYn1sUiMiJovevvERHpLCKTRGShiMwXkV8468Pu2h4h1rC7tiKSLCKfi8hsJ9bfO+tLRGSac41edKarwJnS4kUnnmkiUtzca2iHWJ8UkRVB13WAs7793wOqakuDhcAQ9cuAbkAiMBuoCIO4VgJ5DdbdA9zkPL4JuNt5/G3gbQKzTR4PTHM5tlOBY4F5RxsbkAMsd35mO4+z2ynW24DrG9m3wvn7JwElzvsirr3eI0BH4FjncTqwxIkp7K7tEWINu2vrXJ8053ECMM25Xi8BFznr/w5c5Tz+GfB35/FFwItHeg3tFOuTwNhG9m/394DdkTRuCFCtqstVtRZ4ARjtcUxNGQ38w3n8D2BM0PqnNOAzIEtEOroVhKpOJTCnTGtiGwG8p6rbVHU78B4wsp1ibcpo4AVVPaCqK4BqAu+PdnmPqOp6Vf3CebwbWAgUEobX9gixNsWza+tcnz3OrwnOosDpwCvO+obXtf56vwIMFxE5wmtoj1ib0u7vAUskjSsE1gT9XsOR/yHaiwL/EpEZEpiTHqBAVddD4B8ZyHfWh8NraGlsXsd8jVMU8Hh9UdERYmr3WJ3ilIEEvpGG9bVtECuE4bUVkTgRmQVsIvChugzYoap1jTzvVzE523cCuV7Fqqr11/Uu57reJyJJDWNtEJNrsVoiaZw0si4cmredpKrHAmcBV4vIqUfYN1xfAzQdm5cx/w3oDgwA1gN/ctaHRawikga8CvxSVXcdaddG1rVrvI3EGpbXVlUPqeoAoIjAXUSvIzxvWMUqIn2Am4FyYDCB4qobvYrVEknjaoDOQb8XAes8iuUrqrrO+bkJeI3Am39jfZGV83OTs3s4vIaWxuZZzKq60flnPQz8H/8pnvA8VhFJIPDB/Kyq/tNZHZbXtrFYw/naOvHtACYTqE/IEpH6KciDn/ermJztmQSKR72KdaRTlKiqegB4Ag+vqyWSxk0HejgtOBIJVK5N8DIgEUkVkfT6x8CZwDwnrvrWF+OA8c7jCcBlTguO44Gd9UUh7ailsb0LnCki2U7xx5nOOtc1qD86l8C1rY/1IqfVTgnQA/icdnqPOOXwjwELVfXPQZvC7to2FWs4XlsR8YtIlvO4A3AGgTqdScBYZ7eG17X+eo8FPtRADXZTr8HtWBcFfZEQAnU5wde1fd8DbVFjH40LgZYPSwiUm/4mDOLpRqB1yGxgfn1MBMppPwCWOj9z9D8tPR5y4p8LVLoc3/MEii0OEvjmc8XRxAb8kECFZTXwg3aM9WknljnOP2LHoP1/48S6GDirPd8jwMkEih/mALOc5dvheG2PEGvYXVugHzDTiWkecEvQ/9nnzjV6GUhy1ic7v1c727s19xraIdYPnes6D3iG/7Tsavf3gPVsN8YY0ypWtGWMMaZVLJEYY4xpFUskxhhjWsUSiTHGmFaxRGKMMaZVLJEY0wwR+cT5WSwi32vjc/93Y89lTCSx5r/GhEhEhhEYxfacFhwTp6qHjrB9j6qmtUV8xnjF7kiMaYaI1I+8+kfgFGfuh185A+ndKyLTnYHzfuLsP0wC83I8R6BDGCLyujPY5vz6ATdF5I9AB+d8zwY/l9Mr+V4RmSeBOWguDDr3ZBF5RUQWicizTs9mROSPIrLAieV/2vMamdgW3/wuxhjHTQTdkTgJYaeqDnZGXv1YRP7l7DsE6KOBocUBfqiq25whLqaLyKuqepOIXKOBwfga+i6BQQ77A3nOMVOdbQOB3gTGSfoYOElEFhAYfqRcVbV+SA1j2oPdkRhz9M4kMKbRLALDpecSGGsJ4POgJAJwrYjMBj4jMHBeD47sZOB5DQx2uBGYQmCU1/pz12hgEMRZQDGwC9gPPCoi3wX2tfrVGRMiSyTGHD0Bfq6qA5ylRFXr70j2frVToG7lDOAEVe1PYNyk5BDO3ZQDQY8PAfEamCNjCIGRd8cA77TolRjTCpZIjAndbtzq54AAAADWSURBVAJTyNZ7F7jKGTodEenpjMzcUCawXVX3iUg5geHK6x2sP76BqcCFTj2Mn8D0wE2OKiuBOUAyVXUi8EsCxWLGtAurIzEmdHOAOqeI6kngAQLFSl84Fd6b+c/UrMHeAX4qInMIjBD7WdC2R4A5IvKFql4StP414AQCoz0r8GtV3eAkosakA+NFJJnA3cyvju4lGtNy1vzXGGNMq1jRljHGmFaxRGKMMaZVLJEYY4xpFUskxhhjWsUSiTHGmFaxRGKMMaZVLJEYY4xpFUskxhhjWuX/A4Lybm2ifx42AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f16e0620a90>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_lr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By default the fastai will freeze the weights of all of the layers except a few last layers and the ones that it adds to fine-tune the model for your dataset. So in above epochs, all of learning is done by those unfrozen last layers.\n",
    "Next, we will unfreeze the weights of all of the layers to get more accuracy out of our model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "975f945422c1455781f5f8885b61d9e7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.11419  0.10877  0.90685]                        \n",
      "[ 1.       0.10967  0.10414  0.91113]                        \n",
      "[ 2.       0.10298  0.09952  0.91535]                        \n",
      "[ 3.       0.10733  0.10075  0.91436]                        \n",
      "[ 4.       0.10066  0.09977  0.91328]                        \n",
      "[ 5.       0.09702  0.09785  0.91646]                         \n",
      "[ 6.       0.09365  0.09775  0.91603]                         \n",
      "\n"
     ]
    }
   ],
   "source": [
    "lrs = [lr/9, lr/3, lr]\n",
    "learn.unfreeze()\n",
    "learn.fit(lrs, 3, cycle_len=1, cycle_mult=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have noticed, I have used an array for LR instead of a single value. If you give an array of 3 elements to fastai, it will divide the layers into 3 equal sets. For each set, it will use corresponding value from array.Since we are using a pre-trained model, and in a CNN initial set of layers usually learns simple features(like find a edge, corner, etc) so we don't want our initial layers to change too much, therefore we are using the lowest LR for them.  Higher layers in CNN learns to find complex features (like geometrical patterns,faces etc), so having a higher rate for them would be good idea so they can adopt more rapidly to our data set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJzt3XeYVOXd//H3dzuwdBakLyAoiEhZuiIRpOgvEhOMLdZEfSxJTNEH7CL2aIxPjCVqjEZDsBtBERt26dJ7XTouS1uWbffvjzk7zMzO7s7K7s4y83ldFxdnzpw55zuH4TNn7nPOfZtzDhERiQ8J0S5ARERqj0JfRCSOKPRFROKIQl9EJI4o9EVE4ohCX0Qkjij0RUTiiEJfRCSOKPRFROJIUrQLCNWiRQuXmZkZ7TJERI4p8+bN2+2cy6hsuToX+pmZmcydOzfaZYiIHFPMbGMky6l5R0Qkjij0RUTiiEJfRCSOKPRFROKIQl9EJI4o9EVE4ohCX0QkjkQU+mY2xsxWmtkaM5sQ5vnfm9kyM1tkZh+ZWceA5y4zs9Xen8uqs/hQm3PymLVqV01uQkTkmFZp6JtZIvAEMBboAVxoZj1CFlsAZDnnegGvAQ95r20G3AkMBAYAd5pZ0+orP9iZf57FZc/Ppqi4pKY2ISJyTIvkSH8AsMY5t845VwBMAcYFLuCc+8Q5l+c9/AZo502PBmY653Kcc3uAmcCY6im9rPxCX9gff+t7NbUJEZFjWiSh3xbYHPA425tXnl8Cpakb0WvN7Gozm2tmc3ftUvOMiEhNiST0Lcw8F3ZBs18AWcDDVXmtc+4Z51yWcy4rI6PS/oJEROQHiiT0s4H2AY/bAVtDFzKzkcCtwDnOucNVea2IiNSOSEJ/DtDVzDqZWQpwAfBO4AJm1gd4Gl/g7wx4agYwysyaeidwR3nzakRqkq5AFRGpSKUp6ZwrAm7AF9bLganOuaVmNsnMzvEWexhIB141s4Vm9o732hzgHnxfHHOASd68GjHqpOMAaJha53qMFhGpEyJKR+fcdGB6yLw7AqZHVvDa54Hnf2iBVVFS4jtdUKBLNkVEwoqp9pBpi7cBvtB3Luy5ZhGRuBZToV/KOSgsVuiLiISKydAHNfGIiIQTu6FfpNAXEQkVM6FfXBLcnKPQFxEpK2ZCPzTkDxUWR6kSEZG6K+ZCf2T3VgBs2XMomuWIiNRJsRP63onbNk3SAMg9VBDNckRE6qSYCf0W6SmsmjyWy4dkAke6WRYRkSNipr8CMyMlyUhP870ltemLiJQVM0f6pdKSEwE4rNAXESkj9kI/yRf6+Qp9EZEyYi70kxONBFObvohIODEX+mZGWnKijvRFRMKIudAHX7u+TuSKiJQVk6Gfc7CAl7/dFO0yRETqnJgMfRERCS8mQz+rY1NaN06LdhkiInVOTIZ+x+YNsGgXISJSB8Vk6DdMS+LA4aJolyEiUufEZOinp/pCX+PkiogEi83QT0uixKn/HRGRUDEZ+g1SfZ2uqYlHRCRYTIZ+w9LQz1foi4gEisnQT/dC/+BhNe+IiASKydAvbd7Zf7gwypWIiNQtMRn6Db2BVDZ+nxflSkRE6paIQt/MxpjZSjNbY2YTwjw/zMzmm1mRmY0Pee4hM1tqZsvN7HEzq/H7pkqP9Ce+sbimNyUickypNPTNLBF4AhgL9AAuNLMeIYttAi4HXgl57RBgKNAL6An0B04/6qorkZyo+3FFRMKJZIzcAcAa59w6ADObAowDlpUu4Jzb4D0XOnKJA9KAFMCAZGDHUVddibZN6gHQP7NpTW9KROSYEknzTltgc8DjbG9epZxzXwOfANu8PzOcc8urWmRVmRk92zaiUVpyTW9KROSYEknoh2sriah/AzM7HugOtMP3RXGGmQ0Ls9zVZjbXzObu2rUrklVXKj01if26Tl9EJEgkoZ8NtA943A7YGuH6zwW+cc4dcM4dAN4DBoUu5Jx7xjmX5ZzLysjIiHDVFUtPTdYduSIiISIJ/TlAVzPrZGYpwAXAOxGufxNwupklmVkyvpO4Nd68A5CemqjQFxEJUWnoO+eKgBuAGfgCe6pzbqmZTTKzcwDMrL+ZZQPnAU+b2VLv5a8Ba4HFwHfAd865/9bA+ygjXd0ri4iUEcnVOzjnpgPTQ+bdETA9B1+zT+jrioFrjrLGHyQtKZGcgwU456iFWwNERI4JMXlHLsCzX6wH4PGP1kS5EhGRuiNmQ//pS/oB0KS+LtsUESkVs6E/qHNzAAqLQ+8XExGJXzEb+g1Tk0gw2HtIPW2KiJSK2dBPSDAa10smN0+hLyJSKmZDH6BRvWT25Sv0RURKxXbopyWzT807IiJ+sR369ZLYp/53RET8Yjv0daQvIhIk9kNfbfoiIn4xHfr1UhLJKyiOdhkiInVGzId+fqFCX0SkVGyHfnIihcVOd+WKiHhiOvQ/XrETgGVb90W5EhGRuiGmQ/+C/r4Bv9QVg4iIT0yH/sntGgPoZK6IiCemQ79hqq9bZY2gJSLiE9Ohn57mGxjsgK7VFxEBYjz0G6QmAjrSFxEpFdOhn5qUSEpSAvsV+iIiQIyHPvgGUzmgTtdERIA4CP30tCQ174iIeGI+9Bum6UhfRKRU7Id+ajK5ujlLRASIg9Dv2Lw+G78/GO0yRETqhJgP/S4Z6ew+UMDbC7dEuxQRkaiL+dDv3aEJABNeXxzlSkREoi+i0DezMWa20szWmNmEMM8PM7P5ZlZkZuNDnutgZh+Y2XIzW2ZmmdVTemT6ZzYD4Hyv8zURkXhWaeibWSLwBDAW6AFcaGY9QhbbBFwOvBJmFS8CDzvnugMDgJ1HU/APYQarduyv7c2KiNQ5SREsMwBY45xbB2BmU4BxwLLSBZxzG7zngkYr8b4ckpxzM73lDlRP2VXjHHy19vtobFpEpE6JpHmnLbA54HG2Ny8S3YBcM3vDzBaY2cPeL4eoKClx0dq0iEidEEnoW5h5kaZnEnAa8EegP9AZXzNQ8AbMrjazuWY2d9euXRGuOnIXDugAoOv1RSTuRRL62UDgWdB2wNYI158NLHDOrXPOFQFvAX1DF3LOPeOcy3LOZWVkZES46sgN7tIcgN0HDlf7ukVEjiWRhP4coKuZdTKzFOAC4J0I1z8HaGpmpUl+BgHnAmpL8wYpAOQcLKjtTYuI1CmVhr53hH4DMANYDkx1zi01s0lmdg6AmfU3s2zgPOBpM1vqvbYYX9POR2a2GF9T0d9r5q2Ur2l9X+jP27intjctIlKnRHL1Ds656cD0kHl3BEzPwdfsE+61M4FeR1HjUTuucRoA//pmI9f/6PholiIiElUxf0cuQDOveafE6eodEYlvcRH6pXbs04lcEYlvcRX6AE5H+yISx+Im9K8c2gmAfYc0oIqIxK+4Cf3S3jZ37s+PciUiItETN6FfP9nX+8OjM1dFuRIRkeiJm9DPbFEfgK/XqeM1EYlfcRP6x7dsSFpyAmec2DLapYiIRE3chD5A99aN2LFPbfoiEr/iKvRbpKfy5ZrvddmmiMStuAr95dv2AfDPrzZEtxARkSiJq9C/7WzfKI9m4YYIEBGJfXEV+lmZTaNdgohIVMVV6DdM83Uqul0nc0UkTsVV6Kcm+W7QevLTtVGuREQkOuIq9APNXp8T7RJERGpd3IV+/RTf0f6K7fuiXImISO2Lu9D/7OYfAbBu18EoVyIiUvviLvRbpKcC8IKu1ReROBR3oS8iEs/iMvTP7dMWgJISdccgIvElLkP/rYVbAPjd1IVRrkREpHbFZeiPO6UNADrQF5F4E5eh/+D4XgCkJsXl2xeROBaXqZealEj7ZvV4bV52tEsREalVcRn6AJtzDgGw+8DhKFciIlJ74jb0h5+QAUDOwYIoVyIiUnsiCn0zG2NmK81sjZlNCPP8MDObb2ZFZjY+zPONzGyLmf21OoquDv9zehcAdu/Xkb6IxI9KQ9/MEoEngLFAD+BCM+sRstgm4HLglXJWcw8w64eXWf1K78zdpeYdEYkjkRzpDwDWOOfWOecKgCnAuMAFnHMbnHOLgJLQF5tZP6AV8EE11FttMhp6oa8jfRGJI5GEfltgc8DjbG9epcwsAXgEuKnqpdWsRmlJpCQm6EhfROJKJKEfbkDZSG9rug6Y7pzbXNFCZna1mc01s7m7du2KcNVHx8xokZ7C7v06kSsi8SMpgmWygfYBj9sBWyNc/2DgNDO7DkgHUszsgHMu6GSwc+4Z4BmArKysWrtPNqNhqo70RSSuRBL6c4CuZtYJ2AJcAFwUycqdcxeXTpvZ5UBWaOBHU5sm9Vi1Y3+0yxARqTWVNu8454qAG4AZwHJgqnNuqZlNMrNzAMysv5llA+cBT5vZ0posurp0aFafzXsOqbdNEYkbkRzp45ybDkwPmXdHwPQcfM0+Fa3jBeCFKldYg9o3q09BUQk79x/muMZp0S5HRKTGxe0dueA70gfYlJMX5UpERGqHQh/Y+L3GyxWR+BDXod+mST0AHnx/RZQrERGpHXEd+ilef/q7D+hafRGJD3Ed+oEOHC6KdgkiIjUu7kP/2uG+3jave3l+lCsREal5cR/6p7RrAsBnq2qn+wcRkWiK+9AffVIr//S5f/syipWIiNS8uA99syP9yS3YlMvi7L1RrEZEpGbFfeiH2rEvP9oliIjUGIU+8Mwl/fzTG3V3rojEMIU+MOqk47hxZFcA7nl3WZSrERGpOQp9z40ju/mnDxcVR7ESEZGao9APQ3foikisUugHeOKivgDsOajQF5HYpNAPkOr1xXPLm4ujXImISM1Q6Ac4pb3v7txF2XvZmnsoytWIiFQ/hX6AjIap/uklW3w3ae09VIhzGk5RRGKDQj/E1xPPAGDtroNszsnjlLs/oNPE6ZW8SkTk2BDRGLnx5LhGvrFyH3x/BUXFJf75OQcLaNYgJVpliYhUCx3phwjsi+eRmav8033vmRmNckREqpVCvwr+8eX6aJcgInJUFPph/PeGU/3TDdOS6NvBd1XPi19vjFZJIiLVQqEfxsntGvP6tUPo3KIBC24/kzeuGwrA+t0Hyc3z3biVm1fA3A050SxTRKTKFPrl6NexKR//cThJicG7aMQjswC44oU5jH/qawqKSsK9XESkTlLoR+iTPw4HoNg58gqKWLApF4A9eeqyQUSOHQr9CHVq0YA+HZqQm1fI+Ce/9s+fOmdzFKsSEamaiELfzMaY2UozW2NmE8I8P8zM5ptZkZmND5jf28y+NrOlZrbIzM6vzuJrW7LX1LNs2z7/vEdmrmL64m3RKklEpEoqDX0zSwSeAMYCPYALzaxHyGKbgMuBV0Lm5wGXOudOAsYAj5lZk6MtOlrqJSeGnX/dy/OZt3FPLVcjIlJ1kRzpDwDWOOfWOecKgCnAuMAFnHMbnHOLgJKQ+aucc6u96a3ATiCjWiqPgnvG9aR14zT/48b1kv3TL329ofYLEhGpoki6YWgLBDZcZwMDq7ohMxsApABrq/rauqJD8/p8PXEEzjn25ReRnppEgkGnidN5a+FWHrugT7VtKzevgNfmZXPl0E4kJFjlLxARiUAkR/rhEqdK3U6aWWvgJeAK51yZaxzN7Gozm2tmc3ft2lWVVUeFmdG4XjKJCYaZkeiF8pqdB6ptGze9tojJ05Yz/qmvePyj1dW2XhGJb5GEfjbQPuBxO2BrpBsws0bANOA259w34ZZxzj3jnMtyzmVlZBx7rT83jT4BgGmLjv6E7g2vzCdzwjRmLtsBwPxNuTwa0AeQiMjRiCT05wBdzayTmaUAFwDvRLJyb/k3gRedc6/+8DLrtsuHZAKwbrfvSH/NzgM/uA/+d8v54jhUoMHaReToVRr6zrki4AZgBrAcmOqcW2pmk8zsHAAz629m2cB5wNNmttR7+c+BYcDlZrbQ+9O7Rt5JFKV5V/W8vXAr73y3lZGPzuKN+VuqvJ7DRWWD/drhXQD446vfUVLi+yLZsPsge/MKj6JiEYlXVtdGhcrKynJz586NdhlVljlhWpl5yyeNoV5K+Ms8Q23JPcTwhz+hsPjIv8fvRnajd4cmXPb8bP+8v1+axVUvzqVtk3p8OeGMoy9cRGKCmc1zzmVVtpzuyK0mU64eVGbeiu37wixZVvaePIY+8HFQ4AP8ZsTxdG7RIGjeVS/6vhC35B5i+978iI/4V27fz6XPz+bg4aKIlheR2KTQryaDOjcvM2/p1shCf8W2/UGPLxrYgfX3n4WZ0b5Zfbq2TA+/zfs/ot/kyAZ3Gf3YZ3y2ahdfrNkd0fIiEpsU+tVozb1jef3awbx+7RCACu/S/Z+X5pE5YRobvz/Ihu8P+uf/+JQ23HZ296ARvN6/cRivXDWwzFE/QFGJq/Qk7/rdR9Z/zUvzIn4/IhJ7NEZuNUpKTKBfx2b+x28u2MLI7q04u1dr/7w9BwvoEzD04ukPf+qffuGK/pzeLSMo8AESE4whXVrw8R+HM29jDiu27+fWN5f4n588bRk3juxGRsPUsHUtC/nF4Zwrsw0RiQ860q8hpZdxXv/KfDbn5AHw+EergwI/1PATWlYaxv06NuPigR25/f8d6f7o5W830f/eD8kvLGbVjv1lXvO3T9cAcMaJLQH4w9TvqvReRCR2KPRryA1nHO+fvuS5bznrL59XeJNVYIhH4penduL9G08Lmnfi7e8z6s+fsWv/YQAufOYbHnhvhf/cQmkz0BsLqn45qYjEBoV+DWneIMU/veH7vKDumEOddfJx/PLUTlXeRtsm9cLOX7/7IPvzC/l63fc8NetIV0f3//TkKm9DRGKL2vRriJmx4YGzw16///q1g9mfX0SbJvX4cPkOrht+fJg1VK5hWjIf/eF0ps7dzNOz1vnn/+/ri8pcTfTilQPIbNGAa4Z15unP1rF9bz7HBfQYKiLxQUf6tejuc05i2aTR9OvYjOEntKRbq4Y/OPBLdclIZ+LY7vzpvFP889bvPsi/Z28KWi4rsykAg7v4vgyy9+SVWdeW3EOs3O47J7Bky16G3P9R2OUE8guLKS6pWzc2ikRCoV/DRvVoBcCGB87msiGZ1E+pmR9X4/u1Y8MDZ5f7fOl22zX1NQnd/NoiMidM408zVjJj6XYAhj7wMaMf+4yCohL+3/99wda9+ZzxyCyFW4C8giIOHi7ixNvfp8st09maeyjaJYlUiZp3atiTv+hHUUmZ3qRrzJ0/7sHd/10GwMjurRjSpTmB3fG3bVIfgHXetft//cR3Zc+jPz/yS2Huhhz/dEFRCRNeX8TDAb8k4lmPO2YEPR7ywMcVftmK1DU60q9hiQlGalJk/e9UhyuGduLxC32Dudw4sitXntqJy4ceOUlcXl9AN7+2yP/lsHTrPk5q08j/3Kvzsiksrr0vrrrg75+tI3PCNDVvScxR6Megc05pw8I7zqRn28Zhn//4D6eXmVdU4ihtxdm2N5/ckD59xj/1dbXXWZfdO305AKc++Il/XugXQOmYyVPnbEbkWKHmnRjVpH5Kuc91zgjfl0+p579cD8Dvz+xG68Zp3PTaIr7bnAvgHycglu/oXZy9N+jxqh37Gf3YZ4R2SDuie0veXbSNm19fxF8+Ws2W3ENq6pE6T6Efp9799ansySvgtK4ZFBSV0O229wBIS04gv9DXlDOoc3MGdGrGk7PWknOwAID7pi/n75+vZ1zvNvykT1tObtuYFunhu384FuUXFvPytxuD5o3682dBj1dNHktKUgLPfr7OP+jNFu+EbnGJIzHB2H3gMO8s3MoVQzMr/IIsKCohJUk/uKX26NMWp3q2bcxpXX1DU6YkJdCmcRontGroD3yArI6+yzzPz2pPbl4h+/ML+fvnvl8Bby/cyhX/mEPW5A95dOYq1uz0XepZcoxf6fPnD1cxxWuumfyTnmWeb9+snj+krxxa9oa6b9d/T2FxCVmTP2TSu8tYtm0f17/sGwIzdOyKdxdtpdtt75E5YRpzAk6ei9QkDaIigO+I0wwG3/8xuw8cZsU9YwJGBNvCb6csjHhdk8adxKWDM2uo0ppRUuKY+MZi/jP3SPt8uJvrpl4zmAGdjnSqd/BwETe/tohpi8MPc3lmj1b+8Y4BHh7fi/OyfENOh65bTUNyNDSIilRJSlICyYkJfHrTcD7+w+n+wAffDWCBOjavz6v/M5g25dzR++6ibfx2ygJ27Muv0Zqr0wfLdgQF/hvX+brHLg349fefxYLbzwwKfIAGqUk8cXHfsL8KgKDAB7jptUWAbxzlymzbeyii5USqQm36EiQ9NYn0kJA/4biGQY+vOq0z/TObcePIbtz8ui/E2jerx+YcX7v27PW+poq3F27l8Qv7cM4pbWqh8qrbnJNH0wYpbN+bz//868g4Az/PakffDr6mrSlXDaLY64q6aYPyT45fPLADt721pNznA2XvyWPko7PKzN+29xDJiQkUFJVw9UtzWbLF11/TaV1b8OxlWbV66a/ELjXvSEQOFRRz8bPfMH9Trr/pp6i4hKVb99GrXWMKix1DHvA1DZXn3nN7cvHAjrVYdbA1O/fTpkk96qckkV9YzIm3vw/47lLO3uP7wnpofC9+7jW/VFX2njz+/tk6/vm170Twf64exPnPfEO7pvV4/dohPPT+Sl6fnx30mr9c0Ju8gmImvrG4wnUP6tyMKVcP/kF1SXyItHlHoS/VathDn7App/wbml751UB+P/U7RvZoyV0/PomkxOAWxp3788lIT8U5SEionstCs/fkMeKRWRwu8p2kLq8jvLeuH0rv9k2Oalu7Dxwma/KH/u0E2rX/MP3v/TBo3oYHzg76AqrI+VnteXB8rwqXefyj1TStn8wlx9g5FTl6atOXqPA1QyTw3GXhP3sXPfst2/fl869vNnH8re8Fhe9Xa3cz4N6P6DRxOp1vmV7pMJCR2JdfyKkPfuIPfIAnvK4nQh1t4AO0SE9l9b1jWTZpdJnnMhqmcvc5J5WZH3j+JNCiu0YFPf7P3M10u+29MlcBlTp4uIhHZ67i9reX8vZCjZkg4Sn0pVp1a9WQlZPHMqJ7K64b3iWi15Te6frB0uCTnt3veJ8pIb2FAkydu5krX5iDc46SCsYIzi8sLjPoPMDDM1YCcMmgjv6j8dO6toio1kgkJyaU27HeZUMyuXSwr4lr9i0j/PO/u3MU30w88njGjcNolJZc5tdCQVEJ2XsOMWX2Jt5ccKSp6HBRMSfdeaRfoMCrrdbsPMD4J7+ioCi+utKQ8HQiV2rMRQM78LdP1/KbEV357YiudLllOgC3nd2d4xqn8dD7K9mUk8cvnv2WT2/6UdiO6Sa8sZgxPY/z32F859tL/G3m2/bm8/CMlbzpjQT24e9PJzevgMVb9jJ34x6mLdrGTaNP8K+re+tGLA8YzKa0m+navlRy0rie3H3OSUE3bTWul0zjesmkJCZQUFwSdPL8+cuzWL5tv//L6ty/feU/d/LjXm244ZUFvO/1lBpo94HDtEhP5dy/fcn+/CJmLN3Ojys5qV5UXELOwQJaNtJYC7FKbfpSowLvOM2cMI0hXZrzylWDAPh4xQ6ufMH3b73injFc+vxs/5U/F/Rv779JqrQte92uA5zxSNmrXirSJaMBa3cd5MPfD+P4lg2Z+MYi/j3bt97Pb/4R7ZvVr5b3WV0OHC7C8F0KGsg5xzUvzeODkEtAQ7VtUo9fDOrIg++vYGT3lvz1or5c/Oy3zNu4B4B1951V4bmS309dyBvzt9C7fRPeun7oUb8fqT06kSt1Tn5hMYkJRnLAydvQE6rdWzfi7nNOYkCnZny1ZjcXPfst4Dsan7lsB1e9+MM+GwvvONP/a6GouITEBDvm+g/KKygq07VzoJTEBFbdO5bC4hK63vpe2GXO7tWaJy7qC/huSAv8Ali76wAjAr5Urx7WmeuHH8/KHfvL3J8gdY9O5Eqdk5acGBT44AuhQI+d39sfMEOOP9LOvjknzx/4M383LOg1lw3uGDRyWDiBHdAlJSYcc4EPvoFwhh7fvNznC7zur0P3caBpi7ZRVFxC5oRpdL5lOje9+h378wvJOVgQFPgAz3y2jlMmfcDPn/6aJz9de8x3sSE+EYW+mY0xs5VmtsbMJoR5fpiZzTezIjMbH/LcZWa22vtzWXUVLrEh8GqWlKSEMjeCdWvlu1HstIeOdHHcJSOdXu0ae9MNuHtcT8b3a8dDP+vFoM7NeOz83gD+I9rRJ7Wq0fdQm17+1SBm3zKCpXePZv39Z/HY+b15/dohZZZbOXlM0OOR3Y/sg+MDfgW8Oi+bk+/6gL73zPTPOzHk3wDgwfdX0HfyTOZtzCn36iE5NlQa+maWCDwBjAV6ABeaWY+QxTYBlwOvhLy2GXAnMBAYANxpZk2PvmyJFS3SU7n3XF8XBp1bNCjzfGigtW6cRkKCcf9PT6ZeciIv/nKg/7mf92/PlKsH85M+bdnwwNmc3as1Gx44m6cvqfQX7zGlZaM0GqQmYWb8pE9b/6WmrRod6e009O7dh8f34jcjuka0/vdvHBZ2fm5eIT978mt/z6Kj/jyL+7xxB+TYEcmR/gBgjXNunXOuAJgCjAtcwDm3wTm3CAi9/GI0MNM5l+Oc2wPMBMYgEqBPe99xwLjebcs81zAt2T+dlpzAB17TzkltGrP8njG0bVKvdoqswxITjCcv7lvmC/LxC/tw4YAOfDnhDJo2SOF3I4ND/7s7gu8DAPwjpq297yxe+uUAAE49Pvhy1l//ewGZE6axascBnvls3Q+q+e2FW/hq7W4G3vehv9tuqR2RXLLZFggcGigb35F7JMK9tuz/bIlrPdo0YvmkMeUO5fiPK/ozfdE2jdNbgbEnty4z75xT2gT1e2RmXDE0k398uYGp1wymcf1k7j23J7e+eaTPoH9c3h/wfZGc1jXDfznrtr2HGHz/x2G3nV9YXO4NZuEcKigOuo+g7z0zeeKivmXO70Tq+wOHaVo/pdru4I51lV69Y2bnAaOdc7/yHl8CDHDO/TrMsi8A7zrnXvMe3wSkOucme49vB/Kcc4+EvO5q4GqADh069Nu4MXgQCxGpHs45ikucv/sL5xwLN+dSPyWJRvWSaN244l9OO/blM/C+j8rMX33v2ApPIAfauT+fAfeWXQdU/Z6JzTl5/vM9lV0jgjIyAAALpUlEQVSOGuuq8+qdbCCwB6p2wNYI64jotc65Z5xzWc65rIyMjAhXLSJVZWZB/R2ZGX06NOWE4xpWGvgArRqlMe+2kUz/zWnMumm4f/7IR2fx3Bfrmb9pDzOX7WDdruAuoTfn5LH3kG/c5QP5ReWu//sKOuwLZ+P3R/p5+vecTeQVlL9u8Ykk9OcAXc2sk5mlABcA70S4/hnAKDNr6p3AHeXNE5FjVPP0VHq0aUTH5kdOvG/8Po973l3GT//2FVe9OJc/vvpd0GtOe+gTTrn7A5xzQUENMOLElv7pZz5fR3GEl4YeLirmF899639865tL6HHHjEqD3zkX111SVBr6zrki4AZ8Yb0cmOqcW2pmk8zsHAAz629m2cB5wNNmttR7bQ5wD74vjjnAJG+eiMSAD34X/kqf+Zty+Wrtbs58dBZfrN7tn99p4nSueGEOAC3SU7jlrBN59rIslt7t66Du6Vnr6HLLdPILff0pTX53GZkTprFye9k+lErHbwh15QtzKCgq8a8jv7CYSf/1ref6l+fTb/KHdLvtvbgNft2RKyJH5aVvNnJ7hAPIBFp01ygaBVydFW74yMB5s28dQcuGaWGXX3ffWQz/06dhu/W+cEB7f9cbgfp1bBr2Hodjle7IFZFaMbTLkbuEn700y39TXEV+M6JrUOAD1Au5AuivH68Oejzg3o947ov1bN+bHxT4n9/8IxISjM9u/lHYbYULfIBVAb8eCotLeP6L9f5fB6Fy8wrInDCNiW8sLneZY4WO9EXkqB0q8PWrVNq53uvzsvlDQLv+Ca0acuPIrny8YifHNU7j12d09S9batf+w/zyn3PYtjefXfuPnNBtmJbE/oCTv6N6tArqeC7wip8tuYcY+kD4S0tvO7s7k6f5bib71amdeO7L9ay4ZwypSYk88N4Knpq1lgSDdff71uec48w/f0aXjAYkJhjTFx/pyfT/LuxTaY+l5XlzQTa3vbmEhXeO8l/x5LwhOY9GpEf66lpZRI5a6D0WP+vXjp/1a8eOffms3XWAIV18N3iFu5+gVEbDVN654VScc3SaON0//5M/DqewuMR/n0Bg4E8ce2LQOto2qccp7Zvw3ebc4PqSE/nVaZ1ZsX0/BjRLT8E5eH3eFn7aty1PzVoLQImDZVv30aNNIx6duYo1Ow+EHZz+1/9eQJsmafTrWPWO6H73H9+X4bX/ms+j559Cr7s+AGDZpNHljsNQndS8IyI1plWjNH/gR8rMaJjmC79xvdvQIj2V1o3rBTUbJSUYi+8axTWnlx2oZ+o1g7j/pyfz1vVDWXjHmYzteRyLvVHI/nTeKTx83imM7en78nlvyTbmbAi+tuQnT3wJwLfryl5z8pcLevunf/bk12WeP3C4iBtemc/mCoYMHeI1h324fAcLNx35cppZSbfZ1UWhLyJ1zotXDuDyIZn+zvMA2jc7ch/BfeeeHNRFR6DUpEQuHNCB3u2b0KR+Ck/+ol+ZsZg7ef08fb56N5c8NxvA31Nr/05NyS8sZnbAl8FvzjieZZNGM653W8b2PM4/f/yTXwWtt+edM3h30bagDgL3HCwgv7CYffmF7D1USGHxkauGLn1+tn/6t1MWsjh7byV75uipeUdE6pw+HZrSp0Nw34wdAga8ObPH0fecOuLElny0Yqf/8fh+7fho+Q7eW7KdGd5IZAM7NeM/1wwOet2Tv+jH/E17+OnfvmLuxj3MWLqdf32zMeyQm8Uljj73zKRfx6b+gWwq8uO/flHjI7npSF9EjgmlYyJ0aFafpg1SKlm6cjePOXI+4Iv/9V35U9qsVNo30I0ju4V9bd+AL6RrXprH56t3c9/0FUHLZE6Yxh1v+y5lDQ38wHO2ackJQeMj1zRdvSMix4zNOXk0Skumcf3wTTtVtXN/Phnpqf4rZ0L7BaroqDtwuM9AP+ndhrcWhu+p5qGf9aKguISLB3ZgweZcnHP+k8HfHzhM43rJZZqiIqWrd0Qk5lT3mMaBN3uFPg4cnyCczi3S/dO/P7Mb6alJbM09xGVDMssN/fH92vk7hesb0nzVPL3i7VUXhb6ISIAld49m1spdjAk4YRtOZosG3DiyKy99vZFz+7T1fyE552iRnsLuA75xAu76cQ/qpSSSYFYnegFV846ISA34YvVuNuYc5OKBHWtle2reERGJolO7tuBUqnaPQm3Q1TsiInFEoS8iEkcU+iIicUShLyISRxT6IiJxRKEvIhJHFPoiInFEoS8iEkfq3B25ZrYL2PgDX94C2F2N5dQk1VozVGvNUK01ozpr7eicy6hsoToX+kfDzOZGchtyXaBaa4ZqrRmqtWZEo1Y174iIxBGFvohIHIm10H8m2gVUgWqtGaq1ZqjWmlHrtcZUm76IiFQs1o70RUSkAjER+mY2xsxWmtkaM5sQ7XoAzGyDmS02s4VmNteb18zMZprZau/vpt58M7PHvfoXmVnfGq7teTPbaWZLAuZVuTYzu8xbfrWZXVbL9d5lZlu8/bvQzM4KeG6iV+9KMxsdML9GPydm1t7MPjGz5Wa21Mx+682vc/u2glrr3H71tpFmZrPN7Duv3ru9+Z3M7FtvP/3HzFK8+ane4zXe85mVvY9aqPUFM1sfsG97e/Nr93PgnDum/wCJwFqgM5ACfAf0qAN1bQBahMx7CJjgTU8AHvSmzwLeAwwYBHxbw7UNA/oCS35obUAzYJ33d1Nvumkt1nsX8Mcwy/bwPgOpQCfvs5FYG58ToDXQ15tuCKzy6qlz+7aCWuvcfvW2b0C6N50MfOvts6nABd78p4BrvenrgKe86QuA/1T0Pmqp1heA8WGWr9XPQSwc6Q8A1jjn1jnnCoApwLgo11SeccA/vel/Aj8JmP+i8/kGaGJmrWuqCOfcZ0DOUdY2GpjpnMtxzu0BZgJjarHe8owDpjjnDjvn1gNr8H1Gavxz4pzb5pyb703vB5YDbamD+7aCWssTtf3q1eiccwe8h8neHwecAbzmzQ/dt6X7/DVghJlZBe+jNmotT61+DmIh9NsCmwMeZ1Pxh7e2OOADM5tnZld781o557aB7z8d0NKbXxfeQ1Vrqws13+D9HH6+tMmkgrpqtV6vOaEPvqO8Or1vQ2qFOrpfzSzRzBYCO/EF4Fog1zlXFGbb/rq85/cCzWur3tBanXOl+/Zeb9/+2cxSQ2sNqalGao2F0A83vHxduCRpqHOuLzAWuN7MhlWwbF19D1B+bdGu+UmgC9Ab2AY84s2Per1mlg68DtzonNtX0aLl1BTNWuvsfnXOFTvnegPt8B2dd69g21GtN7RWM+sJTAROBPrja7L532jUGguhnw20D3jcDtgapVr8nHNbvb93Am/i+5DuKG228f7e6S1eF95DVWuLas3OuR3ef6wS4O8c+Yke1XrNLBlfiL7snHvDm10n9224Wuvqfg3knMsFPsXX/t3EzJLCbNtfl/d8Y3xNhLVab0CtY7wmNeecOwz8gyjt21gI/TlAV+8sfgq+kzbvRLMgM2tgZg1Lp4FRwBKvrtIz8JcBb3vT7wCXemfxBwF7S5sDalFVa5sBjDKzpl4TwChvXq0IOedxLr79W1rvBd7VG52ArsBsauFz4rUZPwcsd849GvBUndu35dVaF/erV1eGmTXxpusBI/Gdh/gEGO8tFrpvS/f5eOBj5zs7Wt77qOlaVwR88Ru+cw+B+7b2PgdHeya4LvzBd/Z7Fb42vlvrQD2d8V0h8B2wtLQmfG2KHwGrvb+buSNn+5/w6l8MZNVwff/G99O9EN/RxC9/SG3AlfhOhK0Brqjlel/y6lnk/adpHbD8rV69K4GxtfU5AU7F9/N7EbDQ+3NWXdy3FdRa5/art41ewAKvriXAHQH/12Z7++lVINWbn+Y9XuM937my91ELtX7s7dslwL84coVPrX4OdEeuiEgciYXmHRERiZBCX0Qkjij0RUTiiEJfRCSOKPRFROKIQl9EJI4o9EVE4ohCX0Qkjvx/KmBj82eCTj8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f16e05e80f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_loss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a756960bd4b94119a7249afaf04dc471",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09663  0.09755  0.91627]                         \n",
      "[ 1.       0.09502  0.09557  0.91785]                         \n",
      "[ 2.       0.09251  0.09574  0.91777]                         \n",
      "[ 3.       0.09531  0.09504  0.91819]                         \n",
      "[ 4.       0.09537  0.0935   0.92074]                         \n",
      "[ 5.       0.09129  0.09462  0.9184 ]                         \n",
      "[ 6.       0.09016  0.09305  0.91991]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "dbc1ae665c8345c4a5403afc0928b7c6",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09771  0.09089  0.9234 ]                         \n",
      "[ 1.       0.09984  0.09359  0.92097]                         \n",
      "[ 2.       0.08881  0.08804  0.9251 ]                         \n",
      "[ 3.       0.09628  0.09679  0.91755]                         \n",
      "[ 4.       0.09159  0.09016  0.92407]                         \n",
      "[ 5.       0.08745  0.08681  0.92835]                         \n",
      "[ 6.       0.08303  0.0862   0.92857]                         \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#train for image size128x128\n",
    "learn.set_data(get_data(128, 64))\n",
    "\n",
    "learn.freeze()\n",
    "learn.fit(lr, 3, cycle_len=1, cycle_mult=2)\n",
    "\n",
    "learn.unfreeze()\n",
    "learn.fit(lr, 3, cycle_len=1, cycle_mult=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a944884750bb4805940bf0aeae64bee4",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.0892   0.09124  0.92263]                         \n",
      "[ 1.       0.09028  0.09193  0.92213]                         \n",
      "[ 2.       0.08588  0.08995  0.92386]                         \n",
      "[ 3.       0.09012  0.09022  0.92478]                         \n",
      "[ 4.       0.09102  0.09064  0.92299]                         \n",
      "[ 5.       0.08691  0.08925  0.92487]                         \n",
      "[ 6.       0.0856   0.08926  0.92479]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "9687ca0c34924ba8aa8bf94f0f75bc58",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.08793  0.08575  0.92879]                         \n",
      "[ 1.       0.08913  0.08725  0.92769]                         \n",
      "[ 2.       0.08394  0.08392  0.9297 ]                         \n",
      "[ 3.       0.08719  0.08988  0.92541]                         \n",
      "[ 4.       0.08671  0.0871   0.92625]                         \n",
      "[ 5.       0.08313  0.08325  0.9305 ]                         \n",
      "[ 6.       0.07701  0.08292  0.93139]                         \n",
      "\n"
     ]
    }
   ],
   "source": [
    "#train for image size256x256\n",
    "learn.set_data(get_data(256, 64))\n",
    "\n",
    "learn.freeze()\n",
    "learn.fit(lr, 3, cycle_len=1, cycle_mult=2)\n",
    "\n",
    "learn.unfreeze()\n",
    "learn.fit(lr, 3, cycle_len=1, cycle_mult=2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4xLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvAOZPmwAAIABJREFUeJztnXecFPX5xz/P7t7uVQ7u6PVAEKWJgBRRFGxgwxp7i73EmETz05gYxZioscTYorF3Y0lERRALqChdej3g6HBwwPW92/L9/TFlZ2Zndmfvtt4+79eLF7Mz35l9dm53nu/3qSSEAMMwDMM4Ui0AwzAMkx6wQmAYhmEAsEJgGIZhZFghMAzDMABYITAMwzAyrBAYhmEYAKwQGIZhGBlWCAzDMAwAVggMwzCMjCvVAsRCx44dRVlZWarFYBiGySiWLFmyXwjRKdq4jFIIZWVlWLx4carFYBiGySiIaKudcWwyYhiGYQCwQmAYhmFkWCEwDMMwAFghMAzDMDKsEBiGYRgArBAYhmEYGVYIDMMwDABWCCpCCHyweDu8vkCqRWEYhkkJrBBk5mzYh7s+XIFHZ65PtSgMwzApgRWCjLdZWhnsPNSQYkkYhmFSAysEGbdLuhXN/mCKJWEYhkkNrBAALNl6AL//cAUAwB8UKZaGYRgmNWRUcbtEcdUri1DX5AcArNlVk2JpGIZhUgOvEAxU1TejutGXajEYhmGSTlYohG/XVeKdBdtsjz/9qe8TKA3DMEx6khUmo2teWwQAuHRMb1vjdx5qTKQ4DMMwaUlWrBAYhmGY6LBCYBiGYQCwQgAgla3QMrRHcYokYRiGSR2sEExYubMa/gAnqDEMk12wQrBg1ENfpVoEhmGYpMIKAUB9c6jCaVlpPgDgUAPnIjAMk12wQgDQucijbldUcXE7hmGyk6xXCNOX70JlbVOqxWAYhkk5Wa8Qbn/351SLwDAMkxZkhUK4bExvdCx0x3yeMRyVYRimLWNLIRDRZCJaT0TlRHS3yXEPEb0vH19ARGXyfjcRvUpEK4loORGdqDlnjnzNZfK/znH6TCbyAy2pau0LsEJgGCZ7iKoQiMgJ4FkAUwAMAnAJEQ0yDLsWwEEhRH8ATwJ4RN5/PQAIIYYCOAXA40Skfc/LhBDD5X+Vrfso1jiIWjTbb/Jzf2WGYbIHOyuE0QDKhRCbhRDNAN4DMNUwZiqA1+XtDwGcREQESYF8DQDyA/8QgFHxEDwWHES2Vwib/3o6bpjQDwDQxN3TGIbJIuwohB4Atmte75D3mY4RQvgBVAMoBbAcwFQichFRXwAjAfTSnPeqbC76k6xAEgIRUN3ow5z10RchDgehf6dCAKwQGIbJLuwoBLMHtXG+bTXmFUgKZDGAfwD4EYBfPn6ZbEo6Xv53hembE91ARIuJaPG+fftsiGtyDVm8q19dZDlmypCumP2bCQAAT450W5p8bDJiGCZ7sKMQdkA/q+8JYJfVGCJyASgGcEAI4RdC/Eb2EUwF0B7ARgAQQuyU/68F8A4k01QYQogXhRCjhBCjOnXqZP+TaXDYWHus31OLAV2KAAAel3RbNlbWtej9GIZhMhE7CmERgAFE1JeI3AAuBjDdMGY6gKvk7QsAfCOEEESUT0QFAEBEpwDwCyHWyCakjvL+HABnAlgVh89jisOGRlB6KgOAx+UEANz45hJs48xlhmGyhKgd04QQfiK6DcAsAE4ArwghVhPRNACLhRDTAbwM4E0iKgdwAJLSAIDOAGYRURDAToTMQh55f458za8A/DuOn0uH1jsRCAo4TRSEdozbFdKTe2q86C3XN2IYhmnL2GqhKYSYAWCGYd99mm0vgAtNzqsAMNBkfz0kB3NSII2Lwx8MwulwKnKYjnFotAOXwWYYJlvIikxl7YLAr0k2C2hiUbUrBO0KookVQlbz1Fcb8dRXG1MtBsMkBVsrhExHP+MPKQG/ViFoxns0JiNH4qJhmQzgya82AAB+ffKAFEvCMIkn61YIvmBoxu/TzP61aRBDexRjVJ8OANhkxOh54NPVeOOnilSLwTAJISsUgtYepDUTaVcLJx8ZKqXkcBDuP3uwbvxFL/yEK19ZmGhJmTTn1XkVuO+T1akWg2ESQlYoBN0KQTPj15qM/nimvjyTYiqauXoPAGDBlgP4bkPLEuOYzISr3TLZRpYoBCsfQkg55Dj1t0JxLH+8dGeCpWPSFW1uCsNkA1miEELbWiWgKIe/XzAs7BxnVtwZJhLVjdxXm8kusuKxp3UYa81EivnIuDoA9KuKLfvrEygdk67UekMrhKChXK7xNcO0BbJCIUQLO3U5w0NLG5pDhe3mle9Xt5dsPYinvtqIHzbuDzuHaVs0a6rdNhuizeZuZH8S0/bIEoUQ2n5+7iZ1W1EOLkf4bThcLnQH6B8MOw424MmvNuDylxckQNKW4w8E2QkaZ7QBCPM3V+mO1bN/gWmDZIlCCGmEz1fsRpM/AF8giLs+XA4AcJnUNtLWM3r1xy2JF7IVNPkD6H/vF3jsy/WpFqVNoV0VXPf6Yv2xFvTKWLC5CjVe9ksw6UtWKARjsvGr8yowr3w/Vu+qAWBuMtKy/UCjut29fV7c5WstdbKt+63522I+t2J/PXZXN0YfmIX4LLLaAaAxxl4ZdU1+XPTifNz81pK4yMYwiSArFIKx/ERDk1+3b8dB8wfiy1dJ3T57dggpgYo0dDAr/o6cKIrNjBMfm4Nxf/sm3iK1CSKtArQOZzsEZOWyYnt1q2RimESSJQpB/1pAryR2HTJXCEf3lspXaBXGXR+uiLt8rUV5OJmV9WZaji9C2ZJ2uTkxXSsg+3dq2ffApDHZoRAMD8qG5oDu4VngMa/xV5SbGbX/KqqkVQuZdjJlWopRIWid9s3+2ExG2ppYs+Tsd4ZJN7JCIZDBZPTyD1sgNG2hLxzV0/Q8s/yEdOTDJTsAAPvrmlIsSduiSWMyOm1wF10drIYYfQhaH0Q6mh0ZBsgShWBmSWnyacpWmISdZhInDpR6TR/Vq32Lr1Hr9XGpBgM+3ax+r+6h3thsrhBemLsJD32+Jmx/uaY/N5v2mHQls5+ENjHrafCnT0ItnO30XE5nFOdn0CIPYeaqPXj668hNXobe/yWG/HlW3GXLZHwGp7L2/jZYKIS/fbEO//5+S1jZ9F+9+7O6bRbmzDDpQJYohPB9iqP43tOPRHGetYPQHcFslC6JYIppw2zW2tDsx01vLcHjszckW6yMRxt2CgB//G9oElETpc5RfZP+b+HVmJicGWKKZLKPrPhmGn0IWsYdVhrx3NJCNwDgb+cNVfeN7y+dY3xgpAplhWAWFTPoPutZf7ootHRFSUxTZvQf/xyqfGtW+O7Zb8vV7dom/XG+00wmkB0KIcKxvh0LIp6rJK11yA+tIhRF0BRjpEmiaFIVQmyPHWN9HkaPomifufTosGNNJjkKf58VyhSPlKfQFKNDmmGSRVYohEh9ka1CThWUSKN8d2ic8oNuSfmCRKDIYZRnw97aiOd5m9ND/nTFFwjC5SD07JAfdswb5aFuVAjHalaiZsqEYdKBrFAIOy0Sz+ygmGEKPE51nxIlki4z7OaArKA08tR6fTj1ye9044yOTrPyC9HMSF+t2YslWw+0VNSMwhcIwu1ywOMK/5kYH+rGctgH6vUhwGWlBer3hlcITLqSFQqhNYlABfLKQFs2W5kxpssKQQmh1cqzbPuhsHFGBWamEKIpueveWIzzn/+pJWJmHL6AgMtBukKHCsu2H8L900O9lX1B/X278wN9RrsvEET7vBzk5jjgTZPvDcMYyQqFoA3zKy1wx3Tuf24ah9sm9scxZSXqvmE9iwEAL32fHlVQlYe41qfx2fLdYeM+WroTs1bvwaZ9Ukz8wi1VYWNW7bRXaycbHNLN8gqh0MKs+NqPFer2Owv0hQVH9Omge/32gm2oqm+G1xfExiimPIZJFVmhELSJQFX1zTGd2y43B3eeNlCXq+DJkcxHb87fGh8BY2TD3lq88kNIGSkrBF9AqNm07y/eHnben/63Cje+uQQnPT4XgHl11IVbDtqSwSoOvy3h8weR43SgtNCj7isrDfcnAMDrGuUAAEN7tLO87rfrubkOk55khUIwa4DTGsb0LYk+KIGc/cwPmPbZGnWWrjXzNPoCeGzWenQsjL4SWmmyGrB64AHA+j2hma0/TUJuE4k/KMLKl1RUNaBzkaQgtFVwjR01s0FhMm2PrFAIA7sWRR8UA1on446DDXG9th28is8gEO47OFjfjGe+Lcf+uthWQnY47R8hJ3W6hNwmkuZA0LSkuBK11iE/pHQDBo0wc5Xeb9WjfR7OG9EjAVIyTPzICoXwq5P6AwDycpzo0s4TZXR0tCaoSY/NbfX1WopZQtotby+1de6aXTUY0LkwbL/dma3X1/Ydo4rJCAB6lUirgWcvHYFLx/QGoJ8YGH0qu6u9utf+YBA5DgemDOmKvh0LsLdGf5xh0oGsUAhK7frfnnI4HrvwKHV/bk7LPr42ryGVoaeKQtAWXTMzA5lx+j+/x8bKurC6Og3N9grcZcMKQQk7BYBCj/QdKi1047aJ/dGnNB957lAosrGjmhF/QMDllCKWtuyvx5i/fo1abqfJpBlZoRByc5yoePgMXD+hn+5hrlUOsZAu1nMlFt6YX2Dk+uP7Wh7rWpyre223NWRWrBDksFMgtAIo9LjgcBC6tsvVmeoOmZSy0OYmVNU3I8fp0PkkDjWwQmDSi8zoABNHtPPhWJ3Nj5w/FCUFHuQa4tI/W7ELZw7rHgfpwlm1sxrr9tSistaL4rwcXDamj3qsWVOyotDjsixfPayndVnsDvluXUc4q5nuyz/oQ2yzYYUg+RCkv7VS6VQpZeJ2OdT7Xd3gM81JaQ4Eketwqs749xZtw3kjQr03suEeMplF1ikELT3a50UfpOGiY3qr21OHd8cny3YBADbsrcOW/fXokJ+D9vmx5TlE48ynf9C91ikEeWVQUVVvWmP/1EFdcOMJ/TCidwdd+WUt7fP1lV6NGbcAsKfaiwc/09f4z44VQlDNQehdUoANe+uQnyO9znE6VN/NUdO+ND2/yR9Ebo4TW+SGOF5fUFc9l0tYMOlGVpiMzChwOzFUTjBrCaM1oadCCEx8bA6GT5ud1IStZn8QQgg0NAdMi6l9uWYvRvYpiVjttVOR3sm+eZ+9bl5eXwC+QBAfLtnRZpPUfJoVwuO/OArPXTYCveWwXLfTAZ/f/HP/8YwjAZj3qdBmPa/eVZMQuRmmpWSdQiiWZ8RnD49fCODSbaFkrkQ7mas0bTKb/MFWP1QK3PpForbEs4JZWW2vP4Dn52zCnR8sx/Tlu1olQ7ri8ws17LQ4LwenD+2mHstxOUz/1h/dPE7txa2YhPJl5/PxAzrqwli1eR0Mkw5knUIY3L0Yr15zDP581qC4XXNeeagERKLNACP/8pW6vb+uyXaY6Hd3TTTdX+Bx4fAu4eGnCrsONeL298LNTV5fEPtqJeVk1hugLeALBi37auc4Cc3+YFjV05F9SuBxSQpAWSEoZqJbTuwPtzMUmfTyD1vC8hcYJpVknUIAgIkDOyM3xxl9YATIostCMgve1Xn9aje4Ry8Yhu6GiKGbTzxM3e5dmo+/XzAs7BoFbidm3H48Nvxliul7/N9HK/DztvBCeV5fQA2b/H7j/pZ+hLSmyRc0LWwHSDktXl9At0oY3F0qV6Gcoxyrkc157fJcyHHpvzdVhqqoDJNKslIhxIP+JkldQHIVgtcfUMNE+3YsCHNo5xmU3oWjeuEWjZIAgHyPCy6nw/LB12ThPPb6Algnmzxmr9nbIvnTlbomP/74v5XYeahRl42sJS/HiUZfAAFNCY9HzpcUrrIiUL4LNbLibJebE9aS9YHpkrP+UENzm/XFMJmDLYVARJOJaD0RlRPR3SbHPUT0vnx8ARGVyfvdRPQqEa0kouVEdKLmnJHy/nIi+idF8nymIaMt6hklVSH4gurs3e0MVeXs10nqAmcWeWR88BdokquO6lmMEw7vpDu+sELf++DL30yAx+VAZW0T2urz67V5W9TCf9r7oyXPLa0QtGWvh/SQghSUezx/cxXKK+vU/svt8nLC7v/nK3djy/56DJ82O2XFEhlGIapCICIngGcBTAEwCMAlRGQ0wF8L4KAQoj+AJwE8Iu+/HgCEEEMBnALgcSJS3vN5ADcAGCD/m9y6j5IexNOH8JimJaP5ewXU8gldi3PVGHkFs05xxrBS7QPK5XTAH7SW/4ObxuHwLkXoWOjB/tomHCv3lrZaLWUqWhNYnts8Mjs3x4mgMO86p9zTv85Yh5OfmIsarx9EQJHHZerzqaiSIrva2kqLyTzsrBBGAygXQmwWQjQDeA/AVMOYqQBel7c/BHCSPOMfBOBrABBCVAI4BGAUEXUD0E4I8ZOQ1slvADin1Z8myYw01LwH4rtCeEbTtN2MxuYAAvI0vTgvBy6DOcLMH2p0kmpXES4HRezL3LtEDrl0OeALClURdG2Xa3lOJrJgS2hVZFbcDoDqg6qXS32coYlAMnZYq/X61AznZSb+GKesuINtdcnFZAx2FEIPANri+jvkfaZjhBB+ANUASgEsBzCViFxE1BfASAC95PE7olwz7Xnn+jHq9slHdgYQameZDA7IDVcA6SGUIz/claJ13YrDE+8cBjOS9uG1YMsBLNxi3R4zV46ecTkIny7fhXv/uwoA8EN523IqTxwYMpsZq5YqKPdNqf006YjO6jGjWejVeRVqnkjA5KGvrORYHzCpxo5CMJsiGb+6VmNegfSwXwzgHwB+BOC3eU3pwkQ3ENFiIlq8b196NRbxuJw4spsUWaJkEJuZjKrqmvC3GWvhDwThDwTx7frKuLz/t+sq0eSTzEZEhOI8Kcdi4sDOeOf6MbpZq4LRrxBLZrVHLgZoXIm0JarqmnQrBJ9FWKjy0D9YL/kHtOY6sx7MCueY5L8oSiIZXrSVO6pN80oYBrBXumIHpFm9Qk8AxkwkZcwOInIBKAZwQDYH/UYZREQ/AtgI4KB8nUjXBAAIIV4E8CIAjBo1Ku3mUG/8cjSWbD2oJryZKYQ//m8Vvli1B+MOK8VjX67Hqp01ePu6MRjfv2Or3ntXtRdeX0CtuqnM/h1EOPYw82sr5olLRvfGyD4ddM1+bjrhMPxr7iZ4fQHTsFzlQWdlRmkLXPfGYp2dv12u+U9EmdVf98Zi3WsAulwDI5OHdA3b96mc2GcVyhwvlm47iPOe+xElBW4cqG/Gtcf1xZ/OjF8+DpP52JnqLQIwgIj6EpEbwMUAphvGTAdwlbx9AYBvhBCCiPKJqAAAiOgUAH4hxBohxG4AtUQ0VvY1XAngk3h8oGTTqciDyUO6quGE8zdXhfkRlASuRRUHsGqnlFkcS6XLGbcfb/kQbvQFVFOOMkJEqMfarb1k7x/WsxgXjOypK2vRRy7LcEDTZrTQ40LXdrn4/vcT1bHGktkAwhK0MpWtVfqGR09dfLTpOOMdcOlarFr/rMwivyprk5OLcN5zPwII/X2NBQsZJuoKQQjhJ6LbAMwC4ATwihBiNRFNA7BYCDEdwMsA3iSicgAHICkNAOgMYBYRBQHsBHCF5tI3A3gNQB6AL+R/GYsye35h7mbUef146Nyh6jGlKqZWCVjF/StU7A/VFBrUvZ1cTC300O1Y6EF9kx9eX1Dt6zCsV3t8sGQH+pQWWF73jKHdUHK9G+P6lYYdKymQzEeb99Wju1z4r1txLvp3LkSvklBrzaUmjtFDDT50LW5dsl86oM0crnj4DMtxxkK52ge9VXazFUpYamYFXjNtEVvVToUQMwDMMOy7T7PtBXChyXkVAAZaXHMxgCExyJrWKOUKAIRl9iqRKNqVg9ksW+HT5bvCqpMaZ5bVjc3wBQT21Tap64HLx/TGmL4lOLyLdctQimBO6iJHC13+8gL1Ybixsg4FnuhfE6vS25mG3ZLUxpBerQ8hmrI3ojQ14jIWTKppu97BJKN9CBiLnjXKNumNlXXqvkiOPW2xvOMHSA9vY6luJTz0p81VqpmDiCIqg2iUGBzMimlh2fbwFYGRtqIQlKitYVEq4RpDRJ2aJUOhiQL923lDw/YpKIrgx01VlmNaSzU342FswAohAZRrHvxA6OGtfbBGKkqnVRZD5ezXQXI0k4KxBEU8UKKUFKzaaZrlX9S3EYWgEM1607WdXkFHWvEBCMsAt2Lzvrrog1qA3U54THbDCiFOdDb0FThY36yuDMxMAXe8v8zyWms0Ja0Ve/QDUwfjwXOGqP0LjtAoCG0MfGso0kTUCCFQ32T+EOlu0liorawQ7DLusFKcqMlXMJqQjH0mzKK25t51Yti+SY/PjY+ABqwy0NlMxWjJ6o5p8cSY8HX0g7MBSH2bY31Yap22iu+gKDcHV4ztg4kDO2HD3lq1sBwAdGnnCbtGS9B+hpU7q/H7D1cAAC4b01s3rndJuEJoc92/bHh4x/YrxZz1Um6MsWxI344FanQZEHLYa4nV19AaLntpgen+mkYfOsiyzVi5Gx8t2YGXrz4maXIx6QWvEBLMnR8sb9X5Rlt1zw75mHREF10Pglh7Q0fiktHSw//sZ+apSqfMELWk9F12Ogj/uGg4AMDXxhSCnYCfLZruckanfxQLEoDYo5FagzGcVkFbnO+Wt5fi63WVXHU1i2GFkOb8YNFrQFva2iy2vaWYFao7bbA+meqwjtKY5y4bgWPkxLZIRfHSnZd/2IJJj88BEPIF2KkrNHttqBid0YdgVlhQYeLATjiqZ3FUv0OiuOu0UOCfWe2qRHf9Y9IXNhmlIWWl+aiQZ3SLtx40HdOzQygv4K35W3H/2YPj8t5ms8NeBhPRhaN6YkCXQhzduwMqa7wAzB8smcKDn0k9CfbWeNXVT51Jj2ojXdrlqpFYRh9BJIXw6jWjAejt971L8rHtgPksPt7cOrE/urfPxW/eXw6/WXtUX1AXRs1kD7xCiCPRQhXton243nSCeTTR+SNCNXH8cXQMzt8cXtzO2KqCiHB0bynSSKlrZPZgyTS2ax7ItTb8Pv83OTTTNtYvMvqUzNCu7JIVBfTgOVLqj2JmNAt/buKIpKyFFUIcybUxq7ruuL5h+7y+AG59eym2yauC5kAQXdp5cMawbrqHjpZE9RPKs2gIY4XiTI2nUkoVWlPJGIsGSFoO6xQyr2lXbIA9H4IWrQP667WJ64twuRwgsFde2Zm1P/VadMlj2j6sEOJI/wjN6hX6dirA7ZP6AwiZZ37aVIXPV+7Gn6dL5aR9gSAmD+6KZy8dYevBf834spYLbaCboS/z1OHdI45Xajg9GqWZTyZQo3HUP3bhUVHH9yrJx9I/nYKKh88I8+MU5eZYnBWda19f3OJzo6F8nxZXSKbIdxdKneEqa73qGLvZ2kzbgxVCHLnvzEE4x/AAPW9ED90DO9flhEe2NyszUuWZr0yyff5gTBEoF47sFX2QTX590gDd646FkUNaFcdoMluHxhOtz+T9RVLbj9+ecrhp3oAZZuGkADDt7ME492j7LT4+vuVY3D3lCNvjW4viF1LKlfzv553qMV4hZC+sEOJIbo4zzLk7pHsx/nzWYN0Yxd7c5A9i2fZDeFbujDZ3wz6U3f05fAGBHBsx6koHs0jVNWNFW7eoXa4Lt08aEGF0fCOcUoHWTPStnFMwNA6+oA4Fbjx50XB8cut4rJl2WtTxI3p3sOzfnAiuHFcGADhzmNQz40B9aHXk5RVC1sJRRnHG2HCmq8EE43E51BVCky+Ic56dF3aN5oC9FYISGpkTxzwEADhjWDd8vmI3/nzWYLXPgxVEhJF9OqgVVzMNs4S6SA1uYuWoXu1tj7XjiG4JOw81AtBnoiuTCF9AoLLWi3/N3aQeayulzJnYycxfcYbw2IVHYbIcw68UqfMFguoDJ9IPz22jCY1i7Yi3f1nJcbCbW+B2OjLWZGSWUBdPhRCNT287DrPumAAAWK6pdaW14/sCQYx8cDY+WbYz7Hw7XC/7JM46KmTOVHw/vkAQlTX6fgxeXxBCiDYROcbEBiuEBHLByJ7qrE+xNTc0B1Tn5T+/3mh5rp02le3l2Xu8M15zYowc8uQ4MrZ0hdlnjOY3iSdDexZjYFepQq22B/aMlbvV7ZpGH6rqm3H/9NUteo/9ddIDX/uAV2tkfboGGytrdeOb/AHc98lq9L83o1uUMC2AFUICuPrYsrD8gSvHST2XR/ctwY6D0hL+gyU7LK9h5yH/0lWjcP9Zg8LMUq1FiVH320w2y+gVgsksOFKDoURy4wn91O1HZ4ZHbbU0sFdZQWqdxdrv158/0Ssary+IN+dvBZC5wQJMy2CFkADuP3twWMTIyD4lqHj4DPQqyce4w8K7lRmxYzLqVpyHq8eH5zW0FsVRbLcZu9vlwLo9tXj9x4q4y5Jo7Cq9ZJDvduFaOU9ld7UX1Y0+/OKFn9Ss9Zai9NHWllzXtmStMWRlH2oItVCdvSZxORFM+sEKIQWcOqhL1DHJLHxmRHFW240gUqp2/rmFJo1Uoi0SmA600+QvzFy1Gwu3HMD5z0u9kGPpw63gDwSxq1rKMdDWMLLKbyHS35N6i54YTNuEFUIKsJNslkqFMFHur3BMWfRsXQAZXffmcouy0KliRJ9QVBKZ1Fx9b+E2VNZ4sWTrQWzYWxt23Mjzc0LRQ4qvwop+HQvgdjrUqCQgvM8H07ZhhZCm2MlDSBQTB3bGhr9MwZAe9uLxtVE5wQwrYWGnZlEyOX5AqOnO7z9aoTs2vFd73P3xSlz84nyc//yPOPXJ76JeT1kd2OHpS4+G2+XAx0tD0Uz1TQFdjScrvL4Anv223LaZkUlPWCGkKc4E1SqySyzNW7T26EAG1dJP17r/t59kngyoyLt5f73pcTNi+Rq5nY6wkNtb31mK4x/9Nuq5L363GX+ftV4thcFkJqwQ0gwlXyGTHqxVdSEnZCa1ZFy5s1rdvnFCvwgjk8uI3ubJbMt3VJvuj0Qs0wqX0wG306FLYLNLrVfyO0TqFc6kP6wQ0owOcqZzJplePtbUwcmkqqfah9fNJ0phwqP6dEiVOCr57vgVEIjUl8GIkwi7qr2oNekFkUnfR6blcOmKNOLh84ZiwRapH0EHBBo4AAAgAElEQVQmzbT7dSxQzRiBNArjjIY25LTA48L028ajrGNqchC05MexplEsJqNIXeJ8wSA8juhyZdL3lgmHVwhpxMWje6szukwyGc36zQR1O5NaaWpLQTiJMKxne13YZ6qItSdFJCLpg/KHpqCfRgEWRjAVafs1ROLvbaAMejbDCiFFzLj9eBTnhR4+Snhf53bS//GcJSYabYhsJs0QtZniiSos1xJi+dtHM+XUR7Dpu5wOdQVx95QjIpbsOO6RyI7lf3+/Rd0ur4weDsukJ6wQUsSg7u3we7kb2lvXjsH8e04CANxx8gA8dO4QnD6kWyrFixnlYVK+ry7FksTOsTYyx5NJfo5+pl7x8BmWY42JY58s24n+f5iBVbLDXHEQH27RvKlSnvn37xS9uZNdTn4iejgsk56wQkghl47ujVl3TMBxAzqqM1SPy4nLxvRJqxmrHZrl6pxfrs68UgeP/yJ6d7RkEovJqM6QR/Hr95bBHxS46a0lAKA6iD+46VjT85Xj2+Rcg3RwqjOpgxVCCiGiqNmjmcIdJx8OALaT2dIJ44w81WhzQD771XERx76kMdVo8yqGyn+HD2WzWI5FbSwlzFnpjT2mn3l2+rAoTYPSbZXFtAxWCExcGNtPeiAUejLD96EtBR2tCVAqGdStXcTjeZpWn9rub4UevZKzqkt156mS2XKCnCGtKHYj0YoAZpLviLGGFQITF5QHzj+/Lk+xJPZYtasGgPXMOdW8dOUofPO7E1TTYSeLmkJ9NVFC2p4UgaDQrRis6k0d1as9Kh4+Qw23taqh5fUF8PTXG1FZa14KI1LIKpM5sEJg4oLyHFmzu8b0uC8QxGvztqRNF64G2Rl70TG9UiyJOScP6oJ+GkfvV789AXPvOhFPXTxcN65R03Xvx/L96rYvKFSHcUv5z43j1O3N++vx+OwNOP/5H01LfpiV6P6xfD+WbD3YKhmY5JJexlMmY3Fa9HV+6fvN8AUEft52EF+u2QsB4JoE9HCIlYYm6UF64cj0VAhGivNyUJyXgy7t9M2QtG1YV+8KKeNFWw6oZhwlms0uw3oWo7KmCaP7lmDa1MH43887sXSb1N5z+4FGvPT9FlxvKPVhzFNo8gdwqVxJNlKUFJNesEJg4oJZMT5/IIi/fL5Wt2/nwcawcalAsbcrzeYzhdwcvemnUZNnMFITIbSnxqsqi67tYuuoN/22kCP7ynFl2FrVoCoEAPhx0/4whQBIoa1CABsr6/DmT1tjek8mPcisXwOTtmhtyP/+bjO2VtXjrzPWhY1T+vsCwNrdNSmrOKrMnlNdVbY1uJ0ONGhWCO8v2q47flBuqNPa3houg59F24pToWOhB6PKSjB1eHcACJsIMJkBKwQmLmgjXB6asRYn/H0OXpm3JWxc9/ZSI/l55fsx5anv8eis9SlRCqpCyLB8Dy3NgSC2aWz3X6zaozs+f3MVgNYrBG1oKwB4/eHZz/5gEC4HxVQ2nUk/+K/HxIUBne1lupbKGc1KMbzn52zCW/MTb17YtK9Ol8SV6QrBJcv9+crdYceuGV8GIFRXyNXKz2gMKd15sBFLth7QjwkIuBwOuCx8SUxmwH89Ji4QEUbbaLnZaNKjNxmRKCc9PhcXvfCT+joQY9/odOK7uyZiwR9OUl//vE1//4yz9NZ2MetTmq97XVnbhPOf/0m3zx8UcDkJhxqaYaSReyRkDLYUAhFNJqL1RFRORHebHPcQ0fvy8QVEVCbvzyGi14loJRGtJaJ7NOdUyPuXEdHieH0gJnUMjZLNCgBvLwjvqJWsMh3aKJxMXiH0Ls1XV1pAePmKzfv0HdV8rUwae+6yEab7tYX1/MEgnA7Chr3htayUMhpM+hNVIRCRE8CzAKYAGATgEiIaZBh2LYCDQoj+AJ4E8Ii8/0IAHiHEUAAjAdyoKAuZiUKI4UKIUa36FExacO7RPaKO2S33+H3p+83qPiUENJm0BaeygtKUaExfaYXWs0Oe7ni/VvZ4GNy9GB/fEl4LSes38gcFchykVuvVMnfDvla9fzS+WrMXW2JoK8pYY2eFMBpAuRBisxCiGcB7AKYaxkwF8Lq8/SGAk4iIAAgABUTkApAHoBmAeeYSk/HEUsdoq8YZalWJ0xcIYsSDs/GFiZ08FsxMJpm8QjBy78crAQCHdS5Ex0JPWJhpPOpLjegdXvROyYwOBgWEsM5F+Y1FOYx4cd0bizHxsTkJfY9swY5C6AFAG8+2Q95nOkYI4QdQDaAUknKoB7AbwDYAjwkhFG+UAPAlES0hohta/AmYjMMYVdQcEHjlhy2484Pluv3Pz9mEA/XNuPntpa16v2/XVYbtawsK4VeT+gOQ/DeA5Nh1OqBLXovnx7tibB/d6yY52sgnN0VyOQk3nnBY2HkFCaxv1Vr/CKPHjkIw+0oZjZJWY0YDCADoDqAvgN8RkZLRMl4IMQKSKepWIppgcg0Q0Q1EtJiIFu/bl9ilJ5Mcvt+4X/e6vsmPaZ+tUStzKjwxe0Nc3s+0R3AGO5UVbp0oKYTzRkjzs4CQIn2mDu+uzsr7xrEl6IPnDMGZw0J9OprlFcJ5z/0IQLqXPdrnqbkIHtm5ncg+20poLRMf7CiEHQC0+f09AeyyGiObh4oBHABwKYCZQgifEKISwDwAowBACLFL/r8SwH8hKY8whBAvCiFGCSFGderUye7nYlLEgj+chKV/OgXXH29dnuLKVxbqXm/YG1uHrXnl+7FyR7Xt8WbtSJXIl0xWCLk5TrhdDtWWHwgKOB0EIsINE/qhV0kepk0dEtf31BbJW7b9EJ74cr3qrFfCW5XVl6IQPl1ufFzEjwIPF1uIJ3YUwiIAA4ioLxG5AVwMYLphzHQAV8nbFwD4Rkh2gW0AJpFEAYCxANYRUQERFQGAvP9UAKta/3GYVNOlXS5KCty494xQ3MGvJvXHf02ckgoLtoRi2u0kqV320gKc9cwPKLv7c7UzWCQcJo7jx+XVh9mxTCLf7YRXVm7+oFAfynluJ77//SSM798xru93yqDO6vZt7/yMf34Tqm6rvPdnKySfj1JmQxvdFW+0ZblTlfXeloiqEGSfwG0AZgFYC+A/QojVRDSNiM6Wh70MoJSIygH8FoASmvosgEJID/tFAF4VQqwA0AXAD0S0HMBCAJ8LIWbG8XMxacAbvxyND24ah9+dOhBH9+6Av5wTfbZ627s/o77JH7VXsMIVLy+IOqYoQvP41iZtpZq8HCcaZIUQCAYTHsI7eUg3PHPp0abHnHJG9AUjewIAktEiQetD0Jb/ZlqGrfWWEGIGgBmGffdptr2QQkyN59VZ7N8MIL36FjJxZ8LhehPfSBvtGT9fsRsQwBMX6b8e8zdXqU14tFwyunfUa974Zngc/OTBXTFz9R64WlnWIdXkuZ1qCeyAZoWQSIrzzBsKbZRNf4O7t8OHS8LrVh0ZpdlPS9AqhBqvL6z4HxMbmf1rYDIKuz/Wz1fuDuvQdfGL83Hta4vCVg75MfQfBoAD9VImrcMBHNYpfg7XVJGX40RjcwBXvbIQs1bvxbo9sfljWoJVs5035RIk32iiuiYdIZmYEjV7135PquVifkzLYYXAJI1ck1LTT15kvlA81Bj+4/56XaVu1gnEnuX8/BzJ5u31BZHvznyHZL7biYMNzQlP/tJirH6q8NylUkbzUE3ew+VjpRVcotYt/mBI0TRwiYxWwwqBSRq5JjPLgV3MzQhWkSnaDmEAkBNjMTVFCTQ2B0wVVKaRm+M0DatNJMZ+zQpThkohqedoMtaVZLVEhZ76NCsEr48VQmvJ/F8EkzHkmZh3rHoa75c7cI0y+B2Ms8DmGBOT2udL9u9GX6BN2JuJCBsrw+sHJRJjaQwj7XKle1zkcYWFosYb7QrBG8UsxVFI0WGFwCQNj0mtfKta/TsPSZ3VThvcVbe/3lDIrTlG2/R/f94JQJpN5rUBhfBdEk1FCmYrPS1di3Px6W3HYdEfT1bzPBLVS9vnDz3kmyKsELy+APreMyMs+ZHRwwqBSRpkEvNvZY9Wmr0YSy8bY9r31cXWSH6FnNDm9QVMVyxMdOz4bYb2LEZujlNdISTMZGRzhaD4nu78YDk7nyPACoFJKdEyhXuV6BXCxkp9FM07C7ah7O7PI9qPO+Tn4KJRoWT7Qw3NqKhqwEaTUs2Zxg0mvY3TCSWsN2EmI5s+BO24o6Z9mRBZFA7WN6MqxolKusAKgUkJV4ztg+c1dfa7mJRNBoDeBoXQpci8YfyhCLM+f1Agz+1EgduJs47qrq4+1uzO/MK7Ewd2jj4oCRh9PQoJXyFoTFG//3CFpWkqmUXwjn5wNkb+5aukvV88YYXApITLx/bBlKHd1Jj2/iYtOItyXWG1aoxRRgoirN5iiKCcsNW1OBfBoMA9crno646zrreUKVhF/CSLq48tAwCcONC8zpiyAoxXBFCN14eFWw5gpqzUjUEFj8/eYOo8jjX4IB5kohObFQKTEpQHRUmBG2/8cjSeu2xk2Jgzh0lVM7WNd6xizY2JbLpjctE3t8upS5CKR5+AVKMtLV2U68KHN41L6vtfNqY3nA7CGfLfyoiycnvwszWtfq///bwTw+7/Er944Se1C1uTT/+gf37OJtOcjJpGfTDC375YK2XFxxEhBJ79NlTb6S2T7oDpDisEJiVofQcTDu9kWg7BLTucn7xouLrvtR8rTK9nlgnrDwTx2YpdqkJYu7sGX63dqx5vC2Gn2hXCzDsmYJSNvtbxZECXImz66+mWZbY7FroBSHLuq23C7/6zHI3NAfzz643qSs0O6/bU4I73l4Xtbw4E4TZEqhnLqwPAJf+er3v9wtzNuPWd1vXZMLK1qgF/n7Vefb1yx6G4Xj8ZsEJgkoqSd2BWc2d4r/aGsaGv5/L7To14XbPw0xe/34zb3vnZssZPhte1A6Av/9y92Ny/kkoURXHO0T3w8Bfr8NHSHfhsxS48MXsD3l1ofwb9z683hu1btbMadV5/WDjzyz9ssX3dePoWjEF0ySgjEm9YITBJRZnNuU1yEv5363hUPHyG+nqGpnWmx5BV/MDZg3WvjdFHAPDozNBszay94wkWdu9MQlvLySysN9VI/RmkB+9HS6UcALO/fSTW7akxNRWe+fQPeHP+VnhyHLo2rKcO6qIb19Bsncm9dOvBmGSJhDGSakUMPTvSBVYITFLJkR8GdhrT7Kr2qtvGWWCZwUTx6/fCzQlazPLfrIq0ZRKpUgIPnTsEN54QPeSViOB2OnRO3V2HvBHO0NPQ7Mfkf3yPOeutE/DcTgf+e8t49XWpbKZSeCeCLT+e0U/zysNNVZkGKwQmqRwtm4ViLdNMRDql4PUF8LfzhlqON87WrBrAMy3jsjF9cM+UI22NdTsdOiXwyMx1tt8nUjixgifHGbFz2l8+X2t5LJ769E+frI7fxVIE/0qYpPL0pSPwnxvHoX2+O+pY449V6wQu8rhw8TG9YEVVvT4xiEjqCazQId+8pj8Tf3JcDstihdFCM+2EqxpXj81+gVU7q7H9QEPUcxOVMJepsEJgkkqhx4XRfe1FwmgT1wD9D/+YviUgIkybOth4GoBQ43eFQFCgZ/tQUbYH4txrOJU8fuFReOGK8LDddMEYBaTFFyFcGJDKlEdD+V4cP0BqF9ocCOLMp3/A8Y9+G/XcWGthtXVYITBpi7EfsBJ1dNvE/ur2lePKTM/dcbBR9zoYFDrHdKa3ztRy/sieYUUA0wltroSRaFE+TX47KwTp+i9cMRI9O+ShWXOO0aF8x8kDdK8TrRDKk1yJtrWwQmDSlqJcvVlHqYBqfJYrXbki4Q8KXXXTtqQQ0p1N++otj0VXCNEf2ErUUr7bhQ75biyqCEUOfbW2Ujf2jpMPj/n6reHkJ+Ym9PrxhhUCk3F8b4jmUDKOje01tQSCQueDsCq7zSSXaCUlzHwIT/xC32VP62tyOUltkwoAr2hyEsxyWdhkpId/FUzG0d6Q1VwkR5jURYg3Dwj9CsFO2CuTeKL5EOZvPqB7/fJVo3SlTIz8vE2fHbxse+h1sRxI8MrVo/DUxVL2ux2TlF26pWFiYKywQmAyhtICKTLp3jP04Y7dZWfxZo1pYmy/EhR6XDh/RE8A0upBu0Kw6sPAJBdflBn6v+Zu0r0+6cguYbkXjhhjRycd0QUTZTNjPE1Gh3UKL9CYabBCYDIGJRmtgyFk9YhuRQCAiv2SQggGBeZvPoC6Jj+OlI8pJbAV2GSUPI7oWmR5rKWlI7767QT8xuAPMOOyMb0BAGumnabbr0Q+xVMhaM1PV47rE7frJpPU1s5lGBM+uGkc9lSHZ7O+eMVILNhyAKWF+t4JiilIWf5rf+TKqiAvx6krBMcmo+Tx+e3H47A/zDA91tKy1P07F6G0UDInaaubFuW6UOsNmQ4DQYHORR7ku/WPOkUhxNOHoP0sZsUaMwGeJjFpxzFlJTjrqPByyqWFHpw+tFvYfiUO/f8+Wom5G/bhyPtmqscuGNkT1x7XF1ePL9MpAaMfgkkcTgfhiK5FYSGfQHhZaiPFeTmqj8iI2erilauP0b1evqPadJzDIZXUSNQKoVdJPm4+8TAQZVZfBFYITMbj0fgGrnplobo9tl8JcnOc+NOZg9DRsKroU2perplJDDPvmIA7Tj4cb183Rrd/+Y5DqKyxrm3UtV0uhvdub3pMec6ePjSUgzGit75z29rdNThoUf7C7XLEdYXgCwQxeXBXvHjFSFwwoicKPS4IkZrmPC2FFQKT8RhLFyicM9w6GoVNRqlhfP+O+OTWUCG6h79Yh9F//dpyfFAIS3/PMXLvByVwAIjt7+pxOdAcsBdlNHPVbpTd/TnOfW6e5ZjmQBCeHAdOHdwVDgeppkxvMysEhkkaVklmsZZZZpLDUb3a4zlDWRIrAkKoPTSMDO1ZjJX3n4qTjuxiejwabpcjrOOaFXd+sAJAeFirFp8/qFNeShCDVdvXdIR/MUzGY1UC2mxm+cmt45PeZpIJZ+LA6NnlgGQWihQRZsxmB6QIpD+cfoT6enz/UtNzpRWCPYXQQ1MHa/ryXRBCqP8UmgNB3SQkVy6VEq9+0smAo4yYNoHHFe4gNDMfHNXL3B7NJJfcHHtzUaXb3elDu+LCUdbVbbX071yEao2z+qUrjzEdF4sPYf3eUAOm29/9GT5/EL/7YDny3U6smTYZgORU1hbyU0xGmbRCYIXAtAnOH9kzrBFKBgV3ZB12G/sEhYDDQXju4tiquY7s0wHz7p6Ebu1y4bAwKXpczhbP3lfulLqhaTu5ha8QMk8hsMmIaROYlVgOskbIGGq95pFAwaCIORNZoUf7PEtlAADt8lyo8UYOe7WSbXd1qJqukv9itULwmrT/TFdYITBtgjEmPRZYIWQOVg/moACcCWoT2iHfjYOaQnhWmFVr1Yayzl6zF4GgQNDg78hVEyY5yohhksoUk4S10gKPyUgmHTnUYP5gDgiBRHU/LSlwo8qGQjBLLFu4JVR0r7TAo/oitCYjpf9GPAvoJRpWCEybQRtZ8vQlR1tGlzDpR7VF8lhrTEbR+G7DPlQ3+kzLpMSKqUJw8QqBYVLG5WNDBcVOGNjJtuOSST2HGi0UgkicQlA68u2NkCkNANEMj15/AE1ygpteIcgrBJu5DukAKwSmzZDr0lQzTZSdgUkIhyxWCE3+YMKyyk+V245Gq7ga7d2bfEG1r4Nbk0SnKAQvm4wYJvloI0qssluZ9MQskmfmqj1oaA5g077E9CUulHs93/7uz2rpdDP8ETrxAZKPwNyHIJuM2toKgYgmE9F6IionortNjnuI6H35+AIiKpP35xDR60S0kojWEtE9dq/JMK2BaxWlP2cOCwUCBEwct9+s2wsgcrmI1qCUxN5V7cWJj82xHPfZ8l2m+5Vy6k2+oOo4djtDq1RlhVDX5Mf8zVXxEDnhRFUIROQE8CyAKQAGAbiEiAYZhl0L4KAQoj+AJwE8Iu+/EIBHCDEUwEgANxJRmc1rMkyLYf9B+vPMpSOw4S9TAJj3w1bKUtQ1Rc8VaAkFbnt5ua//tNV0f1GurBD8Afz7O6l386KKUPSRy0FwEPDU1xtx8Yvz8eXqPa2UOPHYWSGMBlAuhNgshGgG8B6AqYYxUwG8Lm9/COAkkn6RAkABEbkA5AFoBlBj85oMw7RxlJXcY19uCDtmLFkebwo8Tt3rsrs/jzj+oXOH6F63kxXWnz5ZjU9XSKsIbVc+IoJWz20/2Ih0x45C6AFgu+b1Dnmf6RghhB9ANYBSSMqhHsBuANsAPCaEOGDzmgzDtHEiWfaUxMLrj++bkPcuMGm8s2TrAZOREpeO7o37zxqEbsW5AEIrBCAUdnrJMb0tzw8E09+XYEchmP3JjOs7qzGjAQQAdAfQF8DviKifzWtKFya6gYgWE9Hiffv2mQ1hGJWf7pmEj285NtViMDaJZNpr8gVABPzh9CMT8t65Oc6wfcu3V+teK6asy8f2BhHh6vF91WQ27WpAIVLJ9b/OWNcacZOCHYWwA4C2zGBPAEYvizpGNg8VAzgA4FIAM4UQPiFEJYB5AEbZvCYAQAjxohBilBBiVKdOnWyIy2Qz3YrzwrpmMZlJoy+AXJczqf6gR2fpH9oPfLoagD4sVlkNNJrUKMr0Hhx2pF8EYAAR9SUiN4CLAUw3jJkO4Cp5+wIA3wgp33sbgEkkUQBgLIB1Nq/JMEwW4/UFbZfJjud7KqzcUa06lL9YFe4QNgtHjaYQ1u6uaaWEiSXq3ZZ9ArcBmAVgLYD/CCFWE9E0IjpbHvYygFIiKgfwWwBKGOmzAAoBrIKkBF4VQqywumYcPxfDMBlOoy+gVgxNFLPumGB57OlvNqrbhRp/g+IIFwCeufRo3TlmVXe1PDk73HmeTtiKuxJCzAAww7DvPs22F1KIqfG8OrP9VtdkGCb76NE+DzsPNeKxWetx52kD1f0fLtmR8Pce2LUIo/uW6IrVKXy5Zq+6/cj5w9TtZy45Gje/vRQOArprOqkB0RMiJw/p2kqJE0tmG7wYhsl4OhZJ4aXPfFuekve346FQIouAUBVTBxECBrORlb9DiZRKVF2meMEKgWGYlLJ8e2Iyke3y0LlDMXlw+Mxdu0+b+a5kIzsI6NexIOK1R/ctwYDOhbjq2DIAsN2y89Qn52LyP76zNTaesEJgGCatEEJg7obkhZj371yIf10RatHZ0CxlRlfVN6n7tI1vlLqJRITSQg/KSvMtr/3u9WPxyW3jVd9Cc5RCeoD0+TfsrcO6PbVRx8YbVggMw6SUcf1CfSu+27APD362Fle9shAAcM34sqTL89ny3QCAytqQQnBpfANK2SVl0TCmryT//HtOCruW00HId7vU6KNolVUB4MXvNrdI7nhgr5gHwzBMgtBWKr9SVgQKqYjrb5If2v5AyD+gLaeuZFAr/oBp5wzG9RP6oqvGz2BEWWHYMRm9u3Bb7ELHCV4hMAyTUiI6WpPYFvulK0cBAHrKkUN+TakJp2aFoMirhMR6XE7071wU8dqKQrCzQvC4EhtqGwlWCAzDpJSJAztbHptuUXo6EXRpJ83wlYQzbQRRjsapPLZfKW48oR/+dv5Q29dWwlGbA9E1nFlJjGTBCoFhmJQSyU/QPt+dNDkUP0EgGMSr87Zgf12z5ljoUel0EO6ZciQ6F1mbiIwQEdxOhy2T0bIURl2xQmAYJqVEqlXUPYJdPt645FWAPyjwwKdrdMfi0XCpORDEv+ZuavV1EgkrBIZhUs6D5wwx3e9IYuc7pS/yFyvD6xYlsyWrNrdBmHSSSySsEBiGSTnDe7Y33W9WojpRlHWU8gnKK8N7OMe7Jeuy7YewTxPWqqVTUagxUINJRdVEwgqBYZiUM7RnMdppGs58cNM4nDqoC+6eckTSZFB6LK/fW6tTAM9dNiLukT/nPDsPkx6fo74+WN+M5+dsQjAosGlfSCH99+edcX3faLBCYBgmLXjrujHqdnFeDl68chR6GIrHJZriPKktphJh1LHQg9OHdovLtft1kkxBlTVeAECtN9Qr+o+frMIjM9dh7sZ9Ome218crBIZhspChPYrV7WhlpBOFselNMI42fKU20pKtB8OOKcrhsVnrdfsP61QYt/e3A2cqMwyTFmijjXJS1XnM4Cro3j5+UU5KyOnNby9V99V4fRh2/5fq69W7pAY6TodUSfWa1xYBACoePiNuckSCVwgMw6QdyYzq0ZJrUESvXj06btduMslB2FbVYDr2mDJ9G1hjme1EwQqBYZi0w+NMTbaudpXidJAu4qe1uEyUXJPf3EfwwNn6MFy7ZbNbCysEhmHSjhxXalYI2hy5SUdYl9RoCU6TBLyb3lpqMhJhhfKS5VxmhcAwTNqRKqfyEV1DReqevGh4XK9tlstglYtg7CVtZm5KBKwQGIZJGwZ3bwcg/olgdvnX5SPx8lWjsPGhKSj0xDfmxq/xA/x+8sAII8N9KFampXjDUUYMw6QN71w3FhVV9RHrGyWS9vlunHRkl4Rcu6bRBwA4Y1i3qIXxjJ9/2fZD6FMauV1nPOAVAsMwaUNxfg6O6mVexiLTeWDqYJxweCfcd+YgTB3ePaZzf/3esgRJpYdXCAzDMEkg3+3C67+MHsaq9WNoEUIkfOXEKwSGYZgUo+3HfNdpkn9BW/UUSE6hO1YIDMMwKUbrQ1eqZWzeX68bUxBnJ7epHAl/B4ZhGCaMru1CjuX2+W61kJ8SjbT4jyejKAlKQAv7EBiGYVJApyIP9siVT90uB4b2KMbOQ41qQb2OhR6sfOA0bNxbm7S+EKwQGIZhUoAx10LxFxsLrA7oYu5kTgRsMmIYhkkB54/oAQB4R+4DoSoEJLdtphZeITAMw6SAy8f2wYWjeqnmIOV/s5pHyYIVAsMwTAogIp1v4L4zB6FzUS5OGZSYTGk7sEJgGIZJA9rnu5PaQ9oM9iEwDMMwAFghMAzDMDKsEBiGYRgArBAYhmEYGW3dO68AAAVSSURBVFYIDMMwDABWCAzDMIwMKwSGYRgGACsEhmEYRoaEsZJSGkNE+wBsbcGpHQHsj7M4iYJlTQwsa2LIJFmBzJI3nrL2EUJ0ijYooxRCSyGixUKIUamWww4sa2JgWRNDJskKZJa8qZCVTUYMwzAMAFYIDMMwjEy2KIQXUy1ADLCsiYFlTQyZJCuQWfImXdas8CEwDMMw0cmWFQLDMAwThTatEIhoMhGtJ6JyIro71fIAABFVENFKIlpGRIvlfSVENJuINsr/d5D3ExH9U5Z/BRGNSIJ8rxBRJRGt0uyLWT4iukoev5GIrkqirPcT0U75/i4jotM1x+6RZV1PRKdp9if8e0JEvYjoWyJaS0SriejX8v60u7cRZE27e0tEuUS0kIiWy7I+IO/vS0QL5Hv0PhG55f0e+XW5fLws2mdIgqyvEdEWzX0dLu9P/ndACNEm/wFwAtgEoB8AN4DlAAalgVwVADoa9j0K4G55+24Aj8jbpwP4AgABGAtgQRLkmwBgBIBVLZUPQAmAzfL/HeTtDkmS9X4Ad5qMHSR/BzwA+srfDWeyvicAugEYIW8XAdggy5R29zaCrGl3b+X7Uyhv5wBYIN+v/wC4WN7/LwA3y9u3APiXvH0xgPcjfYYkyfoagAtMxif9O9CWVwijAZQLITYLIZoBvAdgaoplsmIqgNfl7dcBnKPZ/4aQmA+gPRF1S6QgQojvABxopXynAZgthDgghDgIYDaAyUmS1YqpAN4TQjQJIbYAKIf0HUnK90QIsVsIsVTergWwFkAPpOG9jSCrFSm7t/L9qZNf5sj/BIBJAD6U9xvvq3K/PwRwEhFRhM+QDFmtSPp3oC0rhB4Atmte70DkL3WyEAC+JKIlRHSDvK+LEGI3IP0YAXSW96fLZ4hVvlTLfZu8xH5FMcFEkCnpsspmiqMhzRDT+t4aZAXS8N4SkZOIlgGohPRw3ATgkBDCb/K+qkzy8WoApamSVQih3NeH5Pv6JBF5jLIaZEqYrG1ZIZDJvnQIqRovhBgBYAqAW4loQoSx6foZFKzkS6XczwM4DMBwALsBPC7vTwtZiagQwEcA7hBC1EQaarIvqfKayJqW91YIERBCDAfQE9Ks/sgI75tWshLREAD3ADgCwDGQzED/lypZ27JC2AGgl+Z1TwC7UiSLihBil/x/JYD/QvoC71VMQfL/lfLwdPkMscqXMrmFEHvlH10QwL8RWvanXFYiyoH0gH1bCPGxvDst762ZrOl8b2X5DgGYA8ne3p6IXCbvq8okHy+GZHZMlayTZROdEEI0AXgVKbyvbVkhLAIwQI42cENyIE1PpUBEVEBERco2gFMBrJLlUiIFrgLwibw9HcCVcrTBWADVinkhycQq3ywApxJRB9mscKq8L+EYfCznQrq/iqwXy1EmfQEMALAQSfqeyHbqlwGsFUI8oTmUdvfWStZ0vLdE1ImI2svbeQBOhuTz+BbABfIw431V7vcFAL4RkqfW6jMkWtZ1mgkBQfJ1aO9rcr8D8fBMp+s/SF76DZBsivemgTz9IEUyLAewWpEJkg3zawAb5f9LRCgq4VlZ/pUARiVBxnchmQN8kGYi17ZEPgC/hOSYKwdwTRJlfVOWZYX8g+qmGX+vLOt6AFOS+T0BcBykZf0KAMvkf6en472NIGva3VsAwwD8LMu0CsB9mt/aQvkefQDAI+/PlV+Xy8f7RfsMSZD1G/m+rgLwFkKRSEn/DnCmMsMwDAOgbZuMGIZhmBhghcAwDMMAYIXAMAzDyLBCYBiGYQCwQmAYhmFkWCEwDMMwAFghMAzDMDKsEBiGYRgAwP8DQyxBxuknH3wAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7f16d575b400>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.sched.plot_loss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "fastai has another very good feature called Test Time Augmentation (TTA). The dea is simple; apply simple augmentation on each test image to generate five copies  of it,  and then do the prediction for each copy. You can average these prediction to get a significant(1-2%) decrease in error.\n",
    " \n",
    "So we have trained our first model, let's see how well it performs on validation set using TTA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F2 Score without TTA:0.9281136719526527,   and with TTA:0.9307022849638835\n"
     ]
    }
   ],
   "source": [
    "probs = learn.predict() #returns prediction without TTA\n",
    "f2_without_TTA =f2(probs, data.val_y)\n",
    "probs,y = learn.TTA()\n",
    "probs = np.mean(probs, axis=0)\n",
    "f2_with_TTA = f2(probs, y)\n",
    "print(f\"F2 Score without TTA:{f2_without_TTA},   and with TTA:{f2_with_TTA}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "F2 score without TTA:0.9281136719526527\n"
     ]
    }
   ],
   "source": [
    "dd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('resnet34.weights.1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('resnet34.weights.1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "## do more data augmentation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preparing our first submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In our submission file, we need to place predicted labels against each image. Each image can belong to more than one class.\n",
    "\n",
    "`file_10770,agriculture clear cultivation primary road\n",
    "test_26732,agriculture clear cultivation haze primary`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you look at an example of predictions from our validation set, you will see our original labels are in the form of 1's, 0's, but our predictions are floating point numbers. So, we need to pick a threshold for our predicts to be included in submission files (0.66 for below example). `op_th` function tries multiple threshold in a given range and returns the one which maximizes the F2 score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.66928  0.0029   0.02886  0.0015   0.0011   0.99902  0.00014  0.00053  0.24234  0.02605  0.00069  0.00026\n",
      "  0.99906  0.22009  0.00423  0.00496  0.80653]\n",
      "[ 1.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  1.]\n"
     ]
    }
   ],
   "source": [
    "print(probs[0])\n",
    "print(y[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.17\n"
     ]
    }
   ],
   "source": [
    "threshold = opt_th(probs, y)\n",
    "print(threshold)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 28min 6s, sys: 4min 22s, total: 32min 28s\n",
      "Wall time: 5min 32s\n"
     ]
    }
   ],
   "source": [
    "%time test_preds, _ = learn.TTA(is_test=True)\n",
    "preds = np.mean(test_preds, axis=0)\n",
    "classes = np.array(data.classes)\n",
    "res = np.array([\" \".join(classes[(np.where(pp>threshold))]) for pp in preds])\n",
    "filenames = np.array([os.path.basename(fn).split('.')[0] for fn in data.test_ds.fnames])\n",
    "frame=pd.DataFrame(res, index=filenames, columns=['tags'])\n",
    "frame.to_csv(f'{path}planet_amazon_restnet34_submission1.csv', index_label='image_name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Ensembling\n",
    "Instead of training one model, we will be training multiple models and then averaging their prediction. This techniques is always employed to get more accuracy on data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 231,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_ensumble(nmodels):\n",
    "    models = list()\n",
    "    \n",
    "    for i in range(nmodels):\n",
    "        print(f'-----Training model: {i+1}--------')\n",
    "        val_idx = get_cv_idxs(n, val_pct=0.1, seed=12345) #use 10% of train data as val data\n",
    "        \n",
    "        data = get_data_ens(64, i, val_idx)\n",
    "        learn = ConvLearner.pretrained(f_model, data, metrics=metrics)\n",
    "        print('training for 64x64')\n",
    "        learn.fit(lr, 2, cycle_len=1, cycle_mult=2)\n",
    "        learn.unfreeze()\n",
    "        learn.fit(lrs, 2, cycle_len=1, cycle_mult=2)\n",
    "        \n",
    "        print('training for 128x128')\n",
    "        learn.set_data(get_data_ens(128, i, val_idx))\n",
    "        learn.freeze()\n",
    "        learn.fit(lr, 2, cycle_len=1, cycle_mult=2)\n",
    "        learn.unfreeze()\n",
    "        learn.fit(lrs, 2, cycle_len=1, cycle_mult=2)\n",
    "        \n",
    "        print('training for 256x256')\n",
    "        learn.set_data(get_data_ens(256, i, val_idx))\n",
    "        learn.freeze()\n",
    "        learn.fit(lr, 2, cycle_len=1, cycle_mult=2)\n",
    "        learn.unfreeze()\n",
    "        learn.fit(lrs, 2, cycle_len=1, cycle_mult=2)\n",
    "        \n",
    "        \n",
    "        learn.save(f'ensem_model_{i}.weights')\n",
    "        np.savez_compressed(f'{path}models/ensem_model_{i}_validx', val_idx=val_idx)\n",
    "        models.append(learn)\n",
    "        print(f'-----Training of model {i+1} complete----')\n",
    "    return models\n",
    "        \n",
    "    \n",
    "def get_data_ens(img_sz, model_index, val_idx):\n",
    "    return  ImageClassifierData.from_csv(path, 'train-jpg', f'{path}train_v2.csv', bs, get_transform(model_index, img_sz), suffix='.jpg', val_idxs=val_idx, test_name='test-jpg')\n",
    "\n",
    "    \n",
    "    \n",
    "def get_transform(index, img_sz):\n",
    "    f_model=resnet34\n",
    "    index = (index%5)\n",
    "    print(f'get_transform--{index}: {img_sz}')\n",
    "    tfms = [\n",
    "        tfms_from_model(f_model, img_sz, aug_tfms=transforms_basic, max_zoom=1.05),\n",
    "        tfms_from_model(f_model, img_sz, aug_tfms=transforms_side_on, max_zoom=1.05),\n",
    "        tfms_from_model(f_model, img_sz, aug_tfms=transforms_top_down, max_zoom=1.05),\n",
    "        tfms_from_model(f_model, img_sz, aug_tfms=transforms_top_down, max_zoom=1.1),\n",
    "        tfms_from_model(f_model, img_sz, aug_tfms=transforms_top_down, max_zoom=1.05, crop_type=CropType.RANDOM)]\n",
    "    return tfms[index]\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 230,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----Training model: 1--------\n",
      "get_transform--0: 64\n",
      "training for 64x64\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a00d7ff8e4a48aa9c2834e4d0f6770b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.14123  0.12738  0.88686]                        \n",
      "[ 1.       0.13894  0.12215  0.8923 ]                        \n",
      "[ 2.       0.1381   0.12148  0.89214]                        \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2512996a3b26446aa72d0ae122bcabdf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.11582  0.10072  0.91347]                        \n",
      "[ 1.       0.10864  0.09727  0.91645]                        \n",
      "[ 2.       0.09857  0.09492  0.91977]                         \n",
      "\n",
      "training for 128x128\n",
      "get_transform--0: 128\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "db51cc3bf4db4c6a9d61b22a72ad5df3",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09799  0.09317  0.92097]                         \n",
      "[ 1.       0.09873  0.09252  0.92171]                         \n",
      "[ 2.       0.09702  0.09266  0.92083]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "771d4de3bb4546039adebf8fcaf15d76",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.0919   0.08672  0.9265 ]                         \n",
      "[ 1.       0.0922   0.08715  0.92625]                         \n",
      "[ 2.       0.08603  0.08466  0.93129]                         \n",
      "\n",
      "training for 256x256\n",
      "get_transform--0: 256\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e704caddc2814b8c8f6b7937a15f634c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.0918   0.08919  0.92322]                         \n",
      "[ 1.       0.09305  0.08912  0.92321]                         \n",
      "[ 2.       0.09095  0.08861  0.92276]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "641bf236752b492fab82b6d74fea071c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.08818  0.08304  0.93024]                         \n",
      "[ 1.       0.08729  0.08223  0.93113]                         \n",
      "[ 2.       0.08028  0.08189  0.93266]                         \n",
      "[ 3.       0.08555  0.08365  0.93219]                         \n",
      "[ 4.       0.07971  0.08482  0.92951]                         \n",
      "[ 5.       0.06909  0.08532  0.92912]                         \n",
      "[ 6.       0.06364  0.08631  0.9299 ]                         \n",
      "\n",
      "-----Training of model 1 complete----\n",
      "-----Training model: 2--------\n",
      "get_transform--1: 64\n",
      "training for 64x64\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ef2c3f0264a4411aa9640a90062858d8",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.14316  0.1278   0.88611]                        \n",
      "[ 1.       0.13433  0.1223   0.89097]                        \n",
      "[ 2.       0.13126  0.12012  0.89369]                        \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca8b1030e01c4445a85ea08ef970d8e0",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.11266  0.10309  0.91136]                        \n",
      "[ 1.       0.11255  0.10161  0.90974]                        \n",
      "[ 2.       0.0976   0.09663  0.91787]                         \n",
      "\n",
      "training for 128x128\n",
      "get_transform--1: 128\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "32505b38a16240988e330346e8557b37",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09929  0.09301  0.92042]                         \n",
      "[ 1.       0.09584  0.09294  0.9198 ]                         \n",
      "[ 2.       0.0954   0.09085  0.92148]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6667e292c4374c09ad451b7866761636",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09932  0.08745  0.92374]                         \n",
      "[ 1.       0.09652  0.08676  0.92632]                         \n",
      "[ 2.       0.08692  0.08384  0.92958]                         \n",
      "\n",
      "training for 256x256\n",
      "get_transform--1: 256\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d68fea52011d460d98701c1cd674deb7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09358  0.08915  0.9244 ]                         \n",
      "[ 1.       0.0909   0.08936  0.92324]                         \n",
      "[ 2.       0.08913  0.08796  0.92431]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "fad60166ace14e6ab735268a14add46b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.08721  0.08325  0.93174]                         \n",
      "[ 1.       0.08736  0.08372  0.93143]                         \n",
      "[ 2.       0.08593  0.08274  0.93225]                         \n",
      "[ 3.       0.089    0.08457  0.92996]                         \n",
      "[ 4.       0.08453  0.08174  0.93388]                         \n",
      "[ 5.       0.07834  0.08137  0.93278]                         \n",
      "[ 6.       0.07477  0.0819   0.93276]                         \n",
      "\n",
      "-----Training of model 2 complete----\n",
      "-----Training model: 3--------\n",
      "get_transform--2: 64\n",
      "training for 64x64\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3384e134fe1e4c67a012e2c67f185c13",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.14779  0.12715  0.88653]                        \n",
      "[ 1.       0.1363   0.12225  0.89201]                        \n",
      "[ 2.       0.13544  0.12007  0.89482]                        \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "b2da4c82c1414d51a22398c29226a77b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.11512  0.10323  0.91386]                        \n",
      "[ 1.       0.11114  0.09822  0.91826]                        \n",
      "[ 2.       0.10218  0.09828  0.91502]                         \n",
      "\n",
      "training for 128x128\n",
      "get_transform--2: 128\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "63e5bb97f75240a3aa39c2f50d74d1dc",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09935  0.09328  0.91971]                         \n",
      "[ 1.       0.09829  0.09256  0.92117]                         \n",
      "[ 2.       0.09828  0.09201  0.92158]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d9c696d77bb94ff38d9c7707aabdfbc7",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09707  0.08829  0.92372]                         \n",
      "[ 1.       0.09709  0.08605  0.92876]                         \n",
      "[ 2.       0.09223  0.08488  0.92664]                         \n",
      "\n",
      "training for 256x256\n",
      "get_transform--2: 256\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ab42b3a5d2ae4f759a3a85e56c1489e2",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09218  0.0888   0.92372]                         \n",
      "[ 1.       0.09462  0.08829  0.92334]                         \n",
      "[ 2.       0.09737  0.08805  0.92562]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "903e5444d9fd41c88cda78d7d14d2f3d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09219  0.08388  0.92673]                         \n",
      "[ 1.       0.08893  0.08144  0.93248]                         \n",
      "[ 2.       0.08512  0.08088  0.93208]                         \n",
      "[ 3.       0.08873  0.08169  0.93299]                         \n",
      "[ 4.       0.08617  0.08259  0.9282 ]                         \n",
      "[ 5.       0.08163  0.07919  0.93539]                         \n",
      "[ 6.       0.08074  0.07921  0.93532]                         \n",
      "\n",
      "-----Training of model 3 complete----\n",
      "-----Training model: 4--------\n",
      "get_transform--3: 64\n",
      "training for 64x64\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "8c7cf83d9ba7442fae78320d62bb3095",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.14447  0.12849  0.88602]                        \n",
      "[ 1.       0.1389   0.1233   0.89142]                        \n",
      "[ 2.       0.13561  0.12112  0.89248]                        \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "83cbf8868266445aa003745c8aa09efd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.11718  0.10237  0.91336]                        \n",
      "[ 1.       0.113    0.09816  0.91557]                        \n",
      "[ 2.       0.10239  0.09434  0.91988]                        \n",
      "\n",
      "training for 128x128\n",
      "get_transform--3: 128\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "bc6fe1a26f7a438f8893748f38462c4c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.10027  0.09285  0.91857]                         \n",
      "[ 1.       0.10043  0.09141  0.92097]                         \n",
      "[ 2.       0.09898  0.09158  0.92012]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f02b2ad71d3b4729a687e456eeb3e21b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.097    0.08551  0.92739]                         \n",
      "[ 1.       0.09587  0.08664  0.92863]                         \n",
      "[ 2.       0.09133  0.08428  0.92836]                         \n",
      "\n",
      "training for 256x256\n",
      "get_transform--3: 256\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "13437be9999642a199bc1f45c597e14a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09405  0.08936  0.92308]                         \n",
      "[ 1.       0.09401  0.08762  0.92572]                         \n",
      "[ 2.       0.09423  0.08796  0.9241 ]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "29a1cff76355433aac8355b43457c354",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09043  0.08248  0.93041]                         \n",
      "[ 1.       0.09122  0.08383  0.92914]                         \n",
      "[ 2.       0.0889   0.08152  0.9329 ]                         \n",
      "[ 3.       0.09007  0.08272  0.93147]                         \n",
      "[ 4.      0.0856  0.0815  0.9308]                             \n",
      "[ 5.       0.08346  0.08005  0.93289]                         \n",
      "[ 6.       0.0811   0.0795   0.93379]                         \n",
      "\n",
      "-----Training of model 4 complete----\n",
      "-----Training model: 5--------\n",
      "get_transform--4: 64\n",
      "training for 64x64\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "d1cf6f5663e9444aac658d95868ba07a",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.14541  0.12839  0.88706]                        \n",
      "[ 1.       0.13504  0.12284  0.88916]                        \n",
      "[ 2.       0.13081  0.11959  0.89448]                        \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0bbe727dbb5d47afbd5f96bb7558fa60",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.1157   0.10345  0.91263]                        \n",
      "[ 1.       0.11322  0.09955  0.91581]                        \n",
      "[ 2.       0.10437  0.09441  0.91964]                        \n",
      "\n",
      "training for 128x128\n",
      "get_transform--4: 128\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3a58d80d11cd4e59a5e4c58169188750",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.10273  0.09496  0.91838]                         \n",
      "[ 1.       0.09578  0.09428  0.91875]                         \n",
      "[ 2.       0.09843  0.09235  0.92164]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "182d04ed3b0749e494e6d5a5d7f1a16d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09924  0.0881   0.92566]                         \n",
      "[ 1.       0.09617  0.08657  0.92612]                         \n",
      "[ 2.       0.09097  0.08427  0.92952]                         \n",
      "\n",
      "training for 256x256\n",
      "get_transform--4: 256\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "40e633404a23488280f93cabd8dd969d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n",
       "<p>\n",
       "  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=3), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09458  0.08918  0.92472]                         \n",
      "[ 1.       0.09248  0.08865  0.92466]                         \n",
      "[ 2.       0.09141  0.08844  0.92434]                         \n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a75979c4876e4ef88133c8770d104d39",
       "version_major": 2,
       "version_minor": 0
      },
      "text/html": [
       "<p>Failed to display Jupyter Widget of type <code>HBox</code>.</p>\n"
      ],
      "text/plain": [
       "HBox(children=(IntProgress(value=0, description='Epoch', max=7), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.       0.09029  0.08313  0.92911]                         \n",
      "[ 1.       0.09007  0.08329  0.92949]                         \n",
      "[ 2.       0.08437  0.08032  0.93397]                         \n",
      "[ 3.       0.09159  0.08335  0.93036]                         \n",
      "[ 4.       0.08848  0.08158  0.93119]                         \n",
      "[ 5.       0.08418  0.07971  0.9338 ]                         \n",
      "[ 6.       0.07982  0.07953  0.93427]                         \n",
      "\n",
      "-----Training of model 5 complete----\n",
      "CPU times: user 6h 6min 45s, sys: 51min 7s, total: 6h 57min 52s\n",
      "Wall time: 2h 18min 40s\n"
     ]
    }
   ],
   "source": [
    "%time ens = get_ensumble(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preparing Submission"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 245,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f2 Score: 0.9285756738073999                 \n",
      "f2 Score: 0.9325735931488134                 \n",
      "f2 Score: 0.9345646226806884                 \n",
      "f2 Score: 0.9331467241762751                 \n",
      "f2 Score: 0.9349772800026489                 \n",
      "[0.17000000000000001, 0.21000000000000005, 0.20000000000000004, 0.21000000000000005, 0.21000000000000005]\n",
      "0.2\n"
     ]
    }
   ],
   "source": [
    "#Find optimized Threshold\n",
    "ens_val_probs=list()\n",
    "th_list=list()\n",
    "for mdl in ens:\n",
    "    val_probs, y=mdl.TTA()\n",
    "    val_probs=np.mean(val_probs, axis=0)\n",
    "    acc = f2(val_probs, y)\n",
    "    print (f'f2 Score: {acc}')\n",
    "    th =opt_th(val_probs, y)\n",
    "    th_list.append(th)\n",
    "    ens_val_probs.append(val_probs)\n",
    "print(th_list)   \n",
    "op_th = np.mean(th_list)\n",
    "print(op_th)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 249,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(61191, 17)"
      ]
     },
     "execution_count": 249,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Prepare test predictions\n",
    "ens_test_probs = list()\n",
    "for mdl in ens:\n",
    "    test_probs,_ = mdl.TTA(is_test=True)\n",
    "    test_probs = np.mean(test_probs, axis=0)\n",
    "    ens_test_probs.append(test_probs)\n",
    "ens_test_probs = np.array(ens_test_probs)\n",
    "ens_test_probs = np.mean(ens_test_probs, axis=0)\n",
    "ens_test_probs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 250,
   "metadata": {},
   "outputs": [],
   "source": [
    "#prepare the submission file\n",
    "classes = np.array(ens[0].data.classes)\n",
    "res = np.array([\" \".join(classes[(np.where(pp>op_th))]) for pp in ens_test_probs])\n",
    "filenames = np.array([os.path.basename(fn).split('.')[0] for fn in ens[0].data.test_ds.fnames])\n",
    "frame=pd.DataFrame(res, index=filenames, columns=['tags'])\n",
    "frame.to_csv(f'{path}planet_amazon_restnet34_submission_ens.csv', index_label='image_name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A Machine Learning model for calculating threshold\n",
    "When preparing submission, we used a threshold of ~0.2 to select classes for all of test images, but ideally each test image should have a separate threshold depending on the predictions values from model. I experimented with training ML model to find the better threshold but didn’t succeed. Here is the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.preprocessing import PolynomialFeatures\n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.metrics import accuracy_score, mean_squared_error"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 221,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_optimized_threshold_list(x, y):\n",
    "    th_list = list()\n",
    "    for truth,  preds in zip(y, x):\n",
    "        steps = np.arange(0.1, 0.4, 0.01)\n",
    "        acc = np.array([accuracy_score(truth, (preds > th)) for th in steps])\n",
    "        th_list.append(steps[np.argmax(acc)])\n",
    "    return th_list\n",
    "                   \n",
    "                   \n",
    "def ml_model_threshold():\n",
    "    arr = np.load(f'{path}models/preds_probs.npz')\n",
    "    x = arr['probs']\n",
    "    y = arr['y']\n",
    "    th_list= get_optimized_threshold_list(x, y)\n",
    "    model = make_pipeline(PolynomialFeatures(2), Ridge())\n",
    "    model.fit(x, th_list)\n",
    "    print (mean_squared_error(th_list, model.predict(x)))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Submitting Machine Learning predicitons"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 222,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.00419767227625\n"
     ]
    }
   ],
   "source": [
    "model = ml_model_threshold()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 223,
   "metadata": {},
   "outputs": [],
   "source": [
    "classes = np.array(data.classes)\n",
    "op_th_ml = model.predict(preds)\n",
    "res = np.array([\" \".join(classes[(np.where(preds[i]>op_th_ml[i]))]) for i in range(len(preds))])\n",
    "filenames = np.array([os.path.basename(fn).split('.')[0] for fn in data.test_ds.fnames])\n",
    "frame=pd.DataFrame(res, index=filenames, columns=['tags'])\n",
    "frame.to_csv(f'{path}planet_amazon_restnet34_submission2.csv', index_label='image_name')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}