3. codertimo-bert-with-fastai.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we'll be exploring how to use BERT with fastai for sentence classification."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from pathlib import Path\n",
    "from typing import *\n",
    "\n",
    "import torch\n",
    "import torch.optim as optim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.vision import *\n",
    "from fastai.text import *\n",
    "from fastai.callbacks import *"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Config(dict):\n",
    "    def __init__(self, **kwargs):\n",
    "        super().__init__(**kwargs)\n",
    "        for k, v in kwargs.items():\n",
    "            setattr(self, k, v)\n",
    "    \n",
    "    def set(self, key, val):\n",
    "        self[key] = val\n",
    "        setattr(self, key, val)\n",
    "\n",
    "config = Config(\n",
    "    testing=False,\n",
    "    bert_model_name=\"bert-base-uncased\",\n",
    "    max_lr=3e-5,\n",
    "    epochs=1,\n",
    "    use_fp16=False,\n",
    "    bs=4,\n",
    "    discriminative=False,\n",
    "    max_seq_len=128,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be using the pytorch-pretrained-bert package, so install it if you do not have it yet!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "BERT requires a special wordpiece tokenizer and a vocabulary to go along with that. Thankfully, the pytorch-pretrained-bert package provides all of that within the handy `BertTokenizer` class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Better speed can be achieved with apex installed from https://www.github.com/nvidia/apex.\n"
     ]
    }
   ],
   "source": [
    "from pytorch_pretrained_bert import BertTokenizer\n",
    "bert_tok = BertTokenizer.from_pretrained(\n",
    "    config.bert_model_name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FastAI has its own conventions for handling tokenization, so we'll need to wrap the tokenizer within a different class. This is a bit confusing but shouldn't be that much of a hassle.\n",
    "\n",
    "Notice we add the \\[CLS] and \\[SEP] special tokens to the start and end of the sequence here.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "class FastAiBertTokenizer(BaseTokenizer):\n",
    "    \"\"\"Wrapper around BertTokenizer to be compatible with fast.ai\"\"\"\n",
    "    def __init__(self, tokenizer: BertTokenizer, max_seq_len: int=128, **kwargs):\n",
    "        self._pretrained_tokenizer = tokenizer\n",
    "        self.max_seq_len = max_seq_len\n",
    "\n",
    "    def __call__(self, *args, **kwargs):\n",
    "        return self\n",
    "\n",
    "    def tokenizer(self, t:str) -> List[str]:\n",
    "        \"\"\"Limits the maximum sequence length\"\"\"\n",
    "        return [\"[CLS]\"] + self._pretrained_tokenizer.tokenize(t)[:self.max_seq_len - 2] + [\"[SEP]\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Slightly confusingly, we further need to wrap the tokenizer above in a `Tokenizer` object. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "fastai_tokenizer = Tokenizer(tok_func=FastAiBertTokenizer(bert_tok, max_seq_len=config.max_seq_len), pre_rules=[], post_rules=[])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to make sure fastai uses the same mapping from wordpiece to integer as BERT originally did. Again, fastai has its own conventions on vocabulary so we'll be passing the vocabulary internal to the `BertTokenizer` and constructing a fastai `Vocab` object to use for preprocessing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "fastai_bert_vocab = Vocab(list(bert_tok.vocab.keys()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have all the pieces we need to make BERT work with fastai! We'll load the data into dataframes and construct a validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "DATA_ROOT = Path(\"..\") / \"data\" / \"jigsaw\"\n",
    "\n",
    "train, test = [pd.read_csv(DATA_ROOT / fname) for fname in [\"train.csv\", \"test.csv\"]]\n",
    "train, val = train_test_split(train)\n",
    "\n",
    "if config.testing:\n",
    "    train = train.head(1024)\n",
    "    val = val.head(1024)\n",
    "    test = test.head(1024)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can build the databunch using the tokenizer and vocabulary we build above. Notice we're passing the `include_bos=False` and `include_eos=False` options. This is to prevent fastai from adding its own SOS/EOS tokens that will interfere with BERT's SOS/EOS tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "label_cols = [\"toxic\", \"severe_toxic\", \"obscene\", \"threat\", \"insult\", \"identity_hate\"]\n",
    "\n",
    "databunch = TextDataBunch.from_df(\".\", train, val, test,\n",
    "                  tokenizer=fastai_tokenizer,\n",
    "                  vocab=fastai_bert_vocab,\n",
    "                  include_bos=False,\n",
    "                  include_eos=False,\n",
    "                  text_cols=\"comment_text\",\n",
    "                  label_cols=label_cols,\n",
    "                  bs=config.bs,\n",
    "                  collate_fn=partial(pad_collate, pad_first=False, pad_idx=0),\n",
    "             )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, we can pass our own list of Preprocessors to the databunch (this is effectively what is happening behind the scenes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BertTokenizeProcessor(TokenizeProcessor):\n",
    "    def __init__(self, tokenizer):\n",
    "        super().__init__(tokenizer=tokenizer, include_bos=False, include_eos=False)\n",
    "\n",
    "class BertNumericalizeProcessor(NumericalizeProcessor):\n",
    "    def __init__(self, *args, **kwargs):\n",
    "        super().__init__(*args, vocab=Vocab(list(bert_tok.vocab.keys())), **kwargs)\n",
    "\n",
    "def get_bert_processor(tokenizer:Tokenizer=None, vocab:Vocab=None):\n",
    "    \"\"\"\n",
    "    Constructing preprocessors for BERT\n",
    "    We remove sos/eos tokens since we add that ourselves in the tokenizer.\n",
    "    We also use a custom vocabulary to match the numericalization with the original BERT model.\n",
    "    \"\"\"\n",
    "    return [BertTokenizeProcessor(tokenizer=tokenizer),\n",
    "            NumericalizeProcessor(vocab=vocab)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use our own custom preprocessors, we'll need to modify the `from_df` method to call the function above. Not the cleanest code but it will suffice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BertDataBunch(TextDataBunch):\n",
    "    @classmethod\n",
    "    def from_df(cls, path:PathOrStr, train_df:DataFrame, valid_df:DataFrame, test_df:Optional[DataFrame]=None,\n",
    "                tokenizer:Tokenizer=None, vocab:Vocab=None, classes:Collection[str]=None, text_cols:IntsOrStrs=1,\n",
    "                label_cols:IntsOrStrs=0, label_delim:str=None, **kwargs) -> DataBunch:\n",
    "        \"Create a `TextDataBunch` from DataFrames.\"\n",
    "        p_kwargs, kwargs = split_kwargs_by_func(kwargs, get_bert_processor)\n",
    "        # use our custom processors while taking tokenizer and vocab as kwargs\n",
    "        processor = get_bert_processor(tokenizer=tokenizer, vocab=vocab, **p_kwargs)\n",
    "        if classes is None and is_listy(label_cols) and len(label_cols) > 1: classes = label_cols\n",
    "        src = ItemLists(path, TextList.from_df(train_df, path, cols=text_cols, processor=processor),\n",
    "                        TextList.from_df(valid_df, path, cols=text_cols, processor=processor))\n",
    "        src = src.label_for_lm() if cls==TextLMDataBunch else src.label_from_df(cols=label_cols, classes=classes)\n",
    "        if test_df is not None: src.add_test(TextList.from_df(test_df, path, cols=text_cols))\n",
    "        return src.databunch(**kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "# this will produce a virtually identical databunch to the code above\n",
    "# databunch = BertDataBunch.from_df(\".\", train, val, test,\n",
    "#                   tokenizer=fastai_tokenizer,\n",
    "#                   vocab=fastai_bert_vocab,\n",
    "#                   text_cols=\"comment_text\",\n",
    "#                   label_cols=label_cols,\n",
    "#                   bs=config.bs,\n",
    "#                   collate_fn=partial(pad_collate, pad_first=False, pad_idx=0),\n",
    "#              )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now with the data in place, we will prepare the model and loss functions. Again, the pytorch-pretrained-bert package gives us a sequence classifier based on BERT straight out of the box."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pytorch_pretrained_bert.modeling import BertConfig, BertForSequenceClassification\n",
    "bert_model = BertForSequenceClassification.from_pretrained(config.bert_model_name, num_labels=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is a multilabel classification problem, we're using `BCEWithLogitsLoss`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "loss_func = nn.BCEWithLogitsLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we can build the `Learner`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.callbacks import *\n",
    "\n",
    "learner = Learner(\n",
    "    databunch, bert_model,\n",
    "    loss_func=loss_func,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we're done! All the rest is fastai magic. For example, you can use half-precision training simply by calling `learner.to_fp16()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "if config.use_fp16: learner = learner.to_fp16()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the learning rate finder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LR Finder is complete, type {learner_name}.recorder.plot() to see the graph.\n"
     ]
    }
   ],
   "source": [
    "learner.lr_find()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEKCAYAAAD9xUlFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VNX9//HXZyYJWQgJkCD7vgnIGkDEBZdabFW0VitWqXvdbevys19ba7V+W7WlX/eK1n2h7kWLW1XckX2NgIAsYQ/ZyL6d3x8zCTEmECA3s+T9fDzm8Zi5c2bu5zBkPnPOufdzzTmHiIgIgC/UAYiISPhQUhARkVpKCiIiUktJQUREaikpiIhILSUFERGppaQgIiK1lBRERKSWp0nBzCab2WozW2tmtzTwfC8z+8DMlpnZHDPr7mU8IiKyb+bVGc1m5gfWAD8AsoD5wFTnXGadNi8DbznnnjazE4CLnHMX7Ot909LSXO/evT2JWUQkWi1cuDDbOZe+v3YxHsYwDljrnFsPYGYzgSlAZp02Q4DfBO9/BLyxvzft3bs3CxYsaOZQRUSim5ltbEo7L6ePugGb6zzOCm6raynwk+D9M4FkM+tY/43M7HIzW2BmC3bt2uVJsCIiEvqF5huB48xsMXAcsAWoqt/IOTfDOZfhnMtIT9/v6EdERA6Sl9NHW4AedR53D26r5ZzbSnCkYGZtgbOcc3kexiQiIvvg5UhhPjDAzPqYWRxwLjCrbgMzSzOzmhh+CzzhYTwiIrIfniUF51wlcA3wLvA18JJzbqWZ3WFmpwebTQJWm9ka4DDgLq/iERGR/fPskFSvZGRkOB19JCJyYMxsoXMuY3/tQr3QLCIiYURJQUQkzFVVO/539tcs3ez9cThKCiIiYe7b7CJmfLKetTsLPd+XkoKISJjL3FYAwJCu7Tzfl5KCiEiYW7k1nzi/j37pbT3fl5KCiEiYy9xawIDD2hIX4/1XtpKCiEgYc86RubWAIV28nzoCJQURkbC2a08Zu4vKGdoC6wmgpCAiEtZW1i4yp7TI/pQURETCWObWQFIY3CW5RfanpCAiEsYytxbQs0Mi7eJjW2R/SgoiImEsc1vLLTKDkoKISNgqLKtkw+6iFjlprYaSgohImFq9vQDnaLEjj0BJQUQkbK3c2nLlLWooKYiIhKnMrQW0T4ylc7v4FtunkoKISJjK3FbAkK7tMLMW26eSgohIGKqsqmbV9j0teuQRKCmIiISl9dlFlFdWM7SFzmSuoaQgIhKGVm7NB1p2kRmUFEREwlLm1gLiYnz0TUtq0f0qKYiIhKHMbQUM7pxMjL9lv6aVFEREwkxLX0OhLiUFEZEws72glNziihZfTwAlBRGRsFNTLrsly1vUUFIQEQkzm3KKAejdsWUXmUFJQUQk7GQXluH3Ge0T41p830oKIiJhJntPOR2S4vD5Wq68RQ0lBRGRMLO7qIy0tm1Csm8lBRGRMLOrsJy0ti0/dQRKCiIiYSd7T5SOFMxsspmtNrO1ZnZLA8/3NLOPzGyxmS0zsx95GY+ISLhzzgWnj6JspGBmfuAh4BRgCDDVzIbUa/Y74CXn3CjgXOBhr+IREYkEReVVlFZU0zEKRwrjgLXOufXOuXJgJjClXhsH1JydkQJs9TAeEZGwt7uwDCBk00cxHr53N2BzncdZwPh6bW4H3jOza4Ek4CQP4xERCXvZtUkhyqaPmmgq8JRzrjvwI+BZM/teTGZ2uZktMLMFu3btavEgRURayq495UDoRgpeJoUtQI86j7sHt9V1CfASgHPuSyAeSKv/Rs65Gc65DOdcRnp6ukfhioiE3u6i0E4feZkU5gMDzKyPmcURWEieVa/NJuBEADM7nEBS0FBARFqt7OBIoUNSlE0fOecqgWuAd4GvCRxltNLM7jCz04PNbgAuM7OlwIvAhc4551VMIiLhLruwjJSEWOJiQjO77+VCM8652cDsettuq3M/E5joZQwiIpEklOcoQOgXmkVEpI7sPeUhW08AJQURkbCSXRi6EhegpCAiElYCSUHTRyIirV5ZZRUFpZUhK3EBSgoiImFjd2FoT1wDJQURkbCxNylo+khEpNWrqXuk6SMREWFXMCmkKymIiEjt9FGypo9ERFq97MIyEmL9JMZ5Wmxin5QURETCxO7CspCOEkBJQUQkbGQXhrbEBSgpiIiEjezCMjomKSmIiAiBkUK6po9ERKSq2pFTFNpieKCkICISFnKLy6l20DFEV1yroaQgIhIG9p6joJGCiEirV1PiQtNHIiJSJylo+khEpNXLDoOy2aCkICISFrILy4jxGe3iY0Mah5KCiEgYyN5TRse2cfh8FtI4lBRERMLA7qLQl7gAJQURkbCQXVgW0ovr1Gg1ScE5F+oQREQalb2nLORHHgGErmh3C3t5QRaPfrKOCf06MqFvGuP7dtjvUM05R7UDf4jn+EQkujnnyC4qD+kV12q0mqSQntyGnh0SeX3RFp6buwmA/p3a0i01gQ5JcbRPjKN9YiwFpRVs3F3Mxt3FbMoppto5RvZIZVyfDozt3YHRvdrTtk3D/2zV1Y6t+SXkl1TgMwveoLLakV9SQV5xObnFFRSVVTKqZ3tG9UgN+aKSiITenrJKyiur6aiRQss5fnAnjh/cicqqapZvyefL9btZtDGXXXvKWJ9dSG5RBYVllbSJ8dGrYyI9OyRx9IA0qp1jwYZcHvpoLdXBGai0tm3o3j6Bbu0T6JoST3ZhOWt3FrJ2ZyElFVVNjimtbRwnDj6MHww5jHF9O4T8UDQRCY3sPeFxNjO0oqRQI8bvC/xK79n+e8+VV1YT47MGf70XllWyeFMuSzblkZVbwpa8EjK3FvDfzB10TIqjX6e2nDuuB/07tQ3WQw9MPVU7h8+M1IRYUhJjaZ8YR6zfxxfrsnk/cwf/Wb6Nfy3YDARGM/3T29KvUxJdUhKI8Rn+4C0+1s+gzskM6dKO+Fi/1/9MItKCdheFx4lr0AqTwr7ExTS+7t62TQzHDEjnmAHpzbKvKSO7MWVkN8orq5n3bQ4rtuazdmch63YV8u8lW9lTWtng6/w+Y0CnthzRLYXeaUl0SYmnS0oCXVPj6ZqaQKy/1Rw7IBI1akYKmj4S4mJ8HD0gjaMHpNVuc85RVllNVbWjstpRXe0oLKskc1sBy7PyWb4lnw9X7az9dVEjJSGWyUM7c+qILkzo25EYJQiRiJAd/FuO+oVmM5sM3Af4gcedc3+p9/zfgeODDxOBTs65VC9jigRm9r0povZJcfTokMgPh3au3VZcXsm2/FK255eyJa+Euet2105HdUyKY/Kwzpx0+GFM6NdRU04iYaxmpNAhxNdSAA+Tgpn5gYeAHwBZwHwzm+Wcy6xp45z7dZ321wKjvIonGiXGxdAvvS390tsCcE5GD0orqpizehdvLtvKa4u28PxXm4iP9TGxXxonHN6JHx/RhdTE0P/HE5G9sgvLaJ8YGxajey9HCuOAtc659QBmNhOYAmQ20n4q8AcP42kV4mP9TB7WmcnDOlNaUcVX3+bw4dc7+HD1Tj5YtZM738rk9BFdmTahN8O6pYQ6XBEhcIGdcDibGbxNCt2AzXUeZwHjG2poZr2APsCHHsbT6sTH+jluYDrHDUzndudYubWA57/axBuLt/DSgixG9UzltOFdyejdnsO7tNMitUiI5BSXh8XUEYTPQvO5wCvOuQYP8jezy4HLAXr27NmScUUNM2NYtxT+/JMjuOWUwby2KIvnv9rEHW8FBm4JsX5G9kjlqH4dOWNUN3p0SAxxxCKtR25Ree00cKh5mRS2AD3qPO4e3NaQc4GrG3sj59wMYAZARkaGihgdopSEWC6a2IeLJvZhW34JCzbksnBjLgs25jD9v2v42/trmNC3Iz8d051TjuhMYly4/HYQiU65xeW0bwUjhfnAADPrQyAZnAucV7+RmQ0G2gNfehiLNKJLSgKnjUjgtBFdAcjKLeb1RVt4ZVEWN7y8lD/MWsnlx/blsmP6khCnI5hEmptzjtziCjokhUdFA88mkZ1zlcA1wLvA18BLzrmVZnaHmZ1ep+m5wEynMqZhoXv7RK49cQBzbpzEy1dMYGL/jkx/fw3H/3UOryzMorpaH5NIcyooraSq2tE+TI4K9HRewDk3G5hdb9tt9R7f7mUMcnDMjLG9A0UA532bw13/yeTGl5fyxGffctMPBzFpUDpmKuYncqhygyeuhctCsw43kf0a16cDr181kfunjqKgtIKLnprP6Q9+zrsrt2vkIHKIcooDSSFc1hSUFKRJfD7j9BFd+fCGSdxz1nAKSiv45bML+dH9n/LOim26iJHIQaodKYTJ9JGSghyQuBgf54ztwQe/OY6//2wEFVXVXPHcIs7+x5cs3pQb6vBEIk6Opo8kGsT4fZw5qjvv/upY/vyTI9iwu5gzH/6Ca19czOac4lCHJxIxcjV9JNEkxu9j6riezLlpEtee0J/3M7dz4vSP+cvbqygorQh1eCJhL6eogli/kRQmh3wrKUizaNsmhhtOHsRHN07i1OFd+MfH65h07xyenbuRyqrqUIcnErZyi8ppnxgXNkfzKSlIs+qSksD0c0by5jVHM6BTW37/xgom3/ep1htEGhFOdY9ASUE8ckT3FGZefiQzLhhDSXkV5zz6JU99/q2OUhKpp2akEC6UFMQzZsbJQzsz+7pjApVa38zkmhcXU1jW8KVGRVojjRSk1UlJjGXGBRnccspg3lmxndMf+EzTSSJBuUXltA+TukegpCAtxOczrjiuH89fOp49ZZWc+fAXnPPol7yfuUNnRUurVVXtyC+pCJsT1yB8rqcgrcSRfTvy4Q3H8a/5m3ny8w1c9swC+qQlcf6RvTh+UDp90pLC5igMEa8VlFRQ7cLnHAVoYlIws35AlnOuzMwmAcOBZ5xzeV4GJ9EpOT6WS4/py4VH9ebtFdt5/NP13PlWJne+Bd1SEzi6fxqTBqXzw6Gd8fmUICR61dQ9Cqc1haaOFF4FMsysP4GL3fwbeAH4kVeBSfSL8fs4bURXThvRlY27i/j0m2w++yab2Su28a8FmzltRFf+evZw2sSEx0k9Is2tpu5ROB191NSkUO2cqzSzM4EHnHMPmNliLwOT1qVXxyR6dQxMI1VWVTPj0/Xc885qdhSU8tgFGaQkhs9CnEhzCbe6R9D0heYKM5sK/AJ4K7hNf6XiiRi/j6sm9ee+c0eyeFMuZ/3jC7JyVU9Jok+41T2CpieFi4AJwF3OuW+Dl9h81ruwRGDKyG48c/F4dhSUcubDX7BiS36oQxJpVjlFgfpg4XT0UZOSgnMu0zl3nXPuRTNrDyQ75+72ODYRJvTryKtXHkWsz5g6Yy5frd8d6pBEmk1ucTnxsb6wuv55k5KCmc0xs3Zm1gFYBDxmZtO9DU0kYOBhybxy5VF0ateGaU/M48NVO0IdkkizyAmzEhfQ9OmjFOdcAfATAoeijgdO8i4ske/qmprAS7+cwMDDkrn8mYW8sXhLqEMSOWThVvcImp4UYsysC3AOexeaRVpUx7ZteOGy8WT0bs+v/rWEZ+duDHVIIock3OoeQdOTwh3Au8A659x8M+sLfONdWCINS46P5amLxnHS4Z34/RsrNGKQiBaoexSBScE597Jzbrhz7srg4/XOubO8DU2kYfGxfh48bzQT+nbkxpeXMmf1zlCHJHJQcorK6RBm5+A0daG5u5m9bmY7g7dXzay718GJNCY+1s+MaWMY1DmZK59bxCJVXZUIU1lVTUFpZWSOFIAngVlA1+DtzeA2kZCpmUrq1K4NFz81n2927Al1SCJNllcSPEchQpNCunPuSedcZfD2FJDuYVwiTZKe3IZnLx5PrN/HtCfmsSWvJNQhiTRJONY9gqYnhd1mdr6Z+YO38wGdRSRhoWfHRJ65eByFZZVc8PhXZBeWhTokkf0Kx7pH0PSkcDGBw1G3A9uAnwIXehSTyAE7vEs7nrxwLFvzS/jFE/MoKK0IdUgi+1Rb9ygSRwrOuY3OudOdc+nOuU7OuTMAHX0kYSWjdwf+cf4YVm/fw6VPL6C0oirUIYk0qrbuUYSOFBrym2aLQqSZTBrUiek/G8n8DTlc/fwiyiqVGCQ81YwUUiPxkNRG6JJYEpZOH9GVO6cM44NVOznhrx8zc94mKqqqQx2WyHfkFJWTFOcnPjZ8iuHBoSUFXW1dwtb5R/bi2UvGkdY2jlteW85J0z/m1YVZVCo5SJgIx7OZYT9Jwcz2mFlBA7c9BM5X2Cczm2xmq81srZnd0kibc8ws08xWmtkLB9kPke85ZkA6b1w9kcenZZAUF8MNLy/lpOkf89L8zZRXKjlIaOUUh18xPNjP5Tidc8kH+8Zm5gceAn4AZAHzzWyWcy6zTpsBwG+Bic65XDPrdLD7E2mImXHSkMM4YXAn3svcwQMffsPNry7j//67hism9eOcjB5hN3yX1iEiRwqHaBywNlgnqRyYCUyp1+Yy4CHnXC6Ac05FbMQTPp8xeVhn3rr2aJ68aCxdUhO47d8rOWn6x6zRmdASAjnF4Vf3CLxNCt2AzXUeZwW31TUQGGhmn5vZXDOb3NAbmdnlZrbAzBbs2rXLo3ClNTAzjh/UiVeumMDzl46nvLKasx75gi/WZoc6NGllcosqWt1IoSligAHAJGAqgSu6pdZv5Jyb4ZzLcM5lpKeruoYcOjNjYv80Xr96Il1TEpj2xDxeWZgV6rCklSivrKawrDKsrs1cw8uksAXoUedx9+C2urKAWc65Cufct8AaAklCpEV0S03g5SsncGSwDPf099fgnA6sE2/l1ZzN3MpGCvOBAWbWx8zigHMJVFqt6w0CowTMLI3AdNJ6D2MS+Z528bE8edFYzsnozv0ffMOjn+i/oHgrpzg86x7Bfo4+OhTOuUozu4bAFdv8wBPOuZVmdgewwDk3K/jcyWaWCVQBNznnVGhPWlys38fdZw2nqLyKu99ZxeDOyUwapIPhxBs5YVohFTxMCgDOudnA7Hrbbqtz3xEol6GSGRJyZsa9Px3Oup2FXPfiYmZdczS905JCHZZEodwwrXsEoV9oFgkriXExPDYtA7/PuOyZBRSWVYY6JIlCObVrCq3rkFSRiNSjQyIPnjea9dlF/OZfS6iu1sKzNK9wvcAOKCmINGhi/zT+50eH817mDs57fC5vLt2qiqvSbHKKykmOjyHWH35fwZ6uKYhEsosn9sY5x5Ofb+DaFxfTISmOn4zqxi+O6k2PDomhDk8iWG5xeViuJ4BGCiKNMjMuPaYvn9x8PE9fPI7xfTrw1BcbmPLQ52zOKQ51eBLBcorCsxgeKCmI7JffZxw3MJ1Hzh/DO786hoqqai1CyyHRSEEkSvTvlMxD541mzY49/FqL0HKQcosqwu6KazWUFEQO0LED0/ndj4fwfuYOpr+/JtThSITJL65g154y0pPbhDqUBmmhWeQgXDSxN6u37+HBj9Yy4LC2TBlZvwCwSMOenbuB8qpqpowIz/8zGimIHAQz484zhjG2d3tufmUZmVsLQh2SRIDSiiqe/HwDkwalM6Rru1CH0yAlBZGDFBfj45Hzx5CSEMvVLyxiT2lFqEOSMPfygs3sLirniuP6hTqURikpiByCtLZteGDqKDbuLuKW15ar7LY0qrKqmhmfrmdUz1TG9+kQ6nAapaQgcojG9+3IjT8cxH+WbeO5uRtDHY6EqdkrtrM5p4QrjuuHmYU6nEYpKYg0gyuO7cfxg9K5862vWZaVF+pwJMw453hkzjr6pSfxg8MPC3U4+6SkINIMfD5j+jkjSWsbx1XPLyK/WOsLstcn32Tz9bYCfnlcP3y+8B0lgJKCSLNpnxTHgz8fzfb8Um5/c2Wow5Ew8sictXRuF88ZEXDospKCSDMa3bM915zQn9cXb+GdFdtDHY6E0I6CUp79cgM/f3wuc9fncOkxfYiLCf+vXJ28JtLMrj6+P//9ege3vr6csb3b07FteJ65Kt5YujmP299cyeJNgbWlvulJXHdCfy6Y0CvEkTWNkoJIM4v1+/jb2SM57YHPuPX1FTxy/uiwPtpEmte/l2xl5ZYCbjx5IJOHdaZ/p+RQh3RAwn8sIxKBBnVO5tc/GMg7K7cza+nWUIcjLSivpJxO7dpwzQkDIi4hgJKCiGcuP7Yvo3qmctu/V7KjoDTU4UgLyS8O3wqoTaGkIOIRv8/429kjKKus4v+9ukxnO7cSeSUVpCaE57USmkJJQcRDfdPbcsvkwcxZvYsX5m0KdTjSAvKKy0nRSEFEGjNtQm+OGZDGn976mm+zi0Idjngsv6SC1AQlBRFphM9n3PvTEcT6jd+8tITKqupQhyQecc6RpzUFEdmfzinx3HnGMBZvyuOROetCEsNX63dz+6yVlJRXhWT/rUFReRWV1U5rCiKyf1NGduO0EV2574NvWJ6V3+L7/+dn3/LUFxu45On5FJdXHvT7aMG8cXnF5QBaUxCRprlzylA6to3jkqfnM/291azduadF9uucY9GmPPqmJTF3/W4uenI+RWUHnhj+vWQLR9/9Eau3t0zckSYvWAhRawoi0iSpiXHMuCCD/p3a8sBHazlp+iecct+nPPrxOsorvVtryMotIbuwjIuO7sPffzaS+RtyuPDJeRQGE0NVtWP19j28sXhLo+dU5BSV84dZK9mSV8Jlzyyo/VUse+WXBJNCYuROH6nMhUgLG9EjlRcuO5KdBaX8Z/k2Zi3dyp/fXsXCjbk89PPRxPqb/7faok25AIzumcrQrin4fcb1M5fw00e+IDk+hhVbCiipCKw1DO6czOtXTSQhzv+d97jnnVUUllZy91lH8Ps3VnLNC4t56qKxxHgQb6SqHSlo+khEDlSndvFcNLEPr181kdtPG8J7mTv41b+8OTpp0cZcEuP8DDosUHbh1OFdeei8UewpraSq2vGzsT2Yfs4I/nb2CFbv2MMtr333ZLuFG3OZOX8zlxzdh5+N7cmfzhzGZ2uz+fPbq76zn+zCMh6Zs46PVu9s9j5EgrySwOgpkqePPB0pmNlk4D7ADzzunPtLvecvBO4FtgQ3Peice9zLmETC0YUT+1BR5bhr9tfE+oy/nTMSfzNejGXRpjxGdE/9zq/6ycO6MHlYl++13ZZfwl/fW8PIHqlcNLEPlVXV/O6NFXRJiee6EwcAcE5GDzK3FvDPz75lSJd2HN6lHU9+/i3/XrqV8spq2sT4ePXKoxjWLaXZ+hAJakYK7ZQUvs/M/MBDwA+ALGC+mc1yzmXWa/ov59w1XsUhEikuO7Yv5VXV3PvuamL9Pu4+a3izXKWrpLwqeNWvvk1qf9Wk/izZnM9d//maoV1TWLEln6+3FfDIz0eT1GbvV8atPz6cNTv2cNMrS6l2kBDr55yM7pwxshvXvbiYXz67kDevPZoOSZE7v36g8ksqSIj1Ex/r33/jMOXlSGEcsNY5tx7AzGYCU4D6SUFEgq4+vj9lldXc/8E3dGgbx29POfyQ33NZVh6V1Y7RPds3qb3PZ0z/2QimPPg5Vz2/iNKKKo4bmM7kYZ2/0y7W7+Oh80Zz6xvLGdE9lXPH9qw9FPMfF4zhp//4kmteWMQzF49rNesOecXlEb2eAN6uKXQDNtd5nBXcVt9ZZrbMzF4xsx4NvZGZXW5mC8xswa5du7yIVSRs/PqkAZw3viePfrye91Ye+tXbFgUv9jKqiUkBoF18LI9eMIbi8krKq6r54+lDG7wmRPukOB7++Rh+eVy/7xybP7x7KnedMYwv1u3m7ndWfe910SqvuIKUCJ46gtAvNL8J9HbODQfeB55uqJFzboZzLsM5l5Gent6iAYq0NDPjtlOHcES3FG54eSmbdhcf0vst2pRLn7SkA57GGXhYMs9dOp7Hp2XQOy3pgPd7dkYPpk3oxWOffsu/l2zZ/wuiQKSXuABvk8IWoO4v/+7sXVAGwDm32zlXFnz4ODDGw3hEIkZ8rJ+Hfz4aA658fiGlFQdXmsI5x+JNuYzqmXpQrx/dsz3HDjz4H2K/+/EQxvZuz/UzlzB1xlzeXr4tqms/5ZWUR3SJC/A2KcwHBphZHzOLA84FZtVtYGZ1D304Hfjaw3hEIkqPDolMP2ckK7cW8Mc3D24pbnNOCdmF5U1eT2hucTE+nrhwLDdPHsSmnGKufH4RR9/9EQ988M1BnVEd7jRS2AfnXCVwDfAugS/7l5xzK83sDjM7PdjsOjNbaWZLgeuAC72KRyQSnTTkMK44rh8vztvEa4uyDvj1e09aC01SAEiOj+WqSf355ObjmXHBGAYc1pa/vb+Gyfd9whfrskMWV3NzzpFXUhHRdY/A4/MUnHOzgdn1tt1W5/5vgd96GYNIpLvx5IEs2pTL/7y+nCFd2zG4c7smv3bhxlyS4vwM6hz6awX7fcbJQztz8tDOzPs2h5tfWcp5j33FBUf24pZTBn/ncNdIVFpRTXlltaaPRMRbMX4fD04dRXJ8LFc+t4g9pRVNfu2iTbmM6JHarCfCNYdxfTrw9vXHcsnRfXjuq42c/PdPWBwc1USq2rOZI3ykoKQgEgE6tYvnwamj2JRTzE0vN+16z8XllazaviekU0f7khDn5/enDuHlX07A54NpT8xjxZaWLyneXKKhQiooKYhEjPF9O3LL5MG8s3I7j3/67X7bL92cT1W1Y3SvgzvyqKVk9O7AzMsnkNwmhmlPzGPtzsJQh3RQ9hbD0/SRiLSQS4/pw+ShnfnLO6v4av3ufbatWWQe1SM8Rwp1dUtN4LlLx+MzOP/xr9icc2jnZoRCvqaPRKSlmRn3nj2cnh0SufqFxexs5NoHAIs35dI3LYn2EVJ7qG96W565eDzF5ZWc/8+vyMotpqiskj2lFeQXVxz0uRotJRrKZoOSgkjESY6P5R/nj6GorJKrX1hERQMng324agcfrtp5SCeehcKQru146uJx7NpTxtF3f8TQP7zLEbe/x4g73mPsn/7Lwo05oQ6xUXk1F9iJ8KOPIvsYMJFWalDnZP5y1hFcP3MJf3l7Fb8/dUjtc2t27OG6F5dweJd23Dx5UAijPDije7bn5Ssm8MmabPw+8JlhZjz75QYue2Yhr1151EGV3fBaXnEFcTE+4mMj+7e2koJIhJqesDFDAAANl0lEQVQyshuLNubyz8++ZVTPVE4d3pWconIueXo+CXF+Hv9FBolxkfknPrRrCkO7fvdaDCcM7sRPHv6ci56az2tXHhV202L5JeWkJsQ2WDgwkkR2ShNp5W798RBG90zl5leWkbm1gCueXcjOgjIem5ZBl5SEUIfXrPqkJfHYtIzaa0SH2xpDNJS4ACUFkYgWF+PjoZ+PJjHOzxkPfc68DTnc89PhjOwR3oehHqyM3h2Yfs4IFmzM5YaXl1Jdvf/zNVpKXnFFxK8ngJKCSMTrkpLA/VNHgcH1Jw5gysiGLlsSPU4d3pVbThnMf5Zt49h7P+Led1fxzY49oQ4rKuoegdYURKLCUf3SWHrbySTERe5lIA/EL4/tS9fUBF5esJlH5qzjoY/WMaRLO355XN+QJcX84nKGdW16XapwpaQgEiVaS0KAwPkap4/oyukjurJzTyn/WbaNlxZkcf3MJXy8ehd3njGsxQvs5ZVoTUFEJOQ6Jcdz0cQ+vHXt0fz6pIG8sWQLpz7wWYvWUSqrrKK4vCriS1yAkoKIRAm/z7j+pAG8cNmRlJRX8ZOHv+DRj9e1yMV88oMnrkX69ZlBSUFEosyRfTsy+/pjOHZgGn9+exVH/u8H3D5rJet2eVdoL1pKXIDWFEQkCnVIiuOxaRks2pTLM19u5PmvNvLUFxs4dmA695w1nM4p8c26v71lszV9JCISlsyMMb06cN+5o/jilhMDV7DbmMsF//yK3KLyZt1XXnF0VEgFJQURaQXSk9twzQkDeGxaBhtzirnwqfnNutaQpzUFEZHIM6FfRx6cOooVW/K5/NkFlFU2T6mM/ChaU1BSEJFW5eShnbnnrOF8vnY3v5q5hKpmKJWRV1KO32e0beFzI7ygpCAirc5ZY7rz+1OH8PaK7dzy6rJDrqEUqHsU+RVSQUcfiUgrdcnRfcgvqeD+D74hqU0MfzhtyEF/qUdL3SNQUhCRVuzXJw2guKySxz/7lqQ2fm764eCDep/84EghGigpiEirZWbc+uPDKSqv4qGP1pEYF8PVx/c/4PfJKymnU3LznvsQKkoKItKqmRl/OmMYJeWV3PvuahLj/Fw0sc8BvUdecQUDOyV7FGHLUlIQkVbP7zP+evYISiqq+OObmcT6fZx/ZK8mvz6/uCIqiuGBjj4SEQEgxu/jgamjOXFwJ373xgpmztvUpNdVVFWzp6wyKs5RACUFEZFacTE+Hj5/NMcNTOe3ry/nlYVZ+31NQUn0nLgGSgoiIt/RJsbPoxeM4ej+adz0ylJeX7zvxBBNJS5ASUFE5HviY/3MuCCDI/t05IaX9p0Y9pbN1prCfpnZZDNbbWZrzeyWfbQ7y8ycmWV4GY+ISFMlxPn554UZjO/Tkd+8tJRXG5lKyi8JVkjVSGHfzMwPPAScAgwBpprZkAbaJQPXA195FYuIyMFIjIvhiQvHMrFfGje+spSXFmz+XptousAOeDtSGAesdc6td86VAzOBKQ20uxO4Gyj1MBYRkYOSEOfn8V9kcHT/NP7fq8u+d1RSNF1gB7xNCt2Aumk1K7itlpmNBno45/7jYRwiIockPtbPY9MyOHZAOre8tpxn526sfS6vuBwzSI6PjtO+QrbQbGY+YDpwQxPaXm5mC8xswa5du7wPTkSknvjYwFFJJx3eid+/sYLHPlkPBIvhJcTi80V+hVTwNilsAXrUedw9uK1GMjAMmGNmG4AjgVkNLTY752Y45zKccxnp6ekehiwi0rj4WD+PnD+GHw/vwl2zv+b//ruG3CgqhgfelrmYDwwwsz4EksG5wHk1Tzrn8oG0msdmNge40Tm3wMOYREQOSazfx/3njiIh1s///fcb4mN9DOrcLtRhNRvPRgrOuUrgGuBd4GvgJefcSjO7w8xO92q/IiJe8/uMe84azgVH9qK0ojpqTlwDjwviOedmA7PrbbutkbaTvIxFRKQ5+XzGHVOG0qtjIr07JoU6nGYTHcvlIiIhYGZcekzfUIfRrFTmQkREaikpiIhILSUFERGppaQgIiK1lBRERKSWkoKIiNRSUhARkVpKCiIiUsucc6GO4YCY2S5gY73NKUD+frbt63FD99OA7EMItaGYDqRdU/pUf1tT7rdEv/bVprV8Vg1tb6wfdR/rszr4eJvarrn61dKf1b7aNaVPvZxz+68o6pyL+BswY3/b9vW4ofvAguaO6UDaNaVPTelHA/c979e+2rSWz+pA+lGvL/qsPPysmrNfLf1Z7atdU78vmnKLlumjN5uwbV+PG7t/KJr6Po21a0qf6m/zuk9Nfa99tWktn1VD2/cV+5uNbD8U+qya/tzB9KulP6t9tWvq98V+Rdz0UUsxswXOue9d2yHSRWO/orFPEJ39isY+QXT1K1pGCl6YEeoAPBKN/YrGPkF09isa+wRR1C+NFEREpJZGCiIiUqtVJAUze8LMdprZioN47RgzW25ma83sfjOzOs9da2arzGylmd3TvFHvN65m75OZ3W5mW8xsSfD2o+aPfL+xefJZBZ+/wcycmaU19h5e8ejzutPMlgU/q/fMrGvzR77PuLzo073Bv6llZva6maU2f+T7jc2Lfp0d/J6obug69GHlUA6jipQbcCwwGlhxEK+dBxwJGPA2cEpw+/HAf4E2wcedoqBPtxO4TnZUfVbB53oQuDTsRiAtGvoFtKvT5jrgH1HQp5OBmOD9u4G7o+SzOhwYBMwBMlq6TwdyaxUjBefcJ0BO3W1m1s/M3jGzhWb2qZkNrv86M+tC4A9vrgt8ss8AZwSfvhL4i3OuLLiPnd724rs86lPIedivvwM3AyFZRPOiX865gjpNk2jhvnnUp/dc4PruAHOB7t724vs86tfXzrnVLRH/oWoVSaERM4BrnXNjgBuBhxto0w3IqvM4K7gNYCBwjJl9ZWYfm9lYT6NtmkPtE8A1waH7E2bW3rtQD8gh9cvMpgBbnHNLvQ70AB3y52Vmd5nZZuDnQIPXP29hzfF/sMbFBH5th4Pm7FdYa5XXaDaztsBRwMt1pp3bHODbxAAdCAwVxwIvmVnf4C+EFtdMfXoEuJPAL847gb8R+MMMmUPtl5klAv9DYFoibDTT54Vz7lbgVjP7LXAN8IdmC/IANVefgu91K1AJPN880R285uxXJGiVSYHACCnPOTey7kYz8wMLgw9nEfiSrDt87Q5sCd7PAl4LJoF5ZlZNoP7JLi8D34dD7pNzbked1z0GvOVlwE10qP3qB/QBlgb/oLsDi8xsnHNuu8ex70tz/B+s63lgNiFMCjRTn8zsQuBU4MRQ/ciqp7k/q/AW6kWNlroBvamzcAR8AZwdvG/AiEZeV3/h6EfB7VcAdwTvDwQ2EzzvI4L71KVOm18DM6Phs6rXZgMhWGj26PMaUKfNtcArUdCnyUAmkB6Kz8jr/4NEwEJzyANooQ/4RWAbUEHgF/4lBH49vgMsDf4nvK2R12YAK4B1wIM1X/xAHPBc8LlFwAlR0KdngeXAMgK/fLq0VH+87Fe9NiFJCh59Xq8Gty8jUOemWxT0aS2BH1hLgrcWPaLKw36dGXyvMmAH8G5L96upN53RLCIitVrz0UciIlKPkoKIiNRSUhARkVpKCiIiUktJQUREaikpSMQzs8IW3t/jZjakmd6rKljldIWZvbm/qqBmlmpmVzXHvkUaokNSJeKZWaFzrm0zvl+M21uUzVN1Yzezp4E1zrm79tG+N/CWc25YS8QnrY9GChKVzCzdzF41s/nB28Tg9nFm9qWZLTazL8xsUHD7hWY2y8w+BD4ws0lmNsfMXgnW93++Tm38OTU18c2sMFiUbqmZzTWzw4Lb+wUfLzezPzVxNPMle4v4tTWzD8xsUfA9pgTb/AXoFxxd3Btse1Owj8vM7I/N+M8orZCSgkSr+4C/O+fGAmcBjwe3rwKOcc6NIlBV9H/rvGY08FPn3HHBx6OAXwFDgL7AxAb2kwTMdc6NAD4BLquz//ucc0fw3cqZDQrW0TmRwJnkAKXAmc650QSu3fG3YFK6BVjnnBvpnLvJzE4GBgDjgJHAGDM7dn/7E2lMay2IJ9HvJGBInaqW7YLVLlOAp81sAIFqsLF1XvO+c65uHf15zrksADNbQqAezmf19lPO3sKBC4EfBO9PYO/1HF4A/tpInAnB9+4GfA28H9xuwP8Gv+Crg88f1sDrTw7eFgcftyWQJD5pZH8i+6SkINHKBxzpnCutu9HMHgQ+cs6dGZyfn1Pn6aJ671FW534VDf+9VLi9C3ONtdmXEufcyGCJ73eBq4H7CVwfIR0Y45yrMLMNQHwDrzfgz865Rw9wvyIN0vSRRKv3CFQOBcDMasoep7C3nPGFHu5/LoFpK4Bz99fYOVdM4JKaN5hZDIE4dwYTwvFAr2DTPUBynZe+C1wcHAVhZt3MrFMz9UFaISUFiQaJZpZV5/YbAl+wGcHF10wCpc4B7gH+bGaL8Xak/CvgN2a2DOgP5O/vBc65xQQqnk4lcH2EDDNbDkwjsBaCc2438HnwENZ7nXPvEZie+jLY9hW+mzREDogOSRXxQHA6qMQ558zsXGCqc27K/l4nEmpaUxDxxhjgweARQ3mE+LKmIk2lkYKIiNTSmoKIiNRSUhARkVpKCiIiUktJQUREaikpiIhILSUFERGp9f8BIfpBZKcT3g8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.recorder.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now to actually train the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.397212</td>\n",
       "      <td>0.235468</td>\n",
       "      <td>02:23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learner.fit_one_cycle(config.epochs, max_lr=config.max_lr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See how simple that was?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now to generate predictions. This is where you can get tripped up because the `databunch` does not load data in sorted order. So we'll have to do reorder the generated predictions to match their original order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_preds_as_nparray(ds_type) -> np.ndarray:\n",
    "    \"\"\"\n",
    "    the get_preds method does not yield the elements in order by default\n",
    "    we borrow the code from the RNNLearner to resort the elements into their correct order\n",
    "    \"\"\"\n",
    "    preds = learner.get_preds(ds_type)[0].detach().cpu().numpy()\n",
    "    sampler = [i for i in databunch.dl(ds_type).sampler]\n",
    "    reverse_sampler = np.argsort(sampler)\n",
    "    return preds[reverse_sampler, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_preds = get_preds_as_nparray(DatasetType.Test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can generate a submission if you like, though you'll probably want to use a different set of configurations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sample_submission = pd.read_csv(DATA_ROOT / \"sample_submission.csv\")\n",
    "# if config.testing: sample_submission = sample_submission.head(test.shape[0])\n",
    "# sample_submission[label_cols] = test_preds\n",
    "# sample_submission.to_csv(\"predictions.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}