Assignment 1 - NLP.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import matplotlib\n",
    "from collections import Counter\n",
    "%matplotlib inline\n",
    "from collections import *\n",
    "from random import random\n",
    "import nltk, re, pprint\n",
    "from nltk import word_tokenize, sent_tokenize\n",
    "from collections import Counter\n",
    "import numpy as np\n",
    "import re\n",
    "from nltk.probability import FreqDist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 1: \n",
    "# Topics in Natural Language Processing (202-2-5381) Fall 2018\n",
    "# Prof. Michael Elhadad\n",
    "\n",
    "### Author: Dina Svetlitsky\n",
    "\n",
    "[Assignment description](https://www.cs.bgu.ac.il/~elhadad/nlp18/hw1.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Language Models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part, we will develop two language models and evaluate the performance of the models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Data Exploration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1.1 Gathering and Cleaning Up Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, lets get some data (the book War and Peace be Leo Tolstoy, which I downloaded from Project Gutenberg):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_text_file = open(\"war_and_peace.txt\", \"r\", encoding=\"utf8\")\n",
    "text = input_text_file.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A quick peek into the text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ubt my ability before such an audience,” said he,\\nsmilingly inclining his head.\\n\\nThe princess rested her bare round arm on a little table and considered\\na reply unnecessary. She smilingly waited. All the time the story was\\nbeing told she sat upright, glancing now at her beautiful round arm,\\naltered in shape by its pressure on the table, now at her still more\\nbeautiful bosom, on which she readjusted a diamond necklace. From time\\nto time she smoothed the folds of her dress, and whenever the story\\nproduced an effect she glanced at Anna Pávlovna, at once adopted just\\nthe expression she saw on the maid of honor’s face, and again relapsed\\ninto her radiant smile.\\n\\nThe little princess had also left the tea table and followed Hélène.\\n\\n“Wait a moment, I’ll get my work.... Now then, what are you\\nthinking of?” she went on, turning to Prince Hippolyte. “Fetch me my\\nworkbag.”\\n\\nThere was a general movement as the princess, smiling and talking\\nmerrily to everyone at once, sat down and gaily arranged h'"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text[30000:31000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Replace wierd quotation or apostrophe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "he said: \"don't\"\n"
     ]
    }
   ],
   "source": [
    "s = \"he said: “don‘t”\"\n",
    "s = re.sub(\"‘|’\", \"\\'\", s)\n",
    "s = re.sub(\"“|”\", '\\\"', s)\n",
    "print(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ubt my ability before such an audience,\" said he,\\nsmilingly inclining his head.\\n\\nThe princess rested her bare round arm on a little table and considered\\na reply unnecessary. She smilingly waited. All the time the story was\\nbeing told she sat upright, glancing now at her beautiful round arm,\\naltered in shape by its pressure on the table, now at her still more\\nbeautiful bosom, on which she readjusted a diamond necklace. From time\\nto time she smoothed the folds of her dress, and whenever the story\\nproduced an effect she glanced at Anna Pávlovna, at once adopted just\\nthe expression she saw on the maid of honor\\'s face, and again relapsed\\ninto her radiant smile.\\n\\nThe little princess had also left the tea table and followed Hélène.\\n\\n\"Wait a moment, I\\'ll get my work.... Now then, what are you\\nthinking of?\" she went on, turning to Prince Hippolyte. \"Fetch me my\\nworkbag.\"\\n\\nThere was a general movement as the princess, smiling and talking\\nmerrily to everyone at once, sat down and gaily arranged h'"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text = re.sub(\"‘|’\", \"\\'\", text)\n",
    "text = re.sub(\"“|”\", '\\\"', text)\n",
    "text = re.sub(\"``\", \"''\", text)\n",
    "\n",
    "text[30000:31000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "word_tokenize is a helpful function for text tokenization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tokens = word_tokenize(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many tokens do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "668651"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at the first few tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\ufeff',\n",
       " 'The',\n",
       " 'Project',\n",
       " 'Gutenberg',\n",
       " 'EBook',\n",
       " 'of',\n",
       " 'War',\n",
       " 'and',\n",
       " 'Peace',\n",
       " ',']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is pretty good, but we want to get rid of punctuations and tokens in lower case:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "PUNCTUATION = r\"...?!:;--()[]\\''\\\",``\"\n",
    "tokens = [token.lower() for token in tokens if token not in PUNCTUATION]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have less tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "572491"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can put our tokens in a Counter data structure, which will count occurrences of each token:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "tokens_counter = Counter(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many distinct tokens do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20519"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokens_counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now lets take just the 10,000 most commons tokens, we will use it in cleaning up the data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('the', 34556),\n",
       " ('and', 22150),\n",
       " ('to', 16706),\n",
       " ('of', 14990),\n",
       " ('a', 10491),\n",
       " ('he', 9930),\n",
       " ('in', 8928),\n",
       " ('that', 8067),\n",
       " ('his', 7966),\n",
       " ('was', 7339)]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_common_tokens = tokens_counter.most_common(10000)\n",
    "most_common_tokens[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want to seperate our text to sentences, and then each sentence to tokens. We can use nltk.sent_tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\ufeff',\n",
       " 'The',\n",
       " 'Project',\n",
       " 'Gutenberg',\n",
       " 'EBook',\n",
       " 'of',\n",
       " 'War',\n",
       " 'and',\n",
       " 'Peace',\n",
       " ',',\n",
       " 'by',\n",
       " 'Leo',\n",
       " 'Tolstoy',\n",
       " 'This',\n",
       " 'eBook',\n",
       " 'is',\n",
       " 'for',\n",
       " 'the',\n",
       " 'use',\n",
       " 'of',\n",
       " 'anyone',\n",
       " 'anywhere',\n",
       " 'at',\n",
       " 'no',\n",
       " 'cost',\n",
       " 'and',\n",
       " 'with',\n",
       " 'almost',\n",
       " 'no',\n",
       " 'restrictions',\n",
       " 'whatsoever',\n",
       " '.']"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents = nltk.sent_tokenize(text)\n",
    "sents = [word_tokenize(sent) for sent in sents]\n",
    "sents[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also want to mark numbers as 'N':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "N\n",
      "N\n"
     ]
    }
   ],
   "source": [
    "NUMBERS = '0123456789'\n",
    "n1 = \"123\"\n",
    "n2 = \"800.7655\"\n",
    "print(re.sub('\\d+\\.\\d+|\\d+', 'N', n1))\n",
    "print(re.sub('\\d+\\.\\d+|\\d+', 'N', n2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All good. Now we will do everything we have done above, using a general function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sent_transformation(sent, most_common_tokens_set):\n",
    "    \"\"\"gets a tokenized sentence and returns a cleaned concatanated sentance\"\"\"\n",
    "    cleaned_sentence = []\n",
    "    for token in sent:\n",
    "        if token not in PUNCTUATION:\n",
    "            token = token.lower()\n",
    "            if re.search('\\d', token):\n",
    "                token = re.sub('\\d+\\.\\d+|\\d+', 'N', token)#replace numbers with 'N'\n",
    "            else:\n",
    "                if token not in most_common_tokens_set:\n",
    "                    token = \"<unk>\"\n",
    "            cleaned_sentence.append(token)\n",
    "    return \" \".join(cleaned_sentence)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Test our sent_transformation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Buonapartes', '.', 'But', 'I', 'warn', 'you', ',', 'if', 'you', 'do', \"n't\", 'tell', 'me', 'that', 'this', 'means', 'war', '.']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\"<unk> but i warn you if you do n't tell me that this means war\""
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sent = \"Buonapartes. But I warn you, if you don't tell me that this means war.\"\n",
    "print(word_tokenize(sent))\n",
    "sent_transformation(word_tokenize(sent), set([item[0] for item in most_common_tokens]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def get_common_tokens_set(text, top):\n",
    "    \"\"\"tokenized text to lowercase tokens without punctuations, returns a set of the top most common tokens\"\"\"\n",
    "    tokens = word_tokenize(text)\n",
    "    tokens = [token.lower() for token in tokens if token not in PUNCTUATION]#remove punctuation and lowecase\n",
    "    tokens_counter = Counter(tokens)\n",
    "    most_common_tokens = tokens_counter.most_common(top)\n",
    "    \n",
    "    return set([item[0] for item in most_common_tokens])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def process_file(filename, top):\n",
    "    input_text_file = open(filename, \"r\", encoding=\"utf8\")\n",
    "    text = input_text_file.read()\n",
    "    \n",
    "    common_tokens_set = get_common_tokens_set(text, top)\n",
    "    \n",
    "    sents = nltk.sent_tokenize(text)\n",
    "    sents = [word_tokenize(sent) for sent in sents]\n",
    "    \n",
    "    output_file = open(filename+\".out\", \"w\", encoding=\"utf8\")\n",
    "    for sent in sents:\n",
    "        output_file.write(sent_transformation(sent, common_tokens_set) + \"\\n\")\n",
    "    output_file.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ptb_preprocess takes a list of filenames and for each file dataset:\n",
    "- Keeps only the top most frequent words in the dataset\n",
    "- Replaces all numbers by the token N\n",
    "- Tokenizes the words\n",
    "- Removes all punctuations\n",
    "- Segments the sentencess and prints them one by line\n",
    "- Outputs all the text in lowercase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def ptb_preprocess(filenames, top=10000):\n",
    "    for filename in filenames:\n",
    "        process_file(filename, top)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "ptb_preprocess([\"war_and_peace.txt\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at some cleaned sentences from the file that was created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['with these words she greeted prince vasíli kurágin a man of high rank and importance who was the first to arrive at her reception\\n',\n",
       " 'anna pávlovna had had a cough for some days\\n',\n",
       " 'she was as she said suffering from la grippe grippe being then a new word in st. petersburg used only by the <unk>\\n']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "output_file = open(\"war_and_peace.txt.out\", \"r\", encoding=\"utf8\")\n",
    "text = output_file.readlines()\n",
    "text[5:8]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1.1.2 Gathering Basic Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The Penn Tree Bank (PTB) dataset, which contains 929k training words, 73k validation words, and 82k test words, was peocessed by Tomas Mikolov in the same way as in part 1.1.2.\n",
    "\n",
    "We will get the train, validation and test data of words and characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentences data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_dataset_file_train = open(r\"penn_treebank\\data\\ptb.train.txt\", \"r\", encoding=\"utf8\")\n",
    "input_dataset_file_valid = open(r\"penn_treebank\\data\\ptb.valid.txt\", \"r\", encoding=\"utf8\")\n",
    "input_dataset_file_test = open(r\"penn_treebank\\data\\ptb.test.txt\", \"r\", encoding=\"utf8\")\n",
    "words_train_data = input_dataset_file_train.read()\n",
    "words_valid_data = input_dataset_file_valid.read()\n",
    "words_test_data = input_dataset_file_test.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Characters data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "input_dataset_file_train = open(r\"penn_treebank\\data\\ptb.char.train.txt\", \"r\", encoding=\"utf8\")\n",
    "input_dataset_file_valid = open(r\"penn_treebank\\data\\ptb.char.valid.txt\", \"r\", encoding=\"utf8\")\n",
    "input_dataset_file_test = open(r\"penn_treebank\\data\\ptb.char.test.txt\", \"r\", encoding=\"utf8\")\n",
    "chars_train_data = input_dataset_file_train.read()\n",
    "chars_valid_data = input_dataset_file_valid.read()\n",
    "chars_test_data = input_dataset_file_test.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets look at some sentences in the train dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[' aer banknote berlitz calloway centrust cluett fromstein gitano guterman hydro-quebec ipo kia memotec mlx nahb punts rake regatta rubens sim snack-food ssangyong swapo wachter ',\n",
       " ' pierre <unk> N years old will join the board as a nonexecutive director nov. N ',\n",
       " ' mr. <unk> is chairman of <unk> n.v. the dutch publishing group ']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sents = words_train_data.split(\"\\n\")\n",
    "sents[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And some tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['aer',\n",
       " 'banknote',\n",
       " 'berlitz',\n",
       " 'calloway',\n",
       " 'centrust',\n",
       " 'cluett',\n",
       " 'fromstein',\n",
       " 'gitano',\n",
       " 'guterman',\n",
       " 'hydro-quebec']"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens = words_train_data.split()\n",
    "tokens[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we gather statistics on words:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Total number of tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "887521"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "token_counter = Counter(tokens)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Total number of distinct words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "9999"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(token_counter)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "token/word ratio:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "88.76097609760976"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokens)/float(len(token_counter))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each token repeats 88 times on average"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7821775484749093"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum([item[1] for item in token_counter.most_common(1000)])/float(len(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1000 most frequent words correspond to almost 80% of the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Average number of characters per token:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4.6533670752579379"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.average([len(token) for token in tokens])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standard deviation of number of charachters per token:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2.6019161500774306"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.std([len(token) for token in tokens])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will plot the number of distinct word ngrams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def distinct_ngrams(tokens, n):\n",
    "    nmers = Counter()\n",
    "    for i in range(len(tokens)-n):\n",
    "        nmer = \" \".join(tokens[i:i+n])\n",
    "        nmers[nmer] += 1\n",
    "    return len(nmers)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x2ccf5a72048>"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZwAAAEKCAYAAAAmfuNnAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG1hJREFUeJzt3X+MVed95/H3J+A45AcO2LOUAu6gNdsVthKnjDBKVitv\n2ABpssHb2u5Ecj2bItOuva2zqtRCtVpae5FsqVs3Tmqv2JgY3DQ2dZIaJaHuFCfbRqrBg+PWAQcx\nG9s1LDZThkDS1rTQz/5xn6kvN/PjMuGc6xk+L+nqnvs953nmeTiWPzrnPnNGtomIiKjaWzo9gIiI\nuDgkcCIiohYJnIiIqEUCJyIiapHAiYiIWiRwIiKiFgmciIioRQInIiJqUWngSPqvkvZL+rakL0h6\nm6S5kvolHSrvc5qO3yhpUNJBSaub6sskPV/23S9JpX6ppMdKfY+k7qY2feVnHJLUV+U8IyJiYqrq\nSQOSFgDfBJba/ntJO4CvAUuBYdv3SNoAzLH965KWAl8AlgM/Dvwp8K9sn5W0F/gVYE/p437buyTd\nDrzH9i9J6gX+o+2fkzQXGAB6AAP7gGW2T4w13iuuuMLd3d2V/FtERExX+/bt+xvbXe0cO7PiscwE\nZkn6R+DtwP8DNgLXl/3bgG8Avw6sBR61fRp4UdIgsFzSS8Bs208DSNoO3ADsKm1+s/T1OPCZcvWz\nGui3PVza9ANraATaqLq7uxkYGLggk46IuFhIerndYyu7pWb7CPDbwF8DR4GTtv8EmGf7aDnsVWBe\n2V4AvNLUxeFSW1C2W+vntLF9BjgJXD5OXxER0SGVBU75bmYtsJjGLbJ3SLql+Rg37ud17OmhktZL\nGpA0MDQ01KlhRERcFKpcNPDvgRdtD9n+R+BLwPuB1yTNByjvx8rxR4BFTe0XltqRst1aP6eNpJnA\nZcDxcfo6h+0ttnts93R1tXULMiIiJqnKwPlrYIWkt5fvVVYCLwA7gZFVY33AE2V7J9BbVp4tBpYA\ne8vtt1OSVpR+bm1pM9LXjcBT5arpSWCVpDnlSmtVqUVERIdUtmjA9h5JjwPPAmeAbwFbgHcCOySt\nA14Gbi7H7y8r2Q6U4++wfbZ0dzvwMDCLxmKBXaX+EPBIWWAwDPSWvoYl3Q08U467a2QBQUREdEZl\ny6Knmp6eHmeVWkTE+ZG0z3ZPO8fmSQMREVGLBE5ERNQigRMREbWo+kkDEREXRPeGr3Z6CNPWS/d8\npJafkyuciIioRQInIiJqkcCJiIhaJHAiIqIWCZyIiKhFAiciImqRwImIiFokcCIiohYJnIiIqEUC\nJyIiapHAiYiIWiRwIiKiFgmciIioRQInIiJqUVngSPpJSc81vU5J+qSkuZL6JR0q73Oa2myUNCjp\noKTVTfVlkp4v++6XpFK/VNJjpb5HUndTm77yMw5J6qtqnhER0Z7KAsf2QdvX2r4WWAb8HfBlYAOw\n2/YSYHf5jKSlQC9wNbAGeEDSjNLdg8BtwJLyWlPq64ATtq8C7gPuLX3NBTYB1wHLgU3NwRYREfWr\n65baSuD/2n4ZWAtsK/VtwA1ley3wqO3Ttl8EBoHlkuYDs20/bdvA9pY2I309DqwsVz+rgX7bw7ZP\nAP28EVIREdEBdQVOL/CFsj3P9tGy/Sowr2wvAF5panO41BaU7db6OW1snwFOApeP01dERHRI5YEj\n6a3Ax4A/bN1Xrlhc9RjGImm9pAFJA0NDQ50aRkTERaGOK5wPA8/afq18fq3cJqO8Hyv1I8CipnYL\nS+1I2W6tn9NG0kzgMuD4OH2dw/YW2z22e7q6uiY9wYiImFgdgfNx3ridBrATGFk11gc80VTvLSvP\nFtNYHLC33H47JWlF+X7m1pY2I33dCDxVrpqeBFZJmlMWC6wqtYiI6JCZVXYu6R3Ah4BfbCrfA+yQ\ntA54GbgZwPZ+STuAA8AZ4A7bZ0ub24GHgVnArvICeAh4RNIgMEzjuyJsD0u6G3imHHeX7eFKJhkR\nEW2pNHBs/y2NL/Gba8dprFob7fjNwOZR6gPANaPUXwduGqOvrcDW8x91RERUIU8aiIiIWiRwIiKi\nFgmciIioRQInIiJqkcCJiIhaJHAiIqIWCZyIiKhFAiciImqRwImIiFokcCIiohYJnIiIqEUCJyIi\napHAiYiIWiRwIiKiFgmciIioRQInIiJqkcCJiIhaJHAiIqIWlf6JaUnvBj5L489DG/gF4CDwGNAN\nvATcbPtEOX4jsA44C/yK7SdLfRnwMDAL+Bpwp21LuhTYDiwDjgM/Z/ul0qYP+G9lKP/D9rYq5xpT\nS/eGr3Z6CNPWS/d8pNNDiDepqq9wPgX8se1/DbwXeAHYAOy2vQTYXT4jaSnQC1wNrAEekDSj9PMg\ncBuwpLzWlPo64ITtq4D7gHtLX3OBTcB1wHJgk6Q51U41IiLGU1ngSLoM+LfAQwC2/8H294C1wMjV\nxjbghrK9FnjU9mnbLwKDwHJJ84HZtp+2bRpXNM1tRvp6HFgpScBqoN/2cLl66ueNkIqIiA6o8gpn\nMTAEfE7StyR9VtI7gHm2j5ZjXgXmle0FwCtN7Q+X2oKy3Vo/p43tM8BJ4PJx+oqIiA6pMnBmAj8F\nPGj7fcDfUm6fjShXLK5wDOOStF7SgKSBoaGhTg0jIuKiUGXgHAYO295TPj9OI4BeK7fJKO/Hyv4j\nwKKm9gtL7UjZbq2f00bSTOAyGosHxurrHLa32O6x3dPV1TXJaUZERDsqCxzbrwKvSPrJUloJHAB2\nAn2l1gc8UbZ3Ar2SLpW0mMbigL3l9tspSSvK9zO3trQZ6etG4Kly1fQksErSnLJYYFWpRUREh1S6\nLBr4ZeDzkt4KfBf4BI2Q2yFpHfAycDOA7f2SdtAIpTPAHbbPln5u541l0bvKCxoLEh6RNAgM01jl\nhu1hSXcDz5Tj7rI9XOVEIyJifJUGju3ngJ5Rdq0c4/jNwOZR6gM0fpentf46cNMYfW0Ftp7PeCMi\nojp50kBERNQigRMREbVI4ERERC0SOBERUYsETkRE1CKBExERtUjgRERELRI4ERFRiwRORETUIoET\nERG1SOBEREQtEjgREVGLBE5ERNQigRMREbVI4ERERC0SOBERUYsETkRE1CKBExERtag0cCS9JOl5\nSc9JGii1uZL6JR0q73Oajt8oaVDSQUmrm+rLSj+Dku6XpFK/VNJjpb5HUndTm77yMw5J6qtynhER\nMbE6rnD+ne1rbfeUzxuA3baXALvLZyQtBXqBq4E1wAOSZpQ2DwK3AUvKa02prwNO2L4KuA+4t/Q1\nF9gEXAcsBzY1B1tERNSvE7fU1gLbyvY24Iam+qO2T9t+ERgElkuaD8y2/bRtA9tb2oz09Tiwslz9\nrAb6bQ/bPgH080ZIRUREB1QdOAb+VNI+SetLbZ7to2X7VWBe2V4AvNLU9nCpLSjbrfVz2tg+A5wE\nLh+nr4iI6JCZFff/b2wfkfQvgH5J32neaduSXPEYxlRCcD3AlVde2alhRERcFCq9wrF9pLwfA75M\n4/uU18ptMsr7sXL4EWBRU/OFpXakbLfWz2kjaSZwGXB8nL5ax7fFdo/tnq6urslPNCIiJlRZ4Eh6\nh6R3jWwDq4BvAzuBkVVjfcATZXsn0FtWni2msThgb7n9dkrSivL9zK0tbUb6uhF4qnzP8ySwStKc\nslhgValFRESHVHlLbR7w5bKCeSbwB7b/WNIzwA5J64CXgZsBbO+XtAM4AJwB7rB9tvR1O/AwMAvY\nVV4ADwGPSBoEhmmscsP2sKS7gWfKcXfZHq5wrhERMYHKAsf2d4H3jlI/Dqwco81mYPMo9QHgmlHq\nrwM3jdHXVmDr+Y06IiKqkicNRERELRI4ERFRiwRORETUIoETERG1SOBEREQtEjgREVGLBE5ERNQi\ngRMREbVoK3Ak7W6nFhERMZZxnzQg6W3A24EryjPJVHbNJo/7j4iI8zDRo21+Efgk8OPAPt4InFPA\nZyocV0RETDPjBo7tTwGfkvTLtj9d05giImIaauvhnbY/Len9QHdzG9vbKxpXRERMM20FjqRHgH8J\nPAeM/MkAAwmciIhoS7t/nqAHWFr+uFlERMR5a/f3cL4N/FiVA4mIiOmt3SucK4ADkvYCp0eKtj9W\nyagiImLaaTdwfrPKQURExPTX1i012/9ntFc7bSXNkPQtSV8pn+dK6pd0qLzPaTp2o6RBSQclrW6q\nL5P0fNl3vySV+qWSHiv1PZK6m9r0lZ9xSFJfe/8cERFRlXYfbfN9SafK63VJZyWdavNn3Am80PR5\nA7Db9hJgd/mMpKVAL3A1sAZ4QNKM0uZB4DZgSXmtKfV1wAnbVwH3AfeWvuYCm4DrgOXApuZgi4iI\n+rV7hfMu27NtzwZmAT8LPDBRO0kLgY8An20qrwW2le1twA1N9Udtn7b9IjAILJc0H5ht++mySm57\nS5uRvh4HVparn9VAv+1h2yeAft4IqYiI6IDzflq0G/6Ixv/UJ/K7wK8B/9RUm2f7aNl+FZhXthcA\nrzQdd7jUFpTt1vo5bWyfAU4Cl4/T1zkkrZc0IGlgaGiojelERMRktfuLnz/T9PEtNH4v5/UJ2nwU\nOGZ7n6TrRzvGtiV17Hd7bG8BtgD09PTkd4wiIirU7iq1/9C0fQZ4icbtrPF8APiYpJ8G3gbMlvT7\nwGuS5ts+Wm6XHSvHHwEWNbVfWGpHynZrvbnNYUkzgcuA46V+fUubb0w4y4iIqEy73+F8oul1m+3N\nto9N0Gaj7YW2u2ksBnjK9i3ATmBk1Vgf8ETZ3gn0lpVni2ksDthbbr+dkrSifD9za0ubkb5uLD/D\nwJPAKklzymKBVaUWEREd0u4ttYXAp2lctQD8OXCn7cNjtxrTPcAOSeuAl4GbAWzvl7QDOEDjKuoO\n2yPPbbsdeJjGgoVd5QXwEPCIpEFgmEawYXtY0t3AM+W4u2wPT2KsERFxgbR7S+1zwB8AN5XPt5Ta\nh9ppbPsblFtato8DK8c4bjOweZT6AHDNKPXXm8bUum8rsLWd8UVERPXaXaXWZftzts+U18NAV4Xj\nioiIaabdwDku6Zby1IAZkm6h8eV8REREW9oNnF+g8V3Lq8BRGl/Q/6eKxhQREdNQu9/h3AX0ld/a\nH3l0zG/TCKKIiIgJtXuF856RsIHGKjDgfdUMKSIipqN2A+ctLU91nkv7V0cRERFth8b/BP5C0h+W\nzzcxyvLliIiIsbQVOLa3SxoAPlhKP2P7QHXDioiI6abt22IlYBIyERExKef95wkiIiImI4ETERG1\nSOBEREQtEjgREVGLBE5ERNQigRMREbVI4ERERC0SOBERUYsETkRE1KKywJH0Nkl7Jf2lpP2SfqvU\n50rql3SovDc/FHSjpEFJByWtbqovk/R82Xe/JJX6pZIeK/U9krqb2vSVn3FIUl9V84yIiPZUeYVz\nGvig7fcC1wJrJK0ANgC7bS8BdpfPSFoK9AJXA2uAByTNKH09CNwGLCmvNaW+Djhh+yrgPuDe0tdc\nYBNwHbAc2NQcbBERUb/KAscNPygfLykvA2uBbaW+DbihbK8FHrV92vaLwCCwXNJ8YLbtp20b2N7S\nZqSvx4GV5epnNdBve7j8HZ9+3gipiIjogEq/w5E0Q9JzwDEaAbAHmGf7aDnkVWBe2V4AvNLU/HCp\nLSjbrfVz2tg+A5wELh+nr9bxrZc0IGlgaGho0vOMiIiJVRo4ts/avhZYSONq5ZqW/aZx1dMRtrfY\n7rHd09XV1alhRERcFGpZpWb7e8DXadzWeq3cJqO8HyuHHQEWNTVbWGpHynZr/Zw2kmYClwHHx+kr\nIiI6pMpVal2S3l22ZwEfAr4D7ARGVo31AU+U7Z1Ab1l5tpjG4oC95fbbKUkryvczt7a0GenrRuCp\nctX0JLBK0pyyWGBVqUVERIe0/QfYJmE+sK2sNHsLsMP2VyT9BbBD0jrgZeBmANv7Je2g8UfezgB3\n2D5b+rodeBiYBewqL4CHgEckDQLDNFa5YXtY0t3AM+W4u2wPVzjXiIiYQGWBY/uvgPeNUj8OrByj\nzWZg8yj1AeCaUeqvAzeN0ddWYOv5jToiIqpS5RXORaV7w1c7PYRp66V7PtLpIUTEBZBH20RERC0S\nOBERUYsETkRE1CKBExERtUjgRERELRI4ERFRiwRORETUIoETERG1SOBEREQtEjgREVGLBE5ERNQi\ngRMREbVI4ERERC0SOBERUYsETkRE1CKBExERtagscCQtkvR1SQck7Zd0Z6nPldQv6VB5n9PUZqOk\nQUkHJa1uqi+T9HzZd78klfqlkh4r9T2Supva9JWfcUhSX1XzjIiI9lR5hXMG+FXbS4EVwB2SlgIb\ngN22lwC7y2fKvl7gamAN8ICkGaWvB4HbgCXltabU1wEnbF8F3AfcW/qaC2wCrgOWA5uagy0iIupX\nWeDYPmr72bL9feAFYAGwFthWDtsG3FC21wKP2j5t+0VgEFguaT4w2/bTtg1sb2kz0tfjwMpy9bMa\n6Lc9bPsE0M8bIRURER1Qy3c45VbX+4A9wDzbR8uuV4F5ZXsB8EpTs8OltqBst9bPaWP7DHASuHyc\nviIiokMqDxxJ7wS+CHzS9qnmfeWKxVWPYSyS1ksakDQwNDTUqWFERFwUKg0cSZfQCJvP2/5SKb9W\nbpNR3o+V+hFgUVPzhaV2pGy31s9pI2kmcBlwfJy+zmF7i+0e2z1dXV2TnWZERLShylVqAh4CXrD9\nO027dgIjq8b6gCea6r1l5dliGosD9pbbb6ckrSh93trSZqSvG4GnylXTk8AqSXPKYoFVpRYRER0y\ns8K+PwD8PPC8pOdK7TeAe4AdktYBLwM3A9jeL2kHcIDGCrc7bJ8t7W4HHgZmAbvKCxqB9oikQWCY\nxio3bA9Luht4phx3l+3hqiYaERETqyxwbH8T0Bi7V47RZjOweZT6AHDNKPXXgZvG6GsrsLXd8UZE\nRLXypIGIiKhFAiciImqRwImIiFokcCIiohYJnIiIqEUCJyIiapHAiYiIWiRwIiKiFgmciIioRQIn\nIiJqkcCJiIhaJHAiIqIWCZyIiKhFAiciImqRwImIiFokcCIiohYJnIiIqEUCJyIialFZ4EjaKumY\npG831eZK6pd0qLzPadq3UdKgpIOSVjfVl0l6vuy7X5JK/VJJj5X6HkndTW36ys84JKmvqjlGRET7\nqrzCeRhY01LbAOy2vQTYXT4jaSnQC1xd2jwgaUZp8yBwG7CkvEb6XAecsH0VcB9wb+lrLrAJuA5Y\nDmxqDraIiOiMygLH9p8Bwy3ltcC2sr0NuKGp/qjt07ZfBAaB5ZLmA7NtP23bwPaWNiN9PQ6sLFc/\nq4F+28O2TwD9/HDwRUREzer+Dmee7aNl+1VgXtleALzSdNzhUltQtlvr57SxfQY4CVw+Tl8REdFB\nHVs0UK5Y3KmfDyBpvaQBSQNDQ0OdHEpExLRXd+C8Vm6TUd6PlfoRYFHTcQtL7UjZbq2f00bSTOAy\n4Pg4ff0Q21ts99ju6erq+hGmFRERE6k7cHYCI6vG+oAnmuq9ZeXZYhqLA/aW22+nJK0o38/c2tJm\npK8bgafKVdOTwCpJc8pigVWlFhERHTSzqo4lfQG4HrhC0mEaK8fuAXZIWge8DNwMYHu/pB3AAeAM\ncIfts6Wr22mseJsF7CovgIeARyQN0lic0Fv6GpZ0N/BMOe4u262LFyIiomaVBY7tj4+xa+UYx28G\nNo9SHwCuGaX+OnDTGH1tBba2PdiIiKhcnjQQERG1SOBEREQtEjgREVGLBE5ERNQigRMREbVI4ERE\nRC0SOBERUYsETkRE1CKBExERtUjgRERELRI4ERFRiwRORETUIoETERG1SOBEREQtEjgREVGLBE5E\nRNQigRMREbVI4ERERC2mdeBIWiPpoKRBSRs6PZ6IiIvZtA0cSTOA3wM+DCwFPi5paWdHFRFx8Zq2\ngQMsBwZtf9f2PwCPAms7PKaIiIvWdA6cBcArTZ8Pl1pERHTAzE4PoJMkrQfWl48/kHSw5ZArgL+p\nd1S1mFLz0r1tHzql5nWepszczuN8wRSa1yRMmbn9iOfsJ9ptOJ0D5wiwqOnzwlL7Z7a3AFvG6kDS\ngO2eaobXOZnX1DNd5zZd5wXTd24/yrym8y21Z4AlkhZLeivQC+zs8JgiIi5a0/YKx/YZSf8FeBKY\nAWy1vb/Dw4qIuGhN28ABsP014Gs/Qhdj3m6b4jKvqWe6zm26zgum79wmPS/ZvpADiYiIGNV0/g4n\nIiLeRC76wJG0VdIxSd8eY//1kk5Keq68/nvdY5wMSYskfV3SAUn7Jd05yjGSdH959M9fSfqpToz1\nfLQ5r6l6zt4maa+kvyxz+61RjpmK56ydeU3JcwaNp5pI+pakr4yyb8qdrxETzGtS52taf4fTpoeB\nzwDbxznmz21/tJ7hXDBngF+1/aykdwH7JPXbPtB0zIeBJeV1HfBgeX8za2deMDXP2Wngg7Z/IOkS\n4JuSdtl+uumYqXjO2pkXTM1zBnAn8AIwe5R9U/F8jRhvXjCJ83XRX+HY/jNguNPjuNBsH7X9bNn+\nPo3/cFqftLAW2O6Gp4F3S5pf81DPS5vzmpLKefhB+XhJebV+yToVz1k785qSJC0EPgJ8doxDptz5\ngrbmNSkXfeC06f3lcniXpKs7PZjzJakbeB+wp2XXlH78zzjzgil6zsptjOeAY0C/7WlxztqYF0zN\nc/a7wK8B/zTG/il5vph4XjCJ85XAmdizwJW23wN8GvijDo/nvEh6J/BF4JO2T3V6PBfKBPOasufM\n9lnb19J4MsZySdd0ekwXQhvzmnLnTNJHgWO293V6LBdSm/Oa1PlK4EzA9qmR2wHl93oukXRFh4fV\nlnK//IvA521/aZRDJnz8z5vRRPOayudshO3vAV8H1rTsmpLnbMRY85qi5+wDwMckvUTjafQflPT7\nLcdMxfM14bwme74SOBOQ9GOSVLaX0/g3O97ZUU2sjPkh4AXbvzPGYTuBW8tKmhXASdtHaxvkJLQz\nryl8zrokvbtszwI+BHyn5bCpeM4mnNdUPGe2N9peaLubxqOznrJ9S8thU+58tTOvyZ6vi36VmqQv\nANcDV0g6DGyi8aUmtv8XcCPwnyWdAf4e6PXU+G3ZDwA/Dzxf7p0D/AZwJfzz3L4G/DQwCPwd8IkO\njPN8tTOvqXrO5gPb1PjjgW8Bdtj+iqRfgil9ztqZ11Q9Zz9kGpyvUV2I85UnDURERC1ySy0iImqR\nwImIiFokcCIiohYJnIiIqEUCJyIiapHAiYiIWiRwIiKiFgmciDcxSd2SXpD0v9X4WzJ/Un5bP2LK\nSeBEvPktAX7P9tXA94Cf7fB4IiYlgRPx5vei7ZHH+OwDujs4lohJS+BEvPmdbto+S56BGFNUAici\nImqRwImIiFrkadEREVGLXOFEREQtEjgREVGLBE5ERNQigRMREbVI4ERERC0SOBERUYsETkRE1CKB\nExERtfj/7IzOyC6dp10AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ccf7b83fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar([2,3,4], [distinct_ngrams(tokens, n) for n in [2,3,4]])\n",
    "plt.xlabel(\"n\")\n",
    "plt.ylabel(\"count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What happens if we plot of the log-frequency of words against the log-rank of words?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEOCAYAAACTqoDjAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VOX99vHPNxtZgAQIYQlbJAiEXSIoIAIiiwqu1YKP\n1g0Vt2q1VauP2t9P26q1Wqx7VaytCwqoiCxV2UQUguxr2ElkXwKENcn9/JHYJ+IgEzKTMzO53q9X\nXjJnzkwujpNcnO2+zTmHiIjI8aK8DiAiIqFJBSEiIj6pIERExCcVhIiI+KSCEBERn1QQIiLikwpC\nRER8UkGIiIhPKggREfFJBSEiIj7FeB2gMlJTU12LFi28jiEiElbmz5+/0zlX/2TrhWVBmNkQYEhm\nZiY5OTlexxERCStmttGf9cLyEJNzboJz7ubk5GSvo4iIRKywLAgREQk+FYSIiPgUlgVhZkPM7NWC\nggKvo4iIRKywLAidgxARCb6wLAgREQk+FYSIiPikghAREZ/CsiB0klpEJPjCsiAqe5J6066DzF2/\nm+ISF+BkIiKRIyyH2qisd+Zu4uUZa0mtWYOB7RowqH1DzjqtHrHRYdmXIiJBUS0L4o5+mbRrXJvJ\nS7cyfkE+//52EymJsfRv24DB7RvSq1UqNWKivY4pIuIpcy58D7NkZ2e7yg7Wd/hYMTNX72Dy0q38\nZ8U29h8uomaNGPq2SWNw+4b0aV2fxLhq2aMiEqHMbL5zLvuk64VjQZQbzXVEbm5uwN73aFEJX6/d\nyeSlW5m6fBu7C48SHxvFuafXZ3D7RvRrm0bt+NiAfT8RES9EdEH8IBB7ECdSVFzCvA17mLx0C5OX\nbWXbviPERhs9M1MZ3L4h52c1pG5SXFC+t4hIMKkgAqikxLFg814mL93CpKVbydtziOgoo3tGXQa3\nb8jAdg1Jqx0f9BwiIoGggggS5xzLvt/H5KVbmbR0C2t3FGIGXZvVYVBZWTStm1ilmUREKkIFUUVy\nt+1n0tKtTF66leVb9gHQIT2ZQe0bMqh9Q1rWr+lpPhGR46kgPLBxV2HZnsVWFm7eC8DZp9Xjzn6Z\nnN2yHmbmcUIRERWE577fe4gJi77n9a/Ws33/Ebo2r8Md/TLpc3p9FYWIeEoFESIOHyvmg/l5vDx9\nLfl7D9E+vTZ39G3FgKwGREWpKESk6kV0QQTrPohgOlZcwvgF+bw4bQ0bdh3k9AY1ub1vJhd1bEy0\nikJEqlBEF8QPwmEP4nhFxSVMXLKFF6atYfW2A2SkJjGyT0su7ZKusaBEpEqoIEJcSYlj6vKtPP/l\nGpZ9v4/0lARG9mnJFV2bEB+rcaBEJHhUEGHCOcf0VTsY9WUuCzbtpUHtGtzcuyXDuzUjIU5FISKB\np4IIM8455qzdxfNfrmHOul2kJMaSWb8mdZLiqJMYS53EuBP+OSUxTucxRMRv/haEhikNEWZGj8xU\nemSmkrNhN+98u4ktBYfZvPsgi/OOsufgMY4Wlfh8be34GAa2a8iFHRvRMzNV5zJEJCBUECEou0Vd\nslvU/dEy5xyHjhWzu/Aoew8eY3fhUfYcPMqewqMszitg8tKtfDA/j5TEWAZmlZbF2S01CZKInDoV\nRJgwMxLjYkiMi6FJnZ8+f6SomFmrdzJxyRYmLtnC+zmbqZMYy6D2Dbm4czpnnVav6kOLSFjTOYgI\n9MMkSBOXbOHz5dsoPFrM+VkNeGxoO9JTEryOJyIe00lqAUrL4q2vN/Dc56U3FN7dvxU39MrQoSeR\naszfggjL3xJmNsTMXi0oKPA6SsiLj43mlnNb8vm959KrVSp/mrSSi0Z9Rc6G3V5HE5EQpz2Iambq\nsq089skyvi84zJXZTbizXyvSUxI0LpRINaLLXMWnAe0a0jMzlVFf5PL6V+sZk5NHXEwUTesk0Kxu\nIk3rJtK8XhJXdG1CcoLm3xapzlQQ1VBSjRgevKAtv8huypx1u9i8+yCbdh1k856D5GzYw/4jRewu\nPMJvB7bxOqqIeEgFUY1lptUkM+3HM94557jm9blMXLyF+wa01twVItVYWJ6kluAxMy7s2IgNuw7+\ndwpVEameVBDyEwPbNSQ6ypi4eIvXUUTEQyoI+Ym6SXH0aFmPiUu2EM5XuYlI5aggxKeLOjZi466D\nLPteh5lEqisVhPg0IKvsMNMSHWYSqa5UEOJTnaQ4emamMnGxDjOJVFcqCDmhizo0YtPugyzN12Em\nkepIBSEnNKBdA2KijE+XfO91FBHxQEgVhJklmVmOmV3kdRaBlEQdZhKpzoJaEGb2hpltN7Olxy0f\nZGarzGyNmT1Q7qn7gTHBzCQVc2HHRuTtOcSSfI2cK1LdBHsPYjQwqPwCM4sGXgAGA1nAMDPLMrPz\ngeXA9iBnkgoYmNWQ2GjdNCdSHQW1IJxzM4HjJx7oBqxxzq1zzh0F3gMuBvoAZwHDgRFmFlKHv6qr\n5MRYemWm8qkOM4lUO178Ek4HNpd7nAekO+cecs7dDbwDvOacK/H1YjO7uew8Rc6OHTuqIK5c0KER\n+XsP8cbsDRQV+/zfIiIRKOT+le6cG+2c+/Rnnn/VOZftnMuuX79+VUarti7o0IhuLeryv58uZ8Cz\nM5m4eAslJdqbEIl0Xgz3nQ80Lfe4SdkyCVFJNWJ4/5az+HzFdp6espLb3/mODunJ9G/bADMwIDra\nuLhzOukpCV7HFZEACfqUo2bWAvjUOde+7HEMsBo4j9JimAcMd84tq8B7DgGGZGZmjsjNzQ14Zjmx\n4hLH+AX5PPf5avL2HPrRcy3qJfLR7T1JSYzzKJ2I+MPfKUeDWhBm9i6lJ59TgW3Ao865183sAuA5\nIBp4wzn3xKm8v+ak9o5zjhJX+l8HLNi0l6v/8Q3dM+ox+voziYkOuaOXIlImJAoi2FQQoWXMvM38\nbuxiru/ZgkeHtPM6joicgL8FEZZTjpY7xOR1FCnnyjObsnLrft6YvZ6M1CSGd2umPQmRMKY9CAmo\nouISrh89j1m5O0mIjaZDejJntazHbX1aEh8b7XU8ESHC9yAkdMVER/HatdlMWbaVhZv3smDTXkZ9\nkUtxSQm/HdjG63giUgFhWRA6xBTa4mOjubhzOhd3Tgfg3jGLeGXGOi7q2Ji2jWp7nE5E/BWWB4id\ncxOcczcnJyd7HUX88PCFbUlOiOWBsYsp1g12ImEjLAtCwkudpDgeGZLForwC3vp6g9dxRMRPYXmI\nScLP0E6NGb8gnz9PWsnHC/NpWjeRM1vUZXj3ZsTqSieRkBSWP5lmNsTMXi0o0BwF4cLMeOqKjvyy\nW1NqJ8SyOK+ARz9ZxiUvzGap5poQCUm6zFU8M2nJFh75ZBm7C4/y92FdGNyhkdeRRKoFfy9zDcs9\nCIkMgzs04vN7zqV949o8OH4J2/cf9jqSiJSjghBPJSfG8syVnTh4tJiHxi/VpEQiIUQnqcVzmWm1\n+O2A1jzx2Qr+PHklbRrWIiUhjozUJJrUSdBwHSIeCcuC0I1ykeeGXhnMWL2DV2as+9Hy+Ngozj6t\nHn1ap9GndX2a10vyKKFI9aOT1BIySkocOwuPUHikmF0HjrBuZyFL8wuYsXoHG3cdBKBxcjzN6iUy\nqF1DftWjBWbmcWqR8KPhviWibNhZyPRV21m4eS+rtx1g+ZZ9XHNWcx4b2o7oKJWESEVosD6JKC1S\nk7guNQMo3dN4cvJKXpm5jh37j/DcLztrpFiRINDZPwk7UVHGgxe05ZGLspiyfCtX/+NbNuws9DqW\nSMTRHoSErRt6ZZBWuwYPjF3CgOdm0j2jLimJcZyTmcqFHRuRVEMfb5HKCMtzEOWuYhqRm5vrdRzx\n2LZ9h/nr1NWs2rafrQWH2brvMO3Ta/Pvm84iOSHW63giIUcnqaVacs4xZdk27nz3O1ql1eKBwW3o\nmZmqE9ki5aggpFr7fPk2HvpoCdv2HSE5IZbTG9SkU5MUbu3TktSaNbyOJ+IpFYRUe0eKipmybBuz\nc3eyflch8zfuITEumicv78iArAa6Q1uqLRWEyHHW7jjAb95fyKK8AhJio/n3iO6c0ayO17FEqpxG\ncxU5Tsv6NRlz69k8d1VnSpxjwqLvvY4kEtJUEFKt1IiJ5pIu6Zzdsh7TV+3wOo5ISAvLgtCMclJZ\nfVunsX5nIet1g53ICYVlQTjnJjjnbk5OTvY6ioSpfm3SAHho/BIWbNpDSUn4nosTCZawLAiRympa\nN5GnrujI/I17uPTFr/nNmIWarEjkOBqLQKqtK7Ob0rd1Gm/MXs9L09eSmVaTO/q18jqWSMg4aUGY\nWT3n3K6qCCNS1erXqsHvBrZma8Fh/jJ1NUk1Yri6e3PiYrRzLeLPT8E3ZvaBmV1gmp1FIpCZ8afL\nOtCndX3+MGE57R+bwp8+W8HRohKvo4l4yp+COB14FbgGyDWzP5rZ6cGNJVK14mOjefO6M/nHtdlc\n1KERr8xcx2/GLNTJa6nWTloQrtR/nHPDgBHAr4C5ZjbDzM4OekKRKmJm9M9qwF+v6szvBrXm08Vb\nGLcg3+tYIp45aUGYWT0z+7WZ5QD3AXcCqcC9wDtBzifiiVt7t6Rz0xQen7ic0bPX6wonqZb8OcQ0\nB6gNXOKcu9A5N845V+ScywFeDm48EW9ERRl/+UUnWqXV5LEJy+n3zAxmrNad11K9nHSwPjMzF2L/\nfNKEQVJVnHO8/c1G/jxpJak1a/DFvecSq1FgJcwFcrC+qWaWUu6N65jZlEqlqyTdSS1Vxcy49uwW\nPD+sC5t2H6TP09P51Rtzmb1mp9fRRILOn4Ko75zb+8MD59weIC14kURCz3ltGzBqWBfap9dm9bb9\nXP/mPK5/cy7jF+Tp/IRELH/upC42s2bOuU0AZtYc0E+EVDtDOzVmaKfGFBw8xpNTVjJn7S7ueX8R\nk5Zs5d4BrWndsJbXEUUCyp+CeAj4ysxmAAacA9wc1FQiISw5MZY/XtqBkhLH61+t5+kpq5i6fBu/\n6NqEBwa3oZ6mNJUI4deMcmaWCpxV9vAb51xIHIDVjHISCjbsLOTpqauYuHgLzesl8vHtPUlJjPM6\nlsgJBXpGuRrAbmAfkGVmvSsTTiSStEhN4u/DuvDmdWeSt+cQf/pspdeRRALCn8H6ngSuApYBPwxO\n44CZQcwlElbMjL5t0rjpnAxembGOy85Ip/tp9byOJVIp/uxBXAK0LrtJbkjZ19BgBxMJR3efdzpN\n6iTw8EdLKdY4ThLm/CmIdUBssIOIRIKEuGju6X86udsPsDhv78lfIBLC/LmK6SCw0My+AI78sNA5\nd1fQUomEsX5t0jCDl2es5fa+mWQ1qk2M7r6WMORPQXxS9iUifqiTFMc1ZzXnX99sZMqybcTHRpHV\nqDa/6tGCizunex1PxG/+XuaaADRzzq0KfiT/6TJXCWW7C48yK3cHizYXMCt3B7nbD/DA4Dbc0vs0\nNPeWeMnfy1z9GaxvCPAXIM45l2FmnYH/CYUT1SoICRcHjhRxzevfsmDTXtJTErjpnAwGtW9Io+QE\nr6NJNRTI+yAeA7oBewGccwuB0yqVzgcza2tmL5vZh2Y2MtDvL+KlmjViGDeyB/9zcTuOFJXwhwnL\nGf7atxrHSUKaPwVxzDlXcNwyvybrNbM3zGy7mS09bvkgM1tlZmvM7AEA59wK59ytwJVAT3/eXySc\n/DAy7LyHzuPRIVms31nIBaO+YnfhUa+jifjkT0EsM7PhQLSZtTKz54Gv/Xz/0cCg8gvMLBp4ARgM\nZAHDzCyr7LmhwETgMz/fXyTsmBnDujVjZJ+WrN62nz5PT+P9eZu0NyEhx5+CuBNoR+klru9SOtzG\n3f68uXNuJqVDdJTXDVjjnFvnnDsKvAdcXLb+J865wcDVJ3pPM7vZzHLMLGfHDs3wJeEpPjaa+we1\n4Y3rziQlMY77xy7h3jGL2HngyMlfLFJF/LqKqVLfwKwF8Klzrn3Z4yuAQc65m8oeXwN0Bz4ELqN0\n3KfFzrkXTvbeOkktkaCkxPGHCct4a85GzOCW3i255/xW1IiJ9jqaRCh/T1L7MxbTNHzM/+Cc63eK\n2Xxyzk0HpgfyPUXCQVSU8djQdgztnM4/52zg5Rlrcc7x4AVtvY4m1Zw/N8rdV+7P8cDlQFElvmc+\n0LTc4yZly/xWbk7qSsQQCR1mRtfmdejavA4HDhfx2qx1bNhVyKND2tE4RZfCijdO6RCTmc11znXz\nc90W/PgQUwywGjiP0mKYBwx3zi2raA4dYpJItH3/YUZ9kcvY+fkcOlbMDT0zePjCtkRF6eY6CYyA\n3QdhZnXLfaWa2UAg2c8Q7wJzgNZmlmdmNzrnioA7gCnACmDMqZSDSKRKqxXP45d0YOzIHrRrXJs3\nZq/njdnrKTh4zOtoUs34cyf1ekrPQRilh5bWU3on9VfBj3fCTD8cYhqRm5vrVQyRoDt8rJjzn53B\n5t2HSIqL5r6BrbnmrOYa/E8qJWBDbYQyHWKS6uDwsWK+Xb+b57/IJWfjHhonx/Pqtdm0T/drR17k\nJwI5FtNlP/e8c25cBbMFjApCqhPnHJOWbuXBcUsoOHSMvq3r8+xVnTX/tVRYIAtiItAD+LJsUV9K\n76TeATjn3A2VzFphOsQk1dmmXQd5b94m/jFrPS1SE3nqik50bpridSwJI4EsiKnAr5xzW8oeNwJG\nO+cGBiRpJWgPQqqzaau28+DYJcREG49clEX3jHokJ2ryRzm5QI7m2vSHciizDWh2yslEJCD6tk7j\n2as6s2P/EW5+ez7ZT/yH12au8zqWRBB/9iD+DrSidBwmgKsoHUvpziBnOyntQYjAwaNFzNuw578n\nsds0rMXZLetx0zmnka6b7MSHgF7FZGaXAr3LHs50zo2vZL5K0TkIkZ8qKi7h1Vnr+Cp3J3PX7yap\nRgzPD+tC79Prex1NQkygC6I50Mo597mZJQLRzrn9AchZKdqDEPFt465Crn1jLht3HeSu81pxW5+W\nxMdq8D8pFcg7qUdQOtLqK2WL0oGPKhdPRIKpeb0kJv+6N6fVT2LUF7lkP/4501Zu9zqWhBl/TlLf\nTukMb/sAnHO5QFowQ4lI5SXERfPpnb148eozMOD60fP4zZiFXseSMOJPQRwpm9gH+O9ge+F7+7VI\nNZIYF8MFHRox5Z7edGqawrjv8nlv7iaKiv2aNViqOX8KYoaZ/R5IMLPzgQ+ACcGN9fPMbIiZvVpQ\ncPxU2SLiS+OUBP55QzcyUpN4YNwSzn92Jm9/s1HTnMrP8ucy1yjgRmAApQP2TQH+4ULgk6WT1CIV\nc/hYMR8vzOeduZtZtHkvzeomcmmXdG7ufRpJNfyZHkYiQUCuYjKzaOCfzrkTzhHtJRWEyKkpKXGM\nX5DP2O/y+HrtLuokxnJF1ybcN7C1pjqtBgJyFZNzrhhobmYaDUwkgkRFGZd3bcI7I87inZu60z49\nmddmrefiv89m/sY9XseTEOHPPuU6YLaZfQIU/rDQOffXoKUSkSrTIzOVHpmpvPPtJp6YuJzLX/qa\nTk1TePjCtpzZoq7X8cRDJ9yDMLO3y/44FPi0bN1a5b5EJIIM796MWff349fntWL9jgP84uU5XPC3\nWcxes9PraOKRE56DMLPlQH9gMtDn+Oedc7uDmuxnaKgNkeDaXXiUcd/l8bfPc9l/pIjLz2jCIxdl\nabTYCFHpk9RmdhcwEsgAvi//FKXzQJwWiKCVoZPUIsG1Y/8Rnp6ykjE5edSOj2FYt2ZceWZTWtav\n6XU0qYRAzgfxknNuZMCSBZAKQqRqLM7by58+W8mcdbsAeOqKjgzp2JiEOF3xFI40J7WIBNz6nYXc\n9e4CluQXEBcdxeAODbnmrOZ0bppCTLQ/991KKFBBiEhQHCkq5qvcnUxftYOPFuSz/0gRWY1q888b\nu5Fas4bX8cQPKggRCbr9h4/xj1nreXnGWuomxfHWDd04vYEucgx1gZxyVETEp1rxsdxz/umMHdmD\no0UlDP7bLO55fyFbCw57HU0CICwLQoP1iYSW9unJfHR7Ty4/I52PF+Zz4ahZfLwwn8PHir2OJpWg\nQ0wiElBrtu9n5L++I3f7AZrVTeSF4WfQoUmy17GkHB1iEhFPZKbVYtKvz+GRi7LI33uIS1+czfNf\n5HK0SHNQhBsVhIgEXEx0FDf0ymD+w/3p37YBz/xnNZe8MJs12z2fyl4qQAUhIkGTkhjHy9d05Zlf\ndGLDrkL6/3UmN4yeR96eg15HEz+oIEQk6C7v2oTpv+3DDT0z+HLldgY+O5P35m7yOpachApCRKpE\nWq14HhmSxcS7epFWO54Hxi3hprdymLJsq6Y+DVEqCBGpUu0aJzP+th4M69aUmbk7uOXt+Zzz1DSm\nr9rudTQ5ji5zFRHPHDxaxPvzNvPM1NUcOFLEVdlNeXRoFolxmh87mHSZq4iEvMS4GK7vmcGcB/sx\nuH1D3s/ZzBUvzaHwSJHX0YQwLQjdSS0SWWrFx/LS/+nKo0OyWL5lHwOenamZ7EKADjGJSEgZ910e\nD4xbwtGiEro0S+GJSzqQ1bi217Eiig4xiUhYuuyMJsz7fX+u69GCBZv2cskLs5m8dKvXsaolFYSI\nhJzkxFgeG9qOaff1oU5SLLf+az73vL+Q/YePeR2tWlFBiEjIykhNYtKvezO0U2PGL8in55+/5PPl\n27yOVW2oIEQkpNVNimPUsC7868bu1IqP5aZ/5nD3ewt0pVMVUEGISFjo1SqVj+/oycB2Dfho4fcM\nfG4my7/f53WsiKaCEJGwkVqzBq9ck81vzj+dvD2HuGDULN6es8HrWBFLBSEiYeeu81ox7b4+tKiX\nyP/9eBmXvDCbbfs0zWmgqSBEJCxlpCYx+e7eXNolnYWb99L9j1/wp0krNPBfAKkgRCRsxcdG8+xV\nnXlnRHfaNqrNKzPWcd5fZ/Dtul1eR4sIKggRCXs9Wqby2V29+PV5rVi3o5CrXv2GB8ctZp/um6gU\nFYSIRAQz457zT+frB/rRIT2Zd+du5szHP2dx3l6vo4UtFYSIRJTGKQlMuLMXzw/rwpGiEob+fTYP\njluicxOnIKQKwswuMbPXzOx9MxvgdR4RCV9DOjVm8t3nEGXw7txN9HpyGut3FnodK6wEvSDM7A0z\n225mS49bPsjMVpnZGjN7AMA595FzbgRwK3BVsLOJSGRr07A2qx4fzIUdG5G/9xB9/zKdf32z0etY\nYaMq9iBGA4PKLzCzaOAFYDCQBQwzs6xyqzxc9ryISKXERkfxwvAz+PTOXtRLiuPhj5Zy75hFlJTo\nkNPJBL0gnHMzgd3HLe4GrHHOrXPOHQXeAy62Uk8Ck5xz3wU7m4hUH+3Tk/nq/n60a1ybsd/lcc5T\n01QSJ+HVOYh0YHO5x3lly+4E+gNXmNmtvl5oZjebWY6Z5ezYsSP4SUUkYiTERfPx7T1pn16b/L2H\naPnQZ8xcrd8jJxJSJ6mdc6Occ12dc7c6514+wTqvOueynXPZ9evXr+qIIhLmYqKjGH9bTy7o0BDn\n4No35jLuuzyvY4UkrwoiH2ha7nGTsmV+0ZzUIlIZsdFRvHh1Vz6+vScAvxmziAfHLaFYh5x+xKuC\nmAe0MrMMM4sDfgl84u+LnXMTnHM3JycnBy2giES+Tk1TePP6M4HSS2Fb/v4zdh444nGq0FEVl7m+\nC8wBWptZnpnd6JwrAu4ApgArgDHOuWXBziIicry+rdNY9fggumfUBSD78c9ZsUXzTABYON5daGZD\ngCGZmZkjcnNzvY4jIhHitx8s4oP5pecjnvlFJy7v2sTjRMFhZvOdc9knXS8cC+IH2dnZLicnx+sY\nIhJBPpyfx30fLAKgc9MUPrj1bGKjQ+p6nkrztyAi628tIlJJV3RtwpS7e5OcEMvCzXtp9dAk5m88\n/lau6kEFISJynNYNa7HwkfPp27r0UvrLX5rDU5NXepyq6oVlQegyVxEJNjPjzeu78eZ1pVc5vTh9\nLUP//lW1uhQ2LAtCl7mKSFXp2yaNuQ+dR3SUsTivgJa//4zV2/Z7HatKhGVBiIhUpbRa8eQ+Pphu\nLUovhR3w7Exm5Ub+EB1hWRA6xCQiVS0qyhhz69k8OLgNANe8PpfffbjI41TBpctcRUQq6Ou1Oxn+\n2rcANKmTwCd39KJuUpzHqfyny1xFRIKkR8tUFj0ygLRaNcjbc4gz/vc/TF221etYAaeCEBE5BcmJ\nscx9qD839MwA4Oa35/P+vE0epwosFYSISCU8MiSL164tPVpz/9glPDN1FeF86L68sCwInaQWkVBy\nflYDRpeNCvv8l2u45e35EVESYVkQug9CREJNn9ZpzHmwHwBTl29j+Gvfhv2UpmFZECIioahRcgI5\nD/cHYM66XXT6n6ls23fY41SnTgUhIhJAqTVrsOSxAbRPr83+w0V0/+MXfLNul9exTokKQkQkwGrF\nx/Lpnecwsk9LAH756jdMW7Xd41QVp4IQEQmS+we14anLOwJw/ZvzGDNvs8eJKiYsC0JXMYlIuLjy\nzKY8dUVpSfxu7GKe/yI3bK5wCsuC0FVMIhJOrsxuysS7egHwzH9Wc+NbORw+VuxxqpMLy4IQEQk3\n7Ron88W95wLw5crt9HryS/YdPuZxqp+nghARqSIt69dk5f8OolVaTXYeOErHx6aSt+eg17FOSAUh\nIlKF4mOjmXJ3b849vXQ6015PTmPFln0ep/JNBSEiUsWioow3rzuTy85IB2Dw32axYNMej1P9lApC\nRMQDUVHGX6/szEMXtAXg0he/ZsKi7z1O9WNhWRC6zFVEIsWI3qfxxKXtAbjz3QW8NnNdyFwGG5YF\noctcRSSSXN29Of++qTsAT3y2gnvHLAqJy2DDsiBERCJNz8xUxt/WA4BxC/K59MWv2V141NNMKggR\nkRDRpVkdpt3Xh9rxMazYso8Bz85ka4F3o8GqIEREQkhGahLTf9uX1g1qsfPAEa55/Vs27ir0JIsK\nQkQkxNRNimPsbT3onlGX3O0HOPfp6WzeXfU31KkgRERCUM0aMbx5/Znc3b8VAOc8NY3tVTz5kApC\nRCREJcbFcFufTIZ3bwZAjz9/yaLNe6vs+6sgRERCWFxMFI8NacfvBrWmqMRx41tVN6+ECkJEJMTF\nxUQx8tyy7JzhAAAE0ElEQVSWjOzTkmPFjt+NXcy3VTCNaUzQv0MQmNkQYEhmZqbXUUREqoSZcf+g\nNkSb8dHCfHYeCP49EhYqt3SfiuzsbJeTk+N1DBGRsGJm851z2SdbT4eYRETEJxWEiIj4pIIQERGf\nVBAiIuKTCkJERHxSQYiIiE8qCBER8UkFISIiPoX1jXJmtgPYCCQDx09Qffyy4x+nAjuDGtD39w3G\na0+23s89f6LnQnWbVsX29Gfdim7TUN2evr5vMF6rz2jgX1uZbdrKOXfyOZudc2H/Bbx6smU+Hud4\nlS3Qrz3Zej/3/ImeC9VtWhXbMxjbNFS3Z1VtU31GA//aYGzT478i5RDTBD+W+VqnKlTm+/r72pOt\n93PPn+i5UN2mVbE9/Vm3ots0VLdnZb+vPqMnzxCs1wZjm/5IWB9iqgwzy3F+jEUi/tM2DSxtz8DT\nNq2YSNmDOBWveh0gAmmbBpa2Z+Bpm1ZAtd2DEBGRn1ed9yBERORnqCBERMQnFYSIiPikgihjZklm\n9paZvWZmV3udJ9yZ2Wlm9rqZfeh1lkhhZpeUfT7fN7MBXucJd2bW1sxeNrMPzWyk13lCUUQXhJm9\nYWbbzWzpccsHmdkqM1tjZg+ULb4M+NA5NwIYWuVhw0BFtqdzbp1z7kZvkoaPCm7Tj8o+n7cCV3mR\nN9RVcHuucM7dClwJ9PQib6iL6IIARgODyi8ws2jgBWAwkAUMM7MsoAmwuWy14irMGE5G4//2FP+M\npuLb9OGy5+WnRlOB7WlmQ4GJwGdVGzM8RHRBOOdmAruPW9wNWFP2L9yjwHvAxUAepSUBEb5dTlUF\nt6f4oSLb1Eo9CUxyzn1X1VnDQUU/o865T5xzgwEdVvahOv4iTOf/7ylAaTGkA+OAy83sJbwb8iAc\n+dyeZlbPzF4GupjZg95EC1sn+ozeCfQHrjCzW70IFqZO9BntY2ajzOwVtAfhU4zXAUKFc64QuN7r\nHJHCObeL0mPlEiDOuVHAKK9zRArn3HRguscxQlp13IPIB5qWe9ykbJmcGm3PwNM2DSxtz1NUHQti\nHtDKzDLMLA74JfCJx5nCmbZn4GmbBpa25ymK6IIws3eBOUBrM8szsxudc0XAHcAUYAUwxjm3zMuc\n4ULbM/C0TQNL2zOwNFifiIj4FNF7ECIicupUECIi4pMKQkREfFJBiIiITyoIERHxSQUhIiI+qSBE\nqpCZPWZm93mdQ8QfKgiRU1Q2uqp+hiRi6cMtUgFm1qJs4pl/AkuB180sx8yWmdkfyq23wcz+YGbf\nmdkSM2vj471GmNkkM0uoyr+DiL80mqtIxbUCfuWc+8bM6jrndpdNSvOFmXV0zi0uW2+nc+4MM7sN\nuA+46Yc3MLM7gPOBS5xzR6r8byDiB+1BiFTcRufcN2V/vtLMvgMWAO0onbHsB+PK/jsfaFFu+bWU\nzm52hcpBQpkKQqTiCgHMLIPSPYPznHMdKZ26Mr7cej/88i/mx3vrSygtjCaIhDAVhMipq01pWRSY\nWQNK9wr8sQC4BfjEzBoHK5xIZakgRE6Rc24Rpb/sVwLvALMr8NqvKN37mGhmqcFJKFI5Gu5bRER8\n0h6EiIj4pIIQERGfVBAiIuKTCkJERHxSQYiIiE8qCBER8UkFISIiPqkgRETEp/8HtzVKd/KQzwkA\nAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ccf7b83e10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.loglog([val for word,val in token_counter.most_common(4000)])\n",
    "plt.xlabel('rank')\n",
    "plt.ylabel('frequency');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The graph is almost linear! It seems that the word count distribution follows power law distribution. A few words occur a lot, many words occur rarely."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some comparison between the train, valid datasets and the test dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_tokens = words_test_data.split()\n",
    "len(set(test_tokens)-set(tokens))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_tokens = words_valid_data.split()\n",
    "len(set(valid_tokens)-set(tokens))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are no unseen tokens in the test or validation dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Total number of characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4975414"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chars = chars_train_data.split()\n",
    "len(chars)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of distinct character bigrams:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "958"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distinct_ngrams(chars, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x2ccf8d66da0>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZsAAAEKCAYAAADEovgeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGPlJREFUeJzt3X+s3fV93/HnKzghjAZig+dRm85MWJkALaR4hi5R1MWK\n7YwsoA6QIyV4qQeRoFWibaqg/3gFWQJpKy1bQWLBwdA04JAyWFLCXJNuq1R+XBI6AgTZDSBsAXa5\nDiSZIDV774/zueH45tq+/vG5x755PqSvzue8z/fzOZ+vkHj5+/1+7vekqpAkqad3jXoCkqTZz7CR\nJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqbs6oJ3C0OPXUU2vx4sWjnoYkHVOe\neOKJv62q+Qfaz7BpFi9ezNjY2KinIUnHlCQvTmc/L6NJkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NG\nktRdt7BJ8oEkTw5tbyT5YpJ5STYn2dpe5w71uTbJtiTPJVk5VD8vyVPts5uTpNWPT3JPqz+aZPFQ\nnzXtO7YmWdPrOCVJB9YtbKrquao6t6rOBc4D/i9wH3ANsKWqlgBb2nuSnAWsBs4GVgG3JDmuDXcr\ncAWwpG2rWn0tsLuqzgRuAm5sY80D1gHnA8uAdcOhJkmaWTN1GW058DdV9SJwEbCx1TcCF7f2RcDd\nVfVWVT0PbAOWJTkNOKmqHqmqAu6c1GdirHuB5e2sZyWwuarGq2o3sJl3AkqSNMNm6gkCq4GvtvaC\nqnq5tV8BFrT2QuCRoT7bW+3vWntyfaLPSwBVtSfJ68Apw/Up+kjSjFh8zTdHPYVpeeGGC7t/R/cz\nmyTvAT4FfG3yZ+1MpXrPYV+SXJlkLMnYrl27RjUNSZr1ZuIy2ieA71TVq+39q+3SGO11Z6vvAE4f\n6reo1Xa09uT6Xn2SzAFOBl7bz1h7qarbqmppVS2dP/+Az5GTJB2imQibT/POJTSAB4CJ1WFrgPuH\n6qvbCrMzGCwEeKxdcnsjyQXtfszlk/pMjHUJ8HA7W3oIWJFkblsYsKLVJEkj0PWeTZITgY8Dnx8q\n3wBsSrIWeBG4DKCqnk6yCXgG2ANcXVVvtz5XAXcAJwAPtg3gduCuJNuAcQb3hqiq8STXA4+3/a6r\nqvEuBylJOqCuYVNVP2Fww3649hqD1WlT7b8eWD9FfQw4Z4r6m8Cl+xhrA7Dh4GctSTrSfIKAJKk7\nw0aS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiS\nujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHXXNWySvD/JvUm+n+TZJL+WZF6SzUm2\ntte5Q/tfm2RbkueSrByqn5fkqfbZzUnS6scnuafVH02yeKjPmvYdW5Os6XmckqT9631m84fAt6rq\nHwMfBJ4FrgG2VNUSYEt7T5KzgNXA2cAq4JYkx7VxbgWuAJa0bVWrrwV2V9WZwE3AjW2secA64Hxg\nGbBuONQkSTOrW9gkORn4KHA7QFX9tKp+CFwEbGy7bQQubu2LgLur6q2qeh7YBixLchpwUlU9UlUF\n3Dmpz8RY9wLL21nPSmBzVY1X1W5gM+8ElCRphvU8szkD2AV8Ocl3k3wpyYnAgqp6ue3zCrCgtRcC\nLw31395qC1t7cn2vPlW1B3gdOGU/Y0mSRqBn2MwBfhW4tao+BPyEdslsQjtTqY5z2K8kVyYZSzK2\na9euUU1Dkma9nmGzHdheVY+29/cyCJ9X26Ux2uvO9vkO4PSh/otabUdrT67v1SfJHOBk4LX9jLWX\nqrqtqpZW1dL58+cf4mFKkg6kW9hU1SvAS0k+0ErLgWeAB4CJ1WFrgPtb+wFgdVthdgaDhQCPtUtu\nbyS5oN2PuXxSn4mxLgEebmdLDwErksxtCwNWtJokaQTmdB7/t4GvJHkP8APgcwwCblOStcCLwGUA\nVfV0kk0MAmkPcHVVvd3GuQq4AzgBeLBtMFh8cFeSbcA4g9VsVNV4kuuBx9t+11XVeM8DlSTtW9ew\nqaongaVTfLR8H/uvB9ZPUR8Dzpmi/iZw6T7G2gBsOJj5SpL68AkCkqTuDBtJUneGjSSpO8NGktSd\nYSNJ6s6wkSR1Z9hIkrozbCRJ3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrozbCRJ\n3Rk2kqTuDBtJUneGjSSpO8NGktSdYSNJ6q5r2CR5IclTSZ5MMtZq85JsTrK1vc4d2v/aJNuSPJdk\n5VD9vDbOtiQ3J0mrH5/knlZ/NMnioT5r2ndsTbKm53FKkvZvJs5s/nlVnVtVS9v7a4AtVbUE2NLe\nk+QsYDVwNrAKuCXJca3PrcAVwJK2rWr1tcDuqjoTuAm4sY01D1gHnA8sA9YNh5okaWaN4jLaRcDG\n1t4IXDxUv7uq3qqq54FtwLIkpwEnVdUjVVXAnZP6TIx1L7C8nfWsBDZX1XhV7QY2805ASZJmWO+w\nKeDPkzyR5MpWW1BVL7f2K8CC1l4IvDTUd3urLWztyfW9+lTVHuB14JT9jCVJGoE5ncf/SFXtSPL3\ngc1Jvj/8YVVVkuo8h31qAXglwK/8yq+MahqSNOt1PbOpqh3tdSdwH4P7J6+2S2O0151t9x3A6UPd\nF7XajtaeXN+rT5I5wMnAa/sZa/L8bquqpVW1dP78+Yd+oJKk/eoWNklOTPK+iTawAvge8AAwsTps\nDXB/az8ArG4rzM5gsBDgsXbJ7Y0kF7T7MZdP6jMx1iXAw+2+zkPAiiRz28KAFa0mSRqBnpfRFgD3\ntVXKc4A/qapvJXkc2JRkLfAicBlAVT2dZBPwDLAHuLqq3m5jXQXcAZwAPNg2gNuBu5JsA8YZrGaj\nqsaTXA883va7rqrGOx6rJGk/uoVNVf0A+OAU9deA5fvosx5YP0V9DDhnivqbwKX7GGsDsOHgZi1J\n6sEnCEiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4M\nG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd93DJslxSb6b5Bvt/bwk\nm5Nsba9zh/a9Nsm2JM8lWTlUPy/JU+2zm5Ok1Y9Pck+rP5pk8VCfNe07tiZZ0/s4JUn7Nq2wSbJl\nOrV9+ALw7ND7a4AtVbUE2NLek+QsYDVwNrAKuCXJca3PrcAVwJK2rWr1tcDuqjoTuAm4sY01D1gH\nnA8sA9YNh5okaWbtN2ySvLf9j/vUJHPbWcm8dgax8ECDJ1kEXAh8aah8EbCxtTcCFw/V766qt6rq\neWAbsCzJacBJVfVIVRVw56Q+E2PdCyxvZz0rgc1VNV5Vu4HNvBNQkqQZNucAn38e+CLwy8ATQFr9\nDeC/TGP8PwB+B3jfUG1BVb3c2q8AC1p7IfDI0H7bW+3vWntyfaLPSwBVtSfJ68Apw/Up+kiSZth+\nz2yq6g+r6gzg31fVP6qqM9r2warab9gk+SSws6qe2M/4BdQhzfwISHJlkrEkY7t27RrVNCRp1jvQ\nmQ0AVfWfk/wzYPFwn6q6cz/dPgx8Ksm/AN4LnJTkj4FXk5xWVS+3S2Q72/47gNOH+i9qtR2tPbk+\n3Gd7kjnAycBrrf7rk/r8xRTHdRtwG8DSpUtHFnqSNNtNd4HAXcB/BD4C/NO2Ld1fn6q6tqoWVdVi\nBjf+H66qzwAPABOrw9YA97f2A8DqtsLsDAYLAR5rl9zeSHJBux9z+aQ+E2Nd0r6jgIeAFe0+01xg\nRatJkkZgWmc2DILlrPY/8sN1A7ApyVrgReAygKp6Oskm4BlgD3B1Vb3d+lwF3AGcADzYNoDbgbuS\nbAPGGYQaVTWe5Hrg8bbfdVU1fgTmLkk6BNMNm+8B/wB4+UA7TqWq/oJ2GauqXgOW72O/9cD6Kepj\nwDlT1N8ELt3HWBuADYcyX0nSkTXdsDkVeCbJY8BbE8Wq+lSXWUmSZpXphs1/6DkJSdLsNt3VaP+z\n90QkSbPXtMImyY945+9h3gO8G/hJVZ3Ua2KSpNljumc2P3sCQFt+fBFwQa9JSfrFtPiab456CtPy\nwg0XjnoKx5yDfupzDfw3Bs8fkyTpgKZ7Ge03ht6+i8Hf3bzZZUaSpFlnuqvR/uVQew/wAoNLaZIk\nHdB079l8rvdEJEmz13SfjbYoyX1Jdrbt6+23aiRJOqDpLhD4MoOHXv5y2/57q0mSdEDTDZv5VfXl\nqtrTtjuA+R3nJUmaRaYbNq8l+UyS49r2GQa/GyNJ0gFNN2x+k8FPAbzC4MnPlwD/utOcJEmzzHSX\nPl8HrKmq3QBJ5jH4MbXf7DUxSdLsMd0zm38yETQw+HEy4EN9piRJmm2mGzbvaj+vDPzszGa6Z0WS\npF9w0w2M/wT8VZKvtfeXMsUvakqSNJXpPkHgziRjwMda6Teq6pl+05IkzSbTvhTWwsWAkSQdtIP+\niQFJkg5Wt7BJ8t4kjyX56yRPJ/m9Vp+XZHOSre11eOHBtUm2JXkuycqh+nlJnmqf3dx+wI0kxye5\np9UfTbJ4qM+a9h1bk6zpdZySpAPreWbzFvCxqvogcC6wKskFwDXAlqpaAmxp70lyFrAaOBtYBdyS\n5Lg21q3AFcCStq1q9bXA7qo6E7gJuLGNNQ9YB5wPLAPWDYeaJGlmdQub9oueP25v3922YvA7OBtb\nfSNwcWtfBNxdVW9V1fPANmBZktOAk6rqkaoq4M5JfSbGuhdY3s56VgKbq2q8/X3QZt4JKEnSDOt6\nz6Y9R+1JYCeD//k/CiyoqpfbLq8AC1p7IfDSUPftrbawtSfX9+pTVXuA14FT9jPW5PldmWQsydiu\nXbsO+TglSfvXNWyq6u2qOhdYxOAs5ZxJnxeDs52RqKrbqmppVS2dP9+HWEtSLzOyGq2qfgh8m8Gl\nrFfbpTHa68622w7g9KFui1ptR2tPru/VJ8kc4GQGT6Pe11iSpBHouRptfpL3t/YJwMeB7zP4EbaJ\n1WFrgPtb+wFgdVthdgaDhQCPtUtubyS5oN2PuXxSn4mxLgEebmdLDwErksxtCwNWtJokaQR6Pt/s\nNGBjW1H2LmBTVX0jyV8Bm5KsBV5k8NMFVNXTSTYx+MPRPcDVVfV2G+sq4A7gBODBtgHcDtyVZBsw\nzmA1G1U1nuR64PG233Xt4aGSpBHoFjZV9X+Y4snQVfUasHwffdYzxTPXqmoMOGeK+psMntM21Vgb\ngA0HN2tJUg8+QUCS1J1hI0nqzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nq\nzrCRJHVn2EiSujNsJEndGTaSpO4MG0lSd4aNJKk7w0aS1J1hI0nqzrCRJHVn2EiSuusWNklOT/Lt\nJM8keTrJF1p9XpLNSba217lDfa5Nsi3Jc0lWDtXPS/JU++zmJGn145Pc0+qPJlk81GdN+46tSdb0\nOk5J0oH1PLPZA/y7qjoLuAC4OslZwDXAlqpaAmxp72mfrQbOBlYBtyQ5ro11K3AFsKRtq1p9LbC7\nqs4EbgJubGPNA9YB5wPLgHXDoSZJmlndwqaqXq6q77T2j4BngYXARcDGtttG4OLWvgi4u6reqqrn\ngW3AsiSnASdV1SNVVcCdk/pMjHUvsLyd9awENlfVeFXtBjbzTkBJkmbYjNyzaZe3PgQ8Ciyoqpfb\nR68AC1p7IfDSULftrbawtSfX9+pTVXuA14FT9jOWJGkEuodNkl8Cvg58sareGP6snalU7znsS5Ir\nk4wlGdu1a9eopiFJs17XsEnybgZB85Wq+tNWfrVdGqO97mz1HcDpQ90XtdqO1p5c36tPkjnAycBr\n+xlrL1V1W1Utraql8+fPP9TDlCQdQM/VaAFuB56tqt8f+ugBYGJ12Brg/qH66rbC7AwGCwEea5fc\n3khyQRvz8kl9Jsa6BHi4nS09BKxIMrctDFjRapKkEZjTcewPA58FnkryZKv9LnADsCnJWuBF4DKA\nqno6ySbgGQYr2a6uqrdbv6uAO4ATgAfbBoMwuyvJNmCcwWo2qmo8yfXA422/66pqvNeBSpL2r1vY\nVNVfAtnHx8v30Wc9sH6K+hhwzhT1N4FL9zHWBmDDdOcrSerHJwhIkrozbCRJ3Rk2kqTuDBtJUneG\njSSpO8NGktSdYSNJ6s6wkSR11/MJApI6WnzNN0c9hWl54YYLRz0FHQU8s5EkdWfYSJK6M2wkSd0Z\nNpKk7gwbSVJ3ho0kqTvDRpLUnWEjSerOsJEkdWfYSJK6M2wkSd0ZNpKk7rqFTZINSXYm+d5QbV6S\nzUm2tte5Q59dm2RbkueSrByqn5fkqfbZzUnS6scnuafVH02yeKjPmvYdW5Os6XWMkqTp6Xlmcwew\nalLtGmBLVS0BtrT3JDkLWA2c3frckuS41udW4ApgSdsmxlwL7K6qM4GbgBvbWPOAdcD5wDJg3XCo\nSZJmXrewqar/BYxPKl8EbGztjcDFQ/W7q+qtqnoe2AYsS3IacFJVPVJVBdw5qc/EWPcCy9tZz0pg\nc1WNV9VuYDM/H3qSpBk00/dsFlTVy639CrCgtRcCLw3tt73VFrb25PpefapqD/A6cMp+xvo5Sa5M\nMpZkbNeuXYd6TJKkAxjZAoF2plKj+v42h9uqamlVLZ0/f/4opyJJs9pMh82r7dIY7XVnq+8ATh/a\nb1Gr7WjtyfW9+iSZA5wMvLafsSRJIzLTYfMAMLE6bA1w/1B9dVthdgaDhQCPtUtubyS5oN2PuXxS\nn4mxLgEebmdLDwErksxtCwNWtJokaUTm9Bo4yVeBXwdOTbKdwQqxG4BNSdYCLwKXAVTV00k2Ac8A\ne4Crq+rtNtRVDFa2nQA82DaA24G7kmxjsBBhdRtrPMn1wONtv+uqavJCBUnSDOoWNlX16X18tHwf\n+68H1k9RHwPOmaL+JnDpPsbaAGyY9mQlSV35BAFJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkroz\nbCRJ3Rk2kqTuDBtJUneGjSSpu26Pq5GONouv+eaopzAtL9xw4ainIB1xntlIkrozbCRJ3Rk2kqTu\nDBtJUneGjSSpO8NGktSdYSNJ6s6wkSR1Z9hIkrqb1WGTZFWS55JsS3LNqOcjSb+oZu3japIcB/wR\n8HFgO/B4kgeq6pnRzuzY4KNdJB1Js/nMZhmwrap+UFU/Be4GLhrxnCTpF9JsDpuFwEtD77e3miRp\nhqWqRj2HLpJcAqyqqn/T3n8WOL+qfmtonyuBK9vbDwDPzfhE9+9U4G9HPYkjaLYdD8y+Y5ptxwOz\n75iOtuP5h1U1/0A7zdp7NsAO4PSh94ta7Weq6jbgtpmc1MFIMlZVS0c9jyNlth0PzL5jmm3HA7Pv\nmI7V45nNl9EeB5YkOSPJe4DVwAMjnpMk/UKatWc2VbUnyW8BDwHHARuq6ukRT0uSfiHN2rABqKo/\nA/5s1PM4DEftJb5DNNuOB2bfMc2244HZd0zH5PHM2gUCkqSjx2y+ZyNJOkoYNkeZJKcn+XaSZ5I8\nneQLo57T4Ury3iSPJfnrdky/N+o5HQlJjkvy3STfGPVcjoQkLyR5KsmTScZGPZ/DleT9Se5N8v0k\nzyb5tVHP6XAk+UD7bzOxvZHki6Oe13R5Ge0ok+Q04LSq+k6S9wFPABcfy4/ZSRLgxKr6cZJ3A38J\nfKGqHhnx1A5Lkn8LLAVOqqpPjno+hyvJC8DSqjqa/objkCXZCPzvqvpSW5H696rqh6Oe15HQHse1\ng8HfDr446vlMh2c2R5mqermqvtPaPwKe5Rh/8kEN/Li9fXfbjul/5SRZBFwIfGnUc9HPS3Iy8FHg\ndoCq+ulsCZpmOfA3x0rQgGFzVEuyGPgQ8OhoZ3L42iWnJ4GdwOaqOtaP6Q+A3wH+36gncgQV8OdJ\nnmhP1ziWnQHsAr7cLnV+KcmJo57UEbQa+OqoJ3EwDJujVJJfAr4OfLGq3hj1fA5XVb1dVecyeJLD\nsiTnjHpOhyrJJ4GdVfXEqOdyhH2k/Tf6BHB1ko+OekKHYQ7wq8CtVfUh4CfArPiZkXZJ8FPA10Y9\nl4Nh2ByF2n2NrwNfqao/HfV8jqR2KePbwKpRz+UwfBj4VLvHcTfwsSR/PNopHb6q2tFedwL3MXhy\n+rFqO7B96Az6XgbhMxt8AvhOVb066okcDMPmKNNupt8OPFtVvz/q+RwJSeYneX9rn8DgN4a+P9pZ\nHbqquraqFlXVYgaXMx6uqs+MeFqHJcmJbUEK7XLTCuB7o53VoauqV4CXknyglZYDx+wim0k+zTF2\nCQ1m+RMEjlEfBj4LPNXucQD8bnsawrHqNGBjW0HzLmBTVc2K5cKzyALgvsG/dZgD/ElVfWu0Uzps\nvw18pV12+gHwuRHP57C1fwh8HPj8qOdysFz6LEnqzstokqTuDBtJUneGjSSpO8NGktSdYSNJ6s6w\nkSR1Z9hIkrozbKSjWJLF7bdY/mv7LaD/0Z7CIB1TDBvp6LcE+KOqOhv4IfCvRjwf6aAZNtLR7/mq\nmnh00RPA4hHORTokho109HtrqP02PtNQxyDDRpLUnWEjSerOpz5LkrrzzEaS1J1hI0nqzrCRJHVn\n2EiSujNsJEndGTaSpO4MG0lSd4aNJKm7/w+bY76EggFqEgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ccf8d6cfd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar([2, 3, 4, 5, 6, 7], [distinct_ngrams(chars, n) for n in [2, 3, 4, 5, 6, 7]])\n",
    "plt.xlabel(\"n\")\n",
    "plt.ylabel(\"count\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The number of distinct character 7-grams is similar the the number of distinct word n-grams."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from nltk.probability import MLEProbDist"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 n-gram Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to use an n-gram model, we need to store all distinct n-grams in memory. For each n-grams, the number of times it occurs.\n",
    "\n",
    "In the general case, all n-grams in the dataset could be distinct. Hence, if m is the number of tokens, there could be also m distinct 2,3,4...-grams (there are m starting points). So the worst case memory consumption, for m tokens, L average word length and an n-gram model is:\n",
    "mxnxL.\n",
    "\n",
    "In this dataset we have ~900,000 tokens, so the number of distinct n-grams could be 900,000xnxl.\n",
    "However, we can see that there are only 300,000 bi-grams, 600,000 tri-grams and 800,000 4-grams. Larger n increases the number of n-grams but it can't be more than 887,521 (number of tokens)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "825370"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distinct_ngrams(tokens, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "846671"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "distinct_ngrams(tokens, 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And indeed we can see that the number of distinct ngrams converges to the number of tokens from n=4."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let us build the model by simply counting all n-grams in text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_word_lm(sents, n=2):\n",
    "    \"constructs a word n-gram model given a dataset of sentences\"\n",
    "    counter = defaultdict(FreqDist)\n",
    "    for sent in sents:\n",
    "        pad = [\"~\" for x in range(n-1)]\n",
    "        sent = pad + sent + [\"~\"] #pad from both sided to mark start and end of sentence\n",
    "        for i in range(len(sent)-n+1):\n",
    "            history, word = \" \".join(sent[i:i+n-1]), sent[i+n-1]\n",
    "            counter[history][word] += 1\n",
    "    return counter    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sents_train = words_train_data.split(\"\\n\")\n",
    "sents_train = [sent.split() for sent in sents_train]\n",
    "lm = train_word_lm(sents_train, n=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do we get for the word 'burned'?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FreqDist({'a': 1,\n",
       "          'by': 1,\n",
       "          'if': 1,\n",
       "          'in': 1,\n",
       "          'investors': 1,\n",
       "          'on': 2,\n",
       "          'rand': 1,\n",
       "          'repeatedly': 1,\n",
       "          'when': 1,\n",
       "          '~': 1})"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm['burned']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get the number of times different words follow the word burned. But what is the frequenct of 'burned on' given 'burned'? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.18181818181818182"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm['burned'].freq('on')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And the frequency of 'm given I?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.10380348652931855"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lm['i'].freq('\\'m')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience, the method create_models creates all models from 1 to n. This will make it an easier job to evaluate models for different n values.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def create_models(sents, n):\n",
    "    \"\"\"creates all models from 1 to n\"\"\"\n",
    "    single_word_fd = FreqDist([word for sent in sents for word in sent])\n",
    "    single_word_fd['~'] = len(sents) #number of times a sentence ends\n",
    "                            \n",
    "    models = {1:single_word_fd}\n",
    "    \n",
    "    for i in range(2, n+1):\n",
    "        lm = train_word_lm(sents, i)\n",
    "        models[i] = lm\n",
    "        \n",
    "    return models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Language Model Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to evaluate our model, we will calculate entropy and perplexity (which is simply 2^entropy) on the validation dataset.\n",
    "\n",
    "But before that, we need to 'fix' our model. What happens if a specific n-gram in the validation dataset was not seen in the training dataset? It will get a probability of 0 and it will not be possible to calculate entropy or perplexity. Therefore, we will need to use some 'smoothing' technique in order to solve this problem.\n",
    "\n",
    "The simplest smoothing technique is to add k to each ngram count (Lidstone smoothing). When k equals 1 this is the Laplace smoothing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prob_MLE_addk_smoothing(history, word, n, models, k=1):\n",
    "    \"\"\"\n",
    "    Lidstone smoothing technique. Add k to each ngram count, seen and unseen.\n",
    "    Add |V|*k denominator for normalization.\n",
    "    When k=1 this is the Laplace smoothing\n",
    "    \"\"\"\n",
    "    model = models[n]\n",
    "    history_str = \" \".join(history)\n",
    "    \n",
    "    word_count = 0\n",
    "    history_count = 0\n",
    "    if n == 1:\n",
    "        word_count = model[word]\n",
    "        history_count = model.N()\n",
    "    else:\n",
    "        if history_str in model:\n",
    "            history_count = model[history_str].N()\n",
    "            if word in model[history_str]:\n",
    "                word_count = model[history_str][word]\n",
    "            \n",
    "    V = len(models[1])#number of types in the unigram model\n",
    "    prob = (word_count + k)/(history_count + (V*k))\n",
    "    \n",
    "    return prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the caclulation of entropy and perplexity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from math import log\n",
    "def entropy(sents, n, models, prob_func, alpha=1):\n",
    "        \"\"\"\n",
    "        Calculate the approximate cross-entropy of the n-gram model for a\n",
    "        given evaluation sentences.\n",
    "        This is the average log probability of each word in the text.\n",
    "\n",
    "        :param sents: sentences of tokens to use for evaluation\n",
    "        :type sents: list(list(str))\n",
    "        :param n: the n of n-gram model\n",
    "        :param models: all models from 1 to n. Used for backoff purposes\n",
    "        :param prob_func: probability function to compute P(word|history)\n",
    "        :param alpha: hyperparameter\n",
    "        \"\"\"\n",
    "        \n",
    "        e = 0.0\n",
    "        N = 0\n",
    "        for sent in sents:\n",
    "            pad = [\"~\" for x in range(n-1)]\n",
    "            N += len(sent) + len(pad) #include the end of sent tokens to the total count\n",
    "            sent = pad + sent + pad\n",
    "            for i in range(len(sent)-n+1):\n",
    "                history, word = sent[i:i+n-1], sent[i+n-1]\n",
    "                prob = prob_func(history, word, n, models, alpha)\n",
    "                if prob != 0:\n",
    "                    e += log(prob, 2)\n",
    "        \n",
    "        return -e / N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def perplexity(sents, n, models, prob_func, alpha=1):\n",
    "    return 2**entropy(sents, n, models, prob_func, alpha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now for the part of measuring our model:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Is perplexity reduced when increasing n?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n = 7\n",
    "ngram_models = create_models(sents_train, n) #create all ngram models from 1 to n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def compute_perp_diff_models(n, sents, models, prob_func, alpha=1):\n",
    "    \"\"\"\n",
    "    compute perplexity for each ngram model from 2 to n\n",
    "    \"\"\"\n",
    "    perplexities = []\n",
    "\n",
    "    for i in range(2,n+1):\n",
    "        p = perplexity(sents, i, models, prob_func, alpha)\n",
    "        perplexities.append(p)\n",
    "        \n",
    "    return perplexities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's observe the perplexity results on the validation dataset, using the Laplace smoothing (add 1 to each ngram count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[854.5197234923078,\n",
       " 3283.2826549773554,\n",
       " 5437.201093406795,\n",
       " 6395.441836914597,\n",
       " 6785.036744201539,\n",
       " 6971.589331987894]"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_tokens = [sent.split() for sent in words_valid_data.split(\"\\n\")] \n",
    "n = 7\n",
    "perplexities_valid_MLE_laplace_smoothing = compute_perp_diff_models(n, valid_tokens, ngram_models, prob_MLE_addk_smoothing)\n",
    "perplexities_valid_MLE_laplace_smoothing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x2ccf7bfc390>"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEKCAYAAADq59mMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xt8VPWd//HXh1u4JUEgRCAgIBEF5GZErNZq1YrWqr0h\nWhUtFRVate2vrnS327rddW231UpbrVSrKCClaitrtS1SrXUtYsIdBLlDIhAQSAIh98/vjznBKSpJ\nJGdOZvJ+Ph7zmDPfmTPzmUcr75zv+X7OmLsjIiLSFG2iLkBERJKPwkNERJpM4SEiIk2m8BARkSZT\neIiISJMpPEREpMkUHiIi0mQKDxERaTKFh4iINFm7qAsIS8+ePX3AgAFRlyEiklQKCgr2untWQ69L\n2fAYMGAA+fn5UZchIpJUzGxbY16naSsREWkyhYeIiDSZwkNERJpM4SEiIk0WWniY2RAzWx53KzWz\nO82su5ktNLMNwf0JcftMN7ONZrbezC6JGz/DzFYFz80wMwurbhERaVho4eHu6919lLuPAs4AyoHf\nA3cDi9w9F1gUPMbMhgITgWHAeOAhM2sbvN3DwM1AbnAbH1bdIiLSsERNW10IbHL3bcCVwKxgfBZw\nVbB9JTDP3SvdfQuwERhrZr2BDHdf7LGfPXwybh8REYlAovo8JgJPB9vZ7r4z2N4FZAfbfYHFcfsU\nBmPVwfbR4x9gZlOAKQD9+/dvlsJFRFqq8qoaiksrKS6rZHdpBcVllbx3sJLvXDKEsGf3Qw8PM+sA\nXAFMP/o5d3cza7YfUXf3mcBMgLy8PP04u4gkpYOVNRSXVrC7tJLisgr2xIVDcWklu8sq2FNaSVll\nzQf2bd/WmHrBYLqmhfvPeyKOPC4Flrr77uDxbjPr7e47gymp4mC8COgXt19OMFYUbB89LiKSNNyd\nsiAUjj5aqN+uD4nyqtoP7N+hXRuyM9Lold6RU09M57zcLLLS08jO6Eiv9DR6ZaSRnd6Rbp3bh37U\nAYkJj2t4f8oKYAEwCbgvuH8+bnyumd0P9CF2YnyJu9cGK7XGAW8CNwA/T0DdIiINcndKDle/f1Rw\nJBDqQ6LiSDhUVNd9YP9O7dse+Yd/aJ8Mzh+S9X4gpHc8EhgZndolJBQaK9TwMLMuwMXALXHD9wHz\nzWwysA2YAODua8xsPrAWqAGmuXt9/E4FngA6AS8FNxGR0Lg7+8urKS4Lpo+OTBv989FCcVklVTUf\nDIUuHdqSndGRrPQ0RuZ0e//oIBirD4auaS0rFBrLYguYUk9eXp7rwogicrS6OmdfedWRf/j3HHW0\nsLu0kj1lsVtV7QdDIb1jO3r903TRP9/Xj3cJ+ZxDWMyswN3zGnpdcn47EZFj2F1awYodB1i3q4zd\npfWBEIRFWSU1dR/8ozmzU/sjU0SDsrrQKz0uDDLSjkwjderQ9kM+sfVReIhIUjtQXsXKwhJWFh5g\nRXC/u7TyyPPdu3Q4cmSQm51+1FFDLBCy0tPo2F6h0BQKDxFJGocqa1hdVMKqopIjQbHtvfIjzw/q\n2YWzB/VgRE43RvbLZGjvTB0phEThISItUmVNLet2lv3TEcXG4oPUzzj1yezIiJxuXH1mP0bmdGN4\n30wyO7WPtuhWROEhIpGrrXM2Fh9kReEBVhYeYGVhCet2lh05Yd2jSwdG5GRy6fDejOyXyel9u5GV\nnhZx1a2bwkNEEsrd2b6vPHY0sSMWFKvfLTnSGJee1o7hfTO56dwBjMzpxoicTPp265SUy1lTmcJD\nREK1q6SCFYUHWFVYErsvKuFAeTUAae3aMKxPBhPy+jEiJ5MROd0Y1LMLbdooKFo6hYeINJv9h6pY\nWRQ7oqg/T1FcFlv51LaNMSQ7nUuHn8iI4IjilOx02rfVb9IlI4WHiHws9SufVgZHFCsLS9i+L27l\nU1YXzhnc88gRxbA+GVoOm0IUHiLSoIZWPvXt1okROZlcM7Y/I3MyGZ6TSUZHrXxKZQoPEfkntXXO\nhuIyVu4oYWVR7Iji7Z2lVNfGkqJn1w6MyOnGZaf3ZmRON07PyaRnV618am0UHiKtmLuz7b3yI9NO\nKwsPsLqolMPV7698Oj0nk8nnDmJkTiYj+nWjT2ZHrXwShYdIa1K/8mnlkbAooeTwP698uvrMfozs\nFztPMbCHVj7Jh1N4iKS48qoanvrHNp54Yys7SyoAaNfGGHJiejD1FAuK3OyuWvkkjabwEElRFdW1\nzF68jV/9bRN7D1bxydye3HLeIEb068bQ3lr5JMdH4SGSYiqqa5m3ZDsPvbqJ4rJKzhncg19ddAp5\nA7pHXZqkEIWHSIqoqqljfv4OfvnKRnaWVDB2YHdmXDOacYN6RF2apCCFh0iSq66t47mlhcxYtJGi\nA4cZ078bP/nySD5xcg+tipLQKDxEklRNbR1/WP4uMxZtYPu+ckb268a9Xzid83J7KjQkdAoPkSRT\nW+e8sPJdHnx5A5v3HmJ43wx+c2MeFwzppdCQhFF4iCSJujrnxdU7+dnLG9hYfJBTT0znkevP4DND\nsxUaknAKD5EWzt3585pdPLBwA+t3l5Hbqyu/vHYMlw4/UQ18EhmFh0gL5e4seruYB15+hzXvljKo\nZxcenDiKy0f0oa1CQyIWaniYWTfgUWA44MBXgfXAb4EBwFZggrvvD14/HZgM1AK3u/ufg/EzgCeA\nTsCLwB3u7mHWLhIVd+dv7+zhgYXvsKKwhJN6dOb+CSO5YmQf2qkDXFqIsI88HgT+5O5fMrMOQGfg\nu8Aid7/PzO4G7gb+xcyGAhOBYUAf4GUzO8Xda4GHgZuBN4mFx3jgpZBrF0kod+f/Nr7H/QvXs3T7\nAXJO6MSPvziCz4/pq8uGSIsTWniYWSZwHnAjgLtXAVVmdiVwfvCyWcCrwL8AVwLz3L0S2GJmG4Gx\nZrYVyHD3xcH7PglchcJDUsjize9x/8J3WLJlH30yO3Lv50/nS2fk0KGdQkNapjCPPAYCe4DHzWwk\nUADcAWS7+87gNbuA7GC7L7A4bv/CYKw62D56XCTp5W/dx/0L3+GNTe/RKz2N/7hyGFef2Y+0drru\nlLRsYYZHO2AM8A13f9PMHiQ2RXWEu7uZNdu5CzObAkwB6N+/f3O9rUizW7Z9P/cvfIe/b9hLz65p\n/PvlQ7n2rP66WKEkjTDDoxAodPc3g8fPEAuP3WbW2913mllvoDh4vgjoF7d/TjBWFGwfPf4B7j4T\nmAmQl5enE+rS4qwqLOGBl9/hr+uK6d6lA9+97FSuHzeATh0UGpJcQgsPd99lZjvMbIi7rwcuBNYG\nt0nAfcH988EuC4C5ZnY/sRPmucASd681s1IzG0fshPkNwM/DqlskDGvfLeWBl99h4drddOvcnrvG\nD2HS2QPokqbV8pKcwv5/7jeAOcFKq83ATUAbYL6ZTQa2ARMA3H2Nmc0nFi41wLRgpRXAVN5fqvsS\nOlkuSWL9rjIeXPQOL67aRXrHdnzr4lO46ZwBpHdsH3VpIsfFUrVdIi8vz/Pz86MuQ1qpjcUHeXDR\nBl5Y+S5dOrTjq+cOZPK5A8nspNCQls3MCtw9r6HX6ZhZpBlt3XuIGYs28IflRXRs35bbPnUyN39y\nECd06RB1aSLNSuEh0gx27CtnxqINPLesiPZtjZs/OYgp5w2iR9e0qEsTCYXCQ+Q4FB04zC/+upHf\n5e+gTRtj0tkDuPX8QfRK7xh1aSKhUniIfAy7Sir45SsbmffWdgzjK2f1Z+oFg8nOUGhI66DwEGmC\n4rIKHn51E3Pe3I67MyGvH9MuGEyfbp2iLk0koRQeIo2w92Alj/xtE08t3kZ1rfOlMTl8/dOD6de9\nc9SliURC4SFyDPsPVfHIa5uZ9cZWKmtq+fzoHG6/cDAn9egSdWkikVJ4iHyIkvJqHn19M795fQvl\n1bVcMbIPt1+Yy8lZXaMuTaRFUHiIxCmtqObx17fy6OubKauo4bOn9+bOi3LJzU6PujSRFkXhIQIc\nrKxh1htbmfnaZkoOV3PJsGzuvOgUTuudEXVpIi2SwkNatfKqGp78xzYe+dsm9pdXc9FpvbjzolMY\n3jcz6tJEWjSFh7RKFdW1zF68jV/9bRN7D1bxqVOy+ObFpzCqX7eoSxNJCgoPaXW2vXeIa2Yu5t2S\nCs4d3JNvXpzLGSd1j7oskaSi8JBWZd+hKm58/C3Kq2t5+uZxnH1yj6hLEklKCg9pNSqqa7n5yXyK\nDhxm7tfOIm+AjjZEPq42URcgkgh1dc43f7ucpdv387OrRyk4RI6TwkNahXtffJuXVu/iXy87jctO\n7x11OSJJT+EhKe+J/9vCo69v4cZPDGDyuQOjLkckJSg8JKX9Zc0u7nlhLRcPzeZ7lw/FzKIuSSQl\nKDwkZS3bvp/b5y1jRE43ZkwcTds2Cg6R5qLwkJS07b1DfG1WPr3SO/LYpDw6dWgbdUkiKUXhISmn\nvpej1p3HbzqTnvodcZFmpz4PSSnxvRxzvnaWLqEuEpJQjzzMbKuZrTKz5WaWH4x1N7OFZrYhuD8h\n7vXTzWyjma03s0vixs8I3mejmc0wnfWUD1FX53xr/nIKtu3ngQmjOFO9HCKhScS01QXuPsrd84LH\ndwOL3D0XWBQ8xsyGAhOBYcB44CEzq5+ofhi4GcgNbuMTULckmf9+6W1eXBXr5fjsCPVyiIQpinMe\nVwKzgu1ZwFVx4/PcvdLdtwAbgbFm1hvIcPfF7u7Ak3H7iACxXo5f/30Lk84+ia99Ur0cImELOzwc\neNnMCsxsSjCW7e47g+1dQHaw3RfYEbdvYTDWN9g+elwEeL+X46LTsvn3zw1TL4dIAoR9wvxcdy8y\ns17AQjNbF/+ku7uZeXN9WBBQUwD69+/fXG8rLdjyHQeO9HL8/Br1cogkSqhHHu5eFNwXA78HxgK7\ng6kogvvi4OVFQL+43XOCsaJg++jxD/u8me6e5+55WVlZzflVpAXa/l45k594i6z0NPVyiCRYaOFh\nZl3MLL1+G/gMsBpYAEwKXjYJeD7YXgBMNLM0MxtI7MT4kmCKq9TMxgWrrG6I20daqf2Hqrjx8SXU\nuvPETWPVyyGSYGFOW2UDvw/mn9sBc939T2b2FjDfzCYD24AJAO6+xszmA2uBGmCau9cG7zUVeALo\nBLwU3KSVqu/lKFQvh0hkLLaAKfXk5eV5fn5+1GVIM6urc77x9DL+uGonv7h2NJeP6BN1SSIpxcwK\n4lorPpIuTyJJ5b4/reOPq3by3ctOVXCIREjhIUlj1htbmfnaZm44+yRu/uSgqMsRadUUHpIUFq7d\nzT3/u4aLTuvF99XLIRI5hYe0eCt2HOAbTy/l9L6ZzFAvh0iLoPCQFm37e+VMnhXr5Xh00pl07qAL\nQYu0BAoPabH2H6rixieWUFMX6+XISlcvh0hLofCQFqmiupYpT+VTuO8wM6/PUy+HSAvTqPAws5+a\n2bCwixGBWC/Ht3+3gre27uenE0YydqB+l0OkpWnskcfbwEwze9PMbjWzzDCLktbtR39axx9X7mT6\npafyuZHq5RBpiRoVHu7+qLufQ+y6UgOAlWY218wuCLM4aX2e/MdWHnltM9ePO4kp56mXQ6SlavQ5\nj+BX/U4NbnuBFcC3zGxeSLVJK7Nw7W5+sKC+l2OoejlEWrBGrXs0sweAy4G/Ave6+5LgqR+Z2fqw\nipPWo76XY3jQy9GurdZyiLRkjV00vxL4N3c/9CHPjW3GeqQV2rEv1svRs2saj6mXQyQpNPbPu+uO\nDg4zWwTg7iXNXpW0GgfKq5j0+BKqa9XLIZJMjvknnpl1BDoDPc3sBKB+EjoD/Y64HKcjv8ux7zCz\nv3YWg3upl0MkWTQ0P3ALcCfQB1gaN14K/CKsoiT11dU5/y/o5fj5NaPVyyGSZI4ZHu7+IPCgmX3D\n3X+eoJqkFfjRn9fxwsqd3K1eDpGk1NC01afd/a9AkZl94ejn3f250CqTlPXUP7byyN82c924/tyi\nXg6RpNTQtNWniC3P/dyHPOeAwkOa5OW1u/n+gjVceGovfqDf5RBJWg1NW30/uL8pMeVIKltZeIBv\nPL2MYX0y+fm16uUQSWaNvTDiU/HXszKzk+qX6oo0xo595Xz1ibfo0bUDj92Yp14OkSTX2D/9Xgfe\nNLPLzOxmYCHws/DKklRyoLyKG4/0cpxJr/SOUZckIsepUX/+ufsjZrYGeIXYda1Gu/uuUCuTlFBZ\nU8uUpwrYse8wT00ey+Be6VGXJCLNoLHTVtcDvyF2Vd0ngBfNbGSIdUkKiPVyrGTJln38ZMJIzhrU\nI+qSRKSZNHba6ovAue7+tLtPB24FZjVmRzNra2bLzOyF4HF3M1toZhuC+xPiXjvdzDaa2XozuyRu\n/AwzWxU8N8O0RCcp/PjP6/nfFe/yL+NP5Qr1coiklMb+nsdV7l4c93gJjb8g4h3Efkyq3t3AInfP\nBRYFjzGzocBEYBgwHngouAw8wMPAzUBucBvfyM+WiDy1eBu/+tsmvnJWf279lHo5RFJNY6etTjGz\nRWa2Ong8ArirEfvlAJ8FHo0bvpL3j1pmAVfFjc9z90p33wJsBMaaWW8gw90Xu7sDT8btIy3Qord3\n8/3nV/PpU3txzxXq5RBJRY2dtvo1MB2oBnD3lcSOEhryM2IhUxc3lu3uO4PtXUB2sN0X2BH3usJg\nrG+wffT4B5jZFDPLN7P8PXv2NKI8aW4rCw/w9blBL4d+l0MkZTX2v+zOcT8AVa/mWDuY2eVAsbsX\nfNRrgiMJb2QNDXL3me6e5+55WVlZzfW20kixXo58uneJ9XJ0SVMvh0iqaux/3XvN7GSCf+jN7EvA\nzmPvwjnAFWZ2GdARyDCz2cBuM+vt7juDKan6cylFQL+4/XOCsaJg++hxaUFKyqu58fElVNXUMm/K\nWerlEElxjT3ymAY8ApxqZkXELtN+27F2cPfp7p7j7gOITXH91d2vAxYAk4KXTQKeD7YXABPNLM3M\nBhI7Mb4kmOIqNbNxwSqrG+L2kRagsqaWm5/KZ8e+w/z6hjz1coi0Ao1tEtwMXGRmXYA27l52HJ95\nHzDfzCYD24AJwWesMbP5wFpiU2LT3L022Gcqsf6STsBLwU1agPhejgcnjlIvh0grYbHTDh/xpNm3\njrWzu9/f7BU1k7y8PM/Pz4+6jJT3oz+t4+FXN3HX+CFMPX9w1OWIyHEyswJ3z2vodQ0deWj+QT7S\n7MXbePjVTVx7Vn9u+9TJUZcjIgnU0CXZ70lUIZJcFr29m39/fjUXDMniP9TLIdLqNLZJcJCZ/a+Z\n7TGzYjN73szUNtxK1fdyDO2TwS+uHaNeDpFWqLH/1c8F5gO9gT7A74CnwypKWq74Xo7f3HimejlE\nWqmmNAk+5e41wW02sd4NaUVKyqu56Ym3qKqp1e9yiLRyjf2z8SUzuxuYR6xR8Gpil2XvDuDu+0Kq\nT1qI2O9y5LP9vXKenDyW3GytpRBpzRobHhOC+1uOGp9ILEx0/iOF1dU53/ndSt4MejnGqZdDpNVr\nMDzMrA1wnbv/XwLqkRboJ39Zz4IV73LX+CFcOepDr0kpIq1Mg+c83L0O+EUCapEWaM6b23jo1U1c\nM1a9HCLyvsaeMF9kZl/UL/i1Lq+sK+Z7f4j1cvzwSvVyiMj7GhsetxBbnltlZqVmVmZmpSHWJRFb\nVVjCtLlL1cshIh+qsRdG1NKaVmTHvnK+OustTujcgd9MUi+HiHxQYzvMzcyuM7PvBY/7mVljf8Nc\nkkh9L0dFddDLkaFeDhH5oMbORTwEnA1cGzw+CPwylIokMpU1tdwyO59t7x1i5vV56uUQkY/U2PmI\ns9x9jJktA3D3/WbWIcS6JMHcnbueWcnizbFejrNPVi+HiHy0xh55VJtZW97/GdosoC60qiThfvKX\n9Ty//F2+c4l6OUSkYY0NjxnA74FeZvZfwOvAvaFVJQk1983t/PKVWC/H1PPVyyEiDWvsaqs5ZlYA\nXAgYcJW7vx1qZZIQS7bs43vPr+Z89XKISBMcMzzMrCNwKzAYWAU84u41iShMwldcVsHX5y6lf/fO\nzLhmtHo5RKTRGvrXYhaQRyw4LgV+EnpFkhA1tXV8fe4ySiuqefi6MWR0bB91SSKSRBqathrq7qcD\nmNljwJLwS5JE+PGf17Nkyz4euHokp56YEXU5IpJkGjryqK7f0HRV6nhp1U5mvraZ68edxOdH50Rd\njogkoYaOPEbGXcPKgE7BYwPc3fUna5LZtOcg33lmJSP7dePfLj8t6nJEJEkd88jD3du6e0ZwS3f3\ndnHbxwwOM+toZkvMbIWZrTGze4Lx7ma20Mw2BPcnxO0z3cw2mtl6M7skbvwMM1sVPDdDV/f9eMqr\narhtdgHt2xoPf2UMae3aRl2SiCSpMJfXVAKfdveRwChgvJmNA+4GFrl7LrAoeIyZDSX2y4TDgPHA\nQ0FjIsDDwM1AbnAbH2LdKcndufvZVWwoPsiMa0bTp1unqEsSkSQWWnh4zMHgYfvg5sCVxFZxEdxf\nFWxfCcxz90p33wJsBMaaWW8gw90Xu7sDT8btI4305D+2sWDFu3z74lP4ZG5W1OWISJILdWG/mbU1\ns+VAMbDQ3d8Est19Z/CSXUB2sN0X2BG3e2Ew1jfYPnpcGqlg237+849rufDUXkw9f3DU5YhICgg1\nPNy91t1HATnEjiKGH/W8E1wvqzmY2RQzyzez/D179jTX2ya1vQcrmTZnKb0zO3H/hFG0aaPTRSJy\n/BLSUuzuB4BXiJ2r2B1MRRHcFwcvKwL6xe2WE4wVBdtHj3/Y58x09zx3z8vK0tRMbZ1z+9PL2F9e\nxcPXjSGzsxoBRaR5hBYeZpZlZt2C7U7AxcA6YAEwKXjZJOD5YHsBMNHM0sxsILET40uCKa5SMxsX\nrLK6IW4fOYaf/mU9b2x6jx9eNZxhfTKjLkdEUkiYvy/aG5gVrJhqA8x39xfM7B/AfDObDGwDJgC4\n+xozmw+sBWqAae5eG7zXVOAJoBPwUnCTY1i4djcPvbqJa8b2Y0Jev4Z3EBFpAouddkg9eXl5np+f\nH3UZkdi69xCf+8XrDOjRhd/dejYd26ufQ0Qax8wK3D2vodfpMqop5nBVLbfOLqCNGQ99ZYyCQ0RC\nEea0lSSYu/Nvf1jN+t1l/ObGM+nXvXPUJYlIitKRRwqZu2Q7zy4t5PZP53LBkF5RlyMiKUzhkSJW\n7DjAPQvWct4pWdx+YW7U5YhIilN4pIB9h6qYOmcpWelpPHj1KNqqEVBEQqZzHkmuts65Y94y9pRV\n8sxtZ3NClw5RlyQirYDCI8k9uGgDf9+wl3s/fzojcrpFXY6ItBKatkpir6wrZsaiDXzpjByuGatG\nQBFJHIVHktqxr5w7f7uc03pn8MMrh6PfxxKRRFJ4JKGK6lpum1NAnTu/um4MnTqoEVBEEkvnPJLQ\nDxasYXVRKY/ekMdJPbpEXY6ItEI68kgy89/awby3djDtgpO5aGh2wzuIiIRA4ZFEVheV8L3nV3PO\n4B586+IhUZcjIq2YwiNJlJRXc9ucArp36cCMiaPVCCgikdI5jyRQV+d8c/5ydpVU8NtbzqZH17So\nSxKRVk5HHkngl69s5K/rivne5UMZ0/+EqMsREVF4tHR/37CH+19+h6tG9eH6cSdFXY6ICKDwaNGK\nDhzm9qeXcUqvdO79wulqBBSRFkPh0UJV1tQydc5Sqmudh68bQ+cOOj0lIi2H/kVqoX74wlpW7DjA\nr64bw6CsrlGXIyLyT3Tk0QI9t7SQ2Yu3c8t5gxg/vHfU5YiIfIDCo4V5e2cp3/39Ks4a2J3vXKJG\nQBFpmRQeLUhpRTW3zS4go2N7fn7taNq11f88ItIyhfavk5n1M7NXzGytma0xszuC8e5mttDMNgT3\nJ8TtM93MNprZejO7JG78DDNbFTw3w1Jw2ZG78//mr6Bw/2Ee+soYeqV3jLokEZGPFOaftjXAt919\nKDAOmGZmQ4G7gUXungssCh4TPDcRGAaMBx4ys/prjT8M3AzkBrfxIdYdiUde28xf1u5m+mWnkTeg\ne9TliIgcU2jh4e473X1psF0GvA30Ba4EZgUvmwVcFWxfCcxz90p33wJsBMaaWW8gw90Xu7sDT8bt\nkxLe2LSXH/9pHZ8d0ZuvnjMg6nJERBqUkEl1MxsAjAbeBLLdfWfw1C6g/rrifYEdcbsVBmN9g+2j\nx1PCrpIKbn96GQN7duFHXxyhRkARSQqhh4eZdQWeBe5099L454IjCW/Gz5piZvlmlr9nz57metvQ\nVNXUMW3uUsqrannk+jPomqa2GxFJDqGGh5m1JxYcc9z9uWB4dzAVRXBfHIwXAf3ids8JxoqC7aPH\nP8DdZ7p7nrvnZWVlNd8XCcm9L75Nwbb9/PhLIxjcKz3qckREGi3M1VYGPAa87e73xz21AJgUbE8C\nno8bn2hmaWY2kNiJ8SXBFFepmY0L3vOGuH2S1oIV7/LEG1v56jkDuXxEn6jLERFpkjDnSc4BrgdW\nmdnyYOy7wH3AfDObDGwDJgC4+xozmw+sJbZSa5q71wb7TQWeADoBLwW3pLVhdxl3P7uSvJNOYPpl\np0ZdjohIk1nstEPqycvL8/z8/KjL+ICDlTVc8YvXKT1czR9v/yTZGernEJGWw8wK3D2vodfpDG0C\nuTt3PbOCbe+VM3vyWQoOEUlauv5FAj32+hZeXLWLuy4Zwtkn94i6HBGRj03hkSBLtuzjv19ax/hh\nJzLlvEFRlyMiclwUHglQXFbBtLlL6d+9Mz/+shoBRST56ZxHyKpr6/j63GUcrKhh9uSzyOjYPuqS\nRESOm8IjZP/z5/Us2bKPn109iiEnqhFQRFKDpq1C9NKqncx8bTM3nH0SV41OmctxiYgoPMKyac9B\nvvPMSkb168a/fva0qMsREWlWCo8QlFfVcNvsAjq0a8NDXxlDWru2De8kIpJEFB7NzN25+9lVbCw+\nyIyJo+nTrVPUJYmINDuFRzN78h/bWLDiXb79mSGcm9sz6nJEREKh8GhGBdv2859/XMtFp/Xitk+d\nHHU5IiKhUXg0k70HK5k2Zym9Mzvx0wmjaNNGjYAikrrU59EMamrruP3pZewvr+K5qZ8gs5MaAUUk\ntSk8msFOzKGfAAAG8klEQVT9C9/hjU3v8T9fGsGwPplRlyMiEjpNWx2nv6zZxUOvbuKasf35cl6/\nhncQEUkBCo/jsHXvIb79uxWc3jeT739uaNTliIgkjMLjYzpcVcutswto28Z46Ctj6NhejYAi0nro\nnMfH4O786x9WsX53GY/feCb9uneOuiQRkYTSkcfHMHfJdp5bWsQdF+Zy/pBeUZcjIpJwCo8mWrHj\nAPcsWMv5Q7K4/dO5UZcjIhIJhUcT7DtUxdQ5S8lKT+MBNQKKSCumcx6NVFvn3DFvGXvKKnn2tk9w\nQpcOUZckIhIZhUcjPbhoA3/fsJf//sLpnJ6jRkARad1Cm7Yys9+YWbGZrY4b625mC81sQ3B/Qtxz\n081so5mtN7NL4sbPMLNVwXMzzCzhc0WvrCtmxqINfPmMHCaeqUZAEZEwz3k8AYw/auxuYJG75wKL\ngseY2VBgIjAs2OchM6tvnHgYuBnIDW5Hv2eoduwr587fLmdo7wx+eNVwIsguEZEWJ7TwcPfXgH1H\nDV8JzAq2ZwFXxY3Pc/dKd98CbATGmllvIMPdF7u7A0/G7RO6iupabptTgLvzq+vOUCOgiEgg0aut\nst19Z7C9C8gOtvsCO+JeVxiM9Q22jx7/UGY2xczyzSx/z549x13sDxasYXVRKQ9cPYr+PdQIKCJS\nL7KlusGRhDfze8509zx3z8vKyjqu95r/1g7mvbWDr18wmAtPy254BxGRViTR4bE7mIoiuC8OxouA\n+DPROcFYUbB99HioVheV8G/Pr+bcwT355sWnhP1xIiJJJ9HhsQCYFGxPAp6PG59oZmlmNpDYifEl\nwRRXqZmNC1ZZ3RC3TyhKyqu5bU4BPbt04MGJo2irRkARkQ8Irc/DzJ4Gzgd6mlkh8H3gPmC+mU0G\ntgETANx9jZnNB9YCNcA0d68N3moqsZVbnYCXglso6uqcb85fzq6SCubfcjY9uqaF9VEiIkkttPBw\n92s+4qkLP+L1/wX814eM5wPDm7G0j1TnTm6vrlwwJIvR/U9oeAcRkVZKHeZx2rVtw/TLTou6DBGR\nFk8XRhQRkSZTeIiISJMpPEREpMkUHiIi0mQKDxERaTKFh4iINJnCQ0REmkzhISIiTWaxi9umHjPb\nQ+wSKB9HT2BvM5aTDPSdW4fW9p1b2/eF4//OJ7l7g5clT9nwOB5mlu/ueVHXkUj6zq1Da/vOre37\nQuK+s6atRESkyRQeIiLSZAqPDzcz6gIioO/cOrS279zavi8k6DvrnIeIiDSZjjxERKTJFB4BM+tn\nZq+Y2VozW2Nmd0RdU9jMrKOZLTGzFcF3vifqmhLFzNqa2TIzeyHqWhLBzLaa2SozW25m+VHXkwhm\n1s3MnjGzdWb2tpmdHXVNYTKzIcH/vvW3UjO7M7TP07RVjJn1Bnq7+1IzSwcKgKvcfW3EpYUm+F34\nLu5+0MzaA68Dd7j74ohLC52ZfQvIAzLc/fKo6wmbmW0F8ty91fQ8mNks4O/u/qiZdQA6u/uBqOtK\nBDNrCxQBZ7n7x+13OyYdeQTcfae7Lw22y4C3gb7RVhUujzkYPGwf3FL+rwkzywE+CzwadS0SDjPL\nBM4DHgNw96rWEhyBC4FNYQUHKDw+lJkNAEYDb0ZbSfiC6ZvlQDGw0N1T/jsDPwPuAuqiLiSBHHjZ\nzArMbErUxSTAQGAP8HgwPfmomXWJuqgEmgg8HeYHKDyOYmZdgWeBO929NOp6wubute4+CsgBxprZ\n8KhrCpOZXQ4Uu3tB1LUk2LnB/86XAtPM7LyoCwpZO2AM8LC7jwYOAXdHW1JiBFN0VwC/C/NzFB5x\ngnn/Z4E57v5c1PUkUnBI/wowPupaQnYOcEVwDmAe8Gkzmx1tSeFz96Lgvhj4PTA22opCVwgUxh1J\nP0MsTFqDS4Gl7r47zA9ReASCk8ePAW+7+/1R15MIZpZlZt2C7U7AxcC6aKsKl7tPd/ccdx9A7ND+\nr+5+XcRlhcrMugSLQAimbj4DrI62qnC5+y5gh5kNCYYuBFJ28ctRriHkKSuIHdpJzDnA9cCq4BwA\nwHfd/cUIawpbb2BWsDKjDTDf3VvF0tVWJhv4fezvI9oBc939T9GWlBDfAOYE0zibgZsirid0wR8H\nFwO3hP5ZWqorIiJNpWkrERFpMoWHiIg0mcJDRESaTOEhIiJNpvAQEZEmU3iIiEiTKTxERKTJFB4i\nCWJmA4Lflfh18Pspfwk6+0WSjsJDJLFygV+6+zDgAPDFiOsR+VgUHiKJtcXd6y9/UwAMiLAWkY9N\n4SGSWJVx27Xo+nKSpBQeIiLSZAoPERFpMl1VV0REmkxHHiIi0mQKDxERaTKFh4iINJnCQ0REmkzh\nISIiTabwEBGRJlN4iIhIkyk8RESkyf4/8StdGgZtHKwAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ccf7bf8080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(2,8), perplexities_valid_MLE_laplace_smoothing)\n",
    "plt.xlabel(\"n\")\n",
    "plt.ylabel(\"Perplexity\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The perplexity is ok for n=2, but it should be improved with larger ngram model and it doesn't.\n",
    "\n",
    "Maybe adding k<1 could improve that. We will try to improve perplexity on n=2. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[399.46742337770513,\n",
       " 481.4238597127433,\n",
       " 547.069482374209,\n",
       " 603.5264664100564,\n",
       " 653.8614059439122,\n",
       " 699.7286350257758,\n",
       " 742.143075784467,\n",
       " 781.7815304713514,\n",
       " 819.1225835710153]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_tokens = [sent.split() for sent in words_valid_data.split(\"\\n\")] \n",
    "n = 2\n",
    "perplexities_valid_MLE_addk_n2 = []\n",
    "for k in np.arange(0.1, 1.0, 0.1):\n",
    "    p = perplexity(valid_tokens, n, ngram_models, prob_MLE_addk_smoothing, k)\n",
    "    perplexities_valid_MLE_addk_n2.append(p)\n",
    "perplexities_valid_MLE_addk_n2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x2cce39b2048>"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4VHXaxvHvQwkQeid0Qu8goeiiIuqqyyqu8iJWVhF0\n7brrWi531VXf1dW1rghYUQHFwoIVBcQOGKR3SGihBKSXhJTn/WOG3egbTQKZnJnk/lxXrpw5c2Zy\nMyS5M+f8zu+YuyMiIvJT5YIOICIi0UkFISIi+VJBiIhIvlQQIiKSLxWEiIjkSwUhIiL5UkGIiEi+\nVBAiIpIvFYSIiOSrQtABjke9evW8ZcuWQccQEYkp8+fP3+nu9QvaLqYLomXLliQnJwcdQ0QkppjZ\nhsJsp11MIiKSLxWEiIjkSwUhIiL5UkGIiEi+VBAiIpIvFYSIiORLBSEiIvlSQYiIxJCMrBwmzdvI\nx0u3RvxrxfSJciIiZcXew1m8PmcDL3+9np0HMjm3e2PO7pIQ0a+pghARiWJb9x7mpa9SmTh3IweP\n5HBKu/pce2oiJybWjfjXVkGIiEShNdv3M/aLFKYuTCPX4bfdErjmlNZ0alyjxDKoIEREooS7k7xh\nN2Nmr2PmynSqVCzPpX1bMKJ/K5rViS/xPCoIEZGA5eY6n67YztjP1/H9xj3UqRrHrWe044oTW1C7\nalxguVQQIiIByczO4d8L0hj7RQopOw7SrE4VHhjcmSG9mlElrnzQ8SJbEGZ2K3A14MAS4EogHngT\naAmsB4a6++7w9ncBI4Ac4CZ3nx7JfCIiQdiXkcXEuRt56atU0vdn0rlxDZ65uCfndGlEhfLRc/ZB\nxArCzJoANwGd3P2wmU0GhgGdgJnu/rCZ3QncCdxhZp3C93cGGgMzzKydu+dEKqOISEnavi+Dl75O\nZeKcjezPzObktvV4fGgPftWmLmYWdLz/J9K7mCoAVcwsi9A7hy3AXcCA8P3jgdnAHcBg4A13zwRS\nzWwt0Af4NsIZRUQiam36fsZ9kcKUBWnk5DqDujXmmlMS6dKkZtDRflHECsLd08zsMWAjcBj4xN0/\nMbOG7n70FMBtQMPwchNgTp6n2Bxe9yNmNgoYBdC8efNIxRcROW7zN+xizOcpfLp8O5UrluPiPs0Z\neXJiICOSjkUkdzHVJvSuoBWwB3jLzC7Lu427u5l5UZ7X3ccB4wCSkpKK9FgRkUjLzXVmrUxnzOfr\nSN6wm1rxFbn59LZccWIL6larFHS8IonkLqYzgFR33wFgZu8CJwHbzSzB3beaWQKQHt4+DWiW5/FN\nw+tERKLekexcpi5MY9wXKaxJP0CTWlW479xODO3djPi42BwwGsnUG4F+ZhZPaBfT6UAycBAYDjwc\n/jw1vP00YKKZPU7oIHVbYF4E84mIHLf9GVlMmreRl75az7Z9GXRMqMFTw3owqGtCVI1IOhaRPAYx\n18zeBr4HsoEFhHYNVQMmm9kIYAMwNLz9svBIp+Xh7a/XCCYRiVbp+zJ4+Zv1vD5nA/szsjmpdV3+\nMaQbJ7etF5Ujko6FucfubvykpCRPTk4OOoaIlCEpOw7w/JcpvDM/jezcXM7pmsA1pyTSrWmtoKMV\nmpnNd/ekgraLzR1jIiIlbMHG3Yz9PIXpy7cRV74cQ3s35er+ibSsVzXoaBGjghAR+Rnuzmer0hnz\neQrzUndRs0pFbjitDcNPakm9GBuRdCxUECIiP5GT63y4ZCvPfraWldv207hmZf76205c1LsZVSuV\nnV+bZedfKiJSgOycXN5fvJVnZq1h3Y6DtGlQjceHdufc7o2pGOMjko6FCkJEyrysnFz+vSCN0bPX\nkbrzIB0aVefZS07gnC6NKFeudIxIOhYqCBEps45k5/LO95sZPXstm3YdpnPjGoy9vBdndmxYpovh\nKBWEiJQ5mdk5TE7ezHOfrWXL3gy6N6vFfed2ZmCHBqXmHIbioIIQkTIjIyuHSfM2MubzdWzfl0mv\nFrX5+4XdOKUUndxWnFQQIlLqHTqSzYQ5Gxn7RQo7D2TSt1UdnhjagxNbR+d1GKKFCkJESq0Dmdm8\n9u0Gnv8yhV0Hj9C/TT1uHNiTvol1g44WE1QQIlLq7MvIYvzX63nx61T2HMpiQPv63DiwLb1a1A46\nWkxRQYhIqbHn0BFe+no9L3+dyv6MbM7o2IAbB7ale7PYmScpmqggRCTm7Tp4hBe+TOHVbzdwIDOb\nszs34oaBbaL+kp7RTgUhIjFrx/5Mnv8yhdfnbOBwVg6DuiZww8A2dGhUI+hopYIKQkRizvZ9GYz9\nPIWJ8zZwJDuX87o35oaBbWjToHrQ0UoVFYSIxIwtew4z5vN1vPHdJnJynd/1bML1p7WhVSmecjtI\nKggRiXqbdh1i9Ox1vD1/EwBDejXlD6e2oXnd+ICTlW4qCBGJWut3HmT07LW8+30a5cwY1rs51w5o\nTZNaVYKOViaoIEQk6qzbcYBnZ63l3wvTqFi+HJf1a8G1p7amUc3KQUcrU1QQIhI1Vm/fzzOz1vL+\n4i1UrlCeEf1bMfKURBpUVzEEQQUhIoFbsXUfz8xaw4dLtlE1rjzXntqaq/u3om4ZuKxnNFNBiEhg\nUnYc4IkZa3hv0RaqV6rAjQPbcNWvWlG7alzQ0QQVhIgEYMuewzw9cw1vzd9MpQrluOG0Now8OZGa\n8RWDjiZ5qCBEpMT8cCCT0bPX8dqcDeBwxYktuG5AG+pX166kaKSCEJGI25eRxQtfpvLilykczsph\nSK+m3HR6W5rW1nkM0UwFISIRk5GVw6vfrmf07HXsOZTFoK4J3HpmO9o0qBZ0NCkEFYSIFLusnFwm\nJ2/i6Zlr2L4vk1Pb1edPv25P16aaXTWWqCBEpNjk5jrTFm3h8U9Xs3HXIZJa1ObpYbqCW6xSQYjI\ncXN3ZqxI55+frGLltv10TKjBy7/vzYD29XXN5ximghCR4/LNup08On0VCzbuoVW9qjxzcU8GdU2g\nXDkVQ6xTQYjIMVm0aQ+PfbKKL9fsJKFmZR6+oCsX9mpKxfLlgo4mxUQFISJFsmb7fh77ZBXTl22n\nTtU47hnUkcv6taByxfJBR5NipoIQkULZtOsQT8xYzb8XpBEfV4Fbz2jHiJNbUa2Sfo2UVvqfFZFf\nlL4/g3/NWsukeRspZ8bVJyfyh1Nba76kMkAFISL52nsoizFfrOPlr1PJznGG9m7GTQPb6poMZYgK\nQkR+5GBmNq98s54xn6/jQGY2g7s35pYz2tFS130uc1QQIgJAZnYOE+du5NnP1rLzwBHO6NiQP/66\nHR0TagQdTQKighAp47Jzcnl3QRpPzVhD2p7D9Eusw9jLO9CrRe2go0nAIlYQZtYeeDPPqkTgr8Cr\n4fUtgfXAUHffHX7MXcAIIAe4yd2nRyqfSFnn7ny0dBv//GQV63YcpFvTmjx8YVf6t6mns58FiGBB\nuPsqoAeAmZUH0oApwJ3ATHd/2MzuDN++w8w6AcOAzkBjYIaZtXP3nEhlFCmL3J0v1uzksemrWJK2\nl7YNqjHmsl6c1bmhikF+pKR2MZ0OrHP3DWY2GBgQXj8emA3cAQwG3nD3TCDVzNYCfYBvSyijSKk3\nf8MuHvl4FfNSd9G0dhX++T/dOb9nE8prWgzJR0kVxDBgUni5obtvDS9vAxqGl5sAc/I8ZnN43Y+Y\n2ShgFEDz5s0jElaktNm06xAPf7SSD5ZspV61SvxtcGcu6t2MShV09rP8vIgXhJnFAecBd/30Pnd3\nM/OiPJ+7jwPGASQlJRXpsSJlzf6MLEbPXseLX6VS3oxbzmjLqFMSiY/T+BQpWEl8l5wDfO/u28O3\nt5tZgrtvNbMEID28Pg1oludxTcPrRKSIcnKdt5I38dgnq9l5IJMLejbh9rPbk1CzStDRJIaUREFc\nzH93LwFMA4YDD4c/T82zfqKZPU7oIHVbYF4J5BMpVb5Zu5MHPljBiq37SGpRmxeHJ9G9Wa2gY0kM\nimhBmFlV4EzgmjyrHwYmm9kIYAMwFMDdl5nZZGA5kA1crxFMIoWXuvMg//vhCj5dvp0mtarwr0tC\n12XQyCQ5VhEtCHc/CNT9ybofCI1qym/7h4CHIplJpLTZeyiLp2et4dVv1xNXvhy3n9WeEf1bafpt\nOW46UiUSo7Jzcpk4byNPfLqaPYezuCipGbf9uh0NqmsyPSkeKgiRGDR7VToPfrCCtekHODGxLvf8\ntiOdG9cMOpaUMioIkRiyZvt+HvxgBZ+v3kHLuvGMu7wXZ3bSGdASGSoIkRiw6+ARnpyxmglzNxIf\nV557BnXkihNbEldB13+WyFFBiESxI9m5vPrtep6euYaDR3K4pE9zbjmjLXWrVQo6mpQBKgiRKOTu\nfLp8O3//aCWpOw9ySrv63DOoI+0aVg86mpQhKgiRKLN8yz4e/GA536z7gTYNqvHylb05rX2DoGNJ\nGaSCEIkSO/Zn8s9PVvFm8iZqVqnI/ed15pK+zalYXscZJBgqCJGAZWTl8NLXqYz+bB0ZWTlc9atW\n3DSwLTXjKwYdTco4FYRIQNydD5ds4+8frWDz7sOc0bEBd/+mI4n1qwUdTQRQQYgEYvHmPTzw/nK+\nW7+bDo2qM+HqvvyqTb2gY4n8iApCpARt25vBP6av5N3v06hXLY6/X9CVoUnNdEU3iUoqCJEScPhI\nDuO+SGHM5+vIyXWuPbU115/WmuqVdZxBopcKQiSCcnOdqYvS+MfHq9i6N4PfdG3EnWd3pHnd+KCj\niRSoUAVhZv8EXnL3ZRHOI1JqzN+wm7+9v5xFm/bQpUkNnhrWkz6t6gQdS6TQCvsOYgUwzswqAC8D\nk9x9b+RiicSu9H0ZPPjBCqYt2kKD6pV47H+6c0HPJpTTcQaJMYUqCHd/AXjBzNoDVwKLzexr4Hl3\n/yySAUViRU6uM2HuBh79eBWZObncOLAN157amqqVtCdXYlOhv3PNrDzQIfyxE1gE3GZm17j7sAjl\nE4kJy7bs5e4pS1m0aQ/929TjgfO70Kpe1aBjiRyXwh6DeAL4LTAL+F93nxe+6xEzWxWpcCLR7mBm\nNk/OWM1LX6+ndnxFnryoB4N7NNb1GaRUKOw7iMXAPeFrTP9Un2LMIxIzZizfzr3TlpG25zAX92nG\nHWd3oFZ8XNCxRIpNYQviMnd/Oe8KM5vp7qfrYLWUNVv3Hub+acv5eNk22jWsxtvXnkhSS41OktLn\nFwvCzCoD8UA9M6sNHH3fXANoEuFsIlElJ9d59dv1PDZ9Fdm5zu1ntWfkyYm6qpuUWgW9g7gGuAVo\nDHyfZ/0+4F+RCiUSbZam7eXuKUtYvHkvp7Srz4ODu+hkNyn1frEg3P0p4Ckzu9HdnymhTCJR40Bm\nNo9/sppXvkmlbrVKPHNxT37bLUEHoaVMKGgX00B3nwWkmdkFP73f3d+NWDKRgE1fto37pi1j274M\nLu3bnNvP6kDNKpo7ScqOgnYxnUpoaOu5+dzngApCSp20PYe5d+oyZqzYTodG1Xn20hM4oXntoGOJ\nlLiCdjHdG/58ZcnEEQlOdk4ur3yznsc/XY073HVOB67q30qX/JQyq7Anyr0G3HB0SKuZtSA0ed/p\nkQwnUlIWbdrDXe8uYfnWfQzs0ID7z+tMszo6CC1lW2HPg/gKmGtmtxEa3no78MeIpRIpIfsysvjn\n9FW8OmcDDapX4rlLT+DsLo10EFqEwk/WN9bMlgGfEZqHqae7b4toMpEIcnc+WrqN+99bRvr+TIaf\n2JI//rqdLuAjkkdhdzFdDvwFuALoBnxoZle6+6JIhhOJhE27DnHvtGXMWplOp4QajL08iR7NagUd\nSyTqFHYX04VAf3dPByaZ2RRgPNAjYslEillWTi4vfZXKkzPWYAb3DOrI709qSQUdhBbJV2F3MZ3/\nk9vzzEyT9EnM+H7jbu5+dwkrt+3nzE4Nue+8zjSpVSXoWCJRrbC7mNoBzwEN3b2LmXUDzgMejGQ4\nkeO193AWj05fyYS5G2lUozJjL+/FWZ0bBR1LJCYUdhfT84RGLo0FcPfFZjYRFYREKXfn/cVb+dv7\ny/nhQCZXntSK237djmq6uptIoRX2pyU+vFsp77rsCOQROW4bfzjEPVOX8sXqHXRtUpOXf9+bLk1q\nBh1LJOYUtiB2mllrQtNrYGZDgK0RSyVyDLJycnn+yxSemrGGiuXLcd+5nbj8xJaUL6dzGkSORWEL\n4npgHNDBzNKAVOCyiKUSKaLk9bu4e8oSVm8/wDldGnHvuZ1pVLNy0LFEYlphRzGlAGeYWVWgnLvv\nL8zjzKwW8ALQhdC7j6uAVcCbQEtgPTDU3XeHt78LGAHkADe5+/Si/GOk7Nlz6AiPfLySSfM20aRW\nFV64IokzOjUMOpZIqVDQdN+3/cx6ANz98QKe/yngY3cfYmZxhK5Odzcw090fNrM7gTuBO8ysEzAM\n6EzoAkUzzKydu+cU5R8kZceM5du5893F7D6UxahTErn59LZU1UFokWJT0E9T9WN9YjOrCZwC/B7A\n3Y8AR8xsMDAgvNl4YDZwBzAYeMPdM4FUM1sL9AG+PdYMUjodzMzmwQ9WMGneRjom1GD8VX3o3FgH\noUWKW0HTfd9/HM/dCtgBvGxm3YH5wM2EzqU4eoB7G3B0f0ATYE6ex28mn+tem9koYBRA8+bNjyOe\nxKIFG3dz65sL2bDrENecmshtZ7ajUoXyQccSKZUKNceAmSWa2XtmtsPM0s1sqpklFvCwCsAJwHPu\n3hM4SGh30n+4uxMeGVVY7j7O3ZPcPal+/fpFeajEsKycXJ74dDVDxnxLVo4zaWQ/7jqno8pBJIIK\nOwnNRGAykEDo+MBbwKQCHrMZ2Ozuc8O33yZUGNvNLAEg/Dk9fH8a0CzP45uG10kZl7LjAEPGfMtT\nM9cwuHtjPrrlZPol1g06lkipV9iCiHf319w9O/zxOvCLYwjD04FvMrP24VWnA8uBacDw8LrhwNTw\n8jRgmJlVMrNWQFtgXhH+LVLKuDuvz9nAoKe/Yv3Ogzx7yQk8flEPamhKbpESUdghHx+FRxy9QWiX\n0EWEpvyuA+Duu37mcTcCE8IjmFKAKwmV0mQzGwFsAIaGn2OZmU0mVCLZwPUawVR27difyR3vLGbW\nynRObluPR4d013kNIiXMQocBCtjILPUX7nZ3L+h4REQkJSV5cnJyEF9aIuiTZdu4890lHMzM5q5z\nOnDFiS0pp7OhRYqNmc1396SCtivwHYSZlQMuc/eviyWZyM84mJnN395bzpvJm+jcuAZPXtSDtg2P\neaS1iBynAgvC3XPN7F9AzxLII2XU/A2h4aubdh/iugGtueWMdsRV0IV8RIJU2GMQM83sQuBdL8w+\nKZFCysrJ5emZa3j2s7U0rlWFydecSO+WdYKOJSIUviCuAW4DcszsMGCEjj3UiFgyKfXW7TjArW8u\nZPHmvQzp1ZR7z+1EdY1QEokahZ2sTzuCpdgcHb760IcrqFKxPGMuO4GzuyQEHUtEfqKwlxw14FKg\nlbs/YGbNgAR313kKUiTp+zL48zuLmb1qB6e2q8+jQ7rRoIaGr4pEo8LuYhoN5AIDgQeAA8CzQO8I\n5ZJS6OOl27jr3cUczsrhb4M7c3m/Fv+ZGVhEok9hC6Kvu59gZgsA3H13+OQ3kQIdyMzm/mnLeGv+\nZro2qckTF/WgTYNqQccSkQIUtiCyzKw8/73kaH1C7yhEflHy+l3cOnkhabsPc+PANtx0elsqltfw\nVZFYUNiCeBqYAjQws4eAIcA9EUslMe9Idi5PzVzNc7PX0bR2PG9deyK9Wmj4qkgsKewopglmNp/Q\nhHsGnO/uKyKaTGLW2vT93PLmQpam7eOipGb85dxOVNOV3kRiTkGXHK0MXAu0AZYAY909uySCSexx\nd8Z/s56/f7SSqpUqMPbyXpzVuVHQsUTkGBX0Z914IAv4EjgH6AjcEulQEnu278vgT28t4ss1Ozmt\nfX0eGdKNBtU1fFUklhVUEJ3cvSuAmb2Irs8g+fhwyVbunrKEjKwcHjy/C5f2ba7hqyKlQEEFkXV0\nwd2z9UMvee3PyOK+act55/vNdG8aGr6aWF/DV0VKi4IKoruZ7QsvG1AlfFtzMZVx81J3ceubC9m2\nL4ObTm/LjQPbaPiqSCnziwXh7roivPzIkexcHv90NWO/WEfzOqHhqyc0rx10LBGJAI09lEJbvX0/\nt7yxkOVb93Fxn2bcM6gTVTV8VaTU0k+3FMjdeSU8fLV6pQo8f0USZ3ZqGHQsEYkwFYT8osNHcrj9\n7UW8v3grp3dowMMXdqN+9UpBxxKREqCCkJ+Vtucwo15NZvnWfdx5TgeuOSVRw1dFyhAVhORrbsoP\nXDfhe47k5PLS8N6c1qFB0JFEpISpIOT/eX3OBu6btozmdeN5/ookWuvcBpEySQUh/3EkO5f73lvG\nxLkbOa19fZ66uCc1dI1okTJLBSEA7DyQyR9en89363dz3YDW/PHX7SlfTscbRMoyFYSwNG0vo15N\nZtehIzx9cU/O69446EgiEgVUEGXctEVb+PPbi6gTH8fb155ElyY1g44kIlFCBVFG5eQ6j32yiudm\nr6NPyzqMvuwE6lXT+Q0i8l8qiDJoX0YWN09awGerdnBp3+bce25n4ipooj0R+TEVRBmzbscBRr6a\nzMYfDvHg+V24rF+LoCOJSJRSQZQhn61M56ZJC4irUI6JI/vRp1WdoCOJSBRTQZQB7s6Yz1P4x/SV\ndEqowbgrkmhSq0rQsUQkyqkgSrnDR3L48zuLeW/RFs7t3ph/XNiNKnG6zIeIFEwFUYrlnWzvjrM7\ncO2pmmxPRApPBVFKzUvdxR9en8+RbE22JyLHRgVRCk2Yu4F7py6jeZ14nh+uyfZE5NioIEqRI9m5\n3P/eMiaEJ9t7clhPalbRZHsicmxUEKXEzgOZXPf698xbv4s/DGjNnzTZnogcp4gWhJmtB/YDOUC2\nuyeZWR3gTaAlsB4Y6u67w9vfBYwIb3+Tu0+PZL7SIu9ke08N68HgHk2CjiQipUBJzK9wmrv3cPek\n8O07gZnu3haYGb6NmXUChgGdgbOB0Wam8ZgFmLZoC0PGfAPA29eepHIQkWITxAQ8g4Hx4eXxwPl5\n1r/h7pnungqsBfoEkC8m5OQ6j3y8kpsmLaBrk5pMu7G/ZmIVkWIV6WMQDswwsxxgrLuPAxq6+9bw\n/duAhuHlJsCcPI/dHF4nP7EvI4tb3ljIrJXpXNK3Ofdpsj0RiYBIF0R/d08zswbAp2a2Mu+d7u5m\n5kV5QjMbBYwCaN68efEljREpOw5wtSbbE5ESENE/O909Lfw5HZhCaJfRdjNLAAh/Tg9vngY0y/Pw\npuF1P33Oce6e5O5J9evXj2T8qPPZqnQGP/s1ew9lMeHqvioHEYmoiBWEmVU1s+pHl4FfA0uBacDw\n8GbDganh5WnAMDOrZGatgLbAvEjliyWhyfbWcdUr39GsdjxTb/gVfRPrBh1LREq5SO5iaghMCc/9\nUwGY6O4fm9l3wGQzGwFsAIYCuPsyM5sMLAeygevdPSeC+WJCRlYOd7yzmKkLtzCoWwKPDulGfJxO\nXxGRyIvYbxp3TwG657P+B+D0n3nMQ8BDkcoUa7bsOcyo15JZtmUft5/VnusGtNZkeyJSYvSnaJT6\nbn1osr2MrFxeuCKJ0zs2LPhBIiLFSAURhSbN28hfpy6lae143hjVizYNqgcdSUTKIBVEFMnOyeX+\n95bz2pwNnNquPk9frMn2RCQ4KogokZPr/PGtRUxduIVrTknkz2d30GR7IhIoFUQUyMl1bg+Xw5/P\nbs91A9oEHUlEJJC5mCSP3FznjncW8+6CNP54ZjuVg4hEDRVEgHJznbunLOHt+Zu55Yy23Hh626Aj\niYj8hwoiIO7OPVOX8sZ3m7hxYBtuVjmISJRRQQTA3bl32jImzt3IHwa05rYz2+kEOBGJOiqIEubu\n3P/ecl79dgOjTknkz2e1VzmISFRSQZQgd+fBD1bwyjfrGdG/FXed00HlICJRSwVRQtydhz9ayYtf\npfL7k1pyz6COKgcRiWoqiBLg7jw6fRVjv0jh8n4tuPfcTioHEYl6KogS8MSnqxk9ex2X9G3O/ed1\nVjmISExQQUTYUzPW8PSstVyU1IwHB3ehnKbPEJEYoYKIoH/NWsMTM1YzpFdT/n5BV5WDiMQUFUSE\nPDd7HY99spoLejbhkQu7qRxEJOaoICLg+S9SeOTjlQzu0ZhH/6e7ZmUVkZikgihmL36VykMfrmBQ\ntwT+qXIQkRimgihG479ZzwPvL+ecLo148qIeVCivl1dEYpd+gxWT1+Zs4N5pyzirc0OevrgnFVUO\nIhLj9FusGEycu5G//HspZ3RswDMXn6ByEJFSQb/JjtPk7zZx95QlDOzQgGcvPYG4CnpJRaR00G+z\n4/D2/M3c8e5iTm1Xn9GXnkClCuWDjiQiUmxUEMdoyoLN3P72Ivq3qcfYy3tRuaLKQURKFxXEMZi6\nMI0/Tl7EiYl1GXd5kspBREolFUQRvb94C7e+uZA+rerwwvAkqsSpHESkdFJBFMFHS7Zy8xsLSWpR\nhxeH9yY+rkLQkUREIkYFUUjTl23jxkkL6NGsFi9d2ZuqlVQOIlK6qSAKYcby7dww8Xu6Nq3JK1f2\npprKQUTKABVEAT5bmc51E76nU0INxl/Vh+qVKwYdSUSkRKggfsHnq3dwzevzad+oOq+O6EsNlYOI\nlCEqiJ/x1ZqdjHw1mTb1q/HaiD7UrKJyEJGyRQWRj2/W7mTE+O9IrFeVCVf3pVZ8XNCRRERKnAri\nJ+ak/MCI8cm0rBsqh9pVVQ4iUjapIPKYl7qLq175jqa1qzBhZF/qVqsUdCQRkcCoIMLmb9jFlS/P\nI6FmZSaM7Es9lYOIlHEqCGDBxt0Mf+k7GtaozKSR/WhQvXLQkUREAhfxgjCz8ma2wMzeD9+uY2af\nmtma8Ofaeba9y8zWmtkqMzsr0tkAFm3awxUvzqNutTgmjuxHgxoqBxERKJl3EDcDK/LcvhOY6e5t\ngZnh25hZJ2AY0Bk4GxhtZhGdCW9p2l4uf3EutavGMWlkPxrVVDmIiBwV0YIws6bAIOCFPKsHA+PD\ny+OB8/NqJ8mnAAAGhUlEQVSsf8PdM909FVgL9IlUtmVb9nLpC3OpUaUik0b1o3GtKpH6UiIiMSnS\n7yCeBP4M5OZZ19Ddt4aXtwENw8tNgE15ttscXlfsVm3bz2UvzKVapQpMGtmPJioHEZH/J2IFYWa/\nBdLdff7PbePuDngRn3eUmSWbWfKOHTuOKVvt+Ip0aVKTiSP70qxO/DE9h4hIaRfJaUl/BZxnZr8B\nKgM1zOx1YLuZJbj7VjNLANLD26cBzfI8vml43Y+4+zhgHEBSUlKRyuWoBjUq89qIvsfyUBGRMiNi\n7yDc/S53b+ruLQkdfJ7l7pcB04Dh4c2GA1PDy9OAYWZWycxaAW2BeZHKJyIivyyICxs8DEw2sxHA\nBmAogLsvM7PJwHIgG7je3XMCyCciIoCFDgPEpqSkJE9OTg46hohITDGz+e6eVNB2OpNaRETypYIQ\nEZF8qSBERCRfKggREcmXCkJERPIV06OYzGwHoaGyx6oesLOY4hQn5Soa5Soa5Sqa0pirhbvXL2ij\nmC6I42VmyYUZ6lXSlKtolKtolKtoynIu7WISEZF8qSBERCRfZb0gxgUd4GcoV9EoV9EoV9GU2Vxl\n+hiEiIj8vLL+DkJERH5GqS8IMzvbzFaZ2VozuzOf+zuY2bdmlmlmf4qiXJea2WIzW2Jm35hZ9yjJ\nNTica2H4wk39oyFXnu16m1m2mQ2JhlxmNsDM9oZfr4Vm9tdoyJUn20IzW2Zmn0dDLjO7Pc9rtdTM\ncsysThTkqmlm75nZovDrdWWkMxUhW20zmxL+uZxnZl2K7Yu7e6n9AMoD64BEIA5YBHT6yTYNgN7A\nQ8CfoijXSUDt8PI5wNwoyVWN/+6a7AasjIZcebabBXwIDImGXMAA4P2S+L4qYq5ahKbWbx6+3SAa\ncv1k+3MJXUcm8FzA3cAj4eX6wC4gLkqyPQrcG17uAMwsrq9f2t9B9AHWunuKux8B3gAG593A3dPd\n/TsgK8pyfePuu8M35xC6wl405Drg4e9EoCpFvGRspHKF3Qi8w3+vUhgtuUpaYXJdArzr7hsh9HMQ\nJbnyuhiYFCW5HKhuZkboj6RdhK5bEw3ZOhH6wwh3Xwm0NLOGxfHFS3tBNAE25bm9ObwuaEXNNQL4\nKKKJQgqVy8x+Z2YrgQ+Aq6Ihl5k1AX4HPFcCeQqdK+yk8Nv/j8ysc5TkagfUNrPZZjbfzK6IklwA\nmFk8cDahwo+GXP8COgJbgCXAze6eGyXZFgEXAJhZH6AFxfQHZWkviJhnZqcRKog7gs5ylLtPcfcO\nwPnAA0HnCXsSuKOEfmiL4ntCu3G6Ac8A/w44z1EVgF7AIOAs4C9m1i7YSD9yLvC1u+8KOkjYWcBC\noDHQA/iXmdUINtJ/PAzUMrOFhN5FLwCK5WqcQVxytCSlAc3y3G4aXhe0QuUys27AC8A57v5DtOQ6\nyt2/MLNEM6vn7pGcq6YwuZKAN0J7AKgH/MbMst09kr+QC8zl7vvyLH9oZqOj5PXaDPzg7geBg2b2\nBdAdWB1wrqOGUTK7l6Bwua4EHg7vXl1rZqmE9vfPCzpb+HvsSoDwLrBUIKVYvnqkD7IE+UGoAFOA\nVvz3AE/nn9n2PkruIHWBuYDmwFrgpGh6vYA2/Pcg9Qnhb1YLOtdPtn+FkjlIXZjXq1Ge16sPsDEa\nXi9Cu0tmhreNB5YCXYLOFd6uJqF9/FUj/X9YhNfrOeC+8HLD8Pd9vSjJVovwAXNgJPBqcX39Uv0O\nwt2zzewGYDqh0QAvufsyM7s2fP8YM2sEJAM1gFwzu4XQKIF9P/vEJZAL+CtQFxgd/qs42yM8MVch\nc10IXGFmWcBh4CIPf2cGnKvEFTLXEOAPZpZN6PUaFg2vl7uvMLOPgcVALvCCuy8NOld4098Bn3jo\n3U3EFTLXA8ArZrYEMEK7MyM+w2shs3UExpuZA8sI7ZIuFjqTWkRE8qWD1CIiki8VhIiI5EsFISIi\n+VJBiIhIvlQQIiKSLxWESDEzs5ZmFtEhoyIlQQUhIiL5UkGIRFB4KpIFZtY76CwiRVWqz6QWCZKZ\ntSc0PfPv3X1R0HlEikoFIRIZ9YGpwAXuvjzoMCLHQruYRCJjL6GJ+UrkkqwikaB3ECKRcYTQpHPT\nzeyAu08MOpBIUakgRCLE3Q+a2W+BT8MlMS3oTCJFodlcRUQkXzoGISIi+VJBiIhIvlQQIiKSLxWE\niIjkSwUhIiL5UkGIiEi+VBAiIpIvFYSIiOTr/wBuDoXNj0lQbAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ccf7c06978>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(np.arange(0.1, 1.0, 0.1), perplexities_valid_MLE_addk_n2)\n",
    "plt.xlabel(\"k\")\n",
    "plt.ylabel(\"Perplexity\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "best result is achieved with k=0.1. Lets run again the add k smoothing when k=0.1 for different n values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[399.46742337770513,\n",
       " 1682.285009191801,\n",
       " 3742.725919384281,\n",
       " 5029.421202042892,\n",
       " 5631.551100383158,\n",
       " 5933.2598196671]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "valid_tokens = [sent.split() for sent in words_valid_data.split(\"\\n\")] \n",
    "n = 7\n",
    "perplexities_valid_MLE_addk_smoothing = compute_perp_diff_models(n, valid_tokens, ngram_models, prob_MLE_addk_smoothing, 0.1)\n",
    "perplexities_valid_MLE_addk_smoothing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x2ccf7c324e0>"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY8AAAEKCAYAAADq59mMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8FfX1//HXIUCAkLAvYRMQBFkkQARc2rq0inWBWhfc\noFbFCm61/qx0+drN1q7WfW8FZRERBa1aFW2tVsAAYd83ISQkgJAEJMvN+f1xR00RTQK5meTm/Xw8\n7iNzP3fm3nMVeGfOfGbG3B0REZGqaBB2ASIiUvcoPEREpMoUHiIiUmUKDxERqTKFh4iIVJnCQ0RE\nqkzhISIiVabwEBGRKlN4iIhIlTUMu4BYadu2rXfv3j3sMkRE6pRFixbtcvd2Fa0Xt+HRvXt3MjIy\nwi5DRKROMbOtlVlPbSsREakyhYeIiFRZTMPDzFqa2SwzW2Nmq83sJDNrbWZvmtn64GercutPMrMN\nZrbWzM4uNz7UzJYHr91vZhbLukVE5KvFes/jPuB1d+8LDAJWA3cC89y9NzAveI6Z9QPGAP2BkcDD\nZpYQvM8jwHVA7+AxMsZ1i4jIV4hZeJhZC+DrwFMA7l7s7nuBUcDkYLXJwOhgeRQww92L3H0zsAEY\nZmapQIq7z/fozUemlNtGRERCEMs9jx5AHvB3M1tiZk+aWRLQwd2zg3VygA7BcmdgW7nttwdjnYPl\nQ8dFRCQksQyPhsAQ4BF3HwzsJ2hRfSrYk6i2Wxma2XgzyzCzjLy8vOp6WxEROUQsw2M7sN3dFwTP\nZxENk51BK4rgZ27wehbQtdz2XYKxrGD50PEvcPfH3T3d3dPbtavwHBcRkbjwSXGEpdv2MvPDbfzh\n9TU18pkxO0nQ3XPMbJuZ9XH3tcCZwKrgMQ64J/g5J9hkLjDNzP4CdCJ6YHyhu0fMLN/MRgALgLHA\nA7GqW0SktoqUOR/tOcDanHzW5BSwJruAtTsL2LJ7Px70cJo2SmD813vSslnjmNYS6zPMbwKmmllj\nYBNwNdG9nZlmdg2wFbgEwN1XmtlMouFSCkx090jwPhOAp4GmwGvBQ0Qkbu0uLGJtTkE0JHLyWZtT\nwLqdhXxSEv1n0Qy6t0miT4dkRqV1om/HZPp0TKFb62YkNIj92QzmXm2HHGqV9PR01+VJRKS2O1gS\nYUNuIauzowGxdmcBq7ML2FVY9Nk6bZIa06djMn07pgQhkcxxHZJp2jjhK975yJjZIndPr2i9uL22\nlYhIbVJW5mz/+BNWB3sRa3MKWJ2Tz5Zd+ykLfodPbNiA4zokc1qfdp+FRN+OKbRLTgy3+MNQeIiI\nVLOP9xezJqeAtTn5n+1JrNtZwIHiyGfrdGvdjL4dkzlvYCp9OqbQNzWZ7m2SaqTlVB0UHiIiR6io\nNNpy+vzYRDQwduZ/3nJq2awRfTsmc0l612BPItpySkqs2//81u3qRURqgHu05bQ2OHi9Jmg7bdq1\nn0jQc2qc0IBe7ZtzyrFtoyGRGj0+0T45kXi8HJ/CQ0SknH2flHwhJNbmFFBYVPrZOl1aNaVvx2TO\n7t/xs72J7m2TaJRQfy5UrvAQkXqpuLSMjXmft5w+PXcie9/Bz9ZJadKQvh1TuHBI5/9pOSU3aRRi\n5bWDwkNE4pq7s2Pfwf89sS6ngI15hZQGLadGCcax7ZozvEfr6MHrjsn0TU2mY0qTuGw5VQeFh4jE\nnR17P+HlpTuYtzqX1Tn5FBz8vOXUqUUT+qamcMbx7aMh0TGFHm2TaNyw/rScqoPCQ0Tiwt4Dxby6\nPIc5mVks3LIHdxjQOYULBnUK9iRSOK5DMi2aquVUHRQeIlJnfVIcYd6anby0ZAf/XpdLScTp2TaJ\nW888jlFpnejeNinsEuOWwkNE6pTSSBnvb9zNnMws/rkih/3FEdonJzLupO6MSuvMgM4pOk5RAxQe\nIlLruTtLtu1lbuYOXlm2g12FxSQ3ach5J3RiVFonhvdsU2fOzI4XCg8RqbU25BYyJzOLOZk7+GjP\nARo3bMCZfdszKq0zp/VpR5NG1X9hQKkchYeI1Co5+w7y8tIdvJSZxcod+TQwOPnYttx4Ri9GDuhI\nis6xqBUUHiISun0HSnhtRTZzMncwf/Nu3GFQlxb8/Lx+nH9CKu1TmoRdohxC4SEioThYEmHe6lzm\nZGbxr7V5FEfK6NE2iVvO7M0FgzrRs13zsEuUr6DwEJEaUxop44NNu3lpyQ7+uTKHwqJS2iUncuWI\nYxg9uBMDO7fQTKk6QuEhIjHl7izdvo85mVm8vDSbXYVFJCc25JwBHRk9uDMjNFOqTlJ4iEhMbMor\n5KXMHczNzGLL7gM0TmjAGX3bMyqtE6f3ba+ZUnWcwkNEqs3O/OhMqTmZO1ietQ8zOKlnGyac1ouz\nB3TUpUHiiMJDRI5K/sESXl+ew0uZWXywKTpTamDnFvzs3OM5f1AnOmimVFxSeIhIlR0sifDOmlzm\nZO7g7bW5FJeWcUybZtx0RnSmVK/2mikV7xQeIlIpkTJn/qbdvLQki9dX5FBQVErb5olcMbwbo9I6\nM6iLZkrVJwoPEflS7s7yrH3MydzBy0t3kFtQRPPEhpzdvyOjB3fipJ5taFiPbr0qn1N4iMgXbN61\nnzmZWczN3MGmXftpnNCA0/q0Y1RaZ848XjOlROEhIoHcgoO8vDSbuZlZLN0enSk1okcbxn+9J+cM\nSKVFM82Uks8pPETqsfyDJfxzRQ5zMnfw3427KHPo3ymFn377eM4blEpqi6Zhlyi1VEzDw8y2AAVA\nBCh193Qzaw08B3QHtgCXuPvHwfqTgGuC9W92938G40OBp4GmwKvALe7usaxdJF4VlUZ4Z00ec5dm\n8dbq6Eypbq2bMfH0XoxK60Sv9slhlyh1QE3seZzu7rvKPb8TmOfu95jZncHzH5tZP2AM0B/oBLxl\nZse5ewR4BLgOWEA0PEYCr9VA7SJxwd1ZsHkPLy7O4tUV2RQcLKVt88ZcPqwbo9I6kda1pWZKSZWE\n0bYaBZwWLE8G/gX8OBif4e5FwGYz2wAMC/ZeUtx9PoCZTQFGo/AQqZQDxaX87MUVzF6SRVLjBM4e\n0JFRaZ055VjNlJIjF+vwcKJ7EBHgMXd/HOjg7tnB6zlAh2C5MzC/3Lbbg7GSYPnQcRGpwPqdBUyY\nupgNeYXccmZvfvCNY2naWDOl5OjFOjxOdfcsM2sPvGlma8q/6O5uZtV27MLMxgPjAbp161ZdbytS\nJ72waDs/e2kFSYkJPHvNcE7p1TbskiSOxDQ83D0r+JlrZi8Cw4CdZpbq7tlmlgrkBqtnAV3Lbd4l\nGMsKlg8dP9znPQ48DpCenq4D6lIvfVIc4a65K5iZsZ3hPVrzwGWDdSc+qXYxa3iaWZKZJX+6DJwF\nrADmAuOC1cYBc4LlucAYM0s0sx5Ab2Bh0OLKN7MRFj2iN7bcNiJSzsa8QkY/9D4zM7Zz4+m9mHrt\ncAWHxEQs9zw6AC8GMzgaAtPc/XUz+xCYaWbXAFuBSwDcfaWZzQRWAaXAxGCmFcAEPp+q+xo6WC7y\nBXMys5g0ezmJDRvw9NUnclqf9mGXJHHM4vV0ifT0dM/IyAi7DJGYO1gS4VevrGLago9IP6YVD1w+\nWCf3yREzs0Xunl7RejrDXKQO27JrPxOmLmZVdj7Xf6Mnt5/Vh0aafis1QOEhUke9ujybO2YtI6GB\n8dS4dM48vkPFG4lUE4WHSB1TVBrht/9YzeQPtpLWtSUPXj6YLq2ahV2W1DMKD5E6ZNueA0yctphl\n2/fx/VN6cOc5fWncUG0qqXkKD5E64o2VOdz+/FIcePTKoYwc0DHskqQeU3iI1HIlkTJ+/9oannxv\nMwM7t+Chy4fQrY3aVBIuhYdILZa19xNunLaYJR/tZexJx/DTc48nsaGuTSXhU3iI1FJvr9nJbTOX\nUhpxHrx8MOed0CnskkQ+o/AQqWVKI2X86Y11PPrvjRyfmsLDVwyhR9uksMsS+R8KD5FaJGffQW6a\nvpgPt3zMZcO6cdf5/WjSSG0qqX0UHiK1xLvr8rj1uUwOlkT466VpjB6s29ZI7aXwEAlZpMz561vr\nePCdDfRu35yHrxhKr/bNwy5L5CspPERClFtwkFumZ/LBpt1cPLQLvxo1QHf6kzpB4SESkv9u2MXN\nMzIpLCrhjxedwMXpXSveSKSWUHiI1LBImfPQOxv461vr6NE2ianXDqdPx+SwyxKpEoWHSA3aVVjE\nD5/L5D/rd/GdwZ35zegBJCXqr6HUPfpTK1JDFmzazc0zlvDxgRLuuXAgl57YleBOmyJ1jsJDJMbK\nypxH393In99YR7fWzfj794bRr1NK2GWJHBWFh0gMfby/mNtmZvLO2jzOOyGV3104kOQmjcIuS+So\nKTxEYmTR1j3cOG0JuwuL+fWo/lw54hi1qSRuKDxEqpm78+R/NvP719fQqWVTXrjhZAZ2aRF2WSLV\nSuEhUo32HSjhR88v5a3VOzm7fwf+cNEgWjRVm0rij8JDpJpkbtvLxKmLyS04yP+d14+rT+muNpXE\nLYWHyFFyd57+7xZ+++pq2ic34fkfnExa15ZhlyUSUwoPkaOQf7CEH89axmsrcvjm8e3508WDaNms\ncdhlicScwkPkCK3I2seEqYvJ2vsJP/l2X677Wk+1qaTeUHiIVJG78+yCj/j1y6tondSY58aPIL17\n67DLEqlRDWL9AWaWYGZLzOyV4HlrM3vTzNYHP1uVW3eSmW0ws7Vmdna58aFmtjx47X7Tr3cSksKi\nUm6avoSfv7SCk45tw6u3fE3BIfVSzMMDuAVYXe75ncA8d+8NzAueY2b9gDFAf2Ak8LCZfXpjg0eA\n64DewWNkDdQt8j9WZ+dzwQPv8erybP7f2X34+/dOpHWSjm9I/RTT8DCzLsC5wJPlhkcBk4PlycDo\ncuMz3L3I3TcDG4BhZpYKpLj7fHd3YEq5bURizt2ZsfAjRj/0PoVFpUy7bgQTT+9FgwbaAZb6K9bH\nPP4K3AGUv1lBB3fPDpZzgA7Bcmdgfrn1tgdjJcHyoeNfYGbjgfEA3bp1O9raRThQXMrPXlzB7CVZ\nnNqrLfdemka75MSwyxIJXcz2PMzsPCDX3Rd92TrBnoRX12e6++Punu7u6e3atauut5V6at3OAi54\n8H1ezMzi1m/2ZvL3hyk4RAKx3PM4BbjAzL4NNAFSzOxZYKeZpbp7dtCSyg3WzwLK34ezSzCWFSwf\nOi4SM7MWbefnL60gKTGBZ68Zzim92oZdkkitErM9D3ef5O5d3L070QPhb7v7lcBcYFyw2jhgTrA8\nFxhjZolm1oPogfGFQYsr38xGBLOsxpbbRqRafVIc4Y5ZS7n9+aWc0KUFr978NQWHyGGEcZ7HPcBM\nM7sG2ApcAuDuK81sJrAKKAUmunsk2GYC8DTQFHgteIhUq415hUx4djFrdxZw4+m9uPWbvWmYUBMT\nEkXqHosedog/6enpnpGREXYZUkfMycxi0uzlJDZswL2XpnFan/ZhlyQSCjNb5O7pFa2nM8ylXjtY\nEuFXr6xi2oKPSD+mFQ9cPpjUFk3DLkuk1qtUeJjZn4G/ufvKGNcjUmO27NrPhKmLWZWdz/Xf6Mnt\nZ/WhkdpUIpVS2T2P1cDjZtYQ+Dsw3d33xa4skdh6dXk2d8xaRkID46lx6Zx5fIeKNxKRz1Tq1yx3\nf9LdTyE606k7sMzMppnZ6bEsTiQWXluezYSpi+nVvjn/uPlUBYfIEaj0Pnpwnam+wWMXsBS4zcxm\nxKg2kWq3Imsft81cyuBuLZkxfgRdWjULuySROqmyxzzuBc4D3gZ+6+4Lg5d+b2ZrY1WcSHXKKyhi\n/JQMWjZrxGNXDaVJo4SKNxKRw6rsMY9lwM/cff9hXhtWjfWIxERRaYTrn8lgz4FiZv3gZNonNwm7\nJJE6rbJtqysPDQ4zmwegA+dS27k7k2YvZ/FHe/nzxWkM6Nwi7JJE6ryv3PMwsyZAM6BtcNOmT69B\nncKXXNlWpLZ54j+bmL04enHDc09IDbsckbhQUdvqeuBWoBOwuNx4PvBgrIoSqS5vr9nJ715bw7kD\nU7n5jN5hlyMSN74yPNz9PuA+M7vJ3R+ooZpEqsX6nQXcPD2Tfqkp/OniQbp5k0g1qqhtdYa7vw1k\nmdmFh77u7rNjVpnIUfh4fzHXTM6gSaMEnhibTtPGmlklUp0qalt9g+j03PMP85oDCg+pdUoiZdww\ndRE5+QeZMX4EnVrqWlUi1a2ittVdwc+ra6YckaP3i7krmb9pD3+5ZBBDurUKuxyRuFSpqbpm9oyZ\ntSj3/JhPp+qK1CbPfLCFqQs+4vpv9OTCIV0qXF9Ejkxlz/N4D1hgZt82s+uAN4G/xq4skap7f8Mu\nfvHyKs7s2547zu4bdjkica1SZ5i7+2NmthJ4h+h1rQa7e05MKxOpgs3B5dWPbZfEX8ekkaCZVSIx\nVdm21VXA34heVfdp4FUzGxTDukQqLf9gCddO/pAGBk+OPZHkJo3CLkkk7lX22lbfBU5191xgupm9\nCEwG0mJWmUglRMqcm6YtYevuAzx77XC6tdFVckVqQmXbVqMPeb7QzHRBRAnd715dzb/X5fHb7wxk\nRM82YZcjUm9Utm11nJnNM7MVwfMTgDtiWplIBWZ+uI0n39vM907uzuXDu4Vdjki9UtnZVk8Ak4AS\nAHdfBoyJVVEiFcnYsoefvrScU3u15WfnHh92OSL1TmXDo1m5G0B9qrS6ixGpjO0fH+D6ZxbRpVUz\nHrp8CA0TKn1DTBGpJpX9W7fLzI4lekkSzOwiIDtmVYl8if1FpVw7OYPiSBlPjE2nRTPNrBIJQ2Vn\nW00EHgf6mlkWsBm4MmZViRxGWZnzw+cyWbezgL9fPYxe7ZuHXZJIvVXZ2VabgG+aWRLQwN0LYluW\nyBfd+9Y63li1k/87rx/fOK5d2OWI1GsVXZL9ti8ZB8Dd//IV2zYB3gUSg8+Z5e53mVlr4DmgO7AF\nuMTdPw62mQRcA0SAm939n8H4UKInJzYFXgVucXev5HeUODAnM4sH3t7ApeldufqU7mGXI1LvVXTM\nI7mCx1cpAs5w90FETyYcaWYjgDuBee7eG5gXPMfM+hGdwdUfGAk8bGaf3oThEeA6oHfwGFmF7yh1\n3NJte7lj1jKGdW/Nr0cP+OyXFxEJT0WXZP/lkb5xsGdQGDxtFDwcGAWcFoxPBv4F/DgYn+HuRcBm\nM9sADDOzLUCKu88HMLMpwGjgtSOtTeqOnfkHuW5KBm2bJ/LIlUNo3FAzq0Rqg8qeJNjTzF42szwz\nyzWzOWbWsxLbJZhZJpALvOnuC4AO7v7pTK0coEOw3BnYVm7z7cFY52D50HGJcwdLIoyfksH+olKe\n+l46bZonhl2SiAQq+2vcNGAmkAp0Ap4Hple0kbtH3D0N6EJ0L2LAIa87wfTf6mBm480sw8wy8vLy\nquttJQTuzh2zlrEsax/3XppG344pYZckIuVU5STBZ9y9NHg8CzSp7Ie4+16il3MfCew0s1SA4Gdu\nsFoW0LXcZl2Csaxg+dDxw33O4+6e7u7p7dppNk5d9vC/NjJ36Q5uP6sPZ/XvGHY5InKIyobHa2Z2\np5l1D+4ieAfRy7K3DmZPfYGZtTOzlsFyU+BbwBpgLjAuWG0cMCdYnguMMbNEM+tB9MD4wqDFlW9m\nIyx6pHRsuW0kDr2xMoc//nMto9I6MeG0Y8MuR0QOo7InCV4S/Lz+kPExRNtOhzv+kQpMDmZMNQBm\nuvsrZvYBMNPMrgG2fvre7r7SzGYCq4he+mSiu0eC95rA51N1X0MHy+PW6ux8bn0uk0FdW/L7756g\nmVUitZRVdLqEmTUATnL392umpOqRnp7uGRkZYZchVbCrsIhRD75PaVkZL994Ku1TKt0ZFZFqYmaL\n3D29ovUqbFu5exnwYLVUJfIlikvLuOHZRewqLOKJsekKDpFarrLHPOaZ2XdNPQSJAXfnZy8t58Mt\nH/OniwdxQpeWYZckIhWobHhcT3R6brGZ5ZtZgZnlx7AuqUf+9v4WZmZs5+YzenH+oE5hlyMilVDZ\nCyNWdCkSkSPyr7W53P2PVZzdvwO3fvO4sMsRkUqq7BnmZmZXmtnPg+dddQ9zOVobcgu5adoS+nRM\n4S+XpNGggbqiInVFZdtWDwMnAZcHzwuBh2JSkdQLew8Uc+3kD0ls1IAnxg4lKbGys8ZFpDao7N/Y\n4e4+xMyWALj7x2bWOIZ1SRwriZQxcdpiduw9yPTxw+nSqlnYJYlIFVU2PEqCk/0+vQ1tO6AsZlVJ\nXPvNK6t4f8Nu/njRCQw95rAXKBCRWq6ybav7gReB9mZ2N/Ae8NuYVSVxa+qCrUz+YCvXfa0HF6d3\nrXgDEamVKjvbaqqZLQLOBAwY7e6rY1qZxJ0PNu7mrjkrOa1PO+485/iwyxGRo1DRbWibAD8AegHL\ngcfcvbQmCpP48tHuA9wwdRHd2yZx/2WDSdDMKpE6raK21WQgnWhwnAP8KeYVSdwpOFjCNZM/xB2e\nHJtOSpNGYZckIkeporZVP3cfCGBmTwELY1+SxJNImXPrjEw27drPM98fRve2SWGXJCLVoKI9j5JP\nF9SukiPxh3+uYd6aXH5xQX9O7tU27HJEpJpUtOcxqNw1rAxoGjw3oneR1b1B5Uu9sGg7j/17E1eN\nOIarRhwTdjkiUo2+MjzcPaGmCpH4smjrx0yavZyTerbh/87vF3Y5IlLNKnueh0il7dj7Cdc/s4jU\nlk14+IohNErQHzOReKMLCkm1OlBcynVTMigqiTBj/HBaJekqNiLxSOEh1aaszPnRzKWszs7nqe+d\nSK/2upK/SLxSP0GqzX3z1vPaihwmnXM8p/dpH3Y5IhJDCg+pFv9Yls1989Zz0dAuXPu1HmGXIyIx\npvCQo7Yiax8/ej6Toce04u7vDEC3uheJfwoPOSq5+Qe5bkoGbZISefTKoSQ21OxukfpAB8zliB0s\niTD+mUXsPVDCrBtOol1yYtgliUgNUXjIEXF3fjJ7OZnb9vLolUPo36lF2CWJSA1S20qOyGPvbmL2\nkix+9K3jGDkgNexyRKSGKTykyt5atZPfv76G8wd14sYzeoVdjoiEIGbhYWZdzewdM1tlZivN7JZg\nvLWZvWlm64OfrcptM8nMNpjZWjM7u9z4UDNbHrx2v2k6T2jW5hRwy4wlDOjUgj989wTNrBKpp2K5\n51EK/Mjd+wEjgIlm1g+4E5jn7r2BecFzgtfGAP2BkcDDZvbp1J1HgOuA3sFjZAzrli+xZ38x1075\nkKTEhjwxNp2mjTWzSqS+ill4uHu2uy8OlguA1UBnYBTROxQS/BwdLI8CZrh7kbtvBjYAw8wsFUhx\n9/nu7sCUcttIDSkuLeOGZxexM7+Ix8em07FFk7BLEpEQ1cgxDzPrDgwGFgAd3D07eCkH6BAsdwa2\nldtsezDWOVg+dFxqiLtz19yVLNi8hz9edAJpXVuGXZKIhCzm4WFmzYEXgFvdPb/8a8GehFfjZ403\nswwzy8jLy6uut633pnywlekLP2LCaccyKk25LSIxDg8za0Q0OKa6++xgeGfQiiL4mRuMZwFdy23e\nJRjLCpYPHf8Cd3/c3dPdPb1du3bV90Xqsf+sz+NXr6ziW/06cPtZfcIuR0RqiVjOtjLgKWC1u/+l\n3EtzgXHB8jhgTrnxMWaWaGY9iB4YXxi0uPLNbETwnmPLbSMxtCmvkIlTF9O7fXPuvTSNBg00s0pE\nomJ5hvkpwFXAcjPLDMZ+AtwDzDSza4CtwCUA7r7SzGYCq4jO1Jro7pFguwnA00BT4LXgITG075MS\nrp2cQcOEBjwxNp3miboYgYh8Lmb/Irj7e8CX/ap65pdsczdw92HGM4AB1VedfJXSSBk3TV/Cto8P\nMPXaEXRt3SzskkSkltGvk/IFv311De+uy+P33x3IsB6twy5HRGohXZ5E/seMhR/xt/c38/1TenDp\nid3CLkdEaimFh3xm4eY9/HzOCr5+XDt+8u2+YZcjIrWYwkMA2LbnAD94dhFdWzXjgcsG0zBBfzRE\n5MvpXwihsKiU66ZkUBop48lx6bRo2ijskkSkltMB83ruQHEpN09fwvrcQp6++kR6tmsedkkiUgco\nPOqxdTsLmDB1MRvzCvnN6AF8rbfOyheRylF41FPPZ2zj53NW0DyxEc9eM5xTerUNuyQRqUMUHvXM\nJ8URfj5nBbMWbeeknm2477I02ifr8uoiUjUKj3pkQ260TbU+t5Cbz+zNLWf2JkHXqxKRI6DwqCdm\nL97OT19cQbPGCUz5/jAd3xCRo6LwiHMHSyLcNWclz2VsY1iP1jxw2WA6pKhNJSJHR+ERxzYGl1Rf\nk1PAjaf34tZv9tbJfyJSLRQecWpOZhaTZi+nSaMEJn9/GN84Tm0qEak+Co84c7Akwi9fXsX0hR9x\nYvdW3H/ZYFJbNA27LBGJMwqPOLJ5134mTF3M6ux8bjjtWH70rePUphKRmFB4xImXl+7gzheW0ahh\nA/7+vRM5vW/7sEsSkTim8KjjDpZE+M0/VvHs/I8Y0q0lD14+hE4t1aYSkdhSeNRhW3btZ+K0xazc\nkc/1X+/J7Wf3oZHaVCJSAxQeddQ/lmXz4xeWkdDAeHJsOt/s1yHskkSkHlF41DFFpRHu/sdqpnyw\nlcFBm6qz2lQiUsMUHnXIR7sPMHHaYpZn7ePaU3twx8i+NG6oNpWI1DyFRx3x+ops/t+sZRjw+FVD\nOat/x7BLEpF6TOFRyxWXlvHbV1fz9H+3MKhrSx68bDBdWzcLuywRqecUHrXYtj0HuHHaYpZu38fV\np3Rn0jnHq00lIrWCwqOWemNlDrc/vxQHHr1yKCMHqE0lIrWHwqOWKS4t4/evr+Gp9zYzsHMLHrp8\nCN3aqE0lIrVLzHogZvY3M8s1sxXlxlqb2Ztmtj742arca5PMbIOZrTWzs8uNDzWz5cFr95tZ3N76\nbvvHB7jksQ946r3NjDvpGGbdcJKCQ0RqpVg20J8GRh4ydicwz917A/OC55hZP2AM0D/Y5mEzSwi2\neQS4DugdPA59z7gwb/VOzr3/PTbmFvLwFUP45agBJDZMqHhDEZEQxCw83P1dYM8hw6OAycHyZGB0\nufEZ7l4oZ5N9AAAHdElEQVTk7puBDcAwM0sFUtx9vrs7MKXcNnGhJBKdTXXN5Ay6tGrKyzedyrcH\npoZdlojIV6rpYx4d3D07WM4BPr2mRmdgfrn1tgdjJcHyoeOHZWbjgfEA3bp1q6aSY2fH3k+4afoS\nFm39mKtGHMNPzz2eJo20tyEitV9oB8zd3c3Mq/k9HwceB0hPT6/W965u76zJ5baZmZREnAcuG8z5\ngzqFXZKISKXVdHjsNLNUd88OWlK5wXgW0LXcel2Csaxg+dDxOqskUsaf31jHo//eSL/UFB66Ygg9\n2iaFXZaISJXU9Blnc4FxwfI4YE658TFmlmhmPYgeGF8YtLjyzWxEMMtqbLlt6pzsfZ9w2ePzefTf\nG7l8eDdmTzhZwSEidVLM9jzMbDpwGtDWzLYDdwH3ADPN7BpgK3AJgLuvNLOZwCqgFJjo7pHgrSYQ\nnbnVFHgteNQ5/1qby20zl1JUEuG+MWmMSvvSQzciIrWeRScxxZ/09HTPyMgIuwxKI2Xc+9Y6Hnpn\nI307JvPQFUM4tl3zsMsSETksM1vk7ukVraczzGNoZ/5Bbpq+hIWb9zDmxK784oL+mk0lInFB4REj\n/1mfx60zMjlQHOHeSwfxncFdKt5IRKSOUHhUs0iZc99b63jgnQ30bt+c564YQq/2yWGXJSJSrRQe\n1Sg3/yA3z1jC/E17uHhoF341agBNG6tNJSLxR+FRTd7fsItbZmRSWFTCny4exEVD1aYSkfil8DhK\nkTLngbfXc9+89RzbrjnTrhvOcR3UphKR+KbwOAp5BUXc+twS3t+wmwuHdOY3owfQrLH+k4pI/NO/\ndEfog427uXnGEgoOlvCHi07gkvSuFW8kIhInFB5VVFbmPPTOBu59ax092ibx7DXD6dNRbSoRqV8U\nHlWwq7CIHz6XyX/W72J0Wifu/s5AkhL1n1BE6h/9y1dJCzZF21R7D5Rwz4UDufTErsTxHXFFRL6S\nwqMCZWXOI//eyJ/fWEv3Nkk8ffUwjk9NCbssEZFQKTy+wu7CIn44cynvrsvj/EGd+N2FA2muNpWI\niMLjy3y4ZQ83TVvCngPF3P2dAVw+rJvaVCIiAYXHIcrKnMfe3cSf3lhL11ZNmX3DyQzo3CLsskRE\nahWFRzklkTKuf2YRb6/J5dyBqdzz3YEkN2kUdlkiIrWOwqOcRgkN6NE2iV+P6s+VI45Rm0pE5Eso\nPA7x8/P6hV2CiEit1yDsAkREpO5ReIiISJUpPEREpMoUHiIiUmUKDxERqTKFh4iIVJnCQ0REqkzh\nISIiVWbuHnYNMWFmecDWI9y8LbCrGsupC/Sd64f69p3r2/eFo//Ox7h7u4pWitvwOBpmluHu6WHX\nUZP0neuH+vad69v3hZr7zmpbiYhIlSk8RESkyhQeh/d42AWEQN+5fqhv37m+fV+ooe+sYx4iIlJl\n2vMQEZEqU3gEzKyrmb1jZqvMbKWZ3RJ2TbFmZk3MbKGZLQ2+8y/DrqmmmFmCmS0xs1fCrqUmmNkW\nM1tuZplmlhF2PTXBzFqa2SwzW2Nmq83spLBriiUz6xP8//30kW9mt8bs89S2ijKzVCDV3RebWTKw\nCBjt7qtCLi1mLHqrxCR3LzSzRsB7wC3uPj/k0mLOzG4D0oEUdz8v7Hpizcy2AOnuXm/OeTCzycB/\n3P1JM2sMNHP3vWHXVRPMLAHIAoa7+5Ge7/aVtOcRcPdsd18cLBcAq4HO4VYVWx5VGDxtFDzi/rcJ\nM+sCnAs8GXYtEhtm1gL4OvAUgLsX15fgCJwJbIxVcIDC47DMrDswGFgQbiWxF7RvMoFc4E13j/vv\nDPwVuAMoC7uQGuTAW2a2yMzGh11MDegB5AF/D9qTT5pZUthF1aAxwPRYfoDC4xBm1hx4AbjV3fPD\nrifW3D3i7mlAF2CYmQ0Iu6ZYMrPzgFx3XxR2LTXs1OD/8znARDP7etgFxVhDYAjwiLsPBvYDd4Zb\nUs0IWnQXAM/H8nMUHuUEff8XgKnuPjvsempSsEv/DjAy7Fpi7BTgguAYwAzgDDN7NtySYs/ds4Kf\nucCLwLBwK4q57cD2cnvSs4iGSX1wDrDY3XfG8kMUHoHg4PFTwGp3/0vY9dQEM2tnZi2D5abAt4A1\n4VYVW+4+yd27uHt3orv2b7v7lSGXFVNmlhRMAiFo3ZwFrAi3qthy9xxgm5n1CYbOBOJ28sshLiPG\nLSuI7tpJ1CnAVcDy4BgAwE/c/dUQa4q1VGByMDOjATDT3evF1NV6pgPwYvT3IxoC09z99XBLqhE3\nAVODNs4m4OqQ64m54JeDbwHXx/yzNFVXRESqSm0rERGpMoWHiIhUmcJDRESqTOEhIiJVpvAQEZEq\nU3iIiEiVKTxERKTKFB4iNcTMugf3lXgiuH/KG8GZ/SJ1jsJDpGb1Bh5y9/7AXuC7IdcjckQUHiI1\na7O7f3r5m0VA9xBrETliCg+RmlVUbjmCri8ndZTCQ0REqkzhISIiVaar6oqISJVpz0NERKpM4SEi\nIlWm8BARkSpTeIiISJUpPEREpMoUHiIiUmUKDxERqTKFh4iIVNn/BxGo0VmzJceuAAAAAElFTkSu\nQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ccf7c02208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(range(2,8), perplexities_valid_MLE_addk_smoothing)\n",
    "plt.xlabel(\"n\")\n",
    "plt.ylabel(\"Perplexity\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Still, the best perplexity is for n=2.\n",
    "I think it happens because of the smoothing method we used, the add k smoothing underestimates P(word|history) when n grows. As n grows, there are more words that are not seen in the context of some long history. The more appropriate smoothing would be a backoff smoothing such as Katz backoff.\n",
    "\n",
    "Now it is time to test our model on the test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "test_tokens = [sent.split() for sent in words_test_data.split(\"\\n\")] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "399.46742337770513"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n=2\n",
    "k=0.1\n",
    "perplexity(valid_tokens, n, ngram_models, prob_MLE_addk_smoothing, k)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The worst case perplexity is achieved for the uniform distribution, where there is 1/|V| chance for each word in the vocabulary V."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def uniform_prob(history, word, n, models, alpha=1):\n",
    "    model = models[1]\n",
    "    \n",
    "    return 1/len(model)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10000.000000135005"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 2\n",
    "perplexity(test_tokens, n, ngram_models, uniform_prob)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, this number coincides with the number of words in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The paper [\"Breaking the softmax bottleneck: A high-rank RNN language model\"](https://arxiv.org/pdf/1711.03953.pdf) by Z. Yang et. al describes a new method to improve an RNN model. The authors compare the results to state of the art results in the last 5 years."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"ptb_ppl_results.png\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The perplexity obtained using the language model I created is better than the worst case, but ~8 time worse than the best models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In conclusion - Lidstone smoothing has poor results. Maybe because Lidstone smoothing moves too much probability mass to unkown events."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3.2 Generating Text from a Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Another way to evaluate a language model is to use the model in a generative manner."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generating a word given the word 'burned' using Maximal Likelihood probability:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FreqDist({'a': 1,\n",
       "          'by': 1,\n",
       "          'if': 1,\n",
       "          'in': 1,\n",
       "          'investors': 1,\n",
       "          'on': 2,\n",
       "          'rand': 1,\n",
       "          'repeatedly': 1,\n",
       "          'when': 1,\n",
       "          '~': 1})"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngram_models[2]['burned']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'~'"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "MLEProbDist(ngram_models[2]['burned']).generate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In word generation, we can try to use 'stupid backoff' instead of Lidstone smoothing. If an n-gram is not found, we will try to look for an (n-1)-gram, (n-2)-gram and so on. Eventually we will reach the 1-gram model, and the probability couldn't be 0 as all words in the train data appear in the validation data.\n",
    "\n",
    "The probelm with this method is that the sum of probabilities will not be equal to 1, so entropy can't be measured. Anyway, lets see if it works in practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prob_stupid_backoff(history, word, n, models, alpha=1):\n",
    "    \"\"\"\n",
    "    If an n-gram is not found, we will try to look for an (n-1)-gram, (n-2)-gram and so on.\n",
    "    alpha can be used for normalization\n",
    "    \"\"\"\n",
    "    model = models[n]\n",
    "    \n",
    "    if n == 1:\n",
    "        return model.freq(word) #1-gram model is just the frequency of word\n",
    "    else:\n",
    "        history_str = \" \".join(history)\n",
    "        if history_str in model and word in model[history_str]:\n",
    "            return model[history_str].freq(word)\n",
    "        else:#back-off to shorter history\n",
    "            return prob_stupid_backoff(history[1:], word, n-1, models)*alpha"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_word(models, history, n):\n",
    "    '''\n",
    "    generates a word given history using MLE probability \n",
    "    if history is not found, back-off for shorter history\n",
    "    '''\n",
    "    history = history[-n+1:]\n",
    "    history_str = \" \".join(history)\n",
    "    \n",
    "    model = models[n]\n",
    "   \n",
    "    if n == 1:\n",
    "        return MLEProbDist(model[history_str]).generate()\n",
    "    else:\n",
    "        if model[history_str].N() > 0:\n",
    "            mle = MLEProbDist(model[history_str])\n",
    "            return mle.generate()\n",
    "        else:#back-off to shorter history\n",
    "            return generate_word(models, history[1:], n-1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The generation of words could stop after a perdefined number of words were created.\n",
    "\n",
    "I think that is is more wise to stop when as end of sentence token is generated"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_sent(models, seed, n):\n",
    "    \"\"\"\n",
    "    generates a sentence given a seed using ngram model models[n]\n",
    "    \"\"\"\n",
    "    history = seed\n",
    "    out = history\n",
    "    word = generate_word(models, history, n)\n",
    "    while word != \"~\": #keep generating words until an \"end of sentence\" mark\n",
    "        out.append(word)\n",
    "        word = generate_word(models, history, n)\n",
    "        history = history[-n+1:] + [word]\n",
    "    return \" \".join(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate(models, seed, n, num_of_sents):\n",
    "    \"\"\"\n",
    "    generate num_of_sents sentences using generate_sent\n",
    "    \"\"\"\n",
    "    text = \"\"\n",
    "    for i in range(num_of_sents):\n",
    "        sent = generate_sent(models, seed, n)\n",
    "        sent = sent.replace(\"~ \", \"\")\n",
    "        sent = sent[0].upper() + sent[1:] + \". \"\n",
    "        text = text + sent\n",
    "        seed = [\"~\" for x in range(n-1)]\n",
    "    return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see how good our model is in generating sentences:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"I do n't provide structural damage at the N N the upper manhattan community <unk> frame that 's intent is seeking on thursday 's gains after all of the clean slate of the N N N N billion. The editor of $ N despite the network of <unk> the <unk> materials may be made a provision for example the same risk factors and <unk> or N higher on new <unk> added gross national over-the-counter trading is the markets more complex in the gain of <unk> subsidiaries. \""
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n=2\n",
    "generate(ngram_models, [\"i\"], n, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Daniel 's annual sales so high above to N. <unk> formerly owners contend that level of obtaining or N N pieces of seats for individuals still stands people with banks to N in the world series bonds just get tougher for four years. \""
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "generate(ngram_models, [\"~\"], n, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It doesn't makes much sense. What if we increase n?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'I am today. N develop rental agreements with reuters <unk> hill is for growth not just for the ninth inning of game four when <unk> earthquake damage may top $ N million respectively. '"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n=3\n",
    "generate(ngram_models, [\"i\", \"am\"], n, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"He also noted an absence of <unk> stock-index futures or options. In fiscal N with the chinese government of spain 's nato commitment and in exchange for an <unk> issue of emissions that lead to further advance. \""
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n=3\n",
    "generate(ngram_models, [\"~\" for x in range(n-1)], n, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Not bad, now for n=4:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"He said chrysler fully expects to have them sent home instead. Nevertheless he noted no one will check to determine what the jurors ' attitudes are likely to mirror those mr. bush enacted shortly after the massacre. \""
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n=4\n",
    "generate(ngram_models, [\"~\" for x in range(n-1)], n, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The company also changed its dividend policy under which holders had received an annual N <unk> payment by declaring a <unk> dividend to be paid quarterly on <unk> shares. According to the report completed last month if china 's trade gap continues to widen at the pace seen in the first seven months and the <unk> deficit widens to $ N billion and is set up to handle precisely the kind of emergency the state faces said tom <unk> the governor 's deputy press secretary. \""
      ]
     },
     "execution_count": 90,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n=5\n",
    "generate(ngram_models, [\"~\" for x in range(n-1)], n, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"He believed colon cancer might also arise from multiple hits on cancer <unk> genes because it often seems to develop in stages. Democracy is making a return with a <unk> to latin america 's most <unk> and deeply <unk> country. \""
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n=6\n",
    "generate(ngram_models, [\"~\" for x in range(n-1)], n, 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Westinghouse 's own role as a supplier also is changing. Desperate he has called in the <unk> to help fight the rebels of the <unk> national movement in the north which is only one of several groups picking away at the regime in the capital of <unk>. \""
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n=7\n",
    "generate(ngram_models, [\"~\" for x in range(n-1)], n, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sentences make more sense as n grows. But as n grows, there are less ngrams completions to choose from. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FreqDist({'only': 1})"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ngram_models[7][\"movement in the north which is\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The 6-gram \"movement in the north which is\" can only be completed using the word 'only'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It can be seen that there are quit a few unknowm words. The model could be changed to not generate unkown words.\n",
    "\n",
    "Currently, each sentence is independent of the other sentences. Another thing that can be changed is to create a dependency between the sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### 1.4 RNN language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [The Unreasonable Effectiveness of Recurrent Neural Networks](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) summary\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recurrent Neural Networks (RNNs) are useful because they allow us to operate over sequences of vectors (input, output or both). This article demonstrates the power of RNNs for creation of language models. \n",
    "\n",
    "The author trains RNN character-level language models, so that it allows him to generate new text one character at a time. Each character is encoded into a vector using 1-of-k encoding, standard Softmax classifier is used on every output vector simultaneously and the RNN is trained with mini-batch Stochastic Gradient Descent. At test time, he feeds a character into the RNN and get a distribution over what characters are likely to come next. \n",
    "\n",
    "Using this method, different text is generated using different datasets for training, such as Shakspear, Wikipedia, Baby names etc. The results are pretty impressive and amusing, and demonstrate the model is quite good at learning complex syntactic structures. \n",
    "\n",
    "At last, he tries to understand how does the model work. He looks at how the sampled text evolves while the model trains. At first the generated text is nonsense and gradually the model 'learns' the language. In addition, he looks at the predicted distributions over characters to find nuerons that are sensitive to a specific language feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### [The unreasonable effectiveness of Character-level Language Models](http://nbviewer.jupyter.org/gist/yoavg/d76121dfde2618422139) summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This article presents an unsmoothed maximum-liklihood character level language model and shows it effectiveness at generating convincing natural language outputs. \n",
    "\n",
    "The author builds an n-gram model using the Maximum Likelihood estimates. This models gives the probabilty of a character c given some history h of n-1 characters (P(c|h)). Using a simple python code, he creates a language model and trains it on the text of Shakespears. To generate new text, a letter a time, he takes the history, looks at the last n-1 characters, and then samples a random letter based on the corresponding distribution. \n",
    "\n",
    "The author generates text based on different language-model orders (different value of n). For order 2, the generated text does not make sense. With an order of 4, the results are reasonable. Increasing the order to 7 (~word and a half of history) or 10 (~two short words of history) gets a quite passable Shakepearan text, just as good as the RNN model described in the previous article. \n",
    "\n",
    "Nevertheless, this simple model is not as good as the RNN for linux-kernel code generation, as it cannot model its structure only by looking at the last n-1 characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Character language model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this part we will create a character based language model, and train it on a recipes dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we read all the recipes files, and split them to different recipes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1290,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import os\n",
    "recipes = []\n",
    "for filename in os.listdir('recipes'):\n",
    "    input_text_file = open('recipes/'+filename, \"r\", encoding=\"utf8\", errors='ignore')\n",
    "    text = input_text_file.read()\n",
    "    text = text.split(\"MMMMM----- Recipe via Meal-Master (tm) v8.05\")\n",
    "    recipes = recipes + text[1:]\n",
    "    \n",
    "recipes = [recipe.replace(\"MMMMM\", \"\") for recipe in recipes]    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many recipes do we have?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1291,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "32963"
      ]
     },
     "execution_count": 1291,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(recipes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's tokenize:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1294,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "recipes_tokenized = [word_tokenize(recipe) for recipe in recipes]\n",
    "tokens = [token.lower() for recipe in recipes_tokenized for token in recipe]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1295,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['title',\n",
       " ':',\n",
       " '``',\n",
       " 'be',\n",
       " 'mine',\n",
       " \"''\",\n",
       " 'lollipops',\n",
       " 'categories',\n",
       " ':',\n",
       " 'candies']"
      ]
     },
     "execution_count": 1295,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I decided to keep all punctuations, they may be useful"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Gathering some statistics on the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Number of characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1296,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26551407"
      ]
     },
     "execution_count": 1296,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum([len(token) for token in tokens])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1297,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vocabulary = Counter(tokens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1298,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('.', 358720),\n",
       " (',', 318203),\n",
       " ('--', 197642),\n",
       " ('and', 175811),\n",
       " ('the', 158348),\n",
       " (':', 145566),\n",
       " ('1', 140574),\n",
       " ('in', 100098),\n",
       " ('to', 93754),\n",
       " ('c', 90743)]"
      ]
     },
     "execution_count": 1298,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocabulary.most_common(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vocabulary size:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1299,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "70443"
      ]
     },
     "execution_count": 1299,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocabulary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1300,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "words_per_recipe = Counter([len(recipe) for recipe in recipes_tokenized])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distribution of the size of recipes in words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1301,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x14803dadc50>"
      ]
     },
     "execution_count": 1301,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAHORJREFUeJzt3XuUHWWZ7/HvT8LNgBKkZYVrAivqwjPHgH24iCCICogS\n5AgDBxlAJDCLYWB0lidcDjDjYga56XBQJEgGmEEuchdQRAa5zBkCHQSSECLhJokhaWSUiGMw8Jw/\n6t1QNNXd1d27du29+/dZa69d+62qXc+7u3s/Xe/71luKCMzMzAZ6V90BmJlZe3KCMDOzQk4QZmZW\nyAnCzMwKOUGYmVkhJwgzMytUWYKQtKWkeyQ9IWmhpBNT+caS7pL0VHqelNvnZElLJC2WtHdVsZmZ\n2fBU1XUQkiYDkyPiEUkbAvOAA4AjgZcj4mxJs4BJEfG/JW0HXA3sCGwG/Az4QES8XkmAZmY2pMrO\nICJieUQ8kpZXAYuAzYEZwBVpsyvIkgap/JqIWB0RzwJLyJKFmZnVYEIrDiJpCrA9MBfYNCKWp1Uv\nApum5c2BB3O7LU1lg9pkk01iypQpzQzVzKzrzZs376WI6Bluu8oThKQNgBuAkyLiFUlvrouIkDSi\nNi5JM4GZAFtttRV9fX3NDNfMrOtJer7MdpWOYpK0NllyuCoibkzFK1L/RKOfYmUqXwZsmdt9i1T2\nNhExOyJ6I6K3p2fYBGhmZqNU5SgmAZcBiyLigtyqW4Ej0vIRwC258kMkrStpKjANeKiq+MzMbGhV\nNjHtChwOzJf0aCo7BTgbuE7S0cDzwMEAEbFQ0nXAE8Aa4HiPYDIzq09lCSIiHgA0yOq9BtnnLOCs\nqmIyM7PyfCW1mZkVcoIwM7NCThBmZlbICcLMzAo5QZiZWSEnCDMzK+QEYWZmhZwgzMyskBOEmZkV\ncoIwM7NCThBmZlbICcLMzAo5QZiZWSEnCDMzK+QEYWZmhZwgzMyskBOEmZkVcoIwM7NClSUISXMk\nrZS0IFd2raRH0+O5xr2qJU2R9F+5dd+rKi4zMyunsntSA5cDFwFXNgoi4s8by5LOB36X2/7piJhe\nYTxmZjYClSWIiLhP0pSidZIEHAx8sqrjm5nZ2NTVB7EbsCIinsqVTU3NS/dK2q2muMzMLKkrQRwK\nXJ17vRzYKjUxfRX4gaT3FO0oaaakPkl9/f39LQh19KbMur3uEMzMRq3lCULSBOBA4NpGWUSsjojf\npOV5wNPAB4r2j4jZEdEbEb09PT2tCNnMbFyq4wziU8CTEbG0USCpR9JaaXkbYBrwTA2xmZlZUuUw\n16uB/wA+KGmppKPTqkN4e/MSwO7A42nY6/XAcRHxclWxmZnZ8KocxXToIOVHFpTdANxQVSxmZjZy\nvpLazMwKOUGYmVkhJwgzMyvkBGFmZoWcIDqIL7wzs1ZygmgzTgJm1i6cIMzMrJAThJmZFXKCaKGR\nNB+5qcnM6uYEYWZmhZwgzMyskBOEmZkVcoJoI+53MLN24gTRxpwwzKxOThBmZlbICcLMzAo5QZiZ\nWSEniJq5n8HM2pUThJmZFaosQUiaI2mlpAW5sjMlLZP0aHp8NrfuZElLJC2WtHdVcZmZWTlVnkFc\nDuxTUP6tiJieHncASNoOOAT4cNrnu5LWqjC2juOmKDNrtcoSRETcB7xccvMZwDURsToingWWADtW\nFZuZmQ2vjj6IEyQ9npqgJqWyzYEXctssTWXvIGmmpD5Jff39/VXHamY2brU6QVwMbANMB5YD54/0\nDSJidkT0RkRvT09Ps+NriqLmIDcRmVmnaWmCiIgVEfF6RLwBXMpbzUjLgC1zm26RysYNJxAzazct\nTRCSJudefgFojHC6FThE0rqSpgLTgIdaGZuZmb3dhKreWNLVwB7AJpKWAmcAe0iaDgTwHHAsQEQs\nlHQd8ASwBjg+Il6vKjYzMxteZQkiIg4tKL5siO3PAs6qKh4zMxsZX0ltZmaFnCDakDuszawdVNbE\nZBl/2ZtZpxr2DELSrpImpuUvSbpA0tbVh2bgBGNm9SnTxHQx8AdJHwG+BjwNXFlpVGZmVrsyCWJN\nRATZfEkXRcR3gA2rDcvMzOpWpg9ilaSTgcOB3SS9C1i72rDMzKxuZc4g/hxYDXw5Il4kmwbj3Eqj\nsrdxP4SZ1WHYBJGSwg3AuqnoJeCmKoMyM7P6lRnFdAxwPXBJKtocuLnKoMzMrH5lmpiOB3YFXgGI\niKeA91cZVDdws5CZdboyCWJ1RLzWeCFpAtlke2Zm1sXKJIh7JZ0CrC/p08APgR9VG1bn8RmDmXWb\nMgliFtAPzCebnvsO4LQqg+pUThJm1k2GvQ4iIt6QdAUwl6xpaXG6cM7MzLpYmVFM+5FNr3EhcBGw\nRNK+VQdmxXyWYmatUuZK6vOBPSNiCYCkbYHbgR9XGZiZmdWrTB/EqkZySJ4BVlUUj5mZtYkyCaJP\n0h2SjpR0BNkIpoclHSjpwMF2kjRH0kpJC3Jl50p6UtLjkm6StFEqnyLpvyQ9mh7fG3PNxgk3OZlZ\nVcokiPWAFcAngD3IRjStD3we+NwQ+10O7DOg7C7gv0XEfwd+CZycW/d0RExPj+NKRW9mZpUpM4rp\nqNG8cUTcJ2nKgLKf5l4+CHxxNO9tZmbVGzRBSPp6RJwj6f9ScOV0RPz1GI/9ZeDa3Oupkh4Ffgec\nFhH3DxLXTGAmwFZbbTXGEJrPTT5m1i2GOoNYlJ77mn1QSacCa4CrUtFyYKuI+I2kjwI3S/pwRLwy\ncN+ImA3MBujt7e2K6zGcVMysHQ2aICLiR+n5imYeUNKRZH0XezUuuIuI1WT3nCAi5kl6GvgAFSQn\nMzMrp8yFcnc1Rhul15Mk3Tmag0naB/g6sH9E/CFX3iNprbS8DTCNbDitmZnVpMyFcj0R8dvGi4j4\nT0nDTvct6WqyUU+bSFoKnEE2amld4C5JAA+mEUu7A38v6U/AG8BxEfHySCtjZmbNUyZBvC5pq4j4\nFYCkrSkx3XdEHFpQfNkg295Adte6juQ+BDPrRmUSxKnAA5LuBQTsRhpFZGZm3avMdRA/kbQDsHMq\nOikiXqo2LDMzq1uZTmqRXRG9Q0TcBrxb0o6VR2ZmZrUqM9XGd4FdgEafwirgO5VFZGZmbaFMH8RO\nEbGDpF/Am6OY1qk4LjMzq1mZM4g/pWsUArJrFsiGopqZWRcrkyAuBG4C3i/pLOAB4B8qjcrMzGpX\nZhTTVZLmAXuRDXM9ICIWDbPbuOFrIMysWw2ZIFLT0sKI+BDwZGtCMjOzdjBkE1NEvA4sltR+82qb\nmVmlyoximgQslPQQ8GqjMCL2rywqMzOrXZkE8X8qj8LMzNpOmU7qe1sRiJmZtZcyw1zNzGwccoIw\nM7NCgyYISXen52+2LhwzM2sXQ/VBTJb0MWB/SdeQXST3poh4pNLIzMysVkMliNPJRjBtAVwwYF0A\nn6wqKDMzq9+gTUwRcX1E7AucExF7DngMmxwkzZG0UtKCXNnGku6S9FR6npRbd7KkJZIWS9p7zDUb\nRzzdh5lVYdhO6oj4hqT9JZ2XHp8r+d6Xk91oKG8WcHdETAPuTq+RtB1wCPDhtM930zQfNgpOGGbW\nDGXuKPePwInAE+lxoqRhZ3ONiPuAlwcUzwCuSMtXAAfkyq+JiNUR8SywBGj7u9b5i9jMulmZYa77\nAZ+OiDkRMYfsP/yyZxEDbRoRy9Pyi8CmaXlz4IXcdktT2TtImimpT1Jff3//KMPoPk5WZtZsZa+D\n2Ci3/N5mHDgignQTohHuNzsieiOit6enpxmhdCUnDDMbqzJzMf0j8AtJ95ANdd2d1HcwCiskTY6I\n5ZImAytT+TJgy9x2W6QyMzOrSZlO6quBnYEbgRuAXSLi2lEe71bgiLR8BHBLrvwQSetKmgpMAx4a\n5THMzKwJypxBkPoNbh3JG0u6GtgD2ETSUuAM4GzgOklHA88DB6f3XyjpOrJO8DXA8eleFGZmVpNS\nCWI0IuLQQVbtNcj2ZwFnVRWPmZmNjCfrMzOzQkMmCElrSfK9qM3MxiHfk9rMzAqVaWJq3JP6bkm3\nNh5VB2bN52sjzGwkfE9qMzMrVOqe1JK2BqZFxM8kvRvwRHpmZl2uzGR9xwDXA5ekos2Bm6sMyszM\n6lemD+J4YFfgFYCIeAp4f5VBmZlZ/cokiNUR8VrjhaQJjGKSPWsNd0SbWbOUSRD3SjoFWF/Sp4Ef\nAj+qNiwzM6tbmQQxC+gH5gPHAncAp1UZlI2Ozx7MrJnKjGJ6Q9IVwFyypqXF6V4OZmbWxcqMYtoP\neBq4ELgIWCJp36oDs7HzGYWZjUWZC+XOB/aMiCUAkrYFbgd+XGVgZmZWrzJ9EKsaySF5BlhVUTzW\nZD6LMLPRGvQMQtKBabFP0h3AdWR9EAcBD7cgNjMzq9FQTUyfzy2vAD6RlvuB9SuLyMzM2sKgCSIi\njmplIGZm1l6G7aSWNBU4AZiS3z4i9h/NASV9ELg2V7QNcDqwEXAM2RkKwCkRccdojmFmZmNXZhTT\nzcBlZFdPvzHWA0bEYmA6ZHesA5YBNwFHAd+KiPPGegx7J3dWm9lIlUkQf4yICys6/l7A0xHxvKSK\nDmFmZqNRZpjrP0k6Q9IuknZoPJp0/EOAq3OvT5D0uKQ5kiYV7SBppqQ+SX39/f1Fm9gQfCZhZmWV\nSRB/RtY3cDbZRXPnA2NuBpK0DrA/2eR/ABeT9UdMB5an47xDRMyOiN6I6O3p6RlrGGZmNogyTUwH\nAdvkp/xukn2BRyJiBUDjGUDSpcBtTT6emZmNQJkziAVkI4ya7VByzUuSJufWfSEd18zMalLmDGIj\n4ElJDwOrG4WjHeYKIGki8Gmy6cMbzpE0nexq7ecGrDMzsxYrkyDOaPZBI+JV4H0Dyg5v9nEs445p\nMxuNMveDuLcVgZiZWXspcyX1Kt66B/U6wNrAqxHxnioDMzOzepU5g9iwsazsarYZwM5VBmVmZvUr\nM4rpTZG5Gdi7onjMzKxNlGliOjD38l1AL/DHyiIyM7O2UGYUU/6+EGvIhqDOqCQaMzNrG2X6IHxf\nCDOzcWioW46ePsR+ERHfqCAeawNTZt3Oc2fvV3cYZlazoc4gXi0omwgcTXaRmxOEmVkXG3QUU0Sc\n33gAs8nuQ30UcA3ZrKvjjq9INrPxZMg+CEkbA18FDgOuAHaIiP9sRWBmZlavQc8gJJ0LPAysAv4s\nIs50cuhOPjMysyJDXSj3NWAz4DTg15JeSY9Vkl5pTXhWNScHMxvMoE1METGiq6zNzKy7OAnYm3w2\nYWZ5ThAl+IvTzMYjJwgzMyvkBGFmZoVqSRCSnpM0X9KjkvpS2caS7pL0VHqeVEdslsk3q7mJzWx8\nqvMMYs+ImB4Rven1LODuiJgG3J1etw1/SZrZeNNOTUwzyK7WJj0fUGMsZmbjXl0JIoCfSZonaWYq\n2zQilqflF4FN6wnNzMygvgTx8YiYDuwLHC9p9/zKiAiyJPIOkmZK6pPU19/f34JQxy83q5mNb7Uk\niIhYlp5XAjcBOwIrJE0GSM8rB9l3dkT0RkRvT09Pq0I2Mxt3Wp4gJE2UtGFjGfgMsAC4FTgibXYE\ncEurYxvPGmcLPmsws4Yy96Rutk2BmyQ1jv+DiPiJpIeB6yQdDTwPHFxDbGZmlrQ8QUTEM8BHCsp/\nA+zV6njMzKxYOw1ztQ7lZimz7uQEYWZmhZwgrDSfKZiNL04QZmZWyAnCzMwKOUFY07gJyqy7OEGY\nmVkhJwgzMyvkBGFDcrOR2fjlBGFmZoWcIKwUn0mYjT9OEGZmVsgJYhj+z9nMxisnCDMzK+QEYWZm\nhZwgzMyskBPEENz/YGbjmRPEIJwcipX5XIq28edp1nlaniAkbSnpHklPSFoo6cRUfqakZZIeTY/P\ntjo2MzN7S8vvSQ2sAb4WEY9I2hCYJ+mutO5bEXFeDTGZmdkALU8QEbEcWJ6WV0laBGze6jjMzGxo\ntfZBSJoCbA/MTUUnSHpc0hxJk2oLzJrK/Q9mnam2BCFpA+AG4KSIeAW4GNgGmE52hnH+IPvNlNQn\nqa+/v79l8drQGklguGTgZGHWOWpJEJLWJksOV0XEjQARsSIiXo+IN4BLgR2L9o2I2RHRGxG9PT09\nrQvaRsSJwKzz1TGKScBlwKKIuCBXPjm32ReABa2OzZrPQ17NOlcdo5h2BQ4H5kt6NJWdAhwqaToQ\nwHPAsTXEZmZmSR2jmB4AVLDqjlbHYmZmg/OV1GZmVsgJwirTzL4G91uYtZ4ThJmZFXKCKOD/Vuvh\nEU9m7cUJwtqCE4FZ+3GCMDOzQk4QZmZWyAliADd1VMefrVlncYKwtldmIkAnH7Pmc4LI8ZdM/fI/\nA/88zOrlBGG1G5gIxpIYpsy6fdD9nXDMRsYJwjrKSL7knRDMxsYJwsYdJw6zcpwgrC2NpEPaX/hm\n1XCCSPwlM76538LsnZwgrGuMdQRUO/RvOCFZOxnXCcJ/jN1vLM1RI9nXw3OtG43rBGHjw1BDXwfb\nfrCysu/jJGHdoO0ShKR9JC2WtETSrLrjMTMbr9oqQUhaC/gOsC+wHXCopO2qPOZI/7u08We0vx9F\nv1sjPRMZbNuRxtRuI7/qPr6V01YJAtgRWBIRz0TEa8A1wIyaY7JxZDRf4AP3bVYMY93XN2AaG39W\n7ZcgNgdeyL1emsrM2sZIJw0sm3SK/ssfuG/Z4bhlOs1HM7S3zFQmQ501lfnSLRPXaN6n1V/43ZBg\nFBF1x/AmSV8E9omIr6TXhwM7RcRf5baZCcxMLz8ILC759psALzUx3Lp0Sz3AdWlX3VKXbqkHNL8u\nW0dEz3AbTWjiAZthGbBl7vUWqexNETEbmD3SN5bUFxG9Ywuvft1SD3Bd2lW31KVb6gH11aXdmpge\nBqZJmippHeAQ4NaaYzIzG5fa6gwiItZI+ivgTmAtYE5ELKw5LDOzcamtEgRARNwB3FHBW4+4WapN\ndUs9wHVpV91Sl26pB9RUl7bqpDYzs/bRbn0QZmbWJro+QXTa1B2StpR0j6QnJC2UdGIq31jSXZKe\nSs+TcvucnOq3WNLe9UX/TpLWkvQLSbel151aj40kXS/pSUmLJO3SwXX5m/S7tUDS1ZLW65S6SJoj\naaWkBbmyEccu6aOS5qd1F0pSm9Tl3PQ79rikmyRtVGtdIqJrH2Qd3U8D2wDrAI8B29Ud1zAxTwZ2\nSMsbAr8km3bkHGBWKp8FfDMtb5fqtS4wNdV3rbrrkavPV4EfALel151ajyuAr6TldYCNOrEuZBee\nPgusn15fBxzZKXUBdgd2ABbkykYcO/AQsDMg4MfAvm1Sl88AE9LyN+uuS7efQXTc1B0RsTwiHknL\nq4BFZH/UM8i+pEjPB6TlGcA1EbE6Ip4FlpDVu3aStgD2A76fK+7EeryX7I/5MoCIeC0ifksH1iWZ\nAKwvaQLwbuDXdEhdIuI+4OUBxSOKXdJk4D0R8WBk37BX5vZpmaK6RMRPI2JNevkg2bVgUFNduj1B\ndPTUHZKmANsDc4FNI2J5WvUisGlabuc6fhv4OvBGrqwT6zEV6Af+OTWXfV/SRDqwLhGxDDgP+BWw\nHPhdRPyUDqxLzkhj3zwtDyxvN18mOyOAmurS7QmiY0naALgBOCkiXsmvS/8ptPXwM0mfA1ZGxLzB\ntumEeiQTyJoCLo6I7YFXyZoy3tQpdUnt8zPIkt5mwERJX8pv0yl1KdLJsedJOhVYA1xVZxzdniCG\nnbqjHUlamyw5XBURN6biFel0kvS8MpW3ax13BfaX9BxZ094nJf0rnVcPyP4rWxoRc9Pr68kSRifW\n5VPAsxHRHxF/Am4EPkZn1qVhpLEv462mm3x5W5B0JPA54LCU8KCmunR7gui4qTvSCITLgEURcUFu\n1a3AEWn5COCWXPkhktaVNBWYRtZpVauIODkitoiIKWSf+79FxJfosHoARMSLwAuSPpiK9gKeoAPr\nQta0tLOkd6fftb3I+rk6sS4NI4o9NUe9Imnn9Bn8RW6fWknah6xZdv+I+ENuVT11aXXPfasfwGfJ\nRgI9DZxadzwl4v042Sny48Cj6fFZ4H3A3cBTwM+AjXP7nJrqt5gaRmOUqNMevDWKqSPrAUwH+tLP\n5WZgUgfX5e+AJ4EFwL+QjYzpiLoAV5P1nfyJ7Mzu6NHEDvSm+j8NXES6aLgN6rKErK+h8bf/vTrr\n4iupzcysULc3MZmZ2Sg5QZiZWSEnCDMzK+QEYWZmhZwgzMyskBOEjZmkkHR+7vXfSjqzSe99uaQv\nNuO9hjnOQWmW1nuqPlY63pGSLiq57WaSrq86ppFKU45sV3ccVh0nCGuG1cCBkjapO5C8NBldWUcD\nx0TEnhXEIUmj/luLiF9HRKVJcoSfFQAR8ZWIeKKKeKw9OEFYM6whuyXi3wxcMfAMQNLv0/Meku6V\ndIukZySdLekwSQ+lue23zb3NpyT1SfplmuOpcZ+JcyU9nObOPzb3vvdLupXsaueB8Rya3n+BpG+m\nstPJLlC8TNK5A7b/jqT90/JNkuak5S9LOistfzW93wJJJ6WyKWne/ivJLmLaUtJRqQ4PkU1F0jjG\nQWnfxyTdVxDzFKV7BqQzjxsl/UTZ/Q/OKfqBSDo9fTYLJM1OV9kW/Wy+J2kucI6kicruUfCQskkJ\nZ+Q+6/PSez0u6YRU/nNJvY2fq6RvKbvPxN2SelL5tinWeenn8qGieK1N1X1Vpx+d/wB+D7wHeA54\nL/C3wJlp3eXAF/Pbpuc9gN+S3f9iXbL5Y/4urTsR+HZu/5+Q/TMzjeyK0/WAmcBpaZt1ya5ynpre\n91VgakGcm5FNNdFDNgHfvwEHpHU/B3oL9jkEODctPwQ8mJb/Gdgb+CgwH5gIbAAsJJuBdwrZLLY7\np+0n5469DvDvwEVp3Xxg87S8UUEMU0j3DCC7d8Mz6XNeD3ge2LJgn/zVxP8CfL5gm8uB23jrvgL/\nAHypEQfZDAQTgb8km39qQv69858Z2dX/h6Xl03N1uxuYlpZ3IptypfbfWT/KPXwGYU0R2YyzVwJ/\nPYLdHo7s/heryaYJ+Gkqn0/2pdhwXUS8ERFPkX05fojsxip/IelRsunQ30eWQCCbo+bZguP9D+Dn\nkU1U15gpc/dhYrwf2C21tT/BWxPD7QL8P7Izj5si4tWI+D3Z5He7pX2fj4gH0/JOuWO/BlybO8a/\nA5dLOobsJlfDuTsifhcRf0wxbV2wzZ6S5kqaD3wS+PAg7/XDiHg9LX8GmJU+05+TJaCtyCb4uyR9\nZkTEwPsxQJYMG3X6V+DjymYk/hjww/Sel5AlSusQI253NBvCt4FHyP67blhDaspM7fDr5Natzi2/\nkXv9Bm//3Rw4H0yQ3T3rhIi4M79C0h5kZxBNERHLlN32cR/gPmBj4GCyM6FVBS03eaXiiIjjJO1E\ndnOleZI+GhG/GWKX/Of2OgP+jiWtB3yX7L/7F9KAgfVKxCjgf0bE4gHvV6YaAwXZz/23ETF9NG9g\n9fMZhDVN+s/yOrIO34bnyJphAPYH1h7FWx8k6V2pX2IbssnK7gT+UtnU6Ej6gLKb+AzlIeATkjaR\ntBZwKHBvieM/CJxEliDuJ2tCuz+tux84QNnsqBOBL+TW5c1Nx35fivmgxgpJ20bE3Ig4nezGRFsW\n7D8SjWTwUvovvmwH953ACY3+Cknbp/K7gGMbHdmSNi7Y91254/wv4IF0VvmspIPSfpL0kRHXxmrj\nBGHNdj6QH810KdkX42NkzTKj+e/+V2Rf7j8GjktNK98na155JHXgXsIwZ8SRTY08C7iH7P6+8yKi\nzNTI95O1vy8hO0PaOJUR2e1hL0/xzQW+HxG/GOTYZwL/QdaktCi3+txGxzlZs9VjJWIaVGS3Q72U\nrHP8TrJp78v4BlkCf1zSwvQass/6V6n8MbIEMNCrZLfAXEDWpPX3qfww4Oi030La/Ja/9naezdXM\nxkzS7yNig7rjsObyGYSZmRXyGYSZmRXyGYSZmRVygjAzs0JOEGZmVsgJwszMCjlBmJlZIScIMzMr\n9P8BGhTVGE4YpCEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14803daf198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar([item[0] for item in words_per_recipe.items()], [item[1] for item in words_per_recipe.items()])\n",
    "plt.xlabel(\"Number of words in a recipe\")\n",
    "plt.ylabel(\"Number of recipes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distribution of the size of recipes in characters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1302,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "chars_per_recipe = Counter([len(recipe) for recipe in recipes])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1303,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x1473b84cf60>"
      ]
     },
     "execution_count": 1303,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEKCAYAAAAMzhLIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAGXBJREFUeJzt3XuwZWV55/Hvj5sggoC0XR0RG1NEixhF0/GGQfAWkCjG\niNFS0zEoZsoYcmFMo47BSiUhsTCJUaMYTTrGG96gVRSxFZzMOEBzExAQRUx0gG6MCWhGDPDMH+s9\nsD32WWf1ZZ+z9znfT9Wuvda7bs97LvvZa73rfVeqCkmS5rLLYgcgSZpsJgpJUi8ThSSpl4lCktTL\nRCFJ6mWikCT1MlFIknrtNs6dJ7kJuAO4G7irqtYkOQD4MLAauAl4YVV9b5xxSJK230KcURxdVYdX\n1Zo2vw7YWFWHAhvbvCRpQmWcPbPbGcWaqrptpOx64KiqujnJKuCCqnpE334OPPDAWr169djilKSl\n6NJLL72tqlbs6H7GeukJKODzSe4G3lVVZwIrq+rmtvwWYOV8O1m9ejWbNm0aY5iStPQk+dbO2M+4\nE8VTquo7SR4MnJ/kutGFVVVJtnpKk+Qk4CSAgw8+eMxhSpLmMtY2iqr6TnvfDHwCeDxwa7vkRHvf\nPMe2Z1bVmqpas2LFDp85SZK209gSRZK9k+wzMw08C7ga2ACsbautBc4ZVwySpB03zktPK4FPJJk5\nzgeq6rNJLgHOSnIi8C3ghWOMQZK0g8aWKKrqRuAxWyn/LvD0cR1XkrRz2TNbktTLRCFJ6mWikCT1\nMlFIknqZKCRJvUwUkqReJgpJUi8ThSSpl4lCktTLRCFJ6mWikCT1MlFIknqZKCRJvUwUkqReJgpJ\nUi8ThSSpl4lCktTLRCFJ6mWikCT1MlFIknqZKCRJvUwUkqReJgpJUi8ThSSpl4lCktTLRCFJ6mWi\nkCT1MlFIknqZKCRJvUwUkqReJgpJUi8ThSSpl4lCktTLRCFJ6mWikCT1GnuiSLJrksuTfKrNH5Dk\n/CQ3tPf9xx2DJGn7LcQZxcnAtSPz64CNVXUosLHNS5Im1FgTRZKDgOOAvxspPh5Y36bXA88bZwyS\npB0z7jOKvwJeC9wzUrayqm5u07cAK7e2YZKTkmxKsmnLli1jDlOSNJexJYokvwxsrqpL51qnqgqo\nOZadWVVrqmrNihUrxhWmJGkeu41x30cAz03ybGBPYN8k/wTcmmRVVd2cZBWweYwxSJJ20NjOKKrq\n1Ko6qKpWAy8CvlBVLwU2AGvbamuBc8YVgyRpxy1GP4rTgWcmuQF4RpuXJE2ocV56uldVXQBc0Ka/\nCzx9IY4rSdpx9syWJPUyUUiSepkoJEm9TBSSpF4mCklSLxOFJKmXiUKS1MtEIUnqZaKQJPUyUUiS\nepkoJEm9TBQ7aPW6Ty/KtpK0UEwUkqRe8yaKJEck2btNvzTJW5I8bPyhSZImwZAzir8F/jPJY4A/\nAL4B/ONYo5IkTYwhieKu9mzr44G3VdXbgX3GG5YkaVIMeXDRHUlOBV4G/GKSXYDdxxuWJGlSDDmj\n+DXgTuA3q+oW4CDgzWONSpI0MeZNFC05fAy4Xyu6DfjEOIOSJE2OIXc9vRL4KPCuVvQQ4OxxBiVJ\nmhxDLj29GjgCuB2gqm4AHjzOoJaScXaqs8OepIUwJFHcWVU/mplJshtQ4wtJkjRJhiSKC5O8Dtgr\nyTOBjwCfHG9YkqRJMSRRrAO2AFcBrwLOBd4wzqAkSZNj3n4UVXVPkvXARXSXnK5vHfAkScvAvIki\nyXHAO+mG7ghwSJJXVdVnxh2cJGnxDemZfQZwdFV9HSDJTwOfBkwUkrQMDGmjuGMmSTQ3AneMKR5J\n0oQZckaxKcm5wFl0bRQnAJckeT5AVX18jPFJkhbZkDOKPYFbgacCR9HdAbUX8Bzgl8cW2RIzV+e4\nbS2XpIU25K6nly9EIJKkyTRnokjy2qr6iyR/w1Z6YlfV74w1MknSROg7o7i2vW9aiEAkSZNpzkRR\nVZ9s7+sXLpylb/W6T3PT6cctdhiSNNiQYcbPT7LfyPz+Sc4bsN2eSS5OcmWSa5K8qZUf0PZ5Q3vf\nf8eqIEkapyF3Pa2oqn+fmamq7zFsmPE7gadV1WOAw4FjkjyRbuyojVV1KLCxzUuSJtSQRHF3koNn\nZpI8jAHDjFfn+2129/Yq4Hhg5nLWeuB52xSxJGlBDUkUrwf+Ocn7kvwT8CXg1CE7T7JrkiuAzcD5\nVXURsLKqbm6r3AKsnGPbk5JsSrJpy5YtQw439ew7IWkSDelH8dkkjwOe2Ip+t6puG7LzqrobOLy1\ncXwiyaNmLa8kWz07qaozgTMB1qxZ42i1krRIhjRmBzgGeFxVfQq4f5LHb8tBWhvHF9t+bk2yqu17\nFd3ZhiRpQg259PQO4EnAi9v8HcDb59soyYqZu6WS7AU8E7gO2ACsbautBc7ZxpglSQtoyKCAT6iq\nxyW5HLq7npLsMWC7VcD6JLvSJaSzqupTSb4MnJXkROBbwAu3N3hJ0vgNSRT/1T7sC7ozBeCe+Taq\nqq8Aj91K+XeBp29jnEuanfAkTbIhl57eCnwCeHCSPwH+GfjTsUYlSZoYQ+56en+SS+nOAgI8r6qu\nnWczSdIS0Zso2iWna6rqkXQN0ZKkZab30lPrB3H9aM9sbd2Odpazs52kSTWkMXt/4JokFwM/mCms\nqueOLSpJ0sQYkij+x9ijkCRNrCGN2RcuRCCSpMk05PZYbYe+NodtaY9Yve7Ttl9IWlQmCklSrzkT\nRZKN7f3PFy4cSdKk6WujWJXkycBzk3yIrrPdvarqsrFGJkmaCH2J4o10dzwdBLxl1rICnjauoCRJ\nk2POS09V9dGqOhb4i6o6etbLJMH4O8nN3v/MvI3bkhbSkNtj/zjJc4EjW9EF7QFGkqRlYMgT7v4M\nOBn4anudnMTRYyVpmRjSM/s44PCqugcgyXrgcuB14wxMkjQZhvaj2G9k+oHjCGQp2NaOdJI0DYac\nUfwZcHmSL9LdInsksG6sUUmSJsaQxuwPJrkA+IVW9IdVdctYo5IkTYwhZxRU1c3AhjHHIkmaQI71\ntJPZ9iBpqTFRSJJ69SaKJLsm8VnZkrSM+cxsSVIvn5ktSerlM7MnzOp1n+am048b+zaSNNSgZ2Yn\neRhwaFV9Psn9gV3HH5okaRIMGRTwlcBHgXe1oocAZ48zKEnS5Bhye+yrgSOA2wGq6gbgweMMSpI0\nOYYkijur6kczM0l2o3vC3bJlpzpJy8mQRHFhktcBeyV5JvAR4JPjDUuSNCmGJIp1wBbgKuBVwLnA\nG8YZlCRpcgy56+me9rCii+guOV1fVcv60pMkLSdD7no6DvgG8FbgbcDXkxw77sCmyc5us9jR/Y1u\nb3uKpB015NLTGcDRVXVUVT0VOBr4y/k2SvLQJF9M8tUk1yQ5uZUfkOT8JDe09/13rAqSpHEakiju\nqKqvj8zfCNwxYLu7gD+oqsOAJwKvTnIYXZvHxqo6FNiIT8uTpIk2ZxtFkue3yU1JzgXOomujOAG4\nZL4dt4cd3dym70hyLV1nveOBo9pq64ELgD/cvvAlSePW15j9nJHpW4GntuktwF7bcpAkq4HH0jWI\nr2xJBOAWYOW27EuStLDmTBRV9fKdcYAkDwA+BvxuVd2eZPQYlWSrd1AlOQk4CeDggx3lXJIWy7y3\nxyY5BHgNsHp0/SHDjCfZnS5JvL+qPt6Kb02yqqpuTrIK2Ly1bavqTOBMgDVr1ng7riQtkiHDjJ8N\nvIeuN/Y9Q3ec7tThPcC1VfWWkUUbgLXA6e39nMHRSpIW3JBE8cOqeut27PsI4GXAVUmuaGWvo0sQ\nZyU5EfgW8MLt2LckaYEMSRR/neSPgM8Bd84UVtVlfRtV1T8DmWPx0wdHOKEmsSObDzCSNA5DEsXP\n0Z0ZPI37Lj1Vm5ckLXFDEsUJwMNHhxqXJC0fQ3pmXw3sN+5AJEmTaUii2A+4Lsl5STbMvMYdmHae\nSWxPkTQ9hlx6+qOxRyFJmlhDnkdx4UIEIkmaTEN6Zt/Bfc/I3gPYHfhBVe07zsAkSZNh3jaKqtqn\nqvZtiWEv4FeBd4w9Mu0Q2yUk7SxDGrPvVZ2zgV8aUzySpAkz5NLT80dmdwHWAD8cW0SSpIky5K6n\n0edS3AXcRPfwIUnSMjDkrqed8lwKSdJ06nsU6ht7tquq+uMxxDPxprWR2AEDJW2vvjOKH2ylbG/g\nROBBwLJMFJK03PQ9CvWMmekk+wAnAy8HPgScMdd2kqSlpbeNIskBwO8DLwHWA4+rqu8tRGCSpMnQ\n10bxZuD5dM+t/rmq+v6CRaXtNtqGMq3tKZImS1+Huz8Afgp4A/B/k9zeXnckuX1hwpMkLba+Nopt\n6rUtSVqaTAaSpF4minkstev8Q9swbOuQNMNEIUnqZaKQJPUyUUiSepkoJEm9TBQD2aArabkyUUiS\nepkoJEm9TBSSpF4mim0w7e0UM/FPez0kLSwThSSpl4lCktTLRCFJ6mWiELBz2y9sA5GWlrEliiTv\nTbI5ydUjZQckOT/JDe19/3EdX5K0c4zzjOIfgGNmla0DNlbVocDGNi9JmmBjSxRV9SXg32YVHw+s\nb9PrgeeN6/iSpJ1jodsoVlbVzW36FmDlAh9fkrSNFq0xu6oKqLmWJzkpyaYkm7Zs2TLWWOZrfLVx\nVtJyttCJ4tYkqwDa++a5VqyqM6tqTVWtWbFixYIFKEn6cQudKDYAa9v0WuCcBT6+JGkbjfP22A8C\nXwYekeTbSU4ETgeemeQG4BltXpI0wcZ519OLq2pVVe1eVQdV1Xuq6rtV9fSqOrSqnlFVs++KmgjL\nafC8ba3jcviZSPpx9syWJPUyUUiSepkoJEm9TBQjltP196F13dp6c227nH5+0nJiopAk9TJRSJJ6\nmSgkSb1MFJKkXiYKDbIjjd/bs50N49LkMFFIknqZKCRJvUwUkqReJgoNNtpusC1tCLY/SNPNRCFJ\n6mWikCT1MlFIknot60ThtfIdt3rdp+99be/2kibbsk4UkqT5mSgkSb1MFJKkXiYKSVKvZZkobEDd\nOXbWz3FnPjHP36208y3LRCFJGs5EIUnqZaKQJPUyUWhstnUwwPk67m3Psh0ZvFBSx0QhSeplopAk\n9TJRSJJ6LbtE4XXoyTGkXWH2oIM78hCkufYxLkOO49+jpsGySxSSpG1jopAk9TJRSJJ6mSg0lebr\nU7G1do7R7UbbPuZr95hrX3Ptf2uxbW97Rd+xtnf7nWUc7Svb+7Nbqial7ouSKJIck+T6JF9Psm4x\nYpAkDbPgiSLJrsDbgWOBw4AXJzlsoeOQJA2zGGcUjwe+XlU3VtWPgA8Bxy9CHJKkARYjUTwE+NeR\n+W+3MknSBEpVLewBkxcAx1TVK9r8y4AnVNVvz1rvJOCkNvsI4Pp5dn0gcNtODncSWK/pYr2my1Kv\n18OqasWO7my3HY9nm30HeOjI/EGt7MdU1ZnAmUN3mmRTVa3Z8fAmi/WaLtZrulivYRbj0tMlwKFJ\nDkmyB/AiYMMixCFJGmDBzyiq6q4kvw2cB+wKvLeqrlnoOCRJwyzGpSeq6lzg3J2828GXqaaM9Zou\n1mu6WK8BFrwxW5I0XRzCQ5LUa+oTxbQNB5LkvUk2J7l6pOyAJOcnuaG97z+y7NRWt+uT/NJI+c8n\nuaote2uSLHRdRiV5aJIvJvlqkmuSnNzKp7puSfZMcnGSK1u93tTKp7peIzHtmuTyJJ9q81NfryQ3\ntXiuSLKplS2Feu2X5KNJrktybZInLVi9qmpqX3SN4d8AHg7sAVwJHLbYcc0T85HA44CrR8r+AljX\nptcBf96mD2t1uh9wSKvrrm3ZxcATgQCfAY5d5HqtAh7XpvcBvtbin+q6tRge0KZ3By5qsU11vUbq\n9/vAB4BPLaG/xZuAA2eVLYV6rQde0ab3APZbqHot6h/pTvjBPQk4b2T+VODUxY5rQNyr+fFEcT2w\nqk2vAq7fWn3o7hR7UlvnupHyFwPvWux6zarjOcAzl1LdgPsDlwFPWAr1ouvDtBF4GvcliqVQr5v4\nyUQx1fUCHgh8k9auvND1mvZLT0tlOJCVVXVzm74FWNmm56rfQ9r07PKJkGQ18Fi6b99TX7d2eeYK\nYDNwflUtiXoBfwW8FrhnpGwp1KuAzye5NN0IDzD99ToE2AL8fbtU+HdJ9maB6jXtiWLJqS7NT+2t\naEkeAHwM+N2qun102bTWrarurqrD6b6BPz7Jo2Ytn7p6JfllYHNVXTrXOtNYr+Yp7fd1LPDqJEeO\nLpzSeu1Gd8n6b6vqscAP6C413Wuc9Zr2RDFoOJApcGuSVQDtfXMrn6t+32nTs8sXVZLd6ZLE+6vq\n4614SdQNoKr+HfgicAzTX68jgOcmuYluBOenJfknpr9eVNV32vtm4BN0I1ZPe72+DXy7nc0CfJQu\ncSxIvaY9USyV4UA2AGvb9Fq66/sz5S9Kcr8khwCHAhe3U83bkzyx3bHw6yPbLIoWx3uAa6vqLSOL\nprpuSVYk2a9N70XX7nIdU16vqjq1qg6qqtV0/zdfqKqXMuX1SrJ3kn1mpoFnAVcz5fWqqluAf03y\niFb0dOCrLFS9FqtxZic28jyb7g6bbwCvX+x4BsT7QeBm4L/oviWcCDyIrlHxBuDzwAEj67++1e16\nRu5OANbQ/QN8A3gbsxq5FqFeT6E77f0KcEV7PXva6wY8Gri81etq4I2tfKrrNauOR3FfY/ZU14vu\nDsgr2+uamc+Eaa9Xi+dwYFP7Wzwb2H+h6mXPbElSr2m/9CRJGjMThSSpl4lCktTLRCFJ6mWikCT1\nMlEsY0kqyRkj86ckOW0n7fsfkrxgZ+xrnuOc0EbS/OLA9S9IsmDPSE5yVJInj2nf58708ZgUSX4r\nya8vdhzauUwUy9udwPOTHLjYgYxKsi1PXjwReGVVHT2ueGZsY1wzjgK2KVEMPU5VPbu63uJjsT31\nrap3VtU/jiMeLR4TxfJ2F90jE39v9oLZZwRJvt/ej0pyYZJzktyY5PQkL0n3zIarkvz0yG6ekWRT\nkq+1sYVmBth7c5JLknwlyatG9vs/k2yg63E6O54Xt/1fneTPW9kb6Tr6vSfJm7eyzR+2ba5McvrI\nohNavF9L8ott3dXt+Je115PniivJ2ekGnLsm9w06N/NslMva8TamGxzxt4DfS/dshF9sPb0/1up/\nSZIj2ranJXlfkv8FvC/Jz7YYr2g/p0O3Ur+bkhzYYr82ybtbTJ9rvchnr/+cJBelG1Tu80lWbmWd\n30iyIckX6DpykeS/j/y+3jSy7q+3siuTvG+kHqe06QuS/HWrw9VJHt/K9073XJaLWyzHz45DE2ax\ne4X6WrwX8H1gX7phmR8InAKc1pb9A/CC0XXb+1HAv9MNV3w/unFi3tSWnQz81cj2n6X7MnIoXS/0\nPYGTgDe0de5H19P0kLbfHwCHbCXOnwL+BVhBNzjaF4DntWUXAGu2ss2xwP8G7t/mDxhZ/4w2/Wzg\n8236/sCebfpQYNNIfX8srpF97UXXw/VBLbZ/nVlvZJ3TgFNGtv0A3aB1AAfTDXkys96lwF5t/m+A\nl7TpPWbKZ9XxJuBAumHr7wIOb+VnAS/dyvr7c9/jj18x83OYtc5vtN/VTPzPovsykfa7/BTdM1V+\nlm5EhAPnqm/7Wb+7TR9JG1of+NOZ+OieqfA1YO/F/n/wNfdre06ltYRU1e1J/hH4HeD/DdzskmpD\nGyf5BvC5Vn4VMHoJ6Kyquge4IcmNwCPpPngePXK28kC6D+Yf0Y1F882tHO8XgAuqaks75vvpPnjO\n7onxGcDfV9V/tnr+28iymQELL6X7kIXuoURvS3I4cDfwMyPrz47rd5L8Spt+aIt/BfClmfVmHW92\nXIflvoeK7ZtuxF2ADVU18zv4MvD6JAcBH6+qG3rqCvDNqrpiK/UadRDw4XSDx+1B93yDrTl/JP5n\ntdflbf4BdPV9DPCRqroNeuv7wbb8S0n2bW0qz6IbkPCUts6etKQ5Tx21SEwUgu65BJcBfz9Sdhft\n0mSSXeg+WGbcOTJ9z8j8Pfz439Ts8WGK7pvpa6rqvNEFSY6i++a+EGbivZv74v094Fa6D8BdgB+O\nrH9vXC3OZwBPqqr/THIB3QfdULsAT6yq0f3TEse9x6mqDyS5CDgOODfJq6rqCwPqNFOvn7j0RHeW\n8paq2tDqcdoc+xr9PQT4s6p616x4X9MTy6i5/gZ+taquH7gPLTLbKDTzbfAsuobhGTcBP9+mn0v3\njXtbnZBkl9Zu8XC6wcnOA/5buiHJSfIz6Ub57HMx8NR2PX5XuqdyXTjPNucDL09y/3acA+ZZ/4HA\nze0M6GV0j9mda73vtSTxSLpHSgL8H+DIdCN1jh7vDrpHw874HHDvh2w7g/kJSR4O3FhVb6Ub3fPR\n88Q/xAO5b0jptX0rjjgP+M2Zs54kD0nyYLrLfyckeVArn+vn+2tt+VOA/6iq/2j7fE1adkzy2O2p\njBaOiUIzzqC73j3j3XQfzlfSPUJxe77t/wvdh/xngN9q36L/jq5R+LIkVwPvYp4z23aZax3dsyCu\nBC6tqt6hkavqs3RDLW9K93S6U/rWB94BrG31fSRz1/ezwG5JrgVOp0sQtMtiJwEfb/v4cFv/k8Cv\nzDRm013iW9Magb9K19i9NS8Erm6xPwrYGXcSnQZ8JMmlwG1DNqiqz9G1q3w5yVV0z0HYp6quAf4E\nuLDV9y1z7OKHSS4H3sl9X0T+mO6Lx1eSXNPmNcEcPVbSWLTLcqdU1abFjkU7xjMKSVIvzygkSb08\no5Ak9TJRSJJ6mSgkSb1MFJKkXiYKSVIvE4Ukqdf/B706go1835cNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1473b638860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar([item[0] for item in chars_per_recipe.items()], [item[1] for item in chars_per_recipe.items()])\n",
    "plt.xlabel(\"Number of characters in a recipe\")\n",
    "plt.ylabel(\"Number of recipes\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1185,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "len_per_word = Counter([len(word) for word in vocabulary])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x146bd5d82b0>"
      ]
     },
     "execution_count": 1186,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZUAAAEKCAYAAADaa8itAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG/dJREFUeJzt3Xu8V3Wd7/HXW/CCFjfh8CDAAWdID5pZ7pSyHEo7kqaY\now49KjkeBmqksk49ErooMz2coUejJ21GZ0hTrEZDMmVUvKGYM+dw2YDGTYIEFULALoPZhIKf88f6\n/mTtn/vyY+/1u8H7+Xj8Hr+1vuu71vr8fmz2Z3/Xd63vVxGBmZlZEQ6pdwBmZnbgcFIxM7PCOKmY\nmVlhnFTMzKwwTipmZlYYJxUzMyuMk4qZmRWmaklF0vcl7ZC0Olc2UNIjkjak9wG5bTMkbZS0XtLZ\nufJTJK1K226QpFR+uKQfp/IlkkZW67OYmVllqtlSuQ0YX1Y2HVgYEaOBhWkdSWOAicAJaZ8bJfVK\n+9wETAFGp1fpmJOB30bEnwH/B/hW1T6JmZlVpHe1DhwRP2un9TABGJeW5wCLgCtT+Z0RsRvYJGkj\ncKqkzUDfiFgMIOl24AJgQdpnZjrWPOAfJSm6GCJg0KBBMXJkeVhmZtaZ5cuXvxQRg7uqV7Wk0oEh\nEbEtLb8IDEnLw4DFuXpbUtlrabm8vLTPCwARsUfSfwJHAy91FsDIkSNpbW3tyWcwMzvoSHquknp1\n66hPLYqaDDwmaaqkVkmtO3furMUpzcwOSrVOKtslDQVI7ztS+VZgRK7e8FS2NS2Xl7fZR1JvoB/w\n6/ZOGhGzI6IlIloGD+6y9WZmZt1U66QyH5iUlicB9+bKJ6Y7ukaRdcgvTZfKdkkam+76urRsn9Kx\nLgIe66o/xczMqqtqfSqS7iDrlB8kaQtwNTALmCtpMvAccAlARKyRNBdYC+wBpkXE3nSoy8nuJOtD\n1kG/IJXfAvwgder/huzuMTMzqyMdbH/ct7S0hDvqzcz2j6TlEdHSVT0/UW9mZoVxUjEzs8I4qZiZ\nWWGcVMzMrDC1fqLeKjBy+v1t1jfPOrdOkZiZ7R+3VMzMrDBOKmZmVhgnFTMzK4yTipmZFcYd9Q2g\nvGPezKxZuaViZmaFcVIxM7PCOKmYmVlhnFTMzKwwTipmZlYYJxUzMyuMk4qZmRXGScXMzArjpGJm\nZoVxUjEzs8I4qZiZWWGcVMzMrDAeULJJ5Aed9EyQZtao3FIxM7PCOKmYmVlhnFTMzKwwTipmZlYY\nJxUzMyuMk4qZmRXGtxTXgW8PNrMDlVsqZmZWGCcVMzMrjJOKmZkVxknFzMwK46RiZmaFqUtSkfRF\nSWskrZZ0h6QjJA2U9IikDel9QK7+DEkbJa2XdHau/BRJq9K2GySpHp/HzMwyNU8qkoYBnwdaIuJE\noBcwEZgOLIyI0cDCtI6kMWn7CcB44EZJvdLhbgKmAKPTa3wNP4qZmZWp1+Wv3kAfSb2BI4FfAROA\nOWn7HOCCtDwBuDMidkfEJmAjcKqkoUDfiFgcEQHcntvHzMzqoOZJJSK2Av8APA9sA/4zIh4GhkTE\ntlTtRWBIWh4GvJA7xJZUNiwtl5e/iaSpklolte7cubOwz2JmZm3V4/LXALLWxyjgbcBRkj6Zr5Na\nHlHUOSNidkS0RETL4MGDizqsmZmVqcflr7OATRGxMyJeA+4G3gdsT5e0SO87Uv2twIjc/sNT2da0\nXF5uZmZ1Uo+k8jwwVtKR6W6tM4F1wHxgUqozCbg3Lc8HJko6XNIosg75pelS2S5JY9NxLs3tY2Zm\ndVDzASUjYomkecAKYA+wEpgNvAWYK2ky8BxwSaq/RtJcYG2qPy0i9qbDXQ7cBvQBFqSXmZnVSV1G\nKY6Iq4Gry4p3k7Va2qt/DXBNO+WtwImFB2hmZt3iJ+rNzKwwTipmZlYYJxUzMyuMk4qZmRXGScXM\nzArjpGJmZoVxUjEzs8I4qZiZWWGcVMzMrDBOKmZmVhgnFTMzK4yTipmZFcZJxczMCuOkYmZmhXFS\nMTOzwjipmJlZYfYrqUgaIOmkagVjZmbNrcukImmRpL6SBpJNAfw9SddVPzQzM2s2lbRU+kXELuBC\n4PaIOA04q7phmZlZM6okqfSWNBS4BLivyvGYmVkTqySp/C3wELAxIpZJOhbYUN2wzMysGfXuqkJE\n3AXclVt/FviLagZlZmbNqcOkIum7QHS0PSI+X5WIzMysaXV2+asVWA4cAbyb7JLXBuBk4LDqh2Zm\nZs2mw5ZKRMwBkPTXwPsjYk9a/2fgydqEZ2ZmzaSSjvoBQN/c+ltSmZmZWRtddtQDs4CVkh4HBJwB\nzKxmUGZm1pw6TSqSBDwKLABOS8VXRsSL1Q7MzMyaT6dJJSJC0gMR8Q7g3hrFZGZmTaqSy18rJL0n\nIpZVPRqr2Mjp97dZ3zzr3DpFYma2TyVJ5TTgE5KeA14h61eJiPBoxWZm1kYlSeXsqkdhZmYHhC5v\nKY6I54D+wHnp1T+VmZmZtdFlS0XSFcAU4O5U9ENJsyPiu1WN7ADhvg8zO5hU8vDjZOC0iLgqIq4C\nxpIlmW6T1F/SPEnPSFon6b2SBkp6RNKG9D4gV3+GpI2S1ks6O1d+iqRVadsN6RZoMzOrk0qSioC9\nufW9qawnrgcejIjjgXcC64DpwMKIGA0sTOtIGgNMBE4AxgM3SuqVjnMTWYIbnV7jexiXmZn1QCUd\n9bcCSyT9NK1fANzS3RNK6kf2VP7/BIiIV4FXJU0AxqVqc4BFwJXABODOiNgNbJK0EThV0magb0Qs\nTse9PcW2oLuxmZlZz1Qyn8p1khYB709Fl0XEyh6ccxSwE7hV0jvJRkK+AhgSEdtSnReBIWl5GLA4\nt/+WVPZaWi4vfxNJU4GpAMccc0wPQjczs850eflL0jeBo4FbIuKGHiYUyBLZu4GbIuJdZM++TM9X\niIigk7lc9ldEzI6IlohoGTx4cFGHNTOzMpX0qTwLfBxolbRU0rXpUlV3bQG2RMSStD6PLMlslzQU\nIL3vSNu3AiNy+w9PZVvTcnm5mZnVSSXPqdwaEf8L+CDwQ+Di9N4taTDKFyQdl4rOBNYC84FJqWwS\n+8Yamw9MlHS4pFFkHfJL06WyXZLGpru+LsXjk5mZ1VUlz6ncDIwBtpNNznURsKKH5/0c8CNJh5G1\nhC4jS3BzJU0GngMuAYiINZLmkiWePcC0iCjdjXY5cBvQh6yD3p30ZmZ1VMndX0cDvYDfAb8BXirN\nAtldEfEU0NLOpjM7qH8NcE075a3AiT2JxczMilPJ3V8fA5D038nGAXtcUq+IGN75nmZmdrCp5PLX\nR4EPkD1b0h94DM9Rb2Zm7ajk8td4siRyfUT8qsrxmJlZE6vk8tdnaxGImZk1v0qeUzEzM6uIk4qZ\nmRWmw6QiaWF6/1btwjEzs2bWWZ/KUEnvA86XdCdlw91HRE8fgDQzswNMZ0nlKuAbZGNqXVe2LYAP\nVSsoMzNrTh0mlYiYB8yT9I2I+GYNYzIzsyZVyS3F35R0PtnDjwCLIuK+6oZlZmbNqJL5VP6ebBKt\ntel1haS/q3ZgZmbWfCp5ov5c4OSIeB1A0hxgJfDVagZmZmbNp9LnVPrnlvtVIxAzM2t+lbRU/h5Y\nKelxstuKz6Bs+l8zMzOorKP+DkmLgPekoivT7I1mZmZtVNJSIU3dO7/KsVgPjZx+/xvLm2edW8dI\nzOxg5bG/zMysME4qZmZWmE6TiqRekp6pVTBmZtbcOk0qEbEXWC/pmBrFY2ZmTaySjvoBwBpJS4FX\nSoURcX7VojIzs6ZUSVL5RtWjMDOzA0Ilz6k8IelPgNER8aikI4Fe1Q/NzMyaTSUDSk4B5gH/koqG\nAfdUMygzM2tOldxSPA04HdgFEBEbgP9WzaDMzKw5VZJUdkfEq6UVSb3JZn40MzNro5Kk8oSkrwJ9\nJH0YuAv4t+qGZWZmzaiSpDId2AmsAj4NPAB8vZpBmZlZc6rk7q/X08RcS8gue62PCF/+MjOzN+ky\nqUg6F/hn4Jdk86mMkvTpiFhQ7eDMzKy5VPLw47XAByNiI4CkPwXuB5xUzMysjUr6VF4uJZTkWeDl\nKsVjZmZNrMOWiqQL02KrpAeAuWR9KhcDy2oQm5mZNZnOWirnpdcRwHbgz4FxZHeC9enpidOw+isl\n3ZfWB0p6RNKG9D4gV3eGpI2S1ks6O1d+iqRVadsNktTTuMzMrPs6bKlExGVVPvcVwDqgb1qfDiyM\niFmSpqf1KyWNASYCJwBvAx6V9PY0LP9NwBSyO9MeAMbjvh4zs7qpZOyvUZKuk3S3pPmlV09OKmk4\ncC5wc654AjAnLc8BLsiV3xkRuyNiE7AROFXSUKBvRCxOtzjfntvHzMzqoJK7v+4BbiF7iv71gs77\nHeArwFtzZUMiYltafhEYkpaHAYtz9bakstfScnm5mZnVSSVJ5Y8RcUNRJ5T0UWBHRCyXNK69OhER\nkgp7wFLSVGAqwDHHeBJLM7NqqSSpXC/pauBhYHepMCJWdPOcpwPnSzqH7CaAvpJ+CGyXNDQitqVL\nWztS/a3AiNz+w1PZ1rRcXv4mETEbmA3Q0tLi0QDMzKqkkudU3kHWGT6L7EHIa4F/6O4JI2JGRAyP\niJFkHfCPRcQngfnApFRtEnBvWp4PTJR0uKRRwGhgabpUtkvS2HTX16W5fczMrA4qaalcDBybH/6+\nSmYBcyVNBp4DLgGIiDWS5gJrgT3AtHTnF8DlwG1ktzgvwHd+mZnVVSVJZTXQn32XowoTEYuARWn5\n18CZHdS7BrimnfJW4MSi4zIzs+6pJKn0B56RtIy2fSrnVy0qK8TI6fe/sbx51rl1jMTMDhaVJJWr\nqx6FmZkdECqZT+WJWgRiZmbNr5L5VF5m35z0hwGHAq9ERN+O9zIzs4NRJS2VN556T7fuTgDGVjMo\nMzNrTpU8p/KGyNwDnN1lZTMzO+hUcvnrwtzqIUAL8MeqRWRmZk2rkru/zsst7wE2k10CMzMza6OS\nPpVqz6tyQMk/G2JmdrDpbDrhqzrZLyLim1WIx8zMmlhnLZVX2ik7CpgMHA04qZiZWRudTSd8bWlZ\n0lvJpv+9DLiTbKRiMzOzNjrtU5E0EPjfwCfIpvh9d0T8thaBmZlZ8+msT+XbwIVkk1u9IyJ+X7Oo\nzMysKXX28OOXgLcBXwd+JWlXer0saVdtwjMzs2bSWZ/Kfj1tb2Zm5sRhZmaFcVIxM7PCOKmYmVlh\nnFTMzKwwTipmZlYYJxUzMyuMk4qZmRXGScXMzApTySRddoAon+tl86xz6xSJmR2o3FIxM7PCOKmY\nmVlhnFTMzKwwTipmZlYYJxUzMyuMk4qZmRXGScXMzArjpGJmZoVxUjEzs8LUPKlIGiHpcUlrJa2R\ndEUqHyjpEUkb0vuA3D4zJG2UtF7S2bnyUyStSttukKRafx4zM9unHi2VPcCXImIMMBaYJmkMMB1Y\nGBGjgYVpnbRtInACMB64UVKvdKybgCnA6PQaX8sPYmZmbdU8qUTEtohYkZZfBtYBw4AJwJxUbQ5w\nQVqeANwZEbsjYhOwEThV0lCgb0QsjogAbs/tY2ZmdVDXASUljQTeBSwBhkTEtrTpRWBIWh4GLM7t\ntiWVvZaWy8ttP+QHmfQAk2bWU3XrqJf0FuAnwBciYld+W2p5RIHnmiqpVVLrzp07izqsmZmVqUtS\nkXQoWUL5UUTcnYq3p0tapPcdqXwrMCK3+/BUtjUtl5e/SUTMjoiWiGgZPHhwcR/EzMzaqMfdXwJu\nAdZFxHW5TfOBSWl5EnBvrnyipMMljSLrkF+aLpXtkjQ2HfPS3D5mZlYH9ehTOR34FLBK0lOp7KvA\nLGCupMnAc8AlABGxRtJcYC3ZnWPTImJv2u9y4DagD7AgvczMrE5qnlQi4t+Bjp4nObODfa4Brmmn\nvBU4sbjozMysJ/xEvZmZFcZJxczMCuOkYmZmhXFSMTOzwjipmJlZYZxUzMysME4qZmZWGCcVMzMr\njJOKmZkVpq5D31vj8VD4ZtYTbqmYmVlhnFTMzKwwTipmZlYYJxUzMyuMk4qZmRXGScXMzArjpGJm\nZoVxUjEzs8I4qZiZWWH8RL11Kv+EPfgpezPrnJNKD3lYEzOzfXz5y8zMCuOkYmZmhXFSMTOzwrhP\nxfab+5HMrCNuqZiZWWGcVMzMrDBOKmZmVhgnFTMzK4yTipmZFcZ3f1mPeSgXMytxS8XMzArjlooV\nzi0Xs4OXWypmZlaYpm+pSBoPXA/0Am6OiFl1Dsna4afwzQ4OTZ1UJPUC/gn4MLAFWCZpfkSsrW9k\n1hVfIjM7MDV1UgFOBTZGxLMAku4EJgBVSSr+RVhdbs2YNb9mTyrDgBdy61uA0+oUixWsPMm0l9Q7\nq1NpYqrnHwtOpHagUUTUO4Zuk3QRMD4i/iqtfwo4LSI+W1ZvKjA1rR4HrN/PUw0CXuphuLXSTLFC\nc8XbTLGC462mZooVion3TyJicFeVmr2lshUYkVsfnsraiIjZwOzunkRSa0S0dHf/WmqmWKG54m2m\nWMHxVlMzxQq1jbfZbyleBoyWNErSYcBEYH6dYzIzO2g1dUslIvZI+izwENktxd+PiDV1DsvM7KDV\n1EkFICIeAB6o8mm6femsDpopVmiueJspVnC81dRMsUIN423qjnozM2sszd6nYmZmDcRJpROSxkta\nL2mjpOn1jqecpO9L2iFpda5soKRHJG1I7wPqGWOJpBGSHpe0VtIaSVek8kaN9whJSyU9neL9m1Te\nkPFCNsKEpJWS7kvrjRzrZkmrJD0lqTWVNXK8/SXNk/SMpHWS3tuI8Uo6Ln2npdcuSV+oZaxOKh3I\nDQHzEWAM8HFJY+ob1ZvcBowvK5sOLIyI0cDCtN4I9gBfiogxwFhgWvo+GzXe3cCHIuKdwMnAeElj\nadx4Aa4A1uXWGzlWgA9GxMm5W10bOd7rgQcj4njgnWTfc8PFGxHr03d6MnAK8Afgp9Qy1ojwq50X\n8F7godz6DGBGveNqJ86RwOrc+npgaFoeCqyvd4wdxH0v2ZhtDR8vcCSwgmy0hoaMl+wZrYXAh4D7\nGv1nAdgMDCora8h4gX7AJlIfdKPHm4vvfwD/UetY3VLpWHtDwAyrUyz7Y0hEbEvLLwJD6hlMeySN\nBN4FLKGB402Xk54CdgCPREQjx/sd4CvA67myRo0VIIBHJS1PI15A48Y7CtgJ3JouL94s6SgaN96S\nicAdablmsTqpHMAi+7OkoW7vk/QW4CfAFyJiV35bo8UbEXsju4wwHDhV0oll2xsiXkkfBXZExPKO\n6jRKrDnvT9/tR8guhZ6R39hg8fYG3g3cFBHvAl6h7PJRg8VLehj8fOCu8m3VjtVJpWMVDQHTgLZL\nGgqQ3nfUOZ43SDqULKH8KCLuTsUNG29JRPwOeJys/6oR4z0dOF/SZuBO4EOSfkhjxgpARGxN7zvI\nrvmfSuPGuwXYklqqAPPIkkyjxgtZsl4REdvTes1idVLpWLMOATMfmJSWJ5H1XdSdJAG3AOsi4rrc\npkaNd7Ck/mm5D1n/zzM0YLwRMSMihkfESLKf08ci4pM0YKwAko6S9NbSMtm1/9U0aLwR8SLwgqTj\nUtGZZNNrNGS8ycfZd+kLahlrvTuTGvkFnAP8Avgl8LV6x9NOfHcA24DXyP6amgwcTdZhuwF4FBhY\n7zhTrO8na3L/HHgqvc5p4HhPAlameFcDV6Xyhow3F/c49nXUN2SswLHA0+m1pvR/q1HjTbGdDLSm\nn4d7gAGNGi9wFPBroF+urGax+ol6MzMrjC9/mZlZYZxUzMysME4qZmZWGCcVMzMrjJOKmZkVxknF\nCicpJF2bW/+ypJkFHfs2SRcVcawuznNxGo328QrrL5JUsznLJY2T9L4qHfuB0jM6jU7STElfrncc\nto+TilXDbuBCSYPqHUiepP2Z6XQyMCUiPliteEr2M66SccB+JZVKzxMR50Q2ikBD6eb3ZDXmpGLV\nsIds+tIvlm8ob2lI+n16HyfpCUn3SnpW0ixJn0hzmqyS9Ke5w5wlqVXSL9K4V6XBH78taZmkn0v6\ndO64T0qaT/YUdHk8H0/HXy3pW6nsKrKHNW+R9O129rky7fO0pFm5TReneH8h6QOp7sh0/hXp9b6O\n4pJ0TxpgcU1ukMXSvD4r0vkWpgE5PwN8UdmcGR9IIwD8JH3+ZZJOT/vOlPQDSf8B/EDSCSnGp9L3\nNLqdz7dZ0qAU+zpJ30sxPZxGFyivf56kJcoGW3xU0psGK5R0v6ST0vLK9B0j6W8lTVHm2+nfYZWk\nv+zke/pa+o7/HTiu/FxWZ/V++tOvA+8F/B7oSza8eT/gy8DMtO024KJ83fQ+Dvgd2bDch5ONs/Y3\nadsVwHdy+z9I9gfRaLKRBI4ApgJfT3UOJ3v6eVQ67ivAqHbifBvwPDCYbNDAx4AL0rZFQEs7+3wE\n+L/AkWl9YK7+tWn5HODRtHwkcERaHg205j5vm7hyx+pD9hT/0Sm2F0r1cnVmAl/O7fuvZIM0AhxD\nNhxOqd5yoE9a/y7wibR8WKm87DNuBgaRTauwBzg5lc8FPtlO/QHsm5r8r0rfQ1md6cC09POwjDSt\nBNmYascBfwE8AvQiG0H3+fSz0OZ7IpsjZFX6XvsCG/Pfg1/1f7k5aVUREbsk3Q58HvivCndbFml4\nbkm/BB5O5auA/GWouRHxOrBB0rPA8WTjR52UawX1I/sl/iqwNCI2tXO+9wCLImJnOuePgDPIhuHo\nyFnArRHxh/Q5f5PbVhokcznZL2SAQ4F/lHQysBd4e65+eVyfl/SxtDwixT8Y+FmpXtn5yuMaI6m0\n3lfZiNAA8yOi9G/w/4CvSRoO3B0RGzr5rACbIuKpdj5X3nDgx8oGKjyMbO6Rck+S/SxsAu4HPizp\nSLJksV7SZ4A7ImIv2eCHT5D9++yi7ff0AeCnpe8/tWCsgTipWDV9h2xyq1tzZXtIl10lHUL2S6hk\nd2759dz667T9WS0fWygAAZ+LiIfyGySNI/tLtxZK8e5lX7xfBLaTzRZ4CPDHXP034kpxngW8NyL+\nIGkRWQusUocAYyMif3xSknnjPBHxr5KWAOcCD0j6dEQ8VsFnKn2uN13+Imv9XBcR89PnmNlOnWVA\nC/AsWYtkEDCFLFF1pVb/flYA96lY1aS/queSdXqXbCa7hAHZfA+HduPQF0s6JPWzHEs2q91DwF8r\nG14fSW9XNgJuZ5YCf576D3qRjez6RBf7PAJclv7KRtLALur3A7alltWnyC7vdFTvtymhHE825TLA\nYuAMSaPKzvcy8Nbc/g8DnyutpJbRm0g6Fng2Im4gG6n2pC7ir0Q/9k0LMam9ChHxKtllvIvJWktP\nkl0W/Vmq8iTwl6lvbDBZi3FpO4f6GXCBpD7KRjo+r4D4rUBOKlZt15L9VVryPbJf5E+TTdncnb9C\nnyf7hbMA+Ez66/xmso7cFZJWA/9CFy3xdKltOtl1/aeB5RHR6ZDgEfEg2TDircpmhezqdtYbgUnp\n8x5Px5/3QaC3pHXALLJkQro0NxW4Ox3jx6n+vwEfK3XUk11aakmd72vJOvLbcwmwOsV+InB7F/FX\nYiZwl6TlwEud1HuSbDKx/0rLw9M7ZHOq/Jzs3+Ex4CuRDTnfRkSsIPsOnib7919WQPxWII9SbGZm\nhXFLxczMCuOkYmZmhXFSMTOzwjipmJlZYZxUzMysME4qZmZWGCcVMzMrjJOKmZkV5v8DFznXa84X\n/HEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1476c4bd278>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.bar([item[0] for item in len_per_word.items()], [item[1] for item in len_per_word.items()])\n",
    "plt.xlabel(\"Number of characters in a word\")\n",
    "plt.ylabel(\"Number of words\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Average word length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1188,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7.7467313998552019"
      ]
     },
     "execution_count": 1188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.average([len(word) for word in vocabulary])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standard deviation of word length:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1190,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "3.4825802607693266"
      ]
     },
     "execution_count": 1190,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.std([len(word) for word in vocabulary])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1390,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "char_vocabulary = Counter([ch for recipe in recipes for ch in recipe])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1391,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Counter({'\\x00': 5,\n",
       "         '\\x02': 1,\n",
       "         '\\t': 300,\n",
       "         '\\n': 1126537,\n",
       "         '\\x0b': 1,\n",
       "         '\\x0c': 4,\n",
       "         '\\x14': 1165,\n",
       "         '\\x16': 1,\n",
       "         '\\x1d': 1,\n",
       "         ' ': 8798644,\n",
       "         '!': 3423,\n",
       "         '\"': 17401,\n",
       "         '#': 1789,\n",
       "         '$': 391,\n",
       "         '%': 3190,\n",
       "         '&': 14463,\n",
       "         \"'\": 16458,\n",
       "         '(': 41977,\n",
       "         ')': 41479,\n",
       "         '*': 18527,\n",
       "         '+': 9377,\n",
       "         ',': 319369,\n",
       "         '-': 494997,\n",
       "         '.': 395667,\n",
       "         '/': 134215,\n",
       "         '0': 65465,\n",
       "         '1': 317605,\n",
       "         '2': 184138,\n",
       "         '3': 85463,\n",
       "         '4': 94894,\n",
       "         '5': 53860,\n",
       "         '6': 32681,\n",
       "         '7': 13522,\n",
       "         '8': 28583,\n",
       "         '9': 26906,\n",
       "         ':': 148221,\n",
       "         ';': 55572,\n",
       "         '<': 1584,\n",
       "         '=': 14721,\n",
       "         '>': 2092,\n",
       "         '?': 488,\n",
       "         '@': 4391,\n",
       "         'A': 163433,\n",
       "         'B': 114324,\n",
       "         'C': 231292,\n",
       "         'D': 78070,\n",
       "         'E': 154820,\n",
       "         'F': 79106,\n",
       "         'G': 66430,\n",
       "         'H': 65526,\n",
       "         'I': 113068,\n",
       "         'J': 12580,\n",
       "         'K': 27943,\n",
       "         'L': 80582,\n",
       "         'M': 92479,\n",
       "         'N': 82691,\n",
       "         'O': 109690,\n",
       "         'P': 110484,\n",
       "         'Q': 3538,\n",
       "         'R': 125835,\n",
       "         'S': 244987,\n",
       "         'T': 155213,\n",
       "         'U': 46956,\n",
       "         'V': 26825,\n",
       "         'W': 46636,\n",
       "         'X': 4193,\n",
       "         'Y': 53421,\n",
       "         'Z': 4187,\n",
       "         '[': 2028,\n",
       "         '\\\\': 333,\n",
       "         ']': 2035,\n",
       "         '^': 261,\n",
       "         '_': 2138,\n",
       "         '`': 139,\n",
       "         'a': 1671233,\n",
       "         'b': 400736,\n",
       "         'c': 805895,\n",
       "         'd': 909035,\n",
       "         'e': 2738114,\n",
       "         'f': 342784,\n",
       "         'g': 561162,\n",
       "         'h': 763575,\n",
       "         'i': 1599291,\n",
       "         'j': 28210,\n",
       "         'k': 264501,\n",
       "         'l': 1101131,\n",
       "         'm': 517782,\n",
       "         'n': 1466660,\n",
       "         'o': 1636626,\n",
       "         'p': 565070,\n",
       "         'q': 22770,\n",
       "         'r': 1521732,\n",
       "         's': 1330941,\n",
       "         't': 1772918,\n",
       "         'u': 696299,\n",
       "         'v': 235612,\n",
       "         'w': 270179,\n",
       "         'x': 67472,\n",
       "         'y': 248044,\n",
       "         'z': 45889,\n",
       "         '{': 134,\n",
       "         '|': 373,\n",
       "         '}': 160,\n",
       "         '~': 4327,\n",
       "         '\\x7f': 27})"
      ]
     },
     "execution_count": 1391,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_vocabulary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Training the character language model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1310,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def split_dataset(recipes):\n",
    "    \"\"\"\n",
    "    Splits the dataset into 80% train, 10% validation and 10% test\n",
    "    \"\"\"\n",
    "    num_of_recipes = len(recipes)\n",
    "    permut_recipes = np.random.permutation(recipes)\n",
    "    train = permut_recipes[:int(num_of_recipes*0.8)]\n",
    "    valid = permut_recipes[int(num_of_recipes*0.8):int(num_of_recipes*0.9)]\n",
    "    test = permut_recipes[int(num_of_recipes*0.9):]\n",
    "    return train, valid, test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1311,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train, valid, test = split_dataset(recipes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1312,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "26370"
      ]
     },
     "execution_count": 1312,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1360,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      " \n",
      "      Title: BUTTERMILK BROWN SUGAR POUND CAKE\n",
      " Categories: Cakes\n",
      "      Yield: 12 Servings\n",
      " \n",
      "  2 1/2 c  All-purpose flour\n",
      "      1 ts Baking powder\n",
      "    1/2 ts Baking soda\n",
      "    1/8 ts Salt\n",
      "      1 c  Unsalted butter, at room\n",
      "           -temperature (2 sticks)\n",
      "      2 c  Firmly packed light brown\n",
      "           -sugar\n",
      "      3    Eggs\n",
      "      1 ts Vanilla\n",
      "    1/4 ts Anise extract\n",
      "      1 c  Buttermilk\n",
      "           Orange Marmalade (Optional)\n",
      "           Grated rind of one orange\n",
      "           -(Optional)\n",
      "    1/2 c  Plumped currants *\n",
      " \n",
      "  * NOTE:  To plump currants, cover with boiling water in a small cup\n",
      "  and let stand for 10 minutes.  Drain,\n",
      "  \n",
      "  Preheat the oven to Moderate (350F).  Grease a 10-inch tube pan. Line\n",
      "  the bottom with waxed paper.  Grease and dust with flour, tapping out\n",
      "  the excess.\n",
      "  \n",
      "  Stir together the flour, baking powder, baking soda, and salt in a\n",
      "  large bowl.  Beat the butter and brown sugar in another large bowl\n",
      "  for about 3 minutes, until light and fluffy, scraping down the sides\n",
      "  of the bowl as needed.\n",
      "  \n",
      "  Add the eggs, one at a time, beating for 1 minute after each\n",
      "  addition. Beat in the vanilla and anise extract. Reduce the mixer\n",
      "  speed to the lowest setting; add one-third of the flour mixture,\n",
      "  beating just until blended. Pour in 1/2 cup of the buttermilk,\n",
      "  beating until just combined. Repeat with the remaining flour mixture\n",
      "  and buttermilk, ending with the flour mixture. Fold in the orange\n",
      "  rind and the drained currants.\n",
      "  \n",
      "  Spoon the batter into the prepared pan.  Rotate the pan briskly to\n",
      "  even and settle the batter.  Bake for 60 to 65 minutes or until a\n",
      "  wooden pick inserted in the center comes out clean. Transfer the pan\n",
      "  to a rack and cool for 30 minutes.  Run a thin knife around the\n",
      "  inside of the pan and the tube.  Invert the cake onto the rack;\n",
      "  remove the waxed paper. Re-invert the cake. While still warm, brush\n",
      "  with orange marmalade and garnish with orange zest, if you wish. cool\n",
      "  to room temperature. Slice into 24 thin slices and serve\n",
      "        2    slices per serving. NOTE: The cake can be frozen airtight\n",
      "  for up to one month.\n",
      " \n",
      "\n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will build a model that is very similar to the word model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1353,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def train_char_lm_recipes(recipes, n=7):\n",
    "    \"\"\"\n",
    "    ngram model that counts the number of times the a chracter ch comes after n-1 chracters history\n",
    "    \"\"\"\n",
    "    lm = defaultdict(FreqDist)\n",
    "    for recipe in recipes:\n",
    "        pad = \"~\" * (n-1)\n",
    "        recipe = pad + recipe + \"~\"\n",
    "        for i in range(len(recipe)-n+1):\n",
    "            history, char = recipe[i:i+n-1], recipe[i+n-1]\n",
    "            lm[history][char]+=1\n",
    "    return lm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Character language model evaluation\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As before, for perplexity computation we need smoothing, otherwise the perplexity will be infinity. We will use the add 1 smoothing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1392,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def prob_MLE_addk_smoothing_chars(history, ch, n, model, k=1):\n",
    "    char_count = 0\n",
    "    history_count = 0\n",
    "    if n == 1:\n",
    "        char_count = model[ch]\n",
    "        history_count = model.N()\n",
    "    else:\n",
    "        if history in model:\n",
    "            history_count = model[history].N()\n",
    "            if ch in model[history]:\n",
    "                char_count = model[history][ch]\n",
    "            \n",
    "    V = len(char_vocabulary)\n",
    "    prob = (char_count + k)/(history_count + (V*k))\n",
    "    \n",
    "    return prob"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1397,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from math import log\n",
    "def entropy_chars(texts, n, model, prob_func, alpha=1):\n",
    "        \"\"\"\n",
    "        Calculate the approximate cross-entropy of the n-gram model for a\n",
    "        given evaluation texts.\n",
    "        This is the average log probability of each character in the text.\n",
    "\n",
    "        :param texts: texts for evaluation\n",
    "        :type sents: list(str)\n",
    "        :param n: the n of n-gram model\n",
    "        :param model: ngram model\n",
    "        :param prob_func: probability function to compute P(word|history)\n",
    "        :param alpha: hyperparameter\n",
    "        \"\"\"\n",
    "        \n",
    "        e = 0.0\n",
    "        N = 0\n",
    "        for text in texts:\n",
    "            pad = \"~\" *(n-1)\n",
    "            N += len(text) + 1 #include the end of sent tokens to the total count\n",
    "            text = pad + text + \"~\"\n",
    "            for i in range(len(text)-n+1):\n",
    "                history, ch = text[i:i+n-1], text[i+n-1]\n",
    "                prob = prob_func(history, ch, n, model, alpha)\n",
    "                e += log(prob, 2)\n",
    "        \n",
    "        return -e / N"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1438,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def perplexity_chars(texts, n, model, prob_func, alpha=1):\n",
    "    return 2**entropy_chars(texts, n, model, prob_func, alpha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will train our model on the trainig data for n=5,7,10,12 and measure the perplexity"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1399,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "char_models = {n:train_char_lm_recipes(train, n) for n in [5,7,10,12]}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1439,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "perplexities = []\n",
    "for n in [5,7,10,12]:\n",
    "    perplexities.append(perplexity_chars(valid, n, char_models[n], prob_MLE_addk_smoothing_chars, alpha=1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1441,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.text.Text at 0x149e8b37278>"
      ]
     },
     "execution_count": 1441,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8lOW9/vHPlyVAQsIaICxhlTXsAZe6VLFWrSu2VusC\nYqWe+qutnlakVVu1nmoXe6xtbakiyKHUFqQurRZFW2qrQkAMYUf2PYCQkEDI8v39MSOl1Kxk5pnl\ner9eeSWZPMxzGWGueZb7vs3dERGR5NUk6AAiIhIsFYGISJJTEYiIJDkVgYhIklMRiIgkORWBiEiS\nUxGIiCQ5FYGISJJTEYiIJLlmQQeoi44dO3qvXr2CjiEiEleWLl26z90za9suLoqgV69e5OXlBR1D\nRCSumNmWumynU0MiIklORSAikuRUBCIiSU5FICKS5FQEIiJJTkUgIpLkVAQiIklORSAiEoOOHKvk\ney+t5FBpecT3pSIQEYkxFZVVfG3O+8x8ZzPLtn0U8f2pCEREYoi7c/+LK3lj9R4evGII5w/oFPF9\nqghERGLIk29uYM7irXz10325+cxeUdlnxIrAzKab2V4zKzjhsS+Y2UozqzKz3EjtW0QkHv1+yTYe\nf30d40d141ufHRC1/UbyiGAGcPFJjxUA44FFEdyviEjceXPNHqbOX8G5/TN57JphmFnU9h2x2Ufd\nfZGZ9TrpsdVAVP8DRURi3fJtB7lj9vsMzsrglzeMonnT6J61j9lrBGY22czyzCyvsLAw6DgiIhGx\neV8Jk2YsoWN6CtMnjqF1i+ivDhCzReDu09w9191zMzNrXVdBRCTuFBaXcfP0xQA8N+l0MtNbBJIj\nZotARCSRlZRVMGnGEgqLy3hmQi69O6YFliUuVigTEUkk5ZVVfHX2MlbtKuI3N49mZHa7QPNE8vbR\nOcA7wAAz225mt5rZ1Wa2HTgT+JOZ/SVS+xcRiUXuzr3zVvC3dYU8clUOFwzsHHSkiN41dH01P5of\nqX2KiMS6nyxYx7xl2/nGhadx3djsoOMAukYgIhI1s97dws/f2sD1Y3vw9XGnBR3nOBWBiEgU/GXl\nbr77YgHjBnbi4StzYmo8lYpARCTC8jYf4M457zOse1ue/NJImkV5wFhtYiuNiEiC2bC3mFtn5tG1\nbSumTxxDakrs3aypIhARiZA9RUeZMH0JzZs2YeYtY2mflhJ0pE+kIhARiYCio+VMmL6Yg6XHmHHL\nGLI7pAYdqVqxd4wiIhLnjlVUcfuspWzYe5jpE8eQ061N0JFqpCIQEWlEVVXON//wAf/8cD+PXzuc\nc/vH/lxpOjUkItKIHn1tDS99sJN7Lh7A+FHdg45TJyoCEZFG8szbm5i2aCMTzuzJf53XN+g4daYi\nEBFpBK/k7+T7f1rFxUO68MDlQ2JqwFhtVAQiIqfonQ/3c/fzH5Dbsx3/e90ImjaJnxIAFYGIyClZ\ns7uIybPy6NkhladvHkPL5k2DjlRvKgIRkQbaefAIE6cvITWlKTMmjaVNavOgIzWIikBEpAEOlYYG\njJWUVTBz0li6tW0VdKQG0zgCEZF6OlpeyW3P5bFlfykzJo1hYJeMoCOdkkiuUDbdzPaaWcEJj7U3\ns9fNbH34c7Drs4mI1FNllXPX88tZvPkAP7l2OGf17Rh0pFMWyVNDM4CLT3rsXmChu58GLAx/LyIS\nF9ydh15eyasFu7nvc4O4fHjXoCM1iogVgbsvAg6c9PCVwMzw1zOBqyK1fxGRxvbrRRuZ+c4Wbjun\nN18+p0/QcRpNtC8Wd3b3XeGvdwPVrtpsZpPNLM/M8goLC6OTTkSkGi8s286jr67h8uFdmXrJoKDj\nNKrA7hpydwe8hp9Pc/dcd8/NzIz9SZtEJHH9fX0h98zN58w+HfjxF4bRJM4GjNUm2kWwx8yyAMKf\n90Z5/yIi9VKw4xC3z1pKv06t+fXNo2nRLP4GjNUm2kXwEjAh/PUE4MUo719EpM62HShl4rNLaJua\nwsxJY8loGZ8DxmoTydtH5wDvAAPMbLuZ3Qo8CnzGzNYDF4a/FxGJOQdKjjFh+mLKK6uYOWkMnTNa\nBh0pYiI2oMzdr6/mR+MitU8RkcZw5Fglt85cwo6DR5j95dPp1yk96EgRpZHFIiInqKis4mtzlrF8\n20GeumE0ub3aBx0p4jTXkIhImLtz/4sFvLF6Lw9dMYSLc7oEHSkqVAQiImFPvrmBOYu3ccf5fbnp\nzF5Bx4kaFYGICPD8kq08/vo6xo/qxjcvGhB0nKhSEYhI0ntzzR6+Pb+Ac/tn8tg1w+JqmcnGoCIQ\nkaS2fNtB7pj9PoOzMnjqhlE0b5p8L4vJ918sIhK2aV8Jk2YsoWN6CtMnjiGtRXLeSKkiEJGkVFhc\nxoTpiwF4btLpZKa3CDhRcFQEIpJ0SsoqmDRjCYXFZUyfOIbeHdOCjhSo5DwOEpGkVV5ZxX/NXsaq\nXUX85ubRjOjRNuhIgdMRgYgkDXfn3nkrWLSukEeuyuGCgdUuiZJUVAQikjR+smAd85Zt564L+3Pd\n2Oyg48QMFYGIJIVZ727h529t4PqxPbhzXL+g48QUFYGIJLzXCnbzwIsFXDioEw9fmZN0A8ZqoyIQ\nkYSWt/kAX//d+wzv3pYnrx9FsyQcMFYb/UZEJGFt2FvMrTPz6Nq2FdMnjqFVSuItM9kYAikCM/u6\nmRWY2Uoz+0YQGUQkse0pOsqE6Uto3rQJz00aS/u0lKAjxayoF4GZ5QC3AWOB4cBlZqYrNyLSaIqO\nljNh+mIOlh5jxi1j6NE+NehIMS2II4JBwHvuXuruFcDfgPEB5BCRBFRWUcnts5ayYe9hnrpxNDnd\n2gQdKeYFUQQFwDlm1sHMUoFLgR4B5BCRBFNV5XzrD/n888P9/PDzwzi3f2bQkeJC1KeYcPfVZvYY\nsAAoAZYDlSdvZ2aTgckA2dka+CEitXv0tTW89MFOplw8kPGjugcdJ24EcrHY3Z9x99Hufi7wEbDu\nE7aZ5u657p6bmalWF5GaPfP2JqYt2siEM3ty+3l9go4TVwKZdM7MOrn7XjPLJnR94IwgcohIYnj5\ng508/MoqLsnpwgOXD9GAsXoKavbReWbWASgH7nD3gwHlEJE4986H+/nv33/A2F7t+ekXR9C0iUqg\nvgIpAnc/J4j9ikhiWbO7iMmz8ujZIZXf3JxLy+YaMNYQGlksInFpx8EjTJi+mLSUZsyYNJY2qc2D\njhS3VAQiEncOlZYzcfpiSssqmTFpDN3atgo6UlzTCmUiEleOlldy23N5bNlfyoxJYxjYJSPoSHFP\nRSAicaOyyrnr+eUs3nyAJ68fyVl9OwYdKSHo1JCIxAV356GXV/JqwW7uv2wwlw/vGnSkhKEiEJG4\n8Ku/bWTmO1u47Zze3Hp276DjJBQVgYjEvBeWbeex19ZwxfCuTL1kUNBxEo6KQERi2t/XF3LP3HzO\n6tuBH31hGE00YKzRqQhEJGYV7DjE7bOW0q9Ta35102haNNOAsUhQEYhITNp2oJSJzy6hbWoKMyeN\nJaOlBoxFim4fFZGYc6DkGBOmL6a8sorfTT6dzhktg46U0HREICIx5cixSm6duYQdB4/wzIRc+nVK\nDzpSwlMRiEjMqKis4mtzlvHBtoM8cd1Icnu1DzpSUtCpIRGJCe7O/S8W8MbqvTx8VQ4X53QJOlLS\nqNMRgZn9xMyGRDqMiCSvny3cwJzF27jj/L7cdEbPoOMklbqeGloNTDOz98zsdjNrE8lQIpJcnl+y\nlZ++sY5rRnXnmxcNCDpO0qlTEbj70+7+KeBmoBeQb2a/NbPzG7JTM7vLzFaaWYGZzTEz3RIgkqTe\nXLOHb88v4Nz+mTx6zVAtMxmAOl8sNrOmwMDwxz7gA+BuM/tdfXZoZt2AO4Fcd88BmgLX1ec5RCQx\nLN92kDtmv8/grAyeumEUzZvq/pUg1OlisZn9FLgMeBP4H3dfHP7RY2a2toH7bWVm5UAqsLMBzyEi\ncWzTvhImzVhCZnoLpk8cQ1oL3bsSlLr+5vOB+9y95BN+NrY+O3T3HWb2Y2ArcARY4O4L6vMcIhLf\nCovLmDA99H5y5qSxZKa3CDhRcqvrcdiNJ5eAmS0EcPdD9dmhmbUDrgR6A12BNDO78RO2m2xmeWaW\nV1hYWJ9diEgMKymrYNKMJRQWlzF94hh6d0wLOlLSq7EIzKylmbUHOppZOzNrH/7oBXRr4D4vBDa5\ne6G7lwMvAGedvJG7T3P3XHfPzczMbOCuRCSWlFdW8V+zl7FqVxG/uGEkI3q0DTqSUPupoa8A3yD0\nzn3ZCY8XAT9v4D63AmeYWSqhU0PjgLwGPpeIxAl35955K1i0rpDHrhnKBQM7Bx1JwmosAnd/AnjC\nzL7m7k82xg7d/T0zm0uoWCqA94FpjfHcIhK7frJgHfOWbeeuC/vzxTHZQceRE9RYBGZ2gbu/Ceww\ns/En/9zdX2jITt39u8B3G/JnRST+zHp3Cz9/awPXj+3BneP6BR1HTlLbqaHzCN0yevkn/MwJnd8X\nEanWawW7eeDFAi4c1ImHr8zRgLEYVNupoe+GP98SnTgikkjyNh/g6797n+Hd2/Lk9aNopgFjMamu\nk87NOnF+ITPr+fHtoyIin2TD3mJunZlHt7atmD5xDK1StMxkrKprPb8NvGdml5rZbcDrwP9GLpaI\nxLM9RUeZMH0JzZs2YeaksbRPSwk6ktSgTiOL3f3XZrYSeIvQPEMj3X13RJOJSFwqOlrOhOmLOVh6\njOe/ciY92qcGHUlqUddTQzcB0wnNPjoD+LOZDY9gLhGJQ2UVldw+aykb9h7mVzeNJqebZqyPB3Wd\na+ga4Gx33wvMMbP5wExgRMSSiUhcqapyvvWHfP754X4ev3Y455ymGQHiRV1PDV110veLzaxek82J\nSGJ79LU1vPTBTqZcPJDxo7oHHUfqoa6nhvqb2UIzKwh/Pwy4J6LJRCRuPPP2JqYt2siEM3ty+3l9\ngo4j9VTXu4Z+A0wFygHcPR8tJiMiwMsf7OThV1ZxSU4XHrh8iAaMxaG6XiNIDZ8OOvGxigjkEZE4\nUVFZxdNvb+LxBesY26s9P/3iCJo2UQnEo7oWwT4z60toWgnM7PPAroilEpGYtnLnIabMy6dgRxGf\nHdKZH14znJbNNWAsXtW1CO4gNEPoQDPbAWwC/mMxGRFJbEfLK3li4XqmLdpI+7QUnrphFJcMzQo6\nlpyiut41tBG40MzSgCbuXhzZWCISa97buJ97X1jBpn0lXJvbne9cOpg2qc2DjiWNoLZpqO+u5nEA\n3P3xCGQSkRhSfLScR19dw+z3tpLdPpXZXz6dT/XrGHQsaUS1HRGkRyWFiMSkN1bt4b4/FrC3+Chf\nPrs3d1/Un9SUup5RlnhR2zTUDzb2Ds1sAPD8CQ/1AR5wd01iJxIj9h0u43svreSV/F0M7JLOr24a\nrfWFE1idqt3M+gBPAGcQunPoHeCu8LWDenH3tYSnpjCzpsAOYH59n0dEGp+7M//9HTz0yipKyyr5\n78/05yvn9SWlmdYRSGR1Pcb7LfAL4Orw99cBc4DTT3H/44AP3X3LKT6PiJyi7R+V8u35BSxaV8jo\nnu147Jqh9Ouks8PJoD4Dymad8P3/mdm3GmH/HxeKiASkssqZ9c5mfviXtRjw4BVDuOmMnjTR4LCk\nUdcieNXM7gV+R+jU0BcJTUXdHsDdD9R3x2aWAlxBaOqKT/r5ZGAyQHZ2dn2fXkTqYP2eYqbMy2fZ\n1oOc1z+TR67OoXs7rR+QbMzda9/IbFMNP3Z3r/csU2Z2JXCHu19U27a5ubmel5dX312ISDWOVVTx\n1F8/5BdvbSCtRVMeuHwwV43opnmCEoyZLXX33Nq2q/WIwMyaADe6+z8aJdm/XI9OC4lE3fJtB5ky\nN5+1e4q5YnhXHrh8MB1btwg6lgSo1iJw9yoz+zkwsrF2Gh6h/BngK431nCJSs9JjFfxkwTqe/ccm\nOqW35JkJuYwb1DnoWBID6nqNYKGZXQO84HU5l1QLdy8BOpzq84hI3by9fh9T5+ez7cARbjwjmykX\nDyS9paaHkJC6FsFXgLuBSjM7AhihawMZEUsmIqfsUGk53//TKv6wdDt9Oqbx+6+cydje7YOOJTGm\nrpPO6WZikTjz6opd3P/iSj4qPcZXP92XO8edpqmi5RPVdWSxATcAvd39YTPrAWS5++KIphORettT\ndJQHXizgLyv3kNMtg5mTxjCka5ugY0kMq+upoV8CVcAFwMPAYUIjjcdEKJeI1JO78/ySbTzy59Uc\nq6hi6iUDufXs3jRrqukhpGZ1LYLT3X2Umb0P4O4fhQeEiUgM2LyvhKkvrOCdjfs5o097fjB+GL07\npgUdS+JEXYugPDxB3MdLVWYSOkIQkQBVVFYx/R+bePz1dTRv0oQfjB/KF3N7aHoIqZe6FsHPCM0Q\n2snMHgE+D9wXsVQiUqtVO4uYMi+fFTsO8ZnBnXn4yhy6tGkZdCyJQ3W9a2i2mS0lNFuoAVe5++qI\nJhORT3S0vJIn31zPr/+2kbapzfnFl0Zx6dAumh5CGqy2pSpbArcD/YAVwK/dvSIawUTkPy3ZfIAp\n8/LZWFjC50d3577PDaJtqi7Xyamp7YhgJlAO/B24BBgEfCPSoUTk3xUfLeeHr61l1rtb6N6uFc9N\nGsu5/TODjiUJorYiGOzuQwHM7BlA4wZEouzNNXv4zvwCdhcdZdKnevPfF/UnrYXWDZbGU9vfpvKP\nv3D3Cp2DFIme/YfLeOiVVby4fCf9O7fmFzecxajsdkHHkgRUWxEMN7Oi8NcGtAp/r7mGRCLE3Xlx\n+U4efHklh8sq+MaFp/HVT/fTusESMTUWgbtrYhKRKNpx8Aj3zV/BW2sLGZndlseuGUb/zprqSyJL\nJxpFYkBVlfN/723hsVfXUOXwwGWDmXBWL5pqYJhEgYpAJGAb9h7m3nn55G35iHNO68j/XD2UHu21\nbrBETyBFYGZtgaeBHELTVkxy93eCyCISlPLKKn79tw/52cINtEppyk++MJzxo7RusERfUEcETwCv\nufvnw5PX6e2PJJX87Qe5Z24+a3YX87lhWXzv8iFkpmvdYAlG1IvAzNoA5wITAdz9GHAs2jlEgnDk\nWCWPv76WZ97eRGZ6C35zcy6fGax1gyVYQRwR9AYKgWfNbDiwFPh6eB1jkYT1zw37uPeFFWw9UMqX\nTs/m3ksGkqF1gyUGBHFjcjNgFPCUu48ESoB7T97IzCabWZ6Z5RUWFkY7o0ijOXSknClz8/nS0+/R\nxOB3k8/gf64eqhKQmBHEEcF2YLu7vxf+fi6fUATuPg2YBpCbm+vRiyfSeF4rCK0bfKDkGLef15dv\nXKh1gyX2RL0I3H23mW0zswHuvpbQ1Narop1DJJL2Fh/luy+u5NWC3QzOyuDZiWPI6aZ1gyU2BXXX\n0NeA2eE7hjYCtwSUQ6RRuTt/yNvO9/+0iqMVVdxz8QBuO6cPzbVusMSwQIrA3ZcDuUHsWyRStu4v\nZer8fP6xYT9je7fn0fFD6ZPZOuhYIrXSyGKRU1RZ5Tz7j038eMFamjVpwvevyuFLY7O1brDEDRWB\nyClYs7uIKXPz+WD7IcYN7MT3r84hq02roGOJ1IuKQKQByioq+cWbG/jlXz+kTavm/Oz6kVw+LEvT\nQ0hcUhGI1NPSLQeYMm8FG/YeZvzIbtx/2WDapWndYIlfKgKROjpcVsGPXlvDc+9uoWubVsy4ZQyf\nHtAp6Fgip0xFIFIHf127l+/ML2DnoSNMOLMX3/zsAFpr3WBJEPqbLFKDAyXHePiVVcx/fwf9OrVm\n7u1nMbqn1g2WxKIiEPkE7s7L+bt48KWVHDpSzp3jTuOO8/vSopmmh5DEoyIQOcmuQ0e4b34BC9fs\nZXiPtsy+ZigDu2QEHUskYlQEImFVVc7sxVt57NU1VFY5931uELd8qrfWDZaEpyIQATYWHubeeStY\nvPkAZ/cLrRuc3UEL50lyUBFIUiuvrGLaoo08sXA9LZs14YefH8YXRnfXwDBJKioCSVorth9iyrx8\nVu0q4tKhXfjeFUPolN4y6FgiUacikKRztLySn76xjqf/vokOaSn86sbRXJzTJehYIoFREUhSeefD\n/Ux9IZ/N+0u5bkwPpl46iDattGSkJDcVgSSFoqPl/ODPa5izeCvZ7VP57ZdP56x+HYOOJRITAikC\nM9sMFAOVQIW7a5EaiZgFK3dz/4sFFBaXMfncPtx1YX9apWhgmMjHgjwiON/d9wW4f0lwhcVlfO+l\nlfxpxS4GdknnNzfnMqx726BjicQcnRqShOPuzFu2g4dfWcWRY5V867MDmHyu1g0WqU5QReDAG2ZW\nCfza3acFlEMSzLYDpXx7/gr+vn4fY3q14wfjh9Gvk9YNFqlJUEVwtrvvMLNOwOtmtsbdF524gZlN\nBiYDZGdnB5FR4khllTPzn5v50V/W0sTg4SuHcMPpPbVusEgdBFIE7r4j/Hmvmc0HxgKLTtpmGjAN\nIDc316MeUuLGuj3F3DM3n+XbDnL+gEy+f/VQurXVusEidRX1IjCzNKCJuxeHv74IeCjaOST+lVVU\n8su3PuSXf91AesvmPHHdCK4Y3lXTQ4jUUxBHBJ2B+eF/rM2A37r7awHkkDi2bOtHTJmbz/q9h7lq\nRFfuv2wwHVq3CDqWSFyKehG4+0ZgeLT3K4mhpKyCHy9Yy4x/biYroyXPThzD+QO1brDIqdDtoxI3\nFq0rZOoLK9hx8Ag3n9mTey4eqHWDRRqB/hVJzDtYeoyHX1nNvGXb6ZuZxtzbzyS3V/ugY4kkDBWB\nxCx3508rdvG9l1ZysLSc/3d+P/7fBf1o2VzTQ4g0JhWBxKTdh45y3x8LeGP1HoZ1b8Nzk05ncFet\nGywSCSoCiSlVVc7vlmzjB39eTXlVFd+5dBC3fKoXzTQ9hEjEqAgkZmzaV8K98/J5b9MBzuzTgUev\nGUrPDmlBxxJJeCoCCVxFZRVPv72Jn76+jpRmTXjsmqFcm9tDA8NEokRFIIFauTO0bnDBjiI+O6Qz\nD12ZQ+cMrRssEk0qAgnE0fJKnli4nmmLNtIuNYWnbhjFJUOzgo4lkpRUBBJ1723cz9QXVrBxXwnX\n5nbnO5cOpk2q1g0WCYqKQKKm+Gg5j766htnvbaVH+1b8362nc/ZpWjdYJGgqAomKN1bt4b4/FrC3\n+ChfPrs3d1/Un9QU/fUTiQX6lygRte9wGQ++vIqXP9jJwC7p/Oqm0YzooXWDRWKJikAiwt2Z//4O\nHnplFaVlldz9mf7cfl5fUpppYJhIrFERSKPb/lEp355fwKJ1hYzu2Y5Hxw/ltM7pQccSkWqoCKTR\nVFY5s97ZzA//shaAB68Ywk1naN1gkVgXWBGYWVMgD9jh7pcFlUMax/o9xUyZl8+yrQc5r38mj1yd\nQ/d2qUHHEpE6CPKI4OvAakBTSsaxYxVV/OpvH/LzNzeQ1qIpP/3icK4a0U3TQ4jEkUCKwMy6A58D\nHgHuDiKDnLrl2w4yZW4+a/cUc/nwrnz38sF01LrBInEnqCOC/wXuAXQFMU5UVFaxaV8Jq3YVsXpX\nMat2FfH2+kI6pbfkmQm5jBvUOeiIItJAUS8CM7sM2OvuS83s0zVsNxmYDJCdnR2ldAJw6Eg5a3YV\nsTr8or96dxFrdxdTVlEFQPOmRt/M1tx6dm/uHHca6S01PYRIPDN3j+4OzX4A3ARUAC0JXSN4wd1v\nrO7P5Obmel5eXpQSJo+qKmfrgdLwC34Rq3YVs3pXETsOHjm+Tfu0FAZlpTOoSwaDskIf/Tq11ngA\nkThgZkvdPbe27aJ+RODuU4GpAOEjgm/WVALSOErKKlizu/j4i/7qXaF3+SXHKgFoYtAnszWjerbj\nhjOyGZSVweCsDDqlt9CFX5EEp3EECcbd2XHwSOiUzgkv+lsOlPLxwV96y2YMysrgC7k9Qu/2szLo\n3zldi8KLJKlAi8Dd/wr8NcgM8exoeSXr9hQfP5e/alcRa3YVUXS04vg2PTukMqhLBleP7H78Rb97\nu1Z6ly8ix+mIIA64O4XFZazaVXT8rp3Vu4rYtK+EyqrQ2/zUlKYM6JLOZcO7hk/rpDOgSwatW+h/\nsYjUTK8SMeZYRRUfFh4+4bRO6EV/f8mx49t0a9uKQVnpXJLT5fgF3J7tUzWVg4g0iIogQAdKjp1w\nx07oRX/D3mLKK0Pv8lOaNWFA53TGDep0/AV/UJcMreYlIo1KRRAFlVXOpn2Hj9+e+fHHnqKy49t0\nSm/BoKwMzuufyaCsdAZnZdC7YxrNmuo2TRGJLBVBIys6Ws7qndUPxmrWxOjXqTVn9e14/OLtoKwM\nTc0gIoFRETRQXQZjtUttzqCsDG48o2f4BT+dfp1a06KZbtMUkdihIqiD0mMnD8YqZs2uon8bjNW7\nYxojs9vypdOzGRx+l985Q4OxRCT2qQhO4O7sPHT0X6d2dode9DfvL/nXYKwWocFYnx/d/fhpnf6d\n02mVonf5IhKfkrYIjpZXsn7P4RPu2Clize5iDh0pP75NdvtUBmdlcNWIbhqMJSIJK+GL4MTBWCdO\nu7DxhMFYrZqHBmNdOjSLweEX/AFd0jWrpogkhYQugp8tXM/Mf27+t8FYXdu0ZFBWBp8d0uX4Bdye\nHdJoqsFYIpKkEroIumS05IKBJwzGykqnbWpK0LFERGJKQhfBtWN6cO2YHkHHEBGJaRq2KiKS5FQE\nIiJJTkUgIpLkVAQiIkku6kVgZi3NbLGZfWBmK83swWhnEBGRfwnirqEy4AJ3P2xmzYG3zexVd383\ngCwiIkkv6kXg7g4cDn/bPPzh0c4hIiIhgVwjMLOmZrYc2Au87u7vfcI2k80sz8zyCgsLox9SRCRJ\nmHtwb8bNrC0wH/iauxfUsF0hsKWBu+kI7Gvgnw1CPOWNp6wQX3njKSvEV954ygqnlrenu2fWtlGg\nI4vd/aCZvQVcDFRbBHX5D6mOmeW5e25D/3y0xVPeeMoK8ZU3nrJCfOWNp6wQnbxB3DWUGT4SwMxa\nAZ8B1kQ7h4iIhARxRJAFzDSzpoSK6Pfu/koAOUREhGDuGsoHRkZxl9OiuK/GEE954ykrxFfeeMoK\n8ZU3nrJa9TPOAAAEKUlEQVRCFPIGerFYRESCpykmRESSXEIXgZltNrMVZrbczPKCzlMTM2trZnPN\nbI2ZrTazM4POVB0zGxD+nX78UWRm3wg6V3XM7K7wdCYFZjbHzFoGnakmZvb1cNaVsfZ7NbPpZrbX\nzApOeKy9mb1uZuvDn9sFmfFE1eT9Qvh3W2VmMXP3UDVZfxR+Tcg3s/kf32jT2BK6CMLOd/cRcXC7\n2BPAa+4+EBgOrA44T7XcfW34dzoCGA2UEhoPEnPMrBtwJ5Dr7jlAU+C6YFNVz8xygNuAsYT+Hlxm\nZv2CTfVvZhC63ftE9wIL3f00YGH4+1gxg//MWwCMBxZFPU3NZvCfWV8Hctx9GLAOmBqJHSdDEcQ8\nM2sDnAs8A+Dux9z9YLCp6mwc8KG7N3TAXzQ0A1qZWTMgFdgZcJ6aDALec/dSd68A/kboRSsmuPsi\n4MBJD18JzAx/PRO4KqqhavBJed19tbuvDShStarJuiD89wDgXaB7JPad6EXgwBtmttTMJgcdpga9\ngULgWTN738yeNrO0oEPV0XXAnKBDVMfddwA/BrYCu4BD7r4g2FQ1KgDOMbMOZpYKXArE+nqrnd19\nV/jr3UDnIMMksEnAq5F44kQvgrPDpy8uAe4ws3ODDlSNZsAo4Cl3HwmUEFuH15/IzFKAK4A/BJ2l\nOuHz1VcSKtuuQJqZ3Rhsquq5+2rgMWAB8BqwHKgMNFQ9hCeV1K2IjczMvgNUALMj8fwJXQThd4O4\n+15C57DHBpuoWtuB7SdMvjeXUDHEukuAZe6+J+ggNbgQ2OTuhe5eDrwAnBVwphq5+zPuPtrdzwU+\nInRuOJbtMbMsgPDnvQHnSShmNhG4DLjBI3S/f8IWgZmlmVn6x18DF1HDfEZBcvfdwDYzGxB+aByw\nKsBIdXU9MXxaKGwrcIaZpZqZEfrdxuyFeAAz6xT+nE3o+sBvg01Uq5eACeGvJwAvBpgloZjZxcA9\nwBXuXhqx/STqgDIz68O/7mRpBvzW3R8JMFKNzGwE8DSQAmwEbnH3j4JNVb1wuW4F+rj7oaDz1CS8\nCt4XCR1avw982d3Lgk1VPTP7O9ABKAfudveFAUc6zszmAJ8mNCPmHuC7wB+B3wPZhGYJvtbdT76g\nHIhq8h4AngQygYPAcnf/bFAZP1ZN1qlAC2B/eLN33f32Rt93ohaBiIjUTcKeGhIRkbpREYiIJDkV\ngYhIklMRiIgkORWBiEiSUxGIiCQ5FYGISJJTEYg0gJn1Cq8b8Zvw3PYLzKxV0LlEGkJFINJwpwG/\ncPchhEaoXhNwHpEGURGINNwmd18e/nop0CvALCINpiIQabgT5yuqJDSnlUjcURGIiCQ5FYGISJLT\n7KMiIklORwQiIklORSAikuRUBCIiSU5FICKS5FQEIiJJTkUgIpLkVAQiIklORSAikuT+P6S2quy6\n+CJOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x149e8b1c198>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot([5,7,10,12], perplexities)\n",
    "plt.xlabel(\"n\")\n",
    "plt.ylabel(\"Perplexity\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As can be seen by the plot, the perplexity grows as n grows. Again, like with the word model, I suspect this is because the smoothing underestimates P(char|history) when n grows (the length of history grows). More characters were not seen in the train dataset in the context of some history."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As perplexity is not helpful here, we will test our model by generating recipes for different n."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1442,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generate_recipe(model, seed, n, num_of_chars=1000):\n",
    "    '''\n",
    "    generates a text given a seed using MLE probability\n",
    "    '''\n",
    "    history = seed\n",
    "    out = history\n",
    "  \n",
    "    ch = MLEProbDist(model[history]).generate()\n",
    "    i = 0\n",
    "    while ch != \"~\" or i<num_of_chars: #keep generating words until an \"end of recipe\" mark\n",
    "        out += ch\n",
    "        history = history[-n+2:] + ch\n",
    "        ch = MLEProbDist(model[history]).generate()\n",
    "        i += 1\n",
    "    return out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1433,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~\n",
      " \n",
      "  Cold Reductionalkalike notic\n",
      "     Title: FIR: 233 casses, oranges\n",
      "            1 tb Buttermission\n",
      "  oils. Carrots; brush the Dean's Soy saucepan\n",
      "  celerial, cornstand dill, and\n",
      "           -cubed, formulate flour. OR 15 minutes on each lettuce into and be chopped\n",
      "    1    2 ts Black skillet over med ingreased in a littles, 1 c  Butter\n",
      "  until give oil and with fruity (from head carbohydrate.\n",
      "    2 c  Brown\n",
      "       1 tb Worcested\n",
      "     1 1/2 c  Chopped and well. Skimming accompletely cracker cream rum\n",
      "  :    2/3 c  Water until egg sugar, salt. Put two bit margarine Yeasts up to 375F until\n",
      "         15 minutes.  Sliced\n",
      "  1/4 c  Chop thinner, grapefruits\n",
      "     2 c  Chopped\n",
      "       Title: FARINADE HEN\n",
      " Categories: Meats, or made (Nation\n",
      "     Sprinkle slightly for $12.9g (3%)\n",
      "         10 minutes turn the\n",
      "  TXFT40A@Prodigy, Inc. Electric blender. On high 6\n",
      "  heat oiler top. Margarinator\n",
      "    Yield: 18 g CHOCOLATE-ECLAIR. SPINACHED PANCAKE\n",
      " Categories on packer can hot with a\n",
      "  white, Continue cotta\n",
      " \n",
      "  1/2 c  Fine ground baking fruit up to 1 hour.\n",
      "  \n",
      "  Combine all-purpose flour\n",
      "    3/4-inch fruits: Meanwhiles in wedge from the dip again\n",
      "  cover fish oregano,\n",
      "  Beat until thicken too small very into the 1965.\n",
      "  \n",
      "  SEPARAGUS BREAD (EMERICAN SALAD-\n",
      "      -salt. Red with oil onion; chocolates. Pulse ash in hot reduce logs from heat buttermint; plain,drain warm place they minutes. Set aside. It is knob chopped onion and gelating: Pressing: Califormat butter\n",
      "     2 TBS cinnamon juice\n",
      "  in wok in\n",
      "     8 ea Cheese\n",
      "  stirring to cooking\n",
      "  reserve HOT\" CHAO WATER COLD WORCEMEAT\n",
      " Categories: Salads\" cookies or rice, Soups/stew or pound corn\n",
      "      1 c  Brown or margarinator broiler, melted 400 degreen Approx. 20 to picnic), or lined all skillet\n",
      "  remaining it, Aug 1996\n",
      "  Tasted onion, compote is alls. Loosen.  Recipes until hot over will ingreast\n",
      "     8 oz)\n",
      "    2 tb Butter\n",
      "     1/2 c  Dressing the lean.) Press uncover.\n",
      "  :       1/2 c  Cake-along sugar\n",
      "  minutes; the coriander.\n",
      " \n",
      "\n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "n = 5\n",
    "generated_recipe = generate_recipe(char_models[n], \"~\"*(n-1), n)\n",
    "print(generated_recipe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1402,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "~~~~~~\n",
      " \n",
      "        Shredded cheese, sugar, and cook on high heat 5 minutes more. On\n",
      "  the hot water together use. Then push braid\n",
      "       Salt & peppers magazine, December, Dill, fresh rosemary\n",
      "       1    Shred the wine\n",
      "  then rolls at Home_.\n",
      "  \n",
      "  Serve piping\n",
      " \n",
      "  PREPARE ICING, in small pieces.\n",
      "  \n",
      "  Source: \"Great Good\"\n",
      "  Submitted\n",
      "  1 1/2 c  Unsalted butter or Mondadori Editor's favorite Recipe made in small butter\n",
      "       8 tb Sugar\n",
      "       or until smooth.\n",
      "  \n",
      "  *  The next day.  It is\n",
      "  tender. Remove from: * Oregon\n",
      "  Hazelnut Marketing Board\n",
      "      2 lg Onion; chopped crepe pan. Place in a saucepan and heat sauce\n",
      "    1/4 cup sugar and yeast into 8 servings\n",
      " \n",
      "       1    Eggs, well and add water; bake for the vanilla. Add apple as a medium\n",
      "        2 x  Sm Sweet pepper\n",
      "    1. Place chickened. Finely chopped fresh into julienne straw mushrooms and refrigerate in blended. Add spices\n",
      "        1    Jalapeno\n",
      "  in ham cut into 8 rectangle.\n",
      "  \n",
      "  Oven 5\n",
      "  minutes. Serve with an electric mixer till cup. Be\n",
      "  casserole and bake at 350 degrees. Spread the wild rice. Return to cover and bake for about 3\n",
      "          Title: CRYSTAL ALMO\"\n",
      "      3 c  Egg yolks until complete part of the\n",
      "  pottery gallon string. Fold\n",
      "  baking\n",
      "  the juice\n",
      "      Yield: 4 Servings\n",
      " \n",
      "       4 lg clove; minced jalapeno pepper towels, then pour into your\n",
      "      Yield: 8 Serving. 9. Serves 12-20.\n",
      "  \n",
      "  ^^^^^^^^\n",
      "      4 tb Olive oil\n",
      "      2 tb Vegetables in a plate or till smooth and some\n",
      "  diameter x 2 1/2-inch\n",
      "  spread when clean,\n",
      "  mustard mixture and\n",
      "  bring the skin and seasonings and milk and pour into dry onions) 95 cal; 10 g fat; or kosher, Cakes, Cocktail. Rinse pods, walnut Marketing Board, about 4\" in\n",
      "  pan. Add bacon, onion, finely chop\n",
      " \n",
      "  1. Mix with the\n",
      "      4    Dressing ingredients. Heat 2 minutes.\n",
      "  \n",
      "  Serve hot, cover and blended.\n",
      " \n",
      "\n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "n = 7\n",
    "generated_recipe = generate_recipe(char_models[n], \"~\"*(n-1), n)\n",
    "print(generated_recipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With n=5,7 the recipe doesn't look good. The structure is not as a recipe structure, the instructions come before the ingrediants, the sentences are messy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1413,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "~~~~~~~~~\n",
      " \n",
      "      Title: HOLIDAY OATMEAL COOKIES\n",
      " Categories: Breads2, Brdmachine\n",
      "      2 tb Lemon Juices & Lemon Rind; Grated\n",
      "      1 c  Water\n",
      "             -chiles\n",
      "  cascabel 18 peppercorns, allspice. Warm\n",
      "  tortilla flour or wheat germ and powdered\n",
      " \n",
      "  Rub roast them lightly overlapping. Almond paste and corn syrup or honey.  Pour over the top is a nice brown\n",
      "  sugar would\n",
      "  other seasoning and pepper -- seeded and chopped\n",
      "       margarine\n",
      "      1 tb Wine vinegar\n",
      "     10 oz Grapefruit Halves\n",
      "         -dish 6 layers. Bake for 35 to 45 minutes. Cool\n",
      "  slightly closed jar. Use as much color of rice papers with fingers, to\n",
      "  make a little dollop of whipped cream mixture. Cook over medium\n",
      "  head. EYE PUFFINESS: Teabags, slightly.\n",
      "  Place peas, oil, beaten egg whites until meat thermometer. Stir chicken.\n",
      "  \n",
      "  You can use catsup, tomato paste.\n",
      "  \n",
      "  BLINTZES\n",
      " Categories: Vegetables are tender, brushing with a cover. Freeze 1/2 cup apple juice concentrate (e.g.\n",
      "  :       Georgetown, Delaware, 19947-9622\n",
      " \n",
      "\n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "n = 10\n",
    "generated_recipe = generate_recipe(char_models[n], \"~\"*(n-1), n)\n",
    "print(generated_recipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For n=10 it starts nicely, with title, categories and ingredients. After the instructions it seems that a new recipe description is created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1416,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "~~~~~~~~~~~\n",
      " \n",
      "      Title: CORNSTICKS\n",
      " Categories: Meats\n",
      "      Yield: 1 Servings\n",
      " \n",
      "      1 tb Water\n",
      "      1    Egg\n",
      "     20    minutes.\n",
      "  \n",
      "  Add potatoes (about 2 minutes or until toothpick inserted in center comes out clean, about 300 degrees for 16 minutes, or until\n",
      "  it registers 160 degrees; otherwise,\n",
      "  cut into bite-sized pieces. Can use\n",
      "  leftovers here.\n",
      "  \n",
      "  *  All ingredients and process until smooth, thick frosting. Lay on plate\n",
      "  and ground beef chuck eye steaks\n",
      "      1 lb FRESH GREEN CHILE CHICKEN WITH SALT,\n",
      "  ROSEMARY - Used in fruit sugar.\n",
      "  Add 1/4 C Grand Marnier. Roll\n",
      "  into a measuring spoons, measuring\n",
      "  cup) and stir to coat well.\n",
      "    In medium bowl, with an electric mixer beat whites and water. Chill.  Serve hot.\n",
      " \n",
      "\n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "n=12\n",
    "generated_recipe = generate_recipe(char_models[n], \"~\"*(n-1), n)\n",
    "print(generated_recipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For n=12, this recipe looks like a real one. Let's generate some more."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1417,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "~~~~~~~~~~~\n",
      " \n",
      "      Title: CLASSIC FRENCH-STYLE BAKED CHICKEN CASSEROLE\n",
      " Categories: Cheese/eggs, Vegetables\n",
      "      Yield: 16 Servings\n",
      " \n",
      "     80 g  Plain bread crumbs\n",
      "    1/3 c  Curacao\n",
      " \n",
      "  Blend 1/2 minutes.\n",
      "  \n",
      "  Set a wok over high heat,\n",
      "  whisking constantly, for thirty days straight'.\"\n",
      "  \n",
      "  Storm has since honed her culinary purposes.  In common usage, the fruits with seeds of our choice;\n",
      "  sesame oil\n",
      "    1/2 c  Honey\n",
      "            White angel food cake pan to a\n",
      "  depth of 1/4\". It should rise to double, about 45 minutes.Transfer to a small bowl combine beef, pasta, artichoke leaves. In\n",
      "  a small cloth towel\n",
      "  to drain. Sprinkle grated cheese,\n",
      "             Cornstarch on a plate, pick an old one I have had the tips off the leaves finely. Heat half the bacon mixture, lifting\n",
      "  pan with butter, about 5 minutes.\n",
      "  \n",
      "  Glaze:  In saucepan, and put through\n",
      "  a large bowl until a rich brown solid pudding. Amaretto is also good, though not in use covered).\n",
      "  Spread rolled dough.\n",
      "  Set aside.\n",
      "  \n",
      "  TORTILLA CUPS: To make\n",
      "  gnocchi, pour flour onto a plate covered with plastic wrap;\n",
      "  refrigerator. 3.Whisk the eggs and vanilla.\n",
      "  \n",
      "  (I always find\n",
      "  contradictory as it may be necessary to create swirls with a tablespoon soy sauce\n",
      "      Yield: 2 Quarts\n",
      " \n",
      "      2 c  Minature marshmallows -- cut in pieces\n",
      "      2 ts Oregano\n",
      "    1/4 ts Ground nutmeg\n",
      "    1/4 ts EA:\n",
      "        1 ts Salt\n",
      "    1/2 ts Salt\n",
      "      2 tb Catsup\n",
      "    1/4 c  Sugar\n",
      "      4 tb Oil\n",
      "      2 ts Rice wine vinegar\n",
      "    1/3 c  Golden raisins\n",
      "      1/2 c  Barley\n",
      "     10 oz Cheddar;reduced fat\n",
      "      3 tb Finely chopped. Posted to rec.food.veg\n",
      " \n",
      "\n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "n=12\n",
    "generated_recipe = generate_recipe(char_models[n], \"~\"*(n-1), n)\n",
    "print(generated_recipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, the begining of the recipe looks nice. This is a very time consuming recipe since you have to \"whisk constanly, for thirty days straight\" :)\n",
    "\n",
    "The problem ia that a new recipe starts after the first one. We will change the generate function to stop when a stop token in encountered or when we reached a 1000 characters (the average length of a recipe)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1427,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~~~~~~~~\n",
      " \n",
      "      Title: BAKED ONIONS WITH CHEMICALS Although this torta is from Provence, its superb\n",
      "  blending into the container, combine\n",
      "      8 oz Self-raising flour\n",
      "      2 tb Butter or margarine and milk into egg halves. Put a small saucepan and bring to a boil over medium heat for 25 minutes. Place\n",
      "  rolls, point side down\n",
      "  on it with a grapefruit\n",
      "    1/2 c  Bisquick Baking Mix; Bisquick\n",
      "      1    Teaspoo salt\n",
      " \n",
      "   \"Chile pastes are very tender; remove pans. Cool completely melted and mixture into large bowl, combine\n",
      "  the tofu. Place the\n",
      "  sauce has simmered for 10 minutes.  Add the broccoli and mushroom sp\n",
      "    1/4 ts Salt\n",
      "      2 tb Thyme; minced\n",
      "      1 pt Canned corn; drained\n",
      "     In skillet, transfer to a blender and coarse salt. Set aside. TO PREPARE DIP: In small skillet over low heat until butter melts\n",
      "  and the peppers in large bowl, tossing just\n",
      "  to blend. Drizzle liberally over the countertop to room temperature before using. To blanch, cover with a damp cloth, being careful to separate the eggs whole until light brown.\n",
      "  \n",
      "  Tammie K Nakamura <nakamurt@UCSU.COLORADO.EDU>\n",
      "  \n",
      "  Individual ramekins three-quarters full.\n",
      "  In the remaining ingredients. On\n",
      "  a bread machine according to\n",
      "  dissolved. Leave to set, it\n",
      "  may be added if\n",
      "  necessary to prevent scorching.\n",
      "  \n",
      "  Select a large mixing bowl and sprinkle with 1 cup of the halibut into 1-inch chunks. Dredge the breastbone to\n",
      "  separated. That is how it is meant to remaining butter,\n",
      "           -crushed\n",
      "      1 cn SPAM Luncheon Meat (12 oz)\n",
      "      1    Red bell pepper, mixing until smooth. Add\n",
      "  chicken in 350 degree oven for 25 minutes, spray with\n",
      "  cooking for 20-to-30\n",
      "  minutes to reduce the heat.\n",
      "  \n",
      "  Whisk the egg whites to room temperatures. I have tried chopping veges.\n",
      "  \n",
      "  Per serving (excluding unknown items ): 258 Calories, 3g Fat, 4 Proteins, 2-1/2\n",
      "  cups flour\n",
      "  and whipped cream and whisk in the melted butter\n",
      " \n",
      "  Information, Misc britis\n",
      "      Yield: 10 Servings\n",
      " \n",
      "      2 md Onions coarsely. +++++++++++++++++++++++++++++++++++++++\n",
      "  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n",
      "  ++++++++++++ +++\n",
      "  \n",
      "  Mix all ingredient in Puerto Rican Cookery   ISBN 0-88289-411-0\n",
      "  \n",
      "  From the recipe is presumably the black mustard\n",
      "    1/4 ts Salt\n",
      "      2 tb Chopped fresh tarragon,fine chopped\n",
      "      2 ts White pepper\n",
      " \n",
      "  Soak chicken pieces all around than top of bag securely anchored to steak. Combine the remaining 1/2 teaspoons instant to\n",
      "               - (from 3 lb pheasants)\n",
      "      2    Tomato, chopped\n",
      "    3/4 c  Corn sugar\n",
      "  1 3/4 c  Flour\n",
      "  3 3/4 c  Sweetened cocoa\n",
      "      4    Milk (4-5 tbs)\n",
      "      1 c  -Water\n",
      "      2 tb Cilantro, fresh mint sprigs.\n",
      " \n",
      "\n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "n=12\n",
    "generated_recipe = generate_recipe(char_models[n], \"~\"*(n-1), n)\n",
    "print(generated_recipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Baked onion with chemicals, yummy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1431,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "~~~~~~~~~~~\n",
      " \n",
      "      Title: CORNY STUFFED PEPPERS & TOMATO CREAM\n",
      " Categories: Fruits, Candies\n",
      "      Yield: 1 Cake\n",
      " \n",
      "           Recipe\n",
      " \n",
      "  1 1/2 lb Fresh Asparagus is tender-crisp. Add vegetables and Lemon Rind; Grated\n",
      "    1/3 c  Rasted pepitas or hulled\n",
      "    1/2 ts Dijon mustard\n",
      "  green.\n",
      "  \n",
      "  Wash the squash in a 350F oven, covered, 3-5 minutes, then reduce heat, and simmer for 10 minutes, stirring constantly; cool\n",
      "  mixture and bay leaves. Mix well. Spoon this over rice.  Bake for 1 hour or until fish just flakes when poked\n",
      "  with various meat stuffings either! I got the\n",
      "  recipe calls for pimiento-\n",
      "           Mayonaise\n",
      "    1/2 ts oregano\n",
      "            -Stuffing Mix\n",
      "      4 c  Tomato sauce\n",
      "           -cut into pieces approximately 360F and carefully to remove the meat from marinade and 2 tablespoons marinade. Grill chicken,\n",
      "  turning the basic stuff,\n",
      "  unembellished\n",
      "  it somewhat (don't expect it\n",
      "  to be slightly so they don't burn. (Traditionally the heart-shaped (or similar bowl, stir together the tofu (as in the order given. Roll out pizza dough.  Roll out the dough\n",
      "  comes to a full rolling\n",
      "  pin. Mix together the dressing by combining them. Add the Thai chile. Stir\n",
      "     melted\n",
      "      1 md Green pepper, diced\n",
      "      2    Egg yolks\n",
      "      8    Egg whites\n",
      "      1 oz Turmeric\n",
      "    1/2 ts White pepper\n",
      "      8 oz Plain nonfat yogurt\n",
      "      1 c  Cranberries\n",
      "  and 4 grams fat, .75 gram\n",
      "  carbohydrate, 29 g fat\n",
      "  (11%), 1.5g fiber, 117mg chol, 544 mg Na. From the\n",
      "  Bengal Region\"\n",
      " \n",
      "\n",
      " \n",
      "\n"
     ]
    }
   ],
   "source": [
    "n=12\n",
    "generated_recipe = generate_recipe(char_models[n], \"~\"*(n-1), n)\n",
    "print(generated_recipe)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, the generated recipes are not perfect but they are also not bad. \n",
    "They start with a title, categories an 'yield'. Than the ingrediants and then the instructions. Sometimes there is \\n in the middle of the sentence and suddenly the instructions become a list of ingredients. But overall, most words are existing words, the spacing is good, and on first glance it can look like a real recipe. Until you start to read the instructions and they don't make much sense.\n",
    "\n",
    "We can keep generating recipes and pick only the best ones, and choose a better stop criteria."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Polynomial Curve Fitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Synthetic Dataset Generation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 636,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import math\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 976,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def noise(x, f, sigma):\n",
    "    \"\"\"adds Gaussian noise N(0, sigma) noise to a vector\"\"\"\n",
    "    return np.vectorize(f)(x) + np.array([(sigma * np.random.randn()) for i in range(len(x))]) \n",
    "\n",
    "def generateDataset(N, f, sigma):    \n",
    "    x = np.linspace(0, 1, N)\n",
    "    t = noise(x, f, sigma)\n",
    "    \n",
    "    return (x, t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 977,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def sin2pi(x): return math.sin(2 * math.pi * x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1285,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "N = 100\n",
    "x, t = generateDataset(N, sin2pi, 0.03)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We generated a function (sin(2pix)) with some noise"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1286,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.collections.PathCollection at 0x14781afb518>"
      ]
     },
     "execution_count": 1286,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXd8jef7xz937Fgh9gi1qVVJ1Co1SqhZNb9mq2pVp6Kt\ntmjRXTrMltKqWaP2qi32CrGVpGhExSaSXL8/Pjm/k5AQcnLm9X69zss5z/PknPs5ifu672t8LiMi\nUBRFUTwPL0cPQFEURXEMagAURVE8FDUAiqIoHooaAEVRFA9FDYCiKIqHogZAURTFQ1EDoCiK4qGo\nAVAURfFQ1AAoiqJ4KOkdPYAHkSdPHilevLijh6EoiuIy7N69O1JE8qbkWqc2AMWLF8euXbscPQxF\nURSXwRhzJqXXqgtIURTFQ1EDoCiK4qGoAVAURfFQ1AAoiqJ4KGoAFEVRPBQ1AAri4oAVK4AXXwQC\nAoDGjYFp04Br14C5c4GmTYEKFYA6dYAff+RxRVFcH6dOA1XSnjt3gHbtgDNngNdeAypXBsLDgQkT\ngP79gXLlgHfeASpV4vFJk4BvvgHWrAGKFXP06BVFSQ1qADycd98FMmQAdu3ivwBQvTp3BBcvAsYA\nHTvy34oVgaAg4OuvgTZtgN27eVxRFNdEXUAeTFQUMH06MH68dfIHOPHPnctV/o0bwKZNiX/uzTe5\nc9iwwb7jVRTFtugOwMO4cQOYPx84fZounWrVgHz5El+zfj3wzDOAry/QqROwdClQt671vDFA+/bc\nJTz7rD1HryiKLdEdgAcxeTLg58fVfUwMcPw4V/cDBgB371qvu3sXyJKFz7Nl42r/Xry9E/+Moiiu\nhxoADyA2FhgyBBg6FPjwQ+D334GRI4GJE4HcuYFjx4B+/azX+/vTvRMdDaxdC1Stev97rlzJjCFF\nUVwXdQG5OYsXAwMHAufPA02a0G0zYgTQty+QNStw9SoneRGu6D/6CChbllk/gwYB27dzx5CQ9euB\nQ4eAF15wyC0pimIjbLIDMMb8bIyJMMaEJHPeGGPGGWNOGGMOGGOq2eJzlQezbBnw6qvAG28wg2fx\nYmD5cmD1ambyTJ/OR4ECTP8MDQWefhrYuBFo0AD4/nv6/i9d4vtFRQHffsu00V9/BTJlcuz9KYqS\nOmzlApoGIOgB55sCKB3/6A1gvI0+V0mGCxeAnj3pptmwga4eC6NHAy+/DPz7L1CjBrB1K33++/ax\nKOzZZ4HNm4EFC4DChWkcfH35PDiYBqRhQ4fdmqIoNsImBkBENgL47wGXtAIwXUgwAB9jTEFbfLaS\nmIMHge7dgRIlGLx97jkGctevp8vm+HG6fEaNYobP9Oks6KpWjXGC3btpFF56CWjZEvjuOyAiAjh6\nlAHjwoWBTz7hzmLDBrqOFEVxTewVBC4MICzB6/D4Y4qN2LaNE3e9esAff3DFHh0NZM4M/PILULw4\n/f1du9Ktkz075R3Cw4HLlxkYfvllGoMXXkic++/lBQwbBjRvzrhB+/ZAyZJ8r/LlWTHcqBHf75ln\nWEV8/brDvgpFUVKI02UBGWN6G2N2GWN2Xbx40dHDcQm2buVqvX9/pmdu3gzMmMHJ/Ntvgc8/B778\nkj7+0FD68gHKP2TKxIm9Z0+miAJc1Ses8B0+HDh8mLuAESN47rvvWD9w9izlIYKDWSU8dCgDzf7+\nQFjY/WNVFMWJEBGbPAAUBxCSzLmJADoleH0UQMGHvae/v78oDycwUGTWLJG1a/lcRCQmRsTPT2Tp\nUhEfH5Hz50XmzhXJmlXEGJHu3UUyZRLJkUNk2DCR2Fjr+9WoITJ1qsjOnSIHDojkyiVy9izPzZsn\nUqiQyNatIj16iHTrJlKwID+nalWRjz8WiYsTGTFCJCCAzxVFsR8AdklK5+2UXvjQN3qwAXgewHIA\nBkANADtS8p5qAKzExYls2yby668iS5aI3LrFY1OmcFKvWVPkqadEqlXjORGRadNESpYUaddO5LPP\neGzcOJECBURy5xapXVskKirx53z+uYi3Nyf9p56i8ciRQ2TZMhqV4sVFNm+mQfHxEbl8WeTdd0U+\n/JDGIlMmEV9f/muMSPPmIkeP2ve7UhRP5lEMgE3qAIwxvwN4FkAeY0w4gI8AZIjfYUwAsAxAMwAn\nANwE0NMWn+sprF9Ppc7oaAZrL1wAevQAihRhJk+JEnTzbN5MF03VqszU6d6d0g+DBlHsLToamDkT\nuHKFvvycOYE9e4AqVRgLGD0amDOH1w8bxmumTwd++gno1Yvvlzs3ULs2Ywb16wM+PkCePHQPNW8O\nPPEE0LYtA8Vvvgns3UsZ6cWLGaNQFMV5sIkBEJFODzkvAPrb4rM8jY0bGXSdMgVo0cLqm+/bl5N5\nkyb05depw4l5zhzm/LdsyYyefv14fv9+irydPs33fPJJYOpU4K23eCx3bmYNffUV6wYslCsH/PMP\n6wdq1uT7AonlIrZvZ2HYgAF8feEC/82bl5P+228DHToAJ08C6bX0UFGch5RuFRzxUBcQ/ejz5iU+\ndv06XTh79/LffPlEDh7kufXrRfLmFSlWTGTlSpG7d/l8zhyRSpXom0+KvXsZM4iJSXw8Lk6kShXG\nD2rVEilViscPHWIsYP9+kezZOYY7d0T69RMZPVrkv//oLsqfXyRjRpEMGUSaNBFZvFjk5ElbfkOK\noiQEj+ACcrosIMXKgQNctbdunfj4qlV0u8ydyzTMChWA//2P7qB69bgLiI7mzqF0abp8Bg6kG+eD\nD5L+rNBQrtbTpUt83BjKRfftCxQqBJw7x9V+uXIcQ716/JxGjbiDmDWLRWKVK3M8ixYB69YBBQuy\n/uCFF1htXKsW3VSKojgONQBOzJkzdNUknJQnT2b+/dWrzPHPkIFpoMZYc/JPnGCs4O5d+uinTuV7\nDRyYfAMXb2/WAyRFzZqUldi5k+8ZGMiU0du3WSNw4QIQGUnff+vWjBVcusSfiYvjsUqV6AqaNg3I\nlYuup27dgHnzbP61KYqSQtQAODG5czM4a2HmTFbwLlxIhc9Bg7gC79OHwde6dbnS3raNFb/9+zMI\n27o1kDHjgz+rQQMGihN+XkKqVaPxmDuXNQNXrzLY3LMn1URXrWJl8MyZwKlTfF6vHtC7N3cQJ08C\nzZoBnTszuJw5M/WE+vQBbt2y3XemKErK0ZCcEyHCTJ7Jkzlh5sjBAOyGDaywHTYM+O03uk+qVWMx\n1tSp1OypUIHZQJ98wvf54w/g/fdT/tnZs1PeoUcPum2yZk08rkGD+P6tWvHx6afA7Nks9urTB/jr\nLwZ9a9Zk1lJgIHcmd+/yHrJk4T2EhTEQ/OKL3HVcvUq5iokTudtRFMV+GMYMnJOAgADZtWuXo4dh\nF2Jjqb+zZQuzafz96UMfPpwplmPHciW9fz9X4qGhvMbPj7INefMCgwcDR44AO3awcXvHjo82hpgY\nxgnWr6cxsDSInzyZrqYlSygxkRRRUcxIun6dsYFp0zhei4tn7VrGJRo04G4lIIBVyu+9xyyiAwe4\ns6ldOzXfoqIoxpjdIpKibh26A3ASRo7k6vjAAa6MLbz4Ihuw9+9P984331Db/9df6WLJm5eB3jx5\neDxnTu4SGjR49DGkT8+Je+dO5v5v2kR//fDhlHm4N0CcEB8f7lTGjePOo149Xp81K3c1Pj6c3N9/\nn1pBzz/Pn7t9m2N9+226hzRVVFHsh/5XcwJu3QJ++IF6OgknfwsLFrBJy4UL1N7x8eGKukIFnr92\njee++IKZOo8z+SckMJCPRyVzZuDdd7mLqVCBz0uU4Mp//34auIwZudNo2JDHZ83ivZQvz7EvXUoX\nk6IoaY8aACcgOBgoVYoKm8nRpw/97s89Z109W8ienS6kP/7g6t3RTJ5MI3TyJNC4MdClCw2Ktzfw\n8cfW9M/XXuPx8uX5ukkTFq+pAVAU+6BZQE7AnTsM+N6LCN0wU6cytbNKFUo2r1uXWIc/LIxGoWtX\n+tcdTb58DADXqAGEhDBAPGYM4wSjRln7DB87RqkJC7dvq/tHUeyJBoGdgLAwTohnz1qzb9asYTA4\nfXoGe1evZkHXCy8wzTNHDmYCnT/P1wMHsuG7lxOadBEGkNu1Y7exAgVorBo0sNYlxMYyljFrFlC9\nOusHNm1iBlHevNQdUuOgKA9Hg8AuRGwss2DSpWNGj58fM2QWLqSmf5MmzOxZvJir6r59qQnUvDnz\n7X18OGlmz+7oO0keYzjmtm3ZnWz8+PsN1ZgxDHIHBtKVNWgQ76lcOeDvv5mNNGIEM6UURbENagAc\nyJ07nBQjIijRMHo0hdzmz+f5fPlYePXmm8yuqVKFQdLSpek/d7W+vOPHA02bcjX/2muMeVga01+7\nxjqAXr0oPDdrFusGLDuEPXsoOXHrFjOiFEVJPU7oMPAc3nmHfu/Zs+nC2bGDefRRUXR3BAYyO2jq\nVLpMAKZldu3KNo+uRo4cjAf06QN8/z0L2qZNY73Ctm00DtOmUf9oyBDet6VKuFo1dhobNozGQlGU\n1KMGwAGcOcOc9x9+oHujenVW0O7fzyDvc88xlfLppxkHaNw48c9XrOi67RYzZmQz+rx5udr/7z/u\nfN54gxXIpUpxhX/pEovIqle3ykuXKsX6grlzHXsPiuIuqAGwM6dPsyAqfXpO8CdOMJA7ZAi1+7ds\nYeCzVCkqfK5adf97nD2bfEWuK3DmDDOZxoyhi6dPH9YN9OtH99DXX7NW4MMPGfROWNFcuTLvX1GU\n1KMGwM68/jr93126WDN+0qdn7vvatfST37jBgG/mzJwIExIdDfz8M42Dq7JuHSuLs2ZlcHfFCjai\n8fXlay8vZgytXk0jcPo0YwAAz+fK5djxK4q7oAbAjoSFWbV+qlRh0VNUlPV86dJc8daoQT//nDmJ\nK3Jv36aEcvXqTBt1VWJjgUyZ+Hz1ahqDbNlYy7B1K7+nTJlo/NKlozFYvpzf1R9/8LWiKKlHDYAd\nCQ0FnnqKK9/8+Rn0HDUq8TX16lF4rX17To6rVtFV8sYbTBE1humhrkxgIO8tJoZqoRb5i+zZ2Seg\nbVvGA6pX53FvbwbH27fnzqlQIceNXVHcCU0DtSP//svK2JYtmRHTpAkn98hI5r2XL8+A54kTdIvM\nmsVV78mTTAndvBkoU8bRd5F6qlQBihWju6tWLQaBY2O52n//fWDfPuofLVrEauFJk4CbN+n2+uYb\nR49eUdwHNQB2QIRSzb/8Qv9+UBD9/hMmcNLLkYPBz2vXmPZYpw67aQWkqJbPNZk2jVXBHTpQwXTq\nVBaLTZrEpvWLFwOHD1Oa+s4dxgBKlXLwoBXFzVAXUBpy6xabthQqBHz5JVe5FSsyiNupE/3d7dpR\n8iA8nNr4FSowV96dJ3+Ak/n27ZR8OHUKeOUVoGhRVj1v3sydgZcXv6MlS3TyV5S0QA1AGnHtGit1\nly2jr3vbNrZcbNyYTdVLlwY+/5wpoZcu0d89YwYrfR+ku+9OFC5Ml05UFPsgdO/Oyb52bRqDPXto\nDLVJjKKkDSoGl0b068cdQK9ezPrZu9d67q+/mO3TujULwS5epI9/+XJrdoyncucODUKOHGwjqSjK\no6FicA7myhXg99+Z9XPw4P1FW/Xrs4F7rVr0fU+fzmwfT5/8AX4H+fM/+JrQUBbL5csHVKpk1QtS\nFOXRUBdQGrBnD339BQowa2ffPqumjYVWrej7B+geKlfO/uN0NdatY/V048ZMn23dmmm1S5Y4emSK\n4pqoAUgDjLE2bClWjP79SZMSX2M5f/Ys0z1V5vjBrFjBwPngwXSbrVvHdNnRo9nAfvZsR49QUVwP\nNQBpQLVqTGE8f56vv/mG+f5ffGFVslywgK6OBg0od6DFTUlz8SJrA5o3p2x2v340AhbJiKZNgT//\npLzG7duOHq2iuBZqANKAHDlYtDR4MNMcy5ZlbntwMHcE5cox4Lt0KTByJPX+lfs5c4a7p717+Z3d\nvUu3WVwcjx8+zOuqVaM0xsKFjh2vorgaagDSiDFj6KoICqLsQd68dFd06cKK4GnTmA7aqZOjR+q8\ndO9OaehmzSiRkT4902e//poxgA4drK60wEDg+HHHjldRXA01AGlERAR3AE89RalnPz+6e9KlYz1A\n166avfIgDh7khP7669QIiohIfL57d+4E1q/n64gI526LqSjOiKaB2piQEAqa7dtH7fqICKaFfv45\ng5U66aeMrVvp38+Qgf7/N96gZlKePNZrAgJoXPPl4y7r11+5I9DvWFFShu4AbEhICKt/27ShpPGa\nNaxw/eMPdv8aPtzRI3QdjOEKH2AdRY8ejKvcvMlisQ4dKBaXPTsNQ8WKDBY3a8ZrFEV5OGoAbMg7\n7wAffQT07Zu4qMvfn8bg++8Z2FQeTr16DJLfucPXn3/OTKnSpVlAd/Iks4AOHQJy52YD+f/+Y7po\nrlzcMWjnMEV5MGoAbMTff7PBS3L5/Pnz0+//0092HZbLUrYss3s+/ZSvM2Rg1fRvv3HSj4nhsR9+\n4C5LhAV116+zf8CtWywaO3TIsfehKM6MxgBsxKlTdENkzpz8NQEBWrX6KEydStmMI0eY51+uHHdS\nBQsyrrJtG91CQ4YwW8hC+/bsrVCzJrOs9u/XuICiJIXuAGxE9uxM73yQtl5EBGsElJRRoAAlo2vW\n5ARfrhwweTKP794NXL7MBjp9+iT+OV9f7gS6d2ftwMaNjhm/ojg7NjEAxpggY8xRY8wJY8yQJM4/\na4y5YozZF//40Baf60xUq8ZK1L/+YuP2exHhirZ9e/uPzZXJkYOFcgcOUDZ7xgx+v7lzcwfQtOn9\n8tnbtnEHYAwziIKDHTN2RXF2Um0AjDHpAPwAoCmACgA6GWMqJHHpJhGpGv8YkdrPdSaiooCPP+YE\n1agRdwMvvGCdeETopsialbUAyuPTqBG/79WrGQSOiUl8PjiYLqMWLfj67l1epyjK/djiv0Z1ACdE\n5JSIRAOYBaCVDd7XJbh0iS0cw8K48hw5kkHI//6jamX37qwHsLQ5VF906vDyAiZOZEX1jRvUAYqO\npiGYP58KoRMmABkzsgPbggVqdBUlOWxhAAoDCEvwOjz+2L3UMsYcMMYsN8Y8mdybGWN6G2N2GWN2\nXbx40QbDSxtEqEtTvTrz0MPCgA0bgIED6a6oX5+pibNm0Shs3Uo5CCX1NGrEiX3pUhraEiVYaf31\n18wSahW//PjiC3YW8/d37HgVxVmxVxbQHgB+InLdGNMMwEIApZO6UEQmAZgEsCOYncb3SMTGAr17\n099/4QIno7t3qe8zZgywciXrAQBmqZw+rSt/W1O7NmUgduxgUVjp0sDQoey/sHEj8OOPDBSvW+fo\nkSqK82KLHcA/AIomeF0k/tj/IyJXReR6/PNlADIYY/LARfn8c2rR//gjV5dNmjDYOG8eJ/7nn7cG\ngps2BXbudOx43Znq1bnjatGChXgBAdyFVa/O771o0Ye/h6J4KrYwADsBlDbGPGGMyQigI4DFCS8w\nxhQwhmtgY0z1+M+9ZIPPtjtRUcBnn7FQaeFC5qMn5KWX6I6wSBPfvk0VSyXtyJ6ddQK7d7MHw759\n1GPy8gLGjqUgX4EClIz++uv7f2eK4qmk2gCISAyAAQBWAggFMEdEDhlj+hhjLBnaLwIIMcbsBzAO\nQEdx5m70yfD779TzF+Ekny4d9X+aNmXeuYXOna0FX3PmMBis2A8RBocrVKAh/vpr9hT47ju6jAID\n2VBGUTweEXHah7+/vzgLf/4pUqiQyM8/i9SubT0+eLBI4cIijRuLxMXx2MyZIu3bi8yZI1KkiMit\nW44ZsyeycKFI2bIiWbOKlCsnUrGiSKlS/F1YGDVKpG5dx41RUdISALskhXOswyf5Bz2cxQDExYlU\nrSqydKlIRISIj4/If//xXHS0SOvWIpkyiXzwgciRIyLPPy8SECBSsKDInj2OHbsnMWcOjfSUKfw3\nOpq/u40bRYoWFZk+ndfdvUvDrL8bxR15FAOgJTIpYM8epnlu3Mjm7rVrM8UQoCDZ/PlAx47A9OkM\nCK9YQbfP/v30Pytpz927bB6zcCHloFu25O/GGKbjLlkCvP22NSbTpg2zuBTFk1EDkAQxMfT3169P\naeHAQBYdxcWx8GvHDmDcODYoDwtjsLFxYwYaY2JoJD79VPP+7cmyZUwFDQzk7+leeYjKlWmMFy3i\n6/Tpmc6rKJ6MGoB7iI7m6nDsWKBGDer6jx3LSX7qVKBtWwYQW7TgqrJKFaYavvQScO4c5Z6Tk4RW\n0o5Tp5jlA1A8btkya0MZC089xevi4vi7q1XL/uNUFGdCDcA9DB9Ot8HGjXTl/PQTUwwDAoCePanx\nEx3N9oNZszLLZ9Uq7hSWLKELSLE/Pj5MAQW4C8iTh5IRCTl3jtdNncrfnRoAxdMx4sTZmAEBAbJr\n1y67fd7t21zNBwdzkm/ShI1evLyo89OqFVCyJNCtG7t+ffUVc84vXACKFOHEojiGS5f4uzl2jD2C\njx2jBtCLLwIDBlA9tHhxpuguXgysXUvFUEVxN4wxu0UkICXX6g4gAQcO0ACULEl9n6JFrUqSNWsC\nc+eyzeCQIcDLLzNOMHs2XQ+TJzt27J6Ory/lOTp0AK5epSTEjh0UhatZk+fv3uXKf8cOnfwVBdCO\nYIkID+dqvm1b+okPHWJGibc3z9erx1X/hAmcVK5cobvBkhGkOJbRoykDUbIk1ULLlOHv0Rg2lW/V\nimJxnTqxRWf37mwif2/AWFE8Bd0BxPPJJ8Arr1BdskkTICiIFaVlyjDTx8LSpUwx7NSJrqJ7u1Ep\njiNdOvYIDg4Gcuake87bm+J8Z85QlbVJE2DECKBwYRoAX18G/Zct06wgxfPQGADou//iCypHfvop\nZR1+/pmugoYNOUmcOMG8/saNgc2budL08+N1inPTsSNdP5MmcUfXqhV3b6++ynjAhQs09jlzMk00\ne3ZHj1hRHp9HiQF4vAGIi2Ov2alTWeB17Rrw3HNsPD5kCHDnDleNmTIxSFypEo1B7940Fir05tyc\nPk033dmz3A106ABky0ZjkC4dcOsWDfnWrZTyvnGDPRwUxVV5FAPg8dNXSAiNgCUlMHt27gR+/JF+\n5FOnGAj28WGl6ZNP0iDkzOnYcSspY9kyrvi9vWkM1q6lMbD4/bNkYRexFSsoFufnx8yv4sUdOWpF\nsQ8eHwO4fp054wkbtnh7U1v++HHuAGbNAp5+mjUC7dvr5O9K3LnDxvIAjUHr1tagvoWcObm78/bm\neYuSq6K4Ox5vAEqWBI4epesnKdKnZ1CxXDn7jkuxDZUqsXOYCN09FmNgQYSaQJUq8bXFGCiKJ+Dx\nBiB/fgZ6f/gh6fMREWz12KuXXYel2IiGDbnLW7KEk/yGDZz0LSxezCyvr76irMfkyawBsXR0UxR3\nxuMNAMAWj999x05flp2ACFf+DRsC/fsDpUo5dozK4+HlRQP+0kusDr58GVi+nMHeL79kpXDhwgzq\n9+rFmMC2bVQQvXzZ0aNXlLRFDQCAEiWo/bN9Ozt+1apFl8///sd0T0uDd8U1qV2bek1r13JH17Il\n4z5ffMGYzqZNXPWPGMFq7/XrGfPp2dPRI1eUtMXj00BDQ5kSGBrKXPEGDSgTkCsXXQJeaiLdikuX\nOMF/+y2wZQtjQBcvssr7o4+AatV43e3b1vRQ3f0proRqAaUAEeD996n5ny0bUzzbtGFDkVdfZTBQ\nJ3/3w9eXUh8vvshK4MWLWdexaJF18geAzJmZPrpihePGqihpjcfWAUyaxMBgSAjdARa6dAG+/55S\nECEhFBNT3I/oaCqEPkgULmtWCsgpirvikWvcuDgGfidOTDz5WxgwgIHBhQvtPzbFPvj7A2vWJM4I\nSogIsHq1tvRU3BuPNAAhIawEffrp5K/p2lUNgDtTvz6LxObPT3w8JAQYPBh49lng33/ZW0BR3BWP\nNADXrrHy99tv6QqydJJKSK5cFA5T3BNjgBkz2Nf5s88oCNe9O9CoEUUA9+5ltlDDhuz9oK4gxR3x\nOAOwZg2lnE+eZF74pk1AhQrMAU9YAbp5M48r7ktgIAvDQkLY/GfWLBaNFS1KI/DzzwwQh4cD777r\n6NEqiu3xqDTQLVuY6TNzJnv9lilDfZ+rV1koFBsL/PEHdwSVKwM7dwJPPGGzj1eclPPnaey3bqUI\nXJYsic9fusRU0OPHk44ZKYozoWmgyfDBB8A333Cb//nnrBD94ANu72fOBA4eZAZQ/foUg9PJ3zNY\nuJDunvLl75/8AaaOBgUxVVRR3AmPMQBnzgCHDwPt2vF10aLcEZw9y2KgwEBWg370EfDhh+wFoHgG\nV65QE+pB5M/P6xTFnfCYOoDz57m9T5jXX6QIMH06NV/+/psaMMuWUQJC8RxKlmTK571ERACzZzMb\n6M8/gY8/tvvQFCVN8ZgdQP783AUklc2RKxfzvW/fBgoUsP/YFMfSsiUDwbt383VcHPDee0DZssCu\nXTQE589TF2r48ORrBxTF1fAIA3D3LrM9bt6kxEOhQiz2Ono08TUTJzIVUPEsMmUCxo2jIVi1ii7A\ndeuAI0fYT/jPP4HffqNe1OLFwKhRjh6xotgGt3cB3bnDLk83bnBV9/XXfISGAnXq8D927dqUAy5d\nmscUz6NDBwaABw5ktk/duvy78PZmxlizZrxuwQKgalVep83jFVfH7Q3AiBFc4f35J7t7Va7Myb54\ncaBxY676vL2B558H5sxJ3BpS8SxatmRB2Pz5FAcsWJCTfcK/CT8/9gpYtIi6UYriyri1Abh9mx2e\ntmzh5A9wJXf6NIO9oaF0AzVtCowc6dixKs5BZCTjQZYVf1IUL04JaUVxddw6BhAaysKdy5cZ5IuL\n4/EMGSj1O2QIHwcPOnacivNQpAj/bh5EaCivUxRXx20NQFgYMHQoV/j9+nHCL1eO/tyEWRzGaFaH\nYuWFFygPcvLk/ediYtg1bs8eoEUL+49NUWyNWxqAsDAGc6tVA3x8mMt94gS1Xb79NnE+96JF9Okq\nCsDmQMOH0wV0+DCPnTjBuFHOnEDNmswme/ttphUriitjEwNgjAkyxhw1xpwwxtxXQ2vIuPjzB4wx\n1ZJ6H1tSrY1oAAAgAElEQVQxdCjTOUeNosjbkCF0/9Spw76w48dzZ7BjB7B0qfZ+VRLz2mvAG29Q\nEjowEKhUiemh3t6MKR0/TmNQo4bVSCjK4yJidU/bm1SLwRlj0gE4BuA5AOEAdgLoJCKHE1zTDMBr\nAJoBeBrAWBF5gBo/eRwxuP/+Y2XnqVMs8Lp1ixk+Xl7Uea9dGxg0CDhwgEbg55+B5s0f6SMUD+HW\nLaYGN21Kl09QECvJb93iwmHhQiA4mPGlzJkdPVrFFYiLY43J4sXcQZ49y3qT6GjgySeBvn0pP56a\nToT2FoOrDuCEiJwSkWgAswC0uueaVgCmCwkG4GOMKWiDz76PU6eAEiU4+QPM7V6+nD1g332Xwl4z\nZvDLX7dOJ38lebZuZdvISZOYIpohAzB2LHWkJk2iu+iff1hYOGWKo0erODvh4dxRvv02q8vXr2dj\nqowZgalT+bc1bx7/1u7csc+YbJEGWhhAWILX4eAq/2HXFAaQRCuW1JE1K3cBItb87UyZgD59+ACA\nX37hCq5iRVt/uuJO7NzJVb/l7+ibbzjRb91KKXGA/4Fz5ABGj+bf3CuvOG68ivNy5w7QpAlrRzp1\nYkvS4GAq0B4+DDz3HI3AypWUrP/sM1akpzVOFwQ2xvQ2xuwyxuy6+BjJ1uXKcTu+aVPy1/zyC7M9\nFOVBpE9vXYldvcpakWXLrJM/wFoTPz9u6d9/P3FTIUWxMH8+CwuHDuXusVs3Tv4Ae1F8+y3w6af8\nmxs9mrI09uhCZwsD8A+AogleF4k/9qjXAABEZJKIBIhIQN68eR95MMbwS+7Th1Wd9/Ldd9yKqQFQ\nHkajRmwQFBPDrXmDBiwCs3D7NivMGzSgEahUiTtLRbmXWbPo2weoOvz884nPt27NWNL58/RMZMxo\nnywzW7iAdgIobYx5ApzUOwLofM81iwEMMMbMAt1DV0TE5u4fC926MbhSqRIzfGrXplto+nTg3Dlg\nxYrUBVkUz6BqVQaBR47kyqxcOes5EaB/f8aYAgIY3IuJ4aqtenXGCRTFQlQUdwAAF6mxsYnPZ8jA\notUrV3hdbCwTV9KaVH+EiMQAGABgJYBQAHNE5JAxpo8xJt7rjmUATgE4AWAygH6p/dyH8cEH9LEZ\nw+KvlSsZYT94kEFiRUkJv/3G1f/cufT9Hz8OLFnCRcX06RSRO36cYoO1a7Pu5OmnEyvNKkrJksDe\nvXxevz53lgm5dIl9JwoWZOwpfXqgWDE7DExEnPbh7+8viuJorl8X+fJLkXTpRIoVE6lTRyR/fpEZ\nM6zXnDolkjs3r50wQaR6dYcNV3FCNm8WKVlS5MYNkXPnRHx9RYKDrec//lika1eRmzdF6tYV+eqr\nx/8sALskhXOs0wWBFcXZyJqVqXtDhrBh0GuvcaVm6RwXHk4f7pAhvLZXL6b5PWIJi+LG1KrFHWLL\nlowd/fILU9CHDuXjxx+ZGVSrFt2Hr79un3G5tRqootiSESP4b/fu3NJ//DFdPStXsrjwnXd4Pl06\npvzt2MH4gKIYw6LT4cNZC1C+PAu/vv2W57Nlo2tx2DAuJuzh/wfUAChKivHyAj75hP+Zt27lsQYN\nKC1iKTy0cOeOVYJcUeLiuFho1YoZiocOsaK8YkXHxiT1T1RRHpE2bRgAXrWKq/17uXXLuppTPBsR\n5v1/+SUze3LmBP7+m39Do0Y5vge5xgAU5RGpVo0ZGkn1BhZhQVjt2pptpjAuNH48FwwnTzIT6Ngx\npnzWqcPMH0eiOwBFeQxmzQIaNmRvgL59Odlv20ajEBUFvPUWa09y53b0SBVHsX8/dcdCQhL/HeTN\nC3z+OetGPvoImDDBcWPUHYCiPAaFCjFfu3FjarYEBLDo0NeXWUB79zJQPGqUNhzyVCZNor8/uUXA\noEHsQ379un3HlRDdASjKY5ItG1f/f//NgO+iRTQAFv75hyX/GTNaM4QUz+Hw4QdLzhQsCOTLxwZW\nFl0ge6MGQFFSQUQEV3rHjyee/AGgcGFWfAYG0lBkzeqYMSqOwdub/ciTIy6O7kJvb/uN6V7UBaQo\nqWDePBb05MmT9PkSJagNpCJxnoclWyw5Vq/mIsHPz35juhfdAShKKoiIAJ544sHXFC/O6xT35+hR\nKsSeO8dsn5UrGQOoUYPxgObNWU/yzz/AgAGUgLb0m3AEugNQlFRQpAgQGvrga0JDeZ3ivly5whV/\n3brAxo3M7Nm/H8ienbUilgm/YUPKilStyuZB7ds7dtxqABQlFbRrB6xZk7x2+4ED7PnarJl9x6XY\nj9hYavzky8e04G3bKDkfFgYsWMBrWrVi/ciBA8wQ276dLWodjRoARUkFOXNSerxZM275AaZ9btoE\ntG0L1KxJzZcDBxw7TiXtWLkSuHaNBV8zZzLzq25dnqtXj1XA27cDCxfy2lOn7CT1nAI0BqAoqeSt\nt5jqWbs2mxCdOAFERjI1tEcPIDqaGvA+PlwpvvwyV4OKezBjBv37Xl7cDb72WuLzHTrwb+T8edaL\nGMOq4IStRR2F7gAUJZUYw//0Z89ysi9UiCvBM2e42tu0ifIQefNy+9+mDdC1q316vippT8JEgNhY\ndvdKSObMzPm3tDjPkOH+jmCOQg2AotgIS+bH+vWU9O3Th37hkBBqwqxaxYDwli3sAOUMPmAldYhY\nVWJfeYXP7+32dfMmg8AFCjBLKCrKeXSi1AAoio1YsIBB4cyZWRj211/AxIlWWeg8eZgF8tdfzA+f\nNo16QYprcu4cazxOnGCgv1o1VodPnpxYKPCXX4BnnuHv/4MPKBWSKZPjxp0QNQCKYiOuX7cWhC1c\nyBS/zJkTX+Pra72uQQNg2TL7j1NJPXfvAkFBQIsW9OcHBgIbNgBTp3LyHzYM6NcP+P13Pm/ThjUA\nFy86l0y4GgBFsRHlyjEFEEhsDCyIAMHBvA6wGgPF9Vi4kEH9YcPo9pk7l6v/4sWZ8RUYyN1f5850\n+YwZAzRqxPTQLFkcPXoragAUxUa0acPin127gLJlrcbAwvLlbBZTr979xkBxLWbPpivHUsWbJQsw\nZQoF4Bo1Yr/oHDm4O7h5k77/t966f0foaNQAKIqNyJSJzb1btKDff88eYPduZgb9+it7CU+cyBXj\nsmVsG1mvnqNHrTwOUVFJd/MqWBB46SVmhRUpwh1exoz2H19K0ToARbEhbduy/P/DD6n2WLMm1R6r\nVgUWLwb8/WkM3nyTbgNH6sAoj8aJE9T5uXmTr2fPZiygfPn7s3qiolgJ7OwSIEacuFtFQECA7Nq1\ny9HDUJTH4tgxThjTpzPts2BBq25QpkysBq1ZE3jqKTaPqVePuwPFubh2jcV7f/1Fff8jR5jKGxvL\n3114OP/97jtrcdfo0YwF/P67/cdrjNktIgEpuVb/3BQljShThsJf+/cz5fOff9hBbNUq1gCcOcPW\nkoMGAa+/DpQubdWOUZyDuDjGdrJlY6GfMYzfhIZS8iEkBPjqKz6vVw84dIiGYNw4YORIR4/+4agB\nUJQ0RgQYPJjFYH/8QV2YqVNZMBYZyWDiE0/QSAwYAMyf7+gRKxbWrmWl7+TJDPAuX87eDqVLswNc\nixbcHcyZQ10of3/u+tavB0qVcvToH466gBQljdm8mZN8aChlg594gpIQxYvz/M2bbAqyYwdw4QLQ\nqRMlJNKlc+iwFVCyo0YNoH9/dnUrWhR47z3r+ehouva+/567g969gdOn7+8OZ0/UBaQoTsS6dfQd\nG8PAYePG1skfYJA4KIiFRLVqMbtkzRqHDVdJQEQE4zMAUzr9/ROfz5iRRqFsWaBjRxr3sDD7j/Nx\nUQOgKGlMXJx1NX/2LFCx4v3XpE/P6wAqip49a7/xKclTqBBz+AG6eM6fT3z+9m1O+AUKMCgcEcHr\nXAU1AIqSxtSsCSxZwlhA7tzMGklIdDR14mvU4OuwMF6nOJ4ePdjd6+5dyjr//DN/jxZ++416QIUK\nMT7g5/fwFqHOhBoARUljnnuOfv7Zs6kPNHcucPWq9fy4cXQhREaykCw4mC4hxbHExjJmc+MG3UA7\nd1LL55NPuFtbupSB/REjKP7Xrx/F3lwJDQIrih3Yt4+Tes+eXOFHRrIh+JQpNAiZM3PVf/Ys/cp5\n87KTVNOmjh65ZxIWxtTOLFkYlF+0iNIecXGs4YiJYWpoz57Av/+yyO+LLxjsdzQaBFYUJ6NqVa7s\nb97kynHlSgqGLV3KSaVaNU46r7/OTKCvvuLksmSJo0fueURHA02aMANo+3bgjTdYBHb6NFf/mTIx\nqN++PQv8qlRhlbAzTP6Piu4AFMUBxMayaOjpp7nCrFiRAmL581uvWbeOTUaOH9cKYXsyZw5dcevX\nJ31+9WrgnXe4q3NGKY9H2QGoFpCiOIB06bi6bN6cgcWkqF+fipJ//cVGMop9mDePu6/kaNiQjXxO\nnGBBmCuj6wpFcRCnTtE1lBzGUGPm77/tNiQF1P7Jmzf5815e7PVw7Zr9xpRWqAFQFAeRKxfbCj6I\nc+fYeESxH2XKcHeWHFFRNMrFitltSGlGqgyAMSa3MWa1MeZ4/L+5krnub2PMQWPMPmOMOvUVBQwi\nzp7NNMOEiAAHD1JJUlNC7UtICIPxI0cy2FutGns4REdbr/n+e/5OHCn3YCtSuwMYAmCtiJQGsDb+\ndXLUF5GqKQ1OKIq7U7w40LIlg7+3bvHYokV0CzVvDrz6KoPFTZve311MsT1LlrBPc2AgNX2efJL/\nzpnDlNDISLZ2HD8+cdN3Vya1BqAVgF/in/8CoHUq309RPIrx4xnofeIJTvo9erCa9OpVio/9+y/w\nzDNccU6cmLgKVbEdkZHs2LZkCTB0KH8v3buzVuPiRWb8FC3KHdmmTa5V7fsgUpUGaoyJEhGf+OcG\nwGXL63uuOw3gCoBYABNFZFJK3l/TQBVPYc8eoE4dygtXrgx060ZBuI8+AvLlAzJk4DWlS7NGoEkT\nR4/Yvfj8c8o9T5uW+HhMDL/348eBgQPZ08HZ+vrei03TQI0xawAk0f0S7yd8ISJijEnOmtQRkX+M\nMfkArDbGHBGRjcl8Xm8AvQHAz8/vYcNTFLdg/35KRsyezdeWpiILFlCBUgSoUIE7hO7d2U9Aq4Rt\nx+bN/G7vJX16av1Ur073T2goM7PchYcaABFplNw5Y8y/xpiCInLeGFMQQEQy7/FP/L8RxpgFAKoD\nSNIAxO8OJgHcATz8FhTF9Tl2jEVhALNMPvyQK0+Lq8EYisXlzk0Bsr59uQvQAjHbYOn09SBEnLPw\nKzWk9s9nMYDu8c+7A1h07wXGmKzGmOyW5wAaAwhJ5ecqiluRLRt9zQDbRDZpcr+f+eJFNpxv0IA9\nBDYmuYRSHoe6dYGFC5M/f/Qov//y5e03JnuQWgMwBsBzxpjjABrFv4YxppAxZln8NfkBbDbG7Aew\nA8BSEVmRys9VFLeibVumfd65k3SB2LlzwNatNAzGUH/m9GnHjNUd6dmTcs5bt95/LiaGPZx792Zq\nqDuRKikIEbkE4L4idRE5B6BZ/PNTAKqk5nMUxd0pV45B4FdfZaA3YYHY9etAly5Anz7MGAJ4PleS\nVTfK45A7N/Drr0CrVpR17taNx7ZsAT77jDuvYcMcPUrbo1pAiuIkTJsGtGtnbRYfEEC9mZ9+Apo1\n4+T05pusQt2yBfj6awcP2M0ICmKK53ffMfX25k0a5j59WKuRIYOjR2h7NISkKE5CtmzAsmXAzJlA\n4cJsPn7lCjOBIiLYkSpzZuakBwQwFvDGG9ZWksrjceAAXUA+PmzHuWkT8P77rMEIDmZ2kDtO/oAa\nAEVxKoxhY/j9+4F69ZgW2q0be9E2acJisE6d2ED++HF2qRo50tGjdl0WLwYaNeJKPzSUshxjx7Ii\nOyjIWqHtrmg/AEVxYpYsATp2pAuiWDEagyJFrOfPnKFezdmzQNasjhunKxIZyVacK1dyR5WQuDga\n2mLFWCTmSmhHMEVxE/buZfbJxIl0CSWc/AFOUP7+wKpVjhmfKzN1Klf/Pj73r/S9vIDRo9mrwZ13\nAWoAFMWJuXo1cZewpMifP3GTeeXhLFrECX7ZMrrWChcGBgyw1mIAQIkSPB4a6rhxpjVqABTFiSlV\nCkjOC3rxIguUduwASpa077hckRs3gMmTKanRoQNlHt57Dzh5koHgdOmA2rUZcLcQG+t+1b8JUQOg\nKE5Mx44UhTt2zHps82auWkuXZtvI48e5mp05k7EA5X6OHKG889y51Pv/5Rfq+3zyCTOpChdm8LdV\nK2BIvKj90aM0BhUqOHbsaYkGgRXFyZk0ifrz06dTJ+iVVyhMljEji5Z8fYHLl7nCzZaNAc0xYxgc\nVlhdXb488MEHrJ6+epWT/eXLNKJ58gD9+wOvvcZdVZkynPx79WLFtatlWWkQWFHciN69OaG//DLw\nwgtsJPPRR3xkygT88ANw6RI17MuUoXujSZMHtzX0JObNoyvtpZfo6qlfn8dz5aLu0r//0jiEhtLd\n4+tLZdbYWPes/k2IGgBFcQE6duRkVL06MHw4J7X//mPVcNOmzFrp3p0SEQEBbFvYt682kAGAP/9k\nGi1Ag5kwYN6oEWMoXl40DGXLUvO/fXsGijNmdMyY7YUaAEVxEQ4epI86KIiFYvXrJ/ZPp09PCYMD\nBygpceUKsHu348brLNy+bdVQat6csZKElC7Nx6JFwIoVQIECjAOk9wChHDUAiuIiZM5sXb0eOQLU\nrHn/Ndeu8TovL/YXOHLEvmN0RipXZuU0wJX9kSOMp1i4dIlB9vz5GQt46y1mBHkCagAUxUVo0YI+\n67g4BnsjIxOfv3iROjaN4ls4XbrE6zydXr2AGTNYNZ05M7B0KX3+bdty1T9wINNo69WjUR0wwNEj\nth9qABTFRQgMZCXwsGFAmzbsDBYTw3PR0ZSS7tKFwc2wMOoENUq2n5/n4OfHgHn9+pTWKFcOOHSI\n3+fAgcD8+Tw2dy6zg9w57/9ePMDLpSjugTHAnDmMAWzezGyVF19kxsr48cwOCgjg+W3baCy2bQMa\nNvS81pFxcVRNvX6dGUADBwJFizLv36L8eekSM6Y+/vjh1dbuitYBKIqLER3N9oXTprEvgDFA69bA\nX39xor9yhRWtQUE0DKVL03Xkbt2skkKE1b5jxvB+8+RhwLxQITbcqVGD7R9FaCDdUUBP6wAUxY3J\nmJHBzGXLWBg2fz7bGXp7Ay1bMjV01ixmsfj6AuvW0cWxcCFz292Z4cPZ0GXWLGr558zJbl4FC3L3\nNGcOjcDOne45+T8q6gJSFBfGGCBLFqY5HjrEHcDZs8BTT7H6ddAgnm/ZkpPj+PE0BFmyOHrktmff\nPnZJGzkSuHuXu6KyZdlQJ0MGyjpv2wZs3MjdkY8P00I9Gd0BKIqLs2QJteu9vOjaaNOG1cOLFnGC\na9gQePZZCp/lzs00R3ciJoZ5+zVqAHnzUkK7Y0dO9p07W7t59e3LdFBfX8prf/SRFsqpAVAUF+fO\nHWuh04YNLHx6++3E12TPzlXxuHHA778zAOoKiCTvtjp6lFpI2bMDX3xB11jDhpzcn3uO1b9t21ol\nMbJn547g1ClWT0dEJBbZ80TUACiKi1O5MgPAAGMBHTsmTmW8e5dxgW3bqBpauDDlEZyZAwfYizdb\nNq7gS5YEPvuMWT0AsHYtg7pxcXTlnD/PwrfNm4HGjVkT0bw58OWX3PkANCaRkXxPLy8GgV3FEKYV\nagAUxcVp1456Njt2MEPI29t67vZtykP89x+DwoUKUTV0wADnbXW4fDnrF558Evj7b+4AZs1i4LZe\nPdY4dOzI4HfmzECfPkC+fFTzzJGDBi4sjNIZnToBISFUAd26lYaxUiV+LydO3N9hzdNQA6AoLo63\nN/DTT6wUvnmTPW4BTvo1ajA4un078M03nCRv3WIG0bRpDAo7E1evAl27Mn4xaBB9+sawaGvuXNY5\ndOzImEbdugx4P/kkf/b553lvhQuzycukSdz9lC7N76BXL2DoUK7+f/2Vctl+fg69XYejWUCK4ga0\nbMkA74gRbCBTuDAn05gYdhSrWJHX/fgjV8B16wKzZ1M2ulcva6DU0cyYATRokLTOkTG8v6JFmeoJ\nMM//zBk+T5eOhu355+nu8fGhW+j0aRZ/DRpExdS5c2kIVqyw3305K7oDUBQ3oU4dNoefNYsujpo1\nOblXrEiJ48GDga++YpAUoCEoXtwaP3AGtm3jBJ4c+fMzt9/S+ex//wOmTLEGigsXpgJqoUI0FFFR\n3BVUrUq9/7JlgU8/ZQzE3z/t78fZ0R2AorgZ7dszaNqhA7OC5sxhbKBzZxZHJfR7Fy1KV9HDiIri\njiJv3rStIfDyuj/rJzQU+PZb9kC4fp2B3wkTeE8VKnCy792b7qyMGWnsIiO5m2jb1loUd/06JSEC\nAz1L7+dB6A5AUdwQf38GRzt1ohTChQvsHObry364QUFsLrNiBTNo4uKSfp/Nm4FmzWgoatViRW2f\nPkB4eNqMu0EDBnctrFlDd1XhwpS9ePZZurWiongfc+cy73//fqBYMeDNN5n9kysXx/3ZZzSIL7wA\ndOvGe9bJ34oaAEVxU7p0YTAVoC5OeDgNw++/s69w584MIP/8M1fK0dGJf/6PPyg2164dc+bDw7ka\n9/Ghe+n0aduPuUMHTuhLlrC3QadONAjDhnFyP3qUq/iVK7kjuXiRAd1z5zjJL15M4zBoEN1Elk5g\nSjKIiNM+/P39RVGUx2f4cJHKlUVCQ0UCAkRGjRKJixNZtUokXz6RDz4QefFFkTx5RPz8RBo3FgkK\nEnn5ZZGcOUV27076fT/7TKRZs7QZc3Awx/bssyING4qEh3PcmTOLNGokcusWr9u0SSRTJpH8+UWy\nZOFj/HiR27fTZlyuAoBdksI5VmMAiuLGDBvGCtgaNawpkZUrM100Rw66WDp0APbs4Yr++nXqCEVE\ncEcwfjz97enSMbNm2zZmD126xOKyzZsZfLYlTz/NQG5QEH35AfG6lq1a0a9vyViqU4epnGPG8HmJ\nEvzXE1RPbYW6gBTFjTGGrpOXXqKLpFkzxgIKFqRrZ8sWunratWPxlaUu4J9/qJ1//DjF1a5cYYVt\n9+7MxGnYkP82bQp8+KHtNHXOn6fB+f13xis+/ZTH6tShxtG96aoivEcvL+b0e3pl76OiOwBF8QBi\nYuj/79KFk35EBCf44GAGiEeNohEwhjGBMmW4S5g2jT+3eTMbq6xYYe2XO2MGf27MGObjDxyYsrGc\nOQN8/z2zk65c4a7kpZe4C5k3jwVt+fLRr9+/Pw1B0aLA4cOJ3yc8nFo+Tz3F3c3x4wwWKylHdwCK\n4gFUrWrN91+0iIbAy4tSyV26cALdsoXuIT8/TvbjxjEAmzMn5RR++ME6+YeHM/OmeXMGYceM4WS8\nd+/9vYoTsm2bNQ1zxQoKs736KvDuu3yf996jcfjyS1bvenvTOJQpw3x/ixZQbCwDvd26Udtnzhxe\nU6pU2n6P7oYaAEXxADp04AS/cydX9j4+PH7jBiuIJ06kpEKpUiwYO3qU5xYt4g7hyhWuzgGqj/bp\nQ5fR22/zeWQkUyx79OB7tG9PrZ2E3LrFbKNp06hD5OfHyX/wYJ4vW5bun2LFKG2RNSt7F8TFUbq5\nVi26sGbOpDvq33+5i/nzT+CNN2iElEdDDYCieABZs3JSbd6c6ZMbNvB40aKcdMeM4b9vvEG3kK8v\nJ9/z5znhP/MMBeT69uVuIjiYu4LLl4EjR2hERBgT+PtvBmefeYaGxMKcOfzZZs14bceOFHPr0oWG\nZPlyFqWtXk3Jh7lzWQOwaxeNx8aNFLzr1o0Gx8+PRmfwYBqnpOQjlIeQ0nQhRzw0DVRRbMvmzUyl\nBEQKFBDJkUMkY0aRNWtETp4U8fUVGTtW5JlneP333/NYoUIiZcqIZM3KtNKBA0UiIkR8fEQOHxYp\nUoRpmdWqiXz9NX927FimlVro2VNk4kQ+375dpEQJkehokU6dRH79lcfr1uVY1q0TKVeOKasiIg0a\nMHVVRCQqSmT2bJEpU/iZlmsUgkdIA03VDsAY084Yc8gYE2eMSbYJsTEmyBhz1BhzwhgzJDWfqSjK\n41O7NlfYEyZwFT5hAgXievTgSrp8eWbefPEFMHasdSV+8yY1+PPm5ar/m28YBG7ZksHbokWZqTNj\nBn82MpK7gT17qMwJ8PO84mec334DXn6ZWT0FCjCACzA2IMKK39hYxhTi4vgeBQvympw56WJ6+WV+\nplb2Pj6pzQIKAfACgInJXWCMSQfgBwDPAQgHsNMYs1hEDif3M4qipC2vvkpf+4gRnFy9vRnwTZeO\nWTlNm1J7f/16GoWhQ+nzv3uXbhkvL+rtp0tHt9GSJXzfyEgaCz8/yjFcuQK8/joweTKDv5MmMU//\nyBGrGFu3bsz86dmTgWV/f07qlrTOpUuZZWSRfVZsR6oMgIiEAoB5sAmuDuCEiJyKv3YWgFYA1AAo\nigMJCuLjzBmu6n/7jT74vn25os+f33rtu+/Sz9+7N4vHgoKYGpovHyWYLXr9r71Gw/LBB0wrrVaN\nfv7y5fk+N29Sw2fvXu4OihWjoalbl48WLWg4YmJoJCIjWccwfbqu9NMCewSBCwMIS/A6PP5Ykhhj\nehtjdhljdl28eDHNB6conk6xYgzOvvoqM2rKlUs8+VsoW5aTc/XqXNV/9x1TMAMCqL/z6qvcCURG\nUmp6wwZO4qtXU9dHhLUC+/czAHznDou7fviBgelbt6jl06ULq36vXgXeeYcZSo0b2/978QQeagCM\nMWuMMSFJPFqlxYBEZJKIBIhIQN68edPiIxRFSYJSpYDWrVkxfPly4nMHD1JYbfRorvyffJJxg0yZ\n6EaaPp2T9MGDlKK+cIGN2cuXp5to92768oOD2YMgOprupMuXmQHUsCGlKEJD6eNfv55up7//pjFQ\n0oaHuoBEpFEqP+MfAEUTvC4Sf0xRFCfjxx85IZcowZTRQoVYkLV3L4uzunVjfUDDhsDUqawTaNqU\nE2Wz2HcAAAYWSURBVH6JEnTXdOzIeEGjRvTn37nDHUbt2kwlnTOHuv03bjC/f9QopnHu2cOUT2Po\n93/2WUd/G+6PERuIeBhj1gN4R0R2JXEuPYBjABqCE/9OAJ1F5NDD3jcgIEB27brvLRVFSWP+/ZdV\nwpcvA088wZ1B5szW8zNmUCPIGLqLgoO5G6hUiQ1jbtzgsW7dqPHfowfrAj7+mLuH8uW54vfxoTGp\nVYs7h9Kl+Vz9/Y+PMWa3iCSblZmQVAWBjTFtAHwHIC+ApcaYfSLSxBhTCMAUEWkmIjHGmAEAVgJI\nB+DnlEz+iqI4jvz5WeGbHF270iW0bx8zfU6eZKbP1q1sQ2nJ8PH1BcLC6MoJDWWxlrc3UKUKq5Ib\nNWKxWYsWjC0o9iW1WUALACxI4vg5AM0SvF4GYFlqPktRFOfCy4tZPgAzeL76ii4kEatmUOfOVtXR\n3r05+QM8HxfH4zExVslnxb6oGqiiKKkmXTpmEDVuTLdOSAiNw549zOUPDbW2erx0icdz5WK84JNP\nrAViin3Rr11RFJtQqhRTPHv0YNC4UyfWDPz2G+sESpVigLlFCxqJJk1YX9C5s6NH7rmoAVAUxWZk\nz27t2RsVRXlpf39g1izKRyxfTpXQAQMYF3hQnEFJe9QFpCiKzalfH9i0iZP+k0+yqKtkSTaj79uX\nhkJxPDZJA00rNA1UUdwDS+tGJe15lDRQdQEpipLm6OTvnKgBUBRF8VDUACiKongoagAURVE8FDUA\niqIoHooaAEVRFA/FqdNAjTEXAZyx0dvlARBpo/dydvRe3RdPul+918ejmIikqJmKUxsAW2KM2ZXS\n3FhXR+/VffGk+9V7TXvUBaQoiuKhqAFQFEXxUDzJAExy9ADsiN6r++JJ96v3msZ4TAxAURRFSYwn\n7QAURVGUBLiVATDGBBljjhpjThhjhiRx3hhjxsWfP2CMqeaIcdqKFNzv/+Lv86AxZqsxpoojxmkL\nHnavCa4LNMbEGGNetOf4bElK7tUY86wxZp8x5pAxZoO9x2hLUvB3nNMY86cxZn/8/fZ0xDhTizHm\nZ2NMhDEmJJnz9p+fRMQtHmDD+ZMASgDICGA/gAr3XNMMwHIABkANANsdPe40vt9aAHLFP2/qqveb\nkntNcN06sP/0i44edxr+Xn0AHAbgF/86n6PHncb3+x6Az+Kf5wXwH4CMjh77Y9xrXQDVAIQkc97u\n85M77QCqAzghIqdEJBrALACt7rmmFYDpQoIB+BhjCtp7oDbiofcrIltF5HL8y2AARew8RluRkt8t\nALwGYD6ACHsOzsak5F47A/hDRM4CgIi4+/0KgOzGGAMgG2gAYuw7zNQjIhvBsSeH3ecndzIAhQGE\nJXgdHn/sUa9xFR71Xl4GVxeuyEPv1RhTGEAbAOPtOK60ICW/1zIAchlj1htjdhtjutltdLYnJff7\nPYDyAM4BOAjgdRGJs8/w7Ird5ydtCekBGGPqgwagjqPHkoZ8C2CwiMQZ9+8+kh6AP4CGALIA2GaM\nCRaRY44dVprRBMA+AA0AlASw2hizSUSuOnZYro87GYB/ABRN8LpI/LFHvcZVSNG9GGMqA5gCoKmI\nXLLT2GxNSu41AMCs+Mk/D4BmxpgYEVlonyHajJTcaziASyJyA8ANY8xGAFUAuKIBSMn99gQwRugo\nP2GMOQ2gHIAd9hmi3bD7/OROLqCdAEobY54wxmQE0BHA4nuuWQygW3y0vQaAKyJy3t4DtREPvV9j\njB+APwB0dfHV4UPvVUSeEJHiIlIcwDwA/Vxw8gdS9ne8CEAdY0x6Y4w3gKcBhNp5nLYiJfd7Ftzt\nwBiTH0BZAKfsOkr7YPf5yW12ACISY4wZAGAlmFnws4gcMsb0iT8/AcwOaQbgBICb4MrCJUnh/X4I\nwBfAj/Er4xhxQXGtFN6rW5CSexWRUGPMCgAHAMQBmCIiSaYWOjsp/N2OBDDNGHMQzJAZLCIupxJq\njPkdwLMA8hhjwgF8BCAD4Lj5SSuBFUVRPBR3cgEpiqIoj4AaAEVRFA9FDYCiKIqHogZAURTFQ1ED\noCiK4qGoAVAURfFQ1AAoiqJ4KGoAFEVRPJT/AxlOwdDujFJdAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14781a8a828>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x, t, s=80, facecolors='none',  edgecolors='b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I think the noise is more noticeble in a regular plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1287,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x14781b87898>]"
      ]
     },
     "execution_count": 1287,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VNXdx/HPmZksJGTfF0ISSEjCDiEgsm8KPoq7UKQV\naymu1bZPtcujtrU+1tY+dav7glvVioILirKDrGENEJYkBBIIWSEJ2ZM5zx9JKSBLMDNzZ/m9X6+8\nyMy9yf0e0Pube+655yitNUIIITyPyegAQgghjCEFQAghPJQUACGE8FBSAIQQwkNJARBCCA8lBUAI\nITyUFAAhhPBQUgCEEMJDSQEQQggPZTE6wIWEh4frxMREo2MIIYTL2LJlS4XWOqIz+zp1AUhMTCQ7\nO9voGEII4TKUUoc6u690AQkhhIeSAiCEEB5KCoAQQngoKQBCCOGhpAAIIYSHkgIghBAeSgqAEEJ4\nKCkAAq01X+wsoaax5TvbvtlTyuHKegNSCSHsTQqA4Nu8Su5+byvvbjh8xvv1za3Me2cLf16y16Bk\nQgh7kgIgeG1tAQBbDh0/4/2dxdW0WTWr95fT0mY1IpoQwo6kAHi4vLJaVuwrx9tiYtvh42itT23b\neri9INQ2tpJdePx8v0II4aJsUgCUUq8rpcqUUrvOs10ppZ5RSuUppXYqpYbY4rji0p39Sf71bwvx\ntpi4Z3xvKuuaKapqOLVt2+ETxAV3w9tsYvneUkdHFULYma2uAN4ErrzA9qlASsfXXOAFGx1XdFJj\nSxuPfb6HjIe/4rnlB9BaU1XXzMdbi7l+cByT0qOA/3zq11qz7fBxRiSHMTw5lGV7y4yML4SwA5vM\nBqq1Xq2USrzALtOBt3R7/8IGpVSwUipGa11ii+OLC9t9tJoHPtjO/tKTZMQE8tev95NXdpIeoX40\ntli5fVQSvSK64+dtZuvh41w7OI6iqgYqTjYzOCGY1jYrj362h4MVdSSF+xvdHCGEjTjqHkAcUHTa\n6+KO94SdfbKtmGuf/5bj9S28MWcYX9w3iv++og8Ltx/l2eV5jE4JJzUqALNJMTA++NQVwLai9j+H\nJIQwIa396mC5XAUI4Vac7iawUmquUipbKZVdXl5udByXNn9dIQ98sIPMnqEsuX8M4/tEopTi7vG9\nefHWIfQI7cZ9E1NO7T+kZzC5JbXUN7ey9dBx/LzNpEZ1JyHMj5TI7nIfQAg346gFYY4APU57Hd/x\n3ndorV8GXgbIzMzU59pHXJjWmueW5/HUN/uZnBHFszMH4+tlPmOfK/vFcGW/mDPeG5IQQptVk1Nc\nzdbDJxgYH4zF3P4ZYUJ6JK+tOUhtYwsBvl4Oa4sQwn4cdQXwKfDDjtFAI4Bq6f+3veZWK4u2H+H6\nF9bx1Df7uX5IHC/MGvKdk//5DE4IAWBdfiW5JTUMTgg+tW1iWhStVs2aAxV2yS6EcDybXAEopf4J\njAPClVLFwCOAF4DW+kVgMTANyAPqgTm2OK74jxV7y/jvj3ZScbKJpHB//ji9L7OG98RkUp3+HaH+\n3iSF+/PPTYdptWqGdBQEgCEJwYT6e/PbT3LIOVLNrOEJxAV3o7KumUOVdcSH+BEV6GuPpgkh7MRW\no4BmXmS7Bu62xbHEd+06Us1d724lMdyfp24eyOje4Zd04j/d4IRgPt565NT3/2Yxm5g/J4vnVhzg\npVX5vLQqH38fC7WNrQAE+Fh47bZhZCWFdr1BQgiHcOpF4cXFlVQ38OP5mwn192b+7cOIDOjap/DB\nCSF8vPUIPcP8COvuc8a2/vFBvDQ7kyMnGnh/02FO1LeQFO5PbHA3/rJkL7Nf28gLtw45NWpICOHc\npAC4sJNNrdz+ZjZ1TW0suHN4l0/+0N7V0/5nyHn3iQvuxi+m9DnjvWGJIdz2xmbmvrWFp24eyPRB\nMspXCGfndMNAxXct2FLM00sPnDFPj9aahxbsZH9pLc/PGkKf6ACbHKtPVABjUyO4ZlDsJf1cWHcf\n3vvJcDITQ7j/g+0s3HbOQV5CCCciVwAu4B8r88gvryMiwIcfDE8A4OOtR/h8Zwn/fUUfxqZG2OxY\nFrOJ+bdnfa+fDfD14o3bspjz5iZ+/uF2zCbF1QMvrZAIIRxHrgCcXFlNI/nldXT3sfDIp7vYdvg4\nhyvreXjRLrKSQpk3tpfREc/QzdvM67cNI7NnKPd/sJ0PNxfR2NJmdCwhxDlIAXBy6wsqAXjh1iFE\nB/ly5ztbuff9bZhMiv+7ZRDm7znax578vC28PmcYg3sE86sFOxnw6Nfc/OJ6Xl1TgNUqz/YJ4Syk\nADi5DQVVBPhaGNkrnBdmDeV4fTM7ik7wp+v6Exfczeh459Xdx8K7PxnOG7cNY87lidS3tPLYF7n8\n4fM9Z9zLEEIYR+4BOLkNBZUMTwrFbFL0iwvipdlDyS+v4xoX6Fv3sZgZnxbJ+LRItNY89kUur609\niK+XmQev7INSznf1IoQnkQLgxI5VN3Kwoo5ZHTd+Acb1iWRcnwv8kJNSSvG7q9JpbGnjxVX5+Hmb\nz5iITgjheNIF5CS01tz84noeWrDz1HsbOvr/RySHGRXLppRS/HF6P64bHMffvtlPXlmt0ZGE8GhS\nAJzE+oJKNhVW8f7mIrYXnQDaC0Cgr4X0mECD09mOyaT49bQ0TIpTU04IIYwhBcBJvL62kFB/b8K7\n+/CnL9pvlK4vqCQrKcwpR/p0RWSAL2NSI1i47YiMChLCQFIAnMChyjqW7S1l1vAEfj45lc2Fx3lz\nXSGHKuu5rJd7dP+c7foh8RytbmTDwUqjowjhsaQAOIE31xViMSluHdGTmzPjSY3qzmNf5AIwItk9\nZ9eckhFFgI9FuoGEMJAUAIPVNrbwr+xiruofQ1SgLxazid9MS6fNqgnq5kV6tPv0/5/O18vMtP4x\nfJlTQn1zq9FxhPBIUgAM9mF2cfusnqOSTr03rk8k0wfFcv2QuO89r78ruH5IHHXNbXy9+7trDdc3\nt7JgSzEtbVYDkgnhGeQ5AAMdrKjjpVX5ZPYMYUB88Bnbnp4x2KBUjjMsMZT4kG4s2FrMtYP/M330\n0RMN3DE/mz0lNVjMSqaWFsJO5ArAINuLTnDDC+totWoevaav0XEMYTIprh8cx9q8Ch74YDvL95aS\nXVjF9Oe/5XBVPRaTYk9JjdExhXBbcgVggBV7y7jr3a2EB3gzf04WyRHdjY5kmDvGJFN+spnFOSV8\n0rGGQHxIN969Yzj3v7+d3BJ5WEwIe5EC4GBFVfX89J0tpER25405XV/C0dUF+nrxv9f35/fX9GVt\nXjl7jtYwMyuBsO4+pMcEsvpAudERhXBb0gXkYH9Zsg+Tgld/lOnxJ//TeVtMTEiL4p4JKafWIk6P\nCaC8tony2iaD0wnhnqQAONCOohN8uuMoPx6VREyQ807l7CwyOqbAyJX7AELYhRQAB9Fa86fFuYT5\nezvdKl7OKl0KgBB2JQXAQZbmlrHpYBX3T0ohwNfL6DguIcTfm5ggXykAQtiJFAAHOFRZx+OLc0kO\n92dGVsLFf0Cckh4TKCOBhLATGQVkR7WNLTy3Io831hZiMStemj0UL7PU3EuRHhPA6v3lNLa04etl\nNjqOEG5FCoCdnKhvZurTayipbuSGIfH86so+RAXKqJ9LlRETRKtVk1d2kn5xQUbHEcKtSAGwk693\nl1JS3cj827MYmxphdByXlR4TAMCekhopAELYmPRH2MnXe0qJC+7GmJRwo6O4tJ5h/nTzMsuNYCHs\nQAqAHdQ3t7LmQDmTM6JQyn1n83QEs0nRJzrgjALQ2NKG1rKSmBBdJQXADtYcqKCp1cqUjCijo7iF\njNhA9hytQWvNm98eZMCjX/PmukKjYwnh8qQA2MHXu0sJ6ubFsCT3XM3L0dJjAqlpbOWHr2/i0c/2\nYDLBCyvzaWptMzqaEC5NCoCNtbZZWba3lAlpkTLk00YyOm4Er8uv5NdT03hpdiZltU0s2n7U4GRC\nuDYZBWRj2YeOc6K+Rbp/bGhAfDA/HZvM5PQoMhND0VqTHhPIy6sLuHFIvFuvmiaEPclHVBv7encp\n3hYTY2Top814mU38emo6mYntXWpKKX46Jpm8spOs2FdmcDohXJcUABvSWvNN7jFG9Q7H30curuzp\nqgExxAb58tLqAqOjCOGypADY0O6jNRRVNTBZun/szsts4sejk9l0sIoNBZVGxxHCJUkBsKF3Nx7C\nx2Jiar9oo6N4hBnDehAR4MOtr27kkUW7qKprNjqSEC7FJgVAKXWlUmqfUipPKfXQObaPU0pVK6W2\nd3w9bIvjOpPqhhYWbjvKtYPiCPbzNjqOR/D3sbD4vtHMyOrBOxsPM/bJFazYK/cEhOisLhcApZQZ\neB6YCmQAM5VSGefYdY3WelDH1x+6elxn89GWYhpa2ph9WU+jo3iUiAAfHru2P0vuH02QnxevrJF7\nAkJ0li2uALKAPK11gda6GXgfmG6D3+syrFbNOxsOMSQhWCYsM0jvyACu6BtN9qHjNLbIA2JCdIYt\nCkAcUHTa6+KO9842Uim1Uyn1pVKq7/l+mVJqrlIqWymVXV5eboN49rc2r4KDFXX88LJEo6N4tJG9\nwmhutbL18HGjowjhEhx1E3grkKC1HgA8Cyw8345a65e11pla68yICNcYS//W+kLC/L2Z2l9u/hop\nKykUs0mxPl9GBQnRGbYoAEeAHqe9ju947xStdY3W+mTH94sBL6WUW8yTvOtINcv2ljEjqwc+Flmx\nykgBvl70jwtinRQAITrFFgVgM5CilEpSSnkDM4BPT99BKRWtOuZFVkpldRzXpf8v1VrzwebD3PDC\nOsK7+0j3j5MY2SuMHUUnONnUanQUIZxelwuA1roVuAdYAuQCH2qtdyul5iml5nXsdiOwSym1A3gG\nmKFdeEL3huY2fvHhDh5ckENmYgiL7xstyz06iZG9wmm1ajYXVhkdRQinZ5P5Cjq6dRaf9d6Lp33/\nHPCcLY5ltOZWK/Pe2cLqA+XcPymFeyekYJbJyJzG0J4heJtNrM+vZHyfSKPjCOHUZMKaS2C1an75\nrx2s2l/OE9f3Z0ZWgtGRxFm6eZsZnBDMuvwKo6MI4fRkKohO0lrz6Ge7+XTHUR68Mk1O/k5sZK9w\ndh+t4UR9+9QQWw5VkVd20uBUQjgfuQLopFfWFPDW+kPMHZPMvLHJRscRFzCydxj/txTe+LaQLYeO\nszavgrToAL66f4zR0YRwKnIF0AmbC6v481f7mNovml9PTZOF3p3cwPhgunmZeXrZAfaU1DA6JZy9\nx2opPl5vdDQhnIpcAVxE5ckm7n1vGz1CuvHnGwfIyd8FeFtM/GZaGjWNrfxoZCKlNY1MfGoVK/aW\nMVuG6wpxihSAC7BaNQ98uIOq+mY+uWskgb5eRkcSnXT6id4/3J/EMD+WSQEQ4gxSAM5S3dDCZzuO\nsrP4BNsOn+BA2Ukev64/fWNlkjdXpZRiYnoUb284RH1zK37e8p+9ECD3AM6gtWbe21v43cJdfL2n\nlNjgbvxhel9mZvW4+A8LpzYxLZLmVitrD8jwUCH+TT4KnWbJ7mOsL6jkd1el8+NRSdLf70YyE0MJ\n8LGwfG8ZU/rKpH1CgFwBnNLY0safFueSGtWd20YmysnfzXhbTIxJjWD53jKsVpedhUQIm5IC0OG1\ntQcpqmrgkav7YjHLX4s7mpAWSVltE7uP1hgdRQinIGc6oLSmkedX5DElI4rLe7vFLNXiHMb1iUAp\nWLa31OgoQjgFuQcAPLv8AK1tmt9elW50FGFHYd19GJoQwsurCyisqGNa/xjGpEbg6yXrOAjP5PFX\nAG1WzZc5x7iiXzQ9w/yNjiPs7C83DeTqAbGs2FfO3Le3cO3z39LSZjU6lhCG8PgCsL3oBJV1zUxK\nl6mDPUFSuD9/vnEA2b+bxGPX9mPvsVo+3lpsdCwhDOHxBWBpbikWk2JcqhQAT+JlNjFreAID44N4\nZlkeza1yFSA8j8cXgGW5pQxLDCXIT6Z58DRKKR6YnMqREw18tEWuAoTn8egCcLiynv2lJ5mUEWV0\nFGGQsakRDEkI5rnlB2hqbTM6jhAO5dEFYGlu+3BA6f/3XEopfj65D0erG/lwc5HRcYRwKI8vACmR\n3WX0j4e7vHcYWYmhPL8iX+4FCI/isQWguqGFTQermJgu3T+eTinFneN6caymkSW7jxkdRwiH8dgC\nsGp/Oa1WzeQM6f4R7fcCEkL9eHv9IaOjCOEwHlsAluw6Rpi/N4N6hBgdRTgBk0kxe0RPNhVWkVsi\ncwUJz+CRBWDXkWoW7yrhxsx4zCaZ9VO0uykzHh+LibfkKkB4CI8rAFpr/vDZHkL9vLl7fG+j4wgn\nEuznzbWD4li47QjVDS1GxxHC7jyuACzOOcamwip+MaWPrPErvmP2ZT1paGmTB8OER/CoAtDY0sbj\ni3NJiw7glmGyzKP4rn5xQQxJCOadDYdk4Rjh9jyqALy29iBHTjTw8H9lSN+/OK85lydxsKKOr2RI\nqHBzHlMAahtbeHFVPpPSIxkpi76IC5jWP4bkCH+eWXZArgKEW/OYAvDPTYepbWzl3gkpRkcRTs5s\nUtw7oTd7j9Xy9R5ZPUy4L48oAM2tVl5be5DLksMY2CPY6DjCBVw9IJak8ParAK3lKkC4J48oAAu3\nH6G0pomfjk02OopwERazibvH92ZPSQ3LcsuMjiOEXbh9AbBaNS+tyic9JpCxqRFGxxEu5NpBsSSE\n+vG0XAUIN+X2BWBpbin55XXMG5uMUjLyR3SexWzirnG9yDlSzcaDVUbHEcLm3LoAWK2a51fmEx/S\njav6xxgdR7igawfHEehr4b2Nh42OIoTNuXUBeH9zETuKTnDfxBQsZrduqrATXy8zNwyN58tdJVSe\nbDI6jhA25bZnxdKaRv73y1wuSw7jpqHxRscRLmzW8ARa2rRMDyHcjk0KgFLqSqXUPqVUnlLqoXNs\nV0qpZzq271RKDbHFcS/kkUW7aWq18vj1/aXvX3RJ78gAspJCeW/TYXkwTLiVLhcApZQZeB6YCmQA\nM5VSGWftNhVI6fiaC7zQ1eNeyJLdx/hq9zF+NjGFpHBZ7lF03azhCRyqrGddfqXRUYSwGVtcAWQB\neVrrAq11M/A+MP2sfaYDb+l2G4BgpZRd7srWNrbw8KJdpEUHMHeMjPsXtnFlv2hC/Lx4d6OsFSDc\nhy0KQBxQdNrr4o73LnUfm/DztjB3TC+euGEAXnLjV9iIj8XMTZk9+HpPKTnF1UbHEcImnO4MqZSa\nq5TKVkpll5eXX/LPm02KH49KYpBM+SBs7PbLk4gO9OWWl9ezYq88HSxcny0KwBHg9Mn14zveu9R9\nANBav6y1ztRaZ0ZEyJO7wnlEB/nyyV0jSY7w5463suXZAGEzJdUNvLfxME98uZe7393Kne9scchx\nbVEANgMpSqkkpZQ3MAP49Kx9PgV+2DEaaARQrbUuscGxhXCoyEBfPph7GWNSwvnNJzks2n7OzzFC\nXJKHFuTwm09yeG1tAXtKamhpc8xoM0tXf4HWulUpdQ+wBDADr2utdyul5nVsfxFYDEwD8oB6YE5X\njyuEUfx9LLzyw0xufmk9/7NwF1lJocQEdTM6lnBRdU2trM+v5EeX9eThq/s6dLEqm9wD0Fov1lqn\naq17aa3/1PHeix0nfzpG/9zdsb2/1jrbFscVwigWs4m/3TyIljbNrz7aKc8HiO9tXX4lzW1Wrugb\n7fCVCp3uJrAQriIx3J/fXpXOmgMVvL1BhoeK72fFvjL8vc1kJoY6/NhSAIToglnDExjXJ4L//TKX\nQ5V1RscRLkZrzcq9ZYxKCcfb4vjTsRQAIbpAKcWfbxhAa5vmvU0yKkhcmv2lJzla3ciEtEhDji8F\nQIguigr0ZWxqBJ9tPyr3AsQlWd7xPMm4PlIAhHBZ1wyK5Wh1I5sLZeEY0Xkr9pWRERNIVKCvIceX\nAiCEDUzOiMLP28yiHUeNjiJcRHVDC1sOHTes+wekAAhhE37eFqZkRLE4p4TmVqvRcYQLWHuggjar\nZnyacTMeSAEQwkamD4rjRH0Lq/df+hxWwvMs31tGsJ8Xg3qEGJZBCoAQNjIqJZxQf28WyvQQ4iJq\nG1v4alcJk9OjHP7w1+mkAAhhI15mE1f1j2Fpbiknm1qNjiOc2MJtR6hrbmPWiJ6G5pACIIQNTR8U\nS2OLlSW7jhkdRTgprTVvbzhE/7ggBsYHGZpFCoAQNjS0Zwg9w/z415aii+8sPNLmwuPsLz3JrSMS\nDF+vXAqAEDaklOLmzB5sKKiSqSHEOb294RABvhauGWiXRREviRQAIWzshiHxmBR8mC1XAeJM5bVN\nfLWrhJuG9qCbt9noOFIAhLC16KD2qSE+2lJMm0wNIWjv9y+rbeSVNQW0tGlmjUgwOhJggwVhhBDf\ndcuwHsx7Zyur95cz3sAnPYWxtNY88MF2vtp9jMaW9gcEx/WJoFdEd4OTtZMCIIQdTEiLIszfmw82\nF0kB8GBLc8tYuP0o1wyMJTMxhB4hfoxIDjM61ilSAISwA2+LiesGx/HmukIqTzYR1t3H6EjCwaxW\nzd++2U9imB9P3TwQL7Pz9bg7XyIh3MTNw3rQatXMenUjL67Kp6iq3uhIwoG+2n2M3JIafjYpxSlP\n/iAFQAi7SY0K4MkbB+BjMfHEl3sZ/eQK5q8rNDqWcIC2jk//vSO7O8Vwz/ORAiCEHd2c2YNF94xi\nza/GM6hHMG+uK0RrGRnk7j7bcZS8spM8MCnV0Ll+LkYKgBAO0CPUj5lZPThYUcfuozVGxxF21Npm\n5ellB0iLDmBqv2ij41yQFAAhHOSKvtF4mRWfyaIxbu2LnBIOVtRx/6QUTE786R+kAAjhMMF+3oxJ\nieDznSWydrCbslo1/1iRT+/I7kzJcO5P/yAFQAiHunpgLEdONLCt6LjRUYQdLNtbxr7SWu4a18vp\nP/2DFAAhHGpSRhQ+FhOf7SgxOoqwMa01z63IIz6kG9cMjDU6TqdIARDCgbr7WJiQFsnnO0tkniA3\nsz6/kh1FJ5g3thcWJx33fzZ5ElgIB7t6YCxf7jrGxoJKRvYONzqO+J5a2qz8Zck+GlvaCPT1YsW+\nMiIDfLhxaLzR0TpNCoAQDjYhLRJ/bzNz395CekwAadGBzMxKICM20Oho4hIsyy3j5dUFBPhaqG9u\no82q+f01ffH1Mn6a586SAiCEg/l6mXljThaf7ThKbkkNH20pZuX+Mlb+crxTPzQkzvSv7CKiAn34\n9sEJmE2KplarS538QQqAEIbISgolKykUgC92lnD3e1tZsbeMSRlRBicTnVFa08iKfWXcOe4//f2u\ndvIHuQkshOGm9I0iJsiXN2WeIJfx0ZZirBpuGtrD6ChdIgVACIN5mU3cOqIna/MqOFBaa3QccRFa\na/6VXcTwpFASw/2NjtMlUgCEcAIzsxLwtpiYv77Q6CjiIjYdrKKwsp5bhrn2p3+QAiCEUwj192b6\nwFgWbDlCdUOL0XHEBXyQXUSAj4Wp/WKMjtJlUgCEcBI/GplIQ0sb/8ouMjqKOI+axhYW55Rw9aBY\nunm73k3fs0kBEMJJ9IsLYlhiCPPXF8pTwk7qw81FNLZYmTkswegoNiEFQAgnctvIJIqqGlixt8zo\nKOIsrW1W3vi2kOFJofSPDzI6jk10qQAopUKVUt8opQ50/Blynv0KlVI5SqntSqnsrhxTCHc2pW8U\n0YG+cjPYCS3edYwjJxr4yehko6PYTFevAB4ClmmtU4BlHa/PZ7zWepDWOrOLxxTCbbUPCU1gzYEK\n8spkSKiz0FrzyuoCkiP8mZAWaXQcm+lqAZgOzO/4fj5wbRd/nxAeb0ZWAt5mE/PXHTI6iuiw8WAV\nOUequWNUskvM899ZXS0AUVrrf09sfgw433PsGliqlNqilJp7oV+olJqrlMpWSmWXl5d3MZ4Qrie8\nuw//NTCGBVuLqWmUIaHO4JXVBYT5e3P9kDijo9jURQuAUmqpUmrXOb6mn76f1lrTfqI/l1Fa60HA\nVOBupdSY8x1Pa/2y1jpTa50ZERFxKW0Rwm3cNjKR+uY2PsouNjqKxysoP8myvWXMvqynS873cyEX\nnQxOaz3pfNuUUqVKqRitdYlSKgY459AFrfWRjj/LlFKfAFnA6u+ZWQi3NyA+mCEJwTy97ED7Q2KD\nYlHKfboeXMmXu44B7U9ru5uudgF9Cvyo4/sfAYvO3kEp5a+UCvj398AUYFcXjyuE2/vrTQNJCvfn\n/g+2c9sbmymqqjc6kkdamlvKgPggogJ9jY5ic10tAE8Ak5VSB4BJHa9RSsUqpRZ37BMFrFVK7QA2\nAV9orb/q4nGFcHvJEd1ZcOdIHrk6g82FVVz/wjoamtuMjuVRymub2F50golp7jlNd5fWA9BaVwIT\nz/H+UWBax/cFwMCuHEcIT2U2KeZcnkSf6AB+8MpGFmwt5tYRPY2O5TFW7C1Da5iU4T5DP08nTwIL\n4QIuSw5jQHwQr689iFWmiXCYpbmlxAb5khHjnst1SgEQwgUopbhjdDIFFXUsk2kiHKKxpY01ByqY\nkB7ptjfgpQAI4SKm9YsmLrgbr6wuMDqKR1ifX0lDSxuT0t2z/x+kAAjhMixmE3MuT2RTYRU7ik4Y\nHcftLc0txc/bzIjkMKOj2I0UACFcyC3DehDgY+GVNXIVYE9aa5blljE6JdztHv46nRQAIVxIgK8X\nPxiewOKcEnkuwI52FFdzrKaRiW7c/QNSAIRwOXMuT8JsUnIVYCf1za08+NFOgv28mCwFQAjhTKKD\nfLlucBwfZhdRebLJ6DhuRWvNQwty2F9WyzMzBhPi7210JLuSAiCEC5o7pheNLVbmrys0OopbeePb\nQj7dcZRfTE5lTKr7T0YpBUAIF9Q7sjuTM6KYv/4QdU2tRsdxeXVNrby6poDHF+cyKT2Ku8b1NjqS\nQ0gBEMJFzRvbi+qGFt7fXGR0FJdV19TKX5fsY+QTy3nsi1wyE0P42y0D3WrRlwvp0lxAQgjjDO0Z\nQlZiKC+uyifAx8L4tEgiAnyMjuVSnl2ex0ur87kiI5qfjElmaM9zLmvutuQKQAgX9utpaXibTfxq\nwU6yHl+bfa0tAAAMPklEQVTKrFc3UCuriHXa0txSRvUO58XZQz3u5A9SAIRwaYMTQlj74HgW3zea\nu8f15tu8ShZskVXEOuNQZR15ZSfdapH3SyUFQAgXp5QiIzaQX17RhwHxQby36TDtK7SKC1neMame\nFAAhhFuYNTyB/aUnyT503OgoTm/53jJ6RfjTM8zf6CiGkQIghBu5emAsAT4W3tt42OgoTq2uqZWN\nBVUe/ekfpAAI4Vb8vC1cNySOL3JKOF7XbHQcp7U2r4LmNisT3HSpx86SAiCEm/nB8ASaW60s2Co3\ng89neW4ZAb4WMhM9b+TP6aQACOFm0qIDGdozhPc2ys3gc7FaNcv3lTEmNQIvs2efAj279UK4qVtH\nJFBQUce9/9wmE8bRvrj744tz+WJnCSv3l1Fe28SEPp7d/w/yJLAQbmn6wDiKqhp4dvkBvs2r4JGr\n+zJ9UKzbrm17Ie9vOsyvP8kB4N8XRErBuD7uP9nbxUgBEMINmUyK+yamcGW/aB5csJP7P9hObWML\nsy9LNDqaQ7229iB//HwPY1MjePYHgykoryO7sIruPhbCusu0GcqZ+wgzMzN1dna20TGEcGltVs2P\n529mfX4ln987ipSoAKMj2V1ZTSP/WJnPm+sKmdovmr/PGISPxX2XdjydUmqL1jqzM/vKPQAh3JzZ\npHjyxgH4+1i47/3tNLW2GR3Jbo6caOA3n+Qw6skVvLW+kNkjevLszMEec/K/VFIAhPAAkQG+PHnD\nAHJLanjq6/1Gx/leDpTW0thy/uLVZtXc+upGPsou5sah8az45Tj+eG0/LB4+0udC5G9GCA8xKSOK\nHwxP4OXVBWw6WGV0nEtSXd/CVc+s5cEFO8+7z5e7SjhYUcfTMwbx+HX9PXqKh86SAiCEB/ndVenE\nBPnyxJe5LvWMwJq8cprbrCzafpQNBZXf2a615h8r8kmO8OeKvtEGJHRNUgCE8CB+3hbum5jC1sMn\nTs2G6QpW7C0nqJsXccHdeGTRblrarGdsX32ggj0lNcwb28tjVvOyBSkAQniYG4fGkxjmx1+W7MNq\ndf6rAKtVs2p/OWNSI3j46gz2ldYyf13hGfu8sDKP6EBfrh0UZ0xIFyUFQAgP42U28cDkVPYeq+Xz\nnBKj41zUnpIaKk42MS41gikZUYzrE8Hflx6gqKoegK2Hj7OhoIo7RifhbZFT2qWQvy0hPNDVA2JJ\niw7gb1/v+053yoVUnmyiysGzjK7c195VNSY1AqUUj17dl+ZWK6OfXMHgP3zN3Le2ENTNi5lZCQ7N\n5Q6kAAjhgUwmxS+n9KGwsp6HF+2mofnizwZYrZqZr2xgyv+tpvh4vQNStlu5r5z+cUGnFrxPDPfn\nk7tH8ttp6VzZL4aUyO78Zloa/j4yscGlkr8xITzUxPRIbr88ide/PcjGg5U8ddNABiecf3rklfvL\n2F96ErNJcfubm/nozpEE+nrZNWN1fQtbDx/n7vG9z3i/b2wQfWOD7HpsTyBXAEJ4KKUUD1+dwbt3\nDKexuY0bXljHhL+uZMJTK5n8t1W8v+nMVcVeXl1ATJAvr982jILyOu56Z+sldR99H6sPlGPVME5m\n7rQLKQBCeLjLe4fz1QNj+MmYZDJiA0mPCcTLbOJ3C3exs/gEADnF1WwoqOL2y5MYmxrBEzcMYG1e\nBf+zcJdds63cV06wnxeDegTb9TieSrqAhBAE+nrx66npp15X17dw5dOruf/97Xx+3yheWVNAdx8L\nt2T1ANqHkh6sOMnzK/IZ0jOEmzN72DzTyaZWVu0vZ3RKBGYZ228XXboCUErdpJTarZSyKqXOO/uc\nUupKpdQ+pVSeUuqhrhxTCGF/QX5ePHXzQA5W1vHAB9v5IqeEmVk9zujz//nkPozsFcb/LNxFbknN\n9z5WbkkNmY99w88/3M6+Y7UALN1TyuS/raKyronrB8vYfnvpahfQLuB6YPX5dlBKmYHngalABjBT\nKZXRxeMKIexsZK9w5o5OZsnuUhQw5/KkM7abTYqnZwwmqJsXd7+7ldrGFrTWHD3RwJETDZ0+zj9W\n5nOyqZUvc45xxd9Xc+XfV3PHW9kE+nqx4M6RjE+T/n976VIXkNY6F7jYKkNZQJ7WuqBj3/eB6cCe\nrhxbCGF/P5+Sys7iatJiAogN7vad7REBPjw7czAzX9nA1KfXUNPQQk1jKz4WE4vuuZy06MAL/v6i\nqnoW55Rwx6gk5o3txdsbDrE4p4RfTkll7phe8mCXnTniHkAcUHTa62JguAOOK4ToIh+Lmfd+cuH/\nXYcnh/H76f34bPtRxqZGkBoVwHMr8rjnvW18es/l+Hmf/zTz2tqDmFT71UWIvzf3TUzhvokptm6G\nOI+LFgCl1FLgXNPr/VZrvcjWgZRSc4G5AAkJ8mSfEEbrzDrCs0f0ZPaInqde947szq2vbeTRT3fz\n5I0Dz/kzx+ua+WBzEdMHxREd5GuzvKLzLloAtNaTuniMI8DpQwTiO9473/FeBl6G9iUhu3hsIYQB\nLu8dzj3je/Ps8jxG9grn2nPcyH17wyEaWtqYOybZgIQCHPMcwGYgRSmVpJTyBmYAnzrguEIIA/1s\nYgpZiaE89PFOPtpSfMb6A3VNrcxfV8iEtEhSPWCNYmfV1WGg1ymlioHLgC+UUks63o9VSi0G0Fq3\nAvcAS4Bc4EOt9e6uxRZCODuL2cTzs4YwMD6YX/5rB/d/sJ1j1Y28tCqfsX9ZSWVdM/PG9jI6pkdT\nzrwqUGZmps7OzjY6hhCiC9qsmn+syOPvyw7Q1rH+wOiU9i6i4clhBqdzP0qpLVrr8z6XdTp5ElgI\nYVdmk+LeiSmM7B3GlznH+K+BsTK1g5OQAiCEcIihPUMZ2jPU6BjiNPKUhRBCeCgpAEII4aGkAAgh\nhIeSAiCEEB5KCoAQQngoKQBCCOGhpAAIIYSHkgIghBAeyqmnglBKlQOHvuePhwMVNozjCqTN7s/T\n2gvS5kvVU2sd0ZkdnboAdIVSKruz82G4C2mz+/O09oK02Z6kC0gIITyUFAAhhPBQ7lwAXjY6gAGk\nze7P09oL0ma7cdt7AEIIIS7Mna8AhBBCXIBLFwCl1JVKqX1KqTyl1EPn2K6UUs90bN+plBpiRE5b\n6kSbZ3W0NUcptU4pNdCInLZ0sTaftt8wpVSrUupGR+azh860WSk1Tim1XSm1Wym1ytEZba0T/20H\nKaU+U0rt6GjzHCNy2opS6nWlVJlSatd5ttv//KW1dskvwAzkA8mAN7ADyDhrn2nAl4ACRgAbjc7t\ngDaPBEI6vp/qCW0+bb/lwGLgRqNzO+DfORjYAyR0vI40OrcD2vwb4M8d30cAVYC30dm70OYxwBBg\n13m22/385cpXAFlAnta6QGvdDLwPTD9rn+nAW7rdBiBYKRXj6KA2dNE2a63Xaa2Pd7zcAMQ7OKOt\ndebfGeBeYAFQ5shwdtKZNv8A+FhrfRhAa+3q7e5MmzUQoJRSQHfaC0CrY2PajtZ6Ne1tOB+7n79c\nuQDEAUWnvS7ueO9S93Ell9qeH9P+CcKVXbTNSqk44DrgBQfmsqfO/DunAiFKqZVKqS1KqR86LJ19\ndKbNzwHpwFEgB/iZ1trqmHiGsPv5S9YEdlNKqfG0F4BRRmdxgL8DD2qtre0fDj2CBRgKTAS6AeuV\nUhu01vuNjWVXVwDbgQlAL+AbpdQarXWNsbFclysXgCNAj9Nex3e8d6n7uJJOtUcpNQB4FZiqta50\nUDZ76UybM4H3O07+4cA0pVSr1nqhYyLaXGfaXAxUaq3rgDql1GpgIOCqBaAzbZ4DPKHbO8jzlFIH\ngTRgk2MiOpzdz1+u3AW0GUhRSiUppbyBGcCnZ+3zKfDDjrvpI4BqrXWJo4Pa0EXbrJRKAD4GZrvJ\np8GLtllrnaS1TtRaJwIfAXe58MkfOvff9iJglFLKopTyA4YDuQ7OaUudafNh2q94UEpFAX2AAoem\ndCy7n79c9gpAa92qlLoHWEL7CILXtda7lVLzOra/SPuIkGlAHlBP+ycIl9XJNj8MhAH/6PhE3Kpd\neCKtTrbZrXSmzVrrXKXUV8BOwAq8qrU+53BCV9DJf+c/Am8qpXJoHxnzoNbaZWcJVUr9ExgHhCul\nioFHAC9w3PlLngQWQggP5cpdQEIIIbpACoAQQngoKQBCCOGhpAAIIYSHkgIghBAeSgqAEEJ4KCkA\nQgjhoaQACCGEh/p/BgWR4W805rIAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14781b18160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(x, t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Polynomial Curve Fitting\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will attempt to learn the function y given a synthetic dataset (x, t). We will assume that y is a polinom of degree M and try to learn the sin(2pix) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 981,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def polynom(val, M):\n",
    "    return np.array([val**i for i in range(M+1)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 982,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def OptimizeLS(x, t, M):\n",
    "    cols = M+1\n",
    "    rows = len(t)\n",
    "    \n",
    "    phi = np.array([polynom(x[i], M) for i in range(rows)])\n",
    "    prod = np.dot(phi.T, phi) \n",
    "    m = np.dot(np.linalg.inv(prod), phi.T)   \n",
    "    w = np.dot(m, t)\n",
    "    return w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 983,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def reverseOptimizeLS(x, t, M):\n",
    "    return OptimizeLS(x, t, M)[::-1] #reverse the coefficiants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1455,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_polyfit(x, t, degs, org_func, polyfit_func):\n",
    "    \"\"\"\n",
    "    Plots the original function, x points against org_func(x) with noise, xpoints against predicted polynoms\n",
    "    with different degrees \n",
    "    \"\"\"\n",
    "    \n",
    "    colors = ['b', 'g', 'r', 'c', 'm', 'y', 'k']\n",
    "    \n",
    "    x_sin = np.linspace(0, 1, 101)\n",
    "    plt.plot(x_sin, [org_func(xi) for xi in x_sin], c=colors[0])#original function\n",
    "    plt.scatter(x, t , s=80, facecolors='none',  edgecolors=colors[1])#noisy function\n",
    "    \n",
    "    for i, M in enumerate(degs, 2):\n",
    "        w = polyfit_func(x, t, M)\n",
    "        p = np.poly1d(w)\n",
    "        y = np.array([p(x[j]) for j in range(len(x))])\n",
    "        plt.scatter(x,y, marker='.', c=colors[i])#polynoms\n",
    "        \n",
    "    plt.legend(['sin(2px)', 'noisy sin(2px)']+['M='+str(M) for M in degs], loc='upper right')\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1123,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xdc1dUbwPHPYTu5Llw4cE9ARQNNhRRzj9Ryr8qyrNwj\nNVeOn1ZaWZnlwDJH5cqNKLlQQxPFlaaQO0XvdYHAvef3xxcJFQXhci8Xzvv14nXv/c5Hrftwzvec\n5wgpJYqiKIrykJ21A1AURVGyF5UYFEVRlEeoxKAoiqI8QiUGRVEU5REqMSiKoiiPUIlBURRFeYRK\nDIqiKMojVGJQFEVRHqESg6IoivIIB2sHkBFFixaV5cuXt3YYiqIoNuXQoUM3pJTF0jrOJhND+fLl\nCQ8Pt3YYiqIoNkUIEZ2e41RXkqIoivIIlRgURVGUR6jEoCiKojzCJp8xKIpiPQkJCVy8eJG4uDhr\nh6I8hYuLC+7u7jg6OmbofJUYFEV5LhcvXqRAgQKUL18eIYS1w1EeI6UkJiaGixcv4uHhkaFrqK4k\nRVGeS1xcHEWKFFFJIZsSQlCkSJFMtehUYlAU5bmppJC9ZfbfR3Ul5QJSSiKuRXDu1jl0Ljoal22M\no33G+h4VRcn5zNJiEEIsEkL8K4SIfMp+IYT4QghxVghxVAhRN8W+lkKI00n7xpgjHuU/e//ZS/3v\n6tNpZSeWRixlbMhYys0tx7yD81DrfSs5xRtvvMGJEyfSPG7u3LksXboUgJEjR1KtWjU8PT3p1KkT\ner0+Q/e+fv06LVu2zNC52ZaUMtM/QBOgLhD5lP2tgc2AAHyBA0nb7YG/gQqAExAB1EjrfvXq1ZNK\n2vZE75FFZxWVqyJXyURjYvL2o1ePytpf15aTQydbMTrFVp04ccLaIWRIQkKCrF27tkxISJBSSrl1\n69bk96NGjZKjRo3K8LX79esn9+zZY5Y4zSW1fycgXKbjO90sLQYp5S7g5jMO6QAsTYptP6ATQpQE\nGgBnpZTnpJTxwIqkYxUzGLp1KF+1/oquNbtib2fP3uDLBI2K4PbRImzrvY05++dw+c7lZ15DSrh7\nF65dg/v3tc+KYk337t2jTZs2eHl5UatWLVauXIm/v39ymZz8+fMzbtw4vLy88PX15dq1awDs2LGD\nunXr4uCg9aC3aNEi+b2vry8XL14EYMmSJXTo0AF/f38qV67M5MmTAfjjjz/w9PQkLi6Oe/fuUbNm\nTSIjtU6Sjh07smzZMov+PWQlSz1jKA1cSPH5YtK21La/YKGYcrSj147y771/6Vy9MwB7gy9zd9Qa\nytQ8wt1R3vw9qxOv1XyNoCNBjG08FoCbN2HbNggPhz//hMhIbVti4n/XdXKCYsXA0xPq1IEGDaB5\nc8iX79H734u/x/LI5aw/vZ7YxFhqFavFWz5vUa1oNUv9FSgWMGQIHDli3mt6e8PcuU/fv2XLFkqV\nKsXGjRsBMBgMfPPNN8n77927h6+vL9OmTWPUqFF89913jB8/nr1791KvXr1Ur7lo0SJee+215M8H\nDx4kMjKSvHnzUr9+fdq0aUP9+vVp374948ePJzY2ll69elGrVi0AfHx8GD9+vBn+9NmDzTx8FkIM\nBAYClC1b1srRZH/nbp3Ds7gn9nb2APzzx3ZKzBiOcEzAKcGRf/4oQJ0Wddh/LpIFC+CXX2DnTi0J\nODtrX/zt2kHx4qDTQd68Wsvh1i24ckX7Mti2DYxGcHGBli3h1Vehc2c4oz9Oq2Wt8C7hTS/PXrg6\nu7L7n900XdKU9xq8x/gmOed/IMXyateuzfDhwxk9ejRt27alcePGj+x3cnKibdu2ANSrV4/g4GAA\nrly5QvXq1Z+43rRp03BwcKBnz57J2wIDAylSpAgAr7zyCnv27MHHx4ePPvqI+vXr4+LiwhdffJF8\nvJubG5cvP7v1bUsslRguAWVSfHZP2ub4lO1PkFIuABYA+Pj4qA6NNBRyKcSVu1eSP5d+4SRGEsDe\nBDIBp4qXWPrpixza0Jsld6FyZRgxAjp2hLp1IT0TJuPiICwMVq+GNWtg7VooWcrE/Tq/MXP0bN5u\n/N9vYC9Xepn3GryHf5A/HjoPenr2fMaVFVvxrN/ss0qVKlU4fPgwmzZtYvz48TRr1uyR/Y6OjsnD\nNe3t7UlMavLmyZPnibH9S5YsYcOGDYSEhDwyxPPx4Z4PP8fExHD37l0SEhKIi4sjX1JTOS4ujjx5\n8pj3D2pFlprHsB7okzQ6yRcwSCmvAH8AlYUQHkIIJ6Bb0rFKJjUq24ird68ScTUCAC+f9gg7J+7e\n0bFg4Sx69BnNvpUNaRzwgLAwOH0aZsyAF15IX1IAraUQEABffgn//ANbtoCuzGUMG8cwqs1rzJ4N\n53duJyJoFJf3BlM8f3HmtZrH//b+T42IUjLs8uXL5M2bl169ejFy5EgOHz6crvOqV6/O2bNnkz9v\n2bKFWbNmsX79evLmzfvIscHBwdy8eZPY2FjWrl1Lo0aNAHjrrbeYOnUqPXv2ZPTo0cnH//XXX8nd\nSjmBWVoMQojlgD9QVAhxEZiI1hpASjkf2IQ2MukscB/on7QvUQgxGNiKNkJpkZTyuDliyu0c7BwY\n13gcPVb3ILh3MCUL+nH8xDEmTSrBjRv5qRiwh3IdlhD8wfdmuZ+dHbz8MlSIeZv+zu/z++IWjBoF\nn5cqz7vvnMKv+Bew9zdeaticmNgYzuvPU6FQBbPcW8ldjh07xsiRI7Gzs8PR0ZFvvvmGESNGpHle\nq1at6N27d/LnwYMH8+DBAwIDAwHtAfT8+fMBaNCgAZ07d+bixYv06tULHx8fli5diqOjIz169MBo\nNNKwYUN27NjBSy+9xM6dO2nTpk3W/IGtIT1Dl7Lbjxqumj4mk0lO3zVdFvywqnSve0yClO41oqXb\nkDay7U9t5e2422a/50tBL8ngv4OllFJ+New7WbbsCQlStnx5kdz7zXgppZTV51WXkdcizX5vxTJs\ndbiqlFJ27NhR/vXXX888ZvHixfLdd999rus2btxY3rx5MzOhmV1mhqvazMNn5fkJIageMxb7+aO5\netdIo7eX07DTEbrVnkLdknXTvkAG1CxWk93Ru2leoTkdXylHZf8GLF0xhp9WjOHE0Qd8UfEGV+9e\npZyuXJbcX1GeZebMmVy5coXKlSub7ZrXr19n2LBhFCpUyGzXtDYhbbCv18fHR6qlPZ/NaITx42Hm\nTKhXD378EapZYKTo8X+P02xpM44NOkaxfMW4vDeY62eDibrfhaGzGxAVbaTRwFXs/qZ71gejZImT\nJ0+mOrpHyV5S+3cSQhySUvqkda4qopcD3b4NHTpoSeGtt2DfPsskBYCabjV5q95bBAQFEBoVSsmG\nzfHqO4sXepel5ewx5Km+iz3zu/P22xAfb5mYFEV5PqorKYe5eFF7CHz6NHz9NQwaZPkYJvlPwqOQ\nB+9sfIc78XdwdXbl0p1LdK3RlbP7avLFDC1pnT4N69ZBwYKWj1FRlKdTiSEHOXMGAgP/m8H80kvW\niUMIQT/vfvT16svft/4mNiGW8rryFHAuAGjDYmvWhP79tRi3bIGiRa0Tq6IoT1JdSTnE0aPQuDHc\nu6fNYLZWUkhJCMF1+2JsiCtIZJzpkX29emkT4o4fhyZN4FKq0xqVnOD8rfOEXQjj/K3z1g5FSSeV\nGHKAo0fB3x8cHGD3bu1hc3YQZjDQLCKCCefP0ywigjCD4ZH9bdrA1q1a95dKDjnPruhdNFncBL+F\nfgzdOhS/hX40WdyEXdG7LBrHRx99xPbt281+3fDwcN5///00j4uNjaVp06YYjUaOHDmCn58fNWvW\nxNPTk5UrV2b4/iNGjGDHjh0ZPv+Z0jOmNbv9qHkM/zl1Sko3Nynd3aU8d87a0TxqelSUtN+5U7Jz\np7TfuVNOj4pK9bgDB6QsUEDKatWk/PdfCwepPLf0zGPYfGazdJvtJpcfWy7jE+OllFLGJ8bL5ceW\nS7fZbnLzmc1ZHWa2MW/ePDl37lwppZSnT59Onkdx6dIlWaJECXnr1q0MXTcqKkoGBgY+db/Vy24r\n1hEVpVU2Bdi+HTK47neW8dfpcLKzwx5wsrPDX6dL9bgGDWDDBoiOhhYtIIPrpSjZRKIpkYG/DWRV\nl1V0q9UtebVAR3tHutXqxsouKxn420ASTYlpXOlJUVFRVK9enTfffJOaNWvSokULYmNjAThy5Ai+\nvr7JC+/cunULgH79+vHLL78AMGbMGGrUqIGnpycjRozgzp07eHh4kJCQAMDt27cf+fzQzz//TK1a\ntfDy8qJJkyYAhIaGJhfrmzRpEgMGDMDf358KFSo8UmBv2bJldOigrSZQpUqV5DkUpUqVws3NjevX\nrwNQvnx5Ro0aRe3atWnQoEFy+Y4OHTokLy707bffJhf7K1euHDExMVy9evW5/x7TlJ7skd1+VItB\n+826YkUpCxWSMiLC2tE83T69Xk6PipL79Po0j92yRUpHRykbNpQyNtYCwSkZklaLYf2p9dL3e99n\nHuP7va/87fRvz33v8+fPS3t7e/nnn39KKaXs2rWr/OGHH6SUUtauXVuGhoZKKaWcMGGC/OCDD6SU\nUvbt21f+/PPP8saNG7JKlSrSZDJJKWXyb+r9+vWTa9askVJK+e2338phw4Y9cd9atWrJixcvPnLe\nzp07ZZs2baSUUk6cOFH6+fnJuLg4ef36dVm4cGEZHx8vHzx4IIsXL57qn+XAgQOyWrVq0mg0Siml\nLFeunPz444+llFIGBQUlX/vq1auyYsWKcteuXbJy5coyJiYm+RpvvPGG/OWXX1K9vmox5DKxsdo8\nhUuXYNMmrUR2duXn6srYcuXwc3VN89iXX4afftLmXfTtCyZTmqco2dCpG6do6N7wmcc0dG/Iyesn\nM3R9Dw8PvL29Aa2sdlRUFAaDAb1eT9OmTQHo27cvu3Y9+izD1dUVFxcXXn/9dVavXp1cOO+NN95g\n8eLFACxevJj+/fs/cc9GjRrRr18/vvvuO4xGY6pxtWnTBmdnZ4oWLYqbmxvXrl3jxo0b6FJpKV+5\ncoXevXuzePFi7Oz++xru3r178mtYWBgAxYsXZ8qUKQQEBPDpp59SuHDh5OOzqty3Sgw2xmTSvjT3\n79dmM/v6Wjsi8+rSBWbNglWrYNw4a0ejZEQ+p3zExMY885iY2BjyO+XP0PWdnZ2T36csq50WBwcH\nDh48SJcuXdiwYUPyOs2NGjUiKiqK0NBQjEZjqlVS58+fz8cff8yFCxeoV68eMTFP/vlSiyu1Ut+3\nb9+mTZs2TJs2Dd/H/gd+WunvY8eOUaRIkSeSQFaV+1aJwcaMGwc//6x9eXbubO1ossaIEdqM7Zkz\n4XvzFH9VLKh91fasP72eOw/upLr/zoM7rD+9nvZV25vtnq6urhQqVIjdu3cD8MMPPyS3Hh66e/cu\nBoOB1q1bM2fOHCIiIpL39enThx49eqTaWgD4+++/eeGFF5gyZQrFihXjwoULqR73uEKFCmE0GpOT\nQ3x8PJ06daJPnz506dLlieMfjlJauXIlfn5+gLaa3ObNm/nzzz/55JNPOH/+v2G/WVXuWyUGG7Jy\npfZlOXAgDB9u7WiyjhAwb572IPqdd7TWkWI73Au607FaR15f/zrxxkfrnsQb43l9/et0rNaR0gVL\nm/W+QUFBjBw5Ek9PT44cOcJHH330yP47d+7Qtm1bPD09efHFF/nss8+S9/Xs2ZNbt24ld+U8buTI\nkdSuXZtatWrRsGFDvLy80h1XixYt2LNnDwCrVq1i165dLFmyBG9vb7y9vTmSYm3UW7du4enpyeef\nf86cOXN48OABb775JosWLaJUqVJ8+umnDBgwACklCQkJnD17Fh+fNEsfPTdVRM9GREZqi+jUqQM7\ndmhrL+d0t26Bj4+2UtyhQ1CihLUjUiB9RfRiE2Lp/mt3Tlw/wcB6A6lUuBJnb57l20PfUrNYTZZ3\nXk4ex+yz4tkvv/zCunXr+OGHH8x+7cOHDzNnzpw0r12+fHnCw8Mpms4yAGvWrOHw4cNMnTo11f2Z\nKaJnroV6WgKfoy22872UcuZj+0cCD9dydACqA8WklDeFEFHAHcAIJKYn6NxGr4dOnbSaQj//nDuS\nAkChQtqSob6+2nrSISHpX11Osa48jnlY89oa9l/cz5IjS9gVvYuS+UuytONSfN19n1g605ree+89\nNm/ezKZNm7Lk+nXr1iUgIACj0Yi9vb3ZrpuYmMjwrOo6SM/QpWf9oCWDv4EKgBMQAdR4xvHtgB0p\nPkcBRZ/nnrlpuKrJJGW7dlI6OEi5Z4+1o7GOZcukBCmHDLF2JIqUtr1QT25i7eGqDYCzUspzUsp4\nYAXQ4RnHdweWm+G+ucLnn8Nvv8Gnn0LSsrO5To8e8P772sLzv/1m7WgUJeczR2IoDaR8RH8xadsT\nhBB5gZbAryk2S2C7EOKQEGKgGeJ5PmFhWrnPpDHDFrmlwcCM6Ognagc97tAhGDVKm7Pw3nsWCi6b\nmjVLe77Sv79WW0lRlKxj6bLb7YC9UsqbKba9KKW8JIRwA4KFEKeklE9U2UpKGgMBypYta55owsKg\nWTNtxRgnJ60TO2mIWFZ5WFgu3mTCyc6OEC+vVCd/3bkD3bpB8eKwcKE2Uic3c3aGFSugbl2tMmtI\nCJixu1ZRlBTM0WK4BJRJ8dk9aVtquvFYN5KU8lLS67/AGrSuqSdIKRdIKX2klD7FihXLdNAAhIZq\nScFo1F5DQ81z3WfdUq8n3mTCCMSbTIQ+pTDQ4MFw7pw2E7hIkSwPyyZUqaItPvT77zB9urWjUZSc\nyxyJ4Q+gshDCQwjhhPblv/7xg4QQrkBTYF2KbfmEEAUevgdaAJFmiCl9/P21loK9vfbq75/1t0xH\nYblff4WlS7U1mxs3zvKQbEqfPtCzJ0yZArlsxLKSghCCXr16JX9OTEykWLFiyUXt0mPAgAG4ubll\nyQQxW5fpxCClTAQGA1uBk8AqKeVxIcTbQoi3UxzaCdgmpbyXYltxYI8QIgI4CGyUUm7JbEzp5uen\n9UlMnWqRbiTQageFeHkx1cMj1W6ka9e0Wb/16mmJQXnSvHlaF1ufPlrdKCX3yZcvH5GRkcmVVYOD\ngyld+vkmzPXr148tWyz3dWNLzPKMQUq5Cdj02Lb5j31eAix5bNs5IP1TCLOCn59FEsIjt3R1TfW5\ngpTw5ptw9y788IMas/80Oh0sXqzNjB4/XhuxpeQ+rVu3ZuPGjXTp0oXly5fTvXv35JIY6dGkSROi\noqKyLkAbpkpiWJCUktCoUMbvGM/Y7WNZc3LNIzXplyzRhmPOmAFpTCzN9QIDtXIZc+ZozxyUbC4L\nRv9169aNFStWEBcXx9GjR3nhhReS9+3cuTO55ETKn4YNn131VdFYelRSrnXu1jk6r+rMg8QHdKnR\nBRcnFz4J+4QhW4ewsstKytr5MnQoNG0KH3xg7Whtw6xZsG0bDBgAx45BUhVlJbvJotF/np6eREVF\nsXz5clq3bv3IvoCAgEdqECnPRyUGC9DH6Wm+tDlDfIfwXoP3kssBjG8ynt9O/0a7n9rjFfo3Dx4U\n4PvvwU6149IlXz6t+qq/P0ycCLNnWzsiJVWpjf4zU/dt+/btGTFiBKGhoY+Uwt65cydDhw594vi8\nefOyb98+s9w7J1OJwQIWHl6Ir7sv77+gLRxuCDOgD9Wj89fRzq8dL8V+zarNBZg1CypVsnKwNqZp\nU63a7GefafWU6te3dkTKEx6O/nvYYjDj6L8BAwag0+moXbs2oSmGm6sWQ+ao300tYNmxZbxV7y1A\nSwpH3l3G+RMfc+TdZURtvc2OrzphV+owHwxRS5ZlxKxZWuXV11/XvnuUbCYLR/+5u7vz/vvvZ+jc\n7t274+fnx+nTp3F3d2fhwoVmi8vWqRaDBcTExlDWVZutfTU8BDl9GDgmIBMcGT6xNfpbBbEf+BZG\ndgHZpxSxrXB1hW++0UqH9B1xjH99hhD5byR5HfPSvkp7BjcYTOUila0dZu5m5tF/d+/efWKbv78/\n/s/RGlm+XJVsexrVYrCAMgXLcPz6ce2DdwQ4JoC9iaMnX2D1gfIMeOcWBcudx8XBxbqB2rDWbRNx\n9w1j5fzKdHQbxpG3jrC552YKOBeg4aKGbD271dohKorNUInBAgbUGcDnBz5HSkkJz5YIe2cSE5yZ\n88XXlCkTB02nMKDOgGxVo97WfLLvE8q9Npd8zs5s+bINJfKXpFrRanz80ses67aOnqt7cuP+DWuH\nqSg2QSUGC+hRuwd3HtzhnY3vIFxq4u04l5Dv5xEVVZOX3t3I1n/WMMxvmLXDtFmJpkTmHZzHvG5j\nmTJFsGkT/PjjaaKjZ2AwhNGwTEPaVW3HwsOqD1lR0kMlBgtwcXBha6+tGB4Y6DK0NNcCZvLFqu60\nFWspeG8mv/f7nRL51bqVGXX25lmcHZzxLuHNe+9BrVr3GD48HydOzCQiohkGQxidq3dmZ9ROa4eq\nKDZBJQYLcXVx5afOP/FzscGMTtBqOMwTQ/kizyuU05WzcnS2TUqJvdBqcDs4wOTJv3L9ujtBQeMx\nmeLR60OxF/aYpBr1pSjpoRKDhR0s3Ju1dGKcmEE552sWqeia01UoVAHDAwOnb5wGwLd4SVq1WMKv\nv37AP2er43jRhw1/baBhGVUOQVHSQyUGC0pIgA++rUHF0nEMm1TQYhVdczpnB2ferPsmY0PGYjQZ\nSdhVhTf+MeEsEvlq6AYubHRlxfEVvFn3TWuHqphJZstux8XF0aBBA7y8vKhZsyYTJ07MqlBtkkoM\nFjRvHpw8CXO+ccHlo1EqKZjR+Cbjuf3gNq2WteJ81fMUja5E38TLHLxbjv4Rv/JV668oXfD5yjIr\n2Vdmy247OzuzY8cOIiIiOHLkCFu2bGH//v1ZFa7NUYnBQq5dg0mToFUreI61RJR0cnFwYWOPjbSv\n2p7BVwfzbs93ufPSDAoXu8rt45PpWKmbtUNUzOxh2W0guex2egkhyJ8/PwAJCQkkJCSo4eIpqMRg\nIePGaYvKzJmj1m/OKs4OzgxuMJjIdyKJWBDB0u0LWbGsBJeiXZg719rR5W5hBgMzoqMJMxjMds3M\nlt02Go14e3vj5uZGYGDgI+fndmYpiSGEaAl8DtgD30spZz623x9tSc/zSZtWSymnpOfcnCAiAhYt\ngiFDoGpVa0eTOxy4fYdQvR7/Bjrat3dl+nTo319b+U2xrDCDgWYREcSbTDjZ2aW6cmFGZLbstr29\nPUeOHEGv19OpUyciIyPVMp9JMp0YhBD2wFdAIHAR+EMIsV5KeeKxQ3dLKdtm8FybJSUMGwaFCsGE\nCdaOJnd4/Ito0YQ6bNpUgIkTYf78tM9XzCtUryfeZMIIxJtMhOr1ZkkMYJ6y2zqdjoCAALZs2aIS\nQxJztBgaAGeTlulECLEC6ACk58s9M+fahI0bYccO+OILLTkoWe/xL6LzxW7yzjsFmDcPBg8G9f++\nZfnrdDjZ2SUnan+dzmzXzmjZ7evXr+Po6IhOpyM2Npbg4GBGjx5ttrhsnTmeMZQGLqT4fDFp2+Ma\nCiGOCiE2CyFqPue5CCEGCiHChRDh169fN0PYWS8hAUaMgCpV4O23rR1N7vHwi8gekr+IPvoIChbU\n/j0Uy/JzdSXEy4upHh5m60Z6KKNlt69cuUJAQACenp7Ur1+fwMDAdA91zQ0sVXb7MFBWSnlXCNEa\nWAs8Vx1kKeUCYAGAj4+PNH+I5vftt3D6NKxfD46O1o4m93j4RRSq1+Ov0yV/EX30kdatt2ULtGxp\n5SBzGT9XV7MmhMyW3fb09OTPP/80Wzw5jTlaDJeAMik+uydtSyalvC2lvJv0fhPgKIQomp5zbdXt\n2zB5MgQEqOGp1uDn6srYcuUe+TJ6912oWBFGjdJWmVQUJXXmSAx/AJWFEB5CCCegG7A+5QFCiBIi\naZCwEKJB0n1j0nOurZo1C27c0F7V8NTswckJpk+HY8fgxx+tHY2iZF+ZTgxSykRgMLAVOAmsklIe\nF0K8LYR42LPeBYgUQkQAXwDdpCbVczMbk7VdvqytQdytG/j4WDuaFMLCYMYM7TWX6tpVWxd6wgRt\nXomiKE8yyzOGpO6hTY9tm5/i/TxgXnrPtXWTJkFiIkybZu1IUggLg2bN/luQPZfWaRJCa8UFBMCX\nX2rdSoqiPErNfDazkydh4UIYNAgqVLB2NCmEhmpJwWjUXlMM7ctt/P2hdWut8XTzprWjUZTsRyUG\nM/vwQ8iXD8aPt3Ykj/H311oK9vbaay4v9/2//4HBADNz3Dx7Rck8lRjM6MABWLsWRo6EYsWsHc1j\n/Py07qOpU3NtN1JKtWpB795ad9KlHDEOLnfJbNltgPLly1O7dm28vb3xyVYPA63PUvMYcoUPP9QS\nwpAh1o7kKfz8cn1CSGnyZFi+XMuVqlSGbUlZdjtPnjzPXXb7oZ07d1K0aNEsiNC2qRaDmWzfrpW+\nGDcOChSwdjRKepQvD2+9Bd9/D2fOWDsa5Xllpuy28mwqMZiBlFproWxZVfrC1owfD87O2qxoJesY\nwgxEz4jGEJZ9ym4LIWjevDn16tVjwYIFZosrJ1BdSWawZg388YdWWtvZ2drRKM+jeHGt62/6dBg9\nGry9rR1RzmMIMxDRLAJTvAk7Jzu8Qrxw9bN+2e09e/ZQunRp/v33XwIDA6lWrRpNmjTJdFw5gWox\nZJLRqP22WbWq9jBTsT0jR4JOp1oNWUUfqscUbwIjmOJN6EP1Zrv2w7Lbj3cjpafF8PCZhJubG506\ndeLgwYNmi8vWqRZDJq1cCcePa68O6m/TJul0WnIYN04bWaYW8jIvnb8OOye75BaDzt/6Zbfv3buH\nyWSiQIEC3Lt3j23btvGR+s0gmfoqy4TERJg4ETw9oUsXa0ejZMb772vLrk6YANu2WTuanMXVzxWv\nEC/0oXp0/jqzdCM9lNGy29euXaNTp06ANtS1R48etFQld5OpxJAJS5fC2bOwbh3YqU45m5Y/P4wd\nC8OHw++/Q9Om1o4oZ3H1czVrQshs2e0KFSoQERFhtnhyGvV1lkHx8TBlilaQrV07a0ejmMOgQVCy\npNZqkDZo0zznAAAgAElEQVSx4oeiZA2VGDJo4UKIjoaPP1ZltZ/Jhiq65smjDV/dvRuCg60djaJY\nj0oMGRAXp1VObdQIAgOtHU029rCi64QJ2qsNJIfXX4cyZbRnR6rVoORWKjFkwPffa/V1Jk9WrYVn\nssGKrs7O2uik/fth61ZrR6Mo1mGWxCCEaCmEOC2EOCuEGJPK/p5CiKNCiGNCiH1CCK8U+6KSth8R\nQoSbI56sFBur9Yw0bgwvvWTtaLI5G63o2r8/lCunWg1K7pXpUUlCCHvgKyAQuAj8IYRYL6U8keKw\n80BTKeUtIUQrYAGQcrR4gJTyRmZjsYQFC7QV2n78UbUW0vSwomtoqJYUbKSAn5OT1moYOBA2bYI2\nbawdkaJYljlaDA2As1LKc1LKeGAF0CHlAVLKfVLKW0kf9wPuZrivxcXGavX7mzbVVgBT0sHPTxsH\naiNJ4aF+/bQie5MmqVZDdmSOstsDBgzAzc2NWrVqPbL95s2bBAYGUrlyZQIDA7l169ZTrpBzmSMx\nlAYupPh8MWnb07wObE7xWQLbhRCHhBADn3aSEGKgECJcCBF+/fr1TAWcUd9+C1evas8WlJzN0VF7\nZh4eDhs2WDsa5XEpy24DGSq73a9fP7Zs2fLE9pkzZ9KsWTPOnDlDs2bNmJkLV3Oy6MNnIUQAWmIY\nnWLzi1JKb6AV8K4QItUqVlLKBVJKHymlTzErrIITG6ut+hUQoCY/5Ra9e4OHh/aLgGo1ZD+ZLbvd\npEkTChcu/MT2devW0bdvXwD69u3L2rVrMx+sjTFHYrgElEnx2T1p2yOEEJ7A90AHKWXMw+1SyktJ\nr/8Ca9C6prKdBQu01sLEidaORLEUR0ftWcOhQ9qzBiXjDIYwoqNnYDCYb8hyZstuP821a9coWbIk\nACVKlODatWtmi9lWmKMkxh9AZSGEB1pC6Ab0SHmAEKIssBroLaX8K8X2fICdlPJO0vsWwBQzxGRW\ncXFaa6FpU9VayG369NEmMU6eDK1bqwEHGWEwhBER0QyTKR47Oye8vEJwdc38M6fMlt1ODyEEIhf+\no2c6MUgpE4UQg4GtgD2wSEp5XAjxdtL++cBHQBHg66S/5EQppQ9QHFiTtM0B+ElK+WSnn5V99x1c\nuQLLllk7EsXSHB21RZgGDoQtW6BVK2tHZHv0+lBMpnjAiMkUj14fapbEAP+V3Q4NDSUmJrkjgp07\ndzJ06NAnjs+bNy/79u175jWLFy/OlStXKFmyJFeuXMHNzc0ssdoSsxTRk1JuAjY9tm1+ivdvAG+k\nct45wOvx7dlJXJw2EqlJE5sZhq+YWd++2kz3yZOhZUvVanheOp0/dnZOyS0Gnc7fbNfOaNntZ2nf\nvj1BQUGMGTOGoKAgOnTokPZJOYya+ZyGRYu0eQsTJ6ovhNzKyUlrNRw4oEpyZ4Srqx9eXiF4eEw1\nWzfSQxktuw3QvXt3/Pz8OH36NO7u7ixcuBCAMWPGEBwcTOXKldm+fTtjxjwxZzfHE9IGh1v4+PjI\n8PCsnyQdHw+VKmlrOe/erRJDbqb+W/jPyZMnqV69urXDUNKQ2r+TEOJQUjf+M6kWwzMEBcGFC9qS\nj7n5i8Ammbmqq5MTjBkDe/faRMknRckUlRieIiFBWyC+QQNVQdXmZFFV1wEDtPUapmS7cXOKYl4q\nMTzFsmUQFaV9t6jWgo3JoqquLi4wapR2ud27zXJJm2WLXdC5SWb/fVRiSEViojYKpU4dVUDNJmVh\nVdeBA8HNDaZONdslbY6LiwsxMTEqOWRTUkpiYmJwcXHJ8DXUms+pWLFCW8t59WrVWrBJWVjVNW9e\nGDFCazns3w++vma7tM1wd3fn4sWLWKtmmZI2FxcX3N0zXqtUjUp6jMkENWuCgwNERICdalMpj7l7\nV6u86uurCuwptkWNSsqgX3+FU6e0tX9VUlBSkz8/DB0KGzfC4cPWjkZRzE999aVgMml1capWhS5d\nrB2Nkp0NHgyurtqzKEXJaVRiSGHDBjh6VJvlam9v7WiU7MzVFd5/X3sOFRlp7WgUxbxydWI4f+s8\n+y7s49ytc0ipjTTx8IAePdI+V1E++EDrVpo+3dqRKIp55cpRSbujdzN+53hO3TiFh86DKH0URS73\n4ET4ZyxYoD14VpS0FCkC77wDn3yiLQFapYq1I1IU88h1LYZtf2+j86rOvF3vbS4Ovcj+N/ZzYehF\njKFjsXO9hFvDzWlfRFGSDB8Ozs5a9Q1FySly1e/GRpORgb8NZHnn5TSr0Cx5+749Dpz+sxgfTD7L\nu1vfpHW18zjaO1oxUsVWuLlpk97mzZNU7LgM+8IXqFCoAh2qdcDFIeMTjBTFmszSYhBCtBRCnBZC\nnBVCPFGjVmi+SNp/VAhRN73nmtO2v7fhls/tkaRgCDMw/o1Y3AqbmDGyEh6FPNh0Rq3jqKRPgjEB\nfd3xGIlnwfwKbIkvzKfHt1B2TlmWHVUrOykZF2YwMCM6mjCDweL3znRiEELYA18BrYAaQHchRI3H\nDmsFVE76GQh88xznms2pG6doWOa/NV8NYQaW9dvOnrN56OwSRvwRA37ufpy8cTKrQlBymLc2vMU1\n+0O07pnAhRBf9sTUIrJ4X2Z32cSYkDGsPrna2iEqNijMYKBZRAQTzp+nWUSExZODOVoMDYCzUspz\nUsp4YAXw+JJHHYClUrMf0AkhSqbzXLPJ55SPmNj/lv+7Gh7CjzoXCha8Qbt57bkaHkLM/RjyO+XP\nqhCUHOTUjVNsOrOJX7r+Qs02JxBGaPjVfZqdPcrli7Es7biUsSFjVU0h5bmF6vU8uAfGBEG8yUSo\nXm/R+5sjMZQGLqT4fDFpW3qOSc+5ZtO2Sls2/LUBQ5yWfU/kvUbYwTZ07TqHPAX0JHoeYu3ptXSo\nmvuW8lOe3w8RP9DXqy/5nPLR7vRaWjT7kT8ONKJ//tm8GLoZ//L+2At7Dlw6YO1QFRty8NJBdh6Z\ng2lZaejdAOM9I7ev7SbRlGixGGxmVJIQYqAQIlwIEZ7R4l2lCpSiS/UuvL7+deKN8VR+oQHt239L\nh/bzwd6J7/QHaVulLWVcy5g5eiUnunbvGpUKVwKgoGsMPXtOJz7ehV/XvUtB1xiEEFQsXJFrd69Z\nOVLFVvx49EfaL29PQN6q5F3vQc26Rj4rb2Tv8QV0WtmJBGOCReIwR2K4BKT8JnVP2paeY9JzLgBS\nygVSSh8ppU+xYsUyHOyXrb/EJE3U+KoG39w6QPTQBFYUfIXBcgbhCRWY32Z+hq+t5C4l85fkr5i/\nAChWrwtlikcR0HQVa9a9i0PlbkgpORNzhpIFSlo5UsUWROujGbJlCDv77sR0cBD379jx4/T8DK3Z\ngpA+ISSaEvlk3ycWicUcieEPoLIQwkMI4QR0A9Y/dsx6oE/S6CRfwCClvJLOc83KxcGFX1/9lR9f\n+ZHw+wlEyBr8RE9O2nnTuv6H5HHMk5W3V3KQPl59CIoI4s6DO5RqFEiVAr8xqNllYmML8POBALaf\n246dsKN+qfrWDlWxAd8e+pbenr0pk6c6c+ZA27bg7a3tc7R3ZEazGXwd/rVFupQynRiklInAYGAr\ncBJYJaU8LoR4WwjxdtJhm4BzwFngO+CdZ52b2ZjSIoTA192Xub79yGPvgD3gbGdHgE6X1bdWcpDK\nRSrTuXpnOq7syPV71ynVKJAu44fRsSN8NjeR3ive5X/N/4dQi3oo6bDnnz20r9qe+fMhJgYGtj1I\nRNAoLu8NBsC7hDcCwQXDhTSulHlmmeAmpdyE9uWfctv8FO8l8G56z7UUP1dXQry8CNXr8dfp8HN1\ntUYYig37svWXjN0+lirzqtCiYgtK5i9JlOdd7qz9nnY3fqVd1drWDlGxEUIIYmO1EiuNfWIoUK4J\ntxwTuHXnC9j7GyUbNsckTRb5RcNmHj5nFT9XV8aWK6eSgpIhDnYOzG4xm7PvnaVlxZaUcy3HpO7t\naNnSRPCy2ty7Z+0IFVsRUD6AmV9e59o16Nv8Z3BMAHsTOCRw/Wwwf1z+Ayd7J8q6ls3yWHJVSQxF\nySpF8hahf53+yZ+LTYBGjWDBAm1RH0VJS99abzK1I3jVv0vLNhU5c9cRZAIkOqLz8Kff9tEMbjAY\nO5H1v8/n+haDomSFhg0hIABmz4a4OGtHo9iC7WtKYzKUJsrzdRYZ91PIbjH5/nmf23cm8cqpCZTI\nX4IPXvjAIrGoxKAoWWTCBLhyBRYtsnYkSnaXkKBV6K1fH/ZM+4jLdy7TOnIELW8sZe79bYxpNIZl\nryzD3s4yK4gJW5yu7+PjI8PDw60dhqI8k5Tw4otw4QKcPQtOTtaOSMmuli6Fvn1h3Tpo3z7r7iOE\nOCSl9EnrONViUJQsIoTWarhwQfsfX1FSYzRqa4d7eUG7dtaORqMSg6JkoZdfBh8frZsgMavmJYWF\naTcIC8uiGyhZ6Zdf4K+/YNw47ZeJ7EAlBkXJQkLA+PFw7hz89FMW3CAsDJo105omzZqp5GBjTCZt\nrfnq1aFzZ2tH8x+VGBQli7Vvr3UTTJumdRuYVWgoxMdrF46P1z4rNmPtWjh+XPvlwS4bfRtno1AU\nJWd62Gr46y9YtcrMF/f3155q29trr/7+Zr6BklWk1FoLVarAa69ZO5pHqQluimIBr7wCNWrAxx9r\nXwJm++3Qzw9CQrSWgr+/9lmxCRs2wJEjsGSJltezEzVcVVEsZPly6NEDfv4ZunSxdjSKNUkJDRrA\nzZtw6hQ4Olrmvmq4qqJkM6++qnUbTJ2qPXRUcq8tWyA8HD780HJJ4XmoxKAoFmJvrz1rOHpUm8ik\n5E5SwuTJULYs9O5t7WhSpxKDolhQ9+5QqRJMmaJ9QSi5z7ZtcOCA1lrIrrPhM5UYhBCFhRDBQogz\nSa+FUjmmjBBipxDihBDiuBDigxT7JgkhLgkhjiT9tM5MPIqS3Tk4aK2GI0dgfZauVahkRw9bC2XK\nQP/+aR9vLZltMYwBQqSUlYGQpM+PSwSGSylrAL7Au0KIGin2z5FSeif9WGXBHkWxpJ49oWJF7QtC\ntRpyl+3btTmI2bm1AJlPDB2AoKT3QUDHxw+QUl6RUh5Oen8HbQnP0pm8r6LYLAcHrfzBn39qQxaV\n3OFha8HdPXu3FiDziaG4lPJK0vurQPFnHSyEKA/UAQ6k2PyeEOKoEGJRal1RipIT9eoFFSqoVkNu\nsmMH7N0LY8eCs7O1o3m2NBODEGK7ECIylZ8OKY9LWtf5qf+JCyHyA78CQ6SUt5M2fwNUALyBK8Cn\nzzh/oBAiXAgRfv369bT/ZIqSjTk6as8aDh1SrYbcQEqYOBFKl4YBA6wdTdoyNcFNCHEa8JdSXhFC\nlARCpZRVUznOEdgAbJVSfvaUa5UHNkgpa6V1XzXBTckJEhKgWjXQ6bQx7dmlsma6hIWp2dbPITgY\nWrSAr7+GQYOsF4elJritB/omve8LPDE6WwghgIXAyceTQlIyeagTEJnJeBTFZjg6wkcfweHDNjZC\nSVV0fS5Sav/OZcrYRmsBMp8YZgKBQogzQPOkzwghSgkhHo4wagT0Bl5KZVjqLCHEMSHEUSAAUMum\nK7lKz55QuTJMmmRDzxpURdfnsnUr7N+vDTjI7s8WHlK1khTFyn74Afr0gdWroVMna0eTDg9bDPHx\n2pjLkBDVnfQUUoKvL1y7plXXtfYQVVUrSVFsRPfuWg2liRNtpIbSw4quU6eqpJCGTZvg4EFtoIG1\nk8LzUC0GRckGfvpJ61ZauVIrtqfYPimhXj0wGCxbQfVZVItBUWzIa69BzZraQ8osWxtasag1a7RJ\njBMnZo+k8DxUYlCUbMDeXiusd/p0Fq0NrViU0agl+apVtZagrVGJQVGyiU6doE4dbTZ0QoK1o1Ey\nY+VKbS3nyZOz3+ps6aESg6JkE0JoS3+eO6ct96jYpsREbfhx7drQtau1o8kYlRgUJRtp1Uob5DNl\nCsTFWTsaJSOCguDMGe3f0Gxre1uYjYatKDmTEDBtGly8qJVPUGxLXJzWWmjQADp0SPPwbEslBkXJ\nZgICIDAQpk+H27fTPl7JPr75Rkvq06fbWO2rx6jEoCjZ0PTpEBMDn6VaclLJju7c0f7dmjXTfmyZ\nSgyKkg35+ECXLvDpp6CqzKcQFgYzZmTLwn2ffQY3bmjh2TqVGBQlm5o6Fe7f134LVcjWVV1v3NCS\n+CuvQP361o4m81RiUJRsqlo1bQnIr7+GqChrR5MNZOOqrh9/DPfuaa85gUoMipKNTZqkDXkcP97a\nkWQD/v5aJTp7e+3V39/aEQHavJOvv9bWWqhe3drRmIdKDIqSjbm7w5AhsGyZVncnV8umVV3HjwcH\nB22Wc06hqqsqSjan10PFilqlzm3brB2NklJ4uPZM4cMPtfkn2Z1FqqsKIQoLIYKFEGeSXgs95bio\npJXajgghwp/3fEXJzXQ67XlrcHDqiSHMYGBGdDRhBoPlg8vFpITRo6FoURg1ytrRmFdmu5LGACFS\nyspASNLnpwmQUno/lq2e53xFybUGDYLy5bUvIKPxv+1hBgPNIiKYcP48zSIiVHKwoC1bYMcOLWm7\nulo7GvPKbGLoAAQlvQ8COlr4fEXJFZydYeZMiIjQavE8FKrXE28yYQTiTSZC9XqrxZibJCbC8OFQ\nqRK8/ba1ozG/zCaG4lLKK0nvrwLFn3KcBLYLIQ4JIQZm4HyEEAOFEOFCiPDrasaPkgu9+qq2fvC4\ncXD3rrbNX6fDyc4Oe8DJzg5/nc6qMeYW330HJ0/C7Nm2tWRneqX58FkIsR0okcqucUCQlFKX4thb\nUsonnhMIIUpLKS8JIdyAYOA9KeUuIYQ+Pec/Tj18VnKr/fu1wTgTJmjVO0HrTgrV6/HX6fDLaX0a\n2ZDBoLUUataEnTttqyZSeh8+O6R1gJSy+TNuck0IUVJKeUUIURL49ynXuJT0+q8QYg3QANgFpOt8\nRVE0vr7QrRt88gm8+SaUKQN+rq4qIVjQtGn/1bGypaTwPDLblbQe6Jv0vi+w7vEDhBD5hBAFHr4H\nWgCR6T1fUZRHzZgBJpM2RFKxrHPn4PPPoW9fqFvX2tFkncwmhplAoBDiDNA86TNCiFJCiE1JxxQH\n9gghIoCDwEYp5ZZnna8oytOVL689+PzxR9i3z9rR5C7DhoGjo23MWcgMNcFNUWzQ3btaLaUSJWDb\nXAN3duvR+etw9VNdSlkiLIytC6JpuaQbM2dq8xdskdmeMSiKkv3kzw+zZkl69hRMC7hMW9NVhJOg\nzo46KjmYW1gY8S+15IO4A1QSZxniewPwtXZUWUrVSlIUG3Tqxik+vVOfkoX/ZmFiRe6aHDE+MDJ3\n7lz+vafGcJhVaCizH4znNNWYxmc479tp7YiynEoMimJjLhgu8FLQS7zt8xYrP3XjDo4sFuVxdHHE\nuZEzzZc2537CfWuHmSPceXCHH68VYIYcwgvcoIRsj6FIE2uHleVUYlAUGzN732x6e/bmzXpv0rhf\nAQa8Es86SiO/rsPo90ZT1rUsP0T8YO0wbd788PmUm1uO2Wv9iceewfxNgnAgaO8hEowJ1g4vS6nE\noCg2xCRNLI1YyuAGg5O3jf8sgsJF7jN8HphMgsENBrMkYon1gswBgo4EMXvfbOZUOcrR6Fr0cLiA\nu30sDi6ORHhEMGjjIGuHmKVUYlAUGxKbEEu8MZ4yrmUAMBjCiI72Z+DAQRw6VIAvvjhHlSJVuHLn\nShpXUp7GaDIyYecEfmi3iulj3KlYEf4XrMNjqgfeId588eEXrDu9jr9v/m3tULOMSgyKYkPyOObB\n0d6RS7cvAaDXh2IyxdO8+Q/UqbOTSZNKEv7XBYrnf2rZMSUNu6J3UTx/cUJ+rMdff8FXX4FLnRPQ\n4yeocYJ8TvnoVbsXy44ts3aoWUYlBkWxIXbCjl61e/FN+DcA6HT+2Nk5IYQ9Q4cOJTbWmeEjoK9X\n3zSupDzNv/f+pUisL9OnQ9eu4OsbRkREM86fn0BERDMMhjA8Cnnk6NFfah6DotiYEQ1H4LfQj2pF\nq9Gzdk+8vELQ60Px9H6Rht138/vSppS40sDaYdqsUvnd2T2vD87OkrlzRXKrDIyYTPHo9aGcunGB\nMgXLWDvULKMSg6LYGI9CHmzrvY3uv3Znxp4ZtKrUinhjPKtPdqdi4xpUPtCQIYPzEBgABQpYO1rb\nc3yrH/fP2DF8xilKlaqGwaC1ykymeK115lKHFZGzOTroqLVDzTKqJIai2CgpJb9H/86Biwewt7On\neYXmeJfwJiwMGjWCd96BefOsHaVtuXQJatSActWvc72LF6tf+xW/Mn4YDGHo9aHEO1an96aZNCnX\nhFmBs6wd7nNLb0kMlRgUJQf64AP48kvYtQtefNHa0dgGKaFTJ9i6FY4dg6MJqxmyZQilCpTCq7gX\nl+5cYt+FfQzxHcL4JuOxE7b3iFYlBkXJxe7ehdq1wcEBjhyBfPmsHVH29+OP0Lu3tirbiBHatkRT\nIsF/BxOlj6JwnsK0rtyaAs622z+nEoOi5HKhoRAQAO++q7qU0nLhgpZIa9WC338He3trR5Q10psY\nbK8tpChKuvj7w9Ch2jj84GBrR5N9mUwwYAAkJkJQUOpJIcxgYEZ0NGEGg+UCCwvTVmUKC7PcPZNk\nalSSEKIwsBIoD0QBr0opbz12TNWkYx6qAHwkpZwrhJgEvAlcT9r3oZRyE4qimMW0abBlC/TvD5GR\noNOlfU5u8803sH279lqx4pP7wwwGmkVEEG8y4WRnR4iXV9YvpRoWBs2aQXw8ODlBSIi22LeFZLbF\nMAYIkVJWBkKSPj9CSnlaSuktpfQG6gH3gTUpDpnzcL9KCopiXnnywNKlcPUqDBqkPWBV/nPyJIwc\nCS+/DG+9lfoxoXo98SYTRiDeZCJUr8/6wEJDtaRgNGqvoaFZf88UMpsYOgBBSe+DgI5pHN8M+FtK\nGZ3J+yqKkk4+PjBlCqxYAYsXWzua7CM2Fl57TVv0aNEiECL14/x1Opzs7LAHnOzs8LdEs8vfX2sp\n2Ntrr/7+WX/PFDI7wa24lPJhta6raOs7P0s3YPlj294TQvQBwoHhj3dFKYqSeaNHa70R772n9UhU\nr27tiKxvxAhtWOqmTVCq1NOP83N1JcTLi1C9Hn+dLuu7kUD7RwoJ0VoK/v4W7UaCdIxKEkJsB0qk\nsmscECSl1KU49paUstBTruMEXAZqSimvJW0rDtwAJDAVKCmlHPCU8wcCAwHKli1bLzpaNToU5Xlc\nvgze3lCyJBw4AC4u1o7Ielavhs6dYfhw+OQTa0djORYZriqEOA34SymvCCFKAqFSyqpPObYD8K6U\nssVT9pcHNkgpa6V1XzVcVVEyZvNmaN0a3nwTFiywdjTW8fffWvdapUqwd6/WU5NbWGq46nrgYRnH\nvsC6Zxzbnce6kZKSyUOdgMhMxqMoyjO0agUffgjffQfff2/taCzv3j1tdrMQsGpV7koKzyOziWEm\nECiEOAM0T/qMEKKUECJ5hJEQIh8QCKx+7PxZQohjQoijQAAwNJPxKIqShilToEULbeLbgQPWjsZy\npNRaSpGRsHw5eHhYO6LsS818VpRcKCZG605JSIBDh6B4LljXZ84cGDZMm9vx4YfWjsY61MxnRVGe\nqkgR7QFsTAx07KgN3czJNm3SRiF16gRjx1o7muxPJQZFyaXq1NEKxx04AH37aqUhcqIjR7T5Cl5e\n2mS/p81XUP6jEoOi5GKdO8OsWfDzzzmze+XiRWjbVisFsmGDNplNSZtawU1Rcrnhw+HsWfjf/6BM\nGe2hdFrCDAaLTvhKMCaw7vQ6dp7fiUma8HX35dWar5LHMc9Tz7l1C9q0gdu3Yc+eZ09iUx6lEoOi\n5HJCaGW5r1yBwYO136r79n368ZYuKhdxNYIOKzpQ1rUsnat3xt7OnhXHVzBq+yhWdlmJf3n/J865\nc0cbmnvqlNZS8PTMsvByJJUYFEXBwQFWroR27bQS1HnzQteuTx4Xb4zni9O7iDXmA2FHrDGBeX/t\nwaduCxztHc0e15U7V2i5rCVzX57La7VeS94+uMFgdpzfQdefu7Kr3y6qF/uvxsf9+9qfIzwcfv0V\nAgPNHlaOp54xKIoCaCUy1q7VyvL06AFr1jy6/2bsTRovbszJsytwEmhF5YTg2F8/EBAUgCHO/GsV\nzA+fT6dqnR5JCg+95PESH7zwAXP2z0nedv++NvJo1y744Qfo0MHsIeUKKjEoipIsXz7YuFGb49C1\nqzaK56G+a/vi5+7Hn91/JLROPaZ6eBBapx5HevxELbdavPHbG2aPZ+Xxlbxe5/XkzwZDGNHRMzAY\ntMVrXq/zOisiVyTt08pnBwfDwoXQvbvZw8k1VFeSoiiPcHXVvlw7dtSeNRgMENj9FH9c+oNfuv6C\nEIIanKAUoejwx0748dnLn1F2Tlmi9FGU15U3Wyz6OD0lC2iVcwyGMCIimmEyxWNn54SXVwglCvpy\nN/4uV68Zad3KnshIrbz4q6+aLYRcSSUGRVGekD+/9tC2e3d4/30IOHCXjq92xtnBOdUvaFdXP9pX\nbc/mM5sZVH+Q2eKoUKgCf175k1IFSqHXh2IyxQNGTKZ49PpQzt13oWRcAI1ftOfSJVi3TnvorGSO\n6kpSFCVVLi7a/IZ33oGdy3zYNm0wBgOpfkED5HfKzwPjA7PG8EbdN/hs/2eYpAmdzh87OyfAHjs7\nJ3Q6f4bN207Mlxu5fVtbnlMlBfNQiUFRlKdycICvvoL3Jh/nfHglfH0lZ3/3hzgHSLSDOAccL/og\npSTkfAh1StQx6/171u5JgjGB/uv6E+9QiUr26ygUPYwSCSto84Ge0BnDqVrZkT/+gIYNzXrrXE0l\nBkVR0jR3QnVKvduXK9fjaT3Al1X9t2Na9DqM/JSEXVX45cQvADQp18Ss93V2cGZTz0242LvQbmQ7\njr8Ekf078kpgY/YGtaJT5wTC9tpTtqxZb5vrqWcMiqKkyU7Y8fPIwbQvVIdq6zbxTcSL7F9ek2HO\nZ01KBkYAAAcFSURBVDhReCOTNk1iU89NiCwoRJTfKT/ftvuW44dOsSgB5stKAHza5QZDVxVVtY+y\ngGoxKIqSLg3LNGTnOyupNm4M5Zp+TKSDC30S6vDlpnJseGU3PqXSrOacYbt2QY/llflMVqMyd1nk\nfIjXhzmqpJBFMpUYhBBdhRDHhRAmIcRT/6sQQrQUQpwWQpwVQoxJsb2wECJYCHEm6TXV9aIVRcke\nahevzcquKzi29QP2Rt6gVx84/Vsb2vhW5eOPtTLe5iIl/P67Nou5aVO4ed+eSWNvMmRENO6/lcHV\nL+trNOVWmW0xRAKvALuedoAQwh74CmgF1AC6CyFqJO0eA4RIKSsDIUmfFUXJ5go4F6B+1TIsXaw9\n+PXxgQkTtCJ8gwbB7t1gNGbs2teva8uONmgA/v6wfz9MnQpLDxr4X8tI3mhzi0Dns4QZzD/TWtFk\n6hmDlPIkkFa/YgPgrJTyXNKxK4AOwImkV/+k44KAUGB0ZmJSFMWy6tWDzZvh+HH47DNYvBjmz9dW\nhWvbFl54AerWhZo1tSGwKUkJV6/Cn3/C4cMQEqJ1G5lMULWqdp0+fSBPHpgRrSfeZMIIxJtMhOr1\nFqnsmhtZ4uFzaeBCis8XgReS3heXUl5Jen8VyAULDCpKzlSzplaKYu5cbcW01avhl1+0bQ/lyweF\nCoGzM0lzIiAx8dFrfPghvPIKeHs/uqiOv06Hk51dclVXf53Ocn+4XCbNxCCE2A6USGXXOCnlOnMF\nIqWUQoinLkAthBgIDAQoq8amKUq2VaCAtmLaa69pv/mfP6+1CE6e1BLBrVsQF6ctnqPTQYkS2mpy\nXl5QsODTr+vn6vr/9u4mNI46DuP496k1oChpMVVqX2wM9e1goK7aSpGqFW0updBDUawULyIVjwUP\nevBSbyKiEkoRLxbRohGrQRStUKNNoW+hqLFiTS1WqyjUg4T+PMwImWq6k2Yyk9l9PrCw88Lm97DL\n/GYm/90/H/f2ljoPRLtq2hgiYu00/8ZJYMmE5cXpOoCfJS2MiFOSFgKnL1BHP9AP0Gg0Jm0gZjZ7\nzJkDPT3JowirOjvdEEpQxnDV/cBySd2SOoBNwEC6bQD4d0qQR4HCrkDMzOziTHe46gZJY8Aq4H1J\ng+n6ayXtAYiIcWArMAgcA96MiJH0JbYD90v6FlibLpuZWYUUUb+7Mo1GI4aHh6suw8ysViQdiIim\n30T0N5/NzCzDjcHMzDLcGMzMLMONwczMMtwYzMwso5ajkiT9AvxQ4Et2Ab8W+HqzmbO2JmdtTUVn\nvS4iFjTbqZaNoWiShvMM4WoFztqanLU1VZXVt5LMzCzDjcHMzDLcGBL9VRdQImdtTc7amirJ6v8x\nmJlZhq8YzMwso20ag6QHJX0taVTSf+aWVuLFdPthSSuqqLMIObI+nGY8ImmfpN4q6ixCs6wT9rtd\n0rikjWXWV7Q8eSWtkXRQ0oikz8qusSg5Psedkt6TdCjNuqWKOqdL0k5JpyUdnWR7+cemiGj5B3AJ\n8B1wPdABHAJuOW+fPuADQMBK4Muq657BrHcB89Pn61o564T9PgH2ABurrnuG39t5JPOpL02Xr666\n7hnM+jTwfPp8AfAb0FF17ReR9W5gBXB0ku2lH5va5YrhDmA0Io5HxN/ALmD9efusB16PxBAwL51V\nrm6aZo2IfRHxe7o4RDKrXh3leV8BngTe5gIzBNZEnrwPAbsj4gRARNQ1c56sAVwpScAVJI1hnJqJ\niL0ktU+m9GNTuzSGRcCPE5bH0nVT3acOpprjMZKzkTpqmlXSImAD8EqJdc2UPO/tDcB8SZ9KOiBp\nc2nVFStP1peAm4GfgCPAUxFxrpzySlX6sanpnM/WuiTdQ9IYVlddywx6AdgWEeeSE8uWNxe4DbgP\nuAz4QtJQRHxTbVkz4gHgIHAv0AN8JOnziPiz2rLqr10aw0lgyYTlxem6qe5TB7lySLoV2AGsi4gz\nJdVWtDxZG8CutCl0AX2SxiPinXJKLFSevGPAmYg4C5yVtBfoBerWGPJk3QJsj+RG/Kik74GbgK/K\nKbE0pR+b2uVW0n5guaRuSR3AJmDgvH0GgM3pCICVwB8RcarsQgvQNKukpcBu4JGan0k2zRoR3RGx\nLCKWAW8BT9S0KUC+z/G7wGpJcyVdDtxJMtd63eTJeoLkyghJ1wA3AsdLrbIcpR+b2uKKISLGJW0F\nBklGO+yMiBFJj6fbXyUZsdIHjAJ/kZyN1E7OrM8AVwEvp2fS41HDHyXLmbVl5MkbEcckfQgcBs4B\nOyLif4dBzmY539vngNckHSEZsbMtImr3q6uS3gDWAF2SxoBngUuhumOTv/lsZmYZ7XIryczMcnJj\nMDOzDDcGMzPLcGMwM7MMNwYzM8twYzAzsww3BjMzy3BjMDOzjH8AMvtTJkMpmqwAAAAASUVORK5C\nYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x146c6f0cf60>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "N = 11\n",
    "x, t = generateDataset(N, sin2pi, 0.03)\n",
    "degs = [1, 3, 5, 10]\n",
    "plot_polyfit(x, t, degs, sin2pi, reverseOptimizeLS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can observe that the polynomial function with M=1 does not estimate well the sin(2px) function, which makes sense as it is a linear function.\n",
    "M=3 is very close, and for M>=5 it seems like the coefficiants became finely tuned to the noisy sin(2px) function. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Polynomial Curve Fitting with Regularization\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To solve this overfitting we observed, we can try to pick the best order of a ploynom, or we can use regularization. It involves adding a penalty term to the error function in order to discourage\n",
    "the coefficients from reaching large values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we need a training, dev and test data which will help us choose the penalty (lambda). \n",
    "\n",
    "To construct a development set, we will extend our synthetic dataset construction function to return 3 samples: one for training, one for development and one for testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1456,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def generateDataset3(N, f, sigma):\n",
    "    \"\"\"\n",
    "    returns 3 pairs of vectors of size N each, \n",
    "    (xtest, ttest), (xvalidate, tvalidate) and (xtrain, ttrain).\n",
    "    The target values are generated with Gaussian noise N(0, sigma).\n",
    "    \"\"\"\n",
    "    x = np.linspace(0, 1, 3*N)\n",
    "    np.random.shuffle(x)\n",
    "    \n",
    "    xtrain = x[:N]\n",
    "    xtrain.sort()\n",
    "    ttrain = noise(xtrain, f, sigma)\n",
    "    \n",
    "    xdev = x[N:2*N]\n",
    "    xdev.sort()\n",
    "    tdev = noise(xdev, f, sigma)\n",
    "    \n",
    "    xtest = x[2*N:]\n",
    "    xtest.sort()\n",
    "    ttest = noise(xtest, f, sigma)\n",
    "    \n",
    "    return (xtrain, ttrain), (xdev, tdev), (xtest, ttest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets see what we get"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x146c6875198>]"
      ]
     },
     "execution_count": 1110,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAD8CAYAAAB+UHOxAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd8leX9//HXdU72DtmTJCSMsCEgMhQQkCVYq9ZttZY6\nax1Vv/XX1i5ttVqr1q1Va9U6q8gGlT0MK4yQTSBkkISQReY51++PRIoIEsg55z7j83w88jDnnDu5\n3xfg/bmv+77u61Jaa4QQQngek9EBhBBCGEMKgBBCeCgpAEII4aGkAAghhIeSAiCEEB5KCoAQQngo\nKQBCCOGhpAAIIYSHkgIghBAeysvoAN8nMjJSp6SkGB1DCCFcxtatW2u01lE92dapC0BKSgrZ2dlG\nxxBCCJehlCrt6bZyCUgIITyUFAAhhPBQUgCEEMJDSQEQQggPJQVACCE8lBQAIYTwUFIAhBDCQ0kB\nEADUt3Tw3+2HsFhliVAhPIVTPwgmHMNq1fzive18mVeNUjB/RILRkYQQDmCTHoBS6nWl1GGl1O7T\nfK6UUs8opQqVUjlKqVG22K+wjZfXFvNlXjX+3mb+uX7/8fe3ltZx+QsbyCk7alw4IYTd2OoS0BvA\nzO/5fBaQ0f21AHjBRvsVZ0nrb1/i2X2onieW5TFnaBwPzhzAjoNH2Xagjq2lddz4+hayS+u47e1t\n1DW3G5RYCGEvNikAWus1wJHv2WQ+8JbusgkIU0rF2WLfouc+2V7G+D9/wfYDdcffe2vjfvy8TDx6\n2VAuz0oi2NeLP36+lxtf30JkkA8vXz+a6sY27nl/B1a5PyCEW3HUTeAE4OAJr8u63/sOpdQCpVS2\nUiq7urraIeE8xec7K6iob+W6VzezsaiW1g4LS3ZVMnNIHKH+3gT5enHlmCS2HThKZJAP7y04nxmD\nY/n1JZl8lVfN818VGt0EIYQNOd0oIK31y1rrLK11VlRUj2Y0FcBjS3K5/rXNlNQ0n/Jzi1WzZf8R\npg2KISHcn5vf+JoPtpbR2NbJZaP+V4tvvbAfPx6fwnsLzic21A+A685LZv6IeJ5akc/6whq01uyr\nbOCN9SXUNrU5pH1CCNtz1CigQ0DSCa8Tu98TNrAmv5qXVhdjUjD772v5v9kDue68vphM6vg2eZWN\nNLZ2MmdYLBnRwcx9dh2PLc4lJsSXcWkRx7eLCvblkXmDv/X7lVI8+oOh7C1v4I53thHs58XBIy0A\nHG5s44GZAx3TUCGETTmqB/AZcEP3aKBxQL3WusJB+3Zrja0dPPRRDv2iAvny/smMSe3Dbz7dww2v\nb6H8aMvx7ZbtqQRgbGoEg+NDSI0M5Fi7hfkjEjCfUChOJ9DXixeuG0WovzfpUUE8dtlQhieF8VWe\nXKYTwlXZpAeglHoXmAxEKqXKgN8C3gBa6xeBxcBsoBA4Btxki/0KeHTxPiobWvnwtvH0jQjkzZvG\n8M6WA/xpUS4XP72GRy4ZjFVrnvmigOmZMSSE+QMwd1gcz35RyA9G9nzMf3p0MKt/OeX46yPN7Tyx\nLI/DDa1Eh/jZvG1CCPuySQHQWl99hs81cIct9iX+Z11BDe9uOcDPLkhjVHI40HW55trz+jIxPZL7\nP9jJfR/sBGBSRiTPXj3y+M/eemE/zkuNYFBcyDnvf/KAKJ5YlsdXedVcOSbpzD8ghHAqTncTWPRM\nU1snD36UQ1pUIPdM7/+dz/tGBPLegvP5f3MGccXoRF6+Pgs/b/PxzwN9vZiYEdmrDJlxIaREBPDB\n1oNn3lgI4XRkKggX9djiXMrrW/jw1vHfOrCfyGxS3DIpzW4ZlFJcPTaZx5bso6CqkYyYYLvtSwhh\ne9IDcEHrC2v49+YD3DIxldF9ww3N8sPRiXibFe9sOWBoDiHE2ZMC4GKa2jp54MMc0iIDuW/GAKPj\nEBnky4zBsXy87RCtHRaj4wghzoIUACdWfrSFprbOb733lyX7KK9v4fHLh5320o+jXTs2mfqWDt7e\nVGp0FCHEWZAC4KQsVs2859Yx6+9rKKpuAmBDUQ3/2lTKzRNSyUrpY3DC/xmXFsGkjEj+uCiXf64v\nOe12+2uaOXTCswlCCGNJAXBSe8rrqWlqp+JoK5c9v4Ev8w7z4Ec5pEQEcL8TXPo5kcmkeOWGLGZk\nxvC7hXt5emX+d2Yd1Vpzw+tbmP7Uaj7PKTcoqRDiRFIAnNSGoloA/vOz84kI8uGmf35NWV0Lj18+\nHH8f57j0cyI/bzPPXzuKy0cn8vTKAn63cO+3Zg8trT3GgSPH8PEycec72/n9wr10WKwGJhZCyDBQ\nJ9JpsVLd1EZNYzsr91aRER3E6L7hfHzbeH75YQ7DEkIZm+o8l35O5mU28fgPhxHi583r60toaO3g\n8R8Oo7C6iUU5XTN/vP+z83ln8wFeX19CZUMLz1872uDUQnguKQBO4uixdi55bt3xSdYAbpmYCkBY\ngA+v3JBlVLSzYjIpfj13EOEB3jy5Ip/K+layS+to77TiZVJkRAfxyLzBRAT68OSKfFbnV3Nhf5n1\nVQgjSAFwEk+tyOdQXQu/mZtJYrg/kcG+DIkPNTrWOVFKcddFGYT4e/Pbz/aQEOZPsJ8XUwZGo1TX\nxHMLLkzjo21l/OHzvYy/exLeZrkaKYSjSQFwAvsqG3h7UynXjevLzd1n/e7gxvEpZMaHEBfqR2J4\nwLc+8/Uy8//mZHLLW9m8vamUmya4T7uFcBVy2mUwrTW/+2wvIf7e3HuKOX1c3ZiUPt85+H/jokHR\nTMqI5G8r8jkiaw4L4XBSAAy2ZHclG4truW96f8ICfIyO41BKKX4zN5PmdgtPrcgzOo4QHkcKgIFa\n2i38aVEuA2ODuXpsstFxDJERE8z14/ryzuYD5FY0GB1HCI8iBcBAL60p4tDRFn57yWC8PPgm6D3T\n+hPq783vF+79zgNkQgj78dyjjsEOHW3hxdVFzBkax/n9Is78A24sNMCbe2cMYGNxLcv2VBkdRwiP\nIQXAII8uzkVr+L/ZsqA6wNVjkhgQE8yfFu+VWUWFcBApAAbYWFTLopwKbpvc77QjZDyNl9nEby7J\n5OCRFl5bd/oJ5YQQtiMFwMEsVs3vFnY9HPWzC/oZHcepTEiP5OLBMfzjy0KqGlqNjiOE25MC4GBL\ndlewr7KRB2cNdMpJ3Yz28OxMOi2avyzZZ3QUIdyeFAAH0lrz0upiUiMDmTM0zug4Tik5IoCfXpDK\nx9sPsam41ug4Qrg1KQAOUn+sgwX/2squQ/XcdmE/zCZldCSndeeUDJL6+PPwJ7to65QbwkLYixQA\nB9hQVMPsZ9byVd5hfjM3kyuyEo2O5NT8fcz8Yf4QiqqbeXl1sdFxhHBbUgDsqNNi5eFPdnHNK5vx\nNis+uHU8N09MPT4jpji9yQOimTMsjue+LKRclpEUwi6kANjRol0V/HvzAW6akMKSuy9gRFKY0ZFc\nyv/NGogGnlyeb3QUIdySFAA7+u/2Q8SH+vHrOZky4uccJIYHcNP4FD7eXsbecpknSAhbkwJgQ1/u\nO8z5j63icEMrtU1trCmoYd6IBExyw/ec3T4lnVB/bx5bkmt0FCHcjhQAG6lrbueXH+ZQUd/K9oNH\nWbSrAotVM39EvNHRXFqovzd3Tc1gbUENq/OrjY4jhFuRAmAjv/lsD/UtXYuaFB5u4tMd5QyICWZQ\nXIjByVzf9eP6ktwngEcX5WKxymyhQtiKFAAb+DLvMAt3lvPzqRkkhPnzxb7DbC2tY/5IOfu3BR8v\nEw/MHEBeVSMfbS0zOo4QbkMKgA28traE2BA/bp3cj4yYILaW1gEwb7gUAFuZMzSOEUlh/HV5HkeP\nyfKRQtiCFIBeKjzcyLrCGq4bl4y32URGdBAAY1LCZaZPG1JK8fv5g6k71s597+/EKpeChOg1KQC9\n9K+NpfiYTVzVvaRjRnQwAPNGJBgZyy0NSwzj4dmDWLXvMC+tkSeEhegtKQC90NjawYdby5g7LI7I\nIF8ApmXGcP24vlwqo3/s4sbxKcwZGsdfl+exWSaLE6JXpAD0wifbD9HcbuGG8SnH3+sT6MMfLh1C\nsJ+3ccHcmFKKP/9wKMl9Arjr3e1UN7YZHUkIlyUF4BxprXlzw36GJ4bKFA8OFuznzfPXjqK+pYO7\n39suQ0OFOEdSAM7RhqJaiqqbueH8FKOjeKRBcSH84dIhbCiq5emVMleQEOfCJgVAKTVTKZWnlCpU\nSj10is8nK6XqlVI7ur9+Y4v9GunNDfvpE+jDnGGysItRrsxK4orRiTz7RSG5FTJXkBBnq9cFQCll\nBv4BzAIygauVUpmn2HSt1npE99fve7tfI5XVHWNlbhVXjUnCz1smeTPSw3MG4e9tloXkhTgHtugB\njAUKtdbFWut24D1gvg1+r1PSWvPo4lyUUlw7rq/RcTxeWIAPV2Ql8umOQxyWheSFOCu2KAAJwMET\nXpd1v3ey8UqpHKXUEqXUYBvs1xBvbNjP4l2VPHDxABLC/I2OI4CbJ6TSadW8uXG/0VGEcCmOugm8\nDUjWWg8DngX+e7oNlVILlFLZSqns6mrnmv1x+4E6Hl2cy7RBMSy4IM3oOKJbSmQgMzJjeHvTAY61\ndxodRwiXYYsCcAhIOuF1Yvd7x2mtG7TWTd3fLwa8lVKRp/plWuuXtdZZWuusqKgoG8Szjbrmdu58\nZzsxIX48ecVwWdbRydwyKY36lq4H84QQPWOLAvA1kKGUSlVK+QBXAZ+duIFSKlZ1HzGVUmO79+sy\nj3FarZp7399BdWMbz187itAAecjL2WT1DWd4UhivryuR5wKE6KFeFwCtdSdwJ7AMyAXe11rvUUrd\nqpS6tXuzy4HdSqmdwDPAVVprl/m/9IXVRXyZV82v5w5iWKI89OWMlFL8dFIq+2u7RmgJIc7Myxa/\npPuyzuKT3nvxhO+fA56zxb4cbWNRLU8uz+OS4fFcJ6N+nNrMwbEkhPnz6tpiLh4ca3QcIZyePAn8\nPQ43tvLz97aTEhnIY5cNlev+Ts7LbOLmial8vb+OHQePGh1HCKcnBeB7PLE0j4aWDl64djRBvjbp\nLAk7+9GYJIL9vHh1rUwXLcSZSAE4jaa2Tj7PqeAHIxMYEBtsdBzRQ0G+XlwzNpkluyvZfqBrZbbW\nDovBqYRwTlIATmNxTgUtHRauyEo688bCqfxkUipxoX5c88pmHvlsD0MfWcainAqjYwnhdKQAnMb7\n2QdJiwpkVLKM+nE10cF+fHL7BDJignhjw346rZo/LtorD4kJcRIpAKdQXN1EdmkdV2YlyY1fFxUV\n7Mt7C8bxlx8O5bUbs6iob5VegBAnkQJwCh9sLcNsUlw2Utb1dWUBPl78aEwyUwZEExnkw7rCGqMj\nCeFUpACcpNNi5eNtZUzuH0V0iJ/RcYQNKKWYkB7J+sIaXOj5QyHsTgrASdYW1FDV0MYVWYlGRxE2\nNDE9kpqmdvKqGo2OIoTTkAJwkg+2HqRPoA9TB8YYHUXY0IT0rrkH1xXIZSAhviEFoFuHxUpVQysr\n9lZx6YgEfLzkj8adxIf5kxYVKPcBhDiBPN7a7fZ/b+OLfYexWDVXjpHLP+5oUnok72eX0d5plQIv\nBNIDACB7/xFW7K0iLtSPaYOiGRgbYnQkYQcT0iNp6bDwr02lMmW0EEgPAIAnl+cTGeTD8nsuIMBH\n/kjc1cSMSIYnhvKHz/fyzuZS/vzDYYxJ6WN0LCEM4/E9gA1FNWwsruW2yely8HdzAT5efHL7BF64\ndhSdVs01r2zi35tLjY4lhGE8ugBorXlqeT4xIb5ce16y0XGEA5hMillD4/jszolMSI/k4U92838f\n76K902p0NCEczqMLwJqCGrJL67hzagZ+3maj4wgHCvX35rUbx3D75H68u+UAr8j00cIDeWwB0Frz\n5PI8EsL8+ZHM+OmRzCbFAzMHktU3nIU7y42OI4TDeWwBWJl7mJyyen5+UboMCfRwM4fEsq+ykf01\nzUZHEcKhPPLIZ7VqnlqRT0pEAJeNkjH/nm7mkK71g5fsrjQ4iRCO5ZEFYOmeSnIrGrh7WgbeZo/8\nIxAnSAwPYFhiKEt3y3TRwrN43NHP0n32nx4dxLzhMt2z6HLx4Fh2ltVz6GiL0VGEcBiPKwALd5ZT\neLiJe6b1x2ySxV5El1ndl4EW5cjNYOE5PKoAdFqs/H1VAQNjg4//Dy8EQFpUEGNT+/DymhJZOlJ4\nDI8qAB9vP0RJTTP3Tu+PSc7+xUkeuHgANU1t/HP9fqOjCOEQHlMA2jutPLOqgGGJoUzPlLn+xXdl\npfRh6sBoXlpdRP2xDqPjCGF3HlMA3s8+SFldC/dO7y8LvYvTun/GABpaO3lxTZHRUYSwO48oAK0d\nFp77opDRfcO5sH+U0XGEE8uMD2He8Hj+ub6Eww2tRscRwq48ogC8u+UAlQ2t3Cdn/6IH7p3en06L\n5tkvCo2OIoRduX0BaGm38I8vizg/LYLx3evCCvF9UiIDuXJMEu9uOcCB2mNGxxHCbty+ALy1cT81\nTW3cN6O/0VGEC/n51AzMJsXfVuYbHUUIu3HrAtDU1smLq4u4oH8UWbLykzgLsaF+/Hh8Cv/dcYh9\nlQ1GxxHCLty6ALyxvoS6Yx3cO13O/sXZu/XCfgT5ePHXZdILEO7JbQtAfUsHL68pZtqgGEYkhRkd\nR7ig8EAfFlyQxsrcKrYdqDM6jhA257YF4LW1xTS0dsrZv+iVmyemEhnkwxNL89BaGx1HCJty2wLw\n9uYDXDw4hsz4EKOjCBcW6OvFHVPS2Vhcy7rCGqPjCGFTblkAGls7ONLczqjkcKOjCDdwzXnJJIT5\n88Qy6QUI9+KWBaCivusJzrgwf4OTCHfg62Xm7mkZ5JTVs1RWDRNuxC0LQHn3oh7xoX4GJxHu4rKR\nCfSLCuSvy/PotFiNjiOETdikACilZiql8pRShUqph07xuVJKPdP9eY5SapQt9ns63/QA4qUHIGzE\ny2zi/hkDKKpu5uPth4yOI4RN9LoAKKXMwD+AWUAmcLVSKvOkzWYBGd1fC4AXervf71N+tAWTguhg\nX3vuRniYmUNiGZYYyt9XFtDWaTE6jhC9ZosewFigUGtdrLVuB94D5p+0zXzgLd1lExCmlIqzwb5P\nqfxoKzEhfnjJgu/ChpRS/PLiARw62sK/Nx0wOo4QvWaLI2QCcPCE12Xd753tNjZTUd9CnFz/F3Yw\nMT2S89MieP6rQlk6Urg8pztFVkotUEplK6Wyq6urz+l3VNS3ygggYRdKKe6/eAA1Te2ydKRwebYo\nAIeApBNeJ3a/d7bbAKC1fllrnaW1zoqKOvvFW7TWlB9tkRFAwm5G9w3/39KRLbJ0pHBdtigAXwMZ\nSqlUpZQPcBXw2UnbfAbc0D0aaBxQr7WusMG+v0Nr+ODW87luXF97/HohgK5FYxpaO3ltbbHRUYQ4\nZ70uAFrrTuBOYBmQC7yvtd6jlLpVKXVr92aLgWKgEHgFuL23+z0dk0kxLDGMvhGB9tqFEAxJCGX2\n0FheW1dCbVOb0XGEG9lUXMunOw455HkTL1v8Eq31YroO8ie+9+IJ32vgDlvsSwhnce/0/izdXclL\na4r51exBRscRbuL1dSXsOlTPvOHxdt+X090EFsJVpEcHc+mIBN7csJ8qWUBe2EBLu4U1BdXMyIxx\nyPrlUgCE6IVfTOuPxap5ThaQFzawtqCa1g4rMwbHOmR/UgCE6IXkiAAuH53If7IPyr0A0WvL91YR\n4ufF2FTHLGErBUCIXrplUhrtnVb+tanU6CjChXVarKzKreKiQTF4O2gWAykAQvRSenQQUwdG86+N\npbR2yBxB4txkl9ZRd6yDGZkxDtunFAAhbOCWSanUNrfzX5kpVJyDfZUNPLUiHx8vExf0P/sHYM+V\nTYaBCuHpzk+LIDMuhFfXlRDq782uQ/U8MHOg0bGEk8utaOCZVQUs2V1JoI+Zh2cPItDXcYdlKQBC\n2IBSip9ekMo9/9nJ3e/toN1iZcrAaMakOOZmnnA9z6wq4KkV+QT7enHX1HR+MjGVsAAfh2aQS0BC\n2MicofHEhvjRbrES7OvF31bkGx1JOKnWDguvrClm8oAo1j04lftmDHD4wR+kByCEzfh4mXhk3mDy\nqxoJ8DHzx0W5bC6u5by0CKOjCSezOr+axrZObp6QSmiAt2E5pAcghA3NHBLLzy/K4LpxfYkK9uVv\nK6UXIL7r85wK+gT6ML6fsScHUgCEsAM/bzO3XdiPTcVH2FhUa3Qc4USOtXeycm8Vs4bEGr5qoRQA\nIezkmvOSie7uBXTNhygEfLHvMC0dFi5xwGRvZyIFQAg78fM2c/vkfmwpkV6A+J+FO8uJDvZ1ihFi\nUgCEsKOrxiYTE+LL0ysLpBcgaGzt4Mu8amYPjcNssv9sn2ciBUAIO+rqBaSzZf8RNkgvwOOt2FtF\ne6fVKS7/gBQAIezuR2OSiA3x428r5F6Ap/s8p4KEMH9GJYcZHQWQAiCE3fl5m7ljSj+yS+tYV1hj\ndBxhkKPH2lmTX83cYXEOWeylJ6QACOEAV45JIj5UegGebNmeSjqtmrnDnOPyD0gBEMIhfL3M3D4l\nnW0HjrKmQHoBnmjhzgpSIgIYkhBidJTjpAAI4SBXZiWREOYvvQAPVN3YxoaiGuYOi3eayz8gBUAI\nh/HxMnHHlHR2HDzKV/nVRscRDrR0dwVWjdOM/vmGFAAhHOjy0YkkhPnLcwEeZuHOCjKigxgQG2x0\nlG+RAiCEA/l4mbhzajo7Dx7lqzzpBXiCivoWvi494nRn/yAFQAiHu3x0Ionh/jJHkIdYlFOB1jB3\nWJzRUb5DCoAQDuZtNnHX1HRyyur5Yt9ho+MIO2nrtGCxaj7PqWBwfAhpUUFGR/oOKQBCGOCyUYkk\n9wmQewFuqsNiZebTa/npW9nsOHjUqcb+n0gKgBAG8DZ33QvYdaielbnSC3A3C3eWU1LTfLyH54yX\nf0AKgBCGuWxkAn0jAnha7gW4Fa01L68pJqmPP2aTYkRSGEl9AoyOdUqyJrAQBvEym7hragb3f7CT\n5XuruHhwrNGRhA2szq9mX2UjT14xHB8vE4nh/kZHOi3pAQhhoEtHxJMS0XUvwGqVXoA7eGl1MbEh\nflwyPJ5LhsczMjnc6EinJQVACAN90wvIrWhg+d4qo+OIXsopO8rG4lp+MjEVHy/nP7w6f0Ih3Nz8\nEfGkRgby9Mp86QW4uJfWFBPs58VVY5OMjtIjUgCEMJiX2cTPL0pnX2Ujy/ZUGh1HnKMjze0s3V3J\n1WOTCfbzNjpOj0gBEMIJzBueQFpUoNwLcGHL91RisWrmOeGUD6cjBUAIJ2A2Ke6+KIO8qkaW7JZe\ngCtatKuCvhEBDI53nvn+z0QKgBBOYu6wePpFBfL3VXIvwNUUVTexoaiWOUOdZ7nHnpACIISTMJsU\nd05NJ7+qiS37jxgdR5yFx5fuw8/LxM0TU42OclZ6VQCUUn2UUiuUUgXd/z3lgFel1H6l1C6l1A6l\nVHZv9imEO5vcPxqAraV1BicRPbW19AjL9lTxswv7ERnka3Scs9LbHsBDwCqtdQawqvv16UzRWo/Q\nWmf1cp9CuK3wQB/SIgPZfkAKgCvQWvPY4n1EBftyyyTXOvuH3heA+cCb3d+/CVzay98nhMcbmRzO\ntgNHZX4gF7B8bxXZpXXcM60/AT6uN7NObwtAjNa6ovv7SiDmNNtpYKVSaqtSakEv9ymEWxvVN4wj\nze0UVTcbHUV8j06LlceX7qNfVCBXZiUaHeecnLEAKKVWKqV2n+Jr/onb6a7TldOdskzUWo8AZgF3\nKKUu+J79LVBKZSulsqurZck84XmmDIjGz9vEn5fso7G1g0c+20NtU5vRscRJ3s8uo6i6mQdmDsTL\n7Jrjac7YZ9FaTzvdZ0qpKqVUnNa6QikVB5xyYnOt9aHu/x5WSn0CjAXWnGbbl4GXAbKysqQPLDxO\nfJg/90zrz2NL9vGTN7PZUnKExHB/bpmUZnQ00e1Yeyd/W5nP6L7hzMg83YUP59fbsvUZcGP39zcC\nn568gVIqUCkV/M33wAxgdy/3K4Rb+8nEVIYkhLClpGs46LrCGoMTiRO9traE6sY2fjV7oEuN+z9Z\nbwvAn4HpSqkCYFr3a5RS8Uqpxd3bxADrlFI7gS3AIq310l7uVwi35mU28fgPhzMkIYTx/SLYVFxL\nW6fF6Fge5eCRY8x8eg0PfpjDxqLa4w/n1TS18dKaYmZkxjC6bx+DU/ZOr25ba61rgYtO8X45MLv7\n+2JgeG/2I4QnyowP4fO7JrEqt4oNb9aydX8d49MjjY7lMV5fX0Lh4SYOHjnGf7IPEhfqx/wRCVTU\nt9DSYeGBmQONjthrrjduSQgPMy4tAm+zYk1BjRQAB2lq6+SD7DLmDovjscuGsSK3iv9uP8Qra4ux\nWDXXnJdMenSQ0TF7TQqAEE4u0NeLUcnhrC2o5qFZrn/W6Qo+2lpGU1snP56Qir+PmXnD45k3PJ6a\npjbWF9YwdWC00RFtwjXHLgnhYS7oH8We8gYZDuoAVqvmzQ37GZEUxoiksG99Fhnky/wRCS4z3/+Z\nSAEQwgVckBEFwNMrC+QJYTtbU1BNcU0zN01IMTqK3UkBEMIFDE0M5ZaJqfxrUyl/WpRLS7uF+z/Y\nyTaZM8jm3tiwn6hgX2YNiTM6it3JPQAhXMTDcwbRadW8uq6EtQU15FU1sv1AHUt/cQHeLvokqrMp\nrm7iq7xq7pnW3yUWde8t92+hEG5CKcVvL8nkunHJ5FU1MiAmmKLqZt7csN/oaG7jrY2leJsV15yX\nbHQUh5ACIIQLUUrx+3lDePG6UXxw2/lc2D+Kv68soEZuDvdaY2sHH2Qf5JJh8UQFu9a8/udKCoAQ\nLsZkUswcEkeInze/nptJS4eFJ5bmGR3L5X24tYzmdgs3jk8xOorDSAEQwoWlRwdx04QU3t96kF1l\n9UbHcVlWq+atjaWMSg5j+ElDP92ZFAAhXNxdF2UQEejDIwv3yBDRc/TFvsOU1DTz4wmut6pXb0gB\nEMLFhfh588DFA9laWsenO8qNjuNyrFbN31bm0zcigFlDYo2O41BSAIRwA5ePTmRYYiiPLcmlua3T\n6DguZdnFaoI3AAAM6ElEQVSeSvaUN/CLaRkeN5zWs1orhJsymRSPzBtMVUMb//iy0Og4LsNi1Ty1\nIp/06CDmDU8wOo7DSQEQwk2MSg7nspEJvLq2hNJaWU+4JxbuLKfgcBP3TOuP2eS6C7ucKykAQriR\nB2cNxNuseOijXccXMBGn1mGx8vTKfAbFhXjctf9vSAEQwo3EhPjx20sGs7G4lrc3lxodx6l9vK2M\n/bXHuG96f0weePYPUgCEcDtXZCUyMDaYJbsqjY7itNo6LTyzqpDhSWFcNMg95vY/F1IAhHAzSinG\npUWw/WAd7Z1Wo+M4pf98fZBDR1u4f0Z/l17UvbekAAjhhsam9qG1w8rucnk6+GQt7Rae/aKQsal9\nmOjhS2xKARDCDY1J6QPA1yVHsFq1PCF8grc3lVLd2MZ90z377B+kAAjhlqKCfUnuE8COg0e54qWN\n3PJmttGRnEJTWycvrC5iUkYk56VFGB3HcLIgjBBuKjMuhB0Hj1JR3wrA8j2VzBjsmcMdv/HG+hKO\nNLdz34wBRkdxCtIDEMJNDYwLPn7wB3h9fYmBaRzPatX86pNdzP/HerTW1B/r4KU1xUwbFP2dxd49\nlfQAhHBTA2NDjn8/b3g8S/dU0tphwc/bbGAqx3nuy0Le2XwAgNLaY3y0rYzG1k7umd7f4GTOQ3oA\nQripQXHBQNf9gEtHxtPeaWVbqWcsIl/T1MaLq4tIiwoEuqZ7fn1dCXOGxjE4PtTgdM5DCoAQbiop\nPIBAHzND4kMYk9IHs0mxsbjW6FgO8fyXRbR2WHjputEE+Jh5YlkeLR0W7pmeYXQ0pyIFQAg3ZTIp\nHr1sKHdP60+wnzdDE0LZUOT+BaCivoW3N5fyw1GJZMQEMyQ+lJYOC5eOSCA9OtjoeE5FCoAQbmz+\niITjNzzH94tg58Gjbr9ewDOrCtFac/e0rrP90SnheJkUP79Izv5PJgVACA8xvl8knVbN1/uPGB3F\nbvbXNPN+9kGuGZtMYngAAHdMSWfx3ZNIiQw0OJ3zkQIghIcY3Tccb7NiY/dloBe+KuKnb2XzyfYy\nmtykV/D0yny8zYo7pqYffy/I14v+MXLp51RkGKgQHsLfx8zI5HA2FtdSUd/CUyvyUEqxYm8Vvl67\nmDIgmrnD47hoYAz+Pq43VHRfZQOf7iznZxf0IzrYz+g4LkEKgBAe5Py0CJ79ooCnludj1fDlvRdy\nuLGVz3MqWLSrgqV7KhmZHMYnt08wOmqPHKg9xue7ylmbX8PW0jqCfLy49cI0o2O5DCkAQniQ8f0i\n+PuqAj7YWsZlIxNIjgggOSKArJQ+/HpuJk+vzOfZLwqpqG8hLtTf6Lin1dph4fkvC3lxdTHtFisD\nY4P58YQUfjAygbAAH6PjuQwpAEJ4kBHJYfh5m2jtsHLr5H7f+sxsUswcEsuzXxSyufgIl478/kXS\nj7V3sqGwlmmZMfaM/B3Prirg5TXFNLZ1cumIeH45cyAJYc5brJyZFAAhPIivl5lLhsWjFKe8MToo\nNoRgXy9+8Z8dLNldwYvXjT7tlMl/XZbP6+tLWHL3JAbFhZxyG1vbWFTLkyvymTIgitunpB+f9lqc\nGxkFJISHeeKK4Tx++fBTfmYyqeMH82V7qth+8Ogpt6tubOPf3WsO2+rhMotVc8WLG/hwa9kpP2/t\nsPCrT3aR3CeA568dLQd/G5ACIIT4lscvH8YzV4/E39vMM6sK+N3CPdzxzjbeWF9CflUjWmteXVtM\nh8VKRKAPG4tqzur3t7Rb+CD7IA2tHd96P3v/Eb7eX8dXeYdP+XPPrCqgpKaZxy4b6pKjlJyRXAIS\nQnxLSmQgKZGBrM6r5qNtZWwsqiUi0IdFORUAJIT5U3esnUuGxxPo68XCHeV0Wqx4mb//fFJrzdLd\nlfxxUS6HjrZQ09TObSfch1i0q+v3l9Q0f+dn95Y38NKaYq4YncgED1/G0ZZ6VQCUUlcAjwCDgLFa\n61MuO6SUmgn8HTADr2qt/9yb/Qoh7O+38zK5dlwyg+ND8PUyc/DIMTYW1fLqumIqG6zcOSWdvKpG\n3tl8gN3lDd87x37h4SZ+t3APawtqGBgbTGSQz7fWK7ZYNYt3VQJdBUBrffzeg8WqeejjHMIDvHl4\nziD7NtrD9LYHsBu4DHjpdBsopczAP4DpQBnwtVLqM6313l7uWwhhRyF+3oxKDj/+OqlPAEl9Apg/\nMp7DDW0k9QkgPLBryOWGoppTFoCmtk6eXVXAa+tK8Pcx87t5g7n2vGTueGcbueUNx7fbUnKEmqY2\nxqX1YVPxEaoa2lhXWENxdRPVjW3klNXz7NUjZYinjfWqAGitc4EzLaw8FijUWhd3b/seMB+QAiCE\nC/L1MpPUp2uencggXwbGBrOxqJbbJ/9v+gWtNZ/tLOfRxblUNbRxZVYiD8wcSGSQLwCZcaEs31tF\nc1sngb5eLNpVjp+3iZ9MTGNT8RH+tDiXhTvLUQq0humZMcwdFmdIe92ZI+4BJAAHT3hdBpzngP0K\nIRzg/H4RvLvlAAt3lpMeHYTFqvnjor1sKj7CkIQQXrhu9Ld6EgCD40PQumv6huGJYSzdXclFA2MY\nHN81AmnhznKmDozmpetH09jaSYif15lONMU5OGMBUEqtBE61kvTDWutPbR1IKbUAWACQnJxs618v\nhLCxmYNjeXtTKXe9u/34e6H+3vzx0iFcPTYZs+m7B+7BCV0H+o+3HaK1w0pNUztzhsURG+JHsJ8X\nCWH+PHP1SLzNJvoEymUfezljAdBaT+vlPg4BSSe8Tux+73T7exl4GSArK0v3ct9CCDs7Ly2CXY9c\nTElNMwWHm6htamP+iITvPXDHhfpz04QU/rl+P6vzq/H3NjNlQDQmk+I/C84nPsyPIF8ZpGhvjvgT\n/hrIUEql0nXgvwq4xgH7FUI4iJ+3mUFxIWf1RPCv52RS3djG5zkVzB0Wd3xsf2a8Y54qFr0fBvoD\n4FkgCliklNqhtb5YKRVP13DP2VrrTqXUncAyuoaBvq613tPr5EIIl2YyKZ68cjhpkYHMGxFvdByP\npLR23qssWVlZOjv7lI8WCCGEOAWl1FatdVZPtpWpIIQQwkNJARBCCA8lBUAIITyUFAAhhPBQUgCE\nEMJDSQEQQggPJQVACCE8lBQAIYTwUE79IJhSqhooPcsfiwTObo061+eJbQZpt6fxxHafS5v7aq2j\nerKhUxeAc6GUyu7pU3DuwhPbDNJuo3M4mie2295tlktAQgjhoaQACCGEh3LHAvCy0QEM4IltBmm3\np/HEdtu1zW53D0AIIUTPuGMPQAghRA+4ZAFQSs1USuUppQqVUg+d4nOllHqm+/McpdQoI3LaWg/a\nfW13e3cppTYopYYbkdPWztTuE7Ybo5TqVEpd7sh89tCTNiulJiuldiil9iilVjs6oz304N94qFJq\noVJqZ3e7bzIipy0ppV5XSh1WSu0+zef2O55prV3qi65VxYqANMAH2AlknrTNbGAJoIBxwGajczuo\n3eOB8O7vZ3lKu0/Y7gtgMXC50bkd8HcdBuwFkrtfRxud20Ht/hXwl+7vo4AjgI/R2XvZ7guAUcDu\n03xut+OZK/YAxgKFWutirXU78B4w/6Rt5gNv6S6bgDClVJyjg9rYGduttd6gta7rfrkJSHRwRnvo\nyd83wF3AR8BhR4azk560+RrgY631AQCttae0WwPBSikFBNFVADodG9O2tNZr6GrH6djteOaKBSAB\nOHjC67Lu9852G1dztm36CV1nDa7ujO1WSiUAPwBecGAue+rJ33V/IFwp9ZVSaqtS6gaHpbOfnrT7\nOWAQUA7sAu7WWlsdE88wdjue9WpReOGclFJT6CoAE43O4iBPAw9qra1dJ4YewQsYDVwE+AMblVKb\ntNb5xsayu4uBHcBUoB+wQim1VmvdYGws1+SKBeAQkHTC68Tu9852G1fTozYppYYBrwKztNa1Dspm\nTz1pdxbwXvfBPxKYrZTq1Fr/1zERba4nbS4DarXWzUCzUmoNMBxw5QLQk3bfBPxZd10cL1RKlQAD\ngS2OiWgIux3PXPES0NdAhlIqVSnlA1wFfHbSNp8BN3TfPR8H1GutKxwd1MbO2G6lVDLwMXC9G50J\nnrHdWutUrXWK1joF+BC43YUP/tCzf+OfAhOVUl5KqQDgPCDXwTltrSftPkBXrwelVAwwACh2aErH\ns9vxzOV6AFrrTqXUncAyukYNvK613qOUurX78xfpGgkyGygEjtF11uDSetju3wARwPPdZ8Od2sUn\nz+phu91KT9qstc5VSi0FcgAr8KrW+pTDCF1FD/+u/wC8oZTaRdeomAe11i49Q6hS6l1gMhCplCoD\nfgt4g/2PZ/IksBBCeChXvAQkhBDCBqQACCGEh5ICIIQQHkoKgBBCeCgpAEII4aGkAAghhIeSAiCE\nEB5KCoAQQnio/w/hGtVdpxs96AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x146c6719f28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "N = 101\n",
    "(xtrain, ttrain), (xdev, tdev), (xtest, ttest) = generateDataset3(N, sin2pi, 0.03)\n",
    "\n",
    "plt.plot(xtrain, ttrain)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we will change the OptimizePls function to return optimal parameters w_PLS given M and lambda."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1457,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def optimizePLS(x, t, M, lam):\n",
    "    \"\"\"\n",
    "    returns the optimal parameters wPLS given M and lambda\n",
    "    \"\"\"\n",
    "    cols = M+1\n",
    "    rows = len(t)\n",
    "    \n",
    "    phi = np.array([polynom(x[i], M) for i in range(rows)])\n",
    "    prod = np.dot(phi.T, phi) \n",
    "    \n",
    "    m = np.dot(np.linalg.inv(prod + np.identity(M+1)*lam), phi.T)   \n",
    "    w = np.dot(m, t)\n",
    "    return w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will try and optimize the value of λ by varying the value of ln(λ) from -40 to -20 on the development set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1111,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def norm_error(x, t, y):\n",
    "    \"\"\"return the normalized error of t against y\"\"\"\n",
    "    diff = t-y\n",
    "    return math.sqrt(sum(elem**2 for elem in diff))/len(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1011,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def norm_errors(x, t, polynoms):\n",
    "    \"\"\"return normalized errors of t against y for each y=p(x) in polynoms\"\"\"\n",
    "    norm_errors = []\n",
    "    for p in polynoms:\n",
    "        y = np.array([p(x[j]) for j in range(len(x))])\n",
    "        norm_errors.append(norm_error(x, t, y))\n",
    "    return norm_errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1276,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plot_errors(N):\n",
    "    (xtrain, ttrain), (xdev, tdev), (xtest, ttest) = generateDataset3(N, sin2pi, 0.03)\n",
    "\n",
    "    lambdas = [1/math.exp(i) for i in range(40, 19, -5)]\n",
    "    ln_lambdas = list(range(-40, -19, 5))\n",
    "    M = 9\n",
    "\n",
    "    polynoms = []\n",
    "\n",
    "    for lam in lambdas:\n",
    "        w = optimizePLS(xtrain, ttrain, M, lam)\n",
    "        p = np.poly1d(w[::-1])\n",
    "        polynoms.append(p)\n",
    "\n",
    "    plt.plot(ln_lambdas, norm_errors(xtrain, ttrain, polynoms))\n",
    "    plt.plot(ln_lambdas, norm_errors(xdev, tdev, polynoms))\n",
    "    plt.plot(ln_lambdas, norm_errors(xtest, ttest, polynoms))\n",
    "    plt.legend(['train', 'dev', 'test'], loc='upper right')\n",
    "    plt.xlabel(\"ln(λ)\")\n",
    "    plt.ylabel(\"error\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1280,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYwAAAEKCAYAAAAB0GKPAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmYFOW5xuHfOzvDzjCKbIIGF1xAGHABFExUMIpxiajB\nbJ4gURPjMUbAaFxi1MRjjHHFSIxHkUTjLlH0KIKiIhBQkB0xgAs4KLLMwCzv+aNroBm6hx7o7uqZ\nee7r6mu6vvqq+u2mmGe+quoqc3dERER2JyvsAkREpGFQYIiISEIUGCIikhAFhoiIJESBISIiCVFg\niIhIQhQYIiKSEAWGiIgkRIEhIiIJyQm7gGRq3769d+vWLewyREQajNmzZ3/h7sWJ9G1UgdGtWzdm\nzZoVdhkiIg2GmX2caF/tkhIRkYQoMEREJCEKDBERSUijOoYhIlJfFRUVrF69mvLy8rBLSamCggI6\nd+5Mbm7uHq9DgSEiTdrq1atp2bIl3bp1w8zCLicl3J3S0lJWr15N9+7d93g9KdslZWZdzOx1M/vQ\nzBaY2eUx+piZ3WVmy8zsfTPrEzVvqJktDuaNSVWdItK0lZeXU1RU1GjDAsDMKCoq2utRVCqPYVQC\nV7p7T+AY4FIz61mrzzCgR/AYBdwHYGbZwD3B/J7A+TGWFRFJisYcFjWS8R5TtkvK3T8FPg2ebzSz\nhUAn4MOobmcAj3jkPrHvmFkbM9sP6AYsc/cVAGY2KegbvWzyvPF7yMqG/FaQ1wLyW0J+8DOv5Y7p\n3OaQpfMERKRpSssxDDPrBhwFvFtrVidgVdT06qAtVvvRKSvwzTuhYnMCHW3nQNn+PHjsFDYKHxHZ\nva+++oqJEydyySWX1Gu5U089lYkTJ9KmTZsUVbarlAeGmbUA/gn8wt2/TsH6RxHZnUXXrl33bCXj\n1kDlVti2CbZ+DVs3wdaNwfTGHY9405u/CKaD9urKRCoPAqVFrbBR+Ig0JV999RX33nvvLoFRWVlJ\nTk78X9GTJ09OdWm7SGlgmFkukbB4zN2fitFlDdAlarpz0JYbp30X7j4eGA9QUlLie1go5BZEHs3b\n79EqogqKhE90gGzdlLnhEx08+S0VPiJpNmbMGJYvX07v3r3Jzc2loKCAtm3bsmjRIpYsWcJ3vvMd\nVq1aRXl5OZdffjmjRo0CdlwKadOmTQwbNoyBAwcyY8YMOnXqxLPPPkuzZs2SXmvKAsMiR1geAha6\n+x1xuj0HXBYcozga2ODun5rZOqCHmXUnEhTnARekqtakig4fErqeV3zxwmd7wNQeDX29czilMnx6\nXwDte+zd+xPJMDc8v4APP0nujpCeHVvxm9MPizv/1ltvZf78+cydO5epU6fy7W9/m/nz528//XXC\nhAm0a9eOsrIy+vXrx9lnn01RUdFO61i6dCmPP/44Dz74IOeeey7//Oc/GTlyZFLfB6R2hDEAuBD4\nwMzmBm3jgK4A7n4/MBk4FVgGbAF+FMyrNLPLgJeBbGCCuy9IYa2ZKRPCp2Z687ogjDZC+QZY9AJc\n8k7kZAERSZr+/fvv9F2Ju+66i6effhqAVatWsXTp0l0Co3v37vTu3RuAvn37snLlypTUlsqzpN4E\n6jyPKzg76tI48yYTCRRJhmSGz4Jn4IkfwPyn4MjvJqU8kUxQ10ggXZo3b779+dSpU3n11Vd5++23\nKSwsZPDgwTG/S5Gfn7/9eXZ2NmVlZSmpTTurpf4OHQ77HAZv3AbVVWFXI9KgtWzZko0bN8act2HD\nBtq2bUthYSGLFi3inXfeSXN1O1NgSP1lZcHgq6F0KXzwZNjViDRoRUVFDBgwgMMPP5yrrrpqp3lD\nhw6lsrKSQw89lDFjxnDMMceEVGWERfYKNQ4lJSWuGyilSXU1PDAIKsrg0pmQrcuSScO0cOFCDj30\n0LDLSItY79XMZrt7SSLLa4QheyYrCwaPgfXL4YMnwq5GRNJAgSF77pDToMMRMO33UJXIKbsi0pAp\nMGTPmcHgsbB+Bbz/97CrEZEUU2DI3jn4VNivVzDKqAi7GhFJIQWG7J2aUcaXK2HepLCrEZEUUmDI\n3jtoKHQ8Cqb9QaMMkUZMgSF7r2aU8dXHMHdi2NWINHjXX389t99+e9hl7EKBIcnR42To1Bem3Q6V\n28KuRkRSQIEhyVEzytjwH5j7WNjViDQ4N998MwcddBADBw5k8eLFACxfvpyhQ4fSt29fBg0axKJF\ni9iwYQP7778/1dXVAGzevJkuXbpQUZH63cH6eq4kzze+BZ1KYPr/QO/vQU5e2BWJ1M+/xsBnHyR3\nnR2OgGG31tll9uzZTJo0iblz51JZWUmfPn3o27cvo0aN4v7776dHjx68++67XHLJJbz22mv07t2b\nN954gyFDhvDCCy9wyimnkJubm9y6Y1BgSPKYwZCx8OjZ8O//hX4XhV2RSIMwffp0zjzzTAoLCwEY\nPnw45eXlzJgxg+9+d8cVobdu3QrAiBEj+Pvf/86QIUOYNGlSvW/vuqcUGJJcB34TOvePjDKOGgk5\n+btfRiRT7GYkkE7V1dW0adOGuXPn7jJv+PDhjBs3jvXr1zN79mxOPPHEtNSkYxiSXDWjjK/XwJxH\nwq5GpEE4/vjjeeaZZygrK2Pjxo08//zzFBYW0r17d554InKtNndn3rx5ALRo0YJ+/fpx+eWXc9pp\np5GdnZ4bmaUsMMxsgpmtNbP5ceZfZWZzg8d8M6sys3bBvJVm9kEwT5efbWgOGAJdjoHpd0DFrjd7\nEZGd9enThxEjRtCrVy+GDRtGv379AHjsscd46KGH6NWrF4cddhjPPvvs9mVGjBjBo48+yogRI9JW\nZ8oub25mxwObgEfc/fDd9D0duMLdTwymVwIl7v5FfV5TlzfPICumwiNnwKm3Q/+fhF2NSFy6vHkG\nXN7c3acB6xPsfj7weKpqkRB0PwG6Hhc5lqFRhkijEPoxDDMrBIYC/4xqduBVM5ttZqPCqUz2Ss2x\njI2fwpy/hV2NiCRB6IEBnA685e7Ro5GB7t4bGAZcGuzeisnMRpnZLDObtW7dulTXKvXRbRDsPyA4\nlpGam9KLSPpkQmCcR63dUe6+Jvi5Fnga6B9vYXcf7+4l7l5SXFyc0kKlnmq+/b3pM5j9cNjViMhe\nCjUwzKw1cALwbFRbczNrWfMcOBmIeaaVNADdB0VGGm/+UaMMkQYulafVPg68DRxsZqvN7CIzG21m\no6O6nQlMcffNUW37Am+a2TxgJvCiu7+UqjolDQaPhU2fw6wJYVciInshlWdJne/u+7l7rrt3dveH\n3P1+d78/qs/D7n5ereVWuHuv4HGYu9+cqholTboNgO7Hw5t3wrYtYVcjklG++uor7r333j1a9s47\n72TLlvT9n8qEYxjSFAweB5vXwqyHwq5EJKM0pMDQtaQkPfY/Fg4YHBlllPwY8pqHXZFIRhgzZgzL\nly+nd+/enHTSSeyzzz784x//YOvWrZx55pnccMMNbN68mXPPPZfVq1dTVVXFtddey+eff84nn3zC\nkCFDaN++Pa+//nrKa1VgSPoMHgcTTob3/gIDLg+7GpFd3DbzNhatX5TUdR7S7hCu7n913Pm33nor\n8+fPZ+7cuUyZMoUnn3ySmTNn4u4MHz6cadOmsW7dOjp27MiLL74IwIYNG2jdujV33HEHr7/+Ou3b\nt09qzfFol5SkT9ej4cAT4a0/wdZNYVcjknGmTJnClClTOOqoo+jTpw+LFi1i6dKlHHHEEbzyyitc\nffXVTJ8+ndatW4dSn0YYkl6Dx8FD34L3HoSBV4RdjchO6hoJpIO7M3bsWC6++OJd5s2ZM4fJkyfz\n61//mm9+85tcd911aa9PIwxJry79Infme+su2Lox7GpEQteyZUs2boz8XzjllFOYMGECmzZFRuBr\n1qxh7dq1fPLJJxQWFjJy5Eiuuuoq5syZs8uy6aARhqTf4HHwlxNh5ngYdGXY1YiEqqioiAEDBnD4\n4YczbNgwLrjgAo499lggct+LRx99lGXLlnHVVVeRlZVFbm4u9913HwCjRo1i6NChdOzYMS0HvVN2\nefMw6PLmDchj34XV78Hl70NBq7CrkSZMlzfPgMubi9Rp8Bgo+xJmPhB2JSKSIAWGhKNTXzhoKMy4\nG8o3hF2NiCRAgSHhGTwGyr+Cd8eHXYk0cY1p13w8yXiPCgwJT8ej4OBT4e0/a5QhoSkoKKC0tLRR\nh4a7U1paSkFBwV6tR2dJSbgGj4EHjod37ofB4Z4DL01T586dWb16NY39BmwFBQV07tx5r9ahwJBw\n7dcLDjkN3r4Hjr4YmrUJuyJpYnJzc+nevXvYZTQI2iUl4Tvhati6Ad65L+xKRKQOCgwJ335HRkYZ\n79wbOdVWRDJSKu+4N8HM1ppZzNurmtlgM9tgZnODx3VR84aa2WIzW2ZmY1JVo2SQwWNh69fw9p7d\nF0BEUi+VI4yHgaG76TPd3XsHjxsBzCwbuAcYBvQEzjeznimsUzJBh8Ph0OGR3VJb1oddjYjEkMpb\ntE4D9uR/fn9gWXCr1m3AJOCMpBYnmWnwGNi2MXIAXEQyTtjHMI4zs/fN7F9mdljQ1glYFdVnddAW\nk5mNMrNZZjarsZ8W1+jtexj0/A68e79GGSIZKMzAmAN0dfcjgT8Dz+zJStx9vLuXuHtJcXFxUguU\nEAweA9s2w4w/h12JiNQSWmC4+9fuvil4PhnINbP2wBqgS1TXzkGbNAX7HAqHnRm59Pnm0rCrEZEo\noQWGmXUwMwue9w9qKQXeA3qYWXczywPOA54Lq04JwQlXB6OMu8KuRESipOyb3mb2ODAYaG9mq4Hf\nALkA7n4/cA7wUzOrBMqA8zxyMZdKM7sMeBnIBia4+4JU1SkZaJ9D4PCzYeaDcNzPoHl6bnAvInXT\nDZQkM61bAvceDcdeBiffFHY1Io2WbqAkDV/xQXD4OfDeX2CTzn4TyQQKDMlcJ1wNleXw1p1hVyIi\nKDAkk7X/BhxxLrz3EGz8POxqRJo8BYZkthN+BVXb4K0/hV2JSJOnwJDMVnQgHDkCZmmUIRI2BYZk\nvhOugqoKHcsQCZkCQzJfuwOg1/kwawJs/CzsakSaLAWGNAzH/xKqK+HNP4ZdiUiTpcCQhqFd92CU\n8Vf4+pOwqxFpkhQY0nAc/0vwKo0yREKiwJCGo2036H0BzH4YNugCxiLppsCQhmXQL8Gr4c07wq5E\npMlRYEjD0nZ/OGokzHkENqwOuxqRJkWBIQ3PoF+CO0z/n7ArEWlSFBjS8LTpAn0uhDn/C1/9J+xq\nRJqMlAWGmU0ws7VmNj/O/O+Z2ftm9oGZzTCzXlHzVgbtc81MN7iQXQ26Esw0yhBJo1SOMB4GhtYx\n/yPgBHc/ArgJGF9r/hB3753ojT2kiWndGfp8H/79KHz5cdjViDQJKQsMd58GrK9j/gx3/zKYfAfo\nnKpapJEa+N9gWTD99rArEWkSMuUYxkXAv6KmHXjVzGab2aiQapJM17oT9P0hzJ0IX64MuxqRRi/0\nwDCzIUQC4+qo5oHu3hsYBlxqZsfXsfwoM5tlZrPWrdOtPJucgf8Nlg3T/hB2JSKNXqiBYWZHAn8B\nznD30pp2d18T/FwLPA30j7cOdx/v7iXuXlJcXJzqkiXTtNoPSn4Ecx+H9SvCrkakUQstMMysK/AU\ncKG7L4lqb25mLWueAycDMc+0EgFg4BWQnQvTdCxDJJVSeVrt48DbwMFmttrMLjKz0WY2OuhyHVAE\n3Fvr9Nl9gTfNbB4wE3jR3V9KVZ3SCLTsACU/hnmToHR52NWINFrm7mHXkDQlJSU+a5a+ttEkbfwc\n/tQLDjsTzrwv7GpEGgwzm53o1xdCP+gtkhQt94V+F8H7GmWIpIoCQxqPAZdDdj688fuwKxFplBQY\n0ni02Af6/xd88A/4YmnY1Yg0OgoMaVyOuxxyCjTKEEkBBYY0Li2Kod9/wfwnYd2S3fcXkYQpMKTx\nGXA55DSDN24LuxKRRkWBIY1P8/bQ/ycw/5+wdlHY1Yg0GgoMaZyO+znkNdcoQySJFBjSODUvgv6j\nYMHTsHZh2NWINAq7DQwzyzazK9JRjEhSHfczyGsBU28NuxKRRmG3geHuVcD5aahFJLkK28HRF8OH\nz8DnC8KuRqTBS3SX1FtmdreZDTKzPjWPlFYmkgzHXgr5rTTKEEmCnAT79Q5+3hjV5sCJyS1HJMkK\n28HRo2Ha7+GzD6DDEWFXJNJgJRQY7j4k1YWIpMyxl8C7D0RGGec9FnY1Ig1WQrukzKy1md1RcytU\nM/sfM2ud6uJEkqJZWzjmp7DoBfj0/bCrEWmwEj2GMQHYCJwbPL4G/pqqokSS7pifQn5rHcsQ2QuJ\nBsaB7v4bd18RPG4ADqhrATObYGZrzSzm7VUt4i4zW2Zm70cfRDezoWa2OJg3JvG3IxJHszaRA+CL\nX4RP54VdjUiDlGhglJnZwJoJMxsAlO1mmYeBoXXMHwb0CB6jgPuCdWcD9wTzewLnm1nPBOsUie+Y\n0VCgUYbInko0MEYD95jZSjNbCdwNXFzXAu4+DVhfR5czgEc84h2gjZntB/QHlgUjmW3ApKCvyN4p\naA3H/gwWT4ZP/h12NSINTiLf9M4CDnb3XsCRwJHufpS77+3Rw07Aqqjp1UFbvHaRvXf0xZGD4Bpl\niNRbIt/0rgZ+FTz/2t2/TnlV9WBmo2rO3lq3bl3Y5UimK2gFx14GS16CNbPDrkakQUl0l9SrZvZL\nM+tiZu1qHnv52muALlHTnYO2eO0xuft4dy9x95Li4uK9LEmahKMvhmbtNMoQqadEA2MEcCkwDZgd\nPGbt5Ws/B3w/OFvqGGCDu38KvAf0MLPuZpYHnBf0FUmO/JaRCxMunQKr93YzFmk6Ej2GMdLdu9d6\n7O602seBt4GDzWy1mV1kZqPNbHTQZTKwAlgGPAhcAuDulcBlwMvAQuAf7q4rx0ly9R8VjDJuCbsS\nkQZjt5cGcfdqM7sbOKo+K3b3Oq9w6+5OZNQSa95kIoEikhr5LWDAz+HV62HVTOjSP+yKRDJeoruk\n/s/MzjYzS2k1IunU7ydQWKRRhkiCEg2Mi4F/AFvN7Gsz22hmGXW2lEi95beAAZfD8tfgP++GXY1I\nxks0MFoDPwR+6+6tgMOAk1JVlEja9PsvaF4MU38XdiUiGS/RwLgHOIYdd97bSOTb3iINW17zyChj\nxVT4+O2wqxHJaIkGxtHufilQDuDuXwJ5KatKJJ1KLoLm+2iUIbIbiQZGRXBRQAcws2KgOmVViaRT\nXiEM/AV8NA1WvhV2NSIZK9HAuAt4GtjHzG4G3gT055g0HiU/hhb76owpkTokFBju/hiR60ndAnwK\nfMfdn0hlYSJpldsMBl4BK6fDR9PDrkYkIyU6wsDdF7n7Pe5+t7svTGVRIqHo+0No0SEyynAPuxqR\njJNwYIg0ernNYNB/w8dvRUYaIrITBYZItD4/gJYd4XWNMkRqU2CIRMstiIwy/jMDPnoj7GpEMooC\nQ6S2Pt+HVp00yhCpRYEhUltOfmSUseodWPF62NWIZAwFhkgsR10IrTprlCESRYEhEktOPhx/Jaye\nCcv/L+xqRDJCSgPDzIaa2WIzW2ZmY2LMv8rM5gaP+WZWVXOvcDNbaWYfBPN0H01Jv94joXVXjTJE\nAikLjODaU/cAw4CewPlm1jO6j7v/wd17u3tvYCzwhruvj+oyJJhfkqo6ReLKyYuMMtbMgmWvhl2N\nSOhSOcLoDyxz9xXuvg2YBJxRR//zgcdTWI9I/fW6ANp0hdd/p1GGNHmpDIxOwKqo6dVB2y7MrBAY\nCvwzqtmBV81stpmNivciZjbKzGaZ2ax169YloWyRKDl5MOiX8MkcWDol7GpEQpUpB71PB96qtTtq\nYLCrahhwqZkdH2tBdx/v7iXuXlJcXJyOWqWp6X0BtNlf15iSJi+VgbEG6BI13Tloi+U8au2Ocvc1\nwc+1RC6t3j8FNYrsXnYuHH8VfPJvWPJS2NWIhCaVgfEe0MPMuptZHpFQeK52JzNrDZwAPBvV1tzM\nWtY8B04G5qewVpG69ToP2nbTKEOatJxUrdjdK83sMuBlIBuY4O4LzGx0MP/+oOuZwBR33xy1+L7A\n02ZWU+NEd0/Zn3aTFk3CcXKycsixnMjP4JFt2TumY8zLzcqN3Tdqmeys7FSVLumSnQvH/wqevQQW\nT4ZDvh12RSJpZ96I/loqKSnxWbPq/5WNfo/2o7yqPAUVRRhGdlYQLkHoZGdlxwyhmoCpK6RysnK2\nB1V0SNUVYLlZuZH1Rr1+zHoSeM2crBzys/NT9nllrKpKuKcf5DWHi6dD5A8akQbNzGYn+tWFlI0w\nGpJXv/sqldWVVFZXUuVV259XeuWO58G8iuqK2H1r9a+qrqLSK6morog8j7POSq+MO39r5Va2+BYq\nqyu3v270a8aqp9rTc6v1/h36c9OAm+jYomNaXi8jZOdERhnPjIZFL8Chp4ddkUhaaYTRyFR7dfwA\nq66kwit2mt4lwKJCKV5Ibdi6gYmLJmIY444ex2kHnIY1lb+2qyrhnv6Rmy1dPB2yMuVEQ5E9oxFG\nE5ZlWeRl55GXnZfS1zmrx1lc8+Y1jHtzHFNXTeXaY66lTUGblL5mRsjOgROuhqdHRUYZPYeHXZFI\n2ujPI9kjnVt2ZsIpE/hFn1/w2qrXOOu5s3hrzVthl5UeR5wDRT1g6q1QnZ5dgCKZQIEheyw7K5uL\njriIiadOpFVeK0a/Oprfvfs7yirLwi4ttbKyI6OMtQtg4S5nios0WgoM2WuHFh3K30//OyMPHcnj\nix5nxAsjWFC6IOyyUuvws6D9wfDGbRplSJOhwJCkyM/O5+r+V/PgyQ+yuWIzI18cyQPzHqCyujLs\n0lIjKxtO+BWs/RA+fCbsakTSQoEhSXXMfsfw1PCnOGn/k7h77t388KUfsurrVbtfsCE67EwoPiQY\nZVSFXY1IyikwJOla57fm9yf8ntsG3caKDSs4+/mzeXLJkzSmU7iBHccy1i2CBU+HXY1IyikwJGVO\nPeBUnhr+FEe2P5Ib3r6Bn7/+c0rLSsMuK7l6fgf26alRhjQJCgxJqQ7NOzD+5PH8qt+vmLFmBmc9\ndxZTV00Nu6zkycqKjDK+WALznwq7GpGUUmBIymVZFhf2vJBJp02iuFkxP3vtZ1w/43q2VGwJu7Tk\nOHQ47Hu4RhnS6CkwJG16tO3BxG9P5MeH/5inlj7FOc+fw9y1c8Mua+/VjDJKl8IHT4ZdjUjKKDAk\nrfKy87ii7xX8dehfqfZqfvDSD/jzv/9MRXVF2KXtnUNO2zHKqGqkpxJLk6fAkFD03bcvT57+JKcf\ncDrj3x/PyMkjWbFhRdhl7bmsLBg8BtYvhw+eCLsakZRIaWCY2VAzW2xmy8xsTIz5g81sg5nNDR7X\nJbqsNHwt8lrw24G/5Y+D/8gnmz7h3OfPZeLCiQ339NtDToMOR8C032uUIY1SygLDzLKBe4BhQE/g\nfDPrGaPrdHfvHTxurOey0gh8a/9v8dTwp+jXoR+3zLyFn776U9ZuWRt2WfVnBoPHwvoV8P7fw65G\nJOlSOcLoDyxz9xXuvg2YBJyRhmWlASouLObeb97Lr4/+NbM/n81Zz53FlJVTwi6r/g4+FfbrFYwy\nGvhxGZFaUhkYnYDoa0KsDtpqO87M3jezf5nZYfVcVhoRM2PEISN44vQn6NKiC1e+cSXjpo9j47aN\nYZeWuJpRxpcrYd6ksKsRSaqwD3rPAbq6+5HAn4F6X8XNzEaZ2Swzm7Vu3bqkFyjp1611Nx459RF+\n2uunTP5oMmc/dzbvffZe2GUl7qCh0PEomPYHjTKkUUllYKwBukRNdw7atnP3r919U/B8MpBrZu0T\nWTZqHePdvcTdS4qLi5NZv4QoNyuXS3pfwiPDHiE3K5eLXr6IO2bdwbaqbWGXtns1o4yvPoa5E8Ou\nRiRpUhkY7wE9zKy7meUB5wE73W3GzDpYcDNoM+sf1FOayLLSNBxZfCRPnP4E5xx0Dn9d8FfOf/F8\nlny5JOyydq/HydCpL0y/HSobQMiJJCBlgeHulcBlwMvAQuAf7r7AzEab2eig2znAfDObB9wFnOcR\nMZdNVa2S2QpzC7nu2Ou455v3UFpWynkvnMffFvyNas/gGxdtH2X8B+ZplCGNgzXYc95jKCkp8Vmz\nZoVdhqTQ+vL13DDjBl5b9Rr9OvTj5gE3s1+L/cIuKzZ3+Mu3YNPn8LM5kJMXdkUiuzCz2e5ekkjf\nsA96i9RLu4J23DnkTm487kYWfLGAs587mxdWvJCZX/YzgyFjYcMqeOtOqCgPuyKRvaLAkAbHzDiz\nx5k8OfxJvtH2G4ydPparpl3Fhq0bwi5tVwd+Ew4YDK/fDLf3gKdGwaLJCg9pkLRLShq0quoq/rrg\nr9zz73toV9COmwbexHEdjwu7rJ1VVcBHb8CCZ2Dh81D+FeS3goOHRW7AdOCJkFsQdpXSRNVnl5QC\nQxqFD0s/ZOz0sazYsIILDrmAK/peQUFOBv4S3h4eT8PCFxQeEjoFhjRJ5ZXl/GnOn3h04aN0b92d\nWwfdSs+iDL4EmcJDMoACQ5q0GZ/M4No3r2V9+Xou6X0JPz78x2RnZYddVt0UHhISBYY0eRu2buC3\n7/yWl1a+RO/i3vxu0O/o0rLL7hfMBAoPSSMFhgjg7kz+aDI3v3MzVV7F1f2v5sxvnElwcYGGQeEh\nKabAEIny6aZP+fVbv2bmZzMZ0mUIvzn2NxQ1Kwq7rPpTeEgKKDBEaqn2ah798FH+NOdPtMhrwY3H\n3cgJXU4Iu6w9p/CQJFFgiMSx9MuljJk+hiVfLuGcg87hqpKrKMwtDLusvaPwkL2gwBCpw7aqbdw9\n924env8wXVp24XeDfkev4l5hl5UcVRWw4g34UOEhiVFgiCTgvc/e45o3r+HzLZ/zkyN+wsW9LiY3\nKzfsspJH4SEJUGCIJGjjto3cOvNWnlv+HIcVHcYtg26he+vuYZeVfAoPiUOBIVJPr3z8Cje8fQNb\nK7dyZcmVjDh4RMM6/bY+FB4SRYEhsgfWbVnHtTOu5a01bzGg0wBuOu4migsb+W1/FR5NXsYEhpkN\nBf4EZAMb+Ud2AAAOZUlEQVR/cfdba83/HnA1YMBG4KfuPi+YtzJoqwIqE3lDCgzZW+7OpMWTuGPW\nHRTkFHDdsddx0v4nhV1Weig8mqSMCAwzywaWACcBq4ncp/t8d/8wqs9xwEJ3/9LMhgHXu/vRwbyV\nQIm7f5HoayowJFlWbFjBuOnjWFC6gOEHDmdM/zG0zGsZdlnpo/BoMjIlMI4lEgCnBNNjAdz9ljj9\n2wLz3b1TML0SBYaEqKK6ggfmPcCDHzxIh8IO3DzwZko6JPT/qnFReDRqmRIY5wBD3f2/gukLgaPd\n/bI4/X8JHBLV/yNgA5FdUg+4+/g4y40CRgF07dq178cff5z09yJN27x18xg7fSyrN67mh4f/kMt6\nX0ZedhO9P7fCo9FpcIFhZkOAe4GB7l4atHVy9zVmtg/wCvAzd59W12tqhCGpsqViC3+Y9QeeXPIk\nB7c9mFsG3UKPtj3CLitcCo9GIVMCI6FdUmZ2JPA0MMzdl8RZ1/XAJne/va7XVGBIqk1dNZXfzPgN\nG7dt5PI+l3NhzwvJsqywywqfwqPBypTAyCFy0PubwBoiB70vcPcFUX26Aq8B33f3GVHtzYEsd98Y\nPH8FuNHdX6rrNRUYkg6lZaVc//b1TF01lf4d+nPzwJvp0LxD2GVlDoVHg5IRgREUcipwJ5HTaie4\n+81mNhrA3e83s78AZwM1Bx4q3b3EzA4gMuoAyAEmuvvNu3s9BYaki7vz9LKnuXXmreRYDtcccw3f\nPuDbYZeVeRQeGS9jAiPdFBiSbqu+XsW4N8cxd91chnUbxjXHXEPr/NZhl5WZFB4ZSYEhkkaV1ZVM\nmD+B++beR7tm7fjtgN9ybMdjwy4rs+0uPPY/DvJaQHZO2JU2egoMkRAsKF3A2Olj+WjDR4w8dCSX\n97mcghz9xbxbscKjRlYu5BZCbrPgEeN5XvMY82v/bAa5zWOvIycfGut1wxKgwBAJSVllGXfOvpOJ\niyZyYOsDuWXQLRxadGjYZTUcNeGxbhFUlkFFGWzbAhVbIs+3/4z1PHjUm+0cMHmxwiaRAIrTP68Q\ncppBVmaeTafAEAnZjDUzuPata1m/dT2X9r6UHx32I7KzssMuq/Fzh8ryWoFS62fMAIoVRHGCattm\n8Kr615ZTsJsgqmt0VDvQYswr2LNjZwoMkQywYesGbnz7RqZ8PIUWuS0oLiymfbP2FBUURX42K9pp\nun2z9rQtaEtOlvbbZ7yqiqgA2hwnbGoH0ebdhFNU27YtULU18XoK28Ovlu/RW6lPYGjLFEmR1vmt\nuf2E23n1P6/y3mfvUVpWyhdlX7Bw/UJKy0rZVLFpl2UMo21BW4qaFe0UJEUFRTsCJvjZJr+NvjQY\nluxcyG69x3/VJ6S6avfBUvOw9IxeNcIQCUlZZRmlZaWUlkeCpLSsdHuofFH2BV+U72grryrfZfls\ny6ZdQbvtIbJTwESFS1FBEa3yWjXeG0LJXtEIQ6QBaJbTjM4tO9O5Zec6+7k7mys2bw+WmnD5ouwL\nSst3hMzSL5dSWl5KZXXlLuvIzcrdZXdYTajU3i1WmFuYqrcsDZwCQyTDmRkt8lrQIq8F+7fav86+\n7s7X277eKVRqgqWm7bMtnzG/dD7ry9dT7dW7rKNZTrNdj7PEON5S1KyI/Oz8VL1tyUAKDJFGxMxo\nnd+a1vmtObDNgXX2raqu4qutX+3YHVa+a8B8/PXHzPl8Dl9u/TLmOlrmtow7WtneXtCeds3akZuV\nm4q3LGmkwBBporKzsrf/Ut+diuoK1pet3/l4S61dZIvXL2ZG2Qw2VmyMuY42+W3iH28p2BEwbfPb\n6hTkDKXAEJHdys3KZd/m+7Jv831323dr1da4u8Nq2t9f9z6l5aWUVZbtsnyWZdE8tzm5WblkWzY5\nWTnbf25/WA7ZWdk7Tdc8r7Nvgv2yLZvcrNzY/Wq3BevMzsreaf3R/bItu1GcdKDAEJGkys/Op2OL\njnRs0XG3fbdUbNkpVGqCZVPFJiqrK7c/qrxqx7Tv3L6tahtl1WVUVFdQ6ZVUVe/aN3r5quoqKn3X\nEwNSbadgiQqo2sG0U1gFIVSzTLwQbZXXip/3+Xnq30PKX0FEJI7C3EK65nala6uuaX1dd985RLwq\nEjhR4VJVXbVLCO3ULwieeMFW5zqj2+Kss6bvNt+2y/orqit2Wmfr/NYKDBGRVDCz7X/xS+JS+jVR\nMxtqZovNbJmZjYkx38zsrmD++2bWJ9FlRUQkvVIWGGaWDdwDDAN6AuebWc9a3YYBPYLHKOC+eiwr\nIiJplMrxWH9gmbuvADCzScAZwIdRfc4AHvHI9UneMbM2ZrYf0C2BZUWkkdrdJYsSuaLR7rokclmk\n3a8jkTr2/r0koiA39acipzIwOgGroqZXA0cn0KdTgssmzel/fpPyij24XHETFb19R/+n8xgdYvXd\nua2mn+/SVvt5IuuJXpfHrWPX1h117Po60e0715bYe4/3iylWa7xfHrF+6cTvm2hj4uuN9zst1nuL\n3zfODNlr7VvkM+vX30r56zT4Iz5mNorI7iy6dt2zMy0OLG7OtqpdL5Eg8RkWPbHL0+hzzqPPPq9p\n3rnNdmnbeZ2207I7v07smra/TpyVxq4j/np2fs3Y59PXtXy8M/BjrSru+hNurPXvU8drxVtFzLri\nvFjsvnHE6FyfzyaReuq3jt3b7TqS8P2KRFZR1/stzEvPFx1TGRhrgC5R052DtkT65CawLADuPh4Y\nD5Gr1e5JoXeed9SeLCYi0qSk8iyp94AeZtbdzPKA84DnavV5Dvh+cLbUMcAGd/80wWVFRCSNUjbC\ncPdKM7sMeBnIBia4+wIzGx3Mvx+YDJwKLAO2AD+qa9lU1SoiIrunGyiJiDRh9bmBku7vKCIiCVFg\niIhIQhQYIiKSEAWGiIgkRIEhIiIJaVRnSZnZOuDjPVy8PfBFEstJFtVVP6qrflRX/TTGuvZ39+JE\nOjaqwNgbZjYr0VPL0kl11Y/qqh/VVT9NvS7tkhIRkYQoMEREJCEKjB3Gh11AHKqrflRX/aiu+mnS\ndekYhoiIJEQjDBERSUiTDAwzu9LM3MzaR7WNNbNlZrbYzE6Js1w7M3vFzJYGP9smqZ6bzOx9M5tr\nZlPMrGPQ/r2greZRbWa9Yyx/vZmtiep3aorr6mZmZVGvd3+c5dP9eZ1kZrPN7IPg54lxlk/r5xXM\nC3P7+oOZLQpqe9rM2gTtYW9f8eoKe/uKV1fY21fMuoJ56dm+3L1JPYjcmOllIt/XaB+09QTmAflA\nd2A5kB1j2d8DY4LnY4DbklRTq6jnPwfuj9HnCGB5nOWvB36Zgs8qZl1E7rk+P4Hl0/p5AUcBHYPn\nhwNrMuTzCnv7OhnICZ7fFmu9IW1fMevKgO0rXl1hb1/x6krb9tUURxh/BH7FzrcePgOY5O5b3f0j\nIvfn6B9j2TOAvwXP/wZ8JxkFufvXUZPNiX1b5POBScl4vUQlWFdd0vp5ufu/3f2ToH0B0MzM8pPx\nmntTF+FvX1PcvTKYfIfIHSxrC2P7SqSuuqT188qA7Sve55W27atJBYaZnUHkr4J5tWZ1AlZFTa8O\n2mrb1yN3BAT4DNg3ibXdbGargO8B18XoMgJ4vI5V/CwYqk5I1tB8N3V1D4bbb5jZoDiLh/l5nQ3M\ncfetcVaRzs8r9O0ryo+Bf8VoD2X7qqOuULevOuqqEcr2FaeutG1fjS4wzOxVM5sf43EGMI7Yv1zq\nzSPjuoT/4t5NXbj7Ne7eBXgMuKzWskcDW9x9fpzV3wccAPQGPgX+J8V1fQp0dffewH8DE82sVV2v\nk+bP6zAiQ/aL46w+3Z9XvSX78wr6XANUBrVFLxva9hWnrtC3rzh11bSHtn3VVVd91Pfzil6wSTyI\n7KNdC6wMHpXAf4AOwFhgbFTfl4FjY6xjMbBf8Hw/YHEK6uxKrf23RHajjUtw+W61l09VXVHzpgIl\nmfB5ERmmLwEGZMrnlQnbF/BD4G2gMMa80LavuuoKc/uKV1fY21esutK5fTW6EUY87v6Bu+/j7t3c\nvRuRYVsfd/8MeA44z8zyzaw70AOYGWM1zwE/CJ7/AHg2GbWZWY+oyTOARVHzsoBzqWP/spntFzV5\nJhDvL8Wk1GVmxWaWHTw/gMjntSLGKtL6eQVnjbxI5MDeW3Usn9bPi/C3r6FEjtsNd/ctteaFuX3F\nrCsDtq94dYW9fcX7d0zf9pWs5GtoDyKjjPZR09cQObtgMTAsqv0vBH/dAEXA/wFLgVeBdkmq5Z9E\nNqr3geeBTlHzBgPvxFgmuq7/BT4Iln+O4K+IVNVFZP/tAmAuMAc4PRM+L+DXwOagrprHPmF/Xhmw\nfS0jso+75jO5P2pemNtXzLoyYPuKV1fY21dd/45p2b70TW8REUlIk9klJSIie0eBISIiCVFgiIhI\nQhQYIiKSEAWGiIgkRIEhIiIJUWCI7AUz25RgvyeDL6FhZo+Y2Ttm1jpqfp6ZTTOznFTVKrK3FBgi\nKRZceyjb3VcAuPv3iVze4Uc1fdx9G5EvVY0IpUiRBCgwRJLAzAab2dRgJLHIzB4zMwtmf49dL8Pw\nr6A92jMx2kQyhgJDJHmOAn5B5IY2BwADgvYBwOxafc8DjjSzg6La5gP9Ul2kyJ5SYIgkz0x3X+3u\n1USu9dMtaN8PWFfTycyKgSHAb4kaUbh7FbDNzFqmrWKRelBgiCRP9M10qoCaA9hlQEHUvNHAQ8DD\nRO50Fy0fKE9RfSJ7RYEhknoLgW9A5GwoIvc0uM/dVwFrghsYYWZFwBfuXhFWoSJ1UWCIpN6LRC4j\nDpF7T7zi7qXB9P+yY7fUkKCvSEbS5c1FUszMmgGvE7lLW1Ud/Z4icnOeJWkrTqQeNMIQSTF3LwN+\nA3SK1yfYVfWMwkIymUYYIiKSEI0wREQkIQoMERFJiAJDREQSosAQEZGEKDBERCQh/w+KgClolZMb\ndgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14781798208>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "N = 11\n",
    "plot_errors(N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error on the train dataset is very low and pretty constant, which makes sense because we created a polynom that fits best to the train data.\n",
    "\n",
    "The error on the test and dev data decreases as λ is larger. For λ=-20 the overfitting has been supressed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1282,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZgAAAEKCAYAAAAvlUMdAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3X90VeW95/H3Nz9ICCCB8EMgKFgjBlEjRKRL2yu6VKAW\ndHUq/rpWO2uQZbnXuXPVYm1tO73O2HHq7Ti1spx76a1Vih2nVuqPEW3FOrVUA0UFEzBYlIDWFCT8\nDJDkO3/snXA4nJychDw5Sfi81tor5zz7eZ7z3Sd7ne959n7O3ubuiIiIdLecbAcgIiL9kxKMiIgE\noQQjIiJBKMGIiEgQSjAiIhKEEoyIiAShBCMiIkEowYiISBBKMCIiEkRetgPIphEjRviECROyHYaI\nSJ+yZs2av7r7yI7qndAJZsKECVRVVWU7DBGRPsXMPsikng6RiYhIEEowIiIShBKMiIgEcUKfgxER\n6YrDhw9TV1dHY2NjtkMJqrCwkNLSUvLz87vUXglGRKST6urqGDJkCBMmTMDMsh1OEO7Ojh07qKur\nY+LEiV3qQ4fIREQ6qbGxkZKSkn6bXADMjJKSkuMapSnBiIh0QX9OLq2Odxt1iKwr9n4CG54+8vyo\n204n3YK6z6xr90k3vV4v1ls/KCwHLBdyWv/mRn8tJ36cc6Ss9Xmq+m3rkuu3/rUUZa39dKEvkZgS\nTFc0bIUX7sp2FH1Eb//A6SNJsM+wdpJPTlJy7O7EmQcnnw2T50HJZ7L9JgS3a9culi1bxm233dap\ndnPmzGHZsmUUFxcHiuxo5n3lW2YAlZWV3qVf8jcfhoN72l9/zLc469/r9K01jJYW8GZoaU742xIt\nR5V1tC4uP6Z+unUtCX0m129OsS75dVtS1M9kG7xrsTQfhE+3RO/bqLNg8lwonwujyoPsn9XV1ZSX\nl3d7v5nasmULV155JevXrz+qvKmpiby87h03pNpWM1vj7pUdtdUIpity86FoeLajkP4uJwfIifY3\n6VhDHVT/Gt5dAavuh1X/FUpOjxJN+Rdh7Hn95svQ4sWL2bx5MxUVFeTn51NYWMiwYcOoqalh06ZN\nXHXVVWzdupXGxkZuv/12FixYABy5PNbevXuZPXs2F110Ea+//jrjxo3jmWeeYeDAgd0ap0YwuhaZ\nSP+z5y9Q8yxUr4A/vxaNdIaecmRkU3p+nMC7JvFb/Xd/vYF3t+/ursgBmDz2JL79xbPaXZ84glm1\nahVf+MIXWL9+fdt04p07dzJ8+HAOHDjA+eefz6uvvkpJSclRCeb000+nqqqKiooKrrnmGubOncuN\nN96YdltbZTqCCTqLzMxmmdlGM6s1s8Up1puZPRSvf9vMpnbU1sy+F9ddZ2YrzWxsXH6Zma0xs3fi\nv5eE3DYR6cWGjIbz/z3c9AzcWQvzHo4Ol73xKCy9HP55Mjx3B/z5d9DclO1oj9v06dOP+q3KQw89\nxLnnnsuMGTPYunUr77333jFtJk6cSEVFBQDTpk1jy5Yt3R5XsENkZpYLPAxcBtQBb5rZCnd/N6Ha\nbKAsXi4AHgEu6KDtA+7+rfg1/h64F1gI/BX4ortvN7MpwIvAuFDbJyJ9RNFwOO/GaGlsgE0rofoZ\n+NPj8Ob/gqISmDQnmiAw8W8gb0Cnuk830ugpgwYNanu8atUqXn75Zf7whz9QVFTExRdfnPK3LAUF\nBW2Pc3NzOXDgQLfHFfIczHSg1t3fBzCz5cA8IDHBzAMe8+g43WozKzazMcCE9tq6e+JYdBDxNCB3\n/1NC+QZgoJkVuPvBIFsnIn1P4VA458vRcmgf1L4cnbPZ8Cv408+gYChMmhUdRjv9Usjv3nMS3WXI\nkCHs2ZN6olFDQwPDhg2jqKiImpoaVq9e3cPRHREywYwDtiY8ryMapXRUZ1xHbc3sPuAmoAGYmeK1\nvwSsVXIRkXYNGBSNWibPg8ON8P6q6JxNzXPw9pOQPwjKLovO25RdDgVDsh1xm5KSEi688EKmTJnC\nwIEDGT16dNu6WbNmsWTJEsrLy5k0aRIzZszIWpx9chaZu98D3GNmdwOLgG+3rjOzs4DvA5enamtm\nC4AFAKecckr4YEWk98svjEYuk2ZFP0PY8lo0sql5Ft79FeQWRCOa8rlRnV5g2bJlKcsLCgp44YUX\nUq5rPc8yYsSIo6Y433HHHd0eH4RNMNuA8QnPS+OyTOrkZ9AW4AngeeIEY2alwNPATe6+OVVQ7v4o\n8ChEs8gy3BYROVHk5sNnLomWL/wAPlwdjWyqfw0bn49+1Dn7adg3Mjrkpmnk7QqZYN4EysxsIlFy\nuBa4PqnOCmBRfI7lAqDB3T8ys/r22ppZmbu3TomYB9TE5cXAc8Bid/99wO0SkRNFTi5MuDBaZt0P\n29ZGI5qWw9EVPRq2woDBUFgMA4dCbucmCPR3wRKMuzeZ2SKi2Vy5wFJ332BmC+P1S4hGH3OAWmA/\ncEu6tnHX95vZJKAF+IBoBhlEh8pOB+41s3vjssvd/ZNQ2ygiJxAzKJ0WLdXVMOJUaNwVzUzbXRct\n+YOiRFNYDHkFHffZz+mHlvqhpYh00jE/PjzcGCWbA7ugKZ7umz8wSjSFxdE5nj5Kl4oREcmm/ELI\nPxmGnAxNB48kmz0fRUteYXwYrTh63E8uWdMRJRgRke6UVwCDR0dL06HoEFrjLtj7cbTkFhw5jJZf\n1K+TjRKMiEgoeQNg8MhoaT6ckGzqo/tK5Q6IZqIVFke/y+lisvnOd77D4MGDg0037iolGBGRnpCb\nD4NGREtzExxsiA6j7fsr7KuPpj+3HkYbMLhfjGx0y2QRkZ6WmxddA63kM9GN0opPjZLKgZ2woxY+\nfgd2fRCNeLwlZRf33XcfZ5xxBhdddBEbN24EYPPmzcyaNYtp06bxuc99jpqaGhoaGjj11FNpaYn6\n2bdvH+PHj+fw4cPBN1MjGBGR4/HC4ighdAuPbqBW8hm4YCHs3xndvbPwpGh0UzAEcnJZs2YNy5cv\nZ926dTQ1NTF16lSmTZvGggULWLJkCWVlZfzxj3/ktttu47e//S0VFRW8+uqrzJw5k2effZYrrriC\n/PzwPxBVghER6TUsOlRWcBKcPCW6c27jLjjQAAc+jW4PXXASr/3mRa6+ah5FRUUAzJ07l8bGRl5/\n/XW+/OUvt/V28GB0Ocb58+fz5JNPMnPmTJYvX97pWy13lRKMiMjxmH1/uL4Lh0bLUIdDe6NzNo27\nokNpe3fDjs3ROZuWFlpaWiguLmbdunXHdDN37ly+8Y1vsHPnTtasWcMll/TM7bJ0DkZEpLcziw6P\nFY+H0VP4/BVX8auVr3Fgzy721FXz62d+SZHvZ+Kp4/nfT/4cAHfnrbfeAmDw4MGcf/753H777Vx5\n5ZXk5ub2SNhKMCIifYkZU2dcxPzrb+Tcy69j9i13cX7lNGhp5okf3su/LvmfnDvlTM4qP5Nnnv5l\nW7P58+fz+OOPM3/+/J4LVZeK0aViRKRzUl0+JevcoanxyGG0pvgulvlF0WG0Ll4fTZeKERE50ZlF\n1z/LHwgnjTlyfbTGXbB7e7TkDTySbHrg+mhKMCIi/dEx10drOPr6aIXDYPiEoCEowYiI9Hd5BTB4\nVLQ0x9dHywn/8a8EIyJyIskdAING9shLaRaZiIgEoQQjIiJBKMGIiPQxu3bt4sc//nGX2v7whz9k\n//793RxRakowIiJ9TF9JMDrJLyLSxyxevJjNmzdTUVHBZZddxqhRo/jFL37BwYMHufrqq/nud7/L\nvn37uOaaa6irq6O5uZlvfetb/OUvf2H79u3MnDmTESNG8MorrwSNUwlGROQ4fP+N71Ozs6Zb+zxz\n+Jl8ffrX211///33s379etatW8fKlSt56qmneOONN3B35s6dy+9+9zvq6+sZO3Yszz33HAANDQ0M\nHTqUBx98kFdeeYURI0Z0a8yp6BCZiEgftnLlSlauXMl5553H1KlTqamp4b333uPss8/mpZde4utf\n/zqvvfYaQ4cO7fHYNIIRETkO6UYaPcHdufvuu7n11luPWbd27Vqef/55vvnNb3LppZdy77339mhs\nGsGIiPQxQ4YMYc+ePQBcccUVLF26lL179wKwbds2PvnkE7Zv305RURE33ngjd955J2vXrj2mbWga\nwYiI9DElJSVceOGFTJkyhdmzZ3P99dfz2c9+Foju/fL4449TW1vLnXfeSU5ODvn5+TzyyCMALFiw\ngFmzZjF27NjgJ/mDXq7fzGYB/wPIBf7F3e9PWm/x+jnAfuBmd1+brq2ZfQ+YB7QAn8RttptZCfAU\ncD7wb+6+qKP4dLl+EemKXnm5/kCO53L9wQ6RmVku8DAwG5gMXGdmk5OqzQbK4mUB8EgGbR9w93Pc\nvQJ4Fmg9qNgIfAu4I9Q2iYhI5kKeg5kO1Lr7++5+CFhONPJINA94zCOrgWIzG5OurbvvTmg/CPC4\nfJ+7/z+iRCMiIlkW8hzMOGBrwvM64IIM6ozrqK2Z3QfcBDQAM7svZBGRzLg70VH+/ut4T6H0yVlk\n7n6Pu48HngA6PNeSyMwWmFmVmVXV19eHCVBE+rXCwkJ27Nhx3B/AvZm7s2PHDgoLu37ny5AjmG3A\n+ITnpXFZJnXyM2gLUYJ5Hvh2pkG5+6PAoxCd5M+0nYhIq9LSUurq6ujvX1ILCwspLS3tcvuQCeZN\noMzMJhIlh2uB65PqrAAWmdlyokNgDe7+kZnVt9fWzMrc/b24/Tyge6/RICLSgfz8fCZOnJjtMHq9\nYAnG3ZvMbBHwItFU46XuvsHMFsbrlxCNPuYAtUTTlG9J1zbu+n4zm0Q0TfkDYGHra5rZFuAkYICZ\nXQVc7u7vhtpGERFpX9DfwfR2+h2MiEjnZf13MCIicmJTghERkSCUYEREJAglGBERCUIJRkREglCC\nERGRIJRgREQkCCUYEREJQglGRESCUIIREZEglGBERCQIJRgREQlCCUZERIJQghERkSCUYEREJAgl\nGBERCUIJRkREglCCERGRIJRgREQkCCUYEREJQglGRESCUIIREZEglGBERCQIJRgREQlCCUZERIII\nmmDMbJaZbTSzWjNbnGK9mdlD8fq3zWxqR23N7Htx3XVmttLMxiasuzuuv9HMrgi5bSIikl6wBGNm\nucDDwGxgMnCdmU1OqjYbKIuXBcAjGbR9wN3PcfcK4Fng3rjNZOBa4CxgFvDjuB8REcmCkCOY6UCt\nu7/v7oeA5cC8pDrzgMc8shooNrMx6dq6++6E9oMAT+hrubsfdPc/A7VxPyIikgUhE8w4YGvC87q4\nLJM6adua2X1mthW4gXgEk+HrYWYLzKzKzKrq6+s7tUEiIpK5PnmS393vcffxwBPAok62fdTdK929\ncuTIkWECFBGRoAlmGzA+4XlpXJZJnUzaQpRgvtSJ1xMRkR4SMsG8CZSZ2UQzG0B0An5FUp0VwE3x\nbLIZQIO7f5SurZmVJbSfB9Qk9HWtmRWY2USiiQNvhNo4ERFJLy9Ux+7eZGaLgBeBXGCpu28ws4Xx\n+iXA88AcohPy+4Fb0rWNu77fzCYBLcAHQGt/G8zsF8C7QBPwNXdvDrV9IiKSnrl7x7X6qcrKSq+q\nqsp2GCIifYqZrXH3yo7q9cmT/CIi0vspwYiISBBKMCIiEoQSjIiIBKEEIyIiQSjBiIhIEEowIiIS\nhBKMiIgE0WGCMbNcM/uHnghGRET6jw4TTHy5let6IBYREelHMr0W2e/N7EfAk8C+1kJ3XxskKhER\n6fMyTTAV8d//nFDmwCXdG46IiPQXGSUYd58ZOhAREelfMppFZmZDzezB1lsNm9kPzGxo6OBERKTv\nynSa8lJgD3BNvOwGfhIqKBER6fsyPQfzGXf/UsLz75rZuhABiYhI/5DpCOaAmV3U+sTMLgQOhAlJ\nRET6g0xHMAuBxxLOu3wKfCVMSCIi0h90mGDMLAeY5O7nmtlJAO6+O3hkIiLSp2XyS/4W4K748W4l\nFxERyUSm52BeNrM7zGy8mQ1vXYJGJiIifVqm52Dmx3+/llDmwGndG46IiPQXmZ6DudHdf98D8YiI\nSD+R6TmYH/VALCIi0o9keg7mN2b2JTOzoNGIiEi/kWmCuRX4BXDQzHab2R4z63A2mZnNMrONZlZr\nZotTrDczeyhe/7aZTe2orZk9YGY1cf2nzaw4Lh9gZj8xs3fM7C0zuzjDbRMRkQAyTTBDgZuBf3L3\nk4CzgMvSNTCzXOBhYDYwGbjOzCYnVZsNlMXLAuCRDNq+BExx93OATcDdcfl/AHD3s+PYfhCfPxIR\nkSzI9AP4YWAGR+5suYeOz8tMB2rd/X13PwQsB+Yl1ZkHPOaR1UCxmY1J19bdV7p7U9x+NVAaP54M\n/Dau8wmwC6jMcPtERKSbZZpgLnD3rwGNAO7+KTCggzbjgK0Jz+viskzqZNIW4KvAC/Hjt4C5ZpZn\nZhOBacD45AZmtqD1tgP19fUdbIKIiHRVpgnmcHzYygHMbCTQEiyqDJjZPUAT8ERctJQoEVUBPwRe\nB5qT27n7o+5e6e6VI0eO7KlwRUROOJn+0PIh4GlglJndB/w74JsdtNnG0SOI0rgskzr56dqa2c3A\nlcCl7u4A8WGzf0io8zrRORoREcmCTG+Z/ISZrQEuBQy4yt2rO2j2JlAWH67aBlwLXJ9UZwWwyMyW\nAxcADe7+kZnVt9fWzGYRXRvtb9x9f2tHZlYEmLvvM7PLgCZ3fzeT7eusj/d9zLLqZUcXWvJTS/88\nxYzv5DrHrLfO9dlRDMcbcybb1NvF30+6py+6ry+I3l8zwzByLIeceM5K6+PW9TnkRH/jshzLOapd\nYp3Wx0e1T+6bHLAjj5NjaLfvxDoJfR/1Ohzbvt2+O2gnvV+mIxjcvQao6UT9JjNbBLwI5AJL3X2D\nmS2M1y8BngfmALXAfuCWdG3jrn8EFAAvxTvZandfCIwCXjSzFqKk9LeZxtpZOxp38POanx/Z1qQP\nluQPrWPWp/og8uSnHfTRwWuI9HfJSXbM4DGUDy/nzOFnUl5STvnwcoYVDst2mCc0685vcH1NZWWl\nV1VVZTuMHtFh0utsUuxCQuzOb50djfay1Vd3doVDCy24Oy3egnPkb2tZ8vO2x8Tr3I+0S3icXPeY\n9u3USewbOFIvse+kuqn6ToyprX5CbEf1nRgTqbepuaWZD/d8SPWOarbv2972Fp486GTKh0fJprwk\nSj6ji0ZrBHSczGyNu3c4SzfjEYz0bR0d3urWD0aRLGo42ED1zmpqdtTw7s53qdlZw6qtq9q+9Awv\nHB6NcoaXc2bJmUwePpnSIaVthwml+2gEc4KMYEROZPsP72fjpxup3lEdJZ+dNdR+WktT/JO6QfmD\n2pJO6+G1iUMnkpej7+CpaAQjIhIryi/ivFHncd6o89rKDjUfonZXbVvSqd5ZzVObnqKxuRGAgtwC\nzhh2xlHndMqGlVGQW5CtzehzNILRCEZEYs0tzWzZvSVKOK2jnR017Dm8B4Bcy+W04tOOOa8zKH9Q\nliPvWZmOYJRglGBEJA13p25vHTU7a46MdnZUs6NxR1udU0869YSawaYEkwElGBHpqvr99UePdHbW\nsG3vkd+SnzzoZM4cHk0iaE08/WUGm87BiIgENLJoJCOLRvL50s+3lTUcbGgb6bTOYHt166ttM9iG\nFQxrO6zWOtIZP2R8v53BphGMRjAiEtD+w/vZ9Okm3t0RJZzqndXU7qqlqSX1DLYzh5/JaUNP69Uz\n2DSCERHpBYryi6gYVUHFqIq2stYZbDU7a9oST/IMtrLisraEM7lkcp+cwaYRjEYwItILJM9gaz3U\nlm4G26Rhkxg8YHCPx6qT/BlQghGR3szd2bZ321GTCVLNYGs7xBZfnWB44fCgcSnBZEAJRkT6osQZ\nbK3ndRJnsI0uGk15SXmwGWw6ByMi0k+lm8HWel6nemd12hlsU0qmUDqktL2X6BYawWgEIyL9VOsM\ntsTRznu73qOppYnLTr2MBy9+sEv9agQjInKCSzWD7XDzYWp31fbIb2+UYERETiD5ufmUl5T3yGv1\nz5+PiohI1inBiIhIEEowIiIShBKMiIgEoQQjIiJBKMGIiEgQSjAiIhKEEoyIiAShBCMiIkEETTBm\nNsvMNppZrZktTrHezOyheP3bZja1o7Zm9oCZ1cT1nzaz4rg838x+ambvmFm1md0dcttERCS9YAnG\nzHKBh4HZwGTgOjObnFRtNlAWLwuARzJo+xIwxd3PATYBrYnky0CBu58NTANuNbMJQTZOREQ6FHIE\nMx2odff33f0QsByYl1RnHvCYR1YDxWY2Jl1bd1/p7k1x+9VA6/WmHRhkZnnAQOAQsDvg9omISBoh\nE8w4YGvC87q4LJM6mbQF+CrwQvz4KWAf8BHwIfDf3X1ncgMzW2BmVWZWVV9fn/nWiIhIp/TZk/xm\ndg/QBDwRF00HmoGxwETgH83stOR27v6ou1e6e+XIkSN7LF4RkRNNyMv1bwPGJzwvjcsyqZOfrq2Z\n3QxcCVzqR+6Ydj3wf939MPCJmf0eqATeP+4tERGRTgs5gnkTKDOziWY2ALgWWJFUZwVwUzybbAbQ\n4O4fpWtrZrOAu4C57r4/oa8PgUviOoOAGUBNuM0TEZF0go1g3L3JzBYBLwK5wFJ332BmC+P1S4Dn\ngTlALbAfuCVd27jrHwEFwEtmBrDa3RcSzTr7iZltAAz4ibu/HWr7REQkPTtyhOnEU1lZ6VVVVdkO\nQ0SkTzGzNe5e2VG9PnuSX0REejclGBERCUIJRkREglCCERGRIJRgREQkCCUYEREJQglGRESCUIIR\nEZEglGBERCQIJRgREQlCCUZERIJQghERkSCUYEREJAglGBERCUIJRkREglCCERGRIJRgREQkCCUY\nEREJQglGRESCUIIREZEglGBERCQIJRgREQlCCUZERIJQghERkSCUYEREJIigCcbMZpnZRjOrNbPF\nKdabmT0Ur3/bzKZ21NbMHjCzmrj+02ZWHJffYGbrEpYWM6sIuX0iItK+YAnGzHKBh4HZwGTgOjOb\nnFRtNlAWLwuARzJo+xIwxd3PATYBdwO4+xPuXuHuFcDfAn9293Whtk9ERNILOYKZDtS6+/vufghY\nDsxLqjMPeMwjq4FiMxuTrq27r3T3prj9aqA0xWtfF7cREZEsCZlgxgFbE57XxWWZ1MmkLcBXgRdS\nlM8Hfp4qKDNbYGZVZlZVX1+fdgNERKTr+uxJfjO7B2gCnkgqvwDY7+7rU7Vz90fdvdLdK0eOHNkD\nkYqInJjyAva9DRif8Lw0LsukTn66tmZ2M3AlcKm7e1Kf19LO6EVERHpOyBHMm0CZmU00swFEH/wr\nkuqsAG6KZ5PNABrc/aN0bc1sFnAXMNfd9yd2ZmY5wDXo/IuISNYFG8G4e5OZLQJeBHKBpe6+wcwW\nxuuXAM8Dc4BaYD9wS7q2cdc/AgqAl8wMYLW7L4zXfR7Y6u7vh9ouERHJjB17hOnEUVlZ6VVVVdkO\nQ0SkTzGzNe5e2VG9PnuSX0REejclGBERCUIJRkREglCCERGRIJRgREQkCCUYEREJQglGRESCUIIR\nEZEglGBERCQIJRgREQlCCUZERIJQghERkSBC3g9GRESSuDuHmls43Owcamo5sjQ3c/Co50c/PtjU\nwuHEsoQ6B5PrJz5vTt3nJeWj+C9Xnx10W5VgRKTfcvfog/yYD9/0H+bHlHXwYX64+UgSSNtn/Li7\n5BgMyMthQG4OA/JyKcjLIT/XorK28hxOGpjPgNwcChLKzx43tNviaI8STBfUfLybv1v2p2yHIf1Y\nb72JRm+9vYcDTa0jgqQE0V3MaPvALkj48E79YW5HlUWPc9seH9O+nb6S6xQk1cnL7d1nOZRguqAw\nL5ey0YOzHYb0c4ZlO4TUemlY+Tl21Id5fp5RcMyHeG7aD+x0H/B5OUZ8k0PJkBJMF0wYMYgf3zAt\n22GIiPRqvXt8JSIifZYSjIiIBKEEIyIiQSjBiIhIEEowIiIShBKMiIgEoQQjIiJBKMGIiEgQ1lsv\n/dATzKwe+OA4uhgB/LWbwulOiqtzFFfnKK7O6Y9xneruIzuqdEInmONlZlXuXpntOJIprs5RXJ2j\nuDrnRI5Lh8hERCQIJRgREQlCCeb4PJrtANqhuDpHcXWO4uqcEzYunYMREZEgNIIREZEglGAyYGb/\naGZuZiMSyu42s1oz22hmV7TTbriZvWRm78V/h3VTPN8zs7fNbJ2ZrTSzsXH5DXFZ69JiZhUp2n/H\nzLYl1JsTOK4JZnYg4fWWtNO+p9+vy8xsjZm9E/+9pJ32Pfp+xeuytn/FfT9gZjVxfE+bWXFcnu19\nrL24sr2PtRdXtvexlHHF68LvY+6uJc0CjAdeJPq9zIi4bDLwFlAATAQ2A7kp2v43YHH8eDHw/W6K\n6aSEx38PLElR52xgczvtvwPcEeC9ShkXMAFYn0H7Hn2/gPOAsfHjKcC2XvJ+ZXX/ivu7HMiLH38/\nVd9Z2sdSxtUL9rH24sr2PtZeXD2yj2kE07F/Bu7i6NukzwOWu/tBd/8zUAtMT9F2HvDT+PFPgau6\nIyB3353wdBCpb+F+HbC8O14vUxnGlU6Pvl/u/id33x6XbwAGmllBd7zm8cRFlvevOLaV7t4UP10N\nlKaolo19LJO40gm1j6WMqxfsY+29Xz2yjynBpGFm84i+cbyVtGocsDXheV1clmy0u38UP/4YGN2N\nsd1nZluBG4B7U1SZD/w8TRd/Fw+bl3bzoZX24poYD/1fNbPPtdM8m+/Xl4C17n6wnS568v3K+v6V\n5KvACynKs7KPpYkrq/tYmrhaZWUfayeuHtnHTvgEY2Yvm9n6FMs84Buk/jDqNI/GmBl/o+8gLtz9\nHncfDzwBLEpqewGw393Xt9P9I8BpQAXwEfCDwHF9BJzi7hXAfwKWmdlJ6V6nh9+vs4gOH9zaTvc9\n/X51Wmffr0xii+vcAzTF8SW2zdo+1k5cWd/H2omrtTxr+1i6uDqjK/tYtx7v608L0fHlT4At8dIE\nfAicDNwN3J1Q90Xgsyn62AiMiR+PATYGiPMUko49Ex3W+0aG7Scktw8VV8K6VUBlb3i/iA4ZbAIu\n7C3vV2/69zm9AAAC2ElEQVTZv4CbgT8ARSnWZW0fSxdXNvex9uLK9j6WKq6e2sdO+BFMe9z9HXcf\n5e4T3H0C0RByqrt/DKwArjWzAjObCJQBb6ToZgXwlfjxV4BnuiM2MytLeDoPqElYlwNcQ5pj42Y2\nJuHp1UB730K7JS4zG2lmufHj04jer/dTdNGj71c8o+Y5opOYv0/TvkffL7K8f8WxzSI69zjX3fcn\nrcvmPpYyrl6wj7UXV7b3sfb+jz2zj3VXluzvC9EoZkTC83uIZl5sBGYnlP8L8TcnoAT4DfAe8DIw\nvJti+T9EO+DbwK+BcQnrLgZWp2iTGNfPgHfi9iuIv6GEiovo2PMGYB2wFvhib3i/gG8C++K4WpdR\n2X6/sr1/xX3XEh2jb31fliSsy+Y+ljKuXrCPtRdXtvexdP/H4PuYfskvIiJB6BCZiIgEoQQjIiJB\nKMGIiEgQSjAiIhKEEoyIiAShBCMiIkEowYj0IDPbm2G9p+IfDGJmj5nZajMbmrB+gJn9zszyQsUq\ncryUYER6mfi6Vbnu/j6Au99EdKmPW1rruPshoh/Azc9KkCIZUIIRyQIzu9jMVsUjlRoze8LMLF59\nA8dekuOFuDzRr1KUifQaSjAi2XMe8B+Jbv50GnBhXH4hsCap7rXAOWZ2RkLZeuD80EGKdJUSjEj2\nvOHude7eQnSdqAlx+RigvrWSmY0EZgL/RMKIxd2bgUNmNqTHIhbpBCUYkexJvPFUM9B6wv4AUJiw\nbiHwr8C/Ed1FMlEB0BgoPpHjogQj0vtUA6dDNFuM6H4ej7j7VmBbfLMvzKwE+Ku7H85WoCLpKMGI\n9D7PEV0SH6L7rrzk7jvi5z/jyGGymXFdkV5Jl+sX6WXMbCDwCtEdEJvT1Psl0Y2sNvVYcCKdoBGM\nSC/j7geAbwPj2qsTHzr7lZKL9GYawYiISBAawYiISBBKMCIiEoQSjIiIBKEEIyIiQSjBiIhIEP8f\nbTfxpZOOGPsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x147811e10f0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "N = 101\n",
    "plot_errors(N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this graph we can observe a similar trend. Although the error and the decrease of error in the dev and test datasets is much smaller."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we will write a function that chooses λ that gives the smallest error on the dev dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1283,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def optimizePLS2(xt, tt, xv, tv, M):\n",
    "    \"\"\"\n",
    "    Selects the best value λ given a dataset for training (xt, tt) and a development test (xv, tv)\n",
    "    \"\"\"\n",
    "    lambdas = [1/math.exp(i) for i in range(40, 19, -5)]\n",
    "    ln_lambdas = list(range(-40, -19, 5))\n",
    "\n",
    "    polynoms = []\n",
    "\n",
    "    for lam in lambdas:\n",
    "        w = optimizePLS(xt, tt, M, lam)\n",
    "        p = np.poly1d(w[::-1])\n",
    "        polynoms.append(p)\n",
    "        \n",
    "    lam_index = np.argmin(norm_errors(xv, tv, polynoms))\n",
    "    return ln_lambdas[lam_index]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1101,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-20"
      ]
     },
     "execution_count": 1101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "N = 11\n",
    "(xtrain, ttrain), (xdev, tdev), (xtest, ttest) = generateDataset3(N, sin2pi, 0.03)\n",
    "M = 9 \n",
    "optimizePLS2(xtrain, ttrain, xdev, tdev, M)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Indeed as shown in the graph, the best λ is -20. We will use this λ to find w that best fits our dataset and see how it looks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1104,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x146c5b46e10>]"
      ]
     },
     "execution_count": 1104,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYYAAAD8CAYAAABzTgP2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xd4FFXbx/HvnUavIYRQQ4fQIVTpHSxY6KIoKlIFC8pj\nF0WxK4giKIIFEAVp0kFAUEpAeg0tJLRAICQkpJ73j12fN/AECMlmJ5u9P9eVK7szZ7K/Ea/cOTPn\nzBFjDEoppdS/PKwOoJRSKmfRwqCUUuo6WhiUUkpdRwuDUkqp62hhUEopdR0tDEoppa6jhUEppdR1\ntDAopZS6jhYGpZRS1/GyOkBmlChRwgQGBlodQymlXMr27dsvGGP8btfOJQtDYGAgISEhVsdQSimX\nIiInM9JOLyUppZS6jhYGpZRS19HCoJRS6jpaGJRSSl1HC4NSSqnraGFQSil1HYcUBhGZLiLnRWTv\nTfaLiEwUkVAR2S0iDdPs6yoih+z7xjoij1JKqcxzVI9hBtD1Fvu7AVXtX4OBrwBExBOYbN8fBPQT\nkSAHZVI3Ywz88xMcWQUpyVanUUrlMA6Z4GaM2SAigbdo0gP43tgWmN4sIkVFJAAIBEKNMccARGSO\nve1+R+RSN/HXJFj1mu11AT+o/RDU7Q2lG4KItdmUUpZz1j2GMsCpNO/D7dtutv1/iMhgEQkRkZDI\nyMhsC5rrHVwKq16HoPuh7yyo0AJCvoNp7eGLYFj/AUQdtzqlUspCLnPz2Rgz1RgTbIwJ9vO77aM+\nVHrO7oF5T0Lp+mypN54NHk24dPc38MJhuG8SFAqAP96FifXhm06wdRpcvWh1aqWUkznrWUkRQLk0\n78vat3nfZLtytJhzMKsv5C3C/Oof89x3u/67q1zxfNQtU5u6Fe+iUcOr1I5aRd79v8LSF2D5WKjS\nCer2gurdwTufhSehlHIGZxWGRcAI+z2EpkC0MeaMiEQCVUWkIraC0Bfo76RM7iMpHub0g/gotrWf\nxZjF52hb3Y/BrSqxOyKaPeHR7I64zO97ztgPqEGlEu/RJfAC3VI3UCN8BT6Hl4FPIQi6z3Y/IrAV\neHhaelpKqezhkMIgIrOBtkAJEQkH3sDWG8AYMwVYCnQHQoE44HH7vmQRGQGsADyB6caYfY7IpOyM\ngYXDIWI7JztOZeCyBGoGFGJy/4YUyONFiyol/tv00tVE9kREsyciml2nLrMgIoWvorvgQSeaeRzg\nUe8ttNm9gHw7fyIxvz8edXvhVa8PlKqjN62VykXENlDItQQHBxt97HYGrZsA694j+q5X6Li1IT6e\nHvw2rAUlC+fN0OHnY66xNyKa3eG2r0OnzlEvfgv3e26ircdOfCSFs3kqcqbCfeRr2JdKVWrg4+Uy\nt66Ucisist0YE3zbdloYcrG98+DXQSTW7kv3k/04F5PA/KEtqOpfKNM/0hjD2SvX2B0ezZHjJylw\ndAn1L62gAYcA2JJak22FO3KpQjeqB5anTtkiVC1ZEC9PLRZKWU0Lg7sL3w4zupMa0IBHkv7DtlNx\nzBzUhOaVfR3+UcYYzpw4SMy22ZQ4tgDfaydJNF6sSW3AgpSW/OXZkKoBvrSs6seoDlXx9NDLTkpZ\nIaOFwSVXcFO3cfkUzO6LKejPKz4vselILJ/3rZ8tRQFARChdsSZUHAfmLTizE+9dP9Npz690i9tG\nvGchNsbcxRdrW3AlPok376uVLTmUUo6hhSG3SYiF2f0g+RrTK3/O7K1xvNi1Oj3qpztv0PFEoHQD\npHQDvDq/A8fXkW/3XDodWEKHPCsZsPka3/nm5/G7Kjonj1LqjmlhyE1SU2D+U3B+H2sbTebtjYb+\nTcsztE1la/J4ekGVjrava9HIN52YEvUlXZaUplyx/HQM8rcml1LqlvSOYG6y+k04tJRDDV7hyU1F\naF+jJOPuq4XkhKGkeYsgfX6gkGcS3xb4kmdnb2NvRLTVqZRS6dDCkFv88yP8NZELNR/h/m21qVW6\nCJP6NchZo4H8qiP3TSQoeT+v+Mxh0IxtnL4cb3UqpdQNctBvDZVpJzbC4tFcK9eauw/fS/ECPnz7\nWDAF8uTAK4V1ekKTwfRNWcxdiZsYNGMbsQn66G+lchItDK7u4lH4eQApRQPpfelp4lNg5qDGlCyU\nsQlslug8HsoE86H31ySdP8KIWTtITkm1OpVSyk4LgyuLv2wblgqM9vwPBy95Mu3RYKqUzPwENqfw\n8oFeM/DyzsO84l+y5dAp3ly8D1ecU6NUbqSFwVWlJMMvj2GijjOxxBssPpWXj3rXo2ml7Jmr4HBF\ny8FD31A09ihzy/zCj5tP8u1GXQdCqZxAC4OrWv4SHPuDpYEv8emRkoztVoP76pW2OtWdqdIB2o6l\nzsVljC8XwvilB1ix76zVqZRye1oYXNGWqbDtG/ZVfIzh+4MY0Kw8T7euZHWqzGn9IlTuQP+oyTzo\nf55Rc/5hd/hlq1Mp5da0MLiaI6th+UtElu5Aj4Md6VCjJG/em0PmKmSGhwc8OA0pUJIPUj8mMH8i\nT8wMIUKHsSplGS0MruT8Qfj1ceKL1aDrqUepWboYk/rnsLkKmVHAF3p/j2fsWeaVmklCUhKDvttG\nzLUkq5Mp5ZZc/DeKG7l6AWb1JsUzLz2jnyFfwcJ8+1gw+X1y4FyFzCjbCLq+R4GTa1hUbxtHI2MZ\n9tMOknQYq1JO55DCICJdReSQiISKyNh09o8RkZ32r70ikiIixe37TojIHvs+fZZ2epIT4OcBmNhz\njGQM4am+zHi8Sc6eq5AZjZ+E2j0J3P0p01rF8ueRC7yxSIexKuVsWS4MIuIJTAa6AUFAPxEJStvG\nGPOhMaa+MaY+8B9gvTEmKk2Tdvb9t31OuNsxBhaPhrC/+azAaFZHl7PPVShodTLHE4F7PwffqrTb\n8x/GtCjErC1hTPvzmNXJlHIrjugxNAFCjTHHjDGJwBygxy3a9wNmO+Bz3cPGT2HXLH73fYzPz9Xl\n4971aFKxuNWpsk+egtDnB0iKZ1jkeO6t7ce7Sw+ybM8Zq5Mp5TYcURjKAKfSvA+3b/sfIpIf6ArM\nS7PZAKtFZLuIDHZAntzjwGJY8xb7i3dieEQn/tOtBve62lyFzPCrDj0mIac282nx+TQoX5TRP+/k\nn7BLVidTyi04++bzvcCmGy4jtbRfYuoGDBeR1ukdKCKDRSREREIiIyOdkdVaZ3bB/MFEFqnDA6cf\n5tHmgQx21bkKmVH7IWjyNF5bv2Jm0zOULJyHp74P4VRUnNXJlMr1HFEYIoByad6XtW9LT19uuIxk\njImwfz8P/Ibt0tT/MMZMNcYEG2OC/fz8shw6R7tyBmb1Jd67CHefH0qrmmV5w5XnKmRW53egbGMK\nrxjNj/f7kpicyqAZ24iO12GsSmUnRxSGbUBVEakoIj7YfvkvurGRiBQB2gAL02wrICKF/n0NdAb2\nOiCT60qMgzn9SIm/TJ+Y0QSUqcDEfg3w9HCzogD/fdgeXj5UWD2EqX1rcvzCVYb9tF2HsSqVjbJc\nGIwxycAIYAVwAJhrjNknIkNEZEiapg8AK40xV9Ns8wc2isguYCvwuzFmeVYzuazUVFgwBHN6Jy+k\nPsOlQtX4ZmDj3DNXITOKlIWHvoHzB2i2fzzvPVCbTaEXefU39/77Qans5JDfOMaYpcDSG7ZNueH9\nDGDGDduOAfUckSFXWPcu7F/I1z6P8UdyI+Y93gS/QnmsTmW9yu2h7X9g3bv0Kt+U421b8OW6o3St\nU4p21UtanU6pXEdnPucUu+fChg9Zk7czn8R1YdqjwVT2y4VzFTKr9Rio0hGWvcSzQVepWKIA7yzZ\nr5eUlMoGWhhyglNbMQtHcDhfPYZcHsCnvRvQODAXz1XIDPvD9ijoj/e8x3ijQwBHI6/yw98nrU6m\nVK6jhcFql8NgTn8uefnR+9IwXuxeh7vrBlidKmfKXxx6zYSYM7TZ/yqtqvjy2erDRF1NtDqZUrmK\nFgYrXbsCs/qQmHCNXldG0aN5bZ5sVdHqVDlb2UbQ+R3kyErer3mUq4kpfLrqsNWplMpVtDBYJTUF\n5j1JauQhnogbQaWaDXndHecqZEaTp8C/DqW3vsfAxv78tOUkh87GWJ1KqVxDC4NVVr4GR1YwLmUg\nV8q0YmJfN52rkBkentD1XYg+xZjCqymU15u3l+zXp7Aq5SBaGKywfQZsnswc6cbagvfx7cBg8vl4\nWp3KtVRsDTXuId/mz/lPq6JsDL3AmgPnrU6lVK6ghcHZjm/A/P48Wzwb8iEDmfF4Y0oU1LkKmdL5\nbUhJpHf0DKqULMj4pQdITNbhq0pllRYGZ7oQivn5EcKlNEMTRvD1wKZU0rkKmVe8EjQbiseuWUxo\nnsLxC1eZ+dcJq1Mp5fK0MDhLXBRmVm9iEw39455jfJ8WBOtchaxr/QLk9yX4wIe0q1aCiWuOcCE2\nwepUSrk0LQzOkJIEvwwk5VIYj8ePYmD3NnSro3MVHCJvEWj/KoT9xbs1jxOflMLHK3X4qlJZoYUh\nuxkDS1+A4xt4MeEJajfvyhMtda6CQzV8FPxrE7BlPI81LcXP28LYf/qK1amUcllaGLLb5q9g+wy+\nTL6P2Bq9eO2eIJ2r4GgentDlXbgcxguF1lIknzfjluzT4atKZZIWhux0bj9m5SusTG3M6oDBfK5z\nFbJPpTZQ/W7ybv6Ul1sXZ/OxKFbsO2t1KqVckhaGbJS04jViTV4mFRzJtIFNdK5Cduv8NiQn8NDl\nGVT3L8T4pQe4lpRidSqlXI4WhuxydC3ex1YzMel+3nu4Lb46VyH7+VaGpk/jsfNHJrQwnIqKZ/qm\n41anUsrlaGHIDqkppKx4jXDjx9nqj1K7TBGrE7mP1mMgf3Ea7P+AjjVKMnltKOevXLM6lVIuxSGF\nQUS6isghEQkVkbHp7G8rItEistP+9XpGj3VJu3/G8/xe3k/qw7BOtaxO417yFYV2r8DJjYyveYLE\nlFQ+XHHI6lRKuZQsFwYR8QQmA92AIKCfiASl0/RPY0x9+9e4OzzWdSTGkbp6HLtNFVKCHqBmQGGr\nE7mfhgOhZBD+f7/Nk83L8OuOcPaER1udSimX4YgeQxMg1BhzzBiTCMwBejjh2Jxp82Q8Ys/wdmJ/\nRnWsbnUa9+TpBV3Gw+WTjCq4muL5fXhrsQ5fVSqjHFEYygCn0rwPt2+7UQsR2S0iy0Tk3+srGT0W\nERksIiEiEhIZGemA2Nkg9jzmz09ZbRpTsnY7qpcqZHUi91W5PVTrRt6/PuXVtr6EnLzEkt1nrE6l\nlEtw1s3nHUB5Y0xdYBKw4E5/gDFmqjEm2BgT7Ofn5/CADrHuPVKTr/FuUl9GdaxqdRrV+R1IjqfH\npRnUDCjMhGUHdfiqUhngiMIQAZRL876sfdt/GWOuGGNi7a+XAt4iUiIjx7qMyEOY7TOZk9qRoNoN\nqeavvQXLlagCTZ7GY8f3vH8XRFyOZ9qGY1anUirHc0Rh2AZUFZGKIuID9AUWpW0gIqXE/hwIEWli\n/9yLGTnWZax6g0SPvHyS+ACjOmhvIcdoMwbyFaPu3vfpVsufL9cd5Wy0Dl9V6layXBiMMcnACGAF\ncACYa4zZJyJDRGSIvVlPYK+I7AImAn2NTbrHZjWT0x3fAIeXMTnpPlrUrUFV7S3kHPmKQbuX4cSf\njKsRRkqq4YPlB61OpVSOJq44UiM4ONiEhIRYHcMmNRWmtSX64lmaxrzPkmc7UqWkFoYcJSUZptwF\nyQl8VPV7vthwit+GtaBB+WJWJ1PKqURkuzEm+HbtdOZzVu35Bc7s4t1rvehSL1CLQk7k6WV7+uql\n44ws+Ad+hfIwbsl+Hb6q1E1oYciKpHhYM44z+avza1IzRrbXews5VpUOULULeTZ9xKtt/fgn7DIL\nd562OpVSOZIWhqzYMgWuhDM2pjf31itLlZK6fnOO1vkdSIrjvqgZ1ClThAnLDhKXmGx1KqVyHC0M\nmXX1Avz5CYeLtuTP5JqM1JFIOZ9fNWj8FLJjBu+19ODslWtMWa/DV5W6kRaGzFr/PibxKqMvPkCP\n+mWo7Ke9BZfQ5kXIW4TauydwT51SfL3+KBGX461OpVSOooUhMy4cgZDp7ChxHweTAxjZvorViVRG\n5S8ObV+G4+t5q2Y4ABOW6fBVpdLSwpAZq98k1TMPz5ztwv0NylBJewuuJfhxKFEd341vMbRlORbv\nOq1PX1UqDS0Md+rkX3BwCev9BnA2pYiORHJFnt624atRRxmSfy2F83rx+ZojVqdSKsfQwnAnUlNh\nxSukFAxgdFgL7q9fhoolClidSmVG1Y5QpRN5N33E8KbFWX3gHHsjtNegFGhhuDP75sPpHSzxfYLY\nVB+e6aD3Flxal/GQGMtjibMolNeLidprUArQwpBxSddg9Vsk+dVi7NEgHmxQhgq+2ltwaX7VofET\n5Nk5g+cbCCv3n2P/6StWp1LKcloYMmrrVIgOY3aRwSSmCiN0JFLu0PpF8MpH/7gfKZRHew1KgRaG\njImLgg0fkRDYgXcO+vNQQ+0t5BoF/aDZUHwOLeTF+oks33eWA2e016DcmxaGjNjwISTG8G2+x0hN\nNYxopyORcpUWIyFvEfrGfk/BPF5MWqu9BuXetDDczsWjsHUacbX689kebx5qWJbyvvmtTqUcKV9R\nuGsU3kdX8p86MSzdc5ZDZ2OsTqWUZbQw3M6at8DTh8n0svUW9N5C7tR0CBTwo/eVGRTM48VE7TUo\nN+aQwiAiXUXkkIiEisjYdPY/LCK7RWSPiPwlIvXS7Dth375TRHLI6jt2YVtg/0JiGg1l2j/x9Aou\nS7ni2lvIlXwKQKvn8Q77k9dqRbJ0zxkOn9Neg3JPWS4MIuIJTAa6AUFAPxEJuqHZcaCNMaYO8DYw\n9Yb97Ywx9TOyspDTGAMrX4GCpfj0aldSjWFYW+0t5GqNHofCZXjo0nfk9/bQEUrKbTmix9AECDXG\nHDPGJAJzgB5pGxhj/jLGXLK/3QyUdcDnZq/9CyB8G5ebvciPOy7QK7ic9hZyO++80OZFvM6E8FbN\nCH7fc4Yj2mtQbsgRhaEMcCrN+3D7tpt5AliW5r0BVovIdhEZ7IA8WZecCKvfhJJBfBzZCINheLvK\nVqdSzlD/YShWkfujppPfW5i0NtTqREo5nVNvPotIO2yF4aU0m1saY+pjuxQ1XERa3+TYwSISIiIh\nkZGR2Rt02zdw6QQXWrzGnJDT9AouR9li2ltwC57e0O5lvCL38U61oyzefZrQ87FWp1LKqRxRGCKA\ncmnel7Vvu46I1AW+AXoYYy7+u90YE2H/fh74Ddulqf9hjJlqjAk2xgT7+fk5IPZNxF+C9e9DpXZ8\nctx2WsPb6b0Ft1L7IfCryX1RMyjgBV/oCCXlZhxRGLYBVUWkooj4AH2BRWkbiEh5YD7wiDHmcJrt\nBUSk0L+vgc7AXgdkyrwNH8G1aM41e4VfQk7Rp3E5yhTNZ2kk5WQentD+FTyjQnmvyn4W7TrNsUjt\nNSj3keXCYIxJBkYAK4ADwFxjzD4RGSIiQ+zNXgd8gS9vGJbqD2wUkV3AVuB3Y8zyrGbKtKjjtmci\n1X+Yz/bmRRAdieSuatwDpRvQ/eJMCnil8IXea1BuxMsRP8QYsxRYesO2KWlePwk8mc5xx4B6N263\nzJpxIJ6cbvgcv0w5RL8m5SmtvQX3JALtX8Xzx4f4oOJOhu/0ZGSHqrr+hnILOvP5X+EhtvUWWoxk\nUshVPEQYpiOR3FvlDlC+BZ0v/kBhryTtNSi3oYUB7JPZXoUCfoQHPcUvIeH0a1KOgCLaW3BrItDh\nNTyvnufD8ltZsDOCkxevWp1KqWynhQHg4BII+xvavcykjWfx8BCG6r0FBVChBVTuQIeLsyjiEa+9\nBuUWtDAkJ8Kq16FEdcIq9OTXHeH0b1KeUkXyWp1M5RTtX8Xj2iU+KbuR+f9EEHYxzupESmUrLQzb\nv4OoY9D5bb5YfxxPD2FoW723oNIo0xBq3kvriz/j53GFyX9or0Hlbu5dGK5Fw7oJULE1J4vfxbwd\nEfRvUh7/wtpbUDdo/zoeyfF8FrCaeTvCORWlvQaVe7l3YfjzE9tM587vMOmPo3h5CMO0t6DS41cN\nGgyg6cUFlJdIvlynvQaVe7lvYbgcBpu/grp9OOFdhd/+ieDhphUoqb0FdTNt/4N4ePJJyd/5JSSc\n8Evaa1C5k/sWhjVv/3cS06S1oXh5CEPaVrI6lcrJCpeGpk9T79JKashJvlx31OpESmUL9ywMETtg\nz1xoNozjycX57Z9wBjSrQMlC2ltQt9HyWSRvYT4uvpBfQk4RcTne6kRKOZz7FQZjYOVrkL8EtHyW\nSWuO4OPlwZA2em9BZUC+YtDyWarH/E1jOcBXeq9B5ULuVxgOL4eTG6HtWI7FeLBgZwSPNKuAX6E8\nVidTrqLJ01AogAmF5zF32ynORGuvQeUu7lUYUpJsvQXfKtDoMSatDcXHy4PBrbW3oO6AT35oO5by\ncftozza+0nsNKpdxr8KwYyZcPAKdxnE0KoGFOyN4tHmg9hbUnas/AHyr8lbBefyy9QRno69ZnUgp\nh3GvwpCSBFU7Q/XuTFpzhDxengxurSORVCZ4ekGH1/FPOEkPWc+U9dprULmHexWGZkOh/1xCI6+y\naNdpHm1egRIFtbegMqnmvVAmmLF5f2Pe1lDOXdFeg8od3KswAIgwcc0R8nprb0FlkQh0fJOiyZH0\nZ4X2GlSu4ZDCICJdReSQiISKyNh09ouITLTv3y0iDTN6rKOFno9h8e7TPNo8EF/tLaisqtgKqnRk\nVJ5FLN5ygPPaa1C5QJYLg4h4ApOBbkAQ0E9Egm5o1g2oav8aDHx1B8c61OdrQsmnvQXlSB3eIH9K\nDE/IQr7ecMzqNEplmSN6DE2AUGPMMWNMIjAH6HFDmx7A98ZmM1BURAIyeKzDHD4Xw5LdpxnYIpDi\nBXyy62OUuwmoC3V68YTXClZt+YfImASrE6lcyBjD+sORpKaabP8sRxSGMsCpNO/D7dsy0iYjxzrM\nV+uOkt/bk8GttLegHKzdK3hLKkP4lakb9F6Dcrw1B84zcPpWlu49k+2f5TI3n0VksIiEiEhIZGRk\npn7GK3fXZPLDDSmmvQXlaMUrIsGP08dzPRs3/82FWO01KMdJTTV8tPIQFUsUoGutUtn+eY4oDBFA\nuTTvy9q3ZaRNRo4FwBgz1RgTbIwJ9vPzy1TQEgXz0LZ6yUwdq9RttR6DeOVhJHOYpvcalAMt2nWa\ng2djeK5TNbw8s//veUd8wjagqohUFBEfoC+w6IY2i4BH7aOTmgHRxpgzGTxWKddQsCQed42ku+dW\ndvy9hovaa1AOkJicyierDhMUUJi76wQ45TOzXBiMMcnACGAFcACYa4zZJyJDRGSIvdlS4BgQCkwD\nht3q2KxmUsoyzUeQkrc4o2WW9hqUQ8wNOUVYVBxjulTHw0Oc8plejvghxpil2H75p902Jc1rAwzP\n6LFKuay8hfFs+yJ3LR/Ld5sXEdWmso6AU5kWn5jCxDVHaBxYjLbVM3cJPTNc5uazUi4jeBBJhcox\nmll8s0HXa1CZN/PvE5yPSeDFrjUQcU5vAbQwKOV4Xnnw7vgqtT1OcP7v2Vy6mmh1IuWCouOT+Grd\nUdpV96NxYHGnfrYWBqWyQ51eJBSvwQh+5rsNh61Oo1zQtA3HiI5P4oUu1Z3+2VoYlMoOHp7k6TqO\nQI9zXN08nctx2mtQGRcZk8D0Tce5p24AtUoXcfrna2FQKrtU7UxcQBOG8Cs/rNfBdirjJv8RSkJy\nKs93dn5vAbQwKJV9RMjffTx+Eg2bvyI6LsnqRMoFhF+KY9aWMHoHl6ViiQKWZNDCoFR2KteEKxU6\n8xiLmLXuH6vTKBfw2eojIPBMh6qWZdDCoFQ2K3z32xSQBAps/YzoeO01qJs7ci6G+TvCGdi8AgFF\n8lmWQwuDUtmtZA2iq/Wij1nBvDV/W51G5WAfrzxMfh8vhratYmkOLQxKOUGxu1/Hw0PwDfmIK9e0\n16D+165Tl1m+7yxPtqpo+Wx5LQxKOUORslyuPYh7zQaWrFxtdRqVA3208hDFC/jwZA5YL0YLg1JO\n4tdtLNc88lNmx4fEaK9BpfHX0Qv8eeQCw9pWpmAehzzCLku0MCjlLPmLE91oOG3YzqrlC6xOo3II\nYwwfLD9E6SJ5GdCsgtVxAC0MSjlVQOdnueRZnEo7PyRWew0KWLX/HDtPXWZUx6rk9fa0Og6ghUEp\n5/LJT2yzF6jPITYs+d7qNMpiKfYlOyuVKMBDDctaHee/tDAo5WTl2g/mrFcZqu39lKvxusqbO1u0\nK4LD52J5rrNzluzMqCwlEZHiIrJKRI7YvxdLp005EflDRPaLyD4RGZVm35siEiEiO+1f3bOSRymX\n4OlNXKuXqcIpti780uo0yiL/LtlZq3Rhutd2zpKdGZXVEjUWWGOMqQqssb+/UTLwvDEmCGgGDBeR\noDT7PzXG1Ld/6Upuyi1Uav0wx7yrUfPgF8TFxVodR1ng521hnIqKd+qSnRmV1cLQA5hpfz0TuP/G\nBsaYM8aYHfbXMdjWdi6Txc9VyrWJkNz+dUpxgV3zP7Y6jXKyuMRkJq4NpUnF4rSplsElO5PiYeWr\ncC06e8OR9cLgb4w5Y399FvC/VWMRCQQaAFvSbB4pIrtFZHp6l6KUyq2qNb+XPXkaUjN0KvFXLlkd\nRznRjL9OEBmTwItdqmdsyc7UVPjtafhrEoRtzvZ8ty0MIrJaRPam89UjbTtjjAHMLX5OQWAeMNoY\nc8W++SugElAfOAPc9E8nERksIiEiEhIZGXn7M1PKBXh0fouixHJw/jtWR1FOEh2XxJR1R2lfoyTB\nGV2yc9VrsH8hdH4HqnXJ3oBkoDAYYzoaY2qn87UQOCciAQD27+fT+xki4o2tKPxkjJmf5mefM8ak\nGGNSgWlAk1vkmGqMCTbGBPv5ZbDrpVQOV6tRa/7O14YaJ34gPirC6jjKCab+eZQr15J5IaOL8Gye\nAn9/AU2ehuYjsjecXVYvJS0CBtpfDwQW3thAbP2kb4EDxphPbtiX9lb8A8DeLOZRyuXk6/IGXiaZ\nE/PfsDpuk4mtAAAWkElEQVSKymbnY64xfeMJ7qtXmqDShW9/wIHFsHws1LgHur4HGbns5ABZLQwT\ngE4icgToaH+PiJQWkX9HGN0FPAK0T2dY6gciskdEdgPtgGezmEcpl1O/fiP+KNCNquHzSTh3xOo4\nKhtNXhtKUkoqz3WqdvvGp7bBvCehTCN4cBp4OG9WdJae1mSMuQh0SGf7aaC7/fVGIN0yZ4x5JCuf\nr1RuUbz7ayT+soqz81+h0tC5VsdR2eBUVByztobRu3E5Am+3ZOfFozC7DxQKgP4/g09+54S0yzlT\n7ZRyY8G1a7Cs0ANUOreChLDtVsdR2eDT1YfxEOGZ9rdZsvPqBfipJxgDA+ZBgRLOCZiGFgalcoiy\n3ccSZQpyceHLVkdRDnb4XAy//RPBwBaBlCqS9+YNE+Ngdl+4ctrWU/Ct7LyQaWhhUCqHaFIzkMWF\n+1H64mYSj6y1Oo5yoI9XHqKgjxdD29ziF31qCsx/CsJDbPcUyt10kGa208KgVA4hIlS5+1kijC9X\nFr9qu5SgXN7OU5dZse8cT7WuRLFbLdm54hU4uMQ2+ijoPucFTIcWBqVykBbVSzO/yEBKXNlH0p75\ntz9A5XgfrjiIbwEfBrWsePNGf0+GLV9Bs2HQbKjzwt2EFgalchARoV73pzmYWo745W9Cii7m48o2\nhV5gU+hFhrercvMlO/ctsPUWat4Hncc7N+BNaGFQKodpVd2fecUGUTgujOSQmbc/QOVIxhg+WGFb\nsvPhZuXTbxS2GeYPtt1PeHAqeOSMX8k5I4VS6r9EhLu6Pcy21GokrX0PEq9aHUllwsr959h16jKj\nO1Yjj1c6k9MuhNpGIBUpC31ng3c+54e8CS0MSuVAbaqX5NfiT5Ev4QLJf+liPq4mJdXw0YpDVPYr\nwIMN01llIDYSfnoIxBMG/AoFfJ0f8ha0MCiVA4kIXbvez6qUhqRu/AzioqyOpO7Agn8iOHI+luc7\nV//fJTsTr8Ks3hBzzjZXoXgla0LeghYGpXKottX9WOj7BF7JV0nZoIv5uIrE5FQ+XX2YOmWK0K12\nqet3pqbYnn90Zif0/BbKBlsT8ja0MCiVQ4kID3TpxPyUVrB1Klw+ZXUklQGzt4YRfsm2ZOd1i/AY\nA8tehENLodsHUONu60LehhYGpXKw9jVKstT3cVJSU0n94z2r46jbiEtMZtLaUJpWLE6rqjc84+iv\nibDtG2gxEpo8ZU3ADNLCoFQOJiL063wX3yd3QnbNhvMHrY6kbuG7TSe4EJvAi11rXN9b2DsPVr0O\ntR6AjuOsC5hBWhiUyuE61izJmhIDuEpeUle/ZXUcdRPRcUl8vf4oHWuWpFGFNMvXn9gEvw2B8i3g\n/ik5Zq7CreT8hEq5ORHhsU7BTEm6G4/DSyFsi9WRVDqmbDhKTEIyL3RJs2Rn5CGY0w+KVoC+P4H3\nLZ6smoNoYVDKBXQO8mejXx+ipChm9Rv6gL0c5vyVa3y36Tg96pWmRin7kp0x5+DHnuCZxzZXIX9x\na0PegSwVBhEpLiKrROSI/Xuxm7Q7YV/Cc6eIhNzp8Uq5OxFhSMfafJL4ABL2NxxZaXUklcaktaEk\npxie/XfJzoRYmNUL4i7Y5ioUC7Q0353Kao9hLLDGGFMVWGN/fzPtjDH1jTFpB+7eyfFKubXOQaX4\np8R9REgpzOo3bWPileXCLsYxe2sYfRqXo4JvAUhJhl8fh7N7oOd3UKah1RHvWFYLQw/g36d8zQTu\nd/LxSrkNDw9heMeaTEjoiZzfD3t+sTqSAj5bfRgvT+GZDlVtl/iWPm/r0d39MVTvanW8TMlqYfA3\nxpyxvz4L+N+knQFWi8h2ERmcieOVUkDXWqU4UqITRzwqYda+A8kJVkdya4fOxvDbTtuSnf6F88LG\nT2D7DGj5LAQPsjpept22MIjIahHZm85Xj7TtjDEGWwFIT0tjTH2gGzBcRFrf2OA2xyMig0UkRERC\nIiMjbxdbqVzJw0MY0bEa4671RqJPQch0qyO5tY9WHqJgHvuSnbvnwppxULsntH/d6mhZctvCYIzp\naIypnc7XQuCciAQA2L+fv8nPiLB/Pw/8Bvy7mGmGjrcfO9UYE2yMCfbz87uTc1QqV+leO4Czvs35\nx7MuZsOHcO2K1ZHc0o6wS6zaf46nW1ei6LnNsGAYVGgJ93/pEnMVbiWr6RcBA+2vBwILb2wgIgVE\npNC/r4HOwN6MHq+Uup6HhzCyYzXeiOuFxF2EJaMh/pLVsdyKMYYPlx+iREEfBlVLgDkDwLcy9P0R\nvPJYHS/LsloYJgCdROQI0NH+HhEpLSJL7W38gY0isgvYCvxujFl+q+OVUrd2d50Arpaoyw95+mL2\n/QaTgmHHD5CaanU0t7Ap9CJ/H7vIC80Lk39uH9vEtYd/gXy5Y8S9GBecKBMcHGxCQkJu31CpXGzh\nzghGzdnJtM4+dDr+EZzaDGUbQ/ePoHR9q+PlWsYY7p+8ibiYaFYUfQ+PqGPw+FKX+G8uIttvmDKQ\nLte+EKaUG7unbmmCAgrz1MpEhud9l4udPodLJ2BqW1jynC7uk01W7DvLvvCL/FB4Mh7n90PvmS5R\nFO6EFgalXJSnh/Dr0OY806Eqaw6ep/kyfz6tMZvERk/B9u/gi2DYPlMvLznQv0t2Tiz0A6UiN8E9\nn0LVTlbHcjgtDEq5sPw+XjzXqRp/vNCWe+oE8Pmm8zTf2ZnfW8zF+FaFxc/Atx0hYofVUXOF+TvC\n6Rb1A92TVkHrMdBo4O0PckFaGJTKBQKK5OOTPvVZOPwuKvkVYPiaRLpGv8yh5h/aVn6b1h4Wj9bL\nS1mQkJzCoRVf87z3r5i6faDdK1ZHyjZaGJTKReqVK8rcp5vz5cMNiUtOocsfZRhWfCqX6z4BO76H\nSQ0h5Dt9zlImrFs6l5cSv+Syf3Pkvi8g7UI8uYyOSlIql7qWlMLMv07wxdpQ4pJSeK5uEoNjv8I7\n/G8o3QC6fwxlG1kd0yXEndpF6rddiPLyp9zz65F8Ra2OlCk6KkkpN5fX25On21TmjzFt6du4HB/v\n8qJR+Gj+qPUu5soZ+KYDLBoJVy9aHTVni44g9YeexJq8RD84y2WLwp3QwqBULleiYB7GP1CH5aNb\nU798cR7fHsjdqZ9wvOpjmH9+sl1e2vatXl5Kz7VoUn7sCYkxfF32feoE1bI6kVNoYVDKTVTzL8T3\ng5rw3eONSfQqSLs9nRhTYjKxxWrC78/BtHZwapvVMXOO5ESY+yhcOMSwxFH0vbe71YmcRguDUm6m\nXfWSLB/Vird71GLNRV/qnBjBj2XfIOXKOdvQ1oXD4eoFq2NayxhYPAqOreOV5KfwrdeN6qUKWZ3K\nabQwKOWGvDw9eKR5IOvGtOOpVpV563gNmsZMYHvZRzC75tguL22d5r6Xl9a9B7tmsabUIH5NacOz\nHatZnciptDAo5caK5PPm5e41Wf1cG4Krlueh0G708/yE8wVrwtIXYGobCNtidUzn2vcbrH+f2Jq9\neTqsI/2alKe8b36rUzmVFgalFBV8CzDlkUb8PLgZsYUr0SR8JO8XGkvilUiY3tm21kCsGyyQdekk\nLBoFZYJ5I/UpvDw9GNm+itWpnE4Lg1Lqv5pW8mXR8JZ81Ks+8xMaUz/qXVYW64fZPRcmNYItX9sW\nu8+NUpJg3pOA4Wibz5m/O5LH76pIycJ5rU7mdFoYlFLX8fAQejYqyx8vtOWpDnV45kIPuia8z/E8\n1WHZi7ant4Zttjqm4617D8K3wj2f8t7f8RTM48WQ1pWtTmUJLQxKqXTl9/HiWfsD+mrVa0S786MY\n4/ECV6MjYXoX+G0IxN50NV7Xcmw9/PkJNBjAWu9WrD5wniFtKlMkv7fVySyRpcIgIsVFZJWIHLF/\n/5/li0SkuojsTPN1RURG2/e9KSIRafa5z0BhpVxEQJF8fNK7PotGtOREyQ4EX36X2Xl6kbrnV9vl\npc1fufblpasXYP5g8K3CgfqvMXLWP9QqXZhBd1W0OpllsvSsJBH5AIgyxkwQkbFAMWPMS7do7wlE\nAE2NMSdF5E0g1hjz0Z18rj4rSSlrGGNYvvcs7y47gNelY0wsMps610KgZC3o/iEE3mV1xDtjDMzq\nA8f+4ELfZdzzSzQisGD4XfjnwnsLznpWUg9gpv31TOD+27TvABw1xpzM4ucqpSwgInSrE8Dq59rQ\nt2s7+sePYWjSc1y+dAFmdId5T0HMWatjZtyWKXBkBQnt32LgsnhiriXx7cDGubIo3ImsFgZ/Y8wZ\n++uzgP9t2vcFZt+wbaSI7BaR6eldilJK5Tx5vGwP6Fs3ph2+jR+kRewEpvIgKXt/w0wKhr8n20b5\n5GRndsGq1zHVujLscDAHzlzhi4cbElS6sNXJLHfbS0kishoolc6uV4CZxpiiadpeMsak+8tdRHyA\n00AtY8w5+zZ/4AJggLeBAGPMoJscPxgYDFC+fPlGJ09qp0OpnOLwuRje+f0AYUd2MyH/TzRL2YHx\nq4nc/REEtrQ63v9KiLVN3ku8ygcVv+XLrZd5+/7aPNKsgtXJslVGLyVl9R7DIaCtMeaMiAQA64wx\n1W/Stgcw3BjT+Sb7A4Elxpjat/tcvcegVM607tB5xi/ZT+DF9YzP9xMlU85B7Z7Q+R0oHGB1vP+3\nYDjs/InljaYyZFMBnmhZkdfuCbI6VbZz1j2GRcC/i54OBBbeom0/briMZC8m/3oA2JvFPEopC7Wt\nXpJlo1vT+t6B9DCf8HnygyTtW0TqpEawaWLOuLy051fY+SPHg4Yy7K8CdAry5+XuNa1OlaNktcfg\nC8wFygMngd7GmCgRKQ18Y4zpbm9XAAgDKhljotMc/wNQH9ulpBPA02nuWdyU9hiUyvmi45OY/Eco\nazZt5lWv72knO0j1rYbH3R9BpTbWhIo6DlNacbVYNZqdeZ5AvyL8/HQz8vt4WZPHyZxyKckqWhiU\nch0nL15lwrKDJO7/nXE+P1KGc5haDyKd34EiZZwXJCUJpnch9UIoD6ROINLDnwXD73KrR17o0p5K\nqRyhgm8BvhrQiMFPDueZ4l/xadJDJO5bQsqkYNj4mW1BHGdY+w5EbOc9r6EcTfRl+uON3aoo3Akt\nDEopp2hayZdfRrSn/IPj6OP9GWsTasLqN0ia3ByO/pG9H350LWz6jHUF72b6pXpMfrghNUrpsNSb\n0cKglHIaDw/hoUZlmTWmD3tbT2FwykucjoqBH+4nac6jEB3u+A+NjcTMf5pzeSsy5EJP3u5RmzbV\n/Bz/ObmIFgallNP9+4C+cS88x+SaP/BRUi9SDi4jeWIjUjZ87LjLS6mpsGAIKfHRPBI9hIGta9K/\naXnH/OxcTAuDUsoypYrk5YO+Tek89COeLzGVNYm18Vw7jrjPm0Domqx/wObJELqaNxP6U7lWY17q\nWiPrP9MNaGFQSlmubtmifDH8flJ7/8gLeV7nXHQc/Pggsd/3hcunMvdDI3aQuvotVpnG7AnoySe9\n6+PhIY4NnktpYVBK5Qj/PqBv/JjRrGm/kM9NXzyPriHx80bErXkfkhMy/sMSYkj+ZRDnU4vwcd6R\nfDOwMfl8PLMvfC6jhUEplaPk8fLkybY1GTBmIl/UmsWa5Hrk//Ndoj8OJunQygz9jKRFzyGXT/IS\nI5k0qD1+hfJkc+rcRQuDUipH8i2YhzG9O1F5xHw+9HuPC1eT8J7di3NTH8JcOnHT41L+mYX3vrl8\nkfwATz08gKr+hZwXOpfQwqCUytGq+RdizPBhhPddxTd5HqVQxJ8kfd6Yc4vHQdK169qaC6EkL36O\nLak1KHXva7SsWsKi1K5NC4NSyiW0CSrHYy9+zrK2i1knjfDf/jEXPmzIpZ2LbQ2SE7kwcwBxKZ5s\nb/QBfZq679KcWeUeT45SSuUKXp4ePNSuKdHNFjN7wSyaHpxApQUDOLahFR4FSxIYc4CvS7/NkHtb\nWx3VpWmPQSnlcork86Zfv4F4Dv+L+b6D8b+4lcCweSzNdw8DBw3XYalZpD0GpZTLqlCyGBVGfsg/\nex8jfPM8mvccRV5vHZaaVVoYlFIur0HtWjSoXcvqGLmGXkpSSil1HS0MSimlrpOlwiAivURkn4ik\nishNVwUSka4ickhEQkVkbJrtxUVklYgcsX8vlpU8Simlsi6rPYa9wIPAhps1EBFPYDLQDQgC+olI\nkH33WGCNMaYqsMb+XimllIWyVBiMMQeMMYdu06wJEGqMOWaMSQTmAD3s+3oAM+2vZwL3ZyWPUkqp\nrHPGPYYyQNrn5obbtwH4G2PO2F+fBfydkEcppdQt3Ha4qoisBkqls+sVY8xCRwUxxhgRMbfIMRgY\nDFC+vK7ApJRS2eW2hcEY0zGLnxEBlEvzvqx9G8A5EQkwxpwRkQDg/C1yTAWmAgQHB9+0gCillMoa\nZ0xw2wZUFZGK2ApCX6C/fd8iYCAwwf49Qz2Q7du3XxCRk5nMUwK4kMljXZWes3vQc3YPWTnnChlp\nJMZk/o9vEXkAmAT4AZeBncaYLiJSGvjGGNPd3q478BngCUw3xoy3b/cF5gLlgZNAb2NMVKYDZSxz\niDHmpkNrcyM9Z/eg5+wenHHOWeoxGGN+A35LZ/tpoHua90uBpem0uwh0yEoGpZRSjqUzn5VSSl3H\nHQvDVKsDWEDP2T3oObuHbD/nLN1jUEoplfu4Y49BKaXULeTawnCzB/el2S8iMtG+f7eINLQipyNl\n4Jwftp/rHhH5S0TqWZHTkW53zmnaNRaRZBHp6cx8jpaR8xWRtiKy0/6Ay/XOzuhoGfj/uoiILBaR\nXfZzftyKnI4kItNF5LyI7L3J/uz9/WWMyXVf2IbFHgUqAT7ALiDohjbdgWWAAM2ALVbndsI5twCK\n2V93c4dzTtNuLbaRcT2tzp3N/8ZFgf1Aefv7klbndsI5vwy8b3/tB0QBPlZnz+J5twYaAntvsj9b\nf3/l1h7DrR7c968ewPfGZjNQ1D772lXd9pyNMX8ZYy7Z327GNgvdlWXk3xlgJDCPW8ysdxEZOd/+\nwHxjTBiAMcYdztkAhUREgILYCkOyc2M6ljFmA7bzuJls/f2VWwvDrR7cdydtXMmdns8T2P7icGW3\nPWcRKQM8AHzlxFzZJSP/xtWAYiKyTkS2i8ijTkuXPTJyzl8ANYHTwB5glDEm1TnxLJOtv790zWc3\nJCLtsBWGllZncYLPgJeMMam2PyhzPS+gEbaJo/mAv0VkszHmsLWxslUXYCfQHqgMrBKRP40xV6yN\n5bpya2G41YP77qSNK8nQ+YhIXeAboJuxzTx3ZRk552Bgjr0olAC6i0iyMWaBcyI6VEbONxy4aIy5\nClwVkQ1APcBVC0NGzvlxYIKxXXwPFZHjQA1gq3MiWiJbf3/l1ktJ/31wn4j4YHtw36Ib2iwCHrXf\n3W8GRJv/XxvCFd32nEWkPDAfeCSX/AV523M2xlQ0xgQaYwKBX4FhLloUIGP/Xy8EWoqIl4jkB5oC\nB5yc05Eycs5h2B+tIyL+QHXgmFNTOl+2/v7KlT0GY0yyiIwAVvD/D+7bJyJD7PunYBuh0h0IBeKw\n/dXhsjJ4zq8DvsCX9r+gk40LP4Asg+eca2TkfI0xB0RkObAbSMX2MMt0hzy6ggz+G78NzBCRPdhG\n6bxkjHHpJ66KyGygLVBCRMKBNwBvcM7vL535rJRS6jq59VKSUkqpTNLCoJRS6jpaGJRSSl1HC4NS\nSqnraGFQSil1HS0MSimlrqOFQSml1HW0MCillLrO/wGhNyE/DVGNlQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x146c5fe00b8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "M=9\n",
    "w = optimizePLS(xtrain, ttrain, M, 1/math.exp(20))\n",
    "p = np.poly1d(w[::-1])\n",
    "ytest = np.array([p(xtest[j]) for j in range(len(xtest))])\n",
    "plt.plot(xtest, [sin2pi(xi) for xi in xtest])\n",
    "plt.plot(xtest, ytest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that with M=9 and ln(lambda)=-20 our polynom is a good representation of the function sin(2pix) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Probabilistic Regression Framework"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We further extend the probabilistic model by considering a Bayesian approach to the estimation of this probabilistic model instead of the maximum likelihood estimator (which is known to over-fit the dataset). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the Bayesian approach, we do not learn an optimal value for the parameter w, but instead, marginalize out this parameter and directly learn the predictive distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1274,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def bayesianEstimator(x, t, M, alpha, sigma2):\n",
    "    \"\"\"\n",
    "    Given the dataset (x, t) of size N, and the parameters M, alpha, and sigma2 (the variance),\n",
    "    this function returns a tuple of 2 functions (m(x) var(x)) which are the mean and variance\n",
    "    of the predictive distribution inferred from the dataset, based on the parameters and the\n",
    "    normal prior over w.\n",
    "    \"\"\"\n",
    "    N = len(x)\n",
    "\n",
    "    #inv(S)\n",
    "    prod_mat = np.zeros((M+1, M+1))\n",
    "    for i in range(N):\n",
    "        phi_xi = polynom(x[i], M)\n",
    "        prod_mat += np.outer(phi_xi, phi_xi.T)\n",
    "    S_inv = alpha * np.eye(M+1) + prod_mat/sigma2\n",
    "\n",
    "    S = np.linalg.inv(S_inv)\n",
    "        \n",
    "    def m(new_x):\n",
    "        phi_t = polynom(new_x, M).T \n",
    "        rsum = np.zeros(M+1)\n",
    "        for i in range(N):\n",
    "            rsum += polynom(x[i], M)*t[i]\n",
    "        phi_tS = np.dot(phi_t, S)\n",
    "        return np.dot(phi_tS, rsum)/sigma2\n",
    "\n",
    "    def var(new_x):\n",
    "        phi_x = polynom(new_x, M)\n",
    "        prod = np.dot(np.dot(phi_x.T, S), phi_x)\n",
    "        return sigma2 + prod\n",
    "        \n",
    "    return (m, var)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1251,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def plotBayesianEstimator(N):\n",
    "    (xtrain, ttrain), (xdev, tdev), (xtest, ttest) = generateDataset3(N, sin2pi, 0.03)\n",
    "    M = 9\n",
    "    alpha = 0.005\n",
    "    sigma2 = 1/11.1\n",
    "\n",
    "    m, var = bayesianEstimator(xtrain, ttrain, M, alpha, sigma2)\n",
    "\n",
    "    plt.figure(figsize=(8, 6))\n",
    "    x_sin = np.linspace(0, 1, 101)\n",
    "    \n",
    "    #mean plot\n",
    "    plt.plot(xtest, [m(xi) for xi in xtest], c='r')\n",
    "    #confidence interval\n",
    "    plt.fill_between(xtest, [m(xi)-math.sqrt(var(xi)) for xi in xtest],  [m(xi)+math.sqrt(var(xi)) for xi in xtest], color='r',\n",
    "                            alpha=0.2)\n",
    "    #train data scatter plot\n",
    "    plt.scatter(xtrain, ttrain, s=50, facecolors='none',  edgecolors='b')\n",
    "    #sin2px plot\n",
    "    plt.plot(x_sin, [sin2pi(xi) for xi in x_sin], c='g')\n",
    "    \n",
    "    plt.legend(['Mean', \"sin(2pix)\", \"Confidence interval\", \"Train data\"], loc='upper right')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1275,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe8AAAFpCAYAAAC1YKAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzs3Xd8TecfwPHPuTd7b5EEMZPIEERq1mipVWq0ZrVUVdGh\n/VFqK6WqtKVoqaLDaFGjraKtGqVGBDETEWSI7L3v+f3xkNKauTdLnvfrdV+Sm5Nznptc+Z5nfb+K\nqqpIkiRJklR5aMq7AZIkSZIkPRwZvCVJkiSpkpHBW5IkSZIqGRm8JUmSJKmSkcFbkiRJkioZGbwl\nSZIkqZKRwVuSJEmSKhkZvCVJkiSpkpHBW5IkSZIqGRm8JUmSJKmSMSrvBtyLk5OT6unpWd7NkCRJ\nkqQycezYsURVVZ3vd1yFDt6enp4cPXq0vJshSZIkSWVCUZTLD3KcHDaXJEmSpEpGBm9JkiRJqmRk\n8JYkSZKkSqZCz3lLkiQ9CgoKCoiOjiY3N7e8myJVEGZmZnh4eGBsbFyi75fBW5IkqZRFR0djbW2N\np6cniqKUd3OkcqaqKklJSURHR1O7du0SnUMOm0uSJJWy3NxcHB0dZeCWAFAUBUdHR71GYmTwliRJ\nKgMycEu30vf9IIO3JElSFaAoCoMHDy7+vLCwEGdnZ7p3716OrZJKSgZvSZKkKsDS0pKwsDBycnIA\n2LVrF+7u7uXcKqmkZPCWJEmqIrp27cpPP/0EwNq1axkwYEDx17Kyshg2bBjBwcE0btyYLVu2ABAV\nFUWbNm1o0qQJTZo04a+//gJgz549tGvXjr59++Lt7c2gQYNQVbXsX1QVJVebS5IklaU334TQUMOe\nMzAQPv74vof179+fmTNn0r17d06ePMmwYcPYt28fALNnz6ZDhw6sXLmS1NRUgoODefLJJ3FxcWHX\nrl2YmZkRHh7OgAEDitNWHz9+nNOnT+Pm5karVq04cOAArVu3Nuxrk+5IBm9JkqQqIiAggKioKNau\nXUvXrl1v+9rOnTvZunUr8+fPB8QK+StXruDm5saYMWMIDQ1Fq9Vy4cKF4u8JDg7Gw8MDgMDAQKKi\nomTwLiMyeEulp6AA8vOhsFB8XFAARUWg0YiHVvvPv0ZGYGIiHpL0KHuAHnJp6tGjB//73//Ys2cP\nSUlJxc+rqsrGjRvx8vK67fjp06dTrVo1Tpw4gU6nw8zMrPhrpqamxR9rtVoKCwtL/wVIgAzekqEU\nFUF6OqSlQXIypKaKwP1vN7dHqKr4+Oa/N58zMgJnZ3B1BVtbuOUPhSRJ+hs2bBh2dnb4+/uzZ8+e\n4uefeuopFi1axKJFi1AUhePHj9O4cWPS0tLw8PBAo9GwevVqioqKyq/xUjEZvKWSy88XQfraNfHQ\n6UQgNjcHS0sRfB9WURGkpEBcnAjm1tZQvTo4OoqPtVrDvw5JqkI8PDx4/fXX//P8lClTePPNNwkI\nCECn01G7dm22b9/OqFGj6NOnD2vWrKFz585YWlqWQ6ulf1Mq8urAoKAgVdbzrmBUFZKS4NIl8a+q\n/hOsNaWweSEvDzIzRVDXav/pldvby165VGmcPXsWHx+f8m6GVMHc6X2hKMoxVVWD7ve9suctPRhV\nhcREuHBBDI9bWICT0z9D3qXF1FQ8QPTs09NFL19RRI/c07NkPXxJkqRKTAZv6d50OkhIgPPnIStL\nDF27uJRPWzQasLISj5sjALGxYki9bl1wcCj9mwlJkqQKQAZv6c50Orh+XQTt7GywsSm/oH0nivJP\njzszEw4fFm309hbBXJIk6REmg7d0u6IiiI8XQTs3t+IF7Tu52RvPzoa//xbt9fISowSSJEmPIBm8\npX+kpcGpU2Je2c5OBO7KxMJCPNLTYf9+qFVLDKffshdVkiTpUSCDtySSqERGQkSE6MFWq1beLdKP\njY3odUdHQ0wM+PuLFeqSJEmPCFmYpKpLS4MDB0TwdnYWW74eBYoi5r6trCAkBE6evHPSGEmqwoYP\nH86ZM2fue9zHH3/MmjVrABg3bhze3t4EBATQq1cvUlNT7/v9LVu2vOfXT506xYsvvvhAbZYEGbyr\nspgYEbg1GhG4S2OfdnkzMRFz4NeuiaH0W9JBSlJVt2LFCho2bHjPYwoLC1m5ciUDBw4EoGPHjoSF\nhXHy5EkaNGjAnDlz7nudm5XI7sbf35/o6GiuXLny4I2v4h7Bv9bSfRUVwdmzorKRo6OYJ36U3eyF\nm5qKBW3nzompAkmqQrKysujWrRuNGjXCz8+P9evX065du+IKYVZWVkyaNIlGjRrRvHlz4uPjAfj9\n999p0qQJRkZilrVTp07FHzdv3pzo6GgAVq1aRc+ePWnXrh3169dnxowZxde2srICYPPmzTzxxBOo\nqkpcXBwNGjTg2rVrADz99NOsW7eubH4YjwA5513V5OaKIeSkJDG3XZX2RZuZiZ745ctiG1xgYOVb\nlCdVem/ueJPQa4YtCRroGsjHne9d8GTHjh24ubkV1/NOS0tj6dKlxV/PysqiefPmzJ49m/Hjx7N8\n+XImT57MgQMHaNq06R3PuXLlSvr161f8+eHDhwkLC8PCwoJmzZrRrVs3goL+SRbWq1cvNm7cyGef\nfcaOHTuYMWMGrjfWowQFBTF37lzGjx9f4p9DVSJ73lVJWhr89RdkZIih5KoUuG/SaERmOBBTBjd6\nDZL0qPP392fXrl2888477Nu3D9t/ZSY0MTGhe/fuADRt2pSoqCgA4uLicHZ2/s/5Zs+ejZGREYMG\nDSp+rmPHjjg6OmJubk7v3r3Zv3//f75v0aJFzJkzB1NTUwYMGFD8vIuLC7GxsYZ4qVWC7HlXFTEx\ncOKEWIX9qA+TPwhLSzGMfuKE2Frm5SWLnkhl4n495NLSoEEDQkJC+Pnnn5k8eTJPPPHEbV83NjZG\nuXFDf2t5T3Nzc3Jzc287dtWqVWzfvp3ffvut+HuA2z6+0+cA0dHRaDQa4uPj0el0aG6stcnNzcXc\n3Fz/F1pFyJ73o06nE3O8VWV++2EYGYmpgytX4OhRUQRFkh5RsbGxWFhYMHjwYMaNG0dISMgDfZ+P\njw8RERHFn+/YsYN58+axdetWLP7192TXrl0kJyeTk5PDjz/+SKtWrW77emFhIcOGDWPt2rX4+Piw\nYMGC4q9duHABPz8/PV5h1SKD96OssBCOHxfbwKpVE8FKup2iiJX2mZlw8KDI3y5Jj6BTp04RHBxM\nYGAgM2bMYPLkyQ/0fV26dGHv3r3Fn48ZM4aMjAw6duxIYGAgI0eOLP5acHAwffr0ISAggD59+tw2\n3w3w/vvv06ZNG1q3bs2CBQtYsWIFZ8+eBeCPP/6gW7duBnilVYMsCfqoKigQgTsl5Z85XuneMjPF\nzy04WC5kkwyqspcE7dWrF/PmzaN+/fp3PWbVqlUcPXqUxYsXP/T58/LyaNu2Lfv37y9eyV4V6FMS\nVPa8H0X5+WIYOC1NBu6HYWUl5sEPHhQ3PZIkATB37lzi4uJK7fxXrlxh7ty5VSpw60v2vB81eXki\ncGdnixKZ0sPLzRWL2IKCxJC6JOmpsve8pdIhe96ScLPHnZsrA7c+zMxEYZYjR0S9cEmSpApGBu9H\nRUEBHDsGOTki8Ej6MTERq/OPHxdJXSRJkioQGbwfBUVFYr9yRgbY25d3ax4dRkZi2DwsDMLDoQJP\nMUmSVLXI1QGVnU4ngkti4iM1P5ubp/DzAVuuJxvR2DubYN/s8kkIp9WKbHQXLoitd97eVTMznSRJ\nFYrseVdmqioSsMTGPlKB+9eDNtTs7s+SH5w5ds6CQZNr8/jLDbieXE73mhqN2Cd/6ZLsgUuV1rVr\n1+jfvz9169aladOmdO3alQsXLpToXPv27cPX15fAwEBiYmLo27fvHY+7tfBJWZo6dSq7d+++5zF7\n9uy5b7UzQ9izZ09x2llDkj3vyiw8HKKiRM/wEREVa8LgKZ5snn+R1oEiYYpOBxMXuzNgUm1+Wxpe\nPg27mcwlPFz0xuvWLZ92SI+GQ4fgAepgPzA7O2je/K5fVlWVXr168cILLxRX7jpx4gTx8fE0aNDg\noS/37bffMnHiRAYPHgzADz/8ULJ2l5KZM2fe95g9e/ZgZWV131rjtyosLKww29kM0vNWFGWloijX\nFUUJu8vXFUVRPlUUJUJRlJOKojQxxHWrtOhoEUicnR+pYdwvNjsxpFtyceAG0fGdPSqG85fNOHbW\nHFUVvfPBUzzp+no9pi6rTsx149Jv3M265+fOyUVskn5SU8V7yVCP+9wI/PHHHxgbG9+WDa1Ro0a0\nadMGVVUZN24cfn5++Pv7s379ekAEt3bt2tG3b1+8vb0ZNGgQqqqyYsUKNmzYwJQpUxg0aBBRUVHF\naU1zcnLo378/Pj4+9OrVi5ycnOLr7dy5kxYtWtCkSROeffZZMjMzAfD09GTatGk0adIEf39/zp07\nB0BmZiZDhw7F39+fgIAANm7ceM/z3OrFF18svqG40/mjoqJYtmwZCxcuJDAwkH379pGQkECfPn1o\n1qwZzZo148CBAwBMnz6d559/nlatWvH888/TvHlzTp8+XXytm6MLhw8fpkWLFjRu3JiWLVty/vz5\nh3tPPCRDDZuvAjrf4+tdgPo3HiOApfc4VrqfpCRR1tPJSQSUR0hYhDltGmfc9lxympbX59cgPsmY\n4Bd8cOzQiKEzatGqUSajnk0gOc2IJoN9+DusDPK2a7Xi5x4WJoq9SFIlEBYWdteynps2bSI0NJQT\nJ06we/duxo0bV5yQ5fjx43z88cecOXOGyMhIDhw4wPDhw+nRowcffvgh33777W3nWrp0KRYWFpw9\ne5YZM2Zw7NgxABITE5k1axa7d+8mJCSEoKCg2/KaOzk5ERISwquvvsr8+fMBeO+997C1teXUqVOc\nPHmSDh063Pc8d/Pv83t6ejJy5EjGjh1LaGgobdq04Y033mDs2LEcOXKEjRs3Mnz48OLvP3PmDLt3\n72bt2rX069ePDRs2AKLiWlxcHEFBQXh7e7Nv3z6OHz/OzJkzeffddx/iN/TwDNL/V1V1r6Ionvc4\npCewRhUZYQ4pimKnKEp1VVVLL2XPoyozU+zltrN7JHOVuzkXcD7KDEgDICtHQ/uRDWgZkEmDmrkM\n7prE55ucMTfVkZGlpXubRLq3SaN9UAZDptbm3MbTpT8QYWQkAnho6D/FTSSpktq/fz8DBgxAq9VS\nrVo12rZty5EjR7CxsSE4OBgPDw8AAgMDiYqKonXr1nc91969e3n99dcBCAgIICAgAIBDhw5x5syZ\n4kIl+fn5tGjRovj7evfuDYhSpJs2bQJg9+7dxUP8APb29mzfvv2e57mbO53/33bv3s2ZM2eKP09P\nTy/u1ffo0aO44tlzzz1Hp06dmDFjBhs2bCie709LS+OFF14gPDwcRVEoKCi4b7v0UVZ//d2Bq7d8\nHn3jORm8H0Z+PoSEiCQipqbl3ZoHUqQWEZEZzZnMSE5nXuRMxiVichNIK8wkvTCLtIJMVFRsjayw\nMbJE08yGNadrkxDqRHMXT8IOBeHumk37oAx2H7Yh5JwFk1+Ko+NjGQQO9GFknwRsrHT07pDKlGVu\n/B1mSXP/MiguYmQk9oGHhECzZjINrVSh+fr6lmhe2vSWvzO3lgl9WKqq0rFjR9auXXvP69zvGvc7\nz908yPl1Oh2HDh3CzMzsP1+ztLQs/tjd3R1HR0dOnjzJ+vXrWbZsGQBTpkyhffv2bN68maioKNq1\na/dQbXxYFW7MVVGUEYqiHFUU5WhCQkJ5N6fiUFU4fVqkP7WyKu/W3FNsbgJfXd1Kmx3TMNvUGe8/\n+9D72DimnF/G/qSTKEAtc1da2QcwwK0TA92eoo1DILUt3LCzAvPaR/jo6uc8GzKBGSZPsvPxegw5\n/T/6TZhHbH4ctVzzqVU9n2YNs/kzxBoQ0/6e1fNJTC3D0QhjY7C1FZnYDLn4SJIMrEOHDuTl5fHF\nF18UP3fy5En27dtHmzZtWL9+PUVFRSQkJLB3716Cg4NLdJ3HH3+c7777DhBD9SdPngSgefPmHDhw\noLi0aFZW1n1Xunfs2JHPPvus+POUlJQSnedurK2tycj4Z4quU6dOLFq0qPjz0NDQu35vv379mDdv\nHmlpacWjC2lpabi7uwOiSEtpK6u/dDFAjVs+97jx3H+oqvoF8AWI3Oal37RKIjISrl0rl5Xlx8+Z\ns36XA1k5Gto1zaBH21SM//XOSS/IZF3sTr68uoXDqWIxhzbLlba27env7YtLoTfbNrbg7xBXtq08\nh7Wl7p7X/Ps8LPotiZ/Do3H030OO7y5mJ/wIT0xjcIwfky91JkMdw+Y/7Nj4ux3O9oX8ddKSZe+W\n8UIyU1NRgezoUWjZUtZLlyokRVHYvHkzb775Jh988AFmZmZ4enry8ccf07p1aw4ePEijRo1QFIV5\n8+bh6upavHDsYbz66qsMHToUHx8ffHx8iufZnZ2dWbVqFQMGDCAvLw+AWbNm3XOl++TJkxk9ejR+\nfn5otVqmTZtG7969H/o8d/P000/Tt29ftmzZwqJFi/j0008ZPXo0AQEBFBYW8vjjjxf3qv+tb9++\nvPHGG0yZMqX4ufHjx/PCCy8wa9asMiltarDCJDfmvLerqvqfauqKonQDxgBdgceAT1VVve+tnSxM\nckNCAhw+LFaVarVldllVhbELPPjhN3uaeGVzItyc2ARjTIxVPng9htHPJnAs7QxLLn/P+thdZBfl\n4mddl37VurJgxgv8OdMU//p5t52v7/g6tG2awev9H2xUZflmJzb9YcdPH4dzPiuKb8L/Zt7x3yms\nFgKFpnimdqePdX9+XteNyGhTFo2/ysu9EkvrR3J36ekipWpwsOiRS9It/lOAooy3ikkVkz6FSQzS\n81YUZS3QDnBSFCUamAYYA6iqugz4GRG4I4BsYKghrlslZGWJ/Nr29mUauAHW77RnzzFrurVO5dhZ\nSz5/9wqNvbIZ9UENJm66ykfqVKIsf8dKa8Eg9y68VKMnwXa+/HnMmu2m7vjXOQOJKSL7W2IiSkYG\nU12N+XGTK9RIA3Nz0VO1sBBzxnfotQ7qksTSH5x55f1aTBpmynuBtbEKfZ13lyVi1nwFMb7f8pHx\nRlyeac1S/5cY/9Yg2jXNoH7NvDu8olJkYyN2AYSFQaNGj9wuAMnAZKCV9CRLglZkBQXw998iLae1\ndZlfvsPI+vR4PI0PVrtyfmMYNlY6jqWeZeKZZexKPoCS48Cbns8zPbAXNnmI1denTpH490Xyz1/C\nreiqyLDyoBwdRfKT4GBo0QIaNABFITVDy/QvqrPmJ0dSM7TUcs3H3FTHZxOuEJtSwDmnNSxPXEl8\nXhI1clrSNm0cXw+vcf/rlYb4eKhXT7Rdkm6QJUGlOyn3nrdUSs6eFXW5HR3L5fJXrpkQGWPCoC5J\nFJqkMPLkZ3xxZTMOxjZUOz6DHukd6XzwN2xWvSMqmhUWglaLQ01PNtOCdr2fwrGuvehVOzmBjQ1v\nfFybwFrJDH0yWry27GwxunD9Oly9KhKgLF4sHo6O0KIFdi1a8PFLj7HwLTt0Oli20Zmwi+a0D7qZ\nnKEfE4t68sXlTUw//TXfmPdCOd6VmXXeZP+Bepy/bEZN13z6dUzGxuohbiZK4mYWNktLuLF4RZIk\nydBk8K6orl0TWdTKMfWpV608LsUaozRZhdee90gpyOAN9z6M+9uB69t+JFA3TRxYsyYMGACtWoGv\nLxpzc2LXOdNygwtLn7hC+6AMElON+HBNNX65bMeMmefA+h5lSxMTxZzgX3/Bvn2wfTsoCkrDhmhb\ntqS5fQc+P/40qvpPcjkLrRlv1hlIyDdvEO39IWtjlvFN5AG8IyfxrG1/dh6yYeJid9a9H8mTj2Xc\n/dr60mjEjcrJk2IaQFZ5kySpFMhh84ooJwf27xdbwkxMSvVSGVkaFn5Xje92OJCRraFVo0zGPR9P\nM99sVu3N4JWz08h330tLa1+WhNcn4OtdKFlZhNs3Y7Pakyf/14Qmne98g/HNzw7MXeXKxRhTtBqV\nZ59MYc7oGFydHmKvaFGRGIE4eFA8wsJAp+OqcW0uNh9E25lPoliLrXM7D1kzeEptQr45S9BbCtUG\nvcpJ3QGa2Tbkm8bvER/uS6//1eXcxjCc7IoM8eO7u9xcMarQqpVcgS7JYXPpjvQZNpfBu6JRVTEE\nnZYmVpCWoqwcDe1eaUA9jzzeHhyPi0MB3/zswHtfVkfXcB35T41Goymk9y+tWRm6D0s1hx1mvdhU\nZyzpbg1Jz9Lyy6cR98xopqqQnavB1FhnmIRwaWlw4AC567dgdvoY6Rpb/qr/Al+Yvsah2Jqsn3OJ\n+CQjlvzgzG9LL7AhbhejTn1Ari6Pjxu+zf6v3iCwfi5jB103QGPuIyNDrDyXK9CrPBm8pTvRJ3jL\nJbEVzZUrYv7XgIFbp4MDoZb8tN+G2IR/gsjyzU54uOTz3exLBDXMxsJMx/JfjHF5qT/5PQfjm2HK\nyaWFrDu+i5+UnvhxikHG69l4tTnVHArZ9OHF+6YiVRSwNDdQ4AaRFKVrV8xWf466ajW5gc156vwi\nvj/dkKuPD6ZNrStcjDYlyCcbRVHo59aJk4+vpYV9ACNOzeZ44GBOxf23kEGpsLYWve+TJx9u4Z4k\n3WCot01SUhKBgYEEBgbi6uqKu7t78ef5+fkPdI6hQ4fqVWzDw8OD1Htsj9PpdMydO7fE569qZPCu\nSDIy4MwZgy5Q2xtiRYPevoycU4tF613wfa4hw9+rRV6+wve77RnVN6E4AE/dmELqc0FEO//A5D+0\nHFmaRJ3gzhyf/RPvVv+KBZ8aodWobFsQwaLxV7EwK99RG8XPF5cvZqNs3oS2R3e0WzbBM8/Q9fwC\nbklRjLu5CzsfW8x8nzc5rfmDTQ06cDT1zN1PbEgODmIFeng5lTKVKp3MTHjnHbH20cgI/P1h1Sr9\nysg7OjoSGhpKaGjobQU5QkNDMbkxNaeqKrp73C189dVXeHl5lbwR9yGD98ORwbuiKCr6Z5GTgbqp\nkdEm9H2nDp/+7yon151hx6IILm87RVKaltc/rEFuvlKc6WxL9G5W2nfGtCiaPWu0eOx9iT/H/4b5\n7Ck07eSMkValulMRk1+6xtKNzgZpn8HUqAGTJsH69RAcjP+uhSw/3pSQ+b8X/8XTKBpaZr2C1dr9\nWJkptNr/Mi0WHOHpsXWZt7oaSamluIfe2RkiImQVMum+8vOhc2exVvXgQbGB49NP4aOPYPZsw18v\nIiKChg0bMmjQIHx9fYmLi2PEiBEEBQXh6+t7W13s1q1bExoaSmFhIXZ2dkyYMIFGjRrRokULrl//\n7zRUQkICHTt2xNfXl1deeYVbp2iffvppmjZtiq+vLytWrABgwoQJZGRkEBgYyJAhQ+56nCTI4F1R\nREaKnrcB85Yv3uDCsB5JdG2dXty7trHSsWp6FBt229PcL4sNu215b/9sngmdgH9CPjuPPknekD8Y\nrSyjbfd/8oZrNSo6HXRpmcbfYZb3uGo58vSE+fNhxQqsazvRZN14wp58g6Uf5zFwUm26j63Ht6Ps\nabp3D8bxzTjU4FW0ncdxKtKERgMbcuFyKRV7uXUFenp66VxDeiRs2iTeLl9/LdIFaDTQvj3s3CkC\neFKS4a957tw5xo4dy5kzZ3B3d2fu3LkcPXqUEydOsGvXrtsqbd2UlpZG27ZtOXHiBC1atGDlypX/\nOWbatGm0b9+e06dP07VrV2JjY4u/tnr1ao4dO8aRI0dYsGABKSkpzJ07F2tra0JDQ1mzZs1dj5ME\nGbwrgpQUuHDB4Pu5j5yxoEvLtP88b2ulo6lPNsENr/FnYTumpm7m+fNmvHz5S943Wc+YVUGYmeow\nMhJ3ykfPWJCVo8W/Xg7R101wsCnlldr6CgzEau0K8t6ZSv3cUwxb14nhLCdi0ymuJxuTGOvGtUHz\nGV3rWbYUrCCpay/GvhDJiNm1Sq9NRkZi7/fx46J7JUl3sG0bvPjifxP0Va8ObdvCrl2Gv2bdunUJ\nCvpnfdTatWtp0qQJTZo04ezZs3cM3ubm5nTp0gUQZTajoqL+c8zevXsZPHgwAD179sT6lkRTCxcu\nLO61R0dHc/HixTu27UGPq4rkPu/yVlAgMpPZ2ho8paaDTRFX4/+71UxVIft6JF/FtyHEJ5MX/2rG\n7r9+4Q8LK+ISTejcIg0nuyK+2OREHfd8Rs6pyfujY1AUmLemGs93LYXbf0PTaDB9tge0CYYZM+jw\n6wTI/omtqV8xYeg1rMyMWOz/DgE29Xn11FwS3fpy6douomJN8HQrpeBqaQnJyWJdQ6NGlH7hcamy\nuTV3wb9pNPrNe9/NreUuw8PD+eSTTzh8+DB2dnYMHjyY3Nzc/3yPyS1bWB+2VOju3bvZu3cvhw4d\nwtzcnNatW9/xGg96XFUle97l7fx5EcBvFHo3pOe7JrHwOxdy8/75a5CYqmXujNNkPtWMAy6ZfG3S\nn6/eX8qxTdHs+eJCcRrUo2csGDOvJkOmefJi9ySMjVTavSJSfo7oXQ6FP0rK1RU++wzeegv+/psv\nT7cgIObn4i+PqNWbTUHzOJURQUb/dpxKiC/d9jg4QGys2FUgSf/StSusWfPfIH39Ovz+OzzxROle\nPz09HWtra2xsbIiLi+PXX38t8bluLQ+6bdu24vKbaWlpODg4YG5uzunTpzly5AgARjfW+ty8Ebjb\ncZIgg3d5io+Hy5fFH/RS0LtDKg1r59JiqDdfbXWg/8TajO2/lRV1XiDCsYj6Py4mN3sOAC4OhdT1\nyKduDbF1LGPfcS7+GMboZxPYF2rF1z87MKJXIts/jsDUpOLmBrgjjQYGDoRvviHDypVaH70BM2aI\nZb1AT9d2rPVeSp5JPCPjBxGWHlG67XF0FLXZZQ1w6V+efVZkCx4+XNzfqapINNi5M7z2WuknXGzS\npAkNGzbE29ubIUOG0KpVqxKfa8aMGezevRs/Pz+2b9+Om5sbAN26dSM7O5uGDRsyefJkHnvsseLv\neemllwgICGDIkCH3PE6SSVrKT26uSP1paSlqQpcSnQ62/GnHzCUO9Mt8hUXPf0+umRE/B3+CbeHj\nPDWmPp+ReiQgAAAgAElEQVS/e4XOLavGQqpjJ43YO3odb+bOQ6nhAR9+SIZrffq/W4c8+zAO+Pel\ngHzahf3IxC7VeCL4n1SqCSlGfPOzA1eumeBVK5eBnfXIlZ6TI0ZcWrYs1d+/VDE8TJKWtDSYOlX0\nwHNyRPbht96CV16RMy2PGpmkpbJRVdHz0mhK/Q+3RgNPe4czJbM9i57/nnxLU/7ssIbHPB7D2zOP\nuWNimP9NtVJtQ0XSNKCQWjOG09NqN0mxueQMGMYbnc9xPcWIyONBTNWuw8nShENNetLno3Q8n/bD\nvYs/fs/5UKenHyfDzfGols9vR2yo39uPQ6dKuPLe3FxsD7yR7lWSbrK1hU8+ESvLU1LEzNrIkTJw\nS7eTwbs8xMeLRymnPwXgxAkiXh/Ia8+fJN/Gkj/ar8bP7p9ylR2bpxNyrmrl3u7dIZXvf7XlxNSN\nZHk0YGXOQAZcnMVfy88w8RkrDrRZhlJoTuaznblucoblk6K4fM0UJ7sCalbP5+3B1/n+g0i+mhpF\n73F1bltT8FDs7cVkppz/lu5AoxH3eDJoS3cig3dZy88Xve6yCNy7dhExYQRP9knnupEdW4JW4mdT\n77ZDomJNcbZ7iEIhjwhTE5UOXc1xWr+E4w0H8lbeXFwXvgP5+WRE18dy/S6q2WhRn3+SuTuu079T\nModXn+ezDS5cjhMrbbu2Tsevbi5b/tTjd+noKFafy/lvSZIeggzeZS0iQgyXlvY856ZNRM+ZyBND\nNeTZW9H+3BZ+2dzmtkOKimD2Slde6F4Jtn6VFmNjNjT7kN3NJ4tNtGPHsnWHCcPa2PNnq88xxoSD\ngb1wrX8GZ/tC+nRIYdPv/wTrRg2yiYrVo/KbVityoJ84IebApUdWRV5fJJU9fd8PMniXpdRUiIoq\ntdXlxdauJXnh+zw1wpwUa2N+bf4ZX420Zd1Oe/qOr8PG3+z4+icH2r3SgLRMLWMHlvL2qAquRUA2\nkzImoE6bDkeP8uLPz+GiSaKuRQ3cf/kFIyOVhUZDiMtNJK9A4eufHfDu48tjL3iz+Q87/feFW1hA\nXp5I1CM9kszMzEhKSpIBXAJE4E5KSsLMzKzE55CrzctKUZFIVqyqpVvfedUqsj9fzJNjbDhml8OO\nxxbR3kksXEzP1LDmJ0d2HrLBxFilT4cU+j6ZgnEVT9VTVARBz/vQuWUa073XYTR5Apc1dfi663f8\nHN6AWe/9SKejI6muqcG1jw7wavcCRvVN4NtfHFi4thot/DP55VM9t9Cpqpj/btoUqlWdBYRVRUFB\nAdHR0TLJiFTMzMwMDw8PjP9VLljW865ooqLg3DlRpMJAdDrY9bcNP+23RVFUXsueR63tC3jmNSd2\n2CfzfdO59K7ewWDXe5RdTzZi+Hu1OHjKkmftf2POpX5kam3JX7iY2i3dmfJrGLNyX0Yb9xhjdasJ\nOeVE+BUzti0M5+2Pa9CvYwov99IzeU1+vth73rp1qSTtkSSp4pPBuyLJyhJ7uu3tDVYxLDtXoedb\n9bieYsTgLsk0CVlOh/0z6dKvFr/6XOZz/3cZUau3Qa5VlVy9ZkxUnCluSSdxmjqagnyVl1y28Htm\nMDqf78npPgivvE5Ms1lE7/YZmBirbNljy6INLuxeYoCyn6mpYq9Q06ZymbEkVUFyn3dFoapw9iwY\nGxsscAOMmluTiKumaDUq2p+38sT+mcx+oQ6/+lymXcbbMnCXUA3XAto0zqTuk3WwXfc5Dq4mbE7v\nyK6ha2lc+DSf+L3NObNfOeoxExNjcePraFdIZraB/ivZ2cntY5Ik3VcVn+0sA9evi4cB5zF/2G3H\nmp8cGff8NYbbb6TuojeZ3diPKbXD6GbdjdCls2FAmMGuV2XVrInmqy9h9Gge++IlahpZ0cfqeS54\nXuGjyG+ICPWnUcZgzkeZ0SYw03DXvbl9zMFBrESXJEn6F9nzLk2FheKPsK2twU6Zm6cwck5NTIxU\nPmi5lfpL3+Zoq1rMeuYCZvHNedVkDrHXTWTSLkNxdobly1EaNGBNXn/mjrpM+g+fYBLVmW3Wk9gZ\nf4QffrPn6FkLsnMNNMxtZCQWNZ48Kd5DkiRJ/yKDd2m6fFksQtJjO8C/7Txkg3+9HNpbH6Hozbe5\n4uVKj84pVDd3ZrLlUpaur0GDmnmGri5atdnawqefoq1Ti3mX+xP580U8D6xCSW7AEb8XWffFT1Rz\nKGTiYnfDXdPKCjIyRF4ASZKkf5F/4ktLVpbYt2tvb9DTpmdpaWh5me/zenBZY8fTg4zJUfPZ3mwh\n1c1tOXjSitf6XTfoNSXA1hbls8XEG7nxu0kXtr64nz/afISjlYaJyaOYMuYMX//sWPJUqXfi6AgX\nL4oa4JIkSbeQwbu0nD8PJiYig5YBtagTz+gDA7Egk2debMjJnEhaXljBwk/aMWpuLXzr5DDq2QSD\nXlO6wdGR7ia/orGxxOvD4bRRMtkUNI+onFjeiJxEkU7HB6urcS7KQNnzNBrR65fZ1yRJ+hcZvEtD\ncjLExRk+f3lhIXWXvI2X7iw9OvfjlMsu3nQdzVNOLUnP1GJnU8iWBRflDqNSZOJRjUMjV4ubstGj\naZntTIfEmfyWtpfM4NlcjTeh3StevDDNkwJDTFebm4upl3ADbEOTJOmRIYO3oel0YmuYjY3hz71k\nCfz1F3+MH8jPfmswiujBkc9nMf8bV65eN2HP5xewtyky/HWlYqP6JjD2h1ZkfrgEcnPJHPoaBdue\no+a1Aegen0nPEd8QueUU15KMmLzUjS9/dGT2l65s/dO25GvPHB1Fkh85fC5J0g0yeBtafDykpxs+\nBeru3bBmDTH9ujLI9ifqW7lzbuB43h91jd+WXOCvlefx9swz7DWl/xjWM4lg3ywajH+GT1quQ0lN\nZunVHgQem0agtTeDj08hpvAyvdql8uEaV37ab0tmjoa5q13xfc6Xi9ElKGKiKHL4XJKk28gMa4ZU\nWAh794qhThM9Kk39W0QEDB1KQYN6tB2qciojksOtV+NjXdtw15AeSliEGZv32PHnF+fZqXRGadKY\nK/Mm0vSvF3HUOpH4YQhFeRZc2HQaFwfR5V683pkvNjtxYu3Zkk1tJCWBhwc0bGjYFyNJUoUhM6yV\nhytXxPykIQN3RgZFb49HZ27JpFFeHEwN41XjOYSHBJKWKX995cWvXi5Thl8jukZLIl+ciXLkMLUW\nf8N3jWdxIScSpwGjALC1+mcaY/RzCaiqwt4Qq5Jd1MEBLl0SQVySpCpN/vU3lNxcsajIgFvD9h2z\n4EDPD9DFxBJU/W0+jPsBo5CRhG5+kcUbXPB82p8P18gKVOXp5V4JjDw1isLBL8DGjXT64wo+MW9y\nwelbgvp9cVulMUWBYN8sLlwp4b5/RRGLIOXwuSRVeTJ4G8rFi2Jrj4Hyl/8dZsHBN9bSKn0HMeOG\nc6HvBxgnNMJm/zwmDo1j52fhhH53lq+2ObJmeynXB5fu6o3+17G3LsJ/7xIu1O6Ebv5H+H/fDpPY\n1hzy/B/nM6NuO/5khDme1fVYm2BmJqZn5OpzSarSZPA2hPR0MWRuwF73T/POMD73PQqe7spgj4Pk\nFBaywnseS/6XwPQv3ACoVT2fz965wodfuxrsugZTUCDKW6amQmIiJCTc+5GYKH6OubmimEslYWQE\nG+ZGsmzSVZY3WUqMjQ9rcgfjt20qxpjy3LGJ5BSJGs5rd9iTkq6lQ7MM/S7q4CBXn0tSFScXrOlL\nVeHoUZFRrQRFJKJiTdjypx0FhQpPtUjDv14uceGZMKA/LrWtmDKtBXOivkbZ+C1FK70oLALrxxuT\n8nso5mYqqgomzZuQtf94cZWrcpGbK4K1TieGd83NxXY5S0ux8t7ERFRWu5m3VaeDoiLxr04n1gpk\nZIgAnp4unjM2Ft9vyDUEpe3aNRgyhHTVGi+P97nWZxAt8wbjdGARR89a8MunEQTUz9H/Ojk5ogfe\nurVBq9VJklS+HnTBmvxfr6+kJNFzfMiqYaoKExa58+UWJ3p3SMHMRKXDyAboiuCrnP50Jok29lM4\nFDWGl2o8ww+XnuNq/BkcbcUCqJsxMOa6MZbmRRgblXHgVlVxw5KTIz62sQEfH7GlydJSv4BSWCgC\neGKiSHaTmiqet7AQj4qcuN3VFebPx2bkSKJYyFNZL/On5XJGt23Dd7ObYWluoIox5ubi5xMRAd7e\nhjmnJEmVhgze+tDpRNWwEiRkWb3dkZ2HbLiwKQx7myK+/smB1dsdGGW5ih4Zm1juO44TnWejSalH\nF9vpWHRLZtaX1fGvm8NTzdMxNRG97vdWVOf5rslll1WtsFAEU1UVFbcaNBDTBQYsvoKRkRgadnAQ\n58/JEdeMjxflVXU6cYy1dcXsdQYEwJQpmE6Zws56brRo4826nPEMurYJZyMX6nrkGeb35eAAkZHi\nhsHQ2fwkSarQ5LC5PqKjRdnGEtTqbjrYmzljYqjpms+gybU5F2VGC5cINl0JJpQAZnxgye+Zv/HU\n6V+IO9WStbMv8eSo+iSlGTHr1RhquhawersjcYnG/L6sDDKrFRSIAKrVQv364OZWPsPZOp0YXr92\nTVRt0+lEb78iDq0vWQIrV7K93wh61P8ak9hWOP+0DUszmDMmhl7tU/W/RlaWGIlo0cLgefQlSSp7\ncti8tBUWiuIjJVykduGKGT61c2g5zIcpL8Uxdr47OxwGo00opL9PD+JyxvO2x+uknAjk13BzWg/3\nonOLNBrWyWV/qJhb790hhYGdk7EwK8UbsPx8EbSNjcWwuJub+Li83CzWYWsLdeuKIB4eLtpobS2G\nkyuKkSNJOH6VLutX8M7Y/syt+R1vL5pMo+SXGTS5NmYmOrq0StfvGpaWYjTi8mWoU8cw7ZYkqcKT\nwbukoqNFb7SEPb6arvl8/F01HvPNYkTvRC5/tAGj0KMcf+sN4sxmYhrbmr6eLxMw/gprf3Xg6k+n\nDDdf+iDy8iAtDUxNwd9fDM1WtCFqIyORcczNTaw7CA8XgczWVrS7vGk0vKis5uvqHXn/ix85PaMp\nE84t4kjrZiyZoGXG8ur6B28Quc/PnwcXF1EHXJKkR14FXvlTgd2s8qTH1rARvRL59hdHnghOh4gI\nphdM5lTNTjyl/IWJVkOftE84dsaGr7Y68URwetkF7rw8EQDz8sTcbdu2IkBWtMB9K41GTF20bAmN\nG4u2JyRQ8koghqGq8OsJVyyWfIhiZs6K5fHYGVkx6PhkOrZM4Ph5C8PU/9ZqxZqD06cr1TY7SZJK\nTgbvkrh69Z9FUyU05rnr2NsUMuMzR66NnkmuiQ2taz5GgvURpri9S9zFeuw5asXMFdWZMzrGgI2/\nC1UVK+dzciAwEB5/HNzdK9c8qkYjRgjatBFD/GlpYi90OQY0MxMd6ZZuMHs2LuGxfHmiFqcyIph6\nbjmKAlqtgdpmYyN+f7GxhjmfJEkVmgzeDysvT2RT03N1r1YL3826xMT8mbgmnWFBu7fJefID6mR0\nYfHkN9lzzBpLCx37V5zHr16ugRp/F7m5orft5iYCX/XqlSto/5uREdSqJW5AXF3FKvW8sq+4pijw\n7JMpLFrvAkFB8OqrdFt/nKGF/iy8vJrW3XZjbMgBDQcHsfsht5TfL5IklTsZvB/WlSviXwMMIzfO\nPchrefP51vJFVjZbi4ViSe+sOeh0Gr6fG8mq6ZepX7OUg05yspgGeOwx8POrmKu2S8rMTMzXN2sm\nVmWnpJR5E2a8Esuq7Y6M/ciDU61eIaNxGz786CymGc5cCn6lOPuaQRgbizuGCxcMd05JkiokGbwf\nRm6uQXrdgMhGNnUqGnc3fp9swRXtSdyPfEJ2UnX+WHaBPk8YYBvRvRQViR6pk5PI0uXoWLrXK08u\nLmJEwcFBvOYyLOpR07WAv1edQ6tV6f1OfdrFrwPFlR/3FBKZF8Wkc0uKjy0oFNP0cYlG/HHUinNR\nJVh0Z2cnpnVk5TFJeqTJfd4P49w50fM2RKCbORO2byd08WSaJcymb/UnWNvkff3P+yDy80WP29tb\nbC8qswwv5UxVxZxwWJjopdralk87wsJg+HBGvejMMo9rLHVdzebVPdj9tw06FYy0Kv71cohPNqaW\naz6rpkc93AhMbq74HbdpU7EXGkqS9B+ynreh5eSIYhCGKD6yZw9s3UrBC8/zYu46nEzsWOw3Xv/z\nPojMTJF6tFkzsU+6qgRuEK/V3V0ENSsrMc9fVMrJbe7Ezw/GjmXemjhq5Nsw6tx0ej4ZS5eWaQx4\nKpkPXovmarwJWxdEMOCpZJ54tQHpD1O73cxMBO9Ll0rvNUiSVK4MErwVRemsKMp5RVEiFEWZcIev\nt1MUJU1RlNAbj6mGuG6ZunRJLOLSN692ZibMnQsNGjDvCTNOpF9gmf9EHE3KIL1lUpJof6tWYii5\nqrKwEAvIGjQQW8rKYTEbzz2HVduOfPVtBjq7SA45fUDIeQtWTY9i7KAEJr8Ux+wvqzOmXwKP+WWx\n5qeHHO1xcBB5zzP0rGAmSVKFpHfwVhRFC3wGdAEaAgMURWl4h0P3qaoaeOMxU9/rlqnsbJHByhBz\n3cuWoSYlsaX/CKZHrKSzTWd6urbT/7z3cnN+29kZmjeXiTxA3MTUrQvBwWIxW1kHOUUhb9wkal+u\nxSunzfk6+WuadPijePX5C92T2L7fFlWF7m1SORRm+XDn12hEtrkzZ+Teb0l6BBmi5x0MRKiqGqmq\naj6wDuhpgPNWHJcu3V7OsqTOnkVdv4HvbIfTP/F7NAXWhH6ynCderU98UinNTebni96ll5fYv12e\nqU0rImdnkdzFyKjs62NbWTNY+x0fbM/DNceEPTXGkleUD4CqKsUzGtcSjbG1KsHwvrW1eE1y77ck\nPXIMEbzdgau3fB5947l/a6koyklFUX5RFMXXANctG5mZYpGavr3uoiKKZs/huuLCL6/ak+vyNyub\njeXq+nia+2XR8+26hu8g5eaKnN9VcX77YVhaiq1yDg7iRqeMmJqoWDXzISL4TZZvziXT6ixv/P0N\nAKu2OfJ0mzQysjR8vtmZQZ1LeGNhby963+UxNSBJUqkpqwVrIUBNVVUDgEXAj3c7UFGUEYqiHFUU\n5WhCGf4hvavISLH3Wd/At3Ej2nNn+KTp62y2+JSuLq0Y6N4ZIyOYNSqWtEwte0MMOJydnS2Gg1u0\nqNrz2w/K2FiMTLi5iSmGMhpqnvVqLF1DZ9MkO4j+p4z4POELrPtqeOdTd9yc82k5zJun26TSIiCr\nZBcwNhavJTzcsA2XJKlcGSJ4xwA1bvnc48ZzxVRVTVdVNfPGxz8DxoqiON3pZKqqfqGqapCqqkHO\nzs4GaJ4eMjIgJkavLUVHz1gwa66WnIVLOWrbjk2df0KraFnmPxHlxg2BokCn5umEnLMwTLuzskSv\nu3lzWef5YWi1YiV47doigOtKP598M99stn1yiV453/D+L6bY5SrkdX4ZW9t8lm10pk+HFD5+O1q/\ne0cHB7FmoxyS1EiSVDoMEbyPAPUVRamtKIoJ0B/YeusBiqK4KjcilaIowTeuW/GzSOjR69bpYMTs\nmvR9pw49Q6ZhrMtnoM9TnNceYKrn69Qwd73t+KhYU5zsDFBIIzNTzHM3by7yXUsPR6MR+9+9vMRW\nsjIobhKXaEyeUw3cxr7D0l8KKHA9xpRlU9i9JJwvtzrpv5tNUcR74fTp8tkaJ0mSwekdvFVVLQTG\nAL8CZ4ENqqqeVhRlpKIoI28c1hcIUxTlBPAp0F+tyNlhQATB2NgS97q/2urIiQsWnHt7Bf6RW0l7\nuT+J3eahxDTn0Fdv3XZsyDlz9p+w4pl2emZVy8oSwaZ5c7FYSSoZRYF69UQvPDGx1DOyfferA2Oe\nu47pM13oV/1JnoqAd88sprb3RdydC9h73AC/SwsLMZIUHa3/uSRJKncGWeJ8Yyj85389t+yWjxcD\niw1xrTKj51z3F5udeW/YJcwWzoWaNRnXJJGMuHReNXmPpb878t6KPJp6Z/PXSSs+3+TE8kmXsbbU\nY5g2J0csSmrRQm4FM5RatURd8JAQMf1QSjXC07O0VHMsBEVBmfguS0eE4uuZxGsn51LNoSVpmQYq\nEuPgILIEuriIbWSSJFVaMsPanWRl6T3XfeWaCY+dXA7R0ex58xm+iv2J/9V5ns+G2lPdqYCT4eYs\n2uBCVo6G/SvO07uDHr3u3FzR5uBgGbgNzdVV/FzT00utWlcL/yy277vxXrOxofb/ZjH9D5UfE/bx\ne+ZvBPuKxWo6nZ7r6IyMxEMWLpGkSk8G7zuJivqnQlMJtaweieXaFeR2aMMrRT9Sx8KdKQ2GE5tg\nTE6ehlXTL/PLpxEsfDsaL089tvHk5/+T7rS8cnU/6pycxFREZmapBPARvRL48U87Vm1zFFPSQUG8\n4jGEgGtg3Ollzsbm0/m1epi0aIJ5q8b0n1ib8yUpWgLiPRIdLQuXSFIlJ4P3v2Vni33degbCBUbj\nKSooYmoPDy5kXWGp/0SMVTPe/MiD57smYWlugJXMhYViBXHTpmJIVCo9dnZiL3hGhsH3TLs6FfLr\nonAWb3CmTk8/Wg3zot72pczcU5c0sxSe27aUfh1TyPjzOPE7T9DUJ5u2r3gRfqUEAVxRxHtbLl6T\npEpNVhX7t3PnRElFfYLhiRPw0kt82/p5BrdbT5307jydsITNf9jR2CuHtbMjMTfT8+eu04nV0IGB\notiGVDaSkuDwYRHMS6H2+ZlIM5LSjGhYOwfHxPOMWjOQz5vpONzma5ra+RQf994KVyJjTPlq2uWS\nXej6dfDxAU9PwzRckiSDkFXFSiI3VwyZl2BvdGa2hs83OjFoUi0u/e8zcuxcWPXsWaxNTRhmOhF3\n5wI2fRjJjx9d1D9wq6rIBObtLQN3WXN0FCMdKSmlso2sYZ1c2jTOxNGuiGRnL6z2vItzFnT5fhpN\nh9Rn2Q9i69jQp5PYtlePPfw3F6/l5Biu8ZIklRkZvG919arY5/uQOcwvx5nQaEBDdhy04WXrddRO\nCaG/1zPsTjnKbK9XmdRPYdyQeJr6ZBumnYmJULOmqMUtlT0XFzHikZhYqvvAl3zvzMLsqUw/XJOE\n6pG0HziLb3c4MGSaJ4oCet0C3ly8dv68oZorSVIZksH7prw8sT2sBL3ul2fV4uVeCWyec552B+eS\n7lOHwz03YZHSCNuzIwzbztRU0Wvy8ZG5ysuTmxs0aiQCeCnMHSemavno22oEeOXjGfQ57aMUVuYs\n5NsFhzh21pIZX1Sne+s0/S5iZyd2VcjFa5JU6cjgfVN0tAiG2ofbU3v1mjEh5ywYO/A6bNwIMTFM\nH1KT+PwkplWfyqptBswrnp0tRgUaNXrodkqlwMMDfH3FFIaBU6lu22tHp+bpzB8bzZCvnmB06nAy\nNYVM2jOB+jVz+eYXR94dGqffReTiNUmqtGTwBpFB6+LFEq0wj082pqZrPqZ56bB8OSee8OPT/H28\nUqs33Wp7cS3JQCU4CwvFSucmTUotWYhUAp6eIpVqQoJBi5nk5itYW+hoH5TJxnmRLIv+gAF/ufON\nJoRrBTvo3CJNvy2GN5mbixwBV6/e/1hJkioMGbxBpEHV6cQc4EOq65HHpVgTsj9fgy49jVEd83Aw\ntuF9r9H8GWKNX10DLAhSVTG02aiR3MtdEdWtK3rhBhx+7hCUwfb9tuTmKbRpnMmuZZF8Mnw+tVIh\nOehlnukQb7Br4eAg5r7l4jVJqjRk8NbpICKixNW37G2KGNXmFNoNa1kzMIC/csP5wOc1rsW4MHul\nK2/0v65/GxMTRaUrubK8YlIUaNhQFP9I03Me+gYvzzyeDE7nuQl1iI4XozdKdS8GXhhApFMmiYVv\nG+Q6gMy8JkmVkAzeqiqGzY1LPrw9g2mkm+p42S0Wp8wgfvjoDVq95MX7o2NoFVjCOsw3paaCvT00\naKDfeaTSZWQEjRuLQJ5tmF0FK6dexsszl0YDG1LvGV9qdfcnqvBjnoqzZkbBz1yLM2CNbltbsXgt\nOdlw55QkqdTIJC1FRbBrF5S0dviFCzBoEGPf8OITu/NMKtyCn4UvXVul6VdoBMS+89xcaNUKzMz0\nO5dUNtLT4a+/RDA0UBKX7FyFmOsmONsXYmddRPi5v/A79zoDkqqz6qVtBrmGuNCNm46WLeWCSEkq\nJzJJS1lZtIgwTwsW2V1gRM1evPeMG/06pegfuHU6MQTbpIkM3JWJjY34nSUnG2wFt4WZSv2aedhZ\ni/PV927J23mNWV0tjr92f2WQa4gLWYj87TExhjunJEmlQgZvfRw6hHrwIGMG2WNrbM1s71GGO3dS\nEtSvL4bMpcrFxUXswzfwCvRbTeq5AI8sI8ZcWUZRup514G91M/NaKVVQkyTJMGTwLimdDj79lPVt\n7PnTKJr3vUfhaKJHuspbZWaCtbXMoFaZ1a4t6oEnJpbK6S3NrPmozkiOuxTx2eq3CTlnTnzSw++W\n+A8jIzFvHxGh/7kkSSo1MniX1M6dZEZd4H9PFNHE1pvhNZ8xzHkLC8XcY0CAnHeszBRF9L7t7cWi\nw1LQ2WcoAdfcme56ggXTwvB51pc+4+pwPVnPIG5vLyrrlVK7JUnSnwzeJVFYCMuWMaeHAzGks8h3\nHFrFQIE2OVlsO7K2Nsz5pPKj1Yoc6BqNSIRiQKoKz02sS4PrX5FuCjZBL3F10xHq1cij05j6FOiT\ncl1RwNJSZF4zcOY4SZIMQwbvh6CqkJKuJW/jdi5mRTPfL41B7l1o6dDIMBdISxNVq2rUMMz5pPJn\nagpBQSIBigHrgB8+bUH4FTPWvePAqxZt+dwrk4gNHzD3tRiszHVs/VPPKRwrK/F+jNMzBaskSaVC\nBu8HtPkPOxoP8qFedy+uffg1r3SzQ6uY8IH3a4a5QEGBePj5PXRVM6mCs7YWK9BTUgzWk/3zmDU9\n26ai1cKMx6diX2TM6wVbIPwCvdun8GeIAUZuHBzg7FnIz9f/XJIkGZSMEg9gwy57Xp9fg7ljYkh4\n8+1U9bcAACAASURBVH0u1LnMb/VS0R54l7wkD8NcJClJBG4LC8OcT6pYnJ1F/fWEBIOcztJcR3K6\nmKpxMLFllvco9taC71ePJzlFwcrCANvUjI3FzUZkpP7nkiTJoGTwvg+dDiYtcePb9y7ROTiZotVf\n8kZPE+pYuDOmxiDmf1NN/4ukpIgSk25u+p9Lqrhq1xbbyAywEKx3hxS2/GlXnDr1Ze+BNMKV/zWM\npmDLd/TrmKL3NQCxeO3SJVEUR5KkCkMG7/u4FGtCbr6GNo0zYft2PqsRx1nbfBY2fIsXumSx46Ce\nhUJuzoM2bCjrcz/qNBrw9xe/Zz2LgFR3KmTKS3E8PsKLVdscCb9sQS/jj7hqC0ZNJuNSGENungHe\nTxqNqDx27lyp7VmXJOnhyeB9H1oNFBYqqAWFJKxdwfQOGjo5Nefpao+TX6DBSKvHHzRVFb3ugABZ\n5rOqMDUV89/p6WLXgh7eGnydpROusPF3O3q8VY8/N3enTmwnFrQo4Oe3FlC9cwCvzatBVo6e/82t\nrcV+9esGKLIjSZJByOB9H7Wq51PNsYATH//ONO9rZJrAQt+3UBSF5ZudeKatHkOgyckikYeLi+Ea\nLFV8dnbg62uQEqJPtUhn28KLXNh0GjvrIupfmQ9aLbvb7iDif8tIzdDS4626+neabW3hzBm9bzgk\nSTIMGbzvQ1HgozGXiN+7hM+bwrDqz2KZ6c24T9zZvt+WtwaVsK5yTo5YEOTlZdgGS5VDzZpQvbrB\nqngdOW3B8fMWbJuez7h6Q1jnD+c2T2fVO2e5nmzM70f0XH1uaipWnV++bJD2SlKlkZdXIaeMZPB+\nAB2SNjC/UzSmBZZ8O+FTgp73JitHw4Evz+PqVIKeiKqKYdPAQL1KkUqVmKKI3rexsUFKiO7624Y+\nHVIwNoJ3GgzDXWPHG8FJKF9+Tr+Oyfx60Eb/NtvbQ3i4wUqeSlKFl5MDhw9XyAWbMnjfT2EhW3d9\nxm914IPGr5K16yoJu0+yZMJV3JwLSnbO5GTw9JRFR6o6ExNRAzwzU+/haGMjldw88d/Z0sicuQFv\nccwN1oR9i03ceUyMDdBz0GrFzcb58/qfS5IquqIiCA0V6z1kz7vyyduxnbeDkvCiGq2KBpGYqmca\n1Px8sYK3Xj3DNFCq3GxtxU4DPee/e7VPZcNue1IzxPtzoHtnHrP2ZuKT0Gj3BHq3039+HRDz9XFx\nBpmvl6QKS1XFDou0tAqbe0MG73spLOTTQ4u46ABXv1/BkEle1O/lx4B3a5NU0iCekiKGS01MDNtW\nqfKqWVPv/d/1auQxpFsSHUY2YOch6/+zd+dxNlf/A8dfn3tn3zezMouZMRh7ttCiKKQsRUK2FiQ7\nkUhJZEkIkcqarVKIiJLs2Y1tzNjGYPZ9n3vv5/fH4ZtKv2bmfsbMmPN8POZhu3PuGcZ933PO+7zf\npGda0sfyfeLsVXY0P0Kji+u1m6+Tk6h7rlG/ckkqd2Jj4epV8PAo65n8Kxm8b4tPtmDfSXuu3Pgz\nqMZv/46pDdKpHV+PmHlVObPhHNe2RODpVki7YaHFf+3KzBS1y729tZ28VLHdOf82mcwqRTpr+A2G\n9UhgwkI/Ap+ty5JPO9LU0Jk5LRSurpinXZcwW1vRaCU2VpvxJKk8SU2FiAgRuMtx7Y1KH7wzM6H3\nzHrU6hbOW/Or0rx/Tdq+EUrMDYWJZxeTYwkbuk7E3UVEaicHE3NHx6JTYOu+YhRoMZlE8oMsxiLd\ni62tuO+fklLi8zVFgX7PJnN01QXS95zk1NrzfPf0a+gsLBnfPAs+/VS7+bq5ibPvvDztxpSkspaf\nDydOiOMsCzNb65aySh+8u/fQYW1p4urmCA58Fcn1rRE80SSTd8b8xFe1Mml98wnCXav/5XMUheI3\nf0hJgeBg2epT+ndeXuLev0bXxwCq2nrxVkhf1oerHDi+CU6f1mZgCwvxHyE6WpvxJKmsqaqoZWA0\ngo1NWc/mP1Xq4H3sGJy/AJ8PO4OTg+j2ZGWpMr7vTa62mI5jvg7v1AX3/Ny0LD12NkXsEJWXJ864\ng4K0mrr0oAoLE3eqNbyONTa4D75WHozoaIHpo+naFVpxdYWYGJHUI0kV3Y0bIhnTza2sZ1IklTp4\n798Pz3RQ0f8t92zbnqXsC8qm7cUubP0llLikv26fpGXqWbnVnRfbFnGFlJ7+551eSfr/WFpC/fri\nPEejhDB7C1um136TI14G1thEwTffaDIuigL29iJ5TaNWp5JUJrKy4MwZkZNUQVTa4J2RIWLq9et/\nPX8uNBkYHbeSkBQL/G1nMLpXPI++Hsaa7a5EX7dm468uPPZ6DV5sm0qdkCKc96Wlie3QKlVK6SuR\nHjguLqJ9qIbXsXr7daCxcy3e7mBFzhefiburWnBwEP+R4uK0GU+S7jejEU6dEnkn5fyc+26VLnjn\n5cGwYeJo8euv4cet0ObtxtxMFKvixQfnE+mYz0N7XqZn+wImDIjj4xGxrPjRnafeDOXT9Z683S+O\nj0cWIdPWZBIJEGFhMklNKp7AQLF9p9GWtE7RMaf2KGJtCpjdKA/mztVkXEC82Th3zqxMeUkqM1FR\nYqfLwaGsZ1IslS549+snbricPy/u4C+Yr/JHpAsNe9Vk6XaFSbe+4bHLFngGTOShWuLc8dlH09mx\nIJrLm86we8lFejydWrRYnJoqktQq2DeFVA7caR9aWKjZGfUj7g15wedJZjyi4+b+7ahHjnIm2oaj\n5+zMax9qZSXeqF6+rMk8Jem+SU4W37cVaLv8jkoVvC9cgD17YO3aP69avzFY5cf3j5GRrWf83vlk\nWBfS5mhvVu32Z+oXZtzHLiwUL8AySU0qKTs7qFNHuy1uYEbNoRj0CuPb23Fp2Dy6jgrglQ8C8O9Y\nl0++9ix5FUhXV7hypVzWgJake8rPF+VPnZ3Fa3UFU/FmbIY9e6BDh3+2zv75uAdVa54no+kqXjml\nY+Lcnpz/5ixrdrixYWcJ64+npkKtWrKSmmQeX1+RM6FRgZXq9lXp4dybVbVyyXQ/T2T3SZxae569\nSyP5arMHC9aXMDdDpxPXay5cKJd1oCXpL1RVJKipaoW4FnYvlSp429n98zWwoACWbKuG1RPDsTaa\n+MCmA7i54e1h4KM3bzBvbQl6befmiixcHx9tJi5VXneqrxmNYjdHA+k/TcKq0I3XO7qSu2Ape3fm\nUiMgn7UfXuajFd4UlnSX3skJEhMhIUGTeUpSqYmNhfj4Ct0cqlIF744d4ddf/3o0l5AApmq/c875\nV8btU/Du8er//qxl/SzOXy3Bu7KMDFFJrQJuxUjlkK2t2D7XoHjL5VgrftwVSHDU2xytmsqO0HxS\npy5kwJQAalfPw8ZK5XKs9X8P9G+cnUXymlZ3ySVJa5mZFe5a2L1Uquji6grTpkHr1rBsmSgOdeCQ\nifSWY/HLUBhl0xqqVv3f489dscXXo5irnawskSVcwb8xpHLGx0ckapi5ff7ahwF4OBtY2uEpajkE\nMa6rHe3y1mF5+hirtrmRnqXHwc6MO9s2NuIs8do1s+YpSaXCaBRVBu3sKtS1sHupVMEbYPBg+PJL\n2LgRnn4aJn6zFqPvST7apWLXq9//HpdfoDB5iQ+vdi5mslB2trijK6+GSVpSFLGbY0bzkmu3rIiI\ntmXQ84ksWOvHrFrDibLMYPETTszKH8bsZe7Urp6Ln6eZ2/NubuL6TXa2eeNIktauXRM7ow/ADaBK\nF7wB2rSBLVsg4kIOOQ+/TeN4C2pEt+KROV34dF0VZq7wokHPWni4GHizezHO79LTxerIxaX0Ji9V\nXjY24vpYamqJPj0+2QJ/7wJG944n6ro1X85+mYesWvDuwwaM6RdpE/MVn469bv489XpRKS4y0vyx\nJEkrmZnie/IB2RWtlMH7jo8PfMyNrJvM2Wqg0ZTnGdkznjOXbLkeb8Wi8TGsn36l6Dsrqiq2C0ND\nS3XOUiXn5SW20EuwfR7qn8+lWGvyCxV+W3KRVvWzyd8yh3TyeLNNNabq3qW+u0ZtPl1cRJ1oDa+5\nSVKJmUzinNvOjn/Uw66gKm3wvpV5ixn7Z9D1lguPmKpi8WhLnn8yjSXvxPDpW9dp3TireDvfaWng\n7y+7hkmlS1HEFUSTqdjZ565ORno8lcqgaQFY6FVG9U4g4lMLnnftwroGt7jhmgsL7t2Ip0RcXETd\nc5m8JpW12FjxGv0AbJffUWmD96TdkygwFjCj5XswYIB5meFGo3iBql79vx8rSeaysSlx9vkno65j\nZani37EuvScF0m5oCLtmzMVKb8WE/n7iPEmrtqE2NuLaZEyMNuNJUknk5IgbEBWkW1hRaRK8FUVp\npyhKpKIo0YqijL/HnyuKosy//eenFUVppMXzltTp+NN8deIr3mz6JiE934SHHzZvwLQ0EbhtbbWZ\noCT9F29v8PQs9va5jbXKmg+vsP/LSNo0zeTVTknc+CaRiWF9+M7+Gnvru8DMmZp1NMPVFS5e1LTF\nqSQV2Z0e3ZaWFT67/O/MDt6KouiBhUB7oDbwkqIotf/2sPZA6O2P14HPzH3eklJVldE/j8bV1pVJ\nj04yf0CDQWxlBgaaP5YkFdWd4i0GQ4m2pUP98+n3bDIvtEnD3tbEqOq98bPxZHRXe0yRF+CHH7SZ\np4WFeOG8eFGb8SSpOOLiRDGPBzCJWIuVd1MgWlXVy6qqFgDrgE5/e0wnYKUqHAJcFEUpk/Jj26J+\nYtflXbxQ5V1yUzWorpOWJpLUZBlU6X6ztRXXxzQo3mKnt2FazTc4ot5gbcdAWLRIs45muLjAzZua\ntjiVpP+Uny+S1DSsopaWl0ZkUvm4RaFF8PYD7r5fEnv794r7mFJ34KCJrp+9hWVmCJfXDyY8HAa8\nqpBXUMK/hoICsaq4q7CLJN1Xfn7iLE+DhiC9/TrQyLkmbzfLJDc3AxYv1mCCtzk5ieQ1rbbjJem/\nREWJbXMNF1Yf/v4h9RbX41bmLc3GLKlyl7CmKMrriqIcVRTlaGJiombjxsRA5046Pmj8FT8P+YKd\n261u39dXGLzg77v8RZSaKnp1P2BnKVIFotOJ7fOcHLMDo+j5PZLrhmTmvlIHvvtOu7vatraiaMt1\nDe6RS9J/SUkRL/oaJallZcGwyZeZvW8+RPSk3ws+/PKLJkOXmBbB+wZQ7a5fV739e8V9DACqqn6u\nqmpjVVUbV6lSwg5H9/DZZ9CrF7zVqymPBz0GiMXA8q9MbDroyY0Ey+INmJ8vXpC8zWgbKklacHAQ\nbyI12D5/zP0hOnk9xjTfaOK9HWD2bO26hLm5iTcDubnajCdJ92I0iu1yJyezK10mpen5aH0g1es7\nsjx2PNaWFhz5aCq9e0OfPrBmjUZzLgEtgvcRIFRRlCBFUayAHsDmvz1mM9DndtZ5cyBdVdX7uu9w\n5Ai0a/fP33dwgIdrpXHsvF3xBkxPF2VQH5AL/1IFFxgoClBokNU9s9Yw8kwFTH4lGE6cgB07zJ8f\niB0qCwuZvCaVrmvXxP8DM2//HDhlT+1u4Ww+5IlDzf1k+n+D7tBYDu304+WXxa3K0aNLXK3YbGYH\nb1VVDcCbwA7gPLBBVdWziqIMUhRl0O2HbQMuA9HAUuANc5+3uFxd4cY91vqqCrFJNrg5FyNjNz9f\n3GH18tJugpJkDr1elE7NzDR7pVzDIYA3ArqxVH+KM82DYN487a56OTuL/4gyeU0qDTk54s2hmdvl\nhQbo/nZ1lk++ioOtAYuOo/Fx8OGPOWOZOFFsIDVqJOpyHTig0dyLSZMzb1VVt6mqWkNV1WBVVT+8\n/XuLVVVdfPvnqqqqQ27/eV1VVY9q8bzF0bv3vV+Dtm+HrFwLWtQrRhOF9HSoUUO2/JTKF1dXsQLX\nYPv83Rqv4mRpz9jO9qJH91dfmT8/ENuYTk5iW1NWXpO0pKpw4YJIIjZjR1RVYe5aL/Q6lSPn7Lnk\nvoWo/CN8+MSH1AmzZ8AA0ZUSxOK+wq68K4pnn4WGDaFFC1ixAnbvhnHjoE9/HctGRRQ9DhcWgrW1\nPOuWyqeQEPHClZ9v1jDuVi5MCn2F7XlnWN/xYYyr1hC5N16bOdrainNv2TZU0lJiorjXbcad7vwC\nheffqs7MFV64ORlJzyvgSp0pWKXUJyC9DwcPinXbzZsi9/L0afNrfJVUpQneOp1YPLz3HmzaJH40\nGuHwAROP1i1Gl6Y797rlWbdUHllZidKpZvb9Bnjc0AfLzCAGBCeRrVgSO24BrQfWID5Zg9sVbm5i\ne1O2DZW0YDCIq4jOzmYN88EXPhgMCuumXaHAoODVYQ6q8zUKtsymSyc9r78OQ4eKui/PPSfOvMuq\nnUWlCd4gAnjnzqKX9549IpG2WOXIDQYRtH3KpL6MJBWNp6fIxzAjgGdk6Xh2eDgDHceQ4xrFupHN\neLLgJ/q7/UCn0cHmJ6Dr9WIH6/x57bLZpcrryhWxf21jU+IhjEb4/HsPZo+I5Ykmmegc4pl4Zjn2\nMU+zcEgLjEa4fFlsGu3cCS1bwoQJGn4NxVSpgrfZ7qy65b1uqTy703mshKVTAb7e7kbL+lnMf6Yp\nrdwaMMn1BBkh1Xg5ciI5mQb2ndSgO5OzM8THiw9JKqmsLFGQxcxKalm5OvIKdNQIyEdRoMHAcRh0\nObgcmE7EWT3Z2WBvD+vXi6PXCxfMvolmFhm8i+pODXNf37KeiST9Nzs7s+5+H79gx5NNMlEUhTm1\nR5JQkMpHr9VEiYlhivMcjl8o5tXKf+PqKrY7yyrrR6rYVFXs3tjYmH2U6WBrwsbKxMVr1pzNvMSa\nxG+xOfka3z95Cnt7keZ0/Tp06ya2zA8e1OhrKCEZvIsqPR2Cg2UNc6niqFZNHMiV4JqXh4uBq7fE\n93oTl3B6+7VnTv4erj3VjKfOfoK/PlabOVpb87/9SEkqroQEkahm5lk3iNj/epckRs+tyuiz83G0\nsGPhr7dovOR1Yg7EMm6c+HYFkRenwVOaRQbvojAaxTs8WcNcqkj0epG8VoK73307JvPVZo//VR78\nMGwIqknh6SBXdMZCGu2coV2ZcldXEby1aoQiVQ6FhaLdZwmyyyOibZi+zJtpX3lz6uKfxVzefe0W\nqVV2sSNpP/0OBzMgaxMTlak41KrG0KHiMaoKs2aJip1lSQbvokhLg6CgP992SVJF4eJSorvfNQPz\neatPHM361WTal168NPQxdIdGE+mznYXNuxBwYhNDe6WQma3BS4hOJw4Tz54Fk8n88aTK4epVcdxS\njNdloxH6vx9Au6GhJKZakJxuwTMjQnh5UiAGA+j0haS1GEdArgMztp1iR53RLHMcjl4Pe/eKYoPd\nusEff5RtshrI4P3fTCbxL+7vX9YzkaSSCQ4W+RqFhcX6tNG9E/h+1iU273Xh4jVr+rsNoIqFBxtf\nPIPJ05NxCaOZ8KlG9Q4cHMTK++ZNbcaTHmzZ2RAdXewktTlfe3H1pjVR359hzqhYPh4ZS9TGM8Ql\nWzJzpTdfXt/E2azLfLw5C+sevXh6WQ9OLdpPNT8TEybAhx+KWiH79mnaabREZPD+L+npEBBg1hUE\nSSpT1tai81gJkteahOeQmmHBlk8usWhUGtNrD+JA+hm+GfoEARkR6Lf8QH6BRim3bm5iGzQvT5vx\npAfTnSQ1a+tiJ6kt/KYKs0fEYmfz5zGSrY3KnFHXWbDZikmn59HqGnSt2RlGjABFoYpLIe+Oy+fA\nAfj9dxg1quzudt9NBu//j6qKbZnAwLKeiSSZx8cH3N1L1Pc7NsGS8OqiE1i/as/SwKkG46z2kNe4\nAZMKJpFxS6NCKxYWYgtdNi6R/j+JieJ6YTEzxoxGiImzolHNfyZw1g3JQ6k9ikQlh0/iG6CMf7ts\n74EVgQze/5/0dLFdbqfRtRhJKiuKArVriwoTxTxXrhmYx8EIewD0ip45tUdxLfcW73atjYuaivva\nBdrN08UFYmM1qc8uPYDuVFIrQZKaXg9+noVERP+z29iu45dJarqavpG2NB79cYWooCmD97+5s+oO\nCCjrmUiSNhwdRUnBYgbGYT0SeGt+VZLSxAtaa4/GdKzyGHNyNvNzoxfQff+dOH/Uwp3GJRERaJfO\nLj0wrl37s6tjCQzqmshb8/3+ctRTUADT9gzDwgTTnpxW9nfAikiWCvs3WVmizGR5ONyQJK1Ury5a\ncubnFzlLt88zKVy8ZkPY83V47tE0rC1V9h1fhKlPA+Z3yqPdJQeUmTNhyRJtthptbcXW6JUrotGK\nJIFIUouKMqvd51t94zgVFUStbuH0fDoFnU4l7sD77O4ex+S8Fvg2eETDCZcuufL+N7m5xSx8LkkV\ngKWlSF4rRt1zRYEPh9zk5NfnqBucy/5T9likhRJ661W2m7bQ320wHD8uCj5rxc1NvFBnZGg3plSx\nRUaa3e7T0gLWT7/C2g9FUSCfm4c53upzquZb81bnGVrN9L6QwftecnPF1p0ZreUkqdzy9IQqVYod\nGKt5F3IwwoGm4TnEbovgYP9uOCoufN9+H8doQNL7CzhxQqM56vUi1yQiQt79liApCW7d0uQ1WVGg\nWZ0cpvY8h11mX475wvT6o7Gz+OdZeHkmg/e9ZGb+eTdWkh40dxqX5OYW61z56k0rfjvmwIK3YrCy\nVJk4JxzHI5PI8NzHx5064JF/k9+GfcuGnRpdgHVwEG8wYmK0GU+qmAwG8SZOy8WUyUTW++/wdtNM\nmloH0zO4s3Zj3ycyeP9dYaE4C6xSpaxnIkmlx8FBdMhLLXov+9NRtjSrk42tjcq+kw78fMiJM2Na\nEWQRysbQtay0fJEhuXOYMtlAWqZGLy1ubuJOb1aWNuNJFU9MjFlJave0fDkfWR7mliPMazwRnVLx\nQmHFm3FpS0sTZ90V4KqAJJklMFCcIebnF+nhnm4GLt+wRlXh65/cGPR8Ej/tdyf9u7nkO1xh/gt+\n6KwtmG0aReuBYSXtRvpXFhbiRVuWTq2ccnLEvX8zktT+znT0GFfWfsbsVjp6+bajuWtdzca+n2Tw\nvpvJJLYUfXzKeiaSVPqKmbzWrE42qgrf7HIlI1uPq6OB4bOrsWlIdexjnuFczcUkDexBO8NWGibv\nYuNujbbPnZwgOVlkyUuVy4ULZiep3aGq8MWXkPTGZN5o40C+0ZqC7R+RkFIxL13J4H23tDTRRlE2\nIJEqi2IkrykKrJpylaGzqnH1liUfr/bCyd5I38lBPG94BwMFTAhPJFoXyuSM0Qz90It2Q0NYu93V\n/EWzu7sonVqC9qZSBaVhkhrA5MXehKyazPnAZLaHZ/JWcF+q2Xrz+MAaZOVUvFBY8WZcWlRVJEbI\nBiRSZXIneS0vr0jb0o1r53B67TmebJLJpVhrUjIs+PydqywfasVAv5dYdvNH+vsMI6AwmvcdZjPg\nuWRmr/ZiwJSA4nYl/SsLC/Fx7lyx25tKFZDBAGfOaBa4k9P0JKz+mUeztzKqpyvVbLyYHN6bj0fG\nElotn9XbtNuWv19k8L7jTlEWB4eynokk3V8ODqIYShGT17zcDXww+Ba/fR5JZo6e3pOCePS1Gqx8\n6xPI8SBjyAr+8GzPK0kz6F73PPu+iOTIOXt2HHQyb54uLpCQAHFx5o0jlX8xMeINpQZJaqoKe3bm\nM8swkuXPVuW4RQIzag3FTi/G7tU+he0HK0ZVtbvJ4H1Hbq7o2S1JlVFgoFjZFjF5DeDhejkM7Z6A\nv3chQ7olsnBEOiEXJ3K64CgvB7fFQmeCefOwtVF544VE1u7QYHXj5iZWZLLz2IMrO1uTJDVVha82\nuVPrhXDUmbPIschlWJ1smjjWp4fv0/97XEGhgl5X8XZzZPAGcdfV3r7sG7RKUlmxtBSNS4pReQ1g\n9ohY2rdI582Z/oyZV5Wr2wbjlFGPzI4zyRnQS1Rd++MPvNwKycjW4OXG0lJ0HrtwQW6fP4jutPvU\nIElt1kov5nztxcYOX/C8+h1dWzch2zKNpLULyc4VY5tM8OUmD7q0Lt73fXkggzeIFUdIiCzKIlVu\nXl7Fbhuq18N7A28Ru+00X0+5gpOtwndPDONWYRwzHjaBnx/MmsW2PXa0qKdR61AXF7h5U2yhSw+W\nhATxYeZZd0aWjunLvflp2jFqb3iPC40DOdTsMI6R/fDMq8/nGz04e8mGnu8EUWhQ6Nam6PUOygsZ\nvEGsuGVRFqmyM6NtqI21ypPNsmjfMp3Fc7ryfJV2zLq2lkvD+sOVKwT+vpIBzyVpN1cXF7F9XlCg\n3ZhS2SosLHG7z7/be9KBh2rlUG3tbNS0VEZ2d8LR0oZ5jQaRlmnB2PlVaTcslFD/fHYsiMLaquLt\n4sjgrddDo0Zim0aSKjtHR5H7UYzKa3f7YuI13J0N/PzBZxQU6Klz5iz7HZ/mHeMU3As1TDSzthbH\nXZGR2o0pla3Ll/+scGkmnQL1U/fApk1sfe1xtuecZnKN1+j/hJ4PBt+gfYt0rm+N4IPBN7G3rZjF\nf2TwBrFtLkmSUL26OFcuwarWxlplyTsxXF6dxgC3V8kL/p78RS3RGwth/nxt5+nmJrKS4+O1HVe6\n/zIy4NIlzSqpPRqexJBLI8n0q8bIoIvUdAjkzcAXRRLbZg86PZauyfOUJRm8JUn6Kysrcfe7mMlr\nd/NwMTK/ZXcCbX0ZlrCcwj694aefROtQrSiKOPI6fVpmn1dkJpPYLre3F28aNWC/9kuqmy7xaHgb\nonNj+ThsNDE37XhlSgAJKRb0ap+syfOUJRm8JUn6Jx8fcHY2qyGIrd6GT8JHcTbzMiNCPchw9CN1\n0sdkZ2q4TWltLYL4+fMy+7yiunlTvFHUqsbGlSuwYgWxnVpzvsV6XG48w7OdhtC8X01cnYz8uvgi\ndjYV/3tFBm9Jkv5JpxN1z7OzzQqKjdQ22N9ow+KMJXzeaAyu8ZF82H4vP+03s2DL3VxdRQCQrUMr\nntxcUTVPq8YjJhNMmwZ2doxpY0LRGTn+8usYDh8ncddpPh4Zi7NDxTzj/jsZvCVJujdnZwgItEBP\nkAAAIABJREFUKHHy2qXrVtTtEY7hx/mY9Lm8U+U4F30fY4oymbGTrImJ0zBJ1MNDbL2WcK5SGYmM\nFEnDWuUdbd0KJ06we/hzrE/ew/iQvgTZ+T2Qt4Bl8JYk6d8FB4sfi9nfMy1TT8tXwrCyVEnZkMv4\nGr0oqLOKEWH9UPMLWeEynKXfa3g908JCvNk4dkys5qTyLyFB7JhoVL+c9HSYN4/C+nV503k/QXZ+\nvBXcR5uxyyEZvCVJ+nc2NhAWBikpxfq0ZZvdCfItoH2LDOxsVSaGvkJVGy9in/mIWVZjeSjmB6yO\nH9B+rooiEtiMRm3HlrRVUAAREdoFboBFiyAjgwWv1+dc1hXm1h6Frd782ujllQzekiT9//z8RCZw\nMdpx/nbMkaeaZ3D2snjxtLew5ePaI4jIiuS7zvbctKlO/3Nv0XW4H51HB7N8izv5BRrsbbq4iK3z\nixfNH0sqPdHR4g2WVu2Xz56FjRuJ6/kck1O/p32VFjzr9ag2Y5dTMnhLkvT/0+uhTh1RNrWIyWt2\nNiaqehaQnqVn/c+iZ0A3nzY84d6EU9Vn0FM/nar5l5lqO50eT6Ww+ic3Wg/SqK+yh4co+HHjhvlj\nSdpLSYGrV7VLUjMa4aOPwN2dsS2zyTcVMC98DMqDeNB9Fxm8JUn6b25u4OsrzhWLoHvbVJZ8X4U1\nH15m5JyqdB8fxOLvqmD3yzyM+mxOd91CzqPtqP37YnrUPs3OhVEE+hQwfZm3+XNVFBHAT58WxT+k\n8qOwEE6dAicn7XpJfP89nD/Pb8OeY3X8Tt4K7kOog782Y5djMnhLklQ0NWqIF98inCc/92gaVT0L\nGTe/GovGxRBePY8l33nw4w/N4cBoUoNXUzvhFXJM1hinz0BBZeIrt1ix1V2buVpYiHvDx44Vq82p\nVMqio8V5t62tNuOlpMDChRQ0fYg37HYTZOfHhJD+2oxdzsngLUlS0djZiQBehOQ1vR6+mXGJ7m1T\nmL7chy83u3Phqi3vDLiJ07G3CbD1wf6Vt/k6ZBz6Pw7Brl0E+uaTlKZhqWI7O/FG48yZYjdakUpB\nSoo4znDX6A0awKefQm4uc/rX5HzWFRaEj32gk9TuJoO3JElF5+8vkoyKUI7U0gIGv5DE4RUX6NMh\nhde7JjFlUBxudtYMdXyHc9nRJL2TyEldQwyz5rD7N4WHahY9Ka5IXF1F7fNLl7QdVyqeO9vlzs7a\nbZefPg1btnCtbyemJH5LF+/WdPBqpc3YFYAM3pIkFZ2Fhai8VsSz7zuOnLOjfYt0dDoY0zuer+f1\np63rY3x49XMWNh2HPiWJxOlfMraPhp3H7qhSRWSfy/7fZScqStvtcqMRZswAT0/6hiZhNOp4JG4K\nGVmVJ6RVnq9UkiRtVKkiPoqRDObmZOR6vBUAb3RLpNNjaRyauZTcAlgWtIEvLAbRJ+czOnsf1n6+\nOp1IuDtxwqxa7VIJJSeLeuNabpdv3AiRkfSq9RJ78vbQKH4Me3bXIfC5uqzZ7qrd80C5PXKRwVuS\npOJRFKhZU2ydF/GF7eUOycxb50l2rg5Fgcmv3+LaqmxapI7AGLaR3wfUpdDBDXX6dDAaSUnXM3OF\nF23fCOWZ4SF8tcnMe+BWVqKIy4kTYgtXuj/y8+HkSXH/Xqvt8tRUWLSIw54t2dzyO2o7VGfPgA78\n8PEl9i6NZNQn1Th+QYMVvqpCYqJ402Fvb/54GpPBW5Kk4nN0hKCgItcSb98yg5b1s2jeryZfbXLn\nlz8c6flOEPs+m4x9Zi02e37IRJf3Uc6e5eqirTTqXYuzl20Z2TOe/s8msWaHG23eCCUnz4wA4ODw\nZyMM2YGs9Kmq6PZmMok3TlpZuBA1J5cuTYPIsorl83oTsNKJOvnhwXmMfCmehRs8zXsOk0kEbj8/\neOgh7Wqva0gGb0mSSqZ6dbElXVDwnw9VFFgyIYbpb97gpwPOjJvvx96TDvyy4Co/Pz2GDIsbGD7Y\nz2XflrisnM+Ip86z4v2rdGiVwQtt0vh5QRRebgbmrPYyb87u7qJ4y7Vr5o0j/bebN8WHVsVYQNwc\n2LSJrR07cOuhNQz070pLtwZ/eUirBlmcv2rGmwWDQeRHhIaK4kR6vZmTLh0yeEuSVDJWVlCrlujF\nXASKAh0fSeebGZdpWDOHSa/c4okmWSRFPIrftQHMvbyOft6DsVVz6B317l8+V6eDt/vHsWqbBuem\nHh5i9V3Meu1SMWRliUCr5Tm30QgzZ2LwcGNcs0h0OZ5MDR36j4edu2KLr0cJj0aMRkhKggYNICRE\nu63+UiCDtyRJJefjI67/FDMRLCrGhibhOcxe5cWoT6ryQehQPK3duP7cdKbrxuBxYAvqwUN/+Zyq\nngUkp2uwfanXyw5kpamwUOQW2Nhou928aROcO8eCoc04VxBJyJkZfP1D8F8ekpymZ9YqL17tnFSy\n50hKEm9I/fw0mHDpMit4K4ripijKTkVRom7/eM80P0VRriqKEqEoyklFUY6a85ySJJUjOh3Urg3Z\n2cU6Rw70zWf/SXumfunNb0su0v8pI5/WHc1VzjKtmTNR+jDy35v2l2YoOw460TQ8W5t529iIIH7y\npOxApiVVFbsaubkix0AraWmwcCExLeswUdlN+yot+PG1hsxd60mnUcF8vtGD95b4UL9nbV54MpWn\nHy5BWdzkZBG0g4K0m3cpMnflPR74RVXVUOCX27/+N61VVW2gqmpjM59TkqTyxMVFFG8pYvIawMCu\nSXyy1otHGmZR1UtscXbzaUPNgicxPPE+Q/wmY5V8C9OCRfzyhyMfr/Jk3Kd+jHk5Xrt5OzuL++qR\nkdqNWdnFxIicAi23ywEWLULNymRwZ0tUVBbVHU+ofwFn1p+j4yPpHD5jT2aOni1zopk25Gbxd7sz\nM0VGee3a5Xqr/G6KakbWpaIokcDjqqreUhTFB/hNVdWwezzuKtBYVdVi7WU0btxYPXpULtQlqdzL\ny4PffxcBsYhbpd3GBbHldxde65JEVc8Cth905mrOLRJfaohNYlMmLwhniLqIp6x+Y6/aCksLlRfb\nprJofAzWVhpli6uqqMBWvz5UrarNmJVVaiocPCgCt5bb5efOQd++fP1ac3r7HuST2qMYUb2nduPn\n54tjn5Yty8WVMEVRjhVlkWvuyttLVdVbt38eB/xbKqgK7FIU5ZiiKK+b+ZySJJU3NjYQFlasJLAP\nBt/Ewc6Ih0shiWkWDHo+kcivsphZexjJ7r8xsUlNMh392OTan4yfD3Nr+2mS0y0YOUfDIHt3B7Ji\nVo2T7pKXB8ePF+vNW5GYTDBjBgl+Lgz3P0tdm/ocWTaO4E51aNizFrNWepl3fdBoFFvyDRuWi8Bd\nHP8ZvBVF2aUoypl7fHS6+3GqWML/29vhVqqqNgDaA0MURfnXLumKoryuKMpRRVGOJiYmFudrkSSp\nLFWtKl4Ai5gEVjMwn3YtMjh63p5hPRJ48alUcvJ0RG0chkXsI+S3e4fsqYOwj7+M9aovcLQ38eWk\nq6zd4UZSmobXdywsRIvK48dlB7KSMBrFmx/Q9j43wObNcPYsw171I8OQS+zSFTwUls9P86OYP/Y6\n+0858NSQGuSWNIAnJYmCQ1WqaDvv++A/g7eqqm1UVa1zj49NQPzt7XJu/3jP4sGqqt64/WMC8D3Q\n9P95vs9VVW2sqmrjKhXwL1SSKi29vth1z7969xq1AvNo2Ks21TvVIaBjXRJSrAg9sQD0+Qyx2oX6\nbEdYsQIiI3F3MVInOJezlzSqkX2Hra1Y5UVElNtymOVWVJRI9nJx0XbctDQMcxewoFEY6zmD8bdJ\ndKzpx2tdkqgRkM8jDbP4fvYlHO2NLNviUfzxK1iC2t+Zu22+Geh7++d9gU1/f4CiKPaKojje+Tnw\nFHDGzOeVJKk8cncHX98i3/22slSZOfwGsVtPs3PhRWK2nubrqVcJsA7gBd0Ifoj/jW97NRKBYcoU\nTAUGYuKs8HAxaD93FxdRVSs6WvuxH1RxcaJjm0cJgue/MJlg6z4ntr60ijRDBuOfSsJfqYnfxZEU\nFOhoPagGWTkidCkKDO2ewIadxaxnXgET1P7O3OD9EdBWUZQooM3tX6Moiq+iKNtuP8YL2Kcoying\nD2CrqqrbzXxeSZLKq7Awcde3GFewbG1UgqsW4OwgVr19OiRzYd3bNHKqzZDLn5Iw9g2IjGTNy9sw\nmcDOppRWxx4eYiUZr2FW+4MqK0u0+XR3F1cGNWAwQPfx1Vkz+ybtE1fS58VQsq1TyVm3DHcHHWun\nXcHfu4DPN/75ZsHF0UhOXjGePz9fVAVs0AAsLTWZd1kw629cVdVkVVWfVFU19Pb2esrt37+pqmqH\n2z+/rKpq/dsf4aqqfqjFxCVJKqfs7KBGDbMqmL34VCo1/Q1krV1Oan42oZERfK/rzAuXZtC5+mka\n96nFu4t9tC9RrtOJYHTiRJF3Dyql/HyRI2Bjo2kA/HKTB8mpCkst3uDrMFd+CopkQmhf3mjlz9nL\nNsTEWTGkWyIbdv250v7uV1ceeyizaE+gqiIrvgImqP2drLAmSZL2/P3B2lpkIZeATgcr37/KO894\nwG+TyQj4gXEtHidPseWV08P5ae4FvvvVlW9/0ficFUQwcnSEw4dlBvq9FBaKwF1YqG0hFmDZFncW\nhcwhN/4cg58zUtcxhHdDX2N83zj0epVOo4JJzdSTV6Cj0ABf/uDO6p/cGPpiEXu1p6SImvwPQD6V\nDN6SJGnPwqLYyWt/pygwb50nukNjaGhfj6T275I9cQANcg7y44hfeffVmywwt3vUv7GxESuzw4eL\n1bf8gWcw/NkXXesENcCYmEyNrZ/wZm93sm2y+cBrOtZ6K2xtVIJ8C2jdOJO+7wYSc8sKv/b1WP6j\nOz8viMLfuwi1zHNzxRvK0FDN510WZPCWJKl0VKkCnp4lDuC7jzqSma2nqofK2ibvkmvMZ5DPH6jN\nmjI+8x2yryRw8ZrGV5PuZmsrPg4fFglOlZ3RKM6409K07RR2l49No/iueh7r/JJpVzCciZOf4XSU\nLbHxltxKsiS4aj72dibWT7/MsVXn2fvFRerXKMLVRFUV34f16pXL9p4lIYO3JEmlQ1FEk4f8/BLV\nDz9y1o52LdJJTtfjYQhmes0h/JiwjxWDHsZCb6Let+8S6FOybfkis7MTq7VDhyr3CtxkEne5ExO1\nL316x6FDhOVsZEAHK8JtarOpazcGdk2k3dAQanStQ16+ju9+dWXXoijaNs+kWlFW23ekpIgrYa7F\nzEovx2TwliSp9Njbi9aKxah7foebs5GEFEu6tE5jwkI/hgb24DG3Rgy78QVLmwyiceouZgZ/XgqT\n/ht7e7GNfvBg5WwjajLB2bPiWlhpnRXn5aF+NJ3Xu9uSb1vI9c/X8NyImuw76UBegY5nWqURuTGC\n3UsuUi+0mJ3g8vLEavsB2S6/QwZvSZJKV1CQ6P1dzOplzz+Ryo5DTgx6PpGIS7Y0eTmcplELyc/X\n8WbNQ5xxa0arX94TQaW02dmJ5KzDhyvXNTJVhQsX4Pr10k3yWr6cz71usDkgl1nhQ7m5wsTrXZLo\n+Eg6J74+zzczruDvXYK7/aoqtvnr1avQ18LuRQZvSZJKl4WFKIZRzKtXbs5G5o+5TqfRIXRomc4L\nT6Zy5EBt9D8tBP8D/DAtGMVohClT7k9VNBsbse169CjExpb+85U1VYWLF+HqVZG7UFrFTK5c4cKW\nZYzsoKOtRzOGBfXA3tZE1yfS6N0hhQCfgpKPnZoqbj5oWESmvJDBW5Kk0ufpKVZuxTw3fvmZFLbP\nj+J6nBU7Dzvh713IzkFNeMn3ad6LX8/hkd3gjz/g229LaeJ/Y2kpAsGpU3D5crF6mFcoqiq+vuho\n8e9WWoHbZKJg+lR6dVWxs3ZgeYP30CkahaWCAjHvGjW0Ga+ceTDS7iRJKt/uJK/t3Su2n4tRkatB\nWC5L3on5y++FF45nf+opetn+yslHmuEwb57YGq1ZU+uZ/5OFhQhoFy5ATo54zgckgxkQuxiRkSJ4\ne3pqVj3tnjZvZrLrKY57wcb6k/C10XBr/k4xFmtr7cYsR+TKW5Kk+8PBAYKDNUn6crF0ZFWDKVzO\nucGwl1xEK8px4+5fURW9XgS2GzfEOXhOzv153tJWWCjucV+9Cl5epRu4k5L47buPmdEKXq3WiS4+\nrbUbOz1dvMHy9tZuzHJGBm9Jku6foCCx9axB681H3Rvxdkg/liXsYM27nUUi2aRJJbqWViJ3eoEX\nFMC+feIaVUWWmyuOIJKTS/eM+7bET6bSq0MuIdY+fBI+WruBjUbx/VWBm44UhQzekiTdP5aWovKa\nRnXD368xkJau9RmYupqLYwfAgQPwxRecvWTDK1MCCOsazkO9azJrpRfZuaX0cufkJHYV/vhDNDWp\niC1Fk5LE311+fund476LaddOXq6yj2QHPRuazcbBwk67wVNSRHOcCl67/L/I4C1J0v3l6WlW5bW7\nWegsWNdoGtY6S7pX2U3ecx1g6VI+HBBFSLV8Ns66xJyRsRw4bU+bN0LJySullZi1tfiaoqLg2LGK\ns41eUABnzoitfxsbcfxQ2tLTmfHrB+wIgbnho2ngHKbd2Hl5oipeQIB2Y5ZTMnhLknR/mVl57e+q\n2nqxssH7nMqI4oWm9pzSNWBJbj8CDNEE+BTw2ENZbJx1GU9XA59vLMW7yjqdOCfOyBCJedevl+9V\neGKimOetW2LeNqVYavYu+5ZOYlLzHLo7NGNgUDftBr5TArVOHZGT8ICTwVuSpPvP3t7stqF3e9qj\nFTVvDGWr4Rs6NeyHlRW0WP0GD/fw53qcJYoCw3oksH7nfSiP6eIiPs6cEVXZyltr0Tur7T/+EAHb\nza3Uz4ZNJjh42p5fvtpLD98DBBodWdpyBoqWz5ueDr6+92XbvzyQwVuSpLIRECC2OEvYNvRuy7a4\n43rkfepYNeB6u4lcnv4mgTnnWWv3KoOnVwPA1clAVs59WpFZWIhtdINBBPBz50RCWFlLSBCr7Zs3\n79tqe/dRB0K71GHkFAemZo4nwU6h2bW12CoathM1GMRHmIZb8OWcDN6SJJUNCwuxxZmWZnaxk682\nezCpfxIbm05DLbCjY97XZAzqT53oHwg/toK4JAu+3+3K4w/d5+5gDg7iytKNG7BnT9kF8aws0Vjk\n6FHxhsnd/b5kYl+4ak338dVZOC6G1k+35rfgfGa5DCApuhVvzauq3RPdSVKztdVuzHJOBm9JksqO\nuztUrWp28lpcsiWh/vmEunjyauECLudep3PQVS4GPcnUvLEM7J7Cx6s9eeHJMmgsoihia9rdXQTx\n334TQVyDNy3/KT1dVIPbu1eccXt63teiJfPXeTKkWyI5edP5yCea17LCGP74YFZNucryH91JzdBg\nJyQnRxzDVKtm/lgViAzekiSVrRo1ROKaoQSNJ26rF5LLb8fENuySXtVplzWJ3Tm/0iSwGVcJ5Ivs\nl3it1Xm6vx3MqYtltDrT6UQQ9/AQSWIHD4pAfumSWBlrJTdXbIvv3y+ufyUlidW/i8t9v/f8x1l7\n6oTvoW/aMpolWvNpF9EFztPNQO2gPCKizfy3UFWRIFhJktTu9gDV9JMkqUKytRUlRs+cEeewJTCy\nZzy9JgXRqkEWNQPzeTjnFc7mnuH6o9OZ7PoRa7ZNZl7yy9QfuIFhs6ux5/OLGn8RxaDTiUAKoqLZ\npUuiAYiDg7gzbm0tzqKtrcXRwr0+7gRhVRVZ+zk5IkjfuiV+rijg6ChW2mXI0SWF8ddfw06Bb5vO\nxtpa3L02meBmkiVuTiV/wwaI3Ytq1cSbokpGBm9Jkspe1arialV2domKazz2UBYfDLpJiwE1aV4n\nm/2n7MHyS1xfP8+2Rh9woeFAak6aR9+wyYy9tJKbiZb4VikshS+kmCwt/8yOzs8XwchgEDsR/3bN\nTFHE51lailW2ySSCuKWl+Ltz0DARrIiycnRs/t2Z1AwLmoZn0yQ8B6NqJL/501y1z+bn9C5UDX/4\nf4/fsNMVZwcj4cFmJCve+Xt6wPp0F5UM3pIklT2dTlReO3BA9M4uwfZu/+eSeeHJVLYfdOLwGXu+\n++gmgbWn02xfPzpafMeh3l3xWL2W3q7tyMppVApfhJmsrYt+Hm00ig9X19KtP14E3+5yYeD0AFrU\ny8KvSiEff+1FkG8+YT1f5qDzeSb+Up2JKcsYWiUZL3cDW353Zs0ON7bOjTZvFz8lRZRArURJaneT\nwVuSpPLBxQUCA0Wv7BLe1XW0N9GtTRo//ObCyYv2PN7Ylx+azKb1wUF0bXKZ7483ZPq5wVjlfgWE\naDr9+0qvLxdnvBHRNrwxw59fFl2kQZjIojcaoe0XO/ksfTPDj1kyecx8akcks2aHGxnZeh6um82R\nlefx9zZj5+NOklpVDTPWKxgZvCVJKj9CQkSyVUEBWFmVeJixL8fz1JuhhPrn0aFlPZbVf5eeJyYS\n3qgr0ddisRg/BlatEufCUokt+saToS8m/C9wA/yScojffSfwzEV4K2wqFn7evOSXykvtUrV5UlWF\nzExo3rxcvIEpKzLbXJKk8sPKSpPGJQ3Cclk37Qpj51UluHMdPhozEruDk4gP2chzz3bAcCOOq69O\nJfM+X/t+0Jy7YkOrBn9myp/LvEy3w2OonaDy6p7unKzaRfsnTU8XK+5KmKR2Nxm8JUkqX7y9xbZ5\nRoZZwzzRJJOzG86xZU40n42PoWX6KNxjurO75gqm9ulA4KVfWND+R2au8CI5rfKu4Mzh61HIhaui\nStv13DjaHXgD2+x8thwMYoJpsfZJgXcqqVXSJLW7yeAtSVL5oigiESkvz6y733eGCg/O48BpB/Q6\nhZhXR/K4S3PeD9jKxHqPMC7vfSJWnSC4cx2+/KFy1MTW0qudk5j2lQ+jl1jTZMdw0rNT2L7Rjv0P\nr8TOxYr6NTSuJlcJK6n9Gxm8JUkqfxwcROcxjRqXLN/izqRXbmFnZYn9lvV45dfl4xcOs7e5H/PT\n+rL3/V+ZtNiXP85o2Ff6AXJnwXu3gkKFpd97kF6Qx1xeIYUr/PC1iXnpaxm/sSmrP7iibU2Y3NxK\nWUnt38jgLUlS+eTvL4qWaFB9LCHVgup++cTEWXLoqC9/PD2bAFsfOj+TynmvQmosGM74l66x8Juy\nLWpS3hw9Z0eHYSHYtmyEXauGdBwRzPELYtU7ZakPGXkGmn/QDnyPsOFbEwlW77Db4inG9I6jZmC+\ndhOpxJXU/o0M3pIklU86HdStK64Fmdn3u2FYDr8ccSIqxoa6Ibn4O7mwo9kCrBV7nnzZkqsZF+hx\n/C0ir5Y8w/1Bc/ScHe2HhdCldRoZe06Q9ttJOrZK5+k3Qzly1pYlP7hA15fZmXyQz3+Ezt6P8+LX\nnVj23lW+2OSh7WTS08HPr9Inqd1NBm9JksovJydxfczM7fPRveOZsNCXQgNEXrOm0ABOhqpU+3kr\nels9rd+wI/XCDwwwLNVo4hWXqsLmPc50HBGCk72RYxds+WSNJ23eCGXyEl8MRoUn3qhO8pN9+Sn9\nZ6busqfnjQB47z3Q6WgankPUdQ1bjd7Zs69RQ7sxHwAyeEuSVL4FBYmqazk5JR7iqeaZTHr1Fr0m\nVScvX0eDnrWo3qkOLf2qcfCxRRTa2NCynzWPpoyDiAgNJ1+xqCoMmVGNCQv9SEqzYPHb19h/0pF3\nF/ty6qIdjzXKxNa2gKz2fVHDv2H871UZe1jl6dwf+P6IKJhy6qItAd4F2k0qJUXUvpdJan8hg7ck\nSeWbhYXYPs/M/Pd630Xwaudkrm2JYMbQWFLSLakbkkuQbwGrVz6CuvwX8uzsebKfiYvTRkNysoZf\nQMWx+6gju/5wYu/SC+h0KmlZFuh0Kq5ORqwsVTo+lkxexz5QZwNddzZn+q+xxAycyqx5VgyZUY2M\nLIUJC/0Y9HyiNhPKyRHJi5W4ktq/kcFbkqTyz9VVrMBTzavS5WBn4rWuyVzdEsHArklExVhjaaHy\n+4eWHH7iMwqdHHisUwqnpw4VV9UqmdXb3BjSLRFXZxNtm2Uya6U3jzbMomZgHnXCUhl5fSyp/t/R\n6UQnvtt/iIW2o2m2cig7Dzmh00HY83Wp5lXA0BcTzJ/MnUpqMkntnmR5VEmSKoaQEIiLE0HVxrwz\nVWsrlV7tU+jV/u6z9GB+a/UFT+15lUdbXmTLB6/RcuIXrNvjzVebPbiVZEm9kFyG9Ujg4XrZ5n0t\n5VRalh7fKmLLe1iPeDoMD+XyTSuc3VJI6fACaR77aLirP98fXMVJ18eIbjuSepdyyMjWY21pYnCP\nRMb00SBwg3ij5u8v3rhJ/yBX3pIkVQyWlmL7PC1NrMpKQW3H6hx4cg3eNu60bXKe+aMH8+kqF954\nIZF10y7Tsn4Wz79VndXbHsys52bh2Wzb78yBU/a8/G4QtQNzKbCK43LbdqS5HqLJj6M5vP9rTDVq\n0smwkfh0G9o2y2R8vzhSMizo01Gbe/n/u1QeUoGbx5QyRS2l/wRaaNy4sXr06NGynoYkSeVJRIRY\ngZfitaGkgjSe3NyPM1axLIyswaCRK8XZO3D2kg2PvBZGzI8RONiV/Ay+PEpK01OnezgGo8KyyVfw\nCj1Pi19HYLSN57GfJvDTialE6cIY5L8VuyoOnI62Y/v8KMbOr0qd4FzmjYnVZiIJCeKNWiU861YU\n5Ziqqo3/63Fy5S1JUsVSo4aoe1qgYUbz33hYudAoai8PpYYwuNZFRn7RDUOeyHYPD86jWZ1sfjrg\nVGrPX1Y8XIx8MPgGufkKr6+7SMuDfTFZZtB8wxx+PDGdKwTRxvQzR2N92H3MCSd7I23frEGzOtl8\nPEKjwJ2VBc7O4OurzXgPKHnmLUlSxWJtLVZlx46Bl1epPU1+lhOD3bfRIv9F5lY9QcS6jqzvvBp3\nF188nA1k5TyYSVQ2VibCenzEqcB38deHsMFxCE3ixlDo7caapuvwPW/P18OjCfXPIy33UW7wAAAV\nRklEQVTTglD/fOxtNdqBMJlEhnmLFqJIj/SvZPCWJKni8fQU3cdSU8HFpVSeolWDLLb+7sG3M5fS\n4McPGOiyiSbbn2dDs4X8fLgek169VSrPW1ryCxS+/cWVP87a4epo5JGGWWzd58yxC3a4ORno80wK\nT7WKY4PjCE4EbqGr1xOscOmLw6Bh4OCA5dJFpK30pvPj6bRtfqeXqsZdw1JSxK0CZ2dtx30Aybc2\nkiRVPHc6j5lMpbZ9/nKHZI5dsGP2Ki96Pv0uvzsNJ08t5JGTA6nZ9n283Ao4fsGW63GWpfL8RZWe\npWPdDleWbXbnyo17l3e9etOKOi/WZsWP7gR4F3Duig1th4RyItKW916/SefH0xi/IYVq37/Ojxlb\nCL74Dg1/fB/7IaNEouBnn7HjahgbdrnyWpek0vlC8vLEcwUHl874DxiZsCZJUsUVFwfHj5fa9vmV\nG1a89mEAp6NsCfApwDFhO/r2L7Ir2IhL9GP4nVpB/HVf6tfIZcmEawRXLb1z+Hv57FsPJiz0o1WD\nLJzsjew46Eynx9JYPOEalnftq7Z6JYwurVMZ3TsBoxFCutRh6uAbTFjox+oPLnPC5SveOj8fY7Yz\n77hNZ7ifD4YBr6MUFjCz1Sb2ptUlNsGKNVOv8EhD8xvF3FN8PDRuLHZVKrGiJqzJbXNJkiouLy+R\n2JSUVCr3gYP8Cti1KIprt6yIT7YgpFowc6f8zBOXOvH+E3tICq3LFw2mcml3F54YVINTa8/j4vhn\nExVVhe0HnFi1zZ3UTD1NamczsGsSfp7mbzfvOOjER8u9Obrq/P/eNGTn6nhhXHUmfebHR0NvACI7\nPibOihEvifvX+085YGtt4uxlW9z8Yuh4ZDgZvjvp6PkIrQvmEPd9Dq5XuoN1DpHvL6dOvjdPuN2i\nTdOMOwn32ktLAx+fSh+4i0Num0uSVHEpiuj7DaVaES3Ap4CmdXLIytWx8OSjDBu2hiM/VaVKXCad\nTwxnpd1gQsKjWb7F/X+fo6owcJo/o+dW5bFGmbzZPYGUdAsa9qqlSd/wuWs9mTr45l9W+/a2Jha/\nHcPSHzzIyRPNtK/dsqJ29dz/FSlbu8OVqFg9h+2Xcu6ZRmRU2YfDr58y3moJjyae4L0/nhGTX7yY\nsLYB9OmYQrsWpRi4DQbRNa5mzVJ6ggeTXHlLklSxWVtDw4Zw6BBUqVKqpTT3n3TgicaZ2AT58rnv\nXiYuH0tUk2+Z+uhPFDTbzYnj7zDY8CTWFhZs+d2Zg6cdOLLywv+ysZ9plcHjD2XSZ3IQ5789i6KU\nfC4R0bY8NiHzH78f4FOAm5OBzXucWb/TjX0nHUjNsGDkbD/qh+Wy+uw5TK8P5VePM4QVtKZe9HT6\nP+/GthHb+CB7FHFutbFbPvP+rYKTk6FePdl4pJjkyluSpIrP3V2swJNKKZnqNntbEykZej5Z48Xp\nG+503PE2E5uO4fxn8PR1ldTm7+C/tRdrb2xnxVYXRvaM/8c1quefTENRVI6cNW/17e1eSFTMP8vE\nZmbruJloyfDZ/rR7OIMfP4nGwc7IvN2JDDg3jqwerdHZplHv2GpSPtvOhKd0tNszng+zRrBD1wF1\nydL7F7gzM8W/nbzTXWxy5S1J0oMhKAjS00UAL6Xqa22bZdD//UDOX7Fh69xo7O1U6NEDv5BarBjy\nDvtrGBj4ZC49T0zEruFKwlxfx6g+gl75czdAUcTq+Hq8JU4O1ni5GXB1Mv77k/6LAc8l88EXPrRq\nkIW11Z+Jx9OXeaOqChtnXaJRzRzC3lDxGdyVdPstkO+A+6lx2B0dS0SsG+0tduI95DXU/OvMtxzD\nrReH0z4wXou/qv9mMEBurkhSk3e6i82svzFFUbopinJWURSToij/mh2nKEo7RVEiFUWJVhRlvDnP\nKUmSdE+KAuHhYhs9q3Qyom1tVOaMvM6tJEv2n3Lg3GUbftzrTOtF3ZnY8lc6OzQkbuFN1p+ogaOF\nkekZYwjb/TwfRS/nVp7YFUhM0fPbMUdemRLIc6NCCHquDj3fCSIprXjb/a93SaSKq4HGL9di/roq\nLN/izjPDQ/h6uxsBVTOJ99pCs52jud61MTecf2VCyAAe/f00BTumMqnrTVZav8bWwqdxdNZz4Z1V\nLPabQodH72PDlZQUsVvi4HD/nvMBYtZVMUVRagEmYAkwRlXVf9zrUhRFD1wE2gKxwBHgJVVVz/3X\n+PKqmCRJxZaZCfv3i+ItltrfwVZVcH+yPs3qZHHlhg1VXAvp1zGZfs8mc+GyFWtf28VU00QMBgN9\na3Um4oVLRBiPoVf0tPdoxcGVQ7C92ZojS2/g7WEgPUvHe0t82XPckV8XRxIRbYeNlYlGNXP+8/je\nZIKdh534ZpcrOfkq1Rsf5ILDZjanbabQJhG7Ah9a67qzssNzuFk5s/FXZ74cH8V3TgOwSkvgRJPX\naDTnZWauD2DDLleOrrpg1jl8kWVliX+b5s3lqvtv7stVMVVVz99+sv/vYU2BaFVVL99+7DqgE/Cf\nwVuSJKnYHB2hQQNx/9vTE62jkaLAkG6JnLlkQ8T6s/+7T11ogPGLqtG454vQuSYWc+bw9a4NRJ+p\nwXthCzjcNIqtWetQ23dHjwWPH3yI9l4P0yO0Ae8PzaLe802p9kw9wqvnkpmjJzdfx4KxMXRolXHP\neRhVI5dzbpBQLYLcZw6yM+kQSQVpWORboFzvyLL2bTi5rQvuTipuVnGQkYHPwtlsNa0jMrMWwx2/\npUCtS0JfS6wsVTZ9HH1/ArfRKEqgtmolA7cZ7seZtx9w/a5fxwLN7sPzSpJUWXl7Q/XqcPWqyEDX\n2MRXbvH8W9Vp0LM2vdqloKqw+ic3avjn83b/OLD0hI8+ggMHCJ45k9Xn3iQlqz4L3efwQYYXvi1+\nJMF7F3MT5jL3dvtr/UsBeBlr0LyRB87/1969B0d5nXcc/z66YwlJyAjLXAQSCCMQ4RbMxcZJ8aQm\nHjeXGdJSZ5yMW0/Suq7Tf1q3yUzqjjtNOpM/0thpO7FDPG5n4qZJ2kRN6ozTjEsal8TYg7n5EsAR\nBWx0wwIJJNjV6R/PqsgUWQsr7fu+u7/PzDuw7CIdPYj56T3n7HPKauh5axY7vtPKH5X20dh0moHU\nIGcuDnFypJeDZ4/w6mAXw6MjAMypaGBb4ya2NW7m1xs38vipdr7yyCz++J63+OyjTTxY/xSlj36Z\n9UN9DO64j/Rd9/PT+97Dw5vfZP3yId63bjA/wQ2+u3zpUv8hS67ZpNPmZvZjoOkKT30uhPC9zGue\nY+Jp8+3AthDCfZnH9wAbQggPTPD5PgV8CqC5uXldV1dX9l+NiMiYdBr27PEp2mnofx4CPPdiDZ27\n6jGDD932NretvUIIXrwInZ1c+NqTVPSe5LWaddz0ue2wZQtdqUE++fhb/OzkMVpWv8TpqtcYqT7G\n2dSV156rSiqZUzmL5TWtrJjp1+rapayqXUqJXbqLDQEe+4c63tj5E+4f+hJLwi95tWIlb/3BIxws\nX8NffaOJv/j0Se77SN+U1+VdDQ760aobN07rW/qSLNtp8ylpjzpJeG8CHg4h3JF5/GcAIYQvTPZx\nteYtIjkZHvb176oqvyJ0/yNNbD31Tdb//KssDL+C6mrYupUjHb/Bsi/cy9ymNN/64lE2dJxjNIwy\nmDrHoZMX+MhDCzjw1Alqy6qpKJlkDT8EeP11eOYZ6OyEt99muHUZz697kC8d38HxnkraW4Z58Le6\nuWV1Hjenge8u7+/36XLddU8oTu1RXwDazKwFOAHsAO7Ow+cVkWJXVQVr13oDl7Iypq9N2OT2vlHH\n3Q/exT+t/RgH/vkQDzfvZNF/PMPizk6O8Xn2pj/AzbsqYF8dJfX11NbXU9PXxMqRBcxOVfCOZuXj\n9fXBCy/417h7t79VrrQUtmyB7dup2rCBrWZs5Wh+v+DLje0uV3BPiZy+k83so8CjQCPwAzPbG0K4\nw8zmAk+EEO4MIaTM7AHgR0ApsDOEcDDnkYuIZGPWLD+B7MABXwuPyFhTlT/5nV46l7Tx6ae/zq9q\nAtuv72Tj8W9z+9kfYk+e9i3kGR3AswC3ARUVflRmfb1f6TQcPep9wQFqa2HDBti0yYN7Gnq9X7OB\nAX/vfXNz1CMpGDpVTEQKXwiwf7+fQnb99ZO/fhp07qrjoUfnsfsbr1JbcymgH/+X2fzN0430D5Tx\nm7f3sWNTFyPdZ+j8t4ANvM1f/vYrVJ477SE9/gLflNfa6u1F29vjuY588aKH95YtcF3uPd0LXZym\nzUVEojV2gMmZM75pKoLGIHdtGeDHv5jJmo8v5/e39zC38QI/+K86nntxJs9+9ZfUz0zz2Lca+czj\nK5lROcrHtp3m3g/1Ulm1JO9jnTIh+HT5qlUK7immO28RKR6Dg76BrbbWp6HzLAT42cvV/GPmiNBN\nK4f45F1919QeNRH6+/3Y1pUrp/z99oVKd94iIperqfEGLnv2eAOXPDcJMYNbVw9xa753ekfh/Hnf\nILhsmYJ7Gqi9jYgUlxtugLa2aT+BrKil096mds2aSGY4ioHCW0SKz5Il3nltbOOXTJ0Q/Aej9vZp\naY4jTuEtIsWnpOTSOuz581GPprD098P8+bBwYdQjKWgKbxEpTpWV3sDl7Fnv/iW5O3vWO8ctX651\n7mmm8BaR4lVf73fgvb0+3SvXbmTE39O9Zk2kneyKhcJbRIrbvHk+xdvfH/VIkiudhtOnfSZD7+fO\nC4W3iBQ3M387U3W1N3GRq9fbCytWRNa9rhgpvEVEysp8ujeV8ulfyV5f36XZC8kbhbeICPh07+rV\nPv077nAQeRfaoBYZhbeIyJjGRp9CVwOXyY1tUFu9GsonOWdcppzCW0RkvJYWPzq0pyfqkcRXKuUN\nbtau9TtvyTuFt4jIeCUlfsRmY6Ov58o7pdM+M7FypTaoRUjhLSJyudJSP8ayoUEBPl467TMSHR2w\nYEHUoylqCm8RkSspK/MAr6vTe8DBN/H19PjmNO0sj5zCW0RkIuXlvq47c2Zx34GPBfeyZb4nQCKn\n8BYReTfl5bBunU+hF+MmtrHgbmuD1taoRyMZCm8RkcmUl3sTlxtvhFOniqcPegge3IsX+zGqei93\nbCi8RUSyUVrqO6wXLYLu7sJv5BKCf50tLbB0qYI7ZhTeIiLZKimB9nafQu7uLtyjRMeCe9EiX+dW\ncMeOwltE5GqYeXh3dPgmtuHhqEc0tdJpXxpQcMeaDl0VEbkWCxf6LvSXXvIAr6+PekS5u3DBe7t3\ndEBzs4I7xnTnLSJyrRoa4JZbvEVoT0+yN7KdO+dHoq5f7z+YKLhjTeEtIpKLGTMuBd6pU35YR5KE\n4NP/qRRs2uRtYSX2NG0uIpKr0lLfyFZXB/v2QWUl1NZGParJnT8PAwO+o7ytTaeDJYjCW0Rkqsyd\n6wF+4IDfhTc0xDMQR0e95WtVFWzeDLNmRT0iuUoKbxGRqVRdDTffDCdPwqFDPi3d0BCfNeShIRgc\n9MYrixd7D3dJHP2riYhMNTOYNw9mz4bDh6Gry9fGo5xKT6f9bru62u+2C2F3fBFTeIuITJfKSlix\nAubP9xDv7vap6nyG+Oior2tfvOgtTltbfY1eEk3hLSIy3erq/HCTgQE4csTXw0tLPcSna008lfLP\nNzrq79leuNDvuqUgKLxFRPKlrs6PGB0c9AA/dsybopSXe8OXXNefUyn/2Bcu+MdsbfXp+xkzpmb8\nEhsKbxGRfKup8au11e+O33wTTpx4Z6/0sQ1uIfhd+thVVua/lpT4VPjIiId1CD5NP3cuzJnja9qa\nHi9YCm8RkaiYecjW18NNN3l4X+kaHvaQHh72oB4L7Ouug6Ym//t1df44LrvaZVopvEVE4qCkBCoq\n/BKZhNqjioiIJIzCW0REJGEU3iIiIgmj8BYREUkYhbeIiEjCKLxFREQSRuEtIiKSMApvERGRhFF4\ni4iIJIzCW0REJGEU3iIiIgmj8BYREUkYhbeIiEjCWAgh6jFMyMx6gK5p+NCzgd5p+LjFQvXLjeqX\nG9UvN6pfbqa7fgtDCI2TvSjW4T1dzGxPCOG9UY8jqVS/3Kh+uVH9cqP65SYu9dO0uYiISMIovEVE\nRBKmWMP7a1EPIOFUv9yofrlR/XKj+uUmFvUryjVvERGRJCvWO28REZHEKujwNrNtZvaamR02sz+9\nwvNmZl/JPL/PzNZGMc64yqJ+H8/Ubb+ZPW9mq6IYZ1xNVr9xr1tvZikz257P8cVdNvUzs/eb2V4z\nO2hm/5nvMcZZFv9/68ys08xeztTv3ijGGUdmttPMus3swATPR58dIYSCvIBS4AjQClQALwPLL3vN\nncC/AwZsBH4e9bjjcmVZv83ArMzvP6j6XV39xr3uJ8APge1RjzsuV5bff/XAIaA583hO1OOOy5Vl\n/T4L/HXm941AP1AR9djjcAG3AWuBAxM8H3l2FPKd983A4RDC0RDCBeBp4MOXvebDwFPB7QbqzezG\nfA80piatXwjh+RDC6czD3cD8PI8xzrL5/gP4Q+A7QHc+B5cA2dTvbuC7IYRjACEE1fCSbOoXgJlm\nZkANHt6p/A4znkIIu/B6TCTy7Cjk8J4H/M+4x8czf3a1rylWV1ub38V/EhU3af3MbB7wUeDv8jiu\npMjm+28pMMvMnjOzF83sE3kbXfxlU7/HgHbgJLAf+EwIYTQ/w0u8yLOjLJ+fTAqTmf0aHt63Rj2W\nhPky8FAIYdRvfuQqlQHrgNuBGcB/m9nuEMLr0Q4rMe4A9gJbgcXAs2b20xDCmWiHJdko5PA+ASwY\n93h+5s+u9jXFKqvamNl7gCeAD4YQ+vI0tiTIpn7vBZ7OBPds4E4zS4UQ/jU/Q4y1bOp3HOgLIQwB\nQ2a2C1gFKLyzq9+9wBeDL+IeNrM3gGXAL/IzxESLPDsKedr8BaDNzFrMrALYAXz/std8H/hEZufg\nRmAghPBmvgcaU5PWz8yage8C9+hu5/+ZtH4hhJYQwqIQwiLg28D9Cu7/k83/3+8Bt5pZmZldB2wA\nXsnzOOMqm/odw2ctMLMbgJuAo3kdZXJFnh0Fe+cdQkiZ2QPAj/CdlztDCAfN7Pcyz/89vsP3TuAw\ncA7/SVTIun6fB64H/jZz95gKMWjYHwdZ1k8mkE39QgivmNkzwD5gFHgihHDFt/YUmyy//x4BnjSz\n/fiu6YdCCDptDDCzbwLvB2ab2XHgz4FyiE92qMOaiIhIwhTytLmIiEhBUniLiIgkjMJbREQkYRTe\nIiIiCaPwFhERSRiFt4iISMIovEVERBJG4S0iIpIw/ws0GkYt9oWDHwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x14781497c50>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "N = 101\n",
    "plotBayesianEstimator(N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this graph we can see that the mean is really close to the original function sin(2pix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1256,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAe8AAAFpCAYAAAC1YKAIAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzsnXlYVVX3xz+H8TLP4ICKIgooiIozvqml5dhrWTllatbP\nzJxyyqnUNFNTc0gr8zXLLMsmbTC1zKFMcELAAQdUBBGVQebp/P7YgkOoDHeE/Xme+8i995x91oXr\nWXuvvdZ3KaqqIpFIJBKJxHQwM7QBEolEIpFIyod03hKJRCKRmBjSeUskEolEYmJI5y2RSCQSiYkh\nnbdEIpFIJCaGdN4SiUQikZgY0nlLJBKJRGJiSOctkUgkEomJIZ23RCKRSCQmhnTeEolEIpGYGBaG\nNuBBuLu7qz4+PoY2QyKRSCQSvXDo0KFrqqp6POw4o3bePj4+REREGNoMiUQikUj0gqIoF8pynAyb\nSyQSiURiYkjnLZFIJBKJiSGdt0QikUgkJoZR73lLJBJJVSA/P5/4+HhycnIMbYrESNBoNHh7e2Np\naVmh86XzlkgkEh0THx+Pg4MDPj4+KIpiaHMkBkZVVa5fv058fDz169ev0BgybC6RSCQ6JicnBzc3\nN+m4JQAoioKbm1ulIjHSeUskEokekI5bcieV/T5I5y2RSCTVAEVRGDx4cMnzgoICPDw86NWrlwGt\nklQU6bwlEomkGmBnZ0dUVBTZ2dkA7Nixg9q1axvYKklFkc5bIpFIqgk9evTgp59+AmDTpk0MGDCg\n5L3MzEyGDx9O69atad68OT/88AMAcXFxdOzYkRYtWtCiRQv++usvAHbv3k2nTp3o168f/v7+DBo0\nCFVV9f+hqiky21wikUj0ybhxcPSodscMCYFlyx56WP/+/ZkzZw69evUiMjKS4cOHs3fvXgDmzZtH\nly5dWLduHampqbRu3ZrHHnsMT09PduzYgUajITY2lgEDBpTIVh85coTo6Ghq1apFhw4d2L9/P2Fh\nYdr9bJJSkc5bIpFIqgnBwcHExcWxadMmevTocdd7v/32Gz/++COLFy8GRIb8xYsXqVWrFqNHj+bo\n0aOYm5tz+vTpknNat26Nt7c3ACEhIcTFxUnnrSek85Zoj4wMSE8Hc3OwsCj9XzO5UyOp5pRhhaxL\n+vTpw8SJE9m9ezfXr18veV1VVbZs2ULjxo3vOv6tt97Cy8uLY8eOUVRUhEajKXnP2tq65Gdzc3MK\nCgp0/wEkgHTeEm1QWAhxcXDq1MOds5kZWFuDpSVoNOJfa2uwshL/Psjxy1IbiaTSDB8+HGdnZ4KC\ngti9e3fJ648//jgrVqxgxYoVKIrCkSNHaN68OWlpaXh7e2NmZsann35KYWGh4YyXlCCdt6Ry3LgB\nx49Ddja4uwsn+yCKisSjoAAyM4XjL36oqngoyr//BeHoLS3B1hZq1xbXs5BfYYmkPHh7ezNmzJh/\nvT5z5kzGjRtHcHAwRUVF1K9fn23btjFq1CiefvppNmzYwBNPPIGdnZ0BrJbci2LM2YGhoaGq7Odt\npOTmQmwsXLgATk5gY6P7axY7+fx84fjNzaFePahVCxwcdH99iaSCnDhxgoCAAEObITEySvteKIpy\nSFXV0IedK5ctkvKhqpCYCNHR4rmXl/7C2ebm4mFlBXZ2YvV+8SKcOwfOzlC/vlyNSySSaoG8y0nK\nTkYGxMTAtWvg6ipC2IbEwkLYAZCVBUeOCOdet64Iq8vVuEQiqaJI5y15OHcmpNnYiNW2sWFrKx6F\nhXDpEpw/L8L59euDh4dcjUskkiqFvKNJHkx5E9IMjbn57dV4drYQwzA3Bz8/4cglEomkCiCdt6R0\n7k1I8/AwtEXlx8ZGPAoL4cQJUaZWr56hrZJIJJJKI5235G4MmZCmK8zNRdQgKkrs09eqZWiLJBKJ\npFJIuSvJbTIyICJChJodHET42dQddzHFDvzoUUhONrQ1EolRMGLECGJiYh563LJly9iwYQMAkyZN\nwt/fn+DgYPr27UtqaupDz2/fvv0D3z9+/DhDhw4tk80SgXTeEhFWPnsW9uwRDtzLy/CZ5LrAwgJc\nXMQEJSXF0NZIJAZn7dq1BAYGPvCYgoIC1q1bx8CBAwHo2rUrUVFRREZG0qhRI955552HXqe4E9n9\nCAoKIj4+nosXL5bd+GqOdN7VnRs3YN8+sb/t7g6Ojoa2SLdYWYnPGB4udNglkmpCZmYmPXv2pFmz\nZjRt2pSvvvqKTp06lXQIs7e3Z/r06TRr1oy2bduSlJQEwO+//06LFi2wuFWx0a1bt5Kf27ZtS3x8\nPADr16/nySefpFOnTvj5+TF79uySa9vb2wPw3Xff8eijj6KqKomJiTRq1IgrV64A0Lt3b7788kv9\n/DKqAHLPu7pSFRLSKopGIyRaw8OhbVsh+CKR6Ilxv47j6BXttgQNqRHCsice3PDk119/pVatWiX9\nvNPS0li9enXJ+5mZmbRt25Z58+YxefJkPv74Y2bMmMH+/ftp2bJlqWOuW7eO5557ruT5wYMHiYqK\nwtbWllatWtGzZ09CQ2+LhfXt25ctW7awatUqfv31V2bPnk2NGjUACA0NZcGCBUyePLnCv4fqhFx5\nVzdUFRISRIg8MVGEyPUhbWps2NqKMHpEBOTkGNoaiUTnBAUFsWPHDqZMmcLevXtxcnK6630rKyt6\n9eoFQMuWLYmLiwMgMTERj1Im9/PmzcPCwoJBgwaVvNa1a1fc3NywsbHhqaeeYt++ff86b8WKFbzz\nzjtYW1szYMCAktc9PT1JSEjQxketFmhl5a0oyjqgF3BVVdWmpbyvAO8DPYAsYKiqqoe1cW1JOcjI\nECVTycnGoZBmaOztITUVDh2CVq1ESF0i0TEPWyHrikaNGnH48GF+/vlnZsyYwaOPPnrX+5aWlii3\nElTvbO9pY2NDzj0T3PXr17Nt2zZ27dpVcg5w18+lPQeIj4/HzMyMpKQkioqKMLvViTAnJweb6riQ\nqCDaWnmvB554wPvdAb9bj5eB1Q84VqJtqktCWkVwdhbSqkePCq10iaSKkpCQgK2tLYMHD2bSpEkc\nPly29VNAQABnzpwpef7rr7+ycOFCfvzxR2xtbe86dseOHdy4cYPs7Gy+//57OnTocNf7BQUFDB8+\nnE2bNhEQEMCSJUtK3jt9+jRNm/5r7Se5D1pZeauqukdRFJ8HHPIksEEVLcwOKIrirChKTVVVE7Vx\nfckDMDWFNEPg6ir02o8fh+Bg+TuSVEmOHz/OpEmTMDMzw9LSktWrVzNx4sSHnte9e3eef/75kuej\nR48mNzeXrl27AiJpbc2aNQC0bt2ap59+mvj4eAYPHnzXfjfA/Pnz6dixI2FhYTRr1qxkXzwgIIA/\n/viDnj17avETV230lbBWG7h0x/P4W69J560rjCwhrVAt5HxWAnFZCaQVZJBekElafga5RXk4WNjh\neOvhauWEv70P7lbO+jXQ3R2uXBH74E2bVp36donkFo8//jiPP/74Xa/t3r275OeMjIySn/v160e/\nfv0AqFevHm5ubsTGxuLn53fXKvxevL29+f777//1evHYs2bNKnnNwcGBkydPApCbm0tERATLlhlm\nS8EUMbpsc0VRXkaE1qlbt66BrTFBVFU4oago8dxACmnnsy6zPflv/roRSXTGOU7cPE92UW6Zz/e0\ncqWJQwOCHf3o4hZKZ/dQHCx0nBXu4QHx8WJLoXFj6cAlklssWLCAxMRE/Pz8dDL+xYsXWbBgQUkJ\nmuThKCKSrYWBRNh8230S1j4EdququunW81NAp4eFzUNDQ9XiGkRJGTBgQpqqquy9cYQtib/za/Jf\nnM4UYgs1rd0JcmxIE/sGNHFoQEO7OrhYOuJoYYeThT3W5lbcLMgkPT+T9IJMrubdIObmeaIzzhJ9\n8xxRN8+SVZiDpWJBB9dmPOHRjoG1n6COTQ1dfRBISgJ/f/D11c01JNWOEydOEBAQYGgzJEZGad8L\nRVEOqaoaep9TStDXNOdHYLSiKF8CbYA0ud+tRYpbdp4+LWqY9diy80rONT6N38Ynl34kNvMiGjNr\nOrm1ZFS9Z3jcsx2N7eqVmnF6J7bmGrys3Uqed/e8neSSW5jH/pRjbE/+m1+v/s3Ukyt54+QqnvBo\nx4i6/6WXV0eszLQ4SVEUsQI/eRKsrcHbW3tjSyQSiZbQVqnYJqAT4K4oSjzwJmAJoKrqGuBnRJnY\nGUSp2DBtXFfC3Qlpbm56S7aKuXmOubFr+TpxF4VqIR1dmzO94XD61XwUOwvtlXtYm1vRxb0VXdxb\n8W7AGM5lxvO/+K3879JWnj40GU8rV8Y1GMBon2e1F1Yv1kGPjBR74DV0tMqXSCSSCqK1sLkukGHz\nB1CckHbxopD71FN9ZPTNs8w5vZavE3diZ27D/9V7ipfq/pfG9j56uX4xhWoh26/+zcq4zfyS/Beu\nlk683mAQo32exdHSXjsXyc8XGuitW4uJkURSQWTYXFIaphA2l2iLexPSPD31kliVmHONySfeZ+Pl\nX7Ezt+GNhkOZ0GAQbvrOCr+FuWJOD68weniFEZ4azZzTa5l+6gPeO7eRtxq9xCv1+mFhVsmvt6Wl\nyNSPiBAyqvcoUkkkEomhkPKopsSdLTvt7fXSsrOgqIBl576g8e6n+TpxF1MbDiXu0R+Z5/+qwRz3\nvbRybsLW1ksJD9tASyd/xkQvptW+IfydEln5wa2thfb5wYPi9y+RmChXrlyhf//++Pr60rJlS3r0\n6MHp06crNNbevXtp0qQJISEhXL58uaSs7F7ubHyiT2bNmsXOnTsfeMzu3bsf2u1MG+zevbtEdlab\nyJW3KVBYKOq1T50SCWmennq57N8pkYyMfIfIm7E84dGeFU0n0dCujl6uXRFCnQPZ3mYlWxJ3MS5m\nCe33D2d4nT4sChiLq1UlVs02NuJvUNzIREo4SirLgQNCmldbODuL7+Z9UFWVvn378sILL5R07jp2\n7BhJSUk0atSo3JfbuHEjb7zxBoMHDwbgm2++qZjdOmLOnDkPPWb37t3Y29s/tNf4nRQUFBhNOZtc\neRs7xS07T58W+656aNmZX1TAzFOrCds/gpT8dLa0XMjPrd83asddjKIo9Kv1GCc7fcOkBs+zIf4n\ngv7sz65rBys38K2WhoSHi3wDiaQypKaKqgZtPR4yEfjjjz+wtLRk5MiRJa81a9aMjh07oqoqkyZN\nomnTpgQFBfHVV18Bwrl16tSJfv364e/vz6BBg1BVlbVr17J582ZmzpzJoEGDiIuLK5E1zc7Opn//\n/gQEBNC3b1+ys7NLrvfbb7/Rrl07WrRowTPPPFMi3OLj48Obb75JixYtCAoKKhFuycjIYNiwYQQF\nBREcHMyWLVseOM6dDB06tGRCUdr4cXFxrFmzhqVLlxISEsLevXtJTk7m6aefplWrVrRq1Yr9+/cD\n8NZbb/H888/ToUMHnn/+edq2bUt0dHTJtYqjCwcPHqRdu3Y0b96c9u3bc+rUqfJ9J8qJdN7GSm6u\n2Nc+cEBkP3t46CWT/EzmJcL+epG3Yz9hiHdPojtt5qmaXR5a7mVs2FvYsjBwLP+EfYqDhS2PHRjF\npJj3yS3Mq/igjo4iie3wYfGvRGIiREVF3bet57fffsvRo0c5duwYO3fuZNKkSSQmikreI0eOsGzZ\nMmJiYjh37hz79+9nxIgR9OnTh0WLFrFx48a7xlq9ejW2tracOHGC2bNnc+jQIQCuXbvG22+/zc6d\nOzl8+DChoaF36Zq7u7tz+PBhXnnlFRYvXgzA3LlzcXJy4vjx40RGRtKlS5eHjnM/7h3fx8eHkSNH\nMn78eI4ePUrHjh0ZO3Ys48ePJzw8nC1btjBixIiS82NiYti5cyebNm3iueeeY/PmzYDouJaYmEho\naCj+/v7s3buXI0eOMGfOHKZNm1aOv1D5MY71v+Q2dyakqareEtIAPr20jVej3sXKzJLNLRbwTK3H\n9HJdXdLCyZ/D/9nIxJhlLD73GTuv/cOXLeZXPDve2RmuX4djx6B5c6mDLjF59u3bx4ABAzA3N8fL\ny4tHHnmE8PBwHB0dad26Nd63tA5CQkKIi4sjLCzsvmPt2bOHMWPGABAcHExwcDAABw4cICYmpqRR\nSV5eHu3atSs576mnngJEK9Jvv/0WgJ07d5aE+AFcXFzYtm3bA8e5H6WNfy87d+4kJiam5Hl6enrJ\nqr5Pnz4lHc+effZZunXrxuzZs9m8eXPJfn9aWhovvPACsbGxKIpCvo4n+NJ5GxN3KqS5uOitRWV+\nUQHjo99j1YWv6eTWks9C5uBtoz+hF11ja67hg6CpdPdsz/Bjc2i97wW+aD6Pnl73vwk9EDc38TeK\nioKgIDCTASyJcdOkSZMK7UtbW1uX/Hxnm9DyoqoqXbt2ZdOmTQ+8zsOu8bBx7kdZxi8qKuLAgQNo\nNJp/vWdnd1tDonbt2ri5uREZGclXX31V0pRl5syZdO7cme+++464uDg6depULhvLi7zrGAOFhXDu\nHOzde7tlp54cd3JuCl0PjGLVha+Z2OB5drRZVaUc95309voPhzp+TkO7OvQOH8/82HVUWOfAwwMS\nEoQSmxFrJUgkAF26dCE3N5ePPvqo5LXIyEj27t1Lx44d+eqrrygsLCQ5OZk9e/bQunXrCl3nP//5\nD1988QUgQvWRkaLio23btuzfv7+kqUlmZuZDM927du3KqlWrSp6npKRUaJz74eDgwM2bN0ued+vW\njRUrVpQ8P3r06H3Pfe6551i4cCFpaWkl0YW0tDRq164NiH7nukY6b0Nz4wbs36/XhLRijqadInTf\n8/yTGs3nIXNZFDi28rXRRk5dmxrsbb+W/rW6Mf3UB/Q/PI3MguyHn1gaHh5CljY2Vqs2SiTaRlEU\nvvvuO3bu3Imvry9NmjThjTfeoEaNGvTt25fg4GCaNWtGly5dWLhwITUqqCr4yiuvkJGRQUBAALNm\nzSrZZ/fw8GD9+vUMGDCA4OBg2rVrV5KYdj9mzJhBSkoKTZs2pVmzZvzxxx8VGud+9O7dm++++64k\nYW358uVEREQQHBxMYGBgyYq6NPr168eXX37Js88+W/La5MmTeeONN2jevHmFIxTlQSqsGQoDKaQV\ns/3q3zx9aDIulo58H7qYls7VS/1JVVUWn/uMKSdW0Mo5kJ9av1+xNqRFRXD1KjRpAj4+WrdTUjX4\nl5KWnkvFJMaJVFgzJQyYkFbMxvhfGHrsLZo4+PJL6+XU1Ljr9frGgKIoTPIdQmO7ejx3eBod9g9n\ne+tVREb4s+5HdxKvWRLsl8XoZ5Np1ugBK3MzM7ECj44Wimy3wmYSyQORjlZSSWTYXJ8UK6QdOSLq\nht3c9O64l537gsFHZxLmGsKf7T6qlo77TvrUeIQdbVdxNTeFpr+9xISNKTzVOYVlr1/C1zuXbqP9\n2LLrISvy4kYmx46JVbhEIpHoGOm89YEBE9KKUVWVN06sZHzMEp6q0ZlfWi/HSVsNPEycMNcQlrp+\nTk6eGdf6dsG3/e+0C85k6tAkflkey//Nr0dWzkMmWRYWokLg0CHRzEQikUh0iHTeusaACWnFqKrK\npBPvs+Dsel6u25fNLRegMbd++InViAM72zE5fwteGlce/+c19t0QmaYt/LNp4Z/FL/vLIK9qZSX+\nvgcPQnq6ji2WSCTVGem8dcWdCmnF+6IGEPRQVZXJJ5bz3rnPebXeM6wJmoa5IoVF7iUtw5xAVy92\nt/uI2hoPuv8zhv23HHhN93zSMsr4O9NowNZWOPDMTB1aLJFIqjPSeWsbVYXERNizR9QBe3oarJGF\nqqpMObGcxec+49V6z7Ci6WSTkznVF22DMtm214maGnd2t/uIWhp3nvhnDLuvHuO3A460DSqHI7a1\nFclr4eGQk6M7oyUSSbVFOm9tkplp8IS0YlRV5Y2TK1l07jNG6cNxX78OS5fCokXw0Ufw5Zfw66/w\n998iEzs+Hm7eNFpBkxd6XWffMXvWfOOOp6U7f7T7kBpWHnT9ewx+7X8nsEE5nbC9vch1OHQI8iqh\npy6p1hQVaWec69evExISQkhICDVq1KB27dolz/PK+P0cNmxYpZpteHt7k/qA8riioiIWLFhQ4fGr\nG7LOWxvc27LTAPva9/LOmf8x7eQqXqnXj1VNp+jWce/ZA3PnCues0Ty477W5OTg4gJPT3Q9Hx9Jf\nc3a+XQev44nQyThrhr7lQ0KyFb7euRxJSKFoSGfMHK6yp/3HBDv6lX/QlBTxeVu2FEltkmpJafW8\n9yMjQ/x3WrdOzImbNIHXX4cXXtDOf4G33noLe3t7Jk6ceNfrqqqiqipmOpL79fb2JioqCmfn0qs3\nCgoKcHd3f6CDr2rIOm9DkpICx49DVpZYaRtBo4q1F79n2slVDKz1BCt1ueLOyhKr7e++g0aNYPVq\naNgQCgqEI09Lu/1ITxeiFOnpt5+npUFSkhCrSUuD7AfUU1ta3nbwfn7Qowe0aaNVh+jvk8uB9aeI\nPqsh6YYl/j455Nu9T/v9w3nin9fY3+ET6tuWs47bxeV2I5OQEKP4fkiMl7w8eOIJqFdPBK0aNIA/\n/4QxY0TwasYM7V7vzJkz9OnTh+bNm3PkyBF27NjB7NmzOXz4MNnZ2Tz33HPMmjULgLCwMFauXEnT\npk1xd3dn5MiR/PLLL9ja2vLDDz/g6el519jJyckMHDiQhIQEwsLC7pIi7t27NwkJCeTk5DB+/HhG\njBjB1KlTuXnzJiEhIQQHB7Nhw4ZSj5MI5Mq7ohhYIe1+fJf4B/0OTaGbR1t+aPUeVmaWurlQVBTM\nnCnuKM8/DyNHVr78LS/vtlO/91H8emqq2JZISxOTpe7doVcvMWnQEdE3z9Lxr5dws3Rif4dP8LR2\nLf8gyclCwKVJE9nIpBpS1pX3l1/CBx/A7t13f00SEyEwEM6cEV/7ynDnyvvMmTM0atSIgwcPEhoq\nFns3btzA1dWVgoICOnfuzIcffkhgYOBdztvS0pKff/6Z7t27M2HCBDw9PZk6depd1xk1ahTe3t5M\nmzaNH374gf/+97+kpKTg7Oxcco2srCxCQ0PZv38/Dg4O/1p5l3aci4tL5X4BRoRceesTI1BIux+7\nr0Uw4Mh0WjkH8k3Ld3XjuAsKRDzvk09EBv2aNSIkrA2srITYiftDhGPy82HfPvjpJ3G3+/xzsfLv\n1Qsef7zyd7d7aOLgy7ZWS3nswCi6HxzDH23X4FjeGnl3dzHRsbAAf3+j+c5IjIutW2Ho0H/P72rW\nhEcegR07oH9/7V7T19e3xHEDbNq0iU8++YSCggISEhKIiYkhMDDwrnNsbGzo3r07INps7t2791/j\n7tmzh59//hmAJ598EgcHh5L3li5dyo8//ghAfHw8Z8+eJSQk5F9jlHbcnbZWZ6TzLg+ZmRATo/eW\nnWUh+uZZnox4nQa2tfmp9TLsLHQQCbh4EWbNEhOX7t1h8mSxn6tvLC2hc2fxSE2F7duFI1+yBN5/\nH9q3h549oWNHsNZOPXt712Z80/Jd+kS8ztOHJvNz6+VYlqeJi6KIyc758+J74+urFbskVQtVvf+8\nzsxMN/med7a7jI2N5f333+fgwYM4OzszePBgckqpmLC6495X3lahO3fuZM+ePRw4cAAbGxvCwsJK\nvUZZj6uuyPhdWShWSNuzx2AKaQ/iau4Neh0cj625hu1tVuBWkQYbD0JVxb72wIEiMW/+fJFRYwjH\nfS/OzvDcc7BhA2zeDIMHi8TBqVPF5uH8+WK/WQt3vR5eYawNnsHOawd5LWph+duJFjvwkyfh0qVK\n2yOpevToIb7K9361rl6F33+HRx/V7fXT09NxcHDA0dGRxMREtm/fXuGx7mwPunXr1pL2m2lpabi6\numJjY0N0dDTh4eEAWNzKXymeCNzvOIlAOu+HkZJicIW0O4lPsmTXQQdOxokVZU5hLv+NmEhS7nW2\ntlpKHZuKtfK7LzduiFTXefMgKEiEqbt10+41tEWDBvDaayL2uHIlhIXBzz/Diy/Ce+9pxYEPrdOb\nqb5D+fDityw7/0X5BzAzEyH0yEix/SKR3MEzz4gA34gRItClqvDXX2Ie+tprYpdOl7Ro0YLAwED8\n/f0ZMmQIHTp0qPBYs2fPZufOnTRt2pRt27ZRq1YtAHr27ElWVhaBgYHMmDGDNm3alJzz4osvEhwc\nzJAhQx54nEQmrD2Y69eFQpqTk8ET0tIyzHjp7XrsCnckuGE2sZes8amVg+vQ/mxN/YUtLRfyVM0u\n2r3ovn0wZ46INrz6KgwYYHrJVpmZsGqVWJW/8opw5JWkSC3i2UNT+fbKH/zQ6j16e/2n/IPk54uJ\nYevWWt+jlxgf5SkVS0sTu1MbNogCjLp1YcIE+L//k6kSVQ2ZsKYrCgpEaY8RZJI/M8WXBrVzubjt\nOHY2RRQUQK/vN7I19Rdm+76mXcednS1KwL79VmRxf/CBTrO5dYqdHUycKErXVq8WuQpPPVWpIc0U\nMzY0n8OFv15mwOHp7O/wCc0cG5VvEEtLMSkMDxftIe9T+yqpfjg5idSNpUtFUYtGI5225N+Y2DKq\nehIebUvsJWtWTbmInY2QXPoheRfbNUupkzCQmqfGa+9i0dEwaJDY437+eTH9N1XHXYyZGbz5pkhk\nW7AAdu2q9JC25hp+bLUEF0tH+oRPIDm3Ap3ErK2FElt4uJhcSCR3YGamF20iiYkinbcJcOC4Hd3b\np5Xoe8TcPMfQo7Np6xzEZLc3+ee4FhLHCgpg7VoYPlxM91evhrFjjSoxr1JYWMC770LTpkLpQgvb\nMTU17nwfupik3Bv0PzyNgqKyZ9yWYGMjnHh4uBC9kUgkkjIgnbcJ4OxQSOI1UbOdlp/BfyMmYmdh\nwzct3+XaNXuc7Asrd4H4eHjpJVGz3bWrSEqrirWUNjYiFlmnjkjCO3my0kO2dA5gTdAb/H49nDdO\nrqrYIHZ2YnkVESEmTpIqiTHnF0n0T2W/D9J5mwB9Hknlz8MORJ214vmjMzmfdZmvWyzAJq8ma39w\nZ1D36xUbWFXh++9FItr58/D22+JhDCVgusLJCVasEJ9xzBitlGwNrdObUfWeYfG5z/gq4beKDeLo\nKJLYDh+L2D0UAAAgAElEQVQW/0qqFBqNhuvXr0sHLgGE475+/ToajabCY8hs8weRlCSkOD08DGfD\nLTb+4srI/evJaDuHcS7TaJr8Egs31OCpLim8Mzqh/AOmpIjyr927hULa7NlQQ8tlZsZMXJzIPLe3\nF2pxD1N1ewh5Rfl0/vv/OJp+mgMd1hPkWME8gRs3RFJd8+ZSB70KkZ+fT3x8vBQZkZSg0Wjw9vbG\n0vJuJcyyZptL5/0gjMh5b7/6N90PjqH+9X64//EJtT0KeOm/1+jeIb38g+3bJ0RW0tNFCdjAgaZX\nAqYNoqJE+Zi3t2hjWsmIQ0JOMi33Dsbe3JZDHT8rv4RqMdeuCSGg4ODq+XeRSKoxZXXe8s5gAlzO\nvsrgozNp4tCA44PH8s/603y76Fz5HXdOjkjaGjdOlCZt2CAUyaqrg2jaVPQfP39eFNJWclVUS+PB\n5hYLOJd1mZePz6t4iNTdXXSiOHHCaPufSyQSw1JN79qmQ0FRAQOPzCCrMIfNLRZga17BPZKYGFEC\n9vXX4t8NG0RrzepO27ZCiOboUZg+XWTdV4KObs2Z23gkXyXs4KOL31Z8IA8PIUUbG1speyQSSdVE\nOm8jZ3bsx+y5cZjVQVMJcKhf/gEKCsSe7rBhQnzlgw9g/HitNeyoEnTrJoRc/vxTaKFXcrU7teFQ\nunm0ZWz0exxLP12xQYp10GNjxf68RCKR3IF03kbMzuR/mBe7jqHevRni3av8A8THw8svi5rtLl1E\nCVjr1to3tCrw3HNCUPrHH4WcaiUwU8z4LGQOrpaOPHtoKjcLMis4kJlw4NHRcPlypWySSCRVC+m8\njZQrOdcYdGQmAfb1Wdl0cvlOVlXhhAYOhLNnRXLa/PkGb6pi9Pzf/wnp1PXrRY/wSuBp7coXzd/m\nTGY8rxxfUPGBzM3FHvixY6K1lEQi0R2V3DbTJ9J5GyFFahFDj80mvSCTzS3fKV9v7tRU0Wd7zhzw\n9xer7e7dpcZiWVAUmDJFRCmWLRM9witBJ/dQ3mz0Ehsv/8LG+F8qPpCFhSgfi4gQpWQSiUT75OWJ\n6iITQTpvI2Rl3Ga2J//Ne4HjaOLgW/YT//pLhH/37hUCJKtXQ82aujO0KmJuLoRqWrUSE6B9+yo1\n3LSGw+jg0oxRUQuIy6pAPX4xVla3G5mkV6A8UCKRPJgbN0RLNxNBOm8jIyr9DJNPLKenZxiv1OtX\ntpNycmDhQuGwnZzg009hyBAp8lFRrKxECZmfn1iJR0ZWeCgLMws+bz4XgOePzqJQrYSUrUYDtrZw\n8KBodSqRSLTHxYsybC6pGDmFuQw6MhMnC3vWNZuFUpZQ98mTolZ782Yhc/rZZ9C4se6NrerY24u+\njJ6eoi7+7NkKD+VjW4tVTaew78ZRFpxZXzm7bG1FO9GDB0X1gEQiqTzZ2UIcyYSQzttAqCr8EWHP\ntFW1ePPDmkTG2jDt5Coib8byv2az8LR2ffAAhYXwv//BCy+IVdiqVaLZhiwB0x5ubrBypXCWr70G\nV65UeKhBtbszoNbjvHn6Iw6mRFXOLnt78QU6fFjs00kkkspx7ZrJRSql8zYAmdlmdHvVj9EL62Jt\nqZKTa0aXhRdZev4LRtV7hh5eYQ8e4PJlkRm9ahV07iyS0tq00Y/x1Y3atYUDz8oSUrKpqRUaRlEU\nPgiaSm2NB4OOzOTg6SLW/eDG97udyMmtQDKhk5NYLRw+bFKhPonEKLlwQUyKTQjpvA3A1BW18XTN\nJ3JTDG++nMi0V05h8+wwNOl+BMTOvf+JqgrbtokSsNhY0UzknXfEjVyiO/z8RCvRK1dEXkEF+247\nWzrwQeO5nMm6RJdv/seeI/as+MqTur2C2La3An9DFxeRvHbsmIjESCSS8nPzpnhUosOXIZDOW89k\n5yh8/osri8ZeLonSTIhZQkLuVRbWnce6LXVKPzE1VSRPvfUWNGoEmzZBz56yBExfNG8uauVPnYJJ\nkyrctnP9iqdpmvwymcGrGTL6G3atjuXHJWcZPqceJ+MqsOXh5gbJyUL+tqioQjZJJNWaq1dNLmQO\n0nnrnWupFtjZFFHLQ9z8f0rax7pLPzKl4QsMbOrH+QSrf5904AD07w979sDo0bBmDdSqpWfLJTzy\niNA//+cfmDWr3M4yLsGK3Yfs2f3sEPzs6jL82BzS8zNoG5TJK/2S+eBrz4rZ5e4u+pKfOiUbmUgk\n5aGoSITMTVDASjpvPePhUkBWjhnxSZbcyEtjRORcghwa8qbfSxw6aYuvd+7tg3NyYPFi4bAdHITy\n19ChJjlLrDL06SOS13bsEH+bcjjL6HMaQgOzcLPT8GnIW1zKTuL1mGUAdGp5k8jYcojx3ImiiKz4\n8+fh3LmKjSGRVEfS0yE3VySlmhgWhjaguqGxVnmh53UmLPXG/JkpXMtL5ZfWy8nOsmHaqtq8+kyy\nOPDkSbG6O3dOrLpHjza5PZkqy5AhkJIiJFRdXYUmehmo4VbAmUvWqCq0cwlmku/zvHv2U56q2ZmL\nF56khlvFQvHA7UYmJ0+KG1HduhUfSyKpLiQkCF0HE0SuvA3A/Fcvc87hZ75M/JW2Nybwv48fpfHT\nTQgLyWBoj6u3V9jp6SLTeeJE6biNCUURiWs9eogtjC1bynRaC/8sbDVFbPhJlAHObvR/NHFowItH\n32bR19YMf7KSdaZmZiKEfvy46AcukUjuT2GhaN7k4GBoSyqEVlbeiqI8AbwPmANrVVVdcM/7nYAf\ngPO3XvpWVdU52ri2KZJjlkZ863H4FQXQJedVbNzz2fPRKRpbnYdX3hT6ul26wLRp4OxsaHMlpWFm\nJiIjaWmwYIHI+H/ssQeeoijw+dzzPD7aj53/OPJE+3Q6X1vBSqfeBPd5na5tJlTeLgsLkcR25Ij4\n2cOj8mNKJFWRlBThwE10G7LSzltRFHNgFdAViAfCFUX5UVXVmHsO3auqagX6WlY9JsQsFeHysPdp\n/vg1sW/6889C4hRERrnMJDd+LCzg3Xdh1CiYOVM48FatHnhKUMMcor6KYf1WN7buccLVKZRBdYex\n0ewTfr8exqPuWmjZamkpJn2HDkHbtnICKJGURnw82FQwz8QI0EbYvDVwRlXVc6qq5gFfAk9qYdwq\nyfarf7M+fitTfIfQ3MlfrNzeeAPefBMaNoQvvoBevaTjNhU0GlEDXqeOULg7efKhp7g6FTJh8FW+\nfOc8H0y9xMdhw/Czq8tLkfPILNCS5Km1tQgHhoeLGlaJRHKbvDxISjI5YZY70Ybzrg1cuuN5/K3X\n7qW9oiiRiqL8oihKEy1c1+S4WZDJy8fn4W/vw0y/EaLkqH9/+OMPod710UdC0UtiWjg5idwER0eR\niX7xYrlOtzHX8EnwTM5nXWbGqQ+0Z5dGI5x4eHiFhWUkkirJ9euiTMyEF0n6Slg7DNRVVTUYWAF8\nf78DFUV5WVGUCEVRIpKTk/Vknn6YdnIVl7KTWBswBc2ylcJh29mJBLVhw0x270WCKNVauVJsgYwe\nLYRTykFHt+aMqvcM75//kr9TKt7F7F/Y2Yn9+YgIUXookUjEBNuEV92gHed9GbhTFsz71mslqKqa\nrqpqxq2ffwYsFUVxL20wVVU/UlU1VFXVUI8qlGyz/8ZRVsV9zWjnbnR4bZFQSHvmGVFuFBBgaPMk\n2sDHR3QiS0kRK/ByhqsXBIymjo0XLx6bS26hFhuOODgIRbhDhyqsDCeRVBmys0XvbltbQ1tSKbTh\nvMMBP0VR6iuKYgX0B3688wBFUWoot/pbKorS+tZ1r2vh2iZBXlE+L0fOo06hPfOn7RL73MuXC7lT\nWQJWtWjSRPQCj4uD8ePLtdp1sLDjw6BpnMg4z7tnP9WuXc7OInR+5IhsZCKp3iQni2iUiVPpT6Cq\nagEwGtgOnAA2q6oarSjKSEVRRt46rB8QpSjKMWA50F9Vq4+O48Kjq4jJOM/qL29i36YjfPUVtG9v\naLMkuqJtW5g7VzQMmTatXM7yCc/29K/VjXln1nEqI067drm6iqjA8eNSB11SPVFVMbE28ZA5gGLM\nPjQ0NFSNiIgwnAFJSWKlUonw/elfPic4dxlPnjHnq0bToXdvk06SeCBFRSKL09xcPKrA7LZSfP21\nKCXr3VvUhJfx756Uex3/3f0IcWzE723XoGj7+5KUBPXqQWBg1f0uSiSlcfMm7NsnclRKIzkZnnhC\nvzbdg6Ioh1RVDX3YcVIeVVekp6MueIeRNXegqW3G+0+vhfpBhrZKd+TliX0kFxfxc07Ov3W/73xe\n7ODNzUW9dFV0+M88I34nH38sVr2vvVam07ys3VjoP4aXj89jffxWhtXpo127PD1FMwYLC9GhTjpw\nSXXhyhXxva8CVI1PYWyEh8Nbb7Gh1lX+aA9rAidRoyo77ps3hcNu1eruGW1RkQgZl/bIyRENAYof\neXmQkXG7L/W9jl9RhGMvdva2tqbhdF5+WTjwTz8VE5vBg8t02ot1n2TD5Z+YGPM+vTw74mHtoj2b\ninXQz54Vus7162tvbInEWCkqElnmJthBrDSk89YmubnwwQewcSPXGnnzel87Ojj58lL9pw1tmW5Q\nVeGY7OyE47azu/t9MzPhHMoj/P8gh1/s6DMzRXhLVYUTt7MzXkeuKDB5sujHvmyZSBzr9XChQTPF\njI+CptNszwAmxCzhs+ZztWtXsQ56TIxQZPP21u74EomxkZYmFgly5S0ppqAAdnyehP+6SdTPiuGv\nRkNZNiqZtLRf+TB4GmZKFQoFF1NQANeuib1Tf3/t/Ycoq8PPzxfJV5cuGb8jNzcXCWzp6eJfZ2cI\nC3voaQEO9ZnacChzY9cytE5v7Uin3muXu7tIrLO0BC8v7Y4vkRgTiYkm20GsNKqgV9EvBXlFfDbo\nZx5d1ZcaZlc5PvpDVjzyJF+n/sQgx+E0cfA1tInaJytLrLibNROlUYaYyVpaihB9y5aiiUvz5sJx\nJyeLR2am/m16EFZWov+3n58oETx2rEynTWs4DF9bb144uJher3vTcrA/z8/04Z8oLdWoFjcyOXxY\n/E0lkqpIQYFJdxArDem8K8OVKyQPHMuws7Ow6NgOm+824T+kGVH+E/GgNhFr5v9r69bkSUkR/xHa\ntxehVmNY5VpZQY0aEBoqHHmzZqJ+/upV4ciztaQXXlns7ER9v6cnjBsHZ8489BSNuTXNz7zL5aJz\n2HRexJo3LhIamEXfib6s3+qmHbssLcU+YHi4CC1KJFUNE+8gVhrSeVeU7dthwACcL0Zy6On5mL23\nGFxceP/8JqJunuXj0AlkZzgQGWu6XWvuoqhIlBg5OwvH7eRkaItKx9oaataE1q2hc2cIDhb/Ya9c\nMY7aZldXIaNqbS2yzx/Sd/vwSRsO/NCPPu6Psc1sJe71Yxk74Cp/rDnN+CXepN7U0s1IoxGTi4MH\nReKgRFKVuHTJ5BXV7kU67/Jy8ybMmAHTp4OPDwMb/E1ez76gKFzKvsJbpz+ij9d/eLLGI9TyyCMl\nvQrM9HJzxQq2USNo0UI4HlNAo4FataBdO2jc+PbeuKGpXVs48JwcoW+fknLfQzf+4saIJ6/xQbMJ\nWCjmvBa1CFVVaeyTS5fQm/zwpxYnUTY2IooRHm480QqJpLLk5ooo3L0JtSaOdN7lISJCdAHbsQNG\njoSPP8a7hRc/7xc30PHRSyhSi3i/yUSSUyyIOmtDsJ+J3wTT08X+catWomWpqdZh+/oK7XFjaXbT\nsKFoJZqUBGPH3nePPj3TDC/XfGrbeDK78cv8dHUfPyb9CYCXWz7pGVqeHNrbiwlORIS46Ukkps71\nW0rcxrDFp0VM9E6sZ/LyRMOJV14Rq85PPoERI8DCgjH9r7Jmizuz/zjCliu/M7PRCBzy6jB4pg9D\ne13H1anQ0NZXDFUV2eRWVtChQ6VU5owCRRFZ8TVqiM9lDISEwDvvwKlTMGmS+J7dQ/vgTLbudQbg\nNZ/+BDk0ZEz0YtJzs/l5vxPtgnWQmOfkJBz3kSOykYnE9LlwocqtukE674dz/jy88AJ89hk89RRs\n3AhNm5a87Vc3ly8WnGRewgKs0hqx/b238P1vUxrXy2Xh2HgDGl4JCgrEitDbG9q0qTp7RWZmEBQk\nxFKMJbP6P/8R2zAHD8Kbb94WqbnFc91ucDJOw6INXlBkweqgqVzMvsJ/PttC43o5hAbqqE+3i4uI\nukRG/ssmicRkyMoSGgtV5R52B7LO+34UFcGHH8Lbb4vygqVLoWPHUg8Nd1pDvtNZlnutxd/3Bs39\n43F3NtEbXlaW2Ndv1kzszVaxUBMWFmLFW5xZbQyJd717i33v5ctFQuDkySW/d1uNyq7Vpxk+px5L\nNnrRsE4jrPx2czzgA/43qxOiA6+OcHMTe4VRUWLSY6pbJpLqS3JylcowvxPpvO/H7t1iJdSmjRDW\ncHUt9bCL2Vd4O/YTnq7RhddCQ4Dy9XA2Km7cEM6tQwfjcGq6wspK1IcfOCAyq42hw9CQIcKBf/aZ\ncJojRpS85VMrj9/XxHLmkjUJyZY4eg3jkePfMf38Qn7yeF/7jUvuxNMTLl8WvzN//6o3mZNUXVRV\nhMyN4f+3DpDO+3506QLffCPCLfdx3ACvxywFYEmTCfqyTPsUFoqkDk9PsSVgKtnklUGjEUl4f/0l\nMqttjKCk77XXhANfs0aswPv1u+vthnVyaVgnF3Bidu7/MT5mCVuT9tCnxiO6tcvTE86dE/XgDRvq\n9loSiba4eVMkgt6vg5iJI+NgDyIs7IErjZ3J//BN4i6m+w2nrk0NPRqmRe4sA2vevHo47mLs7EQ9\neEaGcWRWm5mJ/e+wMNFKdOfO+x76qs+zNHFowNjo98guzNGtXYoiboCnTomVjERiCiQlVRkd89KQ\nzruC5BXl81r0InxtvXm9Qdk6RRkdxWVgbdqIUqrquKfp5CSU2VJTRaKeobGwgAULxB7zzJkika0U\nLM0sWNlkMnHZCSw8u0H3dhU3MomOhoQE3V9PIqkMVayDWGlUw7u1dlh+/ktOZsTxfpOJaMxNbLWq\nqmK1bW0t9rfd3Q1tkWFxdxdJbNeuGUdmtUYjOpDVrQsTJ8KJE6Ue1sk9lP61urHgzKecz7qse7uK\nddCPHjWeenmJpDRSU6tUB7HSkM67AiTmXGP26Y/p6RlGT6+Hd4cyKorLwOrWFSHjKlhCUSFq1RJN\nVpKTjUNG1dERVqwQkYExY8QqohQWB4zDXDFjYswy/dhlYSHKyCIixA1SIjFGEhKqVAex0pDOuwK8\ncXIleWo+y5q8bmhTykdmpsgoDwmBwMAqPSutED4+ouuXscioenoKGVVVhdGjS13t1rbxZFrDYXx7\n5Q92XSs9xK51rKzE5OLgQZEUJJEYEwUFwnlX4ZA5SOddbv5JieLT+G2Mrz+QhnZ1DG1O2blxQ6wo\nO3QQ9duS0vHzE1EJYwkL16sn6r9TU4UDT0//1yETGgyigW1txkYvJr9IT/v2Go14HDwotAEkEmOh\nuINYFc/hqdqfTssUqUW8FrWQmtbuTPcbbmhzykZhoRDacHUV3cCq+Gy00igKBASIVW+xJrKhCQyE\nRYtEpveECaKhyR1ozK1ZEjie6JvnWH3hG/3ZZWcnBDDCw/9lk0RiMC5erBbbgdJ5l4MN8T8RnhbD\nuwGv4WBhAlq5OTkiCatxY9ENrIrvAWkNc3OhMOfoaDz7usViQceOwRtv/Cszvo/XI3TzaMubpz8k\nOff+Xcq0joODmCAeOlSqNrtEoleqaAex0pDOu4yk52cw9eRK2rkEM6h2d0Ob83DS0kQ4s00baNBA\nKmOVFwsLUfduZVVqqNogdO0KU6bA3r0wb95d+/KKorAs8HUyCrKYeWq1fu1ychLftaNHjaPcTlJ9\nuX5d3Ouqwf1OOu8yMjf2E67m3mB5k4mYKUb8aysuA7OxEfvbbm6Gtsh0sbYWNeCqet+WnXqnXz94\n6SXYulVko99BgEN9Rvs8y0cXv+NI2kn92uXqKqIUx48bR7a+pHoSF1dl5VDvxYi9kPEQm3GR989v\nYlid3oQ6BxranPuTny/LwLSNjY2QUc3JMZ593ZdfFk58wwahhX4HbzZ6GTcrJ8ZFL0HVd8a8mxtc\nuQIxMcaRrS+pXmRmioijMUgd6wHpvMvAxBPL0JhbM6/xKEObcn8yM8XKp3lzkeBURTvpGAQHBzEZ\nSk83jn1dRRH9vx97TPSZ37at5C1nSwfebvwKe24cZkviLv3b5uEhEoZOn5YOXKJfkpOrfIb5nVSf\nT1pBdqYd4cekPUxvOIwaGiNVIrt+Xdwo27cXYiMS7ePsLDqRpaQYx76uuTnMmSMmFXPnwr59JW+N\nqPtfgh38mHRiOTmFetZsVxThwM+cgfPn9XttSfVFVUXIvBpV00jn/QAKigoYd+FDGtjWZlz9gYY2\n598UFoowuYcHtGtXrb64BsHTU2ShG4uMqpWVKCFr1Egksh09CoC5Ys6yJq8Tl53AknMb9W+XmZn4\nTp44AfHx+r++pPpx86boDliNKmqk834AH5/4nOjsCywKGIu1uZF9KXJyRJjI318oplWjL61BqV1b\n1IEbiwqbnZ0QcfHygvHjxYoX6Oweyn+9OjH/zP9IzLmmf7vMzYVmfGSkmGBKJLokMbHaKUZK530f\nUrJTmBm+kE4OwfSt0dnQ5txNWpqYZbZrJ8vADEH9+uL3fvWqoS0RuLgIGVVra9ET/FbXr8WB48hX\nC5h2cpVh7LKwEFnohw8LhT+JRBcUFcGlS9Uu8iid932Y8+ccUnJTWVbv/1CMxTkWl4HZ2ooyMFdX\nQ1tUPVEUIXzj7W08Mqq1agkHnpMjZFRTUvC182Zc/QGsj99KRGqMYeyytBR14OHhYtIpkWib1FRR\naSNX3pJT106xMnwlL/oPpJldA0ObIyguA6tXT5QuVZNyCKPFzEx0IXN3N55VZcOGsHSp+J6MHQuZ\nmUxvOBxPK1fGRb+n/9KxYqytRXj/4EHIyDCMDZKqy+XL4jtWzZDOuxQm7ZiEjYUNc1tNMbQpgowM\nMbts0ULst8oyMOOgWEbVzs54ZFRDQmDBAjh1CiZNwlG14u3Gr7A/5RjfGKJ0rBgbG5GXER4utnwk\nEm2Qny+2iRwcDG2J3pHO+x52ntvJ1tNbmd5xOl62HoY257bcX4cOULOmoa2R3IulpSghs7AwnvaY\nHTvCzJlipTtrFsNr9yTYwY/Jhigdu5Ni5auICKFBLZFUlpQUseddjeq7i6l+n/gBFBYVMmH7BHyc\nfRjbdqyBjbmnDKwazixNBmtrsZVRUGA87TF79RKh8507MX9vKUsCxxGXncCy85sMa5ejoxC6OXxY\nrJokkspw6VK1VZKUzvsOPjnyCcevHmfhYwvRWGgMZ0hxN7DAQBEGtbQ0nC2SsmFrKwRTsrKMR0b1\n+efF4+uvefT7SHp7dWT+mf+RlGvgVqfOziJKceyY1EGXVJzictlq0EGsNKTzvkVaThozfp9BWN0w\n+gX2M5whqaliT7BtW/DxkWVgpoSjo1iBp6cbz6pyzBjo3Rs+/JDFFwPILsxh5qk1hrZK6KAnJ0sV\nNknFuXZLv6Ca3iOl877F/L3zSc5KZunjSw1TGlZUJOqG7e1lGZgp4+oqEguvXzcOGVVFgenToWNH\nGi34mNFmbVl78XuOpZ82tGXCgZ8+bTzJfhLT4sKFar2dKJ03cD7lPMv+Wcbzwc8TWitU/wbk5wvH\n7eMjWlDKMjDTxssLgoLEysAYwsIWFvDOOxAczKx3D+Ci2DIheqnhSseKMTcXk9Vjx4wnUiExDTIz\nRYRLY8DtTQMjnTcwdddUzBVz5j86X/8XLy4Da9lSloFVJerWFUIuxiKjqtHA0qW4ePnw5o48fr8e\nzk9X9z38PF1jayv2LmNjDW2JxJRISqqWGeZ3Ur0/PfDXpb/YHL2ZSe0n4e3ord+L31kGVqOGfq8t\n0T2+viKaYiwqbI6OsHIlr5x3o1GKGRMjF5FfZAShfTc3sfdtLHKzEuNGVUXIvJrJod5LtXbeRWoR\n47ePp6Z9TSZ3mKy/CxcUiJmjl5csA6vKKIpoHFOjxu3kGkPj4YHl8lUs3mvDqdwEPoz+1NAWid+T\ni4toYmIsmfoS4yU9XXxPqnkzpmrtvL+K+oqDlw8y/9H52FnpqdwgJ0esuAMDIThYloFVdczMxP63\ni4vxyKjWq0evcavpcsGMt059SMr1y4a26La8ZUyMcWwzSIyXK1fkfZNq7Lyz87OZumsqzWs0Z0iz\nIfq5aGqqcN7t2skysOqEhYWo17e1NZrmHEpgIO+FTOGGdRHzNowwjhWvi4u4MV82gsmExDgpLBTC\nLDJaWX2d97IDy7iYdpH3ur2HmaLjX0NxGZiDg9jfdnHR7fUkxoeVlUhKNDMzmuYcIR2eZphFKMsb\nJHNmzjjjKG1zc4PoaJFNLJHcSzXtIFYa1dJ5J2UkMX/ffPo07kPn+jru1Z2XJxx3/fri5l2NSxuq\nPRqNEHHJyzOa5hxvd34bKzNLpjpHwNtvGz5kbWEhJjqRkcZRZicxLi5flvfQW1RL5z3rj1nkFOSw\nqOsi3V4oI0OESUNDReKSLAOT2NkJGdWMDKNozlFT486Uxi+yJRD2Rm6D5csNbZLIIk5NleprkrvJ\nz4fExNsNbqo51c55R12NYu2RtYwKHUUjt0a6uYiqiqQ0MzMICxNZ5RJJMU5OYkKXmmoUoerXfQdT\nW+PJ6/1dKPr8M9iwwdAmifD5qVNGkyMgMQJu3Ki2HcRKo9r9Fib+NhFHa0dmPTJLNxcoLgOrUUPo\nk8tZoqQ03N1FEtu1ayIJx4DYmmuY7z+KcNsUvuzfVKy+t20zqE2Ym4sckaNHjWKCIzECLl7UbQex\n6GiT0hrQivNWFOUJRVFOKYpyRlGUqaW8ryiKsvzW+5GKorTQxnXLy69nfmX72e3M/M9M3GzdtH+B\n7Gyx4m7aVJQHyXIGyYOoVQuaNBEiLgbe3x1cuwctnPyZ2iyZ7LahMHcu7N1rUJtK1NdOG4EOu8Sw\nFK6ymdIAACAASURBVHda1FUHMVUV3/k5c3Qzvg6otPNWFMUcWAV0BwKBAYqiBN5zWHfA79bjZWB1\nZa9bXgqKCpj420R8XXx5tdWr2r9AaqpIRGrXDurVk2VgkrLh4wN+fgaXUTVTzHgvYByXcpJYNqq5\nkHadOlWsfA2JmxvExRmPSp3EMFy7JsLlurqvHjwIZ85Anz66GV8HaGPl3Ro4o6rqOVVV84AvgSfv\nOeZJYIMqOAA4K4pSUwvXLjPrjqwjOjmadx97F2sLa+0NfGcZWPv2sgxMUn78/IQWuoEdVCf3UJ70\neoT5FzaStOgtkasxfry4qRkKRRH9v48dM4oEP4mBiIvT7RbkF1+IiWKnTuU6bf/F/RQYSGJYG867\nNnDpjufxt14r7zE6I6cgh1l/zCKsbhhPBTylvYGLy8B8fUUCkixhkFQERRFNaTw9xbaLAVkYMIac\nolxmJW2ClSvFd3r0aEhIMJxRUn2tepORATdv6u7+ev487N8P/fqVS3L1RPIJHln/CO/sfUc3dj0E\no0tYUxTlZUVRIhRFiUjW0kpEY6Fhy7NbWNl9pfZ6dWdkCI3d0FBo1EhmQEoqh7k5NGt2u0zKQDSy\nr8eoes+w9uIPRNlnwYoVYsU7erRh5V1dXESZkCEnERLDcPWqbstsN20STrtfv3KdNnnnZOys7BgZ\nOlJHhj0YbXicy0CdO55733qtvMcAoKrqR6qqhqqqGurh4aEF8wQd6nagWY1mlR9IVcX+i7m5UEuT\nZWASbWFhAc2bixtJerrBzJjVaASOlnZMOrEcGjaEpUtFBcXYsYZVPnNzg6goqb5WndB1B7HUVPjp\nJ+jRo1xbnrvO7WLb6W1M7zgdDzvt+anyoA3nHQ74KYpSX1EUK6A/8OM9x/wIDLmVdd4WSFNVNVEL\n19YvBQViFlirliwDk+gGa2sRzVFVgzkpNytnZvq9yK/Jf/Hr1b9ESdu774qs70mTxHaRIbCwEL+f\no0cNXl4n0RNpaSLTXFeVO998IyJLAwaU+ZTCokJe/+11fJx9GNNmjG7sKgOVdt6qqhYAo4HtwAlg\ns6qq0YqijFQUpTie8DNwDjgDfAyMqux19U52tggbNm0qHlJbV6IrbGyEjGpOjsEahrxa71l8bb2Z\neGKZSMgJC4OZM0VW7syZhnOeDg5i/zM21jDXl+iXxETdtf7My4OvvxaJxr6+ZT7t02OfcizpGAse\nXYDGwnB5TlrZqFVV9WdVVRupquqrquq8W6+tUVV1za2fVVVVX731fpCqqhHauK7euLMMrG5dWQYm\n0T0ODkJGNT3dICtda3Mr3g14jeib51h36VYgrVcvGDcOdu2CRYsMlzzm5gbnzhk8uU+iYwoLIT5e\ndxHO334T36GBA8t8SkZeBjN+n0Fb77Y82+RZ3dhVRmSW1cPIyBBylh06iJIViURfODuLZjYpKQZR\nGXuqRhfCXEOYeWoN6fm3OqENHgxDhohw48cf690mQCSHOjnBkSPG0cpUohuKv/e6iHKqqigPa9AA\n2rQp82mL9i8iMSORJd2WaC/5uYJI5/0gNBqRQNSixe1yFYlEn3h6iix0A8ioKorCksDxXM27wYKz\nn95+47XXoHdv+OgjEXY0BBqNiIBFR8vysapKfLzuysMiIkQOx8CBZY6kXk6/zKK/FvFsk2dpV6ed\nbuwqB9J5PwgnJ7EXIsvAJIakdm1RB24AFbZWzk0YVLs7S85t5ELWrRxTRYHp06FjR1i4EHbs0KtN\nJTg7iyz4ixcNc32J7sjLgytXdBcy37hRZJd3717mU6b/Pp1CtZAFjy7QjU3lRHolicQUqF9fhPgM\n0Dhhvv+rKChMO7nq9osWFvDOOyIqMHMm/POP3u0CxP53TIxBS+skOqBYU0AXC6e4ONi3D555pswR\n1UMJh/j02KeMazOO+i71tW9TBZDOWyIxBRRF6I17e+tdRrWuTQ1ebzCILxJ+5Z+UqNtvaDSwZInQ\nZ584UThRfWNhIZpVHD0q+j1Lqga67CBWTlEWVVV5/bfXcbd1Z1rHabqxqQJI5y2RmApmZqILmbu7\n3tXOpjR8AS9rNybELEG9M3Tv6ChU2FxcYMwYIaihb+zsROLaqVP6v7ZE+xR3Z9RFB7HUVNHutnt3\ncHUt0yk/nPqBPy/8yZxOc3DSOGnfpgoinbdEYkoUy6ja2elVRtXBwo63G7/CXymRfJO46+43PTyE\nDrqiCBlVQ/REdnUVE4crV/R/bYl2Ke4gpgu+/bZcoix5hXlM2jGJAPcAXmr5km5sqiDSeUskpoal\npSghs7AQgiV6Ylid3gQ5NGTKyRXkFN7T4atuXVi+XChijR4t/tUniiL2vyMjIStLv9eWaJcLF3ST\nqJafD5s3C3XMhg3LdMqqg6s4c+MM73V7Dwsz4xLmks5bIjFFrK2FCltBgd6clblizpLA8ZzPuszy\n81/++4CAAHjvPbh0SbQS1XcNtqWleBw/Llr1SkyPmzd110Hst9/Eqn7QoDIdfj3rOnP2zKGbbzee\naPiE9u2pJNJ5SySmiq2tUGHLytKbo3zMow09PcOYd2YdV3NL2Xdv1QrmzhUOdOpU/YvLODqKfICz\nZ/V7XYl20FUHMVUV5WENGoiVdxmY/eds0nPTWdx1scEFWUpDOm+JxJRxdBQOMz1db9nWiwPHkVWY\nw5unPyz9gMcegylTRDnO3Ln6XwW7uwvtc0O2MJWUn6IikWWuiw5ihw6VS5Tl5LWTfBD+AS+1eIkg\nryDt26MFpPOWSEwdV1ehAnj9ul5Wuv72PrxSrx8fXfiOo6ln2HHAgU++d2PPYfvbGjL9+sHIkaLd\n4ooVOrfpLorlU48eFclJEtMgPV13HcSKRVmeKFv4e+JvE7GzsmNO5znat0VLSOctkVQFvLwgKEjs\n6elhpftmo5ewN7On/VcfM2VlLfYds+eVBXUJGRjAufhbXaBefFEIYXz2GWzYoHOb7kKjEXKyMTFS\nPtVUSEjQTQexCxdg714xoSzDXvpvZ3/jp9ifmNFxBp52ntq3R0sYV/qcRCKpOHXrClnJ06eFJroO\n9+k0ha4oe2aRHTaRue99Qk+vMFQVVnzlQY+xfkRvjsbcXBH9v1NTRSa6i4vQRNcXrq6ipaSHhxC3\nkRgvxR3EdNH86csvxWq+DKIsBWohE7ZPpIFLA4P26i4LcuUtkVQlfH2F4pmOVdi+3O7CfwqG0Miu\nLq/HLCW/qABFgTH9k3FxLODn/bfELMzMYM4c0bnp7bdhzx6d2vUv3NxE8pweS+okFSAlRThwbSer\npaXB1q0iXO7m9tDD1179lejkaBZ1XYS1hXE3o5LOWyKpSigK+PtDjRoihK4jIs/Y0Ll5DosDx3Eq\n8wIfxN3uLtap5U2On7G5fbClpWhg0rgxvPGG2IvWFxYWIis/MtIgbVUlZSQ+HmxsHn5cefnuO7GP\nXobysLT8DGbFf8Yj9R6hr39f7duiZaTzlkiqGmZmYv/bxUVnGdc13fOJvWRNL8+OPObemtmxH3M9\nTyi+nb6goYbb/7d33+FRVVsfgH97JmXSe4GEEEggQOgkgHQELOhnQZGi6NWr6EUQROwFEBVUqmJD\nLKg0G4INFRFFkG7oHQJJCJBOejKZ/f2xMlJMmXLmTFvv8+Qh5czZm0NZs9taV+x89/MDFiygtflJ\nk2g3uFr8/YGSEnXbZKazVQWx6mpg5Uqa9TEhKcuMo4uRq7+Audfav1a3KTh4M+aKPDyAzp1p1GmD\nbGd3XZ+PFT+H4sgpHea2m4yi6hJMPfwetu/3xYZdAbh9UMG/XxQSArz1Fo2wJkwAsrIU71e9wsKA\nEyfsk7qVNSwvjzYVKh0wf/mFlo9Gj2700qMlp/HGyRW4N2IIujbpqmw/bISDN2OuysuL0qhqNDTy\nVFBsVDXmTc5An/uTsHRJf1wtRuHt9K8xZHoVPp6ajkD/ena8N2lCedCrqoCJE9XLwiYEvXnYvZsK\nXzDHcfq08qNuKYFly6iU7lVXNXr5YwfnQaf1xsux9yjbDxvi4M2YK9PpKIlLVZXiQeueG/Ox6YPD\nkBLw3zIV3tIfHcePw419GymYkpBAtcDT02kkrhZvb3ojs28fp091FOXltLSjdPnPXbuAQ4eoAEkj\nRU5+ydmCb89txLOJ9yLay7RKY46Agzdjrs7Pj9KolpQonrSkdfNKvPpIFlbNuIBZ7e/Hxgub8f35\nPxt/YY8edAZ8xQr6j1YtwcG0kS89Xb02Wf1ycmxTQWzpUkrUM3Rog5fpDXo8emAuWvrGYFKLxqfX\nHQkHb8bcQVAQkJJCZ65ttOt6XPxwtPGPx+QD81BlMCFV64QJQNOmwPTp6lYCCwujUZmKJVVZHaSk\nN1EBAcre9/RpSsoyfHijSVneO/019hefwOy2k+CttUGCGBvi4M2YuwgPp01subl0plZhnhoPzG33\nKI6WnsbC9JWNv8DXF5g6lTJrqZlCVaulgJGWplo+eFaHkhKgtJSWM5S0fDlt2GwkKUt+VRFeOPwe\nBoal4JboAcr2QQUcvBlzJ02bAsnJNF1pg3Xf6yN74/qIXph+5P26q45dqWtXYORI4IsvgG3bFO9P\nvXx9aR/AwYOcPtVezp6lIKukCxcuJmUJD2/w0mlHFqGwuhjzkic7xdGwK3HwZszdxMcDrVpRALdB\n4JqXPBllNRV45pCJm9EefphSu774ouK74hsUGkrJQbKz1WuTEVtVEPv6azrBMGpUg5ftu3AMb5/6\nEg82H4ZOga2V7YNKOHgz5o5ataKAaYM0qkn+8ZjYYhQ+zFiDnYUHG3+BTgdMm0ZnsBcsULw/9RKC\nAviePTR9y9RTVEQzH0qOvPV64PPPaXNm6/oDspQSE/fPQaCHH2YkPaRc+yrj4M2YOxICaNuWCpjk\n5Sl+++db3Y8IrxBM2P86pCmj+44dgbvuonSWmzcr3p96eXrSm4fdu22yD4DVIztb+Qpi69bRG8BG\nkrKsOvsb1udtx4ykhxDmZYNCKCrh4M2Yu9JqgU6daOpS4Z3XQZ7+mNV2PP4q2IOlWT+a9qIHHwRa\ntqQCJmoWEgkIoLXS48fVa9Od6fW0XKHkLnMp6XhY8+ZAr171XlZeU4HJB+ahQ0AiHowbplz7dsDB\nmzF35uEBdOlCo6ALFxS99T2xNyI1qB2eOPgGivUmTEt7e9Pu87w8YM4cRfvSqLAwyn1ug1kIdgVb\nVBBLS6PNh6NHN3hufPbxz3CqPBsLkqfAQ+PcFbE5eDPm7ry96Qy4lIqu/WqEBm+0fxzZlbl45ehH\npr0oORm45x7gu++A339XrC+N0mgogUtamnopW91VRobyGdWMSVluuKH+ZsvPYuaxj3B7k0EYGJ6i\nbPt2wMGbMUbFQlJTKXApGLx6hnTA3bE3YO7JpThactq0Fz3wAG2oe+UVdROpGBN67N/Px8dspbKS\n1qX9/JS7Z0YGvdG77bYGk7I8dmA+JIDZbScp17YdcfBmjJGAANqpe+EC7QRWyKttJsBb44VH9s82\nbfOapyftPi8sBF5/XbF+mCQ4mILLwYOc/9wWjMsSSp6rXrGCpuDvuKPeS37N3YYvstfhmcR70dy3\niXJt2xEHb8bYRcHBVImsoECxNKrRunBMbz0Wa3M2Y805E6fCk5KA++8HfvoJ+PVXRfphsogIStt5\n4AAHcKWdOqXsqLu4GFizBrj22nqTslQZqjFh3+to6RuDxxPGKNe2nXHwZoxdLjKSdqErmEZ1fPwI\nJAe0xKT9c1FeY+K0/L33Am3aALNm0ZsJtQhBzyAjg6bQOYAro6yMZlOUXO9etYoqkzVwPOzNkytx\nsOQkFiRPgU6rcCpWO+LgzRj7t5gYOgeuUBY2T40HFiY/gfTyM3j12BLTXuThQdPnJSVUQlTNdWgh\naASemQns3ctnwJWQk6PsDnO9Hli5kjZbJiVd9qOqaoHCYi2yynMw7cgi3BDZBzdG9VWubQfAwZsx\nVrcWLejc9fnzitxuQHgKRja9BrOOL8GJ0kzTXpSYSOe/168Hfv5ZkX6YzDgCz86mLGwcwC0nJU2Z\n+/srd89164Bz54A77/znW7mFWjzwUhzCB3dC3I0dkPTuElTo9Zjfbopy7ToIDt6MsboJQSOa2FjF\n0qi+3nYiPIQWjx6Ya/qL7rqLjpC99hpN5astIoKCBAdwyxUXK1tBTEpg2TJK8du7NwCgrEJg4INJ\n8NUZcGzVPnz39YcobbUCYfsfxWdLU5Vp14Fw8GaM1U+jocAZHg7km1AlrBGxPlF4odX9WHPuD3x7\n7g/TXuThQTW/KyqAGTPsc4wrIoJmINLSbFYP3aWdO6dsHvPdu2lD4SVJWZb+GIb4ppWY/1gmQoIr\nMG7fLDT3aYLN996CBSsikV+k4JS9A+DgzRhrmDGNqp+fIueuJ7UcjXb+LfHIvtkoM3XzWnw8MGEC\nsGkTFZ+wh/BwOurEAdw8tqggVkdSlh82BeKu6/MhBLDg5HLsLz6BN5KnoGWUJ3p3KsFvOxRMx+oA\nOHgzxhrn6UlHyDw8rM477qXxxDsdnkJ6+Rm8fPQD0184YgTlrV6wwH55yMPCaAZi1y6guto+fXA2\nhYXKVhDLzAQ2bACGDaPkQrW0GqBaL5BRfhbTjizCTVH9cFN0fwCAvkZAq3WtxDscvBljpvH2pixs\nej0d+7FCv7CuuDv2Brx+/FMcKkk37UVCUO5zf3/g2WcpW5c9hIVRQOIAbpozZ5StIFZPUpZbBhTi\ng9XhmLhvDgzSgAXJtEntZJYXtu33w6BUFYvdqICDN2PMdL6+lIWtrMzqNKqvtX0Efh4+GLd3lmmZ\n1wAKnC+8ABw7BixcaFX7VgkLo0x0O3cqmo3O5ej1FLyVmjI3JmW55hrah3CJ4YMLcC70Z6w69xse\nDH8I0doYfLcxCNeMb4WpD2QjwM+1zutz8GaMmScwkEbgFy5YNfKM8g7DK0kP47e8HVh+5ifTX9in\nD02hL1+ubu3vK4WGUjDhAF4/YwWxBip9meWbb+iNYx1JWQzaclQMmoSw6kSsnP4SAvt3xozFTTDz\n4SxMHKXMcUdHwsGbMWa+0FCga1fawGXF5q2xzW9FalA7TD4wDwVVZpQknTABSEigXegK7IK3WGgo\nHYHavt1+0/iO7PRp5TKq6fU0Zd6tG2Xeu8Irxz7CyYpMfNF3Cs58fxhVW/7G1iWHcPtgFYvbqIiD\nN2PMMlFRQIcOdPbawhSiWqHFux2fQU5lAZ469KbpL9TpgJdfppGvvY6PGYWE0BLCjh1cTvRSSlcQ\nW7/+X0lZjA4Un8Crx5ZgTMxQlyj3aQoO3owxy8XFAa1bW5VGtWtQG0xqOQqLTq/Cpvw001+YmAg8\n8giwcSPw5ZcWta2Y4GAK3Nu3cwA3ysujTYZKVBCTko6HxcXRssklDNKAsXteRoCHH+a0e9T6tpwE\nB2/GmHUSE+kcthVZ2Ka3fhBxPtEYu+cVVBnMWEc3Hh+bP99+x8eMgoNpD8C2bVQsw92lpyuXDnXP\nHioSM2rUv9bPF5/+BpsKdmN2u4mI8A5Rpj0nwMGbMWYdIWgNMjr6Yr1mM/l7+OLt9k/hQMkJvH78\nE/PanjqV1lWfe87+685BQbQ2u22b1cfpnFppKVBUdNk5bKssXUobJW+88bJvn63IxRMH38CAsG74\nT+z/KdOWk7AqeAshQoUQvwghjtb+WufbHiFEuhBirxAiTQixw5o2GWMOSKOh9e/gYIs3kN0Q1QfD\nmwzGjKMf4GjJadNfGBZGAfzoUeCttyxqW1FBQbQHYOtW9w3gOTnK7TDPyqozKQsATDowB+WGSrzb\n4WkIJabnnYi1T/cpAL9KKVsB+LX26/oMlFJ2llK6x24CxtyNhwfQuTONgouKLLrFguQp0Gm88NDe\nmaaf/QZoHXT4cCpWsWWLRW0rKjCQZgW2bqVRqDuRkqbMlTrbvWIFPcsrkrL8eH4TVp75Bc8m3ock\n/3hl2nIi1gbvmwEYi/MuAXCLlfdjjDkzLy86yiME1eE2UxNdOF5tOwHr87bjo4w15r144kQqYTp1\nKp0vtreAABp9bt1q0bNwWsXFtOavRFa1khJg9WpKyhIZebEJfSke3PMK2vq3wJMJ91jfjhOyNnhH\nSSmzaz8/CyCqnuskgHVCiJ1CiLFWtskYc2Q6HWVhq6qyaOPWA3G3on9oV0w+MA9nKszYBHfp8bEX\nX7Tv8TEjf39K5elOATw7W7k85vUkZXnq4EJkVpzHB52eh7dWwdSrTqTR4C2EWCeE2FfHx82XXidp\njqu+fy19pJSdAVwP4GEhRL8G2hsrhNghhNiRo1ANYcaYyvz8KICXlJi9iUwjNHi/43OoNFSblzoV\nAFq1ogQuGzcCX31lZqdtxN+fCrv89ZfVRV0cnsEAZGQoM2VuTMrStSvQtu0/3/4jbxfePvUFHmkx\nEleFdLS+HSfVaPCWUg6WUrav42M1gHNCiCYAUPtrnTnopJRZtb+eB7AKQPcG2lskpUyRUqZEXJG7\nljHmRIKCgJQUKuJhZha2Vv5xmJH0EFaf+x1fZK8zr90RI4CrrgLmzQNOnDDvtbbi50czA1u2UFpZ\nV1VYSMfllBh5//YbcPbsZaPu8poK3L/nJbTwjcHLSeOsb8OJWTttvgaAccHhHgCrr7xACOEnhAgw\nfg7gGgD7rGyXMeYMwsNpE1tuLuW4NsOkFqOQGtQO4/e9htwqM1JcajSXHx9zlLzjvr4XA7iFG/oc\nXlYWVZ9TwrJlQLNmQN++/3xr2pFFOFp6Gu93fBZ+HgodQ3NS1gbvWQCGCCGOAhhc+zWEEE2FED/U\nXhMF4E8hxG4A2wB8L6Vca2W7jDFn0bQpkJxMx4fMSKPqofHAh51eQGF1MSbtn2Nem+HhVH3syBHH\nOD5m5OtLH1u20CjVlVRXUwWxgADr77VnD7B3LzByJO0ZALCj8ABmH/8M9ze7BYPC6528dRtWBW8p\nZZ6UcpCUslXt9Hp+7ffPSCmH1n5+QkrZqfYjWUr5shIdZ4w5kfh4Wo82M41q+8BEPNvqPizN+hGr\nz24wr82+fen42NKljnF8zMjHh9bBt251jF3xSikooDdnSpzvXraM3gT8HyVeqaypwn92T0O0dxhe\nbzfR+vu7AM6wxhhTR6tWlJvazI2oTyfei86BrTF2zyvmTZ8Dlx8fc6SRrk53MYDbsyqakjIylKkg\nduYMFSG59dZ/7jf1yHvYX3wCizs9h2BPBUb2LoCDN2NMHULQruHISLPSqHppPPFJ5xdRUH0BD+99\n1bw2dTrgpZdok5ijHB8z0ulodLl1q8VpZR1GRYVyFcSMSVlGjAAA/FWwB68f/xT3N7sF10f2tv7+\nLoKDN2NMPVot0KkTHSUyYyTcITAR05MexOfZv2DlmZ/Na7N1a2D8eOCPPxzn+JiRTke78rdutaqw\ni93l5ipTQcyYlGXIECAqCmU1FbgnbRpifSIxp90kZfrqIjh4M8bU5eEBdOlCv5qROvTxlmPQPTgZ\n4/a+irMVuea1OWoU0LMnHR87edLMDtuYtzflhN+xg0avzujUKWU2qq1eTX8nao+Hjf71fRwtPY0L\nS5eg792pmL8sEtXmnTp0WRy8GWPq8/amEXhJick70D00HljSeTrKaiowdu/L5iVv0WiAadNos9iz\nzzrO8TGjSwP4uXP27o15SktpWUKns+4+ej2wciW9sWvXDhO/OIbVVUtwjWYMDrwWhTcfz8B3fwZh\n+JMJ5hxacFkcvBlj9hEcDCQkmLVhq41/PGa2eRjfntuIDzL+lVaiYeHhwPPP0/Gxt982s7Mq8PIC\nQkOBnTspOYkz0OvpbLcSO8w3bKDNaqNH41RBGd6sfgJxXs3w9TVj0SRcj35dS/DDgmNIz/bC2s0K\nFT25VE2N9dP+KuLgzRizn4QEGnWakQP9kRYjMSi8Oybun40jJafMa69/f+D224HPPqN1Zkfj6UkB\nfNcuCmSOSErar3DgAO0KP36c3ohZa9kyICYG6NcPd26eDfhn4fPUGZclY/HylHjgllx8+Wud1aet\nU1AAJCYqf18b4eDNGLMfDw+gY0eadjVxGlwjNFjSaRq8NV646+/nUW0wcxF00iSgRQvHOz5m5OlJ\nNcrT0mhU6yjKyqjU54YNlKc9O5uCdmSk9elQ9+6lxCyjRmFp9s/YJNegfeYT6BHS/l+X+vkYUFGl\ncOiqrKTn3ry5sve1IQ7ejDH7Cg2lJC5mTJ/H+ETi/Y7PYnvRAUw/ssi89ozHx4qKgBkzHOv4mJGH\nx8UAnplpv35UV9MU/pYtwO+/A4cP0/OLjKTAXZv9zGrLlgH+/kgfkopx+2ahm19nZH79AopL/x2i\nPv8lBINSFc4PX1gItGunXDU0FXDwZozZX2Ii/cdZUWHyS25rMgj3NbsJM499jI15f5vXXlIS8PDD\nFJBWrTKzsyrx8KB1+t27gdOn1WvXYKA3Unv30rR4WhoF8chI6o8SdbovlZ0NrF+PmltvxpjDMyEB\nfNFjOoYPLMawxxNwIpPaKyrR4OmFTZGe7YVR1ymY2Ka4mN4oRdVX0doxcfBmjNmflxfQoQONhs0Y\nCS9InoIWvk1xV9rzKKw2s9zm6NF0fGzOHJoOdkTGAL53Lx3HsqWSElq/3rDh4rnz0FAgIkKZzGn1\nWbkSADCzr8Cf+Wl4q/0TaOEbg7eePI2eHUrR4z9t0OKm9oi7oSNOZnnjt3ePwFen0GyJwUD7Ldq2\ndarNagAgzDpuobKUlBS5Y8cOe3eDMaaWPXvoqFRoqMkv2VqwD302/xc3RfXDl91egzDnP+HcXMrk\nFR0NfPSR8qNKpdTUUF/9/KjSVng4nau2NuBUVdF909PpjZNWSwl0PD0V6XajSkuBoUOx8bp2GNBh\nJ+5oOhjLurx82Z9hZZVA1nlPhAbVIDjAvMp0jcrLA2JjacrcQQghdkopUxq7jkfejDHHkZREAcmM\nc9g9QtpjVpsJ+Prsb3gr/XPz2jNWHzt8GHjnHTM7qyKtlqZ1vbyAY8eATZtohHz0KK3XmnPwdAmR\nKAAAHLBJREFU2WCgoLV7N9XM3rOHvhcZSdPHagVuAFizBrmyFKM6H0NL3xi81+GZf7358vaSaBlb\npXzg1uvp71pCgrL3VYnzrM4zxlyftzfQvj0dlTJjDXJyyzuxIW8nHjs4H1eFdES34Lamt9m/P3Db\nbcCnnwJXXQV0d+Byk15eFGABWodOT6dg7ulJx6yioijdal0byYqLafPZqVP0Wp2OZjiUOKNtiZoa\nGFYsx913ByBHlmBLtzcR6OmvXvt5ebRUo1T9cZXxtDljzLFICfz9N527NeP8cF5VIbr8cSc8NR7Y\n1XcpgswJBBUVwF130TTu8uXKnFtWk15Pa9ZVVRS4mzalpQBfX9p8lp5OwdvDg6bFHWFX9fr1eHX1\nE3hqCPBW+ycxLn64em2XldGvvXvb781LPXjanDHmnIzVx2pqaIRoojCvYKzo+gpOlZ/FA3teMi99\nqvH4WEEBMH06BUNn4uFx8cx1cDDlSN++nXbT79tHzzQykkbajhC4AWz6cRGeHQTcHn01/tf8dnUb\nLy4GkpMdLnCbw3l7zhhzXT4+9J+rmbWue4V2witJ4/BF9jq8cXKFeW22aQM8+iiwcSPw3HPOF8CN\ntFqaOo+MpI+ICOvzjivs3O5NGNHlGJojCIs7vWDeJkNrFRbSzIQZmyIdkWO8BWOMsSs1bUopQouK\nKBiZaErCGGwu2IPHDs5H56DW6B/WzfQ2R4ygqecFC2jUP3Om4+5Ad1LVBj3u2DcN+b7A5u5zzFve\nsJZxNqd1a/XatBEeeTPGHJMQNPquqjJrFKwRGnzSeToSfGNwx86nkVluZpWuMWOAxx+nKecpU8xK\nHMMa9/iOV/BHUAHez+uNztGd1W28oABo1cq259ZVwsGbMea4fH1p/dvM6fNAT398kzIHZTUVuG3n\nE6isMbME6IgRwDPPUA7vyZPNKpzC6vdZ5g9YcH4NJm4VuPOGp9Rt3AnzlzeEgzdjzLE1a0absIrN\ny6DWNqAFlnSehm2F+zF+32vmtztsGBUv2bEDmDiRdqIzi6UVHcbYPS+jf4YGr9dcDTRpom4HCgtp\nJsdBNuxZi4M3Y8yxaTR0HresjNYszTCsydV4OvFeLM74xvwELgBw441UvGT3bmD8eLPfQDByrjIP\nt+yYglC9J1auMMBz9Bh1O2DMXx4ZqW67NsTBmzHm+Pz9aTd4bq7ZL52R9BBujOyLR/bNxtrzm81v\n+9prgVmzgIMHgXHjaAMdM1l5TQVu2T4F5yvz8c1qH0QldKREPGpx4vzlDeHgzRhzDs2bWzR9rhVa\nLOv6EtoHJOCOXU9j34Vj5rc9cCAwezYV7njoIdr4xBolpcR9u1/ElsK9+MxzBFJ251BBGDUVFNDf\nncBAddu1MQ7ejDHnoNUCHTvS7m8zz2AHePjhu+7z4KfV4f+2T8b5SgtKSvbpA8ydS+U5H3zQolkA\ndzPtyCKsOPMzZrYZj2Gf76XjfwMGqNcBJ89f3hAO3owx5+HvT1OueXlmv7SZTzTWpM79Z/21oqbS\n/PZ79qQz4NnZwNixVAGN1Wlp5o948ej7uLfZ/+HJihRKeTtypLobxvLyaLnFSfOXN4SDN2PMuRgL\ncBQWmv3S1OBkfNZlBv4q2INRu56F3mBBFrWUFGDhQgoMY8dSIhl2mXU5W3Hv7unoF9oV73Z4BmL5\ncipnetNN6nWirIzKpsbEqNemijh4M8acizF5i5R0dtdMw5pcjQXJU/DNuQ34395Z5uVAN+rUCXj7\nbeDCBeCBB4CMDPPv4aJ2FB7ArTseR5J/c3yTMhte5/OAdeuAW26hmRM1SOkS+csb4pq/K8aYa9Pp\naP27oID+ozbTIy1G4tnE+7A44xs8d/hty/qQnAy8+y6twT/wAFXucnNHSk7h+m2PIMwrCD/1WIgQ\nr0Dg88/pz2jECPU6UlTkEvnLG8LBmzHmnCIjgRYtLFr/BoAZSf/DA3G34pVjH2H+iWWW9SEpCXjv\nPQpOY8dSbW03lVV+HtdsHQ8BgZ97LERTXQRNXa9aBVx9NQVTNbhQ/vKGcPBmjDmv1q1pFG6sz2wG\nIQTe6fAUhkUPxKMH5uLjjG8t60NiIgVwrZZ2oR86ZNl9nFhOZQGu3TYeeVVF+LH7G2jtX5uC9Ntv\nafr6zjvV60x+vsvkL28IB2/GmPPy8AA6d6YAYWb2NYDOgC/t8hKGhPfAfbtfxJKM7yzrR3w88P77\nVMr0f/+jGtpuIqeyAFdveQgnSrOwJnUuugW3pR/U1ADLl1N2vA4d1OlMZSXtLHeR/OUN4eDNGHNu\nQUEWZ18DAJ3WG6tT52BQeCru3T0dn2RaGMBjYymABwYCDz8MpKVZdh8nklNZgEFb/odjpZn4tvs8\nDAxPufjDjRuBzEx1R92FhUC7di6Tv7whHLwZY84vPp5yV1uYutRHq8Pq1Lm4OjwV/0mbjk8zv7es\nH02aAIsWUV8mTKCiJi7KGLiPlmbgu+7zMCi8++UXLFtGz0OtpCwumL+8IRy8GWPOz1i8pLqaPizg\nq9VhTepcDAxLwT1p0/BRxhrL+hIVRQG8SROqRvbXX5bdx4FlV+Q2HLgPHgR27aId5mqMgl00f3lD\nOHgzxlyDry8FcAt3nwMUwL/tPg9DImgNfNaxjy07Bx4eTpvYmjeneuAbN1rcJ0dztOQ0em/+L06U\nZeHb1Ln/DtwAjbr9/OhstxpcNH95Qzh4M8ZcR5MmlFEr34Lc5bV8tTp8mzoPo5pei6cPLcTkA3Nh\nkAbzbxQSArzzDu18njIF+PVXi/vkKHYWHkTvzf9Fsb4U63u+g8ERPf590fnzwM8/UzY1NZKyVFe7\nbP7yhnDwZoy5DiFo6lSjoeQpFvLSeOKzLjPwSPxIzD+5HGP+fgFVBgum44OCKBNbcjLwzDPA2rUW\n98ne1uVsxYC/HoSvVoc/ey1G95B6ynoak7KMHKlOx/Lz6c/cBfOXN4SDN2PMtXh70/GxoiJaC7WQ\nRmgwP/kxzGwzHsvOrMU1Wx5GTqUFpUD9/YE336SUqs8/T2efncz7p1Zh6LaJiPdtgk29PkCSf3zd\nF5aVAV99RSVU1cgpXlZGU+VqJYBxIBy8GWOuJywMaNnSqvVvgBK5PJX4H3za+UVsKdyH1D/vRlrR\nYfNv5OcHvPEG0L07MH068PXXVvVLLVWGaozbOwtj976Mq8NTsbHXYsT4NLCb+7vvaNe3GjW7jfnL\n27Vz2fzlDXG/3zFjzD0kJtKot6TE6lvdFTsUf/ZaDL2sQa9N92HlmZ/Nv4lOR/XAe/cGXnkFWLHC\n6n7Z0vnKfAzZ8jDeOfUlHm85Bt93n49gz4D6X2AwUFKW9u0p77ytuUH+8oZw8GaMuSYPDwoipaWA\n3oLSn1dICW6HHX0+QZegJIzc9QweOzAPlTVV5t3E2xuYPZumlWfPBj75xOp+KWXXIR989kMoftvh\njz9y05CycQy2Fe7H0i4v4bV2E6EV2oZvsHEjVVcbPdr2x7XcJH95Qzh4M8ZcV2AgbRazcvrcKFoX\njvU938W45sMx98RS9Nj0H+wvPm7eTTw9gZkzgSFDaCp98WJF+map7FwP9PlvEm57IgHfbfbFqHUf\nov/msZB6L/zZazFGx1xn2o2WLQOio6kIia25Sf7yhnDwZoy5tmbNgIgISp2pAG+tF97q8CTWpM7F\nmYocpGy8GwtPrjTvPLiHB/DSS8ANN1BZ0bfftqi0qbWkBG6enIhB3S/gx89+wIlrr8a59q9hgMcw\nVC1MQ4K2nh3lVzp0CNi5k3aY2zopixvlL28IB2/GmGvTaGgd1mAAqsyc5m7A/0X1w55+yzEwrBsm\n7H8dg7eMw6GSdNNvoNUCU6dSIpMPP6RRuMoBfOPf/iiu1MN30Cx02zQax8oy8UXXWfht6FO4uiPw\n6fcmricvXUqjYDWSsrhR/vKGcPBmjLk+Hx/KvlZQoGiAjNaF4/vuC/BOh6ews+ggOv4+Ek8fXIhS\nfblpN9Bo6Pz3HXcAn34KvP66VcfbzPXF8TScG94dTx16E4PCU7Gn33Lc3nQwAODq1GKkHTFhWtqY\nlOXmm22flMXN8pc3hIM3Y8w9REfTFLoV2dfqIoTAQ81vx+GBX2F0zHWYdfxjtPt9OFae+dm0zGwa\nDfD448Bdd1GCk5kzbR7AT5Zl4c5dz2Ghz0hUyAp8mzoPa1LnIdYn6p9rjmd6IzLUhMQ0X3xB/R0x\nwoY9hlvmL28IB2/GmPtISqINY+UmjozNEOUdho87T8MfV72PIA9/jNz1DDr+MRIrz/yMGtlIrXEh\nqIjJffcBq1bRWXAL6pM35kRpJu7fPQOtfxuGr86ux1Mt7ofv4r0IPXv5prSMs574cE0Y7rmxkY1+\n5eV0Zn3AACqJakuFhW6Xv7wh7r1owBhzL15elH1t82ba9GSD5B59w7rg735L8WX2r3jxyGKM3PUM\n2vq3wJMJ9+CG0MH4bn0Mjpz2RvPoKoy8Nh9B/rWjbCGAceOoj+++S0ehXnxRkbXdtKLDeCN9BT7J\n/AEeQov/Nb8dTybcgxifSPSfeg43P5aAu67PR5/OJThwUod3vozAM/eeRZv4yrpveOECsGEDJWUp\nKrJ9zW5jpTg3y1/eEGFRxRzji4UYDmAagLYAuksp6yxeK4S4DsACAFoAi6WUs0y5f0pKitzhwvVw\nGWN2cvQocPw47UK3IYM0UBA/+j72F5+AqAxEXM7tGKobjrzD3fHrtkAsf/kEhvQsvvyFS5ZQStWB\nAymhi6en2W0XVZdgWdZafJCxGjuLDkKn8caDzYfhiYS70VR3+e87/YwX3l8VjoPpOjQNr8Z/b85F\nlzZXzE6UlAB//AH88guVOdXrKQXqbbcBY8bYdir73Dk6s2/r0b0DEELslFKmNHqdlcG7LQADgPcA\nTKkreAshtACOABgCIBPAdgCjpJQHGrs/B2/GmE3U1ABbttCILqCBrGEKKa8Amj1cgPbDF2KrYS0q\nDJVo698CHdAPPy4ZgQOzQxAbdkWAXrGCErn06QO8+mqjhTeklDhaeho/5fyFn3K24Nfc7agwVKJT\nYGvc3+xmjI65DqFeQWZ2vBz480/akLZpE+3Wj4qiM+pDhtCub1uvP5eVURu9erlFGlRTg7dV8zFS\nyoO1jTV0WXcAx6SUJ2qvXQHgZgCNBm/GGLMJrZYKhfz5J+1Et/Gxo9W/h6CrrhV+vu55FFZPwrKs\ntfjm7Aaszl+OytuWoOVmb/QOb4/2AQlIDmiJ9gGJSLhlMII8AZ+ZsyEmTwbmzAF0OugNelzQl+Jc\nZT72Fx/H/pIT2F98AtsLDyC9/AwAING3GR6IuwX3xN6IrkFtGvs/+nKVlTSy/uknyppWUUE7vG+9\nFbjmGtq1r1YQNeYv79nTLQK3OdRY844BkHHJ15kA6igCyxhjKvL3p+xre/fSaNKGTmR5o1vbUgBA\nsGcAxsUPx7j44SirqcCjXx3H1oqNKA/Zgo8zvkNJTdnFF3oDHlM1CCzfCs9v+6PYR4MyXL4DXECg\npW8MugQl4fGEMbg24iok+Jk5vVxdDWzdSiPs33+nlLLBwZREZsgQoEsXesOjNjfPX96QRoO3EGId\ngOg6fvSslHK10h0SQowFMBYA4uLilL49Y4xdFBtL55QLCylY2UjLmEp8uCb8X9/31epQvu8G/KfN\nQEzqcx5SSmRUnMP+4uM4VZ6NouoSXNCXoujEQVTt243AwnIE6T0Q2CwR4e1S0LbzYLQJToSvVmd+\np/R6YMcOWsP+7TfahBYQAAwaRCPslBT7JkLh/OUNavRPRko52Mo2sgA0u+Tr2Nrv1dfeIgCLAFrz\ntrJtxhirnxA0+t648WLaTRu4ZUAhHpsfi9UbgnDzgKJ/vv9nmh9+2BSEeZMzarsjEOcTjTifK8ZL\nbQBcZwB27wbWrgW+WQcs+QwI+paC7XXX0S76xqaWa2qAtDQaYa9fT0lr/PyAfv0oYPfsadHmOJvg\n/OUNsmrD2j83EWID6t+w5gHasDYIFLS3Axgtpdzf2H15wxpjTBXnzwPbt9P0uY02YG3f74ubJici\npV0peiSXYs8xH6zfXs9u88YYp7nXrqUjWxUV1PdrrgGuvZbOsxt/HwYDLQ388guwbh2Qm0vlSfv2\npeuvuoq+diSVlfTRt6/bpUFVa7f5rQDeBBABoBBAmpTyWiFEU9CRsKG11w0FMB90VOxDKeXLptyf\ngzdjTDUHDlBJy/B/T28rpaxC4PNfQus+522p8nJap/7pJzq/XlMDxMdTEC8ro6B99iydH+/dm9aw\n+/aljXqO6vx5oGtXm+9FcESqBG9b4+DNGFNNdTUFP43GeadqCwtpOnztWmDXLhq19uxJAbt/f9vn\nHldCcTG9seje3S3ToKpyVIwxxlyGpyetG2/aRKNUZ5yuDQ4Ghg2jj9xc+n04UzpRY/7yrl3dMnCb\ngw/OMcaYUVAQlQ/NzaXd2M4sPNy5AjdAG+g4f7lJOHgzxtil4uIogYsrBHBnUl1No23OX24SDt6M\nMXal2FhKTJKXd7EoBrOtggIq92mj43quhoM3Y4zVpWlToFs3Om9cVWXv3ri2sjJKENO0qb174jQ4\neDPGWH2iooDUVBoVVtZTHpNZx5i/vF07zl9uBn5SjDHWkIgIoEcPSh9aUWHv3rgezl9uEQ7ejDHW\nmLAwCuDFxXSUiSmD85dbjIM3Y4yZIiSEUomWldEHs15+PgVuZ02KY0ccvBljzFRBQZSxrKKCymYy\ny1VU0M5yrh5pEQ7ejDFmjsBACuDV1UBJib1747yKimiTmjNmsnMAHLwZY8xcAQEUwA0GWgdn5iku\npo2AkZH27onT4uDNGGOW8POjTWxC0CiSmcaYv7xNG85fbgUO3owxZilfX6p+5elJFb1Y4woKqGRp\nQIC9e+LUOHgzxpg1fHwokYtOR4GJ1c+Yv7xlS3v3xOlx8GaMMWvpdEBKCk2l5+fbuzeOi/OXK4aD\nN2OMKcHbm3KhBwZSQRN2Oc5frigO3owxphQvL6BrV0r1yQH8Is5frjh+iowxpiRPT6BzZ0qpmpNj\n7944hsJCzl+uMA7ejDGmNA8PCuBRUcD58zTydFc1NYBez/nLFcbBmzHGbEGrBTp2BGJi3DuAc/5y\nm+DgzRhjtqLVAu3bA82b0xS6uwVwzl9uMxy8GWPMljQa2qgVH08jcIPB3j1SD+cvtxkO3owxZmtC\nUDrQhAQagbtDAOf85TbFwZsxxtQgBK39tm5NI/CaGnv3yHYMBjrXzfnLbYaDN2OMqUUIIDGRsozl\n5LhuAC8oAFq04PzlNsTBmzHG1NayJZCcTAFcr7d3b5RlzF+ekGDvnrg0Dt6MMWYP8fF0lCw317UC\nuDF/uZeXvXvi0jh4M8aYvTRrRslccnNpxOrsOH+5ajh4M8aYPcXEUEGT/HygqsrevbGclMCFC7Qc\nwPnLbY6fMGOM2Vt0NJUULSwEKivt3RvLFBUBsbFASIi9e+IWOHgzxpgjiIwEunenIFhRYe/emKem\nhqb9W7Wyd0/cBgdvxhhzFGFhQI8eQEmJcwVwzl+uOg7ejDHmSEJDKYCXltIGMEfH+cvtgoM3Y4w5\nmuBgoGdPCoylpfbuTd0qKoC8vIub1Dh/uar4aTPGmCMKDKQAvnUr7eT297dvf6SkNxLl5fR5cDCl\nPw0NtX/f3BAHb8YYc1QBARTAt22jQh9qpxutqaH198pKypoWEUFr2yEhgE6nbl/YZTh4M8aYI/P3\npwC+fTsVNPH3t+3GsOpqCtjV1TQV3qQJHWULCgI8PW3XLjMLB2/GGHN0vr5Ar160xpyeTkFcq6WA\nqsRac0UFBWyDgUbUcXE0yg4K4oQrDoqDN2OMOQNPTxoBR0fT2vPZsxTIq6oo4AYEmF5+U0rayV5W\nRp8HBABJSXRUzd+fy3g6AQ7ejDHmbPz8qGpXixZUCCQjg4I5QIG4rvVog4GCvvH8eFgYlScNCQF8\nfNTrO1MEB2/GGHNWGg0F4bAw2lSWk3NxWt3Tk4J8WRmNzrXaiyP34GCu+uXkOHgzxpgr8Pam3OIx\nMbQzPSsLOHeOKnxFRdH6tVZr714yhXDwZowxVyIEnREPDKS62swl8TZCxhhjzMlw8GaMMcacDAdv\nxhhjzMlw8GaMMcacDAdvxhhjzMlYFbyFEMOFEPuFEAYhREoD16ULIfYKIdKEEDusaZMxxhhzd9Ye\nFdsHYBiA90y4dqCUMtfK9hhjjDG3Z1XwllIeBADBeXAZY4wx1ai15i0BrBNC7BRCjFWpTcYYY8wl\nNTryFkKsAxBdx4+elVKuNrGdPlLKLCFEJIBfhBCHpJR/1NPeWABjASAuLs7E2zPGGGPuo9HgLaUc\nbG0jUsqs2l/PCyFWAegOoM7gLaVcBGARAKSkpEhr22aMMcZcjc2nzYUQfkKIAOPnAK4BbXRjjDHG\nmAWsPSp2qxAiE8BVAL4XQvxU+/2mQogfai+LAvCnEGI3gG0AvpdSrrWmXcYYY8ydCSkdd2ZaCJED\n4JSCtwwHwMfVrMPP0Hr8DK3Hz1AZ/Bytp/QzbC6ljGjsIocO3koTQuyQUtabTIY1jp+h9fgZWo+f\noTL4OVrPXs+Q06MyxhhjToaDN2OMMeZk3C14L7J3B1wAP0Pr8TO0Hj9DZfBztJ5dnqFbrXkzxhhj\nrsDdRt6MMcaY03O54C2EuE4IcVgIcUwI8VQdPxdCiDdqf75HCNHVHv10dCY8xztrn99eIcRmIUQn\ne/TTkTX2DC+5LlUIoRdC3K5m/5yBKc9QCDGgttzwfiHE72r30dGZ8G85SAjxrRBid+0zvNce/XRk\nQogPhRDnhRB1JhizS1yRUrrMBwAtgOMAWgLwArAbQLsrrhkK4EcAAkBPAFvt3W9H+zDxOfYCEFL7\n+fX8HM1/hpdctx7ADwBut3e/HenDxL+HwQAOAIir/TrS3v12pA8Tn+EzAF6t/TwCQD4AL3v33ZE+\nAPQD0BXAvnp+rnpccbWRd3cAx6SUJ6SUVQBWALj5imtuBvCJJFsABAshmqjdUQfX6HOUUm6WUhbU\nfrkFQKzKfXR0pvxdBIAJAL4CcF7NzjkJU57haABfSylPA1Q/QeU+OjpTnqEEECCotrM/KHjr1e2m\nY5NUSCu/gUtUjyuuFrxjAGRc8nVm7ffMvcbdmfuM/gt618kuavQZCiFiANwK4B0V++VMTPl72BpA\niBBiQ23J4btV651zMOUZLgTQFsAZAHsBTJRSGtTpnstQPa40WlWMsYYIIQaCgncfe/fFCc0H8KSU\n0kCDHmYBDwDdAAwC4APgLyHEFinlEft2y6lcCyANwNUAEkBlmzdKKS/Yt1usIa4WvLMANLvk69ja\n75l7jbsz6RkJIToCWAzgeillnkp9cxamPMMUACtqA3c4gKFCCL2U8ht1uujwTHmGmQDypJSlAEqF\nEH8A6ASAgzcx5RneC2CWpMXbY0KIkwDagApJMdOoHldcbdp8O4BWQogWQggvACMBrLnimjUA7q7d\nHdgTQJGUMlvtjjq4Rp+jECIOwNcAxvAop06NPkMpZQspZbyUMh7AlwDGceC+jCn/nlcD6COE8BBC\n+ALoAeCgyv10ZKY8w9OgmQsIIaIAJAE4oWovnZ/qccWlRt5SSr0QYjyAn0C7LD+UUu4XQjxU+/N3\nQbt6hwI4BqAM9K6TXcLE5/gCgDAAb9eOHPWSCxz8w8RnyBpgyjOUUh4UQqwFsAeAAcBiKWWdx3nc\nkYl/D2cA+FgIsRe0W/pJKSVXGruEEGI5gAEAwmvLYE8F4AnYL65whjXGGGPMybjatDljjDHm8jh4\nM8YYY06GgzdjjDHmZDh4M8YYY06GgzdjjDHmZDh4M8YYY06GgzdjjDHmZDh4M8YYY07m/wFGgn4O\nsCkxjQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x1476f666fd0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "N = 11\n",
    "plotBayesianEstimator(N)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the case where N=11, the height of the confidence interval is larger as we have less data points to learn from."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The height of the band can vary by the density of the sampled data points. For some interval that contains many points, the variance should be smaller leading to a small band height. On the other hand, for an interval with only a few points the variance should be higher leading to a large band height."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}