tghosh_HW04_ist707.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Author: Trina Ghosh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IST 707: Assignment 04"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis using Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random Forest, Logistic Regression and Deep Learning Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Here, the dataset consists of 2 columns, Reviews and Sentiment. We have to perform text mining and run algorithms to predict the sentiment of each word, related to that particular review. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 358,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 359,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/Users/trinaghosh/Desktop/Trina/IST 707/HW 4'"
      ]
     },
     "execution_count": 359,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.getcwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 360,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['doc_term_matrix.csv', 'HW4_yelp_sentiment.csv', 'Untitled.ipynb', 'try 2.R', '.ipynb_checkpoints', 'try 1.R', 'try 2.ipynb']\n"
     ]
    }
   ],
   "source": [
    "print(os.listdir(os.getcwd()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading Dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"HW4_yelp_sentiment.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reviews</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>Wow... Loved this place.</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>Crust is not good.</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>Not tasty and the texture was just nasty.</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>Stopped by during the late May bank holiday of...</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>The selection on the menu was great and so wer...</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             reviews sentiment\n",
       "0                           Wow... Loved this place.         p\n",
       "1                                 Crust is not good.         n\n",
       "2          Not tasty and the texture was just nasty.         n\n",
       "3  Stopped by during the late May bank holiday of...         p\n",
       "4  The selection on the menu was great and so wer...         p"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Converting data set into Document-Term Matrix (DTM). \n",
    "### The conversion was done using R. The R markdown attached with this report contains all the code and respective output.\n",
    "### The dataframe was converted into documents. All punctuations, stop words, stem words and numbers have been removed from the documents. All characters were converted to lowercase. Finally, the DTM contained the count of words in each document, along with the sentiment associated with each document. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 425,
   "metadata": {},
   "outputs": [],
   "source": [
    "DTM = pd.read_csv(\"doc_term_matrix.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 426,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>love</th>\n",
       "      <th>place</th>\n",
       "      <th>wow</th>\n",
       "      <th>crust</th>\n",
       "      <th>good</th>\n",
       "      <th>just</th>\n",
       "      <th>nasti</th>\n",
       "      <th>tasti</th>\n",
       "      <th>textur</th>\n",
       "      <th>...</th>\n",
       "      <th>els</th>\n",
       "      <th>caterpillar</th>\n",
       "      <th>appetit</th>\n",
       "      <th>instant</th>\n",
       "      <th>ninja</th>\n",
       "      <th>draw</th>\n",
       "      <th>hadnt</th>\n",
       "      <th>pour</th>\n",
       "      <th>wound</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>9</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10 rows × 1594 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0  love  place  wow  crust  good  just  nasti  tasti  textur  ...  \\\n",
       "0           1     1      1    1      0     0     0      0      0       0  ...   \n",
       "1           2     0      0    0      1     1     0      0      0       0  ...   \n",
       "2           3     0      0    0      0     0     1      1      1       1  ...   \n",
       "3           4     1      0    0      0     0     0      0      0       0  ...   \n",
       "4           5     0      0    0      0     0     0      0      0       0  ...   \n",
       "5           6     0      0    0      0     0     0      0      0       0  ...   \n",
       "6           7     0      0    0      0     0     0      0      0       0  ...   \n",
       "7           8     0      0    0      0     0     0      0      0       0  ...   \n",
       "8           9     0      0    0      0     0     0      0      0       0  ...   \n",
       "9          10     0      0    0      0     0     0      0      0       0  ...   \n",
       "\n",
       "   els  caterpillar  appetit  instant  ninja  draw  hadnt  pour  wound  \\\n",
       "0    0            0        0        0      0     0      0     0      0   \n",
       "1    0            0        0        0      0     0      0     0      0   \n",
       "2    0            0        0        0      0     0      0     0      0   \n",
       "3    0            0        0        0      0     0      0     0      0   \n",
       "4    0            0        0        0      0     0      0     0      0   \n",
       "5    0            0        0        0      0     0      0     0      0   \n",
       "6    0            0        0        0      0     0      0     0      0   \n",
       "7    0            0        0        0      0     0      0     0      0   \n",
       "8    0            0        0        0      0     0      0     0      0   \n",
       "9    0            0        0        0      0     0      0     0      0   \n",
       "\n",
       "   Sentiment  \n",
       "0          p  \n",
       "1          n  \n",
       "2          n  \n",
       "3          p  \n",
       "4          p  \n",
       "5          n  \n",
       "6          n  \n",
       "7          n  \n",
       "8          p  \n",
       "9          p  \n",
       "\n",
       "[10 rows x 1594 columns]"
      ]
     },
     "execution_count": 426,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DTM.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(988, 1594)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DTM.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "del DTM['Doc']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>love</th>\n",
       "      <th>place</th>\n",
       "      <th>wow</th>\n",
       "      <th>crust</th>\n",
       "      <th>good</th>\n",
       "      <th>just</th>\n",
       "      <th>nasti</th>\n",
       "      <th>tasti</th>\n",
       "      <th>textur</th>\n",
       "      <th>bank</th>\n",
       "      <th>...</th>\n",
       "      <th>els</th>\n",
       "      <th>caterpillar</th>\n",
       "      <th>appetit</th>\n",
       "      <th>instant</th>\n",
       "      <th>ninja</th>\n",
       "      <th>draw</th>\n",
       "      <th>hadnt</th>\n",
       "      <th>pour</th>\n",
       "      <th>wound</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>n</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1593 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   love  place  wow  crust  good  just  nasti  tasti  textur  bank  ...  els  \\\n",
       "0     1      1    1      0     0     0      0      0       0     0  ...    0   \n",
       "1     0      0    0      1     1     0      0      0       0     0  ...    0   \n",
       "2     0      0    0      0     0     1      1      1       1     0  ...    0   \n",
       "3     1      0    0      0     0     0      0      0       0     1  ...    0   \n",
       "4     0      0    0      0     0     0      0      0       0     0  ...    0   \n",
       "\n",
       "   caterpillar  appetit  instant  ninja  draw  hadnt  pour  wound  Sentiment  \n",
       "0            0        0        0      0     0      0     0      0          p  \n",
       "1            0        0        0      0     0      0     0      0          n  \n",
       "2            0        0        0      0     0      0     0      0          n  \n",
       "3            0        0        0      0     0      0     0      0          p  \n",
       "4            0        0        0      0     0      0     0      0          p  \n",
       "\n",
       "[5 rows x 1593 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DTM.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(988, 1593)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DTM.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "n    494\n",
       "p    494\n",
       "Name: Sentiment, dtype: int64"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DTM['Sentiment'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Dummies for Sentiment (target) Column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy = pd.get_dummies(DTM['Sentiment'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>n</th>\n",
       "      <th>p</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   n  p\n",
       "0  0  1\n",
       "1  1  0\n",
       "2  1  0\n",
       "3  0  1\n",
       "4  0  1"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([DTM, dummy['p']],axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>love</th>\n",
       "      <th>place</th>\n",
       "      <th>wow</th>\n",
       "      <th>crust</th>\n",
       "      <th>good</th>\n",
       "      <th>just</th>\n",
       "      <th>nasti</th>\n",
       "      <th>tasti</th>\n",
       "      <th>textur</th>\n",
       "      <th>bank</th>\n",
       "      <th>...</th>\n",
       "      <th>caterpillar</th>\n",
       "      <th>appetit</th>\n",
       "      <th>instant</th>\n",
       "      <th>ninja</th>\n",
       "      <th>draw</th>\n",
       "      <th>hadnt</th>\n",
       "      <th>pour</th>\n",
       "      <th>wound</th>\n",
       "      <th>Sentiment</th>\n",
       "      <th>p</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>n</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>n</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>p</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1594 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   love  place  wow  crust  good  just  nasti  tasti  textur  bank  ...  \\\n",
       "0     1      1    1      0     0     0      0      0       0     0  ...   \n",
       "1     0      0    0      1     1     0      0      0       0     0  ...   \n",
       "2     0      0    0      0     0     1      1      1       1     0  ...   \n",
       "3     1      0    0      0     0     0      0      0       0     1  ...   \n",
       "4     0      0    0      0     0     0      0      0       0     0  ...   \n",
       "\n",
       "   caterpillar  appetit  instant  ninja  draw  hadnt  pour  wound  Sentiment  \\\n",
       "0            0        0        0      0     0      0     0      0          p   \n",
       "1            0        0        0      0     0      0     0      0          n   \n",
       "2            0        0        0      0     0      0     0      0          n   \n",
       "3            0        0        0      0     0      0     0      0          p   \n",
       "4            0        0        0      0     0      0     0      0          p   \n",
       "\n",
       "   p  \n",
       "0  1  \n",
       "1  0  \n",
       "2  0  \n",
       "3  1  \n",
       "4  1  \n",
       "\n",
       "[5 rows x 1594 columns]"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "del df['Sentiment']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>love</th>\n",
       "      <th>place</th>\n",
       "      <th>wow</th>\n",
       "      <th>crust</th>\n",
       "      <th>good</th>\n",
       "      <th>just</th>\n",
       "      <th>nasti</th>\n",
       "      <th>tasti</th>\n",
       "      <th>textur</th>\n",
       "      <th>bank</th>\n",
       "      <th>...</th>\n",
       "      <th>els</th>\n",
       "      <th>caterpillar</th>\n",
       "      <th>appetit</th>\n",
       "      <th>instant</th>\n",
       "      <th>ninja</th>\n",
       "      <th>draw</th>\n",
       "      <th>hadnt</th>\n",
       "      <th>pour</th>\n",
       "      <th>wound</th>\n",
       "      <th>p</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1593 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   love  place  wow  crust  good  just  nasti  tasti  textur  bank  ...  els  \\\n",
       "0     1      1    1      0     0     0      0      0       0     0  ...    0   \n",
       "1     0      0    0      1     1     0      0      0       0     0  ...    0   \n",
       "2     0      0    0      0     0     1      1      1       1     0  ...    0   \n",
       "3     1      0    0      0     0     0      0      0       0     1  ...    0   \n",
       "4     0      0    0      0     0     0      0      0       0     0  ...    0   \n",
       "\n",
       "   caterpillar  appetit  instant  ninja  draw  hadnt  pour  wound  p  \n",
       "0            0        0        0      0     0      0     0      0  1  \n",
       "1            0        0        0      0     0      0     0      0  0  \n",
       "2            0        0        0      0     0      0     0      0  0  \n",
       "3            0        0        0      0     0      0     0      0  1  \n",
       "4            0        0        0      0     0      0     0      0  1  \n",
       "\n",
       "[5 rows x 1593 columns]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(988, 1593)"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Splitting DF into train and test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split, ShuffleSplit\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Splitting data into 80% training set and 20% validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df[df.columns[0:1592]], df[df.columns[1592]], test_size=0.2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>love</th>\n",
       "      <th>place</th>\n",
       "      <th>wow</th>\n",
       "      <th>crust</th>\n",
       "      <th>good</th>\n",
       "      <th>just</th>\n",
       "      <th>nasti</th>\n",
       "      <th>tasti</th>\n",
       "      <th>textur</th>\n",
       "      <th>bank</th>\n",
       "      <th>...</th>\n",
       "      <th>refri</th>\n",
       "      <th>els</th>\n",
       "      <th>caterpillar</th>\n",
       "      <th>appetit</th>\n",
       "      <th>instant</th>\n",
       "      <th>ninja</th>\n",
       "      <th>draw</th>\n",
       "      <th>hadnt</th>\n",
       "      <th>pour</th>\n",
       "      <th>wound</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>17</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>585</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>709</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>854</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>720</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1592 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     love  place  wow  crust  good  just  nasti  tasti  textur  bank  ...  \\\n",
       "17      0      0    0      0     0     0      0      0       0     0  ...   \n",
       "585     0      0    0      0     0     0      0      0       0     0  ...   \n",
       "709     0      0    0      0     0     0      0      0       0     0  ...   \n",
       "854     0      0    0      0     0     0      0      0       0     0  ...   \n",
       "720     0      0    0      0     0     0      0      0       0     0  ...   \n",
       "\n",
       "     refri  els  caterpillar  appetit  instant  ninja  draw  hadnt  pour  \\\n",
       "17       0    0            0        0        0      0     0      0     0   \n",
       "585      0    0            0        0        0      0     0      0     0   \n",
       "709      0    0            0        0        0      0     0      0     0   \n",
       "854      0    0            0        0        0      0     0      0     0   \n",
       "720      0    0            0        0        0      0     0      0     0   \n",
       "\n",
       "     wound  \n",
       "17       0  \n",
       "585      0  \n",
       "709      0  \n",
       "854      0  \n",
       "720      0  \n",
       "\n",
       "[5 rows x 1592 columns]"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>love</th>\n",
       "      <th>place</th>\n",
       "      <th>wow</th>\n",
       "      <th>crust</th>\n",
       "      <th>good</th>\n",
       "      <th>just</th>\n",
       "      <th>nasti</th>\n",
       "      <th>tasti</th>\n",
       "      <th>textur</th>\n",
       "      <th>bank</th>\n",
       "      <th>...</th>\n",
       "      <th>refri</th>\n",
       "      <th>els</th>\n",
       "      <th>caterpillar</th>\n",
       "      <th>appetit</th>\n",
       "      <th>instant</th>\n",
       "      <th>ninja</th>\n",
       "      <th>draw</th>\n",
       "      <th>hadnt</th>\n",
       "      <th>pour</th>\n",
       "      <th>wound</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>196</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>346</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>788</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>316</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>552</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 1592 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     love  place  wow  crust  good  just  nasti  tasti  textur  bank  ...  \\\n",
       "196     0      0    0      0     0     0      0      0       0     0  ...   \n",
       "346     0      0    0      0     0     0      0      0       0     0  ...   \n",
       "788     0      1    0      0     0     0      0      0       0     0  ...   \n",
       "316     0      0    0      0     1     1      0      0       0     0  ...   \n",
       "552     0      1    0      0     0     0      0      0       0     0  ...   \n",
       "\n",
       "     refri  els  caterpillar  appetit  instant  ninja  draw  hadnt  pour  \\\n",
       "196      0    0            0        0        0      0     0      0     0   \n",
       "346      0    0            0        0        0      0     0      0     0   \n",
       "788      0    0            0        0        0      0     0      0     0   \n",
       "316      0    0            0        0        0      0     0      0     0   \n",
       "552      0    0            0        0        0      0     0      0     0   \n",
       "\n",
       "     wound  \n",
       "196      0  \n",
       "346      0  \n",
       "788      0  \n",
       "316      0  \n",
       "552      0  \n",
       "\n",
       "[5 rows x 1592 columns]"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "17     0\n",
       "585    0\n",
       "709    1\n",
       "854    1\n",
       "720    0\n",
       "Name: p, dtype: uint8"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "196    1\n",
       "346    1\n",
       "788    1\n",
       "316    1\n",
       "552    1\n",
       "Name: p, dtype: uint8"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applying ML Algorithm 1 : Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 377,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 378,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg = LogisticRegression()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 379,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/trinaghosh/opt/anaconda3/lib/python3.7/site-packages/sklearn/linear_model/logistic.py:432: FutureWarning: Default solver will be changed to 'lbfgs' in 0.22. Specify a solver to silence this warning.\n",
      "  FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,\n",
       "                   intercept_scaling=1, l1_ratio=None, max_iter=100,\n",
       "                   multi_class='warn', n_jobs=None, penalty='l2',\n",
       "                   random_state=None, solver='warn', tol=0.0001, verbose=0,\n",
       "                   warm_start=False)"
      ]
     },
     "execution_count": 379,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "logreg.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 380,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred=logreg.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 381,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[81, 20],\n",
       "       [28, 69]])"
      ]
     },
     "execution_count": 381,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cnf_matrix = metrics.confusion_matrix(y_test, y_pred)\n",
    "cnf_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 397,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_param_grid = {\n",
    "    'C':[0.01,0.1,0.5,1,5,6,9,10,11,12,15],\n",
    "    'penalty':['l1','l2'],\n",
    "    'solver' : ['liblinear'],\n",
    "    'fit_intercept':[True,False],\n",
    "    'tol': [0.0001,0.001,0.01,0.1,1,10],\n",
    "    'dual' : [False],\n",
    "    'intercept_scaling' : [0.25,0.5,1,2,4],\n",
    "    'max_iter' : [5,10,15,20]\n",
    "    \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 398,
   "metadata": {},
   "outputs": [],
   "source": [
    "log_grid_search = GridSearchCV(estimator = logreg, param_grid = log_param_grid, \n",
    "                          cv = 5, n_jobs = -1, verbose = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 399,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 5280 candidates, totalling 26400 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  25 tasks      | elapsed:    2.8s\n",
      "[Parallel(n_jobs=-1)]: Done 764 tasks      | elapsed:    5.3s\n",
      "[Parallel(n_jobs=-1)]: Done 2388 tasks      | elapsed:   10.2s\n",
      "[Parallel(n_jobs=-1)]: Done 4652 tasks      | elapsed:   16.7s\n",
      "[Parallel(n_jobs=-1)]: Done 7136 tasks      | elapsed:  3.0min\n",
      "[Parallel(n_jobs=-1)]: Done 10052 tasks      | elapsed:  3.2min\n",
      "[Parallel(n_jobs=-1)]: Done 14268 tasks      | elapsed:  3.4min\n",
      "[Parallel(n_jobs=-1)]: Done 19124 tasks      | elapsed:  3.7min\n",
      "[Parallel(n_jobs=-1)]: Done 24636 tasks      | elapsed:  4.2min\n",
      "[Parallel(n_jobs=-1)]: Done 26400 out of 26400 | elapsed:  4.3min finished\n",
      "/Users/trinaghosh/opt/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_search.py:814: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "             estimator=LogisticRegression(C=1.0, class_weight=None, dual=False,\n",
       "                                          fit_intercept=True,\n",
       "                                          intercept_scaling=1, l1_ratio=None,\n",
       "                                          max_iter=100, multi_class='warn',\n",
       "                                          n_jobs=None, penalty='l2',\n",
       "                                          random_state=None, solver='warn',\n",
       "                                          tol=0.0001, verbose=0,\n",
       "                                          warm_start=False),\n",
       "             iid='warn', n_jobs=-1,\n",
       "             param_grid={'C': [0.01, 0.1, 0.5, 1, 5, 6, 9, 10, 11, 12, 15],\n",
       "                         'dual': [False], 'fit_intercept': [True, False],\n",
       "                         'intercept_scaling': [0.25, 0.5, 1, 2, 4],\n",
       "                         'max_iter': [5, 10, 15, 20], 'penalty': ['l1', 'l2'],\n",
       "                         'solver': ['liblinear'],\n",
       "                         'tol': [0.0001, 0.001, 0.01, 0.1, 1, 10]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=2)"
      ]
     },
     "execution_count": 399,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_grid_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 400,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 6,\n",
       " 'dual': False,\n",
       " 'fit_intercept': True,\n",
       " 'intercept_scaling': 1,\n",
       " 'max_iter': 10,\n",
       " 'penalty': 'l1',\n",
       " 'solver': 'liblinear',\n",
       " 'tol': 0.01}"
      ]
     },
     "execution_count": 400,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "log_grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 401,
   "metadata": {},
   "outputs": [],
   "source": [
    "logreg_result = log_grid_search.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 402,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 75.76%\n"
     ]
    }
   ],
   "source": [
    "print(f\"Accuracy: {round(metrics.accuracy_score(y_test, logreg_result)*100, 2)}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.datasets import make_classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 295,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0   0.743119  0.801980  0.771429       101\n",
      "           1   0.775281  0.711340  0.741935        97\n",
      "\n",
      "    accuracy                       0.757576       198\n",
      "   macro avg   0.759200  0.756660  0.756682       198\n",
      "weighted avg   0.758875  0.757576  0.756980       198\n",
      "\n"
     ]
    }
   ],
   "source": [
    "log_report = classification_report(\n",
    "    digits=6,\n",
    "    y_true=y_test, \n",
    "    y_pred=logreg.predict(X_test))\n",
    "print(log_report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 403,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.datasets import make_classification\n",
    "from sklearn.metrics import roc_curve\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from matplotlib import pyplot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 406,
   "metadata": {},
   "outputs": [],
   "source": [
    "# generate a no skill prediction (majority class)\n",
    "ns_probs = [0 for _ in range(len(y_test))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 418,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_probs = log_grid_search.predict_proba(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 419,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_probs = lr_probs[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 420,
   "metadata": {},
   "outputs": [],
   "source": [
    "ns_auc = roc_auc_score(y_test, ns_probs)\n",
    "lr_auc = roc_auc_score(y_test, lr_probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 421,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No Skill: ROC AUC=0.500\n",
      "Logistic: ROC AUC=0.840\n"
     ]
    }
   ],
   "source": [
    "print('No Skill: ROC AUC=%.3f' % (ns_auc))\n",
    "print('Logistic: ROC AUC=%.3f' % (lr_auc))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 422,
   "metadata": {},
   "outputs": [],
   "source": [
    "ns_fpr, ns_tpr, _ = roc_curve(y_test, ns_probs)\n",
    "lr_fpr, lr_tpr, _ = roc_curve(y_test, lr_probs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 423,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hUZfbA8e9JCL0moSch9B4QQwAVBVGKIqyIiiCCqKyurK4VLMti2/Vn17UgKoquCrs0UbCLggoELIQQROkJICC9hbTz++NOYAgpkzKZzMz5PA9P5pa5c26AOfe+73vPK6qKMcaY4BXi6wCMMcb4liUCY4wJcpYIjDEmyFkiMMaYIGeJwBhjglwlXwdQXJGRkRobG+vrMIwxxq/88MMPf6hq/fy2+V0iiI2NZdWqVb4Owxhj/IqIbC1omzUNGWNMkLNEYIwxQc4SgTHGBDm/6yPIT2ZmJmlpaaSnp/s6lAqratWqREVFERYW5utQjDEVTEAkgrS0NGrVqkVsbCwi4utwKhxVZe/evaSlpdG8eXNfh2OMqWC81jQkItNFZLeIJBewXUTkBRHZICJJItKtpJ+Vnp5ORESEJYECiAgRERF2x2SMyZc3+wjeAgYWsn0Q0Nr1ZzzwSmk+zJJA4ez3Y0w+UhNh6dPOT28dq6w+oyxjzcNrTUOqukREYgvZZSjwtjp1sJeLSF0RaayqO70VkzHGnJSaCDMug6wTEBIKcVdD7aYlO9ah7ZA0C3KyTz9WQeuLKWt/GqFr/4toDoRWgTELIDqhZLHmw5d9BE2BVLflNNe6MxKBiIzHuWsgJiamXIIrLhHhzjvv5Omnnwbgqaee4siRI0yZMsWj9+/atYsbbriB1NRUMjMziY2NZdGiRXz99dc89dRTfPTRR6ftv2DBAlJSUpg0aRJTpkyhZs2a3H333YwdO5bBgwczfPjwsj5FYyqO1ETYshRie5f8C3HLUicJoJCTBT+/C5T0ztltXpfTjlXQ+uIdOcT9ONkZTuwBkgjy+23kO0uOqk4DpgHEx8dXyJl0qlSpwty5c7nvvvuIjIws9vsnT57MxRdfzO233w5AUlJSofsPGTKEIUOGlChWY/xa3iv5btdBnejiH2f7j5z2lTP4eYgfW4qYhjhf0qGVT12xF7TeAwePZ/KvReuYuTKVS+pu49+ZUwjNyXSOE9u7ZHEWwJeJIA1w/9uLAnb4KJZSq1SpEuPHj+fZZ5/lscceO23b1q1bGTduHHv27KF+/fq8+eabZ9zZ7Ny5k/79+59cjouLO+MzVq5cyfjx45kzZw5Llixh1apVvPjii945IWO8raRX9Xmv5FdNL4NgQuD43pK/PTrB+ZLPez4FrS9Cdo5yxSvfs2nPEf58QQvuuGggob+fXfq7oAL4MhEsACaIyEygB3CwrPoHrn512RnrBsc1ZnSvWI5nZDP2zTM7W4afHcWV8dHsO5rBLf/54bRts/7cy6PPvfXWW4mLi+Pee+89bf2ECRO47rrrGDNmDNOnT+e2225j/vz5Z7z36quv5sUXX+Siiy7i+uuvp0mTJie3f//99/z1r3/lgw8+ICYmhiVLlngUkzEV0mlX9ZWg+01Qr5ln792ZzGlX8oOedO4KiittJbw7HLLL6Co7OiH/L+iC1udj/9EM6lYPIzREuLt/W5rUrUpcVN1iH6e4vJYIROR9oA8QKSJpwD+AMABVnQosAi4BNgDHgOu9FUt5qV27Ntdddx0vvPAC1apVO7l+2bJlzJ07F4DRo0efkSgABgwYwKZNm/jkk0/4+OOPOeuss0hOdkberlu3jvHjx/PZZ5+dlhyMOaks2sw9PWaZt89nwoqXSxhkCGQchrCqxX9r894w5kOvXWUXh6oy/+ftPPRhChMHtuOahBgGdmpUbp/vzVFD1xSxXYFbvfHZhV3BV6scWuj28BqVPb4DyM/f/vY3unXrxvXXF5zXChrKGR4ezsiRIxk5ciSDBw9myZIlRERE0LhxY9LT0/npp58sEZgzpSbCjMGQleFcXfe6FcJL+eDgvs2w7CWn6cX9mAWtL67dv3DaVX3/f0LXQr8yTtn+I8waVTZX8l68yvbUjgPHeWDeGhav38NZMXWJb1av3GMIiCeLK5Lw8HCuuuoq3njjDcaNGwfAOeecw8yZMxk9ejTvvvsu55133hnv++qrr+jZsyfVq1fn8OHDbNy4kZiYGI4ePUrdunV544036N+/PzVq1KBPnz7lfFbGpwq7Mt+8BNZ/7Lq6xrm6/u65sv38go5ZZp8VAtnpUD3cs91bX1RhruRL64Oft/PAvGSyc5TJgzsw5pxYQkPK/5kfSwRecNddd53WifvCCy8wbtw4nnzyyZOdxXn98MMPTJgwgUqVKpGTk8ONN95I9+7d+frrrwFo2LAhH374IYMGDWL69LLoHDN+wb0tPbQSnHcXRLSEvRvh26edq2IACQEEQsNg+HRoclbpPnfHTzB7nOuq2+2YBa0v1fFLcFVfAa7ky0KdamF0ja7Lv4Z1Jjq8us/iEKeFxn/Ex8dr3olp1q1bR/v27X0Ukf+w31MFVFR7+9Kn4cuHizhICJw9BupG+08fQVkex49kZefwxrebyczOYcKFrQGnf6A8nvwXkR9UNT6/bXZHYIyvnLzaz3Cu9vvcB5GtT9/nYNrpy30fgE5XwM7VMP9myM5yrqi7jiz7L9MyGAVTouMHqJQdh5g4J4k12w9yaVzjkwmgIpR/sURgTHnLvRI+mAZZrkKA2Rnw5UNFvDHEeYAqoqXzp05U0F1R+6MTWdm8+NUGXvl6I3Wrh/HyqG4M6tSoQiSAXJYIjClP7ncBeb8Izr8XOuR5Wvz3ZPjwdmeUTt629CC7ovZXW/44xtRvNjKkaxP+fmkH6tWo7OuQzmCJwJjytGXpqbuA07rnQpyx8I06n75/o87O1b9d+fuVoyey+DxlF386qyltG9Xiyzv7EBPhu87golgiMKY8/PYFrFsAVWpzshiZhDqjfTSn8JEzduXvV5b+tof75q5h+4HjdGpam1YNalXoJACWCIzxvtREeO9K5wvfXUioUx7h+F672g8AB49l8tiiFP67Ko0WkTWYNb4XrRrU8nVYHrHJ68tIzZo1S32MHTt2FFo++sCBA7z88sse7298YONiWHgX/PiO8wTs9h+devR5kwA4NeqP74Xed1kS8HPZOcoVU79nzo/b+Uufliy6vTcJzT18QK4CsDuCCqRJkybMnj27wO25ieAvf/mLR/ubcpaaCP8Zlv+XvruQsKKbg4xf2Hc0g7rVnCJx9wxoS9O61ejUtI6vwyq24L0j8OK0b7m2bt1Kv379iIuLo1+/fmzbtg2AjRs30rNnT7p3787kyZNP3k1s2bKFTp06AbB27VoSEhLo2rUrcXFx/Pbbb0yaNImNGzfStWtX7rnnntP2z87O5u6776Zz587ExcXx73//22vnZfLYtAQ+ngTLX3FLAgJxI+CaWc7Pk9NvhEC3a+HCB8p8lilTflSVOT+k0fepr5m50plfa0DHRn6ZBCAQ7wg+ngS/ryl8nxOHYFey859WQqBhJ1cnXgEadYZBjxc7lILKT99+++3cfvvtXHPNNUydOjXf906dOpXbb7+dUaNGkZGRQXZ2No8//jjJycn8/PPPgJM4ck2bNo3Nmzfz008/UalSJfbt21fseE0JpCbCO0PzuQtQiOkFbQc6NXRSPjg1OUkXLzz8ZcpN2v5j3D8vmSW/7uHsZvX8qgmoIIGXCDyRfvDUf1zNcZYLSwQlVFD56WXLlp2cj2DkyJHcfffdZ7y3V69ePPbYY6SlpTFs2DBat259xj7uvvjiC26++WYqVXL+SsPD/f8fZ4WweSn89hlExUPjLmduX/dhAU1BbhOdlHByElPxzPspjQfnJaPAQ0M6MrpnM0J8UCSurAVeIvDkyj3v9HFXvF4u/zmL8yThyJEj6dGjBwsXLmTAgAG8/vrrtGjRosD9y6teSVBJTYS3hxTd5u+uoPZ/GwIaEMJrVOHs2HD+eXknoupV7CGhxRF4icAT5XSFVlD56Z49ezJnzhyuvvpqZs6cme97N23aRIsWLbjtttvYtGkTSUlJdOnShcOHD+e7f//+/Zk6dSp9+vQ52TRkdwWltGWpWxIIgQ6XQZuBp+/z6ydOs0/uPt2udebPtSv/gJCZncNrSzeRla3c1q81F7Spz/mtIwPuois4EwGU+RXasWPHiIqKOrl85513Flh++rnnnuPaa6/l6aef5tJLL6VOnTM7mGbNmsV//vMfwsLCaNSoEZMnTyY8PJxzzz2XTp06MWjQIG699dS8PjfeeCO//vorcXFxhIWFcdNNNzFhwoQyO7+gFNubkw9/VaoCvSac+W8mohX8+pm1/weg5O0HmTgnibU7DnFZlyYVqkhcWbMy1D5w7NgxqlWrhogwc+ZM3n//fT744IOi31hK/vZ78pmty2HTYqezd96fQbOh74MQPzb//YOwnHIgS8/M5oUvf+PVJZuoV70yj/6pIwM7NfZ1WKVmZagrmNxJaFSVunXr2kQzFUlqIrw16Mx+gU8mQcMO3i3LbCqErXuP8drSTQw7qykPXtqBOtXDfB2S11ki8IHevXuzevVqX4cR2LatcKZxjD0Porp7/r5N3+TfOZyd4Vz12xd+QDp6IotP1/7OsG5RtG1Ui6/u6uPTGcPKW8AkAhs1Uzh/awIsldREeHNg8Ub7FMSeAg543/y6h/vnrmHHwePERdWhVYNaQZUEIEASQdWqVdm7dy8RERGWDPKhquzdu5eqVav6OpTycdpoH4EWfaDZuZ69d+t3Tv8AYKOAAtv+oxk8sjCFuT9up2X9Gvzvz/5TJK6sBUQiiIqKIi0tjT179vg6lAqratWqp41qCmjVItwWFDr8qeCO3rxSL4Bty20UUIDLLRK3de8xJvRtxYQLW1E1LNTXYflMQCSCsLAwmjdv7uswTEWR+0QvcNoTvp6wp4AD2t4jJ6hXvTKhIcKkge1oWq8aHZv4Z32gshQQicCY05xsyxdn/H9x2/ZtFFDAUVX+90Maj36UwsRB7RjVoxn9OzbydVgVhiUC43+KGrcfnQCVa0GVWnDBRPtSD3Kp+45x/7w1LP3tDxJiw+nVIqLoNwUZSwTGv6QmwvQBpzqDq9Vz2vLdZWdAxmHIOFL4+H8T8Ob+mMaD85MR4JE/dWJUQkxAFIkra5YIjH/IvQs4mHb6sNB6zaFx3On77kyC4wcAtfH/QS6yZhUSmofz2OWdaVq3mq/DqbAsEZiKL+9dgLtuY84cEZS3uqyN/w8amdk5vPrNRrJz4PaLWnN+m/qc36a+r8Oq8CwRmIqroLuAkwoYEWQjf4JS8vaD3DM7iXU7DzG0axN7yLQYLBGYiik1EaYPdAq+5eXJ07428idopGdm89wXv/Ha0k2E16jMq6PPZoCNCCoWryYCERkIPA+EAq+r6uN5tscAM4C6rn0mqeoib8Zk/MSWpfknAXva1+Sxbd8x3vh2E8O7RXH/Je2DokhcWfNaIhCRUOAl4GIgDVgpIgtUNcVttweB/6rqKyLSAVgExHorJuNH3J8FCA1zfuZk2dO+BoDD6Zl8kvw7V8ZH06ZhLRbf3SegZgwrb968I0gANqjqJgARmQkMBdwTgQK5kwXXAXZ4MR7jT6ITQEKhThScd6czBNTa/A2w+JfdPDBvDb8fSuesmLq0alDLkkAphXjx2E2BVLflNNc6d1OAa0UkDedu4K/5HUhExovIKhFZZfWEgsTqmU7T0IFtzrMAAL3vsiQQxPYdzeCOWT9z/VsrqVGlErNvOSdoi8SVNW8mgvy66/PWQr4GeEtVo4BLgHdE5IyYVHWaqsaranz9+jYULKAdSIX5f3FmBgNOexbABK3sHGX4K9/z4eod3NavNR/ddh7dYur5OqyA4c2moTQg2m05ijObfm4ABgKo6jIRqQpEAru9GJepiI7vh6XPwIpXneXOw2HdR5Cdac8CBLE9h08QUcMpEnf/Je1pWq8a7RvXLvqNpli8mQhWAq1FpDmwHRgBjMyzzzagH/CWiLQHqgLW9hNMMo9D4jRY+jSkH4Iu10Df+6BujM0FHMRUlf+uSuXRheuYOLAd1/ZsxkUdGvo6rIDltUSgqlkiMgH4FGdo6HRVXSsiDwOrVHUBcBfwmojcgdNsNFaDaiqtIJaTDavfh8X/hEPboXV/6PcPaNTp1D72LEBQ2rb3GJPmJvH9xr30aB7Oea0ifR1SwPPqcwSuZwIW5Vk32e11CuDh1FEmIKjCb5/BF1Ngdwo06QaXvwrNrenHwOwf0vj7/GRCQ4THLu/ENd2tSFx5sCeLTflJWwWfT3amgwxvAVe+5cweZmUAjEvD2lU4p2UEj17eicZ1rEhcebFEYMpGQe35qYmwdj7sXA1bv4Ua9eGSp+Dssa4HxUwwy8jK4ZWvN5Kjyh0Xt6F36/r0bm0jA8ubJQJTettWwFuXOE/+Sii0GQA1G8CR3fDrp6dKRZw1GgY+DlVq+jZeUyGsTj3AvbOTWL/rMMPOampF4nzIEkGwK4uRORu+cJIAOF/6m5dA5RqQcfRUEpBQCG9uScBwPCObZz5fzxvfbqZBraq8fl28jQjyMUsEwWzrcphxqTOCJyQU2g+BWiWo2rhz9enL/R9z5giweQFMPlL3H2PG91sZkRDDpEHtqF3Vmgh9zRJBsMjvyv+XD09dyedkwS8LncneiyvrhNuC2xwBNi+AcTnkKhJ3latI3Nf39KGJzRhWYVgiCAZ5r/w7DYfaTWD7j6fvN+iJM2f78kRhV/72LEDQ++qXXdw/N5ndh9PpFlOPVg1qWhKoYCwRBLLcu4Dfk0+/8l/zX6fN/rR6/wXM9uUJu/I3+dh75AQPf5TCBz/voG3DWkwdfTatGlgfUUVkiSBQbV0OMwafSgDuLn227Nvw7crfuMnOUa6cuozU/ce446I23NKnJZUrebPGpSkNjxKBiFQGYlR1g5fjMWUlZR7kZOazwdrwjffsPpxOZI0qhIYID1zanqh61WnbyEpFV3RFpmgRuRRYA3zuWu4qIvO8HZgppUZdnJ8S4lzth1ZxmoMqVTmzDd/q/JtSyslR3l2xlQuf+oZ3E7cB0K99Q0sCfsKTO4KHgR7AYgBV/VlEWnk1KlNyuf0CNRs7yw06QvcbbYYv4zVb/jjKpLlJLN+0j3NaRnCBPRnsdzxJBJmqeiDPE39WIbQi2roc3h4M2VnOnQDArrXODF9jFjhX/saUof+uSuXv85OpHBrC48M6c3X3aHs62A95kgjWichVQIhrboHbgeXeDcuUSMo8ZyIXcBsRlHNqhi+7EzBlrGndapzfpj6PDO1EozpVfR2OKSFPEsEEYDKQA8zFmV/gPm8GZUqoUWfXixAIdf3V5mTbU72mzJzIyublxRtRVe7s35ZzW0Vyrs0X4Pc8SQQDVHUiMDF3hYgMw0kKpiyU1UxcDTs6Pxt1gvgbrF/AlKmftu1n4pwkft11hCu6RVmRuADiSSJ4kDO/9B/IZ50pidyx/FknnKv4C+6FiBL2xW9e4vz8Pdn6BUyZOZaRxdOf/cr07zbTqHZVpo+N58J2ViQukBSYCERkAM7E8k1F5Bm3TbVxmolMWdiy1FWrx9WW/9WjZXBQ6xcwZWf7/uO8s3wro3rEMHFgO2pZkbiAU9gdwW4gGUgH1rqtPwxM8mZQQSW2t3MnkJ3hTNQy9GW3tv5i+n0NLJjgjBqyfgFTCgePZ/Lxmp2MSIihdcNafHNPH5sxLIAVmAhU9SfgJxF5V1XTyzGm4BKdAP2mwGf3w6CnIe6qkh+rQXuoF2v9AqZUPlv7Ow/OT2bv0QziY8Np1aCmJYEA50kfQVMReQzoAJwcH6aqbbwWVbCp39b52bBD6Y9lNX9MCf1x5ARTFqzlo6SdtGtUi9fHxFuRuCDhSSJ4C3gUeAoYBFyP9REYE1Cyc5Thr3zPjgPp3N2/DX++oCVhoVYkLlh4kgiqq+qnIvKUqm4EHhSRpd4OLOBtWwHJc5wSEHtSnHW7UiC6u2/jMkFl16F06td0isT947KORNWrRuuGVh8o2HiSCE6IM1h4o4jcDGwHGng3rACXmggzLoPsE6ev/+QeaNjemnaM1+XkKO8mbuP/Pv6FiQPbMrpXLH3b2X/rYOVJIrgDqAncBjwG1AHGeTOogLdl6alSEO6ys2zIp/G6TXuOMGnuGhI37+O8VpH0aWsJINgVmQhUdYXr5WFgNICIRHkzqIDnPmQ0JMwpEJdjQz6N981auY3JH6ylSqUQnhgex5VnR9nTwabwRCAi3YGmwLeq+oeIdMQpNXEhYMmgpKIT4MLJ8PmDMPgZqN/OhnyachFVrzp92jpF4hrUtiJxxlHYk8X/Aq4AVuN0EM/DqTz6f8DN5RNeAKvvGn3bsCM0PdsSgPGKE1nZ/PtLZ2LBuwdYkTiTv8LuCIYCXVT1uIiEAztcy+vLJzRjTGn8sHUf985OYuOeo1wVb0XiTMEKSwTpqnocQFX3icgvlgSMqfiOnsjiyU/XM2PZFprUqcaMcQlc0MZmDTMFKywRtBCR3AqjAsS6LaOqw4o6uIgMBJ4HQoHXVfXxfPa5CpiCM+vZalUd6Xn4xpi8dhw4znuJ27iuZzPuGdiOmlU8GRxogllh/0KuyLP8YnEOLCKhwEvAxUAasFJEFqhqits+rXEmuTlXVfeLiI1jM6YEDh7LZOGanYzs4RSJW3pvXxpaZ7DxUGFF574s5bETgA2quglARGbi9DukuO1zE/CSqu53febuUn5mxZaaCJu+gZiezlzC4PxserZv4zJ+7ZPk3/n7B8nsO5pBjxbhtKxf05KAKRZv3jM2BVLdltOAHnn2aQMgIt/hNB9NUdVP8h5IRMYD4wFiYmK8EqzXpSbCW4PPfJp44d3O8FEbNWSKaffhdKYsWMuiNb/ToXFt3hzbnZb1rUicKT5vJoL8hidoPp/fGuiD81zCUhHppKoHTnuT6jRgGkB8fHzeY1RsudNQHkxzHiDLK8eeJjbFl52jXDV1GTsOpnPPgLaMP7+FFYkzJeZxIhCRKqp6oug9T0oDot2Wo3CGoObdZ7mqZgKbRWQ9TmJYWYzPqbgKugsA54lizbGniU2x7Dx4nIa1qjpF4oZ0JLpedSsVbUqtyEsIEUkQkTXAb67lLiLybw+OvRJoLSLNRaQyMAJYkGef+UBf13EjcZqKNhUj/oqtoJpChEC3a+HCB5x5he1uwBQhJ0d567vN9Hv6G/6zYisAfds2sCRgyoQndwQvAINxvrRR1dUi0reoN6lqlohMAD7Faf+frqprReRhYJWqLnBt6y8iKUA2cI+q7i3huVQ8hdUU6jLSEoDxyIbdR5g0J4lVW/dzfpv6XGhVQk0Z8yQRhKjq1jxPJGZ7cnBVXQQsyrNusttrBe50/Qk80QnQdRT88Cacdwe0vthqCplimZm4jckL1lItLJSnr+zCsG5N7elgU+Y8SQSpIpIAqOvZgL8Cv3o3rACxdRn89I7z+vvnnUTQ+y7fxmT8SkxEdS5q34CHhnSifq0qvg7HBChPEsEtOM1DMcAu4AvXOuMud3RQbsdvygew5n9OUxDYXAPGI+mZ2bzw5W8A3DuwHee0jOScllYkzniXJ4kgS1VHeD0Sf5aaCDMGQ1Z+o4MqgaqNDjJFWrVlH/fOSWLTnqOM6B5tReJMufEkEax0DeucBcxV1cNejsn/bFkKWfk8IyCh0G001Im2fgFToCMnsnjyk194e/lWmtatxtvjEjjfisSZcuTJDGUtReQcnOGfD4nIz8BMVZ3p9ej8xcnRQZkQGgbY6CDjud8PHmfmylTG9IrlngFtqWFF4kw5E2fgjoc7O/MSPAeMUtVQr0VViPj4eF21apUvPrpwXz8OX/8LrpgOdaNtdJAp1P6jGXy0ZiejezYDYPehdJsxzHiViPygqvH5bSvy0kNEauIUixsBtAc+AM4p0wgDQUQr52fjOIhsbQnA5EtV+Tj5dyZ/kMyBY5mc0zKClvVrWhIwPuXJPWgy8CHwhKou9XI8xgSs3YfS+fsHyXy6dhedm9bh7XE9rEicqRA8SQQtVDXH65EYE8Cyc5QrX13G7wfTuW9QO244rzmVrEicqSAKm7z+aVW9C5gjImd0JHgyQ5kxwW7HgeM0qu0UiXt4aCei61Wjhd0FmAqmsDuCWa6fxZqZzBjj3AG8vWwLT3yynvsuacd1vWJt3mBTYRU2Q1mi62V7VT0tGbiKyZV2BrPAsneD83NnktNZbILWht2HuXd2Ej9uO0CftvXp176hr0MyplCeNFKOy2fdDWUdiF9LTYQlTzmvP7jFWTZB6b0V27jk+W/Z/MdRnr26C2+O7U7TutV8HZYxhSqsj+BqnCGjzUVkrtumWsCB/N8VhDKOwacPQo5r3gGrKRTUYiOr079jQ6YM6UhkTSsSZ/xDYX0EicBenJnFXnJbfxj4yZtB+Y3URJh3M+zbCCGhzkScVlMoqKRnZvPsF78iCJMGWZE4458K6yPYDGzGqTYa3PJWFt242OkTSJ4NtaPgugUQVs2eJg4yKzbtZdLcNWz+4yijesRYkTjjtwprGvpGVS8Qkf2cPum84MwpE+716CqC1ESYcZmrsqi4zt71WEWbQTBsGlSt7SxbAggKh9Mz+b9PfuE/y7cRE16d927swTmt7C7A+K/CmoZyp6MM7n/hW5a6koA6f3JTooRAdPdTScAEjV2HTjD7hzRuPK85d/ZvQ/XKViTO+LfCmoZynyaOBnaoaoaInAfEAf8BDpVDfL4X29uZUyAn02n/R05VFrW+gKCx72gGC5N2MLpXLK0a1GTpvRfajGEmYHhyKTMf6C4iLYG3gYXAezgT2ge+6AToNQG+exaGvwU161tfQBBRVT5K2smUBWs5lJ7Jua0iaVG/piUBE1A8SQQ5qpopIsOA51T1BREJrlFD4c2dn026Qp2mlgCCxK5D6TwwL5kv1u0iLqoO7w7vYeUhTEDyaKpKEbkSGA38ybUuzHsh+ZD76CD3L/t9m52fO352EoEJeNk5ylWuInEPXNKe68+NtSJxJmB5kgjGAX/BKUO9SUSaA+97NywfcB8dFBIKcdc4X/oHt8Pqd5195nokwFMAABeuSURBVIyDMR/aHUEAS9t/jMZ1qhEaIjwytBMx4dWJjazh67CM8SpPpqpMFpHbgFYi0g7YoKqPeT+0cuY+OignC35+58x9sjPtqeEAlZ2jvPndZp76bD33DWrPmHNibd5gEzQ8maGsN/AOsB1nFH0jERmtqt95O7hy5T46qFLVU1f+qYkwYwhkZ9hIoQC1/vfD3DsnidWpB+jXrgH9O1qROBNcPGkaeha4RFVTAESkPU5iyHfuS7/j3i/QYQgkz4G+D5666o9OgDELbKRQgPrP8q089OFaalUN4/kRXRnSpYk9HWyCjieJoHJuEgBQ1XUiUtmLMZUf934BCeHk02KLH4WYHqcnA0sAASW3HESrBjW5pHNjJg/uQIQViTNBypNE8KOIvIpzFwAwikApOufeL6DZp9ZbX0DAOp6RzTOfryckRLhvUHt6toigZ4sIX4dljE95Mh7uZmAjcC8wEdgE/NmbQZWb3H4BgNAwpw9AQq0vIEAt27iXgc8v4bWlmzl2IhvVM2ZgNSYoFXpHICKdgZbAPFV9onxCKkfRCdD9JljxMoyY6dQNsr6AgHMoPZN/LfqF9xO30SyiOu/d1MNKRRvjprDqo/fjzET2I06JiYdVdXq5RVZe6jVzfjbtBtXDLQEEoN2HTjD/p+2MP78Fd1zUhmqVQ30dkjEVSmFNQ6OAOFW9EugO3FLcg4vIQBFZLyIbRGRSIfsNFxEVkcAYiWR8bu+RE7z1nfNEeKsGNfl2Yl/uv6S9JQFj8lFY09AJVT0KoKp7RKRYz9eLSCjOzGYXA2nAShFZ4D4CybVfLeA2YEWxIjcmH6rKgtU7mLJgLUdOZHF+m/q0qF/TRgQZU4jCEkELt7mKBWjpPnexqg4r4tgJOE8hbwIQkZnAUCAlz36PAE8AdxcncGPy2nHgOA/OT+arX3bTNbouTwyPsyJxxnigsERwRZ7lF4t57KZAqttyGtDDfQcROQuIVtWPRKTARCAi44HxADExMcUMwwSDrOwcRkxbzp7DJ/j74A6MPSeW0BB7MMwYTxQ2Mc2XpTx2fv8LT47XczU1PQuMLepAqjoNmAYQHx9vY/7MSan7jtGkbjUqhYbwz8s7ExNenZiI6r4Oyxi/4s26umk4s5vligJ2uC3XAjoBX4vIFqAnsMA6jI0nsrJzmLZkIxc98w3vLNsCwHmtIy0JGFMC3pxsdSXQ2lW2ejswAhiZu1FVD+I2H7KIfA3craqrvBiTCQDrdh5i4pwkktIOcnGHhgzq3NjXIRnj1zxOBCJSRVVPeLq/qmaJyATgUyAUmK6qa0XkYWCVqi4ofrgm2L2zbAsPfZhCnWphvDjyLC7t3NiKxBlTSp6UoU4A3gDqADEi0gW4UVX/WtR7VXURsCjPuskF7NvHk4BNcMotEtemYS0u69KEvw/uQHiNwKh9aIyveXJH8ALORPXzAVR1tYj09WpUxrgcy8jiqU9/pVKocP8l7enRIoIeViTOmDLlSWdxiKpuzbMuO989jSlD3234gwHPLWH6d5vJyMqxInHGeIkndwSpruYhdT0t/FfgV++GZYLZweOZ/HPhOmatSqV5ZA3+++deJDQP93VYxgQsTxLBLTjNQzHALuALSlB3yBhP/XHkBB8m7eDmC1ryt4taUzXM6gMZ402eTF6/G2fopzFes+fwCT5cvYNx5zWnZf2afDvxQusMNqaceDJq6DXcngjOparjvRKRCSqqyvyft/PQhykcO5FN33YNaB5Zw5KAMeXIk6ahL9xeVwUu5/QaQsaUyPYDx3lg3hq+Xr+HbjFOkbjmkTV8HZYxQceTpqFZ7ssi8g7wudciMkHBKRK3jL1HMphyWQdG97Iiccb4SklKTDQHmpV1ICY4bNt7jKb1nCJxjw+LIya8OtHhVh/IGF8q8jkCEdkvIvtcfw7g3A3c7/3QTCDJys7hla83ctGz3/D2si0AnNsq0pKAMRVAUZPXC9AFp2gcQI4GwlM9qYmnJqnf73pWbvuP0Poi38YVoNbuOMjEOUkkbz/EgI4NudSKxBlToRSaCFRVRWSeqp5dXgF5XWoivHUpZGecvn7WKBjzoU1eX8ZmfL+FRz5KoW71yrwyqptVCjWmAvKkxESiiHTzeiTlZctSyM48c312prPNlIncG8d2jWoxtGtTvrjzfEsCxlRQBd4RiEglVc0CzgNuEpGNwFGcmcdUVf0zOcT2hpBKkJMJoZUBgZws53Vsb19H5/eOnsjiyU/XExYqPHBpBysSZ4wfKKxpKBHoBvypnGIpH9EJ0P0GWDEVRs6CyjVP9RdYs1CpLPl1D/fNXcOOg8cZ0yv2ZOloY0zFVlgiEABV3VhOsZSfujHOz6ZnQ9U6lgBK6eCxTB5ZmMLsH9JoUd8pEtc91orEGeMvCksE9UXkzoI2quozXojH+KE/jp7g4zU7+UufltzWz4rEGeNvCksEoUBNXHcGxrjbfTidBT/v4MbeLU4Wiatn9YGM8UuFJYKdqvpwuUVi/IKqMufH7TzyUQrHM7Pp174hzSNrWBIwxo8V2UdgTK7Ufce4f94alv72B/HN6vH4FVYkzphAUFgi6FduUZgKLys7h2teW87+oxk8MrQjo3o0I8SKxBkTEApMBKq6rzwDMRXTlj+OEh1enUqhITwx3CkSF1XP6gMZE0g8ebLYBKHM7BxeWryB/s8uOVkk7pyWkZYEjAlAJSlDbQJc8vaD3Ds7iZSdh7i0c2MGxzXxdUjGGC+yRGBO8+Z3m3l04TrCa1Rm6rVnM7BTI1+HZIzxMksEBuBkOYiOTeow7KymPHhpB+pUD/N1WMaYchA8icB9DoID25x123+Alhf6Ni4fO3Iiiyc++YXKoSE8OLgDCc3DSWhu5SGMCSbBkQhSE2HGYMg6wWmPR7x/TVDPQfD1+t08MC+ZHQePM+7c5lYkzpggFRyJYMtSyMqdiMZtgrXcOQiCLBHsP5rBIwtTmPvjdlo1qMnsm8/h7Gb1fB2WMcZHgiMRxPaGkNBT8w4E+RwE+49l8NnaXdx2YStuvbAVVSpZkThjgplXE4GIDASexylg97qqPp5n+53AjUAWsAcYp6pbyzyQ6AQ4eyysfB1GzYawakE3B8HuQ+nM/3k7N/VuQYv6Nflu4oXWGWyMAbyYCEQkFHgJuBhIA1aKyAJVTXHb7ScgXlWPicgtwBPA1V4JKHcOgqh4qFwjaBKAqvK/VWk8sjCFjKwcLu7QiOaRNSwJGGNO8uYdQQKwQVU3AYjITGAocDIRqOpit/2XA9d6MZ6gk7rvGPfNXcO3G/4goXk4jw/rbEXijDFn8GYiaAqkui2nAT0K2f8G4OP8NojIeGA8QExMTFnFF9Byi8QdOJbJo3/qxMiEGCsSZ4zJlzcTQX7fOprPOkTkWiAeuCC/7ao6DZgGEB8fn+8xjGPzH0eJcRWJe3J4F5pFVKdJ3Wq+DssYU4F5s+hcGhDtthwF7Mi7k4hcBDwADFHVE16MJ6BlZufw7y9/Y8CzS5jx/RYAerWMsCRgjCmSN+8IVgKtRaQ5sB0YAYx030FEzgJeBQaq6m4vxhLQktIOcO/sJH75/TCXdWnCkK5WJM4Y4zmvJQJVzRKRCcCnOMNHp6vqWhF5GFilqguAJ3HmRf6f64nWbao6xFsxBaLp327m0YUp1K9Vhdeui+fiDg19HZIxxs949TkCVV0ELMqzbrLb64u8+fmBLLccRFxUHa7uHs2kQe2pU82GhBpjii84niwOIIfTM3n841+oUimUyZd1ID42nPhYKxJnjCk5m6HMjyz+ZTf9n13C+4nbqBQqqNoAKmNM6dkdgR/YdzSDhz9cy/yfd9CmYU1eHnUOZ8VYkThjTNmwROAHDh7P5Mt1u7m9X2tu7duKypXsRs4YU3YsEVRQvx90isT9+fwWNI+swbeTLrTOYGOMV1giqGBUlZkrU/nnwnVk5uQwsGMjYiNrWBIwxniNJYIKZOveo0yas4Zlm/bSs0U4jw+LI9aKxBljvMwSQQWRlZ3DyNdWcPB4Jv+8vDMjukdbkThjTLmwROBjG/ccoZmrSNzTVzlF4hrXsfpAxpjyY8NPfCQjK4fnvviVgc8t4e1lzqRsPVtEWBIwxpQ7uyPwgZ9TDzBxdhLrdx1maNcm/Omspr4OyRgTxCwRlLM3vt3MYwtTaFCrKm+MiadfeysSZ4zxLUsE5SS3SFzX6DqMSIhh0qB21K5qQ0KNMb5nicDLDqVn8q9Fv1A1LIR/XNaRs5uFc3YzKxJnjKk4rLPYi75I2cXFz3zDrJXbqFwpxIrEGWMqJLsj8IK9R07w0IcpLFi9g3aNajFtdDxdouv6OixjjMmXJQIvOJyexeL1u7njojbc0qelFYkzxlRolgjKyI4Dx5n303b+0qclsZE1+G7ShdYZbIzxC5YISiknR3kvcRuPf/wL2TnKpZ0bExtZw5KAMcZvWCIohc1/HGXSnCRWbN7Hua0i+NflccREVPd1WMYYUyyWCEooKzuHa19fwaH0TJ64Io4r46MQsSJxxhj/Y4mgmDbsPkxsRA0qhYbw7NVdaRZRnYa1q/o6LGOMKTEbzuKhE1nZPPP5rwx8bikzXEXiEpqHWxIwxvg9uyPwwI/b9jNxdhK/7T7CsLOaMsyKxBljAoglgiK8tmQT//x4HY1rV+XN67vTt20DX4dkjDFlyhJBAXJylJAQoVuzuozqEcPEge2oZUNCjTEByBJBHgePZ/LYwhSqhYXy0NBOViTOGBPwrLPYzadrf+fiZ75hzo/bqVGlkhWJM8YEBbsjAP44coJ/fLCWhWt20qFxbaaP7U6npnV8HZYxxpQLSwTAkfQslv62h3sGtGX8+S0IC7UbJWNM8Aieb7wD25yfaasA2H7gOC9+9RuqSmxkDb6/rx+39m1lScAYE3S8+q0nIgNFZL2IbBCRSflsryIis1zbV4hIrFcCSU2EH94CQN+7io8//oD+z3zDS4s3snXvMQBqVrGbI2NMcPJaIhCRUOAlYBDQAbhGRDrk2e0GYL+qtgKeBf7PK8FsWQo52QDkZJ1gzbcL6dasHp/dcT6xkTW88pHGGOMvvHlHkABsUNVNqpoBzASG5tlnKDDD9Xo20E+8UbmtWgSKMwIoBKVvt3a8PS6B6HCrFGqMMd5MBE2BVLflNNe6fPdR1SzgIBCR90AiMl5EVonIqj179hQ/kuN7EXLzSwjdG6hVCjXGGBdvJoL8vmnzDsz3ZB9UdZqqxqtqfP369YsfSWxvqFQVJBSpVMVZNsYYA3h3+GgaEO22HAXsKGCfNBGpBNQB9pV5JNEJMGaB01cQ29tZNsYYA3g3EawEWotIc2A7MAIYmWefBcAYYBkwHPhKvfU4b3SCJQBjjMmH1xKBqmaJyATgUyAUmK6qa0XkYWCVqi4A3gDeEZENOHcCI7wVjzHGmPx5dfC8qi4CFuVZN9ntdTpwpTdjMMYYUzh7jNYYY4KcJQJjjAlylgiMMSbIWSIwxpggJ/42+YqI7AG2lvDtkcAfZRiOP7BzDg52zsGhNOfcTFXzfSLX7xJBaYjIKlWN93Uc5cnOOTjYOQcHb52zNQ0ZY0yQs0RgjDFBLtgSwTRfB+ADds7Bwc45OHjlnIOqj8AYY8yZgu2OwBhjTB6WCIwxJsgFZCIQkYEisl5ENojIpHy2VxGRWa7tK0QktvyjLFsenPOdIpIiIkki8qWINPNFnGWpqHN222+4iKiI+P1QQ0/OWUSucv1drxWR98o7xrLmwb/tGBFZLCI/uf59X+KLOMuKiEwXkd0iklzAdhGRF1y/jyQR6VbqD1XVgPqDU/J6I9ACqAysBjrk2ecvwFTX6xHALF/HXQ7n3Beo7np9SzCcs2u/WsASYDkQ7+u4y+HvuTXwE1DPtdzA13GXwzlPA25xve4AbPF13KU85/OBbkByAdsvAT7GmeGxJ7CitJ8ZiHcECcAGVd2kqhnATGBonn2GAjNcr2cD/cS/JzEu8pxVdbGqHnMtLseZMc6fefL3DPAI8ASQXp7BeYkn53wT8JKq7gdQ1d3lHGNZ8+ScFajtel2HM2dC9CuquoTCZ2ocCrytjuVAXRFpXJrPDMRE0BRIdVtOc63Ldx9VzQIOAhHlEp13eHLO7m7AuaLwZ0Wes4icBUSr6kflGZgXefL33AZoIyLfichyERlYbtF5hyfnPAW4VkTScOY/+Wv5hOYzxf3/XiSvTkzjI/ld2ecdI+vJPv7E4/MRkWuBeOACr0bkfYWes4iEAM8CY8sroHLgyd9zJZzmoT44d31LRaSTqh7wcmze4sk5XwO8papPi0gvnFkPO6lqjvfD84ky//4KxDuCNCDabTmKM28VT+4jIpVwbicLuxWr6Dw5Z0TkIuABYIiqniin2LylqHOuBXQCvhaRLThtqQv8vMPY03/bH6hqpqpuBtbjJAZ/5ck53wD8F0BVlwFVcYqzBSqP/r8XRyAmgpVAaxFpLiKVcTqDF+TZZwEwxvV6OPCVunph/FSR5+xqJnkVJwn4e7sxFHHOqnpQVSNVNVZVY3H6RYao6irfhFsmPPm3PR9nYAAiEonTVLSpXKMsW56c8zagH4CItMdJBHvKNcrytQC4zjV6qCdwUFV3luaAAdc0pKpZIjIB+BRnxMF0VV0rIg8Dq1R1AfAGzu3jBpw7gRG+i7j0PDznJ4GawP9c/eLbVHWIz4IuJQ/POaB4eM6fAv1FJAXIBu5R1b2+i7p0PDznu4DXROQOnCaSsf58YSci7+M07UW6+j3+AYQBqOpUnH6QS4ANwDHg+lJ/ph//vowxxpSBQGwaMsYYUwyWCIwxJshZIjDGmCBnicAYY4KcJQJjjAlylghMhSMi2SLys9uf2EL2jS2oSmMxP/NrV4XL1a7yDG1LcIybReQ61+uxItLEbdvrItKhjONcKSJdPXjP30Skemk/2wQuSwSmIjquql3d/mwpp88dpapdcAoSPlncN6vqVFV927U4Fmjitu1GVU0pkyhPxfkynsX5N8ASgSmQJQLjF1xX/ktF5EfXn3Py2aejiCS67iKSRKS1a/21butfFZHQIj5uCdDK9d5+rjr3a1x14qu41j8up+Z3eMq1boqI3C0iw3HqOb3r+sxqriv5eBG5RUSecIt5rIj8u4RxLsOt2JiIvCIiq8SZh+Ah17rbcBLSYhFZ7FrXX0SWuX6P/xORmkV8jglwlghMRVTNrVlonmvdbuBiVe0GXA28kM/7bgaeV9WuOF/Eaa6SA1cD57rWZwOjivj8y4A1IlIVeAu4WlU74zyJf4uIhAOXAx1VNQ541P3NqjobWIVz5d5VVY+7bZ4NDHNbvhqYVcI4B+KUlMj1gKrGA3HABSISp6ov4NSh6auqfV1lJx4ELnL9LlcBdxbxOSbABVyJCRMQjru+DN2FAS+62sSzcWro5LUMeEBEooC5qvqbiPQDzgZWukprVMNJKvl5V0SOA1twShm3BTar6q+u7TOAW4EXceY3eF1EFgIel7lW1T0isslVI+Y312d85zpuceKsgVNywX12qqtEZDzO/+vGOJO0JOV5b0/X+u9cn1MZ5/dmgpglAuMv7gB2AV1w7mTPmGhGVd8TkRXApcCnInIjTsneGap6nwefMcq9KJ2I5DtHhav+TQJOobMRwATgwmKcyyzgKuAXYJ6qqjjfyh7HiTNT1+PAS8AwEWkO3A10V9X9IvIWTvG1vAT4XFWvKUa8JsBZ05DxF3WAna4a86NxroZPIyItgE2u5pAFOE0kXwLDRaSBa59w8Xy+5l+AWBFp5VoeDXzjalOvo6qLcDpi8xu5cxinFHZ+5gJ/wqmjP8u1rlhxqmomThNPT1ezUm3gKHBQRBoCgwqIZTlwbu45iUh1Ecnv7soEEUsExl+8DIwRkeU4zUJH89nnaiBZRH4G2uFM55eC84X5mYgkAZ/jNJsUSVXTcSo7/k9E1gA5wFScL9WPXMf7BuduJa+3gKm5ncV5jrsfSAGaqWqia12x43T1PTwN3K2qq3HmKl4LTMdpbso1DfhYRBar6h6cEU3vuz5nOc7vygQxqz5qjDFBzu4IjDEmyFkiMMaYIGeJwBhjgpwlAmOMCXKWCIwxJshZIjDGmCBnicAYY4Lc/wMCNH/jd2v/lwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "pyplot.plot(ns_fpr, ns_tpr, linestyle='--', label='No Skill')\n",
    "pyplot.plot(lr_fpr, lr_tpr, marker='.', label='Logistic')\n",
    "# axis labels\n",
    "pyplot.xlabel('False Positive Rate')\n",
    "pyplot.ylabel('True Positive Rate')\n",
    "# show the legend\n",
    "pyplot.legend()\n",
    "# show the plot\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applying ML Algorithm 2 : ANN "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 427,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense\n",
    "from keras.wrappers.scikit_learn import KerasClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 428,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_model(optimizer='rmsprop', init='glorot_uniform'):\n",
    "\t# create model\n",
    "\tmodel = Sequential()\n",
    "\tmodel.add(Dense(12, input_dim=1592, kernel_initializer=init, activation='relu'))\n",
    "\tmodel.add(Dense(8, kernel_initializer=init, activation='relu'))\n",
    "\tmodel.add(Dense(1, kernel_initializer=init, activation='sigmoid'))\n",
    "\t# Compile model\n",
    "\tmodel.compile(loss='binary_crossentropy', optimizer=optimizer, metrics=['accuracy'])\n",
    "\treturn model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 429,
   "metadata": {},
   "outputs": [],
   "source": [
    "ann_model = KerasClassifier(build_fn=create_model, verbose=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 430,
   "metadata": {},
   "outputs": [],
   "source": [
    "ann_param_grid = {\n",
    "    'optimizer':['SGD', 'RMSprop', 'Adagrad', 'Adadelta', 'Adam', 'Adamax', 'Nadam'],\n",
    "    'init' : ['uniform', 'lecun_uniform', 'normal', 'zero', 'glorot_normal', 'glorot_uniform', 'he_normal', 'he_uniform'],\n",
    "    'epochs' : [7,8,9,10,11,12,13],\n",
    "    'batch_size' : [4,5,6,7],\n",
    "#    'learn_rate' : [0.001, 0.01, 0.1, 0.2, 0.3],\n",
    "#    'momentum' : [0.0, 0.2, 0.4, 0.6, 0.8, 0.9],\n",
    "#    'activation' : ['softmax', 'softplus', 'softsign', 'relu', 'tanh', 'sigmoid', 'hard_sigmoid', 'linear'],\n",
    "#    'weight_constraint' : [1, 2, 3, 4, 5],\n",
    "#    'dropout_rate' : [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9],\n",
    "#    'neurons' : [1, 5, 10, 15, 20, 25, 30]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 339,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 1344 candidates, totalling 6720 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  25 tasks      | elapsed:   12.1s\n",
      "[Parallel(n_jobs=-1)]: Done 146 tasks      | elapsed:   38.6s\n",
      "/Users/trinaghosh/opt/anaconda3/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py:706: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.\n",
      "  \"timeout or by a memory leak.\", UserWarning\n",
      "[Parallel(n_jobs=-1)]: Done 349 tasks      | elapsed:  1.7min\n",
      "[Parallel(n_jobs=-1)]: Done 632 tasks      | elapsed:  3.6min\n",
      "[Parallel(n_jobs=-1)]: Done 997 tasks      | elapsed:  6.6min\n",
      "[Parallel(n_jobs=-1)]: Done 1442 tasks      | elapsed: 11.1min\n",
      "[Parallel(n_jobs=-1)]: Done 1969 tasks      | elapsed: 16.1min\n",
      "[Parallel(n_jobs=-1)]: Done 2576 tasks      | elapsed: 19.8min\n",
      "[Parallel(n_jobs=-1)]: Done 3265 tasks      | elapsed: 26.1min\n",
      "[Parallel(n_jobs=-1)]: Done 4034 tasks      | elapsed: 30.4min\n",
      "[Parallel(n_jobs=-1)]: Done 4885 tasks      | elapsed: 36.7min\n",
      "[Parallel(n_jobs=-1)]: Done 5816 tasks      | elapsed: 41.9min\n",
      "[Parallel(n_jobs=-1)]: Done 6720 out of 6720 | elapsed: 48.9min finished\n"
     ]
    }
   ],
   "source": [
    "ann_grid = GridSearchCV(estimator=ann_model, param_grid=ann_param_grid, \n",
    "                        cv = 5, n_jobs = -1, verbose = 2)\n",
    "grid_result = ann_grid.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 340,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'batch_size': 5, 'epochs': 10, 'init': 'he_normal', 'optimizer': 'Adam'}"
      ]
     },
     "execution_count": 340,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ann_grid.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 341,
   "metadata": {},
   "outputs": [],
   "source": [
    "ann_result = ann_grid.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 342,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 75.76%\n"
     ]
    }
   ],
   "source": [
    "print(f\"Accuracy: {round(metrics.accuracy_score(y_test, ann_result)*100, 2)}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 367,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0   0.778947  0.732673  0.755102       101\n",
      "           1   0.737864  0.783505  0.760000        97\n",
      "\n",
      "    accuracy                       0.757576       198\n",
      "   macro avg   0.758406  0.758089  0.757551       198\n",
      "weighted avg   0.758821  0.757576  0.757502       198\n",
      "\n"
     ]
    }
   ],
   "source": [
    "ann_report = classification_report(\n",
    "    digits=6,\n",
    "    y_true=y_test, \n",
    "    y_pred=ann_grid.predict(X_test))\n",
    "print(ann_report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 417,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No Skill: ROC AUC=0.500\n",
      "ANN: ROC AUC=0.877\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deXhU5fXA8e9J2HeSsCchYd9kMwZFXBBUQIQWUSluaC1u1LqLYK1brT83qtW6U9EqaBEQt9pWRVBBNpVNUPYEkCVAWBOynN8fd4JDmCQTMnfW83mePM69c2fm3BDnvO9733teUVWMMcbErrhQB2CMMSa0LBEYY0yMs0RgjDExzhKBMcbEOEsExhgT46qFOoDKSkpK0rS0tFCHYYwxEWXJkiW7VLWJr+ciLhGkpaWxePHiUIdhjDERRUQ2lfWcDQ0ZY0yMs0RgjDExzhKBMcbEuIi7RuBLQUEB2dnZ5OXlhTqUsFSrVi2Sk5OpXr16qEMxxoShqEgE2dnZ1K9fn7S0NEQk1OGEFVUlJyeH7Oxs0tPTQx2OMSYMuTY0JCKTRWSHiKwo43kRkWdEZK2ILBOR3if6WXl5eSQmJloS8EFESExMtN6SMaZMbl4jeA0YVM7zg4H2np+xwPNV+TBLAmWz340Je1kLYd6Tzn+9H5d1TCxy8fxdGxpS1bkiklbOIcOB19Wpg71ARBqJSAtV3eZWTMaYMJS1EP4xBIoLgJJGi4LEQbNuULMB5O+D7StAi4/dHyOK8nKJ27ESUYVqteCq2ZCSGbD3D+WsoVZAltd2tmffcURkrIgsFpHFO3fuDEpwlSUi3H777Ue3n3jiCe6//36/X799+3aGDh1Kjx496NKlC0OGDAFgzpw5DB069LjjZ8+ezaOPPgrA/fffzxNPPAHAmDFjmD59ehXOxBgX+WrVbpznSQIA6vnB+dLPy3Ue5+U626X3x4DcwwVs37Hdc/4KRUec31kAhfJisa/xCp+r5KjqS8BLABkZGWG5kk7NmjWZMWMG99xzD0lJSZV+/X333ce5557LH/7wBwCWLVtW7vHDhg1j2LBhJxSrMSHh3fL3btXvLzUIEFfd+dKLrwEXveK0fLMWwpRhzpeg9/4olnu4gL989APT1mQxpNFm/lZwP/HFBc75p50R0M8KZY8gG0jx2k4GtoYoliqrVq0aY8eOZdKkScc9t2nTJgYMGED37t0ZMGAAmzdvPu6Ybdu2kZycfHS7e/fuxx2zaNEievXqxfr163nttdcYN25cYE/CmLIEYnzau+Xv3aovLvQ6KA56Xw7nTDx2+CMl09kuvT9KFRUrFz3/Ne8szuK6s9rw1O1jiR/zvmvnH8oewWxgnIhMA/oAuYG6PnDpi/OP2ze0ewuuOC2Nw0eKGPOP4/+YR56czMUZKew+eIQb/rnkmOfevu40vz73pptuonv37tx1113H7B83bhxXXnklV111FZMnT+bmm29m1qxZx7320ksv5dlnn2XgwIFcffXVtGzZ8ujzX3/9Nb///e957733SE1NZe7cuX7FZEyVldWSr6zSLf9+t0HGmONb+z1G+/6iS8mM+gSw5+ARGtWpTnyccMd5HWnZqBbdkxs5T7p4/m5OH50KzAc6iki2iPxWRK4Xkes9h3wErAfWAi8DN7oVS7A0aNCAK6+8kmeeeeaY/fPnz2f06NEAXHHFFXz55ZfHvfb8889n/fr1/O53v2P16tX06tWLkushP/zwA2PHjuX9998nNTXV/RMx4SsUM2fKaslXVumW/+Ec52GMtfZ9UVVmfptN/yfnMG2Rc+l0ULfmvyQBl7k5a+g3FTyvwE1ufHZ5LfjaNeLLfT6hbg2/ewC+3HLLLfTu3Zurr766zGPKms6ZkJDA6NGjGT16NEOHDmXu3LkkJibSokUL8vLy+Pbbb4/pJZgYE6iWeWWV1ZKvrNItf+9x7hho7Zdl697DTJy5nM/X7KRXaiMyWjcOegxWayjAEhISuOSSS3j11VeP7uvbty/Tpk0D4M0336Rfv37Hve6zzz7j0KFDAOzfv59169Ydbf03atSIDz/8kAkTJjBnzhz3T8IERqBb74FqmVdWWS35yrKW/3He+24L502ay4L1u7lvaBemX9+X9s3qBz2OqCgxEW5uv/12nn322aPbzzzzDNdccw2PP/44TZo04R//+Mdxr1myZAnjxo2jWrVqFBcXc+2113LKKacc/eJv1qwZ77//PoMHD2by5MnBOhVzotxovQeqZV5Z5bXkKyuGW/6+NKxdnZ4pjfjLiJNISagTsjjEGaGJHBkZGVp6YZoffviBzp07hyiiyGC/owDIWui0ytPOqPjLbN6T8OmDv2w3ag0NU8o+3h+5WbC3ZG2ROBhwL5xxe7kvCZjKnLspU2FRMa9+uYGComLGndMecK4PBOPufxFZoqoZvp6zHoEx/qhsC9+N1nsgW+aVZS35Klu1dR93v7uM5VtyuaB7i6MJIBxKwFgiMLGrMq1cX+Pz5SWCQI2reysZY7eWeUTJLyzi2c/W8vycdTSqU52/X9abwd2ah0UCKGGJwMQmt1v4brXerWUecTbuOsQLX6xjWM+W/PGCLjSuWyPUIR3HEoGJTW638K31HtMO5hfy31Xb+VWvVnRsXp9Pbzub1MTQXQyuiCUCE/18DQGlnYFT7kohvmbFtWtOpIVvrfeYNO+nndwzYzlb9h6mW6sGtGtaP6yTAFgiMNGurCGg/H38UuPQj5lz1sI3Fcg9VMCfP1rFO4uzaZNUl7fHnka7psG/J+BE2A1lATRz5kxEhNWrVwOwceNGRIS//e1vR48ZN24cr732GuCUjG7VqhX5+fkA7Nq1i7S0tGCHHXkqc6NWWTdhed+MVVzkX1nflExnuqYlAVNKUbFy0Qtf8+7SLdx4dls++sMZZKYnhDosv1mPIICmTp1Kv379mDZt2tG1CJo2bcrTTz/NddddR40ax18kio+PZ/Lkydxwww1BjjZCBeoibyinYpqosfvgERrVdorE3Xl+R1o1qk23Vg1DHValxW6PIMC3/x84cICvvvqKV1999Wg5CYAmTZowYMAApkyZ4vN1t9xyC5MmTaKwsNDn88aj5N/r+6mVK7Nghc6MC1SVd5dk0/+JX4rEnd+1eUQmAYjGHsHH4+Hn5eUfU9ll75qfBIMfLfctZ82axaBBg+jQoQMJCQksXbqUhASnazh+/HgGDx7MNddcc9zrUlNT6devH2+88QYXXnhhhacXk45ZyrCUqkzjtIu55gRk7znEhJkrmPvjTk5u3TiihoDKEn2JwB++lr2rYh2YqVOncssttwAwatQopk6dyk03OcVV09PTyczM5K233vL52gkTJjBs2DAuuOCCKsUQdUpm++Rm+04CNo3TBNnMb7O5d+YKFHhgWFeuOLU1cXHhc2PYiYq+RFBByx0I+LJ3OTk5fPbZZ6xYsQIRoaioCBHhxht/WWJhwoQJjBw5kjPPPPO417dr146ePXvyzjvvnHAMUae8XoD3UoY2jdMEUULdmpyclsAjv+5GcuPwnhJaGdGXCPwR4Fbi9OnTufLKK3nxxReP7jvrrLPIzs4+ut2pUye6dOnCBx98QGbm8Z83ceLE2OoRVFTe4ZgFzb15ljJsmGItfOO6gqJiXp63nsIi5eYB7TmrQxPObJ8UVuUhAiE2EwEEtJU4depUxo8ff8y+iy66iEceeeSYfRMnTqRXr14+36Nr16707t2bpUuXBiSmsObPzJ/yFjQvaylDYwJoxZZc7n53GSu37uPCHi3DqkhcoFkZ6hgRst+Rr5a/PyWaS5dczrjKegEmKPIKinjm0594ce56GtepwcO/6sqgbi1CHVaVWRlqExpltfz9KeDm74LmxgTYppxDvDxvPSN6teLeC7rQsE71UIfkOksEJvB8zfbxnp3lTwE3m+1jguhgfiGfrPyZEb2T6di8Pp/dfnZIVwwLtqhJBMFa5ScSBXX4z585//7e1WuzfUwQfPHjTibMWM7W3MN0T25Iu6b1YyoJQJQkglq1apGTk0NiYqIlg1JUlZycHGrVqhWcDyxvtk/pu3qttW9CaM/BIzz04SpmLN1C2yZ1+dd1kVMkLtCiIhEkJyeTnZ3Nzp07Qx1KWKpVqxbJycnB+bC0M5zrASUzfBBnKMju6jVhpKRI3KacQ4zr345x57SjVvX4UIcVMlGRCKpXr056enqow4ht3rODGqfDoRwY+AA062ItfxM2cg7k07hODeLjhPGDOtGqcW26tozM+kCBFBWJwITYMdcFPIu9APx7vDMEdMbtoYzOGFSVfy3J5uEPVnH34E5c1qc153VtHuqwwoYlAlN1x1wX8LowXXTEec56AiaEsnYfYsLM5cz7aReZaQmc1iYx1CGFHUsEpur8vS5gTJDNWJrNvbNWIMBDv+rGZZmpUVEkLtAsEZiqS8l0bhbLy3UK+IFdFzBhIaleTTLTE/jzr0+iVaPaoQ4nbFkiMIFRs4HzU/LFbwnAhEBBUTEvfrGOomL4w8D2nNmhCWd2aBLqsMKeJQJz4rxnCuXvc3oEWQstCZiQWLEllzunL+OHbfsY3rOl3WRaCZYIzIkpa6bQlGG29KMJqryCIv76v594ed56EurW4MUrTuZ8mxFUKa4mAhEZBDwNxAOvqOqjpZ5PBaYAjTzHjFfVj9yMyVSRz1XDbKaQCZ3Nuw/x6pfrGdk7mQlDOsdEkbhAcy0RiEg88BxwLpANLBKR2aq6yuuwe4F3VPV5EekCfASkuRWTqaJArhpmTBXszyvg3yt+5uKMFDo0q8/nd5wdVSuGBZubPYJMYK2qrgcQkWnAcMA7EShQsiJJQ2Cri/GYqrJVw0wY+Hz1DibOXM7P+/LoldqIdk3rWxKoIjcTQSsgy2s7G+hT6pj7gf+IyO+BusBAX28kImOBsQCpqakBD9T4qbz7BWy9AOOy3QeP8NAHq5j57RbaN63H9Bv6xmyRuEBzMxH4ulxfuh7yb4DXVPVJETkNeENEuqlq8TEvUn0JeAmcFcpciTZWVbR2sLeUTKsjZEKiqFgZ+fzXbN59iJsHtOem/m2pWS12i8QFmpuJIBvwXn8wmeOHfn4LDAJQ1fkiUgtIAna4GJcp4c/awd7y98Hudc5jqyNkgmDn/nwS6zpF4iYM6UyrxrXp3KKcv1FzQuJcfO9FQHsRSReRGsAoYHapYzYDAwBEpDNQC7Ba0sHiPeZfsoJYebyfL5kdZIwLVJW3F23mnCfn8NbCzQAM7NLMkoBLXOsRqGqhiIwDPsGZGjpZVVeKyIPAYlWdDdwOvCwit+IMG43RoC6nFeO8x/yr1XbKQ5Q3xOPvymLGVMHmnEOMn7GMr9fl0Cc9gX7tkkIdUtSTSPvezcjI0MWLF4c6jOjxwhm/1AjyZ5y/MtcUjKmk6Uuy+eOsFcTHCfcM6cRvTrEicYEiIktUNcPXc3ZncawrXSOoIraymHFRswY16ds2kYd/3Y0WDa1IXLBYIogVZbXkrUaQCaEjhcU8P2cdxarcem4HzmjfhDPaW5G4YLNEEAvKmh2Uvw9+XuYcYzWCTJB9n7WXu6YvY832/Yzo1cqKxIWQJYJo46vl72t2UM0GvmcBWSIwLjt8pIin/ruGV7/cQNP6tXjlygwGdmkW6rBimiWCaFJWy3//tmOP63cbZIyxWUAmJLL2HGLK15sYlZnK+MGdaFDLisSFmiWCSFWZln9xodcL4+BwjvMwJdMZDrJZQMZl+zxF4i7xFImbc+fZtLQVw8KGJYJIFMiWv80CMi77bPV2JsxYwY79efRObUy7pvUsCYQZSwSRxNdaANbyN2Eq50A+D36wive+20rHZvV54YqTade0XqjDMj5YIogU5a0FYC1/E2aKipWLX5hP1p5D3DqwAzec3ZYa1dysaGOqwq9E4KkVlKqqa12Ox5SlvLUArOVvwsSO/Xkk1a1JfJww8YLOJDeuQ8fmVio63FWYokXkAmA58F/Pdk8Rmel2YKaUkrpA4LT242uCxEO1mse3/M+43ZKACariYuXNbzZxzhNf8KanSNyAzs0sCUQIf3oED+IsKPM5gKp+JyLtXI3KHC8l07koXFIXCKzlb8LCxl0HGT9jGQvW76Zv20TOsjuDI44/iaBAVfeWuuMvsirVRYvSdYEsAZgQe2dxFn+ctYIa8XE8OuIkLj0lxe4OjkD+JIIfROQSIE5E0oE/AAvcDcsAx98rYHWBTJhp1ag2Z3ZowkPDu9G8Ya1Qh2NOUIVlqEWkLnAfcJ5n1yfAA6p62OXYfIqZMtSl7xVonP7L6mDValtdIBMS+YVF/P3zdagqt53XMdThmEqoahnq81X1buBurzccAcwIUHzGW1n3ChzK+eUYqwtkQuDbzXu4+91l/Lj9ABf1TrYicVHEn0RwL8d/6U/0sc9UVXn3Cpw8Br550eoCmaA7dKSQJ//zI5O/2kDzBrWYPCaDczpZkbhoUmYiEJHzcRaWbyUiT3k91QAodjuwmOKrF3CMOKjVwO4RMCGxZc9h3liwicv6pHL3oE7UtyJxUae8HsEOYAWQB6z02r8fGO9mUDGlvF5AXHVnWKikB2B3B5sgyT1cwMfLtzEqM5X2zerzxZ1n24phUazMRKCq3wLfisibqpoXxJhiS3l3DPe+HBqmWA/ABNV/Vv7MvbNWkHPwCBlpCbRrWs+SQJTz5xpBKxH5M9AFODo/TFU7uBZVLCm5Y7ik5Y84xePia0CP0ZYATNDsOpDP/bNX8sGybXRqXp9XrsqwInExwp9E8BrwMPAEMBi4GrtGEDh2x7AJA0XFysjnv2br3jzuOK8D153VlurxViQuVviTCOqo6ici8oSqrgPuFZF5bgcWNcpaNL4sdh3ABNH2fXk0qecUifvThV1Jblyb9s2sPlCs8ScR5IszWXidiFwPbAGauhtWlChrARlvtoC8CYHiYuXNhZv5v49Xc/egjlxxWhr9O9n/1rHKn0RwK1APuBn4M9AQuMbNoCJeRQvIeLMF5E2Qrd95gPEzlrNww276tUvi7I6WAGJdhYlAVb/xPNwPXAEgIsluBhXR/FlApvTxtoC8CZK3F23mvvdWUrNaHI+N7M7FJyfb3cGm/EQgIqcArYAvVXWXiHTFKTVxDmDJwJs/N4Udzjl+ty0mY4IouXEdzu7oFIlr2sCKxBlHeXcW/wW4CPge5wLxTJzKo/8HXB+c8CJEZW4K88UuEBuX5BcW8bdPnYUF7zi/I6e3S+L0dkkhjsqEm/J6BMOBHqp6WEQSgK2e7TXBCS2C2E1hJgwt2bSbu6YvY93Og1ySYUXiTNnKSwR5JaWmVXW3iKy2JFAGuynMhJGD+YU8/skapszfSMuGtZlyTSZndbBVw0zZyksEbUSkpMKoAGle26jqiIreXEQGAU8D8cArqvqoj2MuAe7HWfXse1Ud7X/4YcJuCjNhZOvew7y1cDNXntqaOwd1ol5NfyYHmlhW3l/IRaW2n63MG4tIPPAccC6QDSwSkdmqusrrmPbAPcDpqrpHRCJ3HpstI2lCKPdQAR8u38boPk6RuHl39aeZXQw2fiqv6NynVXzvTGCtqq4HEJFpONcdVnkd8zvgOVXd4/nMHVX8TGNizr9X/Mwf31vB7oNH6NMmgbZN6lkSMJXiZjGRVkCW13a2Z5+3DkAHEflKRBZ4hpKOIyJjRWSxiCzeuXOnS+EaE1l27M/jxjeXcP0/l9CkXk3eu+l02jaxInGm8twcPPQ1PaH0AsnVgPbA2Tj3JcwTkW6quveYF6m+BLwEzprFgQ+1EsqqHWQLy5sgKipWLnlhPltz87jz/I6MPbONFYkzJ8zvRCAiNVU1vxLvnQ2keG0n40xBLX3MAlUtADaIyBqcxLCoEp8TPGXVDrJ6QSZItuUepln9Wk6RuGFdSWlcx0pFmyqrsAkhIpkishz4ybPdQ0T+5sd7LwLai0i6iNQARgGzSx0zC+jved8knKGi9ZWIP7i87xcoqR0EvusFGRNAxcXKa19tYMCTX/DPbzYB0L9jU0sCJiD86RE8AwzF+dJGVb8Xkf4VvUhVC0VkHPAJzvTRyaq6UkQeBBar6mzPc+eJyCqgCLhTVX3UYQgT3vcLVKvtTBVNybR6QcZVa3ccYPy7y1i8aQ9ndmjCOVYl1ASYP4kgTlU3lbojscifN1fVj4CPSu27z+uxArd5fsJf6fsFvKeKWr0g44JpCzdz3+yV1K4ez5MX92BE71Z2d7AJOH8SQZaIZALquTfg98CP7oYVxkrfL1DC6gUZF6Qm1mFg56Y8MKwbTerXDHU4Jkr5kwhuwBkeSgW2A//z7Isd3jOFbHaQcVFeQRHPfPoTAHcN6kTftkn0bWtF4oy7/EkEhao6yvVIwtUxlUWFozNgbXaQCbDFG3dz17vLWL/zIKNOSbEicSZo/EkEizzTOt8GZqjqfpdjCg8+1xfwuoXBVhMzAXIgv5DH/72a1xdsolWj2rx+TSZnWpE4E0T+rFDWVkT64kz/fEBEvgOmqeo016MLlaquL2BMJfyce5hpi7K46rQ07jy/I3WtSJwJMnEm7vh5sLMuwV+By1Q13rWoypGRkaGLFy925829ewGLJ/s4IA4yrrL1BUyV7Tl4hA+Wb+OKU1sDsGNfnq0YZlwlIktUNcPXcxU2PUSkHk6xuFFAZ+A9oG9AIwwH/vYCbH0BUwWqyscrfua+91aw91ABfdsm0rZJPUsCJqT86YOuAN4HHlPV6L1l1lYZMy7bsS+PP763gk9WbuekVg15/Zo+ViTOhAV/EkEbVS12PZJQs1XGjIuKipWLX5zPz7l53DO4E7/tl041KxJnwkR5i9c/qaq3A++KyHEXEvxZoSyi2CpjxgVb9x6meQOnSNyDw7uR0rg2bawXYMJMeT2Ctz3/rdTKZBHNVhkzAVJUrLw+fyOP/XsN9wzpxJWnpdm6wSZslbdC2ULPw86qekwy8BSTq+oKZsZEpbU79nPX9GUs3byXszs2YUDnZqEOyZhy+TNIeY2Pfb8NdCDGRIO3vtnMkKe/ZMOug0y6tAf/GHMKrRrVDnVYxpSrvGsEl+JMGU0XkRleT9UH9vp+lTGxLS2pDud1bcb9w7qSVM+KxJnIUN41goVADs7KYs957d8PfOtmUMZEiryCIib970cEYfxgKxJnIlN51wg2ABtwqo0aY0r5Zn0O42csZ8Oug1zWJ9WKxJmIVd7Q0BeqepaI7OHYRecFZ02ZBNejMyYM7c8r4P/+vZp/LthMakId3rq2D33bWS/ARK7yhoZKlqOM7r9wW2vAVNL2fflMX5LNtf3Sue28DtSpYUXiTGSrsOiciKQBW1X1iIj0A7oD/1TVfe6Hd7yAFp0ra62BarVtrQFzjN0Hj/Dhsq1ccVoaADv359uKYSailFd0zp/po7NwlqlsC7yOU3jurQDGFzrH1BfysdaAiXmqyvvfb+Xcp77gwQ9WsX7nAQBLAiaq+NOnLVbVAhEZAfxVVZ8RkeiYNVRefSFbayDmbd+Xx8SZK/jfD9vpntyQN0f2sfIQJir5tVSliFwMXAH8yrOvunshBZHVFzJlKCpWLvEUiZs4pDNXn55mReJM1PInEVwD3IhThnq9iKQDU90NK4isvpDxkr3nEC0a1iY+TnhoeDdSE+qQllQ31GEZ46oKmziqugK4GVgsIp2ALFX9s+uRGRNERcXKK/PWM/CpL/jngk0AnNmhiSUBExP8WaHsDOANYAvO1JrmInKFqn7ldnDGBMOan/dz17vL+D5rLwM6NeW8rlYkzsQWf4aGJgFDVHUVgIh0xkkMPqchGRNJ/rlgEw+8v5L6tarz9KieDOvR0u4ONjHHn0RQoyQJAKjqDyJSw8WYjHFdSTmIdk3rMeSkFtw3tAuJViTOxCh/EsFSEXkRpxcAcBlWdM5EqMNHinjqv2uIixPuGdyZU9skcmqbxFCHZUxI+TMf7npgHXAXcDewHrjOzaCMccP8dTkMenouL8/bwKH8Iiq6q96YWFFuj0BETgLaAjNV9bHghBQEVl8opuzLK+AvH61m6sLNtE6sw1u/62Oloo3xUl710Qk4K5EtBU4RkQdVdXLQInNLWfWFpgyz+kJRase+fGZ9u4WxZ7bh1oEdqF0jPtQhGRNWyhsaugzorqoXA6cAN1T2zUVkkIisEZG1IjK+nONGioiKiHszkbIWwrwn4fupVl8oBuQcyOe1rzYA0K5pPb68uz8ThnS2JGCMD+UNDeWr6kEAVd0pIpW6v15E4nFWNjsXyAYWichs7xlInuPq49yw9k2lIq+MY3oBpcRV/6XWkNUXiniqyuzvt3L/7JUcyC/kzA5NaNOkns0IMqYc5SWCNl5rFQvQ1nvtYlUdUcF7ZwJrVXU9gIhMA4YDq0od9xDwGHBHZQKvlGOqjHqLg96XQ8MUqy8UBbbuPcy9s1bw2eod9ExpxGMju1uROGP8UF4iuKjU9rOVfO9WQJbXdjbQx/sAEekFpKjqByJSZiIQkbHAWIDU1NRKhkH5VUZ7jLYEEAUKi4oZ9dICdu7P549DuzCmbxrxcXZjmDH+KG/N4k+r+N6+/i88OijvGWqaBIyp6I1U9SXgJXAWpql0JFZlNGpl7T5Ey0a1qRYfxyO/PonUhDqkJtYJdVjGRBQ319jLBlK8tpOBrV7b9YFuwBzPLf3NgdkiMkxVA7QEmRerMhpVCouKmfzVBp78z4/cM7gTY05Pp197mxJqzIlwMxEsAtp7ylZvAUYBo0ueVNVcvNZDFpE5wB2uJAETVX7Yto+7313Gsuxczu3SjMEntQh1SMZENL8TgYjUVNV8f49X1UIRGQd8AsQDk1V1pYg8CCxW1dmVD9fEujfmb+SB91fRsHZ1nh3diwtOamFF4oypIn/KUGcCrwINgVQR6QFcq6q/r+i1qvoR8FGpffeVcezZ/gRsYlNJkbgOzepzYY+W/HFoFxLqWu1DYwLBnx7BM8BQnEXsUdXvRaS/q1EZ43HoSCFPfPIj1eKFCUM606dNIn2sSJwxAeXPTWJxqrqp1L4iN4IxxttXa3dx/l/nMvmrDRwpLLYicca4xJ8eQZZneEg9dwv/HvjR3bBMLMs9XMAjH/7A24uzSE+qyzvXnUZmekKowzImavmTCG7AGR5KBRJ9RBMAABI+SURBVLYD/+ME6g4Z469dB/J5f9lWrj+rLbcMbE+t6lYfyBg3VZgIVHUHztRPY1yzc38+73+/lWv6pdO2ST2+vPscuxhsTJD4M2voZY4p0+lQ1bGuRGRiiqoy67stPPD+Kg7lF9G/U1PSk+paEjAmiPwZGvqf1+NawK85toaQMSdky97DTJy5nDlrdtI71SkSl55UN9RhGRNz/Bkaett7W0TeAP7rWkQmJjhF4uaTc+AI91/YhStOsyJxxoTKiZSYSAdaBzoQExs25xyiVWOnSNyjI7qTmlCHlAQrEmdMKFV4H4GI7BGR3Z6fvTi9gQnuh2aiSWFRMc/PWcfASV/w+vyNAJzeLsmSgDFhoKLF6wXogVM0DqBY7a4eU0krt+Zy97vLWLFlH+d3bcYFViTOmLBSbiJQVRWRmap6crACMtFlytcbeeiDVTSqU4PnL+ttlUKNCUP+XCNYKCK9VXWp69GYqFFSJK5T8/oM79mKPw7tTKM6NiXUmHBUZiIQkWqqWgj0A34nIuuAgzgrj6mq9g5SjCaCHMwv5PFP1lA9Xph4QRcrEmdMBCivR7AQ6A38KkixmAg398ed3DNjOVtzD3PVaWlHewXGmPBWXiIQAFVdF6RYTITKPVTAQx+uYvqSbNo0cYrEnZJmReKMiRTlJYImInJbWU+q6lMuxGMi0K6D+Xy8fBs3nt2WmwdYkThjIk15iSAeqIenZ2CMtx3785j93VauPaPN0SJxja0+kDERqbxEsE1VHwxaJCYiqCrvLt3CQx+s4nBBEQM6NyM9qa4lAWMiWIXXCIwpkbX7EBNmLmfeT7vIaN2YRy+yInHGRIPyEsGAoEVhwl5hUTG/eXkBew4e4aHhXbmsT2virEicMVGhzESgqruDGYgJTxt3HSQloQ7V4uN4bKRTJC65sdUHMiaa+LN4vYlBBUXFPPf5Ws6bNPdokbi+bZMsCRgThU6kDLWJciu25HLX9GWs2raPC05qwdDuLUMdkjHGRZYIzDH+8dUGHv7wBxLq1uCFy09mULfmoQ7JGOMySwQG+KVIXNeWDRnRqxX3XtCFhnWqhzosY0wQWCKIcQfyC3ns36upER/HvUO7kJmeQGa6lYcwJpbYxeIYNmfNDs6fNJc3FmxCcXoFxpjYYz2CGLTn4BEe+nAVM5ZuoV3Teky/vi8nt24c6rCMMSFiiSAG7Tl0hP+s3M7N57TjpnPaUbOaFYkzJpa5OjQkIoNEZI2IrBWR8T6ev01EVonIMhH5VERauxlPLNuxL4+X5q5DVWnTpB5f3X0Ot53X0ZKAMca9RCAi8cBzwGCgC/AbEelS6rBvgQxV7Q5MBx5zK55Ypaq8syiLAU99wZP/+ZGNOYcAbEaQMeYoN4eGMoG1qroeQESmAcOBVSUHqOrnXscvAC53MZ6Yk7X7EPfMWM6Xa3eRmZ7AoyNOsiJxxpjjuJkIWgFZXtvZQJ9yjv8t8LGvJ0RkLDAWIDU1NVDxRbWSInF7DxXw8K+6MToz1YrEGWN8cjMR+PrW8Tk/UUQuBzKAs3w9r6ovAS8BZGRk2BzHcmzYdZBUT5G4x0f2oHViHVo2qh3qsIwxYczNi8XZQIrXdjKwtfRBIjIQmAgMU9V8F+OJagVFxfzt0584f9Jcpny9EYDT2iZaEjDGVMjNHsEioL2IpANbgFHAaO8DRKQX8CIwSFV3uBhLVFuWvZe7pi9j9c/7ubBHS4b1tCJxxhj/uZYIVLVQRMYBn+CsfzxZVVeKyIPAYlWdDTyOsy7yv0QEYLOqDnMrpmg0+csNPPzhKprUr8nLV2ZwbpdmoQ7JGBNhXL2hTFU/Aj4qte8+r8cD3fz8aFZSJK57ckMuPSWF8YM707C2TQk1xlSe3VkcYfbnFfDox6upWS2e+y7sQkZaAhlpViTOGHPirOhcBPl89Q7OmzSXqQs3Uy1erEicMSYgrEcQAXYfPMKD769k1ndb6dCsHn+/rC+9Uq1InDEmMCwRRIDcwwV8+sMO/jCgPTf1b0eNataRM8YEjiWCMPVzbh6zvtvCdWe2IT2pLl+OP8cuBhtjXGGJIMyoKtMWZfHIhz9QUFzMoK7NSUuqa0nAGOMaSwRhZFPOQca/u5z563M4tU0Cj47oTpoViTPGuMwSQZgoLCpm9MvfkHu4gEd+fRKjTkmxInHGmKCwRBBi63YeoLWnSNyTlzhF4lo0tPpAxpjgseknIXKksJi//u9HBv11Lq/P3wTAqW0SLQkYY4LOegQh8F3WXu6evow12/czvGdLftWrVahDMsbEMEsEQfbqlxv484eraFq/Fq9elcGAzlYkzhgTWpYIgqSkSFzPlIaMykxl/OBONKhlU0KNMaFnicBl+/IK+MtHq6lVPY4/XdiVk1sncHJrKxJnjAkfdrHYRf9btZ1zn/qCtxdtpka1OCsSZ4wJS9YjcEHOgXweeH8Vs7/fSqfm9Xnpigx6pDQKdVjGGOOTJQIX7M8r5PM1O7h1YAduOLutFYkzxoQ1SwQBsnXvYWZ+u4Ubz25LWlJdvhp/jl0MNsZEBEsEVVRcrLy1cDOPfryaomLlgpNakJZU15KAMSZiWCKogg27DjL+3WV8s2E3p7dL5C+/7k5qYp1Qh2WMMZViieAEFRYVc/kr37Avr4DHLurOxRnJiFiROGNM5LFEUElrd+wnLbEu1eLjmHRpT1on1qFZg1qhDssYY06YTWfxU35hEU/990cG/XUeUzxF4jLTEywJGGMinvUI/LB08x7unr6Mn3YcYESvVoywInHGmChiiaACL89dzyMf/0CLBrX4x9Wn0L9j01CHZIwxAWWJoAzFxUpcnNC7dSMu65PK3YM6Ud+mhBpjopAlglJyDxfw5w9XUbt6PA8M72ZF4owxUc8uFnv5ZOXPnPvUF7y7dAt1a1azInHGmJhgPQJg14F8/vTeSj5cvo0uLRowecwpdGvVMNRhGWNMUFgiAA7kFTLvp53ceX5Hxp7Zhurx1lEyxsSO2PnGy98HuVmQtRCALXsP8+xnP6GqpCXV5et7BnBT/3aWBIwxMcfVbz0RGSQia0RkrYiM9/F8TRF52/P8NyKS5kogWQth+wrYuwmdMoyPP36P8576guc+X8emnEMA1KtpnSNjTGxyLRGISDzwHDAY6AL8RkS6lDrst8AeVW0HTAL+z5VgNs4DLQaguDCP5V9+SO/WjfnPrWeSllTXlY80xphI4WaPIBNYq6rrVfUIMA0YXuqY4cAUz+PpwABxo3Jb7UQUUCAOpX/vTrx+TSYpCVYp1Bhj3EwErYAsr+1szz6fx6hqIZALJJZ+IxEZKyKLRWTxzp07Kx/J4RwEcDJMHKc0VasUaowxHm4mAl/ftKUn5vtzDKr6kqpmqGpGkyZNKh9J2hlQrTZIPFKtprNtjDEGcHf6aDaQ4rWdDGwt45hsEakGNAR2BzySlEy4arZzrSDtDGfbGGMM4G4iWAS0F5F0YAswChhd6pjZwFXAfGAk8Jm6dTtvSqYlAGOM8cG1RKCqhSIyDvgEiAcmq+pKEXkQWKyqs4FXgTdEZC1OT2CUW/EYY4zxzdXJ86r6EfBRqX33eT3OAy52MwZjjDHls9tojTEmxlkiMMaYGGeJwBhjYpwlAmOMiXESaYuviMhOYNMJvjwJ2BXAcCKBnXNssHOODVU559aq6vOO3IhLBFUhIotVNSPUcQSTnXNssHOODW6dsw0NGWNMjLNEYIwxMS7WEsFLoQ4gBOycY4Odc2xw5Zxj6hqBMcaY48Vaj8AYY0wplgiMMSbGRWUiEJFBIrJGRNaKyHgfz9cUkbc9z38jImnBjzKw/Djn20RklYgsE5FPRaR1KOIMpIrO2eu4kSKiIhLxUw39OWcRucTzb71SRN4KdoyB5sffdqqIfC4i33r+voeEIs5AEZHJIrJDRFaU8byIyDOe38cyEeld5Q9V1aj6wSl5vQ5oA9QAvge6lDrmRuAFz+NRwNuhjjsI59wfqON5fEMsnLPnuPrAXGABkBHquIPw79we+BZo7NluGuq4g3DOLwE3eB53ATaGOu4qnvOZQG9gRRnPDwE+xlnh8VTgm6p+ZjT2CDKBtaq6XlWPANOA4aWOGQ5M8TyeDgyQyF7EuMJzVtXPVfWQZ3MBzopxkcyff2eAh4DHgLxgBucSf875d8BzqroHQFV3BDnGQPPnnBVo4HnckONXQowoqjqX8ldqHA68ro4FQCMRaVGVz4zGRNAKyPLazvbs83mMqhYCuUBiUKJzhz/n7O23OC2KSFbhOYtILyBFVT8IZmAu8uffuQPQQUS+EpEFIjIoaNG5w59zvh+4XESycdY/+X1wQguZyv7/XiFXF6YJEV8t+9JzZP05JpL4fT4icjmQAZzlakTuK/ecRSQOmASMCVZAQeDPv3M1nOGhs3F6ffNEpJuq7nU5Nrf4c86/AV5T1SdF5DScVQ+7qWqx++GFRMC/v6KxR5ANpHhtJ3N8V/HoMSJSDac7WV5XLNz5c86IyEBgIjBMVfODFJtbKjrn+kA3YI6IbMQZS50d4ReM/f3bfk9VC1R1A7AGJzFEKn/O+bfAOwCqOh+ohVOcLVr59f97ZURjIlgEtBeRdBGpgXMxeHapY2YDV3kejwQ+U89VmAhV4Tl7hklexEkCkT5uDBWcs6rmqmqSqqapahrOdZFhqro4NOEGhD9/27NwJgYgIkk4Q0XrgxplYPlzzpuBAQAi0hknEewMapTBNRu40jN76FQgV1W3VeUNo25oSFULRWQc8AnOjIPJqrpSRB4EFqvqbOBVnO7jWpyewKjQRVx1fp7z40A94F+e6+KbVXVYyIKuIj/POar4ec6fAOeJyCqgCLhTVXNCF3XV+HnOtwMvi8itOEMkYyK5YSciU3GG9pI81z3+BFQHUNUXcK6DDAHWAoeAq6v8mRH8+zLGGBMA0Tg0ZIwxphIsERhjTIyzRGCMMTHOEoExxsQ4SwTGGBPjLBGYsCMiRSLynddPWjnHppVVpbGSnznHU+Hye095ho4n8B7Xi8iVnsdjRKSl13OviEiXAMe5SER6+vGaW0SkTlU/20QvSwQmHB1W1Z5ePxuD9LmXqWoPnIKEj1f2xar6gqq+7tkcA7T0eu5aVV0VkCh/ifPv+BfnLYAlAlMmSwQmInha/vNEZKnnp6+PY7qKyEJPL2KZiLT37L/ca/+LIhJfwcfNBdp5XjvAU+d+uadOfE3P/kfll/UdnvDsu19E7hCRkTj1nN70fGZtT0s+Q0RuEJHHvGIeIyJ/O8E45+NVbExEnheRxeKsQ/CAZ9/NOAnpcxH53LPvPBGZ7/k9/ktE6lXwOSbKWSIw4ai217DQTM++HcC5qtobuBR4xsfrrgeeVtWeOF/E2Z6SA5cCp3v2FwGXVfD5FwLLRaQW8BpwqaqehHMn/g0ikgD8Guiqqt2Bh71frKrTgcU4LfeeqnrY6+npwAiv7UuBt08wzkE4JSVKTFTVDKA7cJaIdFfVZ3Dq0PRX1f6eshP3AgM9v8vFwG0VfI6JclFXYsJEhcOeL0Nv1YFnPWPiRTg1dEqbD0wUkWRghqr+JCIDgJOBRZ7SGrVxkoovb4rIYWAjTinjjsAGVf3R8/wU4CbgWZz1DV4RkQ8Bv8tcq+pOEVnvqRHzk+czvvK8b2XirItTcsF7dapLRGQszv/XLXAWaVlW6rWnevZ/5fmcGji/NxPDLBGYSHErsB3ogdOTPW6hGVV9S0S+AS4APhGRa3FK9k5R1Xv8+IzLvIvSiYjPNSo89W8ycQqdjQLGAedU4lzeBi4BVgMzVVXF+Vb2O06clboeBZ4DRohIOnAHcIqq7hGR13CKr5UmwH9V9TeViNdEORsaMpGiIbDNU2P+CpzW8DFEpA2w3jMcMhtniORTYKSINPUckyD+r9e8GkgTkXae7SuALzxj6g1V9SOcC7G+Zu7sxymF7csM4Fc4dfTf9uyrVJyqWoAzxHOqZ1ipAXAQyBWRZsDgMmJZAJxeck4iUkdEfPWuTAyxRGAixd+Bq0RkAc6w0EEfx1wKrBCR74BOOMv5rcL5wvyPiCwD/oszbFIhVc3Dqez4LxFZDhQDL+B8qX7geb8vcHorpb0GvFBysbjU++4BVgGtVXWhZ1+l4/Rce3gSuENVv8dZq3glMBlnuKnES8DHIvK5qu7EmdE01fM5C3B+VyaGWfVRY4yJcdYjMMaYGGeJwBhjYpwlAmOMiXGWCIwxJsZZIjDGmBhnicAYY2KcJQJjjIlx/w+8v+lrUCBdhgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# generate a no skill prediction (majority class)\n",
    "ns_probs = [0 for _ in range(len(y_test))]\n",
    " \n",
    "ann_probs = ann_grid.predict_proba(X_test)\n",
    " \n",
    "ann_probs = ann_probs[:, 1]\n",
    " \n",
    "ns_auc = roc_auc_score(y_test, ns_probs)\n",
    "ann_auc = roc_auc_score(y_test, ann_probs)\n",
    " \n",
    "print('No Skill: ROC AUC=%.3f' % (ns_auc))\n",
    "print('ANN: ROC AUC=%.3f' % (ann_auc))\n",
    " \n",
    "ns_fpr, ns_tpr, _ = roc_curve(y_test, ns_probs)\n",
    "ann_fpr, ann_tpr, _ = roc_curve(y_test, ann_probs)\n",
    "\n",
    "\n",
    "pyplot.plot(ns_fpr, ns_tpr, linestyle='--', label='No Skill')\n",
    "pyplot.plot(ann_fpr, ann_tpr, marker='.', label='ANN')\n",
    "pyplot.xlabel('False Positive Rate')\n",
    "pyplot.ylabel('True Positive Rate')\n",
    "pyplot.legend()\n",
    "pyplot.show()\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applying ML Algorithm 3  : Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import GridSearchCV, cross_val_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = RandomForestClassifier()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/trinaghosh/opt/anaconda3/lib/python3.7/site-packages/sklearn/ensemble/forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "                       max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "                       min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                       min_samples_leaf=1, min_samples_split=2,\n",
       "                       min_weight_fraction_leaf=0.0, n_estimators=10,\n",
       "                       n_jobs=None, oob_score=False, random_state=None,\n",
       "                       verbose=0, warm_start=False)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 371,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_param_grid = {\n",
    "    'bootstrap': [True,False],\n",
    "    'max_depth': [10,15,20,25,50],\n",
    "    'max_features': [5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20],\n",
    "    'n_estimators': [100,125,150,175,200,225,250,275,300],\n",
    "    'criterion': ['gini','entropy']\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 372,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_grid_search = GridSearchCV(estimator = rf_model, param_grid = rf_param_grid, \n",
    "                          cv = 5, n_jobs = -1, verbose = 2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 373,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 5 folds for each of 2880 candidates, totalling 14400 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  25 tasks      | elapsed:    4.0s\n",
      "[Parallel(n_jobs=-1)]: Done 146 tasks      | elapsed:   11.2s\n",
      "[Parallel(n_jobs=-1)]: Done 349 tasks      | elapsed:   23.5s\n",
      "[Parallel(n_jobs=-1)]: Done 632 tasks      | elapsed:   43.1s\n",
      "[Parallel(n_jobs=-1)]: Done 997 tasks      | elapsed:  1.1min\n",
      "[Parallel(n_jobs=-1)]: Done 1442 tasks      | elapsed:  1.7min\n",
      "[Parallel(n_jobs=-1)]: Done 1969 tasks      | elapsed:  2.3min\n",
      "[Parallel(n_jobs=-1)]: Done 2576 tasks      | elapsed:  3.1min\n",
      "[Parallel(n_jobs=-1)]: Done 3265 tasks      | elapsed:  4.2min\n",
      "[Parallel(n_jobs=-1)]: Done 4034 tasks      | elapsed:  5.2min\n",
      "[Parallel(n_jobs=-1)]: Done 4885 tasks      | elapsed:  6.1min\n",
      "[Parallel(n_jobs=-1)]: Done 5816 tasks      | elapsed:  7.3min\n",
      "[Parallel(n_jobs=-1)]: Done 6829 tasks      | elapsed:  8.6min\n",
      "[Parallel(n_jobs=-1)]: Done 7922 tasks      | elapsed:  9.9min\n",
      "[Parallel(n_jobs=-1)]: Done 9097 tasks      | elapsed: 11.1min\n",
      "[Parallel(n_jobs=-1)]: Done 10352 tasks      | elapsed: 12.7min\n",
      "[Parallel(n_jobs=-1)]: Done 11689 tasks      | elapsed: 14.4min\n",
      "[Parallel(n_jobs=-1)]: Done 13106 tasks      | elapsed: 15.9min\n",
      "[Parallel(n_jobs=-1)]: Done 14400 out of 14400 | elapsed: 17.9min finished\n",
      "/Users/trinaghosh/opt/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_search.py:814: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise-deprecating',\n",
       "             estimator=RandomForestClassifier(bootstrap=True, class_weight=None,\n",
       "                                              criterion='gini', max_depth=None,\n",
       "                                              max_features='auto',\n",
       "                                              max_leaf_nodes=None,\n",
       "                                              min_impurity_decrease=0.0,\n",
       "                                              min_impurity_split=None,\n",
       "                                              min_samples_leaf=1,\n",
       "                                              min_samples_split=2,\n",
       "                                              min_weight_fraction_leaf=0.0,\n",
       "                                              n_estimators='warn', n_jobs=None,\n",
       "                                              oob_score=Fa...\n",
       "                                              random_state=None, verbose=0,\n",
       "                                              warm_start=False),\n",
       "             iid='warn', n_jobs=-1,\n",
       "             param_grid={'bootstrap': [True, False],\n",
       "                         'criterion': ['gini', 'entropy'],\n",
       "                         'max_depth': [10, 15, 20, 25, 50],\n",
       "                         'max_features': [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15,\n",
       "                                          16, 17, 18, 19, 20],\n",
       "                         'n_estimators': [100, 125, 150, 175, 200, 225, 250,\n",
       "                                          275, 300]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=2)"
      ]
     },
     "execution_count": 373,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_grid_search.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 374,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': False,\n",
       " 'criterion': 'entropy',\n",
       " 'max_depth': 20,\n",
       " 'max_features': 15,\n",
       " 'n_estimators': 150}"
      ]
     },
     "execution_count": 374,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_grid_search.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 375,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_result = rf_grid_search.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 376,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 78.79%\n"
     ]
    }
   ],
   "source": [
    "print(f\"Accuracy: {round(metrics.accuracy_score(y_test, rf_result)*100, 2)}%\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 312,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import classification_report\n",
    "from sklearn.datasets import make_classification"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 327,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0   0.768519  0.821782  0.794258       101\n",
      "           1   0.800000  0.742268  0.770053        97\n",
      "\n",
      "    accuracy                       0.782828       198\n",
      "   macro avg   0.784259  0.782025  0.782156       198\n",
      "weighted avg   0.783941  0.782828  0.782400       198\n",
      "\n"
     ]
    }
   ],
   "source": [
    "sk_report = classification_report(\n",
    "    digits=6,\n",
    "    y_true=y_test, \n",
    "    y_pred=rf_grid_search.predict(X_test))\n",
    "print(sk_report)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROC Curve"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 424,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No Skill: ROC AUC=0.500\n",
      "Random Forest: ROC AUC=0.876\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3dd3hUZfbA8e9JCIQaCAGkJCT0XkMQO4oKiqBYQFFh1WVV0F3FgmVdRN3151pWVteuKIsVUFBRXBEFpARQSgBBSiABpRMpCaSc3x93EoeQMoFMP5/nyZO5d+7MnEuZc9/7vu95RVUxxhgTviL8HYAxxhj/skRgjDFhzhKBMcaEOUsExhgT5iwRGGNMmKvi7wAqKi4uThMTE/0dhjHGBJXly5fvUdUGJT0XdIkgMTGRZcuW+TsMY4wJKiKytbTn7NaQMcaEOUsExhgT5iwRGGNMmLNEYIwxYc4SgTHGhDmvJQIReVNEdolIWinPi4hMFJGNIrJKRHp4KxZjjDGl82aLYBLQv4znBwCtXT+jgJe8GIsxxvhfRirMf8b57cvXlsNr8whUdZ6IJJZxyGDgHXXqYC8Wkboi0lhVf/FWTMYY4zdbF8PbA6EgDyIioctQqNPUo5fm7c8kcs2HiBZAZDUYMRPiUyotNH9OKGsKZLhtZ7r2nZAIRGQUTquBhIQEnwRnTEjISIX0+ZB4dqV+cXhVRWMOlnNcNxMKcp3HBXmwYgog5b5MgQjc1o3JP+acb4gkgpL+BEpcJUdVXwVeBUhOTraVdIzxREaqcwWad8y5Au1xI8TE+zuqsmVlwA/vQEG+ZzFX9Hh/2rnm+O2Bz0PyyFIPz8rO5R+z1vH+0gwuqbuNf+eOJ7IgFyKrOkmvEvkzEWQC7n9jzYAdforFmNCTPt9JAqhzBbrsTX9HVDEVjTmozjECsveW+mx+gXLlSwvZvPsQfzq3BXf160/krz291vLxZyKYCYwRkfeB3kCW9Q8YU4kSzwaJAM2HKtEwfCo06+XvqMqWuRSmXAX5uRAZVX7MFT3en46LteSr+v2Hj1G3RhSREcI9F7WlSd1oujSr6zwZn+K1W1/irTWLReQ94DwgDtgJ/A2IAlDVl0VEgBdwRhYdAf6gquVWk0tOTlYrOmdMMaXdJ3/jYtjxAwz4Z5m3IQJKqPYRQKmxqiqfrNjOo5+u5f7+7bg2pfL7QkVkuaoml/hcsC1eb4nAmGIyUuHtyyDvKERUgT6jITYJ9m2BhRNBC6BK9UofaWIqx44D2Tz08Wrmrt9N94S6PHVlF1o3ql3pn1NWIgi6MtTGhIWKXOWmz4e8HOdxQS58/68Tj/HCSBNz6mas2M5DH6eRX6A8MrADI85IJDKi/JFElc0SgTGBxv0KP7IKnDUW6rcs/fi9W47f7jfeGaO+40eYelOZ96SNf8VUj6JbfF3+MaQz8bE1/BaHJQJjfOFkr/Dzc+G7JyvwQRHOraA6TZyfEZ8Gz/3zMJCXX8AbC7aQm1/AmPNbc17bhpzbpgFOl6n/WCIwxtuKrvCPOVf45z0Aca1LPz4r8/jtvg9BpytLP/6XlfDJrZCfd+KVvxdHmpiKWbvjN+6ftorV27O4tEtjVBUR8XsSAEsEJpQE6uiR467wj8GcRyvw4ghnolRZt4bqt4SYZoF57oajefm88M1GXvp2E3VrRPGf4T0Y0Om0gEgAhSwRmNBQNIs217nq7jceGrT1d1SOQ7uO3z7nPugwqPTjf02DT//sTJDy9N6+XfkHrPQ9R3j5u00M6taEv17agXo1q/o7pBNYIjChIX2+07kKzlX37Af9G0+pIiAqGk7rXPohp3V2rvLtCj9oHT6ax//W7uTy7k1pe1pt5tx9Hgn1/dcZXB5LBCY0ND/r98cRUXDJ09Cog//icbdzLXxxn13hh4n5P+/mgemr2X4gm05N69CqYe2ATgJgicAEu93rIW06rHj3930iThIIlC/S+BQnHrvCD2lZR3J5YtZaPlyWSYu4mnwwqg+tGlb+xDBvsERggsP6L2Hj/6BxN6fS5NpPYH867N8CCNRNcH6jzvOBNnnKrvBDWn6BcuXLC9my5zC3n9eSOy9oTXRUpL/D8pglAhP4MlLhvaEn7pcIOH00nHGHU4747UFO/4BNnjI+su/wMepWd4rE3XtxW5rWrU6npjH+DqvCLBGYwJc+323DddVf+LhmfajT2PkZMdNuvxifUFWm/7CdCZ85ReKu653AxR1P83dYJ80SgQl8RVf34pQaRkrueLXbL8YHMvcf4cGP05i3YTc9m9cjJSnW3yGdMksExv82zYVN30CzZGjc9cTnazV0qmrWbgJnj7WOV+M3H/+YycMfp6HAo4M6csPpzYnwQ5G4ymaJwPhXRipMvtyzY7My4Mtxzi2gs8d6Ny5jShBbsxo9E2P5+xWdaFYvsIeEVoQlAuNfx93/j4AOl0Gb/scfs+FLWDsTUCunbHwqN7+A1+ZvJi9fufOC1pzbpgHntI4LqPIQlcESgfEv9/v/VapBnzEnfsnXbwUbvrIRQcan0rZncf+0VazZ8RuXdW0SUEXiKpslAuNfhV/6SWfD+X8t+Uo/PsVGBBmfycnNZ+Kcn3ll3mbq1ajKy9f3oH+nxv4Oy6ssEZjAkNCn7C94GxFkfGTr3iO8Nn8zQ7o35eFLOxBTI8rfIXmdJQLjO1sXu67qz4JmvY5/btsip+PYvuyNHxw+msfsNb8ypEcz2p5Wm2/GnufXFcN8zRKB8Y2ti+Gt/vw+GayYLfOdmcG2wLrxse827ObB6avZkZVNl2YxtGpYO6ySAECEvwMwISgjFeY/4/wulD6P42YEt+gLfR92fhfOFi4cEWSMD+w/fIy7P1zBiDdTiY6K4KM/BU+RuMpmLQJTubYugrcGUPSlL5FOTaCCfLeDFDpcDskjIeNc2LbYRgQZnyosErd17xHG9G3FmPNbBVWRuMpmicBUrs3fctyVf2EncEYqbF3g2h8B2XudhzYiyPjQ3kNHqVejKpERwrj+7WharzodmwRfkbjKZonAVK6EPq4HAlWiod/ffk8EpVUHtRFBxstUlY+WZ/L4Z2u5f0A7hvduzkVBXCSuslkiMJWjcOH4Jj2d7diWTnnowi94u/I3fpKx7wgPfrya+T/vISUxlj4t6vs7pIBjicCcuq2LYNIAULcRQfs2OXWB3FcKsyt/42PTf8jk4U/SEOCxyzsxPCUhJIrEVTZLBOZ3hVf1Fb1i3zTn+CQAWF0gEwjialUjJSmWJ67oTNO61f0dTsCyRGAcm+fB5EGuL3SB2o2ce/yeyM46fjsiCrTARgEZn8vNL+CV7zaRXwB/7teac9o04Jw2DfwdVsCzRBDuClsBO9e6XdUr1GwADTt49h671sKv+10bEdDjeoiJt74A41Np27O4d+oq1v3yG4O7/V4kzpTPEkE42zwPJg92rt6LS77FGefvieIjgrpeZwnA+ExObj7/+vpnXpu/mdiaVXnlhp5BvWykP3g1EYhIf+B5IBJ4XVWfLPZ8AvA2UNd1zDhVneXNmIybn2eXnATcx/l7wkYEGT/atu8IbyzYzFU9mvHgJe3DokhcZfNaIhCRSOBF4EIgE1gqIjNVda3bYQ8DH6rqSyLSAZgFJHorJlNMU9dQTyIgsgqlrgXsCRsRZHzoYE4uX6b9ytXJ8bRpVJu595wXUiuG+Zo3WwQpwEZV3QwgIu8DgwH3RKBAHdfjGGCHF+MJbyWNCCpcH7hhO0j5k60FbILC3J928dDHq/n1txy6J9SlVcPalgROkTcTQVMgw207E+hd7JjxwFcicgdQE+hX0huJyChgFEBCQkKlBxryNsyG94Y5t4EkAhp1gmp14PAu5/ld62wtYBPw9h0+xmOfreXjH7fTumEtpt52RtgWiats3qw+WlJ3ffHB5tcCk1S1GXAJMFlETohJVV9V1WRVTW7QwIaCVUhBgfMlX9gXoAWQ4xrumXfUdZBV/jSBLb9AueqlhXy6cgd3XtCaz+48ix4J9fwdVsjwZosgE4h3227Gibd+bgb6A6jqIhGJBuKAXV6MK7x8/y/Yt/n4sf1Xvl5+/R9jAsDug0epX9MpEvfgJe1pWq867RvXKf+FpkK8mQiWAq1FJAnYDgwDrit2zDbgAmCSiLQHooHdXowpvGxbAt88Dh2vgN63OdU/3e//22gfE6BUlQ+XZfD45+u4v387rj+9Of06NPJ3WCHLa4lAVfNEZAwwG2do6JuqukZEJgDLVHUmMBZ4TUTuwrltNFL1hFoF5mQc2QfTboa68XDZ8xAdAwnFu2iw0T4m4Gzbe4Rx01excNNeeifFclarOH+HFPK8Oo/ANSdgVrF9j7g9Xguc6c0YwpIqzLwDDv4KN892koAxQWDq8kz++kkakRHCE1d04tpeViTOF2xmcShKfRV++gwu/rvbXAFjAl+jOtU4o2V9Hr+iE41jrEicr1giCFalVQr98b/w5QMQ3xtOv91/8RnjgWN5Bbz07SYKVLnrwjac3boBZ7e2kYG+ZokgGG2ZD5Mvd2YBSyS0uRhqNYRDu2D9F4DCLyshc6nd/zcBa2XGAe6buor1Ow8ypHtTKxLnR5YIgklhK2D7D04SANB82DIPqtaEY4cpmqqRn2trAZiAlH0sn2f/t543FmyhYe1oXr8x2UYE+ZklgmBR1ArIP/G5i55wKoXavAATBDL2H+HthVsZlpLAuAHtqBNtReL8zRJBIHPvB0ib+nsr4DhulUJtXoAJUL+5isRd4yoS9+2959HEVgwLGJYIAlVGKrx9mVMGIiLSWQzeXWmrgNm8ABNgvvlpJw9OT2PXwRx6JNSjVcNalgQCjCWCQJU+H/KOAeq0BPZscHvSVgEzgW/voaNM+GwtM1bsoG2j2rx8Q09aNazl77BMCSwRBKrEs501Agrv9w/4p1M8zlYBM0Egv0C5+uVFZOw/wl392nDbeS2pWsWbNS7NqfAoEYhIVSBBVTd6OZ7wVNKcgPgU6PVHWPwipIxyOoNtvQAT4HYdzCGuZjUiI4SHLm1Ps3o1aHualYoOdOUmAhG5FHgWqAokiUg34G+qeoW3gwsLx/UFVIHkm6BuAhzYBktfd45JfRU6DLb7/yZgFRQo7y3dxj9m/cT9A9pxw+nNuaC9DQkNFp60CCbgLCgzF0BVV4hIK69GFU6O6wvIhdRXTjymIN/mBJiAlb7nMOOmr2Lx5n2c0bI+59rM4KDjSSLIVdUDxWb8WYXQylK8L+C6D536QNuXO6uK5efanAATsD5clsFfP0mjamQETw7pzNBe8TY7OAh5kgjWicg1QIRrbYE/A4u9G1YYcO8X6HkTpL4Mp4+Gln2d51v2hRGfWp+ACWhN61bnnDYNeGxwJ06LifZ3OOYkSXnl/0WkJvAIcJFr12zgUVXN9nJsJUpOTtZly5b546Mrj3u/QOHKnJoPkdVg5Gf2pW8C1tG8fP4zdxOqyt0XtfV3OKYCRGS5qiaX9Jwn47kuVtX7VbW762ccMKByQwwz7v0Cmu/8gDNfwNYNNgHqx237uezfC3h+zs9sP5CDrSEVOjy5NfQwML3YvodK2Gc85d4vEFHFaRUU5FtfgAlIR47l8cxXG3jz+y2cVieaN0cmc347GxEUSkpNBCJyMc7C8k1F5Fm3p+oABd4OLKTFp8D5j8D/HoaBz0GDdtYXYALW9v3ZTF68leG9E7i/fztqW5G4kFNWi2AXkAbkAGvc9h8ExnkzqLDQoI3zu1FHZ5SQJQATQLKyc/li9S8MS0mgdaPafHfvebZiWAgrNRGo6o/AjyIyRVVzfBiTMcaPvlrzKw9/ksbew8dIToylVcNalgRCnCd9BE1F5AmgA1A0PkxV23gtKmOMz+05dJTxM9fw2apfaHdabV4fkWxF4sKEJ4lgEvA48DTOaKE/YH0ExoSU/ALlqpcWsuNADvdc1IY/nduSqEgrEhcuPEkENVR1tog8raqbgIdFxMY4emrbEkibBg07OkXjCm382vm9c43TR2CMH+z8LYcGtZwicX+7rCPN6lWndSMrEhduPEkER8WZM75JRG4FtgMNvRtWiMhIhbcHOsNESzPrHmfUkHUWGx8qKFCmpG7j/774ifv7t+WGPon0bWf/rcOVJ4ngLqAWcCfwBBAD3OTNoILetiWwdgb8tt2pFQSAOGsIdBoCadNh5XtAAeTnWUE541Obdx9i3PTVpG7Zx1mt4jivrSWAcFduIlDVJa6HB4EbAESkmTeDCmqltgLU+bJv3Q+i68Ca6bbIvPG5D5Zu45EZa6hWJYKnrurC1T2bWZE4U3YiEJFeQFNggaruEZGOwP3A+YAlg5Kkz3drBbizReaN/zWrV4Pz2jpF4hrWsSJxxlHWzOJ/AFcCK3E6iD/GqTz6f8CtvgkvCCWeDZFRrqv9KCDCqSFki8wbPzial8+/5zgLC95zcVvObBXHma3i/ByVCTRltQgGA11VNVtEYoEdru31vgktyGSkwobZ0LSHs5rY6o/grHug1fl25W/8YvnWfdw3dRWbdh/mmuRmqKrdBjIlKisR5BSWmlbVfSLykyWBUmSkwqRLT+wX+P5ZJxGcPdY/cZmwdPhoHv+cvZ63F6XTJKY6b9+UwrltbNUwU7qyEkELESmsMCpAots2qjqkvDcXkf7A80Ak8LqqPlnCMdcA43FWPVupqtd5Hn6AKK1fwEYEGT/YcSCbd1O3cePpzbm3fztqVfNkcKAJZ2X9C7my2PYLFXljEYkEXgQuBDKBpSIyU1XXuh3TGngAOFNV94tIcI5jq16f41bvjIgCLbARQcZnso7k8vnqX7iut1Mkbv59fWlkncHGQ2UVnZtziu+dAmxU1c0AIvI+Tr/DWrdj/gi8qKr7XZ+56xQ/0z8KRwMBEAE9roeYeOsXMD7xZdqv/HVGGvsOH6N3i1haNqhlScBUiDfbjE2BDLftTKB3sWPaAIjI9zi3j8ar6pfF30hERgGjABISErwS7CkpGimUC1WinIljlgCMl+06mMP4mWuYtfpXOjSuw1sje9GygRWJMxXnzapSJQ1PKL62XRWgNXAecC3wuojUPeFFqq+qarKqJjdoEICdXvEpcPY9zuPBL1kSMF6XX6Bc8/Iivl63i3svbsuMMWfSqWmMv8MyQcrjFoGIVFPVoxV470wg3m27Gc4Q1OLHLFbVXGCLiKzHSQxLK/A5gaF+K+d34y7+jcOEtF+ysmlUO9opEjeoI/H1alipaHPKym0RiEiKiKwGfnZtdxWRf3vw3kuB1iKSJCJVgWHAzGLHfAL0db1vHM6tos0ViN+YsFBQoEz6fgsXPPMd/12yFYC+bRtaEjCVwpMWwURgIM6XNqq6UkT6lvciVc0TkTHAbJz7/2+q6hoRmQAsU9WZrucuEpG1QD5wr6ruLf1djQk/G3cdYty0VSzbup9z2jTgfKsSaiqZJ4kgQlW3FpuRmO/Jm6vqLGBWsX2PuD1W4G7XT3Db60zj55dVENfav7GYkPF+6jYembmG6lGRPHN1V4b0aGqzg02l8yQRZIhICqCuuQF3ABu8G1aQyUiF+U87j2fcBnXjrcPYVIqE+jXo174hjw7qRIPa1fwdjglRniSC23BuDyUAO4GvXftMofT5zixisNnE5pTk5OYzcc7PANzXvx1ntIzjjJZWJM54lyeJIE9Vh3k9kmCWeDZEVnHmEURWsdnE5qQsS9/HfdNWsXn3YYb1irciccZnPJlHsFREZonICBGxxUxLYvMIzCk4dDSPv81I4+pXFnEsr4B3bkrhySu7WBIwPuPJCmUtReQMnOGfj4rICuB9VX3f69EFE5tHYE7Sr1nZvL80gxF9Ern34rbUtCJxxsc8mlmsqgtV9U6gB/AbMMWrURkT4vYfPsbkxc58gFYNnSJx4wd1tCRg/KLcf3UiUgunWNwwoD0wAzjDy3EZE5JUlS/SfuWRGWkcOJLLGS3r07JBLVs20viVJ5cfacCnwFOqOt/L8RgTsnb9lsNfZ6Qxe81OOjeN4Z2beluROBMQPEkELVS1wOuRGBPC8guUq19ZxK9ZOTwwoB03n5VElUhv1nw0xnNlLV7/jKqOBaaJSPGqoR6tUBbyMlJ/X4/YZhabEuw4kM1pdZwicRMGdyK+XnVaWCvABJiyWgQfuH5XaGWysJGRCm8PhLyjHFdx22YWG5wWwDuL0nnqy/U8cEk7buyTaOsGm4BV1gplqa6H7VX1uGTgKiZ3qiuYBQ/3K//CL/j0+ZBXuFi9W4PJZhaHvY27DnLf1FX8sO0A57VtwAXtG/k7JGPK5EkfwU2c2Cq4uYR9ocn9yl8iIP50qF4PDmzD1ik2xb27ZBvjZ66hZrVInhvalcu7WZE4E/jK6iMYijNkNElEprs9VRs44O3AAsbGOa7bPzhf9Ps2Qc2GcNh9eWVbp9g4EuNqcFHHRowf1JG4WlYkzgSHsloEqcBenJXFXnTbfxD40ZtBBQxVp0UATmsgshoM/a/zRZ+RCm8PgvxjTivA1ikOSzm5+Tz39QYEYdwAKxJnglNZfQRbgC041UbDT0YqLPw3bP4GeoyAes2Pv9qPT4ERM0/sOzBhY8nmvYybvpotew4zvHeCFYkzQausW0Pfqeq5IrKf4xedF5w1ZWK9Hp2/FO8X6HotNO9z4nHxKZYAwtDBnFz+78uf+O/ibSTE1uDdW3pzRitrBZjgVdatocLlKMPvX/hxI4IEti0sORGYsLTzt6NMXZ7JLWclcfdFbahR1eoDmeBW6tRGt9nE8UCkquYDfYA/ATV9EJv/JJ4NEa7/3JFRNgrIsO/wMSYvSgegVcNazL/vfB4e2MGSgAkJnsxx/wRnmcqWwDs4hefe9WpU/hafAn1udx5fPclu/4QxVeXTlTu48NnvmPDZWjbvPgRgy0aakOLJ5UyBquaKyBDgX6o6UURCf9RQvUTnd5Pufg3D+M/O33J46OM0vl63ky7NYphyVW8rD2FCkkdLVYrI1cANwOWufVHeC8mP3GcQ70939u34EdoO8GtYxvfyC5RrXEXiHrqkPX84M9GKxJmQ5enM4ttxylBvFpEk4D3vhuUHGanw9mW/jxQqHCj10UgY8andHgoTmfuP0DimOpERwmODO5EQW4PEuNDuEjOm3EscVU0D7gSWiUg7IENVn/B6ZL6WPt81g1hB851ZxOAsSJ9uyzCEuvwC5fX5m+n37Hf817Vy2DltGlgSMGHBkxXKzgYmA9tx5hCcJiI3qOr33g7OpwpHChXkOjOFESjIs9pBYWD9rwe5b9oqVmYc4IJ2DbmooxWJM+HFk1tDzwGXqOpaABFpj5MYkr0ZmM+49wt0GARp0+D8RyCht80aDgP/XbyVRz9dQ+3oKJ4f1o1BXZvY7GATdjxJBFULkwCAqq4TkapejMl3SusXmPu40y9w9li/hme8p7AcRKuGtbikc2MeGdiB+lYkzoQpTxLBDyLyCk4rAGA4oVJ0rni/QKHCfgFrCYSc7GP5PPu/9URECA8MaM/pLepzeov6/g7LGL/yZDzcrcAm4D7gfmAzzuzi4Fd8BnFkVZBI6xcIUYs27aX/8/N4bf4WjhzNR/WEFViNCUtltghEpDPQEvhYVZ/yTUg+FJ8Cvf4IS/4Dw96H6DrWLxCCfsvJ5R+zfuK91G00r1+Dd//Y20pFG+OmrOqjD+KsRPYD0EtEJqjqmz6LzFfqNXd+N+0BNWItAYSgXb8d5ZMftzPqnBbc1a8N1atG+jskYwJKWbeGhgNdVPVqoBdwW0XfXET6i8h6EdkoIuPKOO4qEVERCY2RSMbv9h46yqTvtwBOkbgF9/flwUvaWxIwpgRl3Ro6qqqHAVR1t4hUaH69iETirGx2IZAJLBWRme4jkFzH1caZsLakQpEbUwJVZebKHYyfuYZDR/M4p00DWjSoZSOCjClDWYmghdtaxQK0dF+7WFWHlPPeKcBGVd0MICLvA4OBtcWOewx4CrinIoEbU9yOA9k8/Eka3/y0i27xdXnqqi5WJM4YD5SVCK4stv1CBd+7KZDhtp0J9HY/QES6A/Gq+pmIlJoIRGQUMAogISGhgmGYcJCXX8CwVxez++BR/jqwAyPPSCQywiaGGeOJstYsnnOK713S/8Ki8XquW03PASPLeyNVfRV4FSA5OdnG/JkiGfuO0KRudapERvD3KzqTEFuDhPo1/B2WMUHFm3V1M3FWNyvUDNjhtl0b6AR8KyLpwOnATOswNp7Iyy/g1Xmb6Pfsd0Urh53VOs6SgDEnwZvr7C0FWrvKVm8HhgHXFT6pqlm4rYcsIt8C96jqMi/GZELAul9+4/5pq1iVmcWFHRoxoHNjf4dkTFDzOBGISDVVPerp8aqaJyJjgNlAJPCmqq4RkQnAMlWdWfFwK8nWxbBmOjTuAtuXOvu2/wCt+/ktJOOZyYvSefTTtcRUj+KF67pzaefGViTOmFMk5U2zF5EU4A0gRlUTRKQrcIuq3uGLAItLTk7WZctOodGQkQpvXeKUm3ZXJdoWoAlghUXilmzey/tLM/jrwA7E1gyN2ofG+IKILFfVEm+9e9IimAgMxFnEHlVdKSJ9KzE+30qf76wzADj92a5EaIXmAtKRY3k8PXsDVSKFBy9pT+8W9eltReKMqVSedBZHqOrWYvvySzwyGJxQaK6aFZoLUN9v3MPF/5rHm99v4VhegRWJM8ZLPGkRZLhuD6lrtvAdwAbvhuVF8SnQ62ZY8jJc9wFUrWWF5gJMVnYuf/98HR8syyApriYf/qkPKUmx/g7LmJDlSSK4Def2UAKwE/iak6g7FFDquialNe0J0TGWAALMnkNH+XTVDm49tyV/6dea6CirD2SMN5WbCFR1F87QT2O8ZvfBo3y6cgc3nZVEywa1WHD/+dYZbIyPeLJ4/Wu4zQgupKqjvBKRCSuqyicrtvPop2s5cjSfvu0akhRX05KAMT7kya2hr90eRwNXcHwNIWNOyvYD2Tz08Wq+Xb+bHglOkbikuJr+DsuYsOPJraEP3LdFZDLwP69FZMKCUyRuEXsPHWP8ZR24oY8ViTPGX06mxEQS0LyyAzHhYdveIzSt5xSJe3JIFxJiaxAfa/WBjPGncucRiMh+Ednn+jmA0xp40PuhmVCSl1/AS99uot9z3/HOomjMxWYAABn6SURBVHQAzmwVZ0nAmABQ3uL1AnTFKRoHUKDBOqsnI/X3+QIHtjn7ti+Hluf7N64wsGZHFvdPW0Xa9t+4uGMjLrUiccYElDITgaqqiHysqj19FZBXZKTCpEsh/9jx+9+71uoLednbC9N57LO11K1RlZeG97BKocYEIE9KTKSKSA+vR+JN6fOdWkLFFdYXMpWusOHY7rTaDO7WlK/vPseSgDEBqtQWgYhUUdU84CzgjyKyCTiMq1KbqgZPckg8GyIinWJzkVUB+f2x1ReqVIeP5vHP2euJihQeurSDFYkzJgiUdWsoFegBXO6jWLwnPgV6joSlr8PwqRBV3eoLecG8Dbt5YPpqdmRlM6JPYlHpaGNMYCsrEQiAqm7yUSzeVVhfqFkyVK1pCaASZR3J5bHP1zJ1eSYtGjhF4nolWpE4Y4JFWYmggYjcXdqTqvqsF+LxnsKRQpnLoMW5/o0lxOw5fJQvVv/C7ee15M4LrEicMcGmrM7iSKAWziLzJf0Ej4xUWD7JefzuNc62OSW7Dubw+vzNAEVF4u7r386SgDFBqKwWwS+qOsFnkXhT+nwocK2lYyuRnRJVZdoP23nss7Vk5+ZzQftGJMXVpJ4ViTMmaJXbRxASjhs1FGUjhU5Sxr4jPPjxaub/vIfk5vV48korEmdMKCgrEVzgsyi8zX3U0HUfWmvgJOTlF3Dta4vZf/gYjw3uyPDezYmwInHGhIRSE4Gq7vNlIF7nPmrIeCx9z2HiY2tQJTKCp65yisQ1q2f1gYwJJZ7MLA4N7qOGTLly8wt4ce5GLnpuXlGRuDNaxlkSMCYEhUcisFFDFZK2PYvBL3zPP2ev58IOjRjYpYm/QzLGeNHJrEcQfGzUkMfe+n4Lj3++jtiaVXn5+p7073Sav0MyxnhZeCQCGzVUrsJyEB2bxDCke1MevrQDMTWi/B2WMcYHwiMR2KihUh06msdTX/5E1cgIHh7YgZSkWFKSrDyEMeEkPPoIwEYNleDb9bu4+Ll5TF68FeX30tHGmPASHi0Cc5z9h4/x2Odrmf7Ddlo1rMXUW8+gZ/N6/g7LGOMnlgjC0P4jx/hqzU7uPL8Vo89vRbUqVh/ImHDm1VtDItJfRNaLyEYRGVfC83eLyFoRWSUic0SkuTfjCWe7fsvh1XmbUFVaNKjF9/efz90XtbUkYIzxXiIQkUjgRWAA0AG4VkQ6FDvsRyBZVbsAU4GnvBVPuFJVPlyawQXPfsczX20gfe8RABsRZIwp4s1bQynARlXdDCAi7wODgbWFB6jqXLfjFwPXezGesJOx7wgPTF/Ngo17SEmK5ckhna1InDHmBN5MBE2BDLftTKB3GcffDHxR0hMiMgoYBZCQkFBZ8YW0wiJxB47k8vjlnbguJcGKxBljSuTNRFDSt06J4xNF5HogGShx6TBVfRV4FSA5OfnkxjiGyQplW/YcJsFVJO6fV3Wlef0aNKlb3d9hGWMCmDc7izOBeLftZsCO4geJSD/gIWCQqh71SiRhUGsoN7+Af8/5mYufm8fbC9MB6NOyviUBY0y5vNkiWAq0FpEkYDswDLjO/QAR6Q68AvRX1V1eiyTEaw2tyjzAfVNX8dOvB7msaxMGdbMiccYYz3ktEahqnoiMAWbjrH/8pqquEZEJwDJVnQn8E2dd5I9EBGCbqg6q9GBCuNbQmwu28Pjna2lQuxqv3ZjMhR0a+TskY0yQ8eqEMlWdBcwqtu8Rt8f9vPn5RUKw1lBhkbguzWIY2iuecQPaE1PdhoQaYyoufGYWh0itoYM5uTz5xU9UqxLJI5d1IDkxluREKxJnjDl54VN0LgRWKJv70y4uem4e76Vuo0qkWJE4Y0ylCI8WQfFRQyM+DarbQ/sOH2PCp2v4ZMUO2jSqxX+Gn0H3BCsSZ4ypHOGRCIJ81FBWdi5z1u3izxe0ZnTfVlStEj4NOWOM94VHIgjCUUO/ZuXwyYrt/OmcFiTF1WTBuPOtM9gY4xXhcWlZOGoIAn7UkKryXuo2Lnz2O/719Qa2FhaJsyRgjPGS8GgRQFCMGtq69zDjpq1m0ea9nN4ilieHdCHRisQZP8jNzSUzM5OcnBx/h2IqKDo6mmbNmhEV5fnFY/gkggCXl1/Ada8tISs7l79f0ZlhveKtSJzxm8zMTGrXrk1iYiKuyZ4mCKgqe/fuJTMzk6SkJI9fZ4nAzzbtPkRzV5G4Z65xisQ1jrH6QMa/cnJyLAkEIRGhfv367N69u0KvC48+ggB0LK+Af329gf7/msc7i7YCcHqL+pYETMCwJBCcTubvzVoEfrAi4wD3T13F+p0HGdytCZd3b+rvkIwxYcxaBD72xoItDPnP92Rl5/LGiGSeH9ad2JpV/R2WMQFHRBg7dmzR9tNPP8348eM9fv3OnTsZOHAgXbt2pUOHDlxyySUAfPvttwwcOPCE42fOnMmTTz4JwPjx43n66acBGDlyJFOnTj2FMwl81iLwkcIicd3iYxiWksC4Ae2oE21DQo0pTbVq1Zg+fToPPPAAcXFxFX79I488woUXXsif//xnAFatWlXm8YMGDWLQoMovfhwMLBF42W85ufxj1k9ER0Xwt8s60rN5LD2bW5E4E1yGvrLohH0DuzTmhj6JZB/LZ+RbJy72dFXPZlydHM++w8e47b/Lj3vugz/1Kfczq1SpwqhRo3juued44oknjntu69at3HTTTezevZsGDRrw1ltvnbCM7S+//MJFF11UtN2lS5cTPmPp0qWMGjWKadOmMW/ePJYtW8YLL7xQbmyhxm4NedHXa3dy4bPf8cHSbVStEmFF4oypoNGjRzNlyhSysrKO2z9mzBhuvPFGVq1axfDhw7nzzjtLfO3NN99M3759eeKJJ9ix4/gFEhcuXMitt97KjBkzaNGihVfPI9BZi8AL9h46yqOfrmXmyh20O602r96QTNf4uv4Oy5iTVtYVfPWqkWU+H1uzqkctgJLUqVOHG2+8kYkTJ1K9+u8j6hYtWsT06dMBuOGGG7jvvvtOeO3FF1/M5s2b+fLLL/niiy/o3r07aWlpAKxbt45Ro0bx1Vdf0aSJrehnLQIvOJiTx9z1u7irXxtmjjnLkoAxp+Avf/kLb7zxBocPHy71mNKGTMbGxnLdddcxefJkevXqxbx58wBo3Lgx0dHR/Pjjj16JOdhYIqgkOw5k8+LcjagqiXE1+X7c+fy5X2urFGrMKYqNjeWaa67hjTfeKNp3xhln8P777wMwZcoUzjrrrBNe980333DkiFOr6+DBg2zatKmoH6Fu3bp8/vnnPPjgg3z77bfeP4kAZ99Sp6igQPnv4q1c9Nw8XvhmY1GROBsRZEzlGTt2LHv27CnanjhxIm+99RZdunRh8uTJPP/88ye8Zvny5SQnJ9OlSxf69OnDLbfcQq9evYqeb9SoEZ9++imjR49myZIlPjmPQCXB1oGZnJysy5adxCpj3z8P/3sEHtwBVSunkNuWPYcZN20VS7bs48xW9fnHFV1IqF+jUt7bGH9at24d7du393cY5iSV9PcnIstVtcSqm9ZZfJLy8gu4/vUl/JaTy1NXduHq5GY2Jd8YE5QsEVTQxl0HSaxfkyqRETw3tBvN69egUZ1of4dljDEnzfoIPHQ0L59n/7eB/v+az9uuInEpSbGWBIwxQc9aBB74Ydt+7p+6ip93HWJI96YMsSJxxpgQYomgHK/N28zfv1hH4zrRvPWHXvRt29DfIRljTKWyRFCKggIlIkLo0bwuw3sncH//dtS2IaHGmBBkfQTFZGXnct/UlTz66RoAejaP5fHLO1sSMMbHIiMj6datG506deKyyy7jwIEDlfK+6enpdOrUqVLey9348eNp2rQp3bp1o1u3bowbN67SP6PQihUrmDVrVqW9nyUCN7PX/MqFz37HtB+2U7NaFSsSZ0xFZKTC/Gec35WgevXqrFixgrS0NGJjY3nxxRcr5X296a677mLFihWsWLGiaG0DT+Tn51focyo7EditIWDPoaP8bcYaPl/9Cx0a1+HNkb3o1DTG32EZExi+GAe/ri77mKO/wc400AKQCGjUCarVKf340zrDAM+/KPv06VO0nsChQ4cYPHgw+/fvJzc3l8cff5zBgweTnp7OgAEDOOuss1i4cCFNmzZlxowZVK9eneXLl3PTTTdRo0aN48pR5OTkcNttt7Fs2TKqVKnCs88+S9++fZk0aRKffPIJ+fn5pKWlMXbsWI4dO8bkyZOpVq0as2bNIjbWs3Lyc+bM4Z577iEvL49evXrx0ksvUa1aNRITE7npppv46quvGDNmDL169WL06NHs3r2bGjVq8Nprr9GuXTs++ugjHn30USIjI4mJieHrr7/mkUceITs7mwULFvDAAw8wdOhQj/8sS2ItAuBQTh7zf97NvRe3ZcaYMy0JGFNROVlOEgDnd05W2cdXQH5+PnPmzClaNCY6OpqPP/6YH374gblz5zJ27Nii1vvPP//M6NGjWbNmDXXr1mXatGkA/OEPf2DixIksWnT8ugqFrYzVq1fz3nvvMWLECHJycgBIS0vj3XffJTU1lYceeogaNWrw448/0qdPH955550SY33uueeKbg3Nnj2bnJwcRo4cyQcffMDq1avJy8vjpZdeKjo+OjqaBQsWMGzYMEaNGsW///1vli9fztNPP83tt98OwIQJE5g9ezYrV65k5syZVK1alQkTJjB06FBWrFhxykkAwqlFcGCb8ztzGbQ4l+0Hsvn4h0xG921FYlxNFj5wAbWqhc8fhzEe8+TKPSMV3h4E+ccgsipc+TrEp5zSx2ZnZ9OtWzfS09Pp2bMnF154IeCs9vfggw8yb948IiIi2L59Ozt37gQgKSmJbt26AdCzZ0/S09PJysriwIEDnHvuuYBTtvqLL74AYMGCBdxxxx0AtGvXjubNm7NhwwYA+vbtS+3atalduzYxMTFcdtllAHTu3LnU1c7uuusu7rnnnqLtlStXkpSURJs2bQAYMWIEL774In/5y18Air7EDx06xMKFC7n66quLXnv06FEAzjzzTEaOHMk111zDkCFDTunPtDRe/eYTkf7A80Ak8LqqPlns+WrAO0BPYC8wVFXTKz2QjFRYPgkAffcavuz5KvcsqkaBwsAuTUiMq2lJwJhTEZ8CI2ZC+nxIPPuUkwD83keQlZXFwIEDefHFF7nzzjuZMmUKu3fvZvny5URFRZGYmFh0FV+tWrWi10dGRpKdnV20TGxJyuoHdH+viIiIou2IiAjy8vI8Oofy+hlr1nTqnhUUFFC3bl1WrFhxwjEvv/wyS5Ys4fPPP6dbt24lHnOqvHZrSEQigReBAUAH4FoR6VDssJuB/araCngO+D+vBJM+HwqczpiCvKOsXvA5PZrX46u7ziExrnIK0BkT9uJT4OyxlZIE3MXExDBx4kSefvppcnNzycrKomHDhkRFRTF37ly2bt1a5uvr1q1LTEwMCxYsAJyy1YXOOeecou0NGzawbds22rZtW2mxt2vXjvT0dDZu3AjA5MmTi1om7urUqUNSUhIfffQR4CSQlStXArBp0yZ69+7NhAkTiIuLIyMjg9q1a3Pw4MFKi9ObfQQpwEZV3ayqx4D3gcHFjhkMvO16PBW4QLxRua16fRQnM0eg9O3RjnduSiE+1iqFGhMMunfvTteuXXn//fcZPnw4y5YtIzk5mSlTptCuXbtyX//WW28xevRo+vTpc9xKZ7fffjv5+fl07tyZoUOHMmnSpONaAqcqOjqat956i6uvvprOnTsTERHBrbfeWuKxU6ZM4Y033qBr16507NiRGTNmAHDvvffSuXNnOnXqxDnnnEPXrl3p27cva9eupVu3bnzwwQenHKfXylCLyFVAf1W9xbV9A9BbVce4HZPmOibTtb3JdcyeYu81ChgFkJCQ0LO8K4ATzH8G5jwGKEoEcsHDzpWLMaZEVoY6uFW0DLU3WwQlXdkXzzqeHIOqvqqqyaqa3KBBg4pHkng2VIkGiUSqVHO2jTHGAN7tLM4E4t22mwE7SjkmU0SqADHAvkqPxAsdWcYYEyq8mQiWAq1FJAnYDgwDrit2zExgBLAIuAr4Rr11ryo+xRKAMRVQ1mgbE7hO5ivUa7eGVDUPGAPMBtYBH6rqGhGZICKDXIe9AdQXkY3A3YD3inMYYzwWHR3N3r17rcxKkFFV9u7dS3R0xdZJCZ81i40xHsvNzSUzM7NofL4JHtHR0TRr1oyoqOMLZdqaxcaYComKiiIpKcnfYRgfsVpDxhgT5iwRGGNMmLNEYIwxYS7oOotFZDdQwanFReKAPeUeFVrsnMODnXN4OJVzbq6qJc7IDbpEcCpEZFlpveahys45PNg5hwdvnbPdGjLGmDBnicAYY8JcuCWCV/0dgB/YOYcHO+fw4JVzDqs+AmOMMScKtxaBMcaYYiwRGGNMmAvJRCAi/UVkvYhsFJETKpqKSDUR+cD1/BIRSfR9lJXLg3O+W0TWisgqEZkjIs39EWdlKu+c3Y67SkRURIJ+qKEn5ywi17j+rteIyLu+jrGyefBvO0FE5orIj65/35f4I87KIiJvisgu1wqOJT0vIjLR9eexSkR6nPKHqmpI/QCRwCagBVAVWAl0KHbM7cDLrsfDgA/8HbcPzrkvUMP1+LZwOGfXcbWBecBiINnfcfvg77k18CNQz7Xd0N9x++CcXwVucz3uAKT7O+5TPOdzgB5AWinPXwJ8gbPC4+nAklP9zFBsEaQAG1V1s6oeA94HBhc7ZjDwtuvxVOACCe4VOMo9Z1Wdq6pHXJuLcVaMC2ae/D0DPAY8BYRCPWVPzvmPwIuquh9AVXf5OMbK5sk5K1DH9TiGE1dCDCqqOo+yV2ocDLyjjsVAXRFpfCqfGYqJoCmQ4bad6dpX4jHqLKCTBdT3SXTe4ck5u7sZ54oimJV7ziLSHYhX1c98GZgXefL33AZoIyLfi8hiEenvs+i8w5NzHg9cLyKZwCzgDt+E5jcV/f9erlBcj6CkK/viY2Q9OSaYeHw+InI9kAyc69WIvK/McxaRCOA5YKSvAvIBT/6eq+DcHjoPp9U3X0Q6qeoBL8fmLZ6c87XAJFV9RkT6AJNd51zg/fD8otK/v0KxRZAJxLttN+PEpmLRMSJSBac5WVZTLNB5cs6ISD/gIWCQqh71UWzeUt451wY6Ad+KSDrOvdSZQd5h7Om/7RmqmquqW4D1OIkhWHlyzjcDHwKo6iIgGqc4W6jy6P97RYRiIlgKtBaRJBGpitMZPLPYMTOBEa7HVwHfqKsXJkiVe86u2ySv4CSBYL9vDOWcs6pmqWqcqiaqaiJOv8ggVQ3mdU49+bf9Cc7AAEQkDudW0WafRlm5PDnnbcAFACLSHicR7PZplL41E7jRNXrodCBLVX85lTcMuVtDqponImOA2TgjDt5U1TUiMgFYpqozgTdwmo8bcVoCw/wX8anz8Jz/CdQCPnL1i29T1UF+C/oUeXjOIcXDc54NXCQia4F84F5V3eu/qE+Nh+c8FnhNRO7CuUUyMpgv7ETkPZxbe3Gufo+/AVEAqvoyTj/IJcBG4Ajwh1P+zCD+8zLGGFMJQvHWkDHGmAqwRGCMMWHOEoExxoQ5SwTGGBPmLBEYY0yYs0RgAo6I5IvICrefxDKOTSytSmMFP/NbV4XLla7yDG1P4j1uFZEbXY9HikgTt+deF5EOlRznUhHp5sFr/iIiNU71s03oskRgAlG2qnZz+0n30ecOV9WuOAUJ/1nRF6vqy6r6jmtzJNDE7blbVHVtpUT5e5z/wbM4/wJYIjClskRggoLryn++iPzg+jmjhGM6ikiqqxWxSkRau/Zf77b/FRGJLOfj5gGtXK+9wFXnfrWrTnw11/4n5ff1HZ527RsvIveIyFU49ZymuD6zuutKPllEbhORp9xiHiki/z7JOBfhVmxMRF4SkWXirEPwqGvfnTgJaa6IzHXtu0hEFrn+HD8SkVrlfI4JcZYITCCq7nZb6GPXvl3AharaAxgKTCzhdbcCz6tqN5wv4kxXyYGhwJmu/fnA8HI+/zJgtYhEA5OAoaraGWcm/m0iEgtcAXRU1S7A4+4vVtWpwDKcK/duqprt9vRUYIjb9lDgg5OMsz9OSYlCD6lqMtAFOFdEuqjqRJw6NH1Vta+r7MTDQD/Xn+Uy4O5yPseEuJArMWFCQrbry9BdFPCC6554Pk4NneIWAQ+JSDNguqr+LCIXAD2Bpa7SGtVxkkpJpohINpCOU8q4LbBFVTe4nn8bGA28gLO+wesi8jngcZlrVd0tIptdNWJ+dn3G9673rUicNXFKLrivTnWNiIzC+X/dGGeRllXFXnu6a//3rs+pivPnZsKYJQITLO4CdgJdcVqyJyw0o6rvisgS4FJgtojcglOy921VfcCDzxjuXpROREpco8JV/yYFp9DZMGAMcH4FzuUD4BrgJ+BjVVVxvpU9jhNnpa4ngReBISKSBNwD9FLV/SIyCaf4WnEC/E9Vr61AvCbE2a0hEyxigF9cNeZvwLkaPo6ItAA2u26HzMS5RTIHuEpEGrqOiRXP12v+CUgUkVau7RuA71z31GNUdRZOR2xJI3cO4pTCLsl04HKcOvofuPZVKE5VzcW5xXO667ZSHeAwkCUijYABpcSyGDiz8JxEpIaIlNS6MmHEEoEJFv8BRojIYpzbQodLOGYokCYiK4B2OMv5rcX5wvxKRFYB/8O5bVIuVc3Bqez4kYisBgqAl3G+VD9zvd93OK2V4iYBLxd2Fhd73/3AWqC5qqa69lU4TlffwzPAPaq6Emet4jXAmzi3mwq9CnwhInNVdTfOiKb3XJ+zGOfPyoQxqz5qjDFhzloExhgT5iwRGGNMmLNEYIwxYc4SgTHGhDlLBMYYE+YsERhjTJizRGCMMWHu/wE5xWYIOpyB6AAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# generate a no skill prediction (majority class)\n",
    "ns_probs = [0 for _ in range(len(y_test))]\n",
    " \n",
    "rf_probs = rf_grid_search.predict_proba(X_test)\n",
    " \n",
    "rf_probs = rf_probs[:, 1]\n",
    " \n",
    "ns_auc = roc_auc_score(y_test, ns_probs)\n",
    "rf_auc = roc_auc_score(y_test, rf_probs)\n",
    " \n",
    "print('No Skill: ROC AUC=%.3f' % (ns_auc))\n",
    "print('Random Forest: ROC AUC=%.3f' % (rf_auc))\n",
    " \n",
    "ns_fpr, ns_tpr, _ = roc_curve(y_test, ns_probs)\n",
    "rf_fpr, rf_tpr, _ = roc_curve(y_test, rf_probs)\n",
    "\n",
    "\n",
    "pyplot.plot(ns_fpr, ns_tpr, linestyle='--', label='No Skill')\n",
    "pyplot.plot(rf_fpr, rf_tpr, marker='.', label='Random Forest')\n",
    "pyplot.xlabel('False Positive Rate')\n",
    "pyplot.ylabel('True Positive Rate')\n",
    "pyplot.legend()\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Algorithm Performance Comparison"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train_test_split  is only done once before running the differnet models and algorithms to ensure that the models are being fit and predicted on the same set of data. Running train_test_split again and again causes re-sampling and the datasets are different. Hence, our accuracy is hampered."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The Random Forest gives us the highest accuracy, around 78%. ANN gives us a slightly lower accuracy of 75% and Logistic Regression gives us the lowest accuracy of 73%. All the 3 algorithms have approximately the same accuracy around 75%. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Here, no overfitting occurs as the accuracy is not perfect, rather is at an optimal amount of 75% , which allows maximum number of correct prediction with some room for misclassification. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ANN gives us maximum AUC for the ROC curve at 0.877. While Logistic Regression gives us the least AUC 0.84. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Out of the 3 models, ANN runs the slowest because of its large computation power. Logistic Regression runs the fastest out of the three because of strict convex feature. Random forest is a type of ensemble learning so it runs relatively slower. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment analysis is hard due to things such as subjectivity, tone, lack of context, irony, and sarcasm. So, it’s not only hard for algorithms to do sentiment analysis but also for humans. Therefore any accuracy within the range of 60 to 80% is considered good. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}