Abstractive_Summary (1).ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Abstractive_Summary.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GFIJC_Rk_qTL",
        "colab_type": "text"
      },
      "source": [
        "## Problem Statement\n",
        "#### Customer reviews can often be long and descriptive. Analyzing these reviews manually, as you can imagine, is really time-consuming. This is where the brilliance of Natural Language Processing can be applied to generate a summary for long reviews.\n",
        "\n",
        "We will be working on a really cool dataset. Our objective here is to generate a summary for the Amazon Fine Food reviews using the abstraction-based approach we learned about above. You can download the dataset from Kaggle"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HCEz_oJ5CWbV",
        "colab_type": "code",
        "outputId": "97cd0cbe-3a87-42e7-8236-e2e4a9ade99e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "source": [
        "# Load the Drive helper and mount\n",
        "from google.colab import drive\n",
        "\n",
        "# This will prompt for authorization.\n",
        "drive.mount('/content/gdrive')\n",
        "#* updating filepath to your Google Drive\n",
        "\n",
        "#Make sure you have this path available in your google drive\n",
        "data=\"/content/gdrive/My Drive/colab_notebooks/amazon_food.csv\""
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly\n",
            "\n",
            "Enter your authorization code:\n",
            "··········\n",
            "Mounted at /content/gdrive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cHjnQlkpCDPZ",
        "colab_type": "text"
      },
      "source": [
        "## Read the dataset\n",
        "### This dataset consists of reviews of fine foods from Amazon. The data spans a period of more than 10 years, including all ~500,000 reviews up to October 2012. These reviews include product and user information, ratings, plain text review, and summary. It also includes reviews from all other Amazon categories.\n",
        "\n",
        "### We’ll take a sample of 100,000 reviews to reduce the training time of our model. Feel free to use the entire dataset for training your model if your machine has that kind of computational power."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "VsoUo6Q6AKxp",
        "colab_type": "text"
      },
      "source": [
        "#### This dataset has 578413 Reviews but due to lack of time and system capability i am using only on 100000 if u want bothered about your acccuracy you can also use a lots of data for training ."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZXFeI6wTCpaE",
        "colab_type": "code",
        "outputId": "29b77599-75bc-4553-cc1e-90663f8dbd53",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 80
        }
      },
      "source": [
        "import pandas as pd # import pandas for read the file\n",
        "import re # import regex for remove the useless symbols.... \n",
        "from bs4 import BeautifulSoup # if you want to take the data fromA API or web scrapping then use this \n",
        "from keras.preprocessing.text import Tokenizer # import tokenizer to token word\n",
        "from keras.preprocessing.sequence import pad_sequences   # import pad sequence works as it fill zero on the blank value\n",
        "from nltk.corpus import stopwords # use stopwords  for preprocessing \n",
        "from tensorflow.keras.layers import Input, LSTM, Embedding, Dense, Concatenate, TimeDistributed ## use lstm embadding dense\n",
        "from tensorflow.keras.models import Model  \n",
        "from tensorflow.keras.callbacks import EarlyStopping # use earlystopping  to avoid overfitting\n",
        "import warnings\n",
        "pd.set_option(\"display.max_colwidth\", 200)\n",
        "warnings.filterwarnings(\"ignore\")"
      ],
      "execution_count": 3,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using TensorFlow backend.\n"
          ],
          "name": "stderr"
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "<p style=\"color: red;\">\n",
              "The default version of TensorFlow in Colab will soon switch to TensorFlow 2.x.<br>\n",
              "We recommend you <a href=\"https://www.tensorflow.org/guide/migrate\" target=\"_blank\">upgrade</a> now \n",
              "or ensure your notebook will continue to use TensorFlow 1.x via the <code>%tensorflow_version 1.x</code> magic:\n",
              "<a href=\"https://colab.research.google.com/notebooks/tensorflow_version.ipynb\" target=\"_blank\">more info</a>.</p>\n"
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2B9LbynGCbWA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# read te dataset\n",
        "data = pd.read_csv(data)[:200000]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ids83GDaC5pz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# drop duplicate and NA values\n",
        "data.drop_duplicates(subset=['Text'],inplace=True)#dropping duplicates\n",
        "data.dropna(axis=0,inplace=True)#dropping na"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3ksfzGTpCSnI",
        "colab_type": "text"
      },
      "source": [
        "### Information about dataset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vyevZ2r2CZog",
        "colab_type": "code",
        "outputId": "dc621f4d-c05d-461a-d2bd-d00e7f9a42e3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 272
        }
      },
      "source": [
        "data.info()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "<class 'pandas.core.frame.DataFrame'>\n",
            "Int64Index: 162834 entries, 0 to 199999\n",
            "Data columns (total 10 columns):\n",
            "Id                        162834 non-null int64\n",
            "ProductId                 162834 non-null object\n",
            "UserId                    162834 non-null object\n",
            "ProfileName               162834 non-null object\n",
            "HelpfulnessNumerator      162834 non-null int64\n",
            "HelpfulnessDenominator    162834 non-null int64\n",
            "Score                     162834 non-null int64\n",
            "Time                      162834 non-null int64\n",
            "Summary                   162834 non-null object\n",
            "Text                      162834 non-null object\n",
            "dtypes: int64(5), object(5)\n",
            "memory usage: 13.7+ MB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "tApirT1zCd0s",
        "colab_type": "code",
        "outputId": "16101ec0-c436-4847-8174-11e3843cfd93",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 297
        }
      },
      "source": [
        "data.describe()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Id</th>\n",
              "      <th>HelpfulnessNumerator</th>\n",
              "      <th>HelpfulnessDenominator</th>\n",
              "      <th>Score</th>\n",
              "      <th>Time</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>count</th>\n",
              "      <td>162834.000000</td>\n",
              "      <td>162834.000000</td>\n",
              "      <td>162834.00000</td>\n",
              "      <td>162834.000000</td>\n",
              "      <td>1.628340e+05</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>mean</th>\n",
              "      <td>94083.608945</td>\n",
              "      <td>1.700922</td>\n",
              "      <td>2.17314</td>\n",
              "      <td>4.168466</td>\n",
              "      <td>1.296507e+09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>std</th>\n",
              "      <td>57457.267552</td>\n",
              "      <td>6.784802</td>\n",
              "      <td>7.39892</td>\n",
              "      <td>1.312606</td>\n",
              "      <td>4.770465e+07</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>min</th>\n",
              "      <td>1.000000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>1.000000</td>\n",
              "      <td>9.393408e+08</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>25%</th>\n",
              "      <td>43533.250000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>0.00000</td>\n",
              "      <td>4.000000</td>\n",
              "      <td>1.271462e+09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>50%</th>\n",
              "      <td>91353.500000</td>\n",
              "      <td>0.000000</td>\n",
              "      <td>1.00000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>1.311293e+09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>75%</th>\n",
              "      <td>143102.750000</td>\n",
              "      <td>2.000000</td>\n",
              "      <td>2.00000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>1.332634e+09</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>max</th>\n",
              "      <td>200000.000000</td>\n",
              "      <td>866.000000</td>\n",
              "      <td>878.00000</td>\n",
              "      <td>5.000000</td>\n",
              "      <td>1.351210e+09</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                  Id  HelpfulnessNumerator  ...          Score          Time\n",
              "count  162834.000000         162834.000000  ...  162834.000000  1.628340e+05\n",
              "mean    94083.608945              1.700922  ...       4.168466  1.296507e+09\n",
              "std     57457.267552              6.784802  ...       1.312606  4.770465e+07\n",
              "min         1.000000              0.000000  ...       1.000000  9.393408e+08\n",
              "25%     43533.250000              0.000000  ...       4.000000  1.271462e+09\n",
              "50%     91353.500000              0.000000  ...       5.000000  1.311293e+09\n",
              "75%    143102.750000              2.000000  ...       5.000000  1.332634e+09\n",
              "max    200000.000000            866.000000  ...       5.000000  1.351210e+09\n",
              "\n",
              "[8 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0t2FCtGlCk4J",
        "colab_type": "code",
        "outputId": "1f073d02-baff-49d7-e1d0-b693013710c8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        }
      },
      "source": [
        "data.dtypes"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Id                         int64\n",
              "ProductId                 object\n",
              "UserId                    object\n",
              "ProfileName               object\n",
              "HelpfulnessNumerator       int64\n",
              "HelpfulnessDenominator     int64\n",
              "Score                      int64\n",
              "Time                       int64\n",
              "Summary                   object\n",
              "Text                      object\n",
              "cleaned_text              object\n",
              "cleaned_summary           object\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 47
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jw0wSftUCpiy",
        "colab_type": "text"
      },
      "source": [
        "## import stopwords form nltk libray to remove stopwords"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SnDtg-IADBkc",
        "colab_type": "code",
        "outputId": "9809d2ca-2dcf-46c2-f380-d5b8ee69773b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "import nltk\n",
        "nltk.download('stopwords')"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package stopwords to /root/nltk_data...\n",
            "[nltk_data]   Unzipping corpora/stopwords.zip.\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "diXfvzQYDDzh",
        "colab_type": "text"
      },
      "source": [
        "# Now, Perform most important processing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SZRlPqFhDJau",
        "colab_type": "text"
      },
      "source": [
        "#####  Performing basic preprocessing steps is very important before we get to the model building part. Using messy and uncleaned text data is a potentially disastrous move. So in this step, we will drop all the unwanted symbols, characters, etc. from the text that do not affect the objective of our problem.\n",
        "\n",
        "Here is the dictionary that we will use for expanding the contractions:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vBwyDw-RC-Zq",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "contraction_mapping = {\"ain't\": \"is not\", \"aren't\": \"are not\",\"can't\": \"cannot\", \"'cause\": \"because\", \"could've\": \"could have\", \"couldn't\": \"could not\",\n",
        "                           \"didn't\": \"did not\",  \"doesn't\": \"does not\", \"don't\": \"do not\", \"hadn't\": \"had not\", \"hasn't\": \"has not\", \"haven't\": \"have not\",\n",
        "                           \"he'd\": \"he would\",\"he'll\": \"he will\", \"he's\": \"he is\", \"how'd\": \"how did\", \"how'd'y\": \"how do you\", \"how'll\": \"how will\", \"how's\": \"how is\",\n",
        "                           \"I'd\": \"I would\", \"I'd've\": \"I would have\", \"I'll\": \"I will\", \"I'll've\": \"I will have\",\"I'm\": \"I am\", \"I've\": \"I have\", \"i'd\": \"i would\",\n",
        "                           \"i'd've\": \"i would have\", \"i'll\": \"i will\",  \"i'll've\": \"i will have\",\"i'm\": \"i am\", \"i've\": \"i have\", \"isn't\": \"is not\", \"it'd\": \"it would\",\n",
        "                           \"it'd've\": \"it would have\", \"it'll\": \"it will\", \"it'll've\": \"it will have\",\"it's\": \"it is\", \"let's\": \"let us\", \"ma'am\": \"madam\",\n",
        "                           \"mayn't\": \"may not\", \"might've\": \"might have\",\"mightn't\": \"might not\",\"mightn't've\": \"might not have\", \"must've\": \"must have\",\n",
        "                           \"mustn't\": \"must not\", \"mustn't've\": \"must not have\", \"needn't\": \"need not\", \"needn't've\": \"need not have\",\"o'clock\": \"of the clock\",\n",
        "                           \"oughtn't\": \"ought not\", \"oughtn't've\": \"ought not have\", \"shan't\": \"shall not\", \"sha'n't\": \"shall not\", \"shan't've\": \"shall not have\",\n",
        "                           \"she'd\": \"she would\", \"she'd've\": \"she would have\", \"she'll\": \"she will\", \"she'll've\": \"she will have\", \"she's\": \"she is\",\n",
        "                           \"should've\": \"should have\", \"shouldn't\": \"should not\", \"shouldn't've\": \"should not have\", \"so've\": \"so have\",\"so's\": \"so as\",\n",
        "                           \"this's\": \"this is\",\"that'd\": \"that would\", \"that'd've\": \"that would have\", \"that's\": \"that is\", \"there'd\": \"there would\",\n",
        "                           \"there'd've\": \"there would have\", \"there's\": \"there is\", \"here's\": \"here is\",\"they'd\": \"they would\", \"they'd've\": \"they would have\",\n",
        "                           \"they'll\": \"they will\", \"they'll've\": \"they will have\", \"they're\": \"they are\", \"they've\": \"they have\", \"to've\": \"to have\",\n",
        "                           \"wasn't\": \"was not\", \"we'd\": \"we would\", \"we'd've\": \"we would have\", \"we'll\": \"we will\", \"we'll've\": \"we will have\", \"we're\": \"we are\",\n",
        "                           \"we've\": \"we have\", \"weren't\": \"were not\", \"what'll\": \"what will\", \"what'll've\": \"what will have\", \"what're\": \"what are\",\n",
        "                           \"what's\": \"what is\", \"what've\": \"what have\", \"when's\": \"when is\", \"when've\": \"when have\", \"where'd\": \"where did\", \"where's\": \"where is\",\n",
        "                           \"where've\": \"where have\", \"who'll\": \"who will\", \"who'll've\": \"who will have\", \"who's\": \"who is\", \"who've\": \"who have\",\n",
        "                           \"why's\": \"why is\", \"why've\": \"why have\", \"will've\": \"will have\", \"won't\": \"will not\", \"won't've\": \"will not have\",\n",
        "                           \"would've\": \"would have\", \"wouldn't\": \"would not\", \"wouldn't've\": \"would not have\", \"y'all\": \"you all\",\n",
        "                           \"y'all'd\": \"you all would\",\"y'all'd've\": \"you all would have\",\"y'all're\": \"you all are\",\"y'all've\": \"you all have\",\n",
        "                           \"you'd\": \"you would\", \"you'd've\": \"you would have\", \"you'll\": \"you will\", \"you'll've\": \"you will have\",\n",
        "                           \"you're\": \"you are\", \"you've\": \"you have\"}"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y6-sRaVKDKJz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "stop_words = set(stopwords.words('english')) \n",
        "\n",
        "def text_cleaner(text,num):\n",
        "    newString = text.lower()\n",
        "    newString = BeautifulSoup(newString, \"lxml\").text\n",
        "    newString = re.sub(r'\\([^)]*\\)', '', newString)\n",
        "    newString = re.sub('\"','', newString)\n",
        "    newString = ' '.join([contraction_mapping[t] if t in contraction_mapping else t for t in newString.split(\" \")])    \n",
        "    newString = re.sub(r\"'s\\b\",\"\",newString)\n",
        "    newString = re.sub(\"[^a-zA-Z]\", \" \", newString) \n",
        "    newString = re.sub('[m]{2,}', 'mm', newString)\n",
        "    if(num==0):\n",
        "        tokens = [w for w in newString.split() if not w in stop_words]\n",
        "    else:\n",
        "        tokens=newString.split()\n",
        "    long_words=[]\n",
        "    for i in tokens:\n",
        "        if len(i)>1:                                                 #removing short word\n",
        "            long_words.append(i)   \n",
        "    return (\" \".join(long_words)).strip()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zlmodkPLDR_s",
        "colab_type": "text"
      },
      "source": [
        "### We will perform the below preprocessing tasks for our data:\n",
        "\n",
        "1.Convert everything to lowercase\n",
        "\n",
        "2.Remove HTML tags\n",
        "\n",
        "3.Contraction mapping\n",
        "\n",
        "4.Remove (‘s)\n",
        "\n",
        "5.Remove any text inside the parenthesis ( )\n",
        "\n",
        "6.Eliminate punctuations and special characters\n",
        "\n",
        "7.Remove stopwords\n",
        "\n",
        "8.Remove short words\n",
        "\n",
        "Let’s define the function"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "I08or-h4DK9S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cleaned_text = []\n",
        "for t in data['Text']:\n",
        "    cleaned_text.append(text_cleaner(t,0))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gzr67PoTDOHK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#call the function\n",
        "cleaned_summary = []\n",
        "for t in data['Summary']:\n",
        "    cleaned_summary.append(text_cleaner(t,1))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IuCJGNdyDcJw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "data['cleaned_text']=cleaned_text\n",
        "data['cleaned_summary']=cleaned_summary"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4kYjoME9DfsF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "data.replace('', np.nan, inplace=True)\n",
        "data.dropna(axis=0,inplace=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "U61RuUT_DaEv",
        "colab_type": "text"
      },
      "source": [
        "## Understanding the distribution of the sequences\n",
        "Here, we will analyze the length of the reviews and the summary to get an overall idea about the distribution of length of the text. This will help us fix the maximum length of the sequence:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1OYRT-OFDjF5",
        "colab_type": "code",
        "outputId": "686ed7c7-0b54-45e3-ca64-caa5cf114f20",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 281
        }
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "\n",
        "text_word_count = []\n",
        "summary_word_count = []\n",
        "\n",
        "# populate the lists with sentence lengths\n",
        "for i in data['cleaned_text']:\n",
        "      text_word_count.append(len(i.split()))\n",
        "\n",
        "for i in data['cleaned_summary']:\n",
        "      summary_word_count.append(len(i.split()))\n",
        "\n",
        "length_df = pd.DataFrame({'text':text_word_count, 'summary':summary_word_count})\n",
        "\n",
        "length_df.hist(bins = 30)\n",
        "plt.show()"
      ],
      "execution_count": 14,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYMAAAEICAYAAAC9E5gJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3df5BV5Z3n8fdHUGPsGDCajgFnYEaS\nLSITf7DCVFI7HY2IODOYKpPguAGNFZIVZ8wulYiZrSWjcRdrVjOyY0iRwAiOkThGIxNxSAe5lbU2\noKBEBXXoKIlNoUT5YdpEHZzv/nGe1tO37+17G7rvr/68qm7de77nxz1P1+n+9vOc5zmPIgIzMxvZ\njqr3CZiZWf05GZiZmZOBmZk5GZiZGU4GZmaGk4GZmeFkYGZmOBmYWYOTtEvSJ4fgOLdL+sZQnFMr\ncjKwqkkaXe9zMLPh4WRQY5KulbRb0m8kPSvpvOL/WCR1SOrOLe+S9BVJT0h6TdIKSe2SHkzH+Ymk\nsWnbCZJC0hWSXpC0X9KXJP3HtP8BSX+fO/YfSnpI0iuSXpZ0p6QxRd99raQngNfSefygqExLJd06\nrD84G5Ek3QH8HvDPknokfVXSdEn/L13LP5fUkbY9UVK3pD9Ly22SuiTNlTQfuAz4ajrOP9etUI0q\nIvyq0Qv4MPAC8MG0PAH4Q+B24Bu57TqA7tzyLmAT0A6MA/YCjwFnAu8CHgIW544ZwLfTuhnA68AP\ngffn9v+TtP1pwPnAscDJwE+Bvyv67m3AqcBxwCnAa8CYtH50Ot7Z9f75+tWar3QNfjJ9Hge8Aswi\n+2f2/LR8clo/A3gxXevfAe7JHafP75lffV+uGdTWW2R/dCdLOjoidkXEL6rc9/9ExEsRsRv4v8Dm\niHg8Il4H7iNLDHk3RMTrEfFjsj/ed0XE3tz+ZwJERFdEdEbEGxHxa+AW4E+KjrU0Il6IiN9FxB6y\nhPHptG4m8HJEbB3UT8Ls8PxnYF1ErIuIf4+ITmALWXIgXe//BGxIsS/W7UybjJNBDUVEF/Bl4OvA\nXklrJH2wyt1fyn3+XYnltsPZPjU3rUlNV68C/wicVHSsF4qWV5H9UpLe76iyDGZH6veBT6cmogOS\nDgAfJ6ux9loOnA7cHhGv1OMkm5GTQY1FxPci4uNkF3UAN5H95/7u3GYfqOEp/c90HlMi4gSyP+4q\n2qb40bY/BP5I0unAnwJ3DvtZ2kiWv/5eAO6IiDG51/ERsQRA0iiyZLAauErSaWWOY0WcDGpI0ocl\nnSvpWLJ2/N8B/07WJj8r3QD7AFntoVbeA/QAByWNA75SaYfUNHUP8D3gkYj41fCeoo1wLwF/kD7/\nI/Bnki6QNErSu1KHi/Fp/dfI/uh/HvhbYHVKEMXHsSJOBrV1LLAEeJl3bnJdR9bM8nOyG2U/Br5f\nw3P6G+As4CDwAHBvlfutAqbgJiIbfv8L+O+pSeizwGyyP/q/JqspfAU4StLZwH8D5kbEW2S17gAW\npeOsILtfd0DSD2tchoandJfdbFAk/R7wDPCBiHi13udjZkfGNQMbNElHkf0HtsaJwKw1eESpDYqk\n48naXn9J1q3UzFqAm4nMzMzNRDZySVopaa+kp3Kxv5X0THp0x31Fj+a4Lj3e4FlJF+TiM1OsS9Ki\nXHyipM0p/n1Jx6T4sWm5K62fUJsSm5XXtDWDk046KU4++WSOP/74ep/KEXnttddchjrZunXrQeBc\nYHVEnA4gaQbwUEQcknQTQERcK2kycBdwDvBB4CfAh9Kh/pXssQjdwKPApRGxQ9LdwL0RsUbSt4Gf\nR8QySVcBfxQRX5I0B/hURHy20vmedNJJMWHChH7xZv35V+JyDY+tW7e+HBEn91tR7+dhHO7r7LPP\njo0bN0azcxnqh+wxBhOAp6LENQZ8Crgzfb4OuC63bj3wx+m1Phe/Lr1E1oV4dIq/vV3vvunz6LSd\nSp1DFF3zpTTrz78Sl2t4AFuixPXlG8hm5X2ed8Z8jCN7WGCv7hSDvo/r6AamAe8DDkTEoRLbj+vd\nJ7IayMG0/cvFJ5CetjkfoL29nUKh0O8ke3p6SsabnctVW04GZiVI+mvgEHV+1EZELCd7vAJTp06N\njo6OftsUCgVKxZudy1VbTgZmRSRdTvbMpfNStRpgN9ljvHuNTzHKxF8BxkganWoH+e17j9WtbMKg\n96btzerGvYnMciTNBL4K/HlE/Da3ai0wJ/UEmghMAh4hu2E8KfUcOgaYA6xNSWQjcEnafx5wf+5Y\n89LnS8huWDdnTw5rGa4Z2Eg2EfgZcJKymeUWk938PRbolASwKSK+FBHbU++gHWTNRwsie/4Nkq4m\nuyk8ClgZEdvT8a8F1iibxe5xsmfjkN7vkNQF7CNLIGZ15WRgI9nzETG1KLai5JZARNwI3Fgivg5Y\nVyL+HFlX1OL467wzOZBZQ3AzkZmZORmYmVkVySBNHvGIpJ9L2i7pb1L8dknPS9qWXmekuCQtTUPt\nn5B0Vu5Y8yTtTK95ufjZkp5M+yxVaqw1M7PaqOaewRvAuRHRI+lo4GFJD6Z1X4mIe4q2v5Csp8Uk\nssE3y4Bpkk4ku0E3lWzCia2S1kbE/rTNF4DNZG2vM4EHGSITFj3QZ3nXkouG6tBmDcHXuB2pijWD\nNIK5Jy0enV4DdYObTfasl4iITWR9rU8BLgA6I2JfSgCdwMy07oSI2JS6160GLj6CMpmZ2SBV1Zso\nzSG6FTgNuC0iNkv6L8CNkv4HsAFYFBFvkBtqn/QOwx8o3l0iXuo8+gzNr3ZY98Iph/osN9JQ8EYd\nmj4YrVAGs5GuqmSQ+lOfkR7ne5+k08n6Y78IHEM2XP5a4PrhOtF0Hn2G5re1tVU1rPvy4ir0ZZX3\nqZVGHZo+GK1QBrORblC9iSLiANmoypkRsSc1Bb0B/APv9KcuN2x/oPj4EnEzM6uRanoTndw7wYek\n48ie2/5Mausn9fy5GOidIGQtMDf1KpoOHIyIPWQjNGdIGitpLDCD7JG+e4BXJU1Px5rLO8P2zcys\nBqppJjoFWJXuGxwF3B0RP5L0kKSTyZ7bvg34Utp+HTAL6AJ+C1wBEBH7JN1A9iwXgOsjYl/6fBVw\nO3AcWS+iIetJZGZmlVVMBhHxBHBmifi5ZbYPYEGZdSuBlSXiW4DTK52LmZkND49ANjMzJwMzM3My\nMDMznAzMzAwnAzMzw8nAzMxwMjAzM5wMzMwMJwMzM8PJwMzMcDIwMzOcDMzMDCcDMzPDycDMzHAy\nMDMznAzMzAwnAxvZJkjaK6l3ylYknSipU9LO9D42xSVpqaQuSU9IOiu3z7y0/U5J83LxsyU9mfZZ\nmqZ1LfsdZvXkZGAj2cvAzKLYImBDREwCNqRlgAuBSek1H1gG2R92YDEwDTgHWJz7474M+EJuv5kV\nvsOsbqqZA7nlTFj0QL/YriUX1eFMrM56gH1FsdlAR/q8CigA16b46jSt6yZJYySdkrbt7J3PW1In\nMFNSATghIjal+GrgYrL5vct9h1ndjMhkYDaA9ojYkz6/CLSnz+OAF3LbdafYQPHuEvGBvqMfSfPJ\naiK0t7dTKBT6bdPT08PCKW/1iZXartn09PS0RDmKNWq5KiYDSe8Cfgocm7a/JyIWS5oIrAHeB2wF\nPhcRb0o6FlgNnA28Anw2InalY10HXAm8BfxVRKxP8ZnArcAo4LsRsWRIS2l2GCIiJEU9vyMilgPL\nAaZOnRodHR39tikUCtz88Gt9Yrsu679dsykUCpQqb7Nr1HJVc8/gDeDciPgocAZZFXg6cBPwzYg4\nDdhP9kee9L4/xb+ZtkPSZGAO8BGyttNvSRolaRRwG1mb7GTg0rStWT28lJp/SO97U3w3cGpuu/Ep\nNlB8fIn4QN9hVjcVk0FketLi0ekVwLnAPSm+iqw9FLL20FXp8z3AeakXxWxgTUS8ERHPA11kN9zO\nAboi4rmIeJOstjH7iEtmdnjWAr09guYB9+fic1OvounAwdTUsx6YIWlsunE8A1if1r0qaXq6/ucW\nHavUd5jVTVW9idJ/8NvI/oPpBH4BHIiIQ2mTfHvo222oaf1Bsqakwba5mg23icDPgA9L6pZ0JbAE\nOF/STuCTaRlgHfAc2T8x3wGuAkg3jm8AHk2v63tvJqdtvpv2+QXZzWMG+A6zuqnqBnJEvAWcIWkM\ncB/wH4b1rMoovplW7Y2YhVMOVdymXjd0GvVm0mA0cRmej4ipJeLnFQdSL6IFpQ4SESuBlSXiW4DT\nS8RfKfUdZvU0qN5EEXFA0kbgj4Exkkan//7z7aG9bajdkkYD7yW7kVyubZUB4sXf3+dmWltbW1U3\nYi4v0ZW0WL1uuDXqzaTBaIUymI10FZuJJJ2cagRIOg44H3ga2AhckjYrblvtbQ+9BHgo/Ve1Fpgj\n6djUE2kS8AhZ1XqSpImSjiG7ybx2KApnZmbVqaZmcAqwKvX6OQq4OyJ+JGkHsEbSN4DHgRVp+xXA\nHZK6yAb0zAGIiO2S7gZ2AIeABan5CUlXk92IGwWsjIjtQ1ZCMzOrqGIyiIgngDNLxJ8j6wlUHH8d\n+HSZY90I3Fgivo7sBp2ZmdWBn01kZmZOBmZm5mRgZmY4GZiZGU4GZmaGk4GZmeFkYGZmOBmYmRlO\nBmZmhpOBmZnhZGBmZjgZmJkZTgZmZoaTgZmZ4WRgZmY4GZiZGU4GZmaGk4GZmeFkYGZmOBmYmRkw\nut4n0CgmLHqgz/KuJRfV6UzMzGqvYs1A0qmSNkraIWm7pGtS/OuSdkvall6zcvtcJ6lL0rOSLsjF\nZ6ZYl6RFufhESZtT/PuSjhnqgpqZWXnVNBMdAhZGxGRgOrBA0uS07psRcUZ6rQNI6+YAHwFmAt+S\nNErSKOA24EJgMnBp7jg3pWOdBuwHrhyi8pmZWRUqJoOI2BMRj6XPvwGeBsYNsMtsYE1EvBERzwNd\nwDnp1RURz0XEm8AaYLYkAecC96T9VwEXH26BzIaCpP+aasJPSbpL0rvK1WAlHZuWu9L6CbnjDKqW\nbFYvg7pnkC7yM4HNwMeAqyXNBbaQ1R72kyWKTbndunknebxQFJ8GvA84EBGHSmxf/P3zgfkA7e3t\n9PT0UCgUKp73wimHKm5TrJrjDoVqy9DIWqEMeZLGAX8FTI6I30m6m6y2O4usBrtG0rfJarDL0vv+\niDhN0hyymu5ni2rJHwR+IulD6WtuA84nu94flbQ2InbUsJhmfVSdDCS1AT8AvhwRr0paBtwARHq/\nGfj8sJxlEhHLgeUAU6dOjba2Njo6Oirud3nRzeFq7Lqs8nGHQqFQqKoMjawVylDCaOA4Sf8GvBvY\nQ1aD/Yu0fhXwdbJkMDt9hqyG+/epxvt2LRl4XlJvLRlSLRlA0pq0rZOB1U1VyUDS0WSJ4M6IuBcg\nIl7Krf8O8KO0uBs4Nbf7+BSjTPwVYIyk0al2kN/erOYiYrek/w38Cvgd8GNgK+VrsONItd6IOCTp\nIFmNd7C15H6Ka8OlamA9PT0snPJWn1gr1NRarcbZq1HLVTEZpP9wVgBPR8QtufgpEbEnLX4KeCp9\nXgt8T9ItZFXjScAjgIBJkiaS/bGfA/xFRISkjcAlZPcR5gH3D0XhzA6HpLFk/6lPBA4A/0TWGaLm\nimvDpWpghUKBmx9+rU+sVjXb4dSiNc6GLVc1NYOPAZ8DnpS0LcW+RtYb6AyyZqJdwBcBImJ7amPd\nQdYTaUFEvAUg6WpgPTAKWBkR29PxrgXWSPoG8DhZ8jGrl08Cz0fErwEk3Uv2e1CuBttbG+6WNBp4\nL1mNd7C1ZLO6qZgMIuJhsv/qi60bYJ8bgRtLxNeV2i+1nZ5THDerk18B0yW9m6yZ6DyyThLlarBr\n0/LP0vqHUo13ULXkGpXNrCSPQDYrEhGbJd0DPEZWu32crKnmAUrXYFcAd6QbxPvI/rgfbi3ZrC6c\nDMxKiIjFwOKicMkabES8Dny6zHEGVUs2qxc/qM7MzJwMzMzMycDMzHAyMDMznAzMzAwnAzMzw8nA\nzMxwMjAzM5wMzMwMJwMzM8PJwMzMcDIwMzOcDMzMDCcDMzPDycDMzHAyMDMznAzMzAwnAzMzo4pk\nIOlUSRsl7ZC0XdI1KX6ipE5JO9P72BSXpKWSuiQ9Iems3LHmpe13SpqXi58t6cm0z1JJGo7CmplZ\nadXUDA4BCyNiMjAdWCBpMrAI2BARk4ANaRngQmBSes0HlkGWPMjmlJ1GNo/s4t4Ekrb5Qm6/mUde\nNDMzq1bFZBAReyLisfT5N8DTwDhgNrAqbbYKuDh9ng2sjswmYIykU4ALgM6I2BcR+4FOYGZad0JE\nbIqIAFbnjmVmZjUwejAbS5oAnAlsBtojYk9a9SLQnj6PA17I7dadYgPFu0vES33/fLLaBu3t7fT0\n9FAoFCqe98IphypuU6ya4w6FasvQyFqhDGYjXdXJQFIb8APgyxHxar5ZPyJCUgzD+fUREcuB5QBT\np06NtrY2Ojo6Ku53+aIHBv1duy6rfNyhUCgUqipDI2uFMpiNdFX1JpJ0NFkiuDMi7k3hl1ITD+l9\nb4rvBk7N7T4+xQaKjy8RNzOzGqmmN5GAFcDTEXFLbtVaoLdH0Dzg/lx8bupVNB04mJqT1gMzJI1N\nN45nAOvTulclTU/fNTd3LDMzq4Fqmok+BnwOeFLSthT7GrAEuFvSlcAvgc+kdeuAWUAX8FvgCoCI\n2CfpBuDRtN31EbEvfb4KuB04DngwvczMrEYqJoOIeBgo1+//vBLbB7CgzLFWAitLxLcAp1c6FzMz\nGx4egWxWgqQxku6R9IykpyX9sQdaWisbVNfSkWRCUQ+kXUsuqtOZWJ3cCvxLRFwi6Rjg3WTNoxsi\nYomkRWQDLa+l70DLaWSDKKflBlpOBQLYKmltGmfTO9ByM1nT6kzcPGp15JqBWRFJ7wX+E1nHCSLi\nzYg4gAdaWgtzzcCsv4nAr4F/kPRRYCtwDQ0w0LLU4L6enh4WTnmrT6wVBgG26mDGRi2Xk4FZf6OB\ns4C/jIjNkm7lnWdvAfUbaFlqcF+hUODmh1/rE6vVoMnh1KqDGRu1XG4mMuuvG+iOiM1p+R6y5OCB\nltaynAzMikTEi8ALkj6cQucBO/BAS2thbiYyK+0vgTtTT6LnyAZPHoUHWlqLcjIwKyEitpF1CS3m\ngZbWktxMZGZmTgZmZuZkYGZmOBmYmRlOBmZmRgv2Jip+wJyZmVXmmoGZmTkZmJmZk4GZmeFkYGZm\nOBmYmRlVJANJKyXtlfRULvZ1SbslbUuvWbl116V5XZ+VdEEuPjPFutKUgb3xiZI2p/j304PBzMys\nhqqpGdxONj9rsW9GxBnptQ5A0mRgDvCRtM+3JI2SNAq4jWyu2MnApWlbgJvSsU4D9gNXHkmBzMxs\n8Comg4j4KbCv0nbJbGBNRLwREc+TPdL3nPTqiojnIuJNYA0wOz3L/VyyyUOg77yyZmZWI0cy6Oxq\nSXOBLcDCNOH3OGBTbpv83K7Fc8FOA94HHIiIQyW276d4PthSc4kunHKoxJ5HbrjmLG3U+VAHoxXK\nYDbSHW4yWAbcAER6vxn4/FCdVDnF88G2tbX1m0v08mEagTxcc8o26nyog9EKZTAb6Q4rGUTES72f\nJX0H+FFaLDfnK2XirwBjJI1OtQPPBWtmVgeH1bW0d1Lw5FNAb0+jtcAcScdKmghMAh4hm/ZvUuo5\ndAzZTea1aYaojcAlaf/8vLJmZlYjFWsGku4COoCTJHUDi4EOSWeQNRPtAr4IEBHbJd1NNnn4IWBB\nRLyVjnM12QTho4CVEbE9fcW1wBpJ3wAeB1YMWenMzKwqFZNBRFxaIlz2D3ZE3AjcWCK+jmzi8OL4\nc2S9jczMrE48AtnMzJwMzMzMycDMzHAyMDMznAzMzAwnAzMzw8nAzMxwMjArKz1+/XFJP0rLJefe\nSCPuv5/imyVNyB1jUPN7mNXLkTy1dESZUOIBeLuWXFSHM7EaugZ4GjghLffOvbFG0rfJ5t5Ylt73\nR8Rpkuak7T5bNL/HB4GfSPpQOtZtwPlkT+p9VNLaiNhRq4KZFXPNwKwESeOBi4DvpuWB5t6YnZZJ\n689L2w9qfo/hL5VZea4ZmJX2d8BXgfek5YHm3hhHmq8jIg5JOpi2H+z8Hv0Uz+FRat6Inp4eFk55\nq0+sFeaXaNV5Mhq1XE4GZkUk/SmwNyK2Suqo57kUz+FRat6IQqHAzQ+/1ic2XPNv1FKrzpPRqOVy\nMjDr72PAn0uaBbyL7J7BrZSfe6N3Ho9uSaOB95LN1THY+T3M6sb3DMyKRMR1ETE+IiaQ3QB+KCIu\no/zcG2vTMmn9Q2mujkHN71GDopmV5ZqBWfXKzb2xArhDUhewj+yP++HO72FWF04GZgOIiAJQSJ9L\nzr0REa8Dny6z/6Dm9zCrFzcTmZmZk4GZmTkZmJkZTgZmZkYVyUDSSkl7JT2Vi50oqVPSzvQ+NsUl\naWl6+NYTks7K7TMvbb9T0rxc/GxJT6Z9lqZh/GZmVkPV1AxuB2YWxRYBGyJiErAhLQNcSNaXehLZ\nEPplkCUPYDHZkPtzgMW9CSRt84XcfsXfZWaDNGHRA31eZpVUTAYR8VOyvtN5+QdzFT+wa3VkNpGN\n2DwFuADojIh9EbEf6ARmpnUnRMSmNEhnde5YZmZWI4c7zqA9Ivakzy8C7enz2w/sSnofzDVQvLtE\nvKTih3aVeuDTwimHSuw5PIbiYVON+tCqwWiFMpiNdEc86CwiQlIMxclU8V19HtrV1tbW74FPl9ew\nSjwUDwNr1IdWDUYrlMFspDvc3kQvpSYe0vveFC/3YK6B4uNLxM3MrIYONxnkH8xV/MCuualX0XTg\nYGpOWg/MkDQ23TieAaxP616VND31IpqbO5aZmdVIxWYiSXcBHcBJkrrJegUtAe6WdCXwS+AzafN1\nwCyyGZ1+C1wBEBH7JN1A9rRGgOsjovem9FVkPZaOAx5MLzMzq6GKySAiLi2z6rwS2wawoMxxVgIr\nS8S3AKdXOg8zMxs+HoFsZmZOBmZm5mRgZmY4GZiZGU4GZmaGp708IsUPANu15KI6nYmZ2ZFxzcDM\nzJwMzMzMycDMzHAyMDMznAzMzAwnAzMzw8nAzMxwMjAzM5wMzPqRdKqkjZJ2SNou6ZoUP1FSp6Sd\n6X1sikvSUkldkp6QdFbuWPPS9jslzcvFz5b0ZNpnaZrcyaxunAzM+jsELIyIycB0YIGkycAiYENE\nTAI2pGWAC4FJ6TUfWAZZ8iCbDGoacA6wuDeBpG2+kNtvZg3KZVaWk4FZkYjYExGPpc+/AZ4GxgGz\ngVVps1XAxenzbGB1ZDYBY9Lc4BcAnRGxLyL2A53AzLTuhIjYlCaEWp07llld+NlEZgOQNAE4E9gM\ntKd5uwFeBNrT53HAC7ndulNsoHh3iXip759PVtugvb2dQqHQb5uenh4WTnlrwHKU2q/R9fT0NOV5\nV9Ko5XIyMCtDUhvwA+DLEfFqvlk/IkJSDPc5RMRyYDnA1KlTo6Ojo982hUKBmx9+bcDj7Lqs/36N\nrlAoUKq8za5Ry+VmIrMSJB1NlgjujIh7U/il1MRDet+b4ruBU3O7j0+xgeLjS8TN6uaIkoGkXalH\nxDZJW1JsyHpcmNVD6tmzAng6Im7JrVoL9F6f84D7c/G56RqfDhxMzUnrgRmSxqbfgxnA+rTuVUnT\n03fNzR3LrC6GopnoExHxcm65t8fFEkmL0vK19O1xMY2sN8W0XI+LqUAAWyWtTTfcmkrx/AbgOQ6a\n1MeAzwFPStqWYl8DlgB3S7oS+CXwmbRuHTAL6AJ+C1wBEBH7JN0APJq2uz4i9qXPVwG3A8cBD6aX\nWd0Mxz2D2UBH+rwKKJAlg7d7XACbJPX2uOgg9bgAkNRJ1s3urmE4N7OKIuJhoFy///NKbB/AgjLH\nWgmsLBHfApx+BKdpNqSO9J5BAD+WtDX1eoCh63FhZmY1cqQ1g49HxG5J7wc6JT2TXznUPS6Ku9mV\n6qK1cMqhofq6IVGpC1mjdjMbjFYog9lId0TJICJ2p/e9ku4jG2X5kqRTImLPIHpcdBTFC2W+r083\nu7a2tn5dtC4v0W5fT5W69DVqN7PBaIUymI10h91MJOl4Se/p/UzWU+IphqjHxeGel5mZDd6R1Aza\ngfvSQJzRwPci4l8kPcrQ9bgwM7MaOOxkEBHPAR8tEX+FIepxYWZmteERyGZm5mRgZmZOBmZmhp9a\nOuyKH1Hhx1OYWSNyzcDMzJwMzMzMycDMzHAyMDMzfAPZbETwXBtWiWsGZmbmZGBmZm4mqrsndx/s\n99htV9/NrNZcMzAzMycDMzNzMjAzM5wMzMwM30BuSH64nZnVmmsGZmbmZGBmZm4magp+lIANBzdH\nWp5rBmZm1jg1A0kzgVuBUcB3I2JJnU+pofm/uubna94aSUMkA0mjgNuA84Fu4FFJayNiR33PrHk4\nOTSXRrzm3Rw5sjVEMgDOAboi4jkASWuA2YCTwWEq9YtdDf/y10xTXPOVriNfL62jUZLBOOCF3HI3\nMK14I0nzgflpsecTn/jEK8DLw396Q0c39QudRAOVocT5VaOhyjAIv1/H7z6sa17SsyWOVbef/2Fe\nL9Vq1uuqknqXq+R13yjJoCoRsRxY3rssaUtETK3jKR0xl8EGUnzNl9KqP3+Xq7YapTfRbuDU3PL4\nFDNrVb7mraE0SjJ4FJgkaaKkY4A5wNo6n5PZcPI1bw2lIZqJIuKQpKuB9WTd7FZGxPYqdh2w+twk\nXIYR6Aiu+VJa9efvctWQIqLe52BmZnXWKM1EZmZWR04GZmbWvMlA0kxJz0rqkrSo3udTDUkrJe2V\n9FQudqKkTkk70/vYep5jJZJOlbRR0g5J2yVdk+JNVY5W0Yy/B70k7ZL0pKRtkrakWMnrSJmlqZxP\nSDqrvmf/jsH8Xg9UDknz0vY7Jc2rdTmaMhnkhvJfCEwGLpU0ub5nVZXbgZlFsUXAhoiYBGxIy43s\nELAwIiYD04EF6WffbOVoek38e5D3iYg4I9fvvtx1dCEwKb3mA8tqfqbl3U71v9clyyHpRGAx2cDD\nc4DFtf6HqimTAbmh/BHxJtA7lL+hRcRPgX1F4dnAqvR5FXBxTU9qkCJiT0Q8lj7/BniabDRtU5Wj\nRTTl70EF5a6j2cDqyGwCxpt5BOAAAAGpSURBVEg6pR4nWGyQv9flynEB0BkR+yJiP9BJ/wQzrJo1\nGZQayj+uTudypNojYk/6/CLQXs+TGQxJE4Azgc00cTmaWLP/HgTwY0lb02M3oPx11GxlHWw56l6+\nhhhnYJmICElN0ddXUhvwA+DLEfGqpLfXNVM5rK4+HhG7Jb0f6JT0TH5lq1xHzVKOZq0ZtNJQ/pd6\nq7vpfW+dz6ciSUeTJYI7I+LeFG66crSApv49iIjd6X0vcB9Zs1e566jZyjrYctS9fM2aDFppKP9a\noLfnwDzg/jqeS0XKqgArgKcj4pbcqqYqR4to2t8DScdLek/vZ2AG8BTlr6O1wNzUG2c6cDDXDNOI\nBluO9cAMSWPTjeMZKVY7EdGUL2AW8K/AL4C/rvf5VHnOdwF7gH8jaxO8EngfWW+DncBPgBPrfZ4V\nyvBxsrbeJ4Bt6TWr2crRKq9m/D1I5/0HwM/Ta3vvuZe7jgCR9Zz6BfAkMLXeZciVperf64HKAXwe\n6EqvK2pdDj+OwszMmraZyMzMhpCTgZmZORmYmZmTgZmZ4WRgZmY4GZiZGU4GZmYG/H98XXVbkrrm\nqwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "LXk5iBkcDtX1",
        "colab_type": "code",
        "outputId": "6814f5cd-8c63-4782-9906-3f585a1b29ea",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "cnt=0\n",
        "for i in data['cleaned_summary']:\n",
        "    if(len(i.split())<=8):\n",
        "        cnt=cnt+1\n",
        "print(cnt/len(data['cleaned_summary']))"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.9439564829896432\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RKadDFfTDwRm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "max_text_len=30\n",
        "max_summary_len=8"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XYZA7JAKDyx3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "cleaned_text =np.array(data['cleaned_text'])\n",
        "cleaned_summary=np.array(data['cleaned_summary'])\n",
        "\n",
        "short_text=[]\n",
        "short_summary=[]\n",
        "\n",
        "for i in range(len(cleaned_text)):\n",
        "    if(len(cleaned_summary[i].split())<=max_summary_len and len(cleaned_text[i].split())<=max_text_len):\n",
        "        short_text.append(cleaned_text[i])\n",
        "        short_summary.append(cleaned_summary[i])\n",
        "        \n",
        "df=pd.DataFrame({'text':short_text,'summary':short_summary})"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "BrewAeAJD2Ar",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "df['summary'] = df['summary'].apply(lambda x : 'sostok '+ x + ' eostok')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H78d1_yzDusL",
        "colab_type": "text"
      },
      "source": [
        "## train test split"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uCYN90gJD45C",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "x_tr,x_val,y_tr,y_val=train_test_split(np.array(df['text']),np.array(df['summary']),test_size=0.1,random_state=0,shuffle=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OB_Nf93uD27i",
        "colab_type": "text"
      },
      "source": [
        "# Preparing the Tokenizer\n",
        "A tokenizer builds the vocabulary and converts a word sequence to an integer sequence. Go ahead and build tokenizers for text and summary:\n",
        "\n",
        "# Text Tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0zweZZ9OD7ow",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from keras.preprocessing.text import Tokenizer \n",
        "from keras.preprocessing.sequence import pad_sequences\n",
        "\n",
        "#prepare a tokenizer for reviews on training data\n",
        "x_tokenizer = Tokenizer() \n",
        "x_tokenizer.fit_on_texts(list(x_tr))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8W88FtEGD1l_",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "## Rarewords and its Coverage\n",
        "Let us look at the proportion rare words and its total coverage in the entire text\n",
        "\n",
        "Here, I am defining the threshold to be 4 which means word whose count is below 4 is considered as a rare word\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uyGG0euiD_N1",
        "colab_type": "code",
        "outputId": "00f3ec31-2745-4b33-812a-a1ba2fcd7f15",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "thresh=4\n",
        "\n",
        "cnt=0\n",
        "tot_cnt=0\n",
        "freq=0\n",
        "tot_freq=0\n",
        "\n",
        "for key,value in x_tokenizer.word_counts.items():\n",
        "    tot_cnt=tot_cnt+1\n",
        "    tot_freq=tot_freq+value\n",
        "    if(value<thresh):\n",
        "        cnt=cnt+1\n",
        "        freq=freq+value\n",
        "    \n",
        "print(\"% of rare words in vocabulary:\",(cnt/tot_cnt)*100)\n",
        "print(\"Total Coverage of rare words:\",(freq/tot_freq)*100)"
      ],
      "execution_count": 21,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "% of rare words in vocabulary: 66.73746690331403\n",
            "Total Coverage of rare words: 2.1848453887258286\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "P6O9yV5WEGIS",
        "colab_type": "text"
      },
      "source": [
        "## tot_cnt gives the size of vocabulary (which means every unique words in the text)\n",
        "\n",
        "* cnt gives me the no. of rare words whose count falls below threshold\n",
        "\n",
        "* tot_cnt - cnt gives me the top most common words\n",
        "\n",
        "* Let us define the tokenizer with top most common words for reviews."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KiM2gAoyECoD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#prepare a tokenizer for reviews on training data\n",
        "x_tokenizer = Tokenizer(num_words=tot_cnt-cnt) \n",
        "x_tokenizer.fit_on_texts(list(x_tr))\n",
        "\n",
        "#convert text sequences into integer sequences\n",
        "x_tr_seq    =   x_tokenizer.texts_to_sequences(x_tr) \n",
        "x_val_seq   =   x_tokenizer.texts_to_sequences(x_val)\n",
        "\n",
        "#padding zero upto maximum length\n",
        "x_tr    =   pad_sequences(x_tr_seq,  maxlen=max_text_len, padding='post')\n",
        "x_val   =   pad_sequences(x_val_seq, maxlen=max_text_len, padding='post')\n",
        "\n",
        "#size of vocabulary ( +1 for padding token)\n",
        "x_voc   =  x_tokenizer.num_words + 1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4mY8uDKOENxU",
        "colab_type": "code",
        "outputId": "cb75492b-e0ea-4883-fdc1-d71c96334a68",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "x_voc"
      ],
      "execution_count": 23,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "11433"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 23
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q0tW0BCpEQ4s",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#x_tr_seq"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S7qaVxpoEamy",
        "colab_type": "text"
      },
      "source": [
        "## summary tokenizer"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "82GZnIbKEFDn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#prepare a tokenizer for reviews on training data\n",
        "y_tokenizer = Tokenizer()   \n",
        "y_tokenizer.fit_on_texts(list(y_tr))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6Lm2-XuaEgeJ",
        "colab_type": "text"
      },
      "source": [
        "# Rarewords and its Coverage\n",
        "Let us look at the proportion rare words and its total coverage in the entire summary\n",
        "\n",
        "Here, I am defining the threshold to be 6 which means word whose count is below 6 is considered as a rare word"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Nf7M3NvPEIfK",
        "colab_type": "code",
        "outputId": "0b4107ab-c819-45d2-c270-4046bf160681",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "thresh=6\n",
        "\n",
        "cnt=0\n",
        "tot_cnt=0\n",
        "freq=0\n",
        "tot_freq=0\n",
        "\n",
        "for key,value in y_tokenizer.word_counts.items():\n",
        "    tot_cnt=tot_cnt+1\n",
        "    tot_freq=tot_freq+value\n",
        "    if(value<thresh):\n",
        "        cnt=cnt+1\n",
        "        freq=freq+value\n",
        "    \n",
        "print(\"% of rare words in vocabulary:\",(cnt/tot_cnt)*100)\n",
        "print(\"Total Coverage of rare words:\",(freq/tot_freq)*100)"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "% of rare words in vocabulary: 76.95611432141185\n",
            "Total Coverage of rare words: 3.969070293687\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2UZLHGZxEa8C",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#prepare a tokenizer for reviews on training data\n",
        "y_tokenizer = Tokenizer(num_words=tot_cnt-cnt) \n",
        "y_tokenizer.fit_on_texts(list(y_tr))\n",
        "\n",
        "#convert text sequences into integer sequences\n",
        "y_tr_seq    =   y_tokenizer.texts_to_sequences(y_tr) \n",
        "y_val_seq   =   y_tokenizer.texts_to_sequences(y_val) \n",
        "\n",
        "#padding zero upto maximum length\n",
        "y_tr    =   pad_sequences(y_tr_seq, maxlen=max_summary_len, padding='post')\n",
        "y_val   =   pad_sequences(y_val_seq, maxlen=max_summary_len, padding='post')\n",
        "\n",
        "#size of vocabulary\n",
        "y_voc  =   y_tokenizer.num_words +1"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IES-ifU0EiAW",
        "colab_type": "code",
        "outputId": "911abf32-bc84-47ea-b89f-ad3d07f82d6d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "y_tokenizer.word_counts['sostok'],len(y_tr)"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(78858, 78858)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ZxL4zwE4ElBo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ind=[]\n",
        "for i in range(len(y_tr)):\n",
        "    cnt=0\n",
        "    for j in y_tr[i]:\n",
        "        if j!=0:\n",
        "            cnt=cnt+1\n",
        "    if(cnt==2):\n",
        "        ind.append(i)\n",
        "\n",
        "y_tr=np.delete(y_tr,ind, axis=0)\n",
        "x_tr=np.delete(x_tr,ind, axis=0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gapHEGP9Epxz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "ind=[]\n",
        "for i in range(len(y_val)):\n",
        "    cnt=0\n",
        "    for j in y_val[i]:\n",
        "        if j!=0:\n",
        "            cnt=cnt+1\n",
        "    if(cnt==2):\n",
        "        ind.append(i)\n",
        "\n",
        "y_val=np.delete(y_val,ind, axis=0)\n",
        "x_val=np.delete(x_val,ind, axis=0)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TyodtFN4E8mr",
        "colab_type": "text"
      },
      "source": [
        "## Attention Layer:-"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8yQ0ElSQEy2P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "import os\n",
        "from tensorflow.python.keras.layers import Layer\n",
        "from tensorflow.python.keras import backend as K\n",
        "\n",
        "\n",
        "class AttentionLayer(Layer):\n",
        "    \"\"\"\n",
        "    This class implements Bahdanau attention (https://arxiv.org/pdf/1409.0473.pdf).\n",
        "    There are three sets of weights introduced W_a, U_a, and V_a\n",
        "     \"\"\"\n",
        "\n",
        "    def __init__(self, **kwargs):\n",
        "        super(AttentionLayer, self).__init__(**kwargs)\n",
        "\n",
        "    def build(self, input_shape):\n",
        "        assert isinstance(input_shape, list)\n",
        "        # Create a trainable weight variable for this layer.\n",
        "\n",
        "        self.W_a = self.add_weight(name='W_a',\n",
        "                                   shape=tf.TensorShape((input_shape[0][2], input_shape[0][2])),\n",
        "                                   initializer='uniform',\n",
        "                                   trainable=True)\n",
        "        self.U_a = self.add_weight(name='U_a',\n",
        "                                   shape=tf.TensorShape((input_shape[1][2], input_shape[0][2])),\n",
        "                                   initializer='uniform',\n",
        "                                   trainable=True)\n",
        "        self.V_a = self.add_weight(name='V_a',\n",
        "                                   shape=tf.TensorShape((input_shape[0][2], 1)),\n",
        "                                   initializer='uniform',\n",
        "                                   trainable=True)\n",
        "\n",
        "        super(AttentionLayer, self).build(input_shape)  # Be sure to call this at the end\n",
        "\n",
        "    def call(self, inputs, verbose=False):\n",
        "        \"\"\"\n",
        "        inputs: [encoder_output_sequence, decoder_output_sequence]\n",
        "        \"\"\"\n",
        "        assert type(inputs) == list\n",
        "        encoder_out_seq, decoder_out_seq = inputs\n",
        "        if verbose:\n",
        "            print('encoder_out_seq>', encoder_out_seq.shape)\n",
        "            print('decoder_out_seq>', decoder_out_seq.shape)\n",
        "\n",
        "        def energy_step(inputs, states):\n",
        "            \"\"\" Step function for computing energy for a single decoder state \"\"\"\n",
        "\n",
        "            assert_msg = \"States must be a list. However states {} is of type {}\".format(states, type(states))\n",
        "            assert isinstance(states, list) or isinstance(states, tuple), assert_msg\n",
        "\n",
        "            \"\"\" Some parameters required for shaping tensors\"\"\"\n",
        "            en_seq_len, en_hidden = encoder_out_seq.shape[1], encoder_out_seq.shape[2]\n",
        "            de_hidden = inputs.shape[-1]\n",
        "\n",
        "            \"\"\" Computing S.Wa where S=[s0, s1, ..., si]\"\"\"\n",
        "            # <= batch_size*en_seq_len, latent_dim\n",
        "            reshaped_enc_outputs = K.reshape(encoder_out_seq, (-1, en_hidden))\n",
        "            # <= batch_size*en_seq_len, latent_dim\n",
        "            W_a_dot_s = K.reshape(K.dot(reshaped_enc_outputs, self.W_a), (-1, en_seq_len, en_hidden))\n",
        "            if verbose:\n",
        "                print('wa.s>',W_a_dot_s.shape)\n",
        "\n",
        "            \"\"\" Computing hj.Ua \"\"\"\n",
        "            U_a_dot_h = K.expand_dims(K.dot(inputs, self.U_a), 1)  # <= batch_size, 1, latent_dim\n",
        "            if verbose:\n",
        "                print('Ua.h>',U_a_dot_h.shape)\n",
        "\n",
        "            \"\"\" tanh(S.Wa + hj.Ua) \"\"\"\n",
        "            # <= batch_size*en_seq_len, latent_dim\n",
        "            reshaped_Ws_plus_Uh = K.tanh(K.reshape(W_a_dot_s + U_a_dot_h, (-1, en_hidden)))\n",
        "            if verbose:\n",
        "                print('Ws+Uh>', reshaped_Ws_plus_Uh.shape)\n",
        "\n",
        "            \"\"\" softmax(va.tanh(S.Wa + hj.Ua)) \"\"\"\n",
        "            # <= batch_size, en_seq_len\n",
        "            e_i = K.reshape(K.dot(reshaped_Ws_plus_Uh, self.V_a), (-1, en_seq_len))\n",
        "            # <= batch_size, en_seq_len\n",
        "            e_i = K.softmax(e_i)\n",
        "\n",
        "            if verbose:\n",
        "                print('ei>', e_i.shape)\n",
        "\n",
        "            return e_i, [e_i]\n",
        "\n",
        "        def context_step(inputs, states):\n",
        "            \"\"\" Step function for computing ci using ei \"\"\"\n",
        "            # <= batch_size, hidden_size\n",
        "            c_i = K.sum(encoder_out_seq * K.expand_dims(inputs, -1), axis=1)\n",
        "            if verbose:\n",
        "                print('ci>', c_i.shape)\n",
        "            return c_i, [c_i]\n",
        "\n",
        "        def create_inital_state(inputs, hidden_size):\n",
        "            # We are not using initial states, but need to pass something to K.rnn funciton\n",
        "            fake_state = K.zeros_like(inputs)  # <= (batch_size, enc_seq_len, latent_dim\n",
        "            fake_state = K.sum(fake_state, axis=[1, 2])  # <= (batch_size)\n",
        "            fake_state = K.expand_dims(fake_state)  # <= (batch_size, 1)\n",
        "            fake_state = K.tile(fake_state, [1, hidden_size])  # <= (batch_size, latent_dim\n",
        "            return fake_state\n",
        "\n",
        "        fake_state_c = create_inital_state(encoder_out_seq, encoder_out_seq.shape[-1])\n",
        "        fake_state_e = create_inital_state(encoder_out_seq, encoder_out_seq.shape[1])  # <= (batch_size, enc_seq_len, latent_dim\n",
        "\n",
        "        \"\"\" Computing energy outputs \"\"\"\n",
        "        # e_outputs => (batch_size, de_seq_len, en_seq_len)\n",
        "        last_out, e_outputs, _ = K.rnn(\n",
        "            energy_step, decoder_out_seq, [fake_state_e],\n",
        "        )\n",
        "\n",
        "        \"\"\" Computing context vectors \"\"\"\n",
        "        last_out, c_outputs, _ = K.rnn(\n",
        "            context_step, e_outputs, [fake_state_c],\n",
        "        )\n",
        "\n",
        "        return c_outputs, e_outputs\n",
        "\n",
        "    def compute_output_shape(self, input_shape):\n",
        "        \"\"\" Outputs produced by the layer \"\"\"\n",
        "        return [\n",
        "            tf.TensorShape((input_shape[1][0], input_shape[1][1], input_shape[1][2])),\n",
        "            tf.TensorShape((input_shape[1][0], input_shape[1][1], input_shape[0][1]))\n",
        "        ]\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8S09mmA6EpAY",
        "colab_type": "text"
      },
      "source": [
        "# Model building\n",
        "We are finally at the model building part. But before we do that, we need to familiarize ourselves with a few terms which are required prior to building the model.\n",
        "\n",
        "* Return Sequences = True: When the return sequences parameter is set to True, LSTM produces the hidden state and cell state for every timestep\n",
        "\n",
        "* Return State = True: When return state = True, LSTM produces the hidden state and cell state of the last timestep only\n",
        "\n",
        "* Initial State: This is used to initialize the internal states of the LSTM for the first timestep\n",
        "\n",
        "* Stacked LSTM: Stacked LSTM has multiple layers of LSTM stacked on top of each other. This leads to a better representation of the sequence. I encourage you to experiment with the multiple layers of the LSTM stacked on top of each other (it’s a great way to learn this)\n",
        "\n",
        "Here, we are building a 3 stacked LSTM for the encoder:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dWejpYocEsEk",
        "colab_type": "code",
        "outputId": "f9d49523-4401-4697-84b7-dc52fc9b2270",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 853
        }
      },
      "source": [
        "from keras import backend as K \n",
        "K.clear_session()\n",
        "\n",
        "latent_dim = 300\n",
        "embedding_dim=100\n",
        "\n",
        "# Encoder\n",
        "encoder_inputs = Input(shape=(max_text_len,))\n",
        "\n",
        "#embedding layer\n",
        "enc_emb =  Embedding(x_voc, embedding_dim,trainable=True)(encoder_inputs)\n",
        "\n",
        "#encoder lstm 1\n",
        "encoder_lstm1 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)\n",
        "encoder_output1, state_h1, state_c1 = encoder_lstm1(enc_emb)\n",
        "\n",
        "#encoder lstm 2\n",
        "encoder_lstm2 = LSTM(latent_dim,return_sequences=True,return_state=True,dropout=0.4,recurrent_dropout=0.4)\n",
        "encoder_output2, state_h2, state_c2 = encoder_lstm2(encoder_output1)\n",
        "\n",
        "#encoder lstm 3\n",
        "encoder_lstm3=LSTM(latent_dim, return_state=True, return_sequences=True,dropout=0.4,recurrent_dropout=0.4)\n",
        "encoder_outputs, state_h, state_c= encoder_lstm3(encoder_output2)\n",
        "\n",
        "# Set up the decoder, using `encoder_states` as initial state.\n",
        "decoder_inputs = Input(shape=(None,))\n",
        "\n",
        "#embedding layer\n",
        "dec_emb_layer = Embedding(y_voc, embedding_dim,trainable=True)\n",
        "dec_emb = dec_emb_layer(decoder_inputs)\n",
        "\n",
        "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True,dropout=0.4,recurrent_dropout=0.2)\n",
        "decoder_outputs,decoder_fwd_state, decoder_back_state = decoder_lstm(dec_emb,initial_state=[state_h, state_c])\n",
        "\n",
        "# Attention layer\n",
        "attn_layer = AttentionLayer(name='attention_layer')\n",
        "attn_out, attn_states = attn_layer([encoder_outputs, decoder_outputs])\n",
        "\n",
        "# Concat attention input and decoder LSTM output\n",
        "decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([decoder_outputs, attn_out])\n",
        "\n",
        "#dense layer\n",
        "decoder_dense =  TimeDistributed(Dense(y_voc, activation='softmax'))\n",
        "decoder_outputs = decoder_dense(decoder_concat_input)\n",
        "\n",
        "# Define the model \n",
        "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
        "\n",
        "model.summary()"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:107: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
            "\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:111: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.\n",
            "\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:66: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
            "\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/keras/initializers.py:119: calling RandomUniform.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/resource_variable_ops.py:1630: calling BaseResourceVariable.__init__ (from tensorflow.python.ops.resource_variable_ops) with constraint is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "If using Keras pass *_constraint arguments to layers.\n",
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.sparse_tensor_dense_matmul is deprecated. Please use tf.sparse.sparse_dense_matmul instead.\n",
            "\n",
            "Model: \"model\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_1 (InputLayer)            [(None, 30)]         0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding (Embedding)           (None, 30, 100)      1143300     input_1[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "lstm (LSTM)                     [(None, 30, 300), (N 481200      embedding[0][0]                  \n",
            "__________________________________________________________________________________________________\n",
            "input_2 (InputLayer)            [(None, None)]       0                                            \n",
            "__________________________________________________________________________________________________\n",
            "lstm_1 (LSTM)                   [(None, 30, 300), (N 721200      lstm[0][0]                       \n",
            "__________________________________________________________________________________________________\n",
            "embedding_1 (Embedding)         (None, None, 100)    295200      input_2[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "lstm_2 (LSTM)                   [(None, 30, 300), (N 721200      lstm_1[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "lstm_3 (LSTM)                   [(None, None, 300),  481200      embedding_1[0][0]                \n",
            "                                                                 lstm_2[0][1]                     \n",
            "                                                                 lstm_2[0][2]                     \n",
            "__________________________________________________________________________________________________\n",
            "attention_layer (AttentionLayer ((None, None, 300),  180300      lstm_2[0][0]                     \n",
            "                                                                 lstm_3[0][0]                     \n",
            "__________________________________________________________________________________________________\n",
            "concat_layer (Concatenate)      (None, None, 600)    0           lstm_3[0][0]                     \n",
            "                                                                 attention_layer[0][0]            \n",
            "__________________________________________________________________________________________________\n",
            "time_distributed (TimeDistribut (None, None, 2952)   1774152     concat_layer[0][0]               \n",
            "==================================================================================================\n",
            "Total params: 5,797,752\n",
            "Trainable params: 5,797,752\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rELOtraEFJQ6",
        "colab_type": "text"
      },
      "source": [
        "#### I am using sparse categorical cross-entropy as the loss function since it converts the integer sequence to a one-hot vector on the fly. This overcomes any memory issues."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zbrieMn7Ewfm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.compile(optimizer='rmsprop', loss='sparse_categorical_crossentropy',metrics = ['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uLBs5K0eFLWV",
        "colab_type": "text"
      },
      "source": [
        "#### Remember the concept of early stopping? It is used to stop training the neural network at the right time by monitoring a user-specified metric. Here, I am monitoring the validation loss (val_loss). Our model will stop training once the validation loss increases:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RBi9UbrkFSPU",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "es = EarlyStopping(monitor='val_loss', mode='min', verbose=1,patience=2)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WMstwzCdFUG-",
        "colab_type": "text"
      },
      "source": [
        "#### We’ll train the model on a batch size of 128 and validate it on the holdout set (which is 10% of our dataset):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ryYviI2VFVS3",
        "colab_type": "code",
        "outputId": "a6fa454d-270f-4534-9925-125f5dcb9e8f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 972
        }
      },
      "source": [
        "history=model.fit([x_tr,y_tr[:,:-1]], y_tr.reshape(y_tr.shape[0],y_tr.shape[1], 1)[:,1:] ,epochs=50,callbacks=[es],batch_size=128, validation_data=([x_val,y_val[:,:-1]], y_val.reshape(y_val.shape[0],y_val.shape[1], 1)[:,1:]))"
      ],
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/ops/math_grad.py:1424: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
            "Instructions for updating:\n",
            "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
            "Train on 77266 samples, validate on 8554 samples\n",
            "Epoch 1/50\n",
            "77266/77266 [==============================] - 101s 1ms/sample - loss: 2.8125 - acc: 0.5773 - val_loss: 2.5445 - val_acc: 0.6024\n",
            "Epoch 2/50\n",
            "77266/77266 [==============================] - 98s 1ms/sample - loss: 2.4757 - acc: 0.6081 - val_loss: 2.3692 - val_acc: 0.6163\n",
            "Epoch 3/50\n",
            "77266/77266 [==============================] - 97s 1ms/sample - loss: 2.3426 - acc: 0.6199 - val_loss: 2.2789 - val_acc: 0.6238\n",
            "Epoch 4/50\n",
            "77266/77266 [==============================] - 97s 1ms/sample - loss: 2.2633 - acc: 0.6267 - val_loss: 2.2186 - val_acc: 0.6310\n",
            "Epoch 5/50\n",
            "77266/77266 [==============================] - 99s 1ms/sample - loss: 2.2067 - acc: 0.6317 - val_loss: 2.1810 - val_acc: 0.6344\n",
            "Epoch 6/50\n",
            "77266/77266 [==============================] - 98s 1ms/sample - loss: 2.1607 - acc: 0.6352 - val_loss: 2.1425 - val_acc: 0.6382\n",
            "Epoch 7/50\n",
            "77266/77266 [==============================] - 97s 1ms/sample - loss: 2.1226 - acc: 0.6385 - val_loss: 2.1191 - val_acc: 0.6410\n",
            "Epoch 8/50\n",
            "77266/77266 [==============================] - 97s 1ms/sample - loss: 2.0919 - acc: 0.6413 - val_loss: 2.1001 - val_acc: 0.6426\n",
            "Epoch 9/50\n",
            "77266/77266 [==============================] - 97s 1ms/sample - loss: 2.0649 - acc: 0.6437 - val_loss: 2.0878 - val_acc: 0.6440\n",
            "Epoch 10/50\n",
            "77266/77266 [==============================] - 97s 1ms/sample - loss: 2.0401 - acc: 0.6455 - val_loss: 2.0718 - val_acc: 0.6446\n",
            "Epoch 11/50\n",
            "77266/77266 [==============================] - 97s 1ms/sample - loss: 2.0173 - acc: 0.6475 - val_loss: 2.0617 - val_acc: 0.6456\n",
            "Epoch 12/50\n",
            "77266/77266 [==============================] - 96s 1ms/sample - loss: 1.9971 - acc: 0.6494 - val_loss: 2.0602 - val_acc: 0.6466\n",
            "Epoch 13/50\n",
            "77266/77266 [==============================] - 96s 1ms/sample - loss: 1.9793 - acc: 0.6510 - val_loss: 2.0426 - val_acc: 0.6483\n",
            "Epoch 14/50\n",
            "77266/77266 [==============================] - 96s 1ms/sample - loss: 1.9617 - acc: 0.6522 - val_loss: 2.0373 - val_acc: 0.6480\n",
            "Epoch 15/50\n",
            "77266/77266 [==============================] - 96s 1ms/sample - loss: 1.9448 - acc: 0.6537 - val_loss: 2.0335 - val_acc: 0.6479\n",
            "Epoch 16/50\n",
            "77266/77266 [==============================] - 95s 1ms/sample - loss: 1.9280 - acc: 0.6549 - val_loss: 2.0253 - val_acc: 0.6486\n",
            "Epoch 17/50\n",
            "77266/77266 [==============================] - 96s 1ms/sample - loss: 1.9126 - acc: 0.6559 - val_loss: 2.0152 - val_acc: 0.6497\n",
            "Epoch 18/50\n",
            "77266/77266 [==============================] - 95s 1ms/sample - loss: 1.9015 - acc: 0.6573 - val_loss: 2.0104 - val_acc: 0.6520\n",
            "Epoch 19/50\n",
            "77266/77266 [==============================] - 95s 1ms/sample - loss: 1.8911 - acc: 0.6579 - val_loss: 2.0131 - val_acc: 0.6507\n",
            "Epoch 20/50\n",
            "77266/77266 [==============================] - 95s 1ms/sample - loss: 1.8799 - acc: 0.6592 - val_loss: 2.0104 - val_acc: 0.6509\n",
            "Epoch 21/50\n",
            "77266/77266 [==============================] - 99s 1ms/sample - loss: 1.8715 - acc: 0.6596 - val_loss: 2.0140 - val_acc: 0.6500\n",
            "Epoch 22/50\n",
            "77266/77266 [==============================] - 104s 1ms/sample - loss: 1.8613 - acc: 0.6604 - val_loss: 2.0061 - val_acc: 0.6513\n",
            "Epoch 23/50\n",
            "77266/77266 [==============================] - 101s 1ms/sample - loss: 1.8497 - acc: 0.6619 - val_loss: 2.0002 - val_acc: 0.6515\n",
            "Epoch 24/50\n",
            "77266/77266 [==============================] - 99s 1ms/sample - loss: 1.8410 - acc: 0.6623 - val_loss: 2.0083 - val_acc: 0.6499\n",
            "Epoch 25/50\n",
            "77266/77266 [==============================] - 99s 1ms/sample - loss: 1.8332 - acc: 0.6631 - val_loss: 2.0055 - val_acc: 0.6509\n",
            "Epoch 00025: early stopping\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "NV0tUpgFFcav",
        "colab_type": "text"
      },
      "source": [
        "# Understanding the Diagnostic plot\n",
        "#### Now, we will plot a few diagnostic plots to understand the behavior of the model over time:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iNhrcWR-FcVR",
        "colab_type": "code",
        "outputId": "efa2d982-1dbd-454f-974d-d279906e2cbd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 265
        }
      },
      "source": [
        "from matplotlib import pyplot\n",
        "pyplot.plot(history.history['loss'], label='train')\n",
        "pyplot.plot(history.history['val_loss'], label='test')\n",
        "pyplot.legend()\n",
        "pyplot.show()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3deXzU1b3/8ddnksm+75AEEtYkgICE\nRUQEAUFtUWur1brWSm211bY/b62tt+3ttbXtrbe11irurVxbi1itC5tlcwFMAgok7CQQyL6QfT+/\nP74TlpBlApNMMvk8H488ZpLv+c58wvLOyfme7zlijEEppdTgZ3N3AUoppVxDA10ppTyEBrpSSnkI\nDXSllPIQGuhKKeUhvN31xlFRUSYpKcldb6+UUoNSZmZmqTEmurNjbgv0pKQkMjIy3PX2Sik1KIlI\nXlfHdMhFKaU8hAa6Ukp5CA10pZTyEG4bQ1dKqfPR3NxMfn4+DQ0N7i6lT/n5+ZGQkIDdbnf6HA10\npdSgkp+fT3BwMElJSYiIu8vpE8YYysrKyM/PJzk52enzdMhFKTWoNDQ0EBkZ6bFhDiAiREZG9vq3\nEA10pdSg48lh3u58vsdBF+j7i6r5xTvZNLa0ursUpZQaUAZdoOdX1PHCh0fYerjc3aUopYagyspK\nnn766V6fd/XVV1NZWdkHFZ026AJ99ugo/O1erM8ucncpSqkhqKtAb2lp6fa89957j7CwsL4qC3Ai\n0EUkUUQ2iEi2iOwRkQc6aRMqIv8Skc8cbe7qm3LBz+7F3HFRrM8pQndbUkr1t4cffphDhw4xZcoU\npk+fzmWXXcbSpUtJS0sD4LrrrmPatGlMmDCB5cuXnzovKSmJ0tJScnNzSU1N5Z577mHChAlceeWV\n1NfXu6Q2Z6YttgA/MMZkiUgwkCki64wx2We0uQ/INsZ8UUSigX0issIY0+SSKjtYlBbHmj1F7DlR\nxcT40L54C6XUIPDzf+0h+0SVS18zbXgIP/3ihC6PP/744+zevZudO3eyceNGrrnmGnbv3n1qeuGL\nL75IREQE9fX1TJ8+nRtuuIHIyMizXuPAgQO89tprPPfcc9x444288cYb3HrrrRdce489dGNMgTEm\ny/G8GsgB4js2A4LFuiwbBJRj/SDoE/PHR2MTWKvDLkopN5sxY8ZZc8WffPJJJk+ezKxZszh27BgH\nDhw455zk5GSmTJkCwLRp08jNzXVJLb26sUhEkoCpwLYOh54C3gZOAMHATcaYtk7OXwYsAxgxYkTv\nq3WIDPJl2shw1mcX8f1F4877dZRSg1t3Pen+EhgYeOr5xo0bWb9+PZ988gkBAQHMmzev07nkvr6+\np557eXm5bMjF6YuiIhIEvAE8aIzp+DvOYmAnMByYAjwlIiEdX8MYs9wYk26MSY+O7nQ5X6ctSosl\nu6CK45Wu+YNQSilnBAcHU11d3emxkydPEh4eTkBAAHv37mXr1q39WptTgS4idqwwX2GMWdVJk7uA\nVcZyEDgCpLiuzHMtTI0F0NkuSql+FRkZyaWXXsrEiRN56KGHzjq2ZMkSWlpaSE1N5eGHH2bWrFn9\nWpv0NFPEMS7+ClBujHmwizZ/BoqMMT8TkVggC5hsjCnt6nXT09PNhW5wseB3Gxke5s9f7555Qa+j\nlBo8cnJySE1NdXcZ/aKz71VEMo0x6Z21d2YM/VLgNmCXiOx0fO0RYASAMeYZ4BfAyyKyCxDgh92F\nuassTIvlxQ+PUNXQTIif8yuSKaWUJ+ox0I0xH2KFdHdtTgBXuqooZy1KjeXZTYfZtK+EL04e3t9v\nr5RSA8qgu1P0TFNHhBMZ6MP6HB1HV0qpQR3oXjbhipQYNuwtprn1nFmSSik1pAzqQAdrHL2qoYVP\nj+hiXUqpoW3QB/plY6Pw9bbpXaNKqSFv0Ad6gI83c8boYl1Kqf5xvsvnAvz+97+nrq7OxRWdNugD\nHay7RvMr6tlX1PndW0op5SoDOdA9YpPoK1JjAFi3p4iUuHNWHFBKKZc5c/ncRYsWERMTw+uvv05j\nYyPXX389P//5z6mtreXGG28kPz+f1tZWHn30UYqKijhx4gTz588nKiqKDRs2uLw2jwj0mGA/piSG\nsT6niO8sGOvucpRS/eX9h6Fwl2tfM24SXPV4l4fPXD537dq1rFy5ku3bt2OMYenSpWzevJmSkhKG\nDx/Ou+++C1hrvISGhvLEE0+wYcMGoqKiXFuzg0cMuYA17PJZ/kmKqnq3S7ZSSp2vtWvXsnbtWqZO\nncrFF1/M3r17OXDgAJMmTWLdunX88Ic/ZMuWLYSG9s++DR7RQwcr0H+7Zh/rc4r42syR7i5HKdUf\nuulJ9wdjDD/60Y/45je/ec6xrKws3nvvPX7yk5+wYMEC/vM//7PP6/GYHvrYmCBGRgbo6otKqT51\n5vK5ixcv5sUXX6SmpgaA48ePU1xczIkTJwgICODWW2/loYceIisr65xz+4LH9NBFhIWpsfx1ax61\njS0E+nrMt6aUGkDOXD73qquu4pZbbuGSSy4BICgoiFdffZWDBw/y0EMPYbPZsNvt/PnPfwZg2bJl\nLFmyhOHDh/fJRdEel8/tK65YPrejTw6VcfNzW3nm1otZMnGYS19bKTUw6PK5XS+f6zFDLgDTk8IJ\n9bfrXaNKqSHJowLd28t2arGuFl2sSyk1xHhUoIM126Wirpmso5XuLkUp1UeGwjIf5/M9elygzx0X\njY+XjXXZhe4uRSnVB/z8/CgrK/PoUDfGUFZWhp+fX6/O87ipIEG+3swaHcm67CIeuToVa0tUpZSn\nSEhIID8/n5KSEneX0qf8/PxISEjo1TkeF+hgDbs8+s/dHCqpYUxMsLvLUUq5kN1uJzk52d1lDEge\nN+QCsLB9sa7sYjdXopRS/ccjA31YqD8T40N0r1Gl1JDikYEOsCg1jqyjFZRUN7q7FKWU6hceG+gL\n02IwBjbs1WEXpdTQ4LGBnjYshPgwf71rVCk1ZHhsoFuLdcXw4cES6pta3V2OUkr1OY8NdICFabE0\nNLfx0cFSd5eilFJ9zqMDfWZyJMG+3qzTYRel1BDg0YHu423j8vHRfLC3iLY2z71NWCmlwMMDHay7\nRktrmthxTBfrUkp5No8P9HnjYvC2id5kpJTyeIMz0Gudv8gZGmBnRnKE7jWqlPJ4PQa6iCSKyAYR\nyRaRPSLyQBft5onITkebTa4v1eHzf8DvUqDskNOnLEqL5UBxDbmltX1WllJKuZszPfQW4AfGmDRg\nFnCfiKSd2UBEwoCngaXGmAnAV1xeabvky0AEtj7t9CkLU2MBdNhFKeXRegx0Y0yBMSbL8bwayAHi\nOzS7BVhljDnqaNd399sHx8FFN8KOFVBX7tQpiREBpMQF6/RFpZRH69UYuogkAVOBbR0OjQPCRWSj\niGSKyO1dnL9MRDJEJOOCFqe/5H5oqYdPX3D6lEVpsXyaW05FbdP5v69SSg1gTge6iAQBbwAPGmOq\nOhz2BqYB1wCLgUdFZFzH1zDGLDfGpBtj0qOjo8+/6phUGLMIti+H5ganTlmYGkubgQ37dLEupZRn\ncirQRcSOFeYrjDGrOmmSD6wxxtQaY0qBzcBk15XZidn3Q20x7HrdqeaT4kOJDfHVYRellMdyZpaL\nAC8AOcaYJ7po9hYwR0S8RSQAmIk11t53ki+HuEnw8VPQ1tZjc5tNWJAay6b9JTQ062JdSinP40wP\n/VLgNuAKx7TEnSJytYjcKyL3AhhjcoDVwOfAduB5Y8zuPqsarJkus78Lpfvg4HqnTlmUFktdUytb\nD5f1aWlKKeUOPW4SbYz5EBAn2v0W+K0rinLahOth/c/g4ydh3JU9Nr9kVCQBPl6syy5i3viYvq9P\nKaX60eC8U7Sdlx1m3gu5W+DEzh6b+9m9mDs2mvU5RRiji3UppTzL4A50gGl3gE8wfPKUU80XpcVS\nVNXIruMn+7gwpZTqX4M/0P1CrVDfvQoqj/XYfH5KDDZB13ZRSnmcwR/oYA27AGx7psemEYE+pI+M\nYF2OzkdXSnkWzwj0sESY+CXIfAUaeh5KWZQWS05BFcfK6/qhOKWU6h+eEehgLQfQVG2Feg8WplmL\ndX2gi3UppTyI5wT68CmQdJk17NLa3G3T5KhAxsQEsU4DXSnlQTwn0AFmfweqjsOeN3tsujA1lm2H\nyzlZ3334K6XUYOFZgT5mEUSNt2406mGe+aK0GFraDJv2X8Cqj0opNYB4VqDbbNaiXYW74MjmbptO\nSQwnKshHF+tSSnkMzwp0gEk3QmA0fPzHbpt52YQrUmLYuK+YppaeF/dSSqmBzvMC3e4HM74JB9dB\ncfcLPl41cRjVDS28ntHzDUlKKTXQeV6gA0y/G7z9e1wOYN74aGaPjuQ3q/dSUt3YT8UppVTf8MxA\nD4iAqV+Dz1+H6q7HyEWEX1w3kYbmNv773ex+LFAppVzPMwMdYNa3rfno25d322x0dBD3zhvNWztP\n8OGB0n4qTimlXM9zAz1yNKR+AT59Hppqu2367XmjSY4K5NG3dutuRkqpQctzAx3gku9AQyXsWNFt\nMz+7F7+4diJHSmt5euOhfipOKaVcy7MDfcRMSJgBW/8Ebd33vOeMjeLaKcN5ZuMhDpXU9FOBSinl\nOp4d6GDdaFSRC3vf6bHpT65Jw9du49F/7tYdjZRSg47nB3rKFyA8CT7ueUej6GBffrgkhY8PlfHm\njuN9X5tSSrmQ5we6zctaWjd/Oxzd1mPzW2aMYOqIMB57N4fKuqZ+KFAppVzD8wMdYMot4BdmLdrV\nA5tNeOy6SVTWN/Pr1Xv7oTillHKNoRHoPoEw/Ruw910o63kWS9rwEL5+aRKvbT9GZl55PxSolFIX\nbmgEOsCMZeBlh61PO9X8wYXjGB7qxyOrdtPcqot3KaUGvqET6MGxcNGN1pz0up573YG+3vxs6QT2\nFVXzwodH+qFApZS6MEMn0MG6ONpSD5++4FTzKyfEsSgtlt+v368bSiulBryhFegxqTD2Stj+LDQ3\nOHXKz5ZOwCbCz97eo3PTlVID2tAKdLB66bUlsOt1p5rHh/nzvYXj+GBvMWv2FPZxcUopdf6GXqAn\nz4W4i6wbjdqcu9h516VJpA4L4WdvZ1PT2NLHBSql1PkZeoEuArO/C6X7rF2NnODtZeOx6ydSVN3A\nE2v393GBSil1foZeoANMuA5C4nvcd/RMF48I55YZI3j54yPsPn6yD4tTSqnz02Ogi0iiiGwQkWwR\n2SMiD3TTdrqItIjIl11bpot52WHWtyB3C5zY4fRp/7E4hYhAH3785i5a2/QCqVJqYHGmh94C/MAY\nkwbMAu4TkbSOjUTEC/g1sNa1JfaRi28Hn2CnFu1qFxpg59EvpPFZ/klWbMvrw+KUUqr3egx0Y0yB\nMSbL8bwayAHiO2n6HeANoNilFfYVv1CYdgfseRMOrnf6tKWThzNnTBS/Xb2P4irnpj4qpVR/6NUY\nuogkAVOBbR2+Hg9cD/y5h/OXiUiGiGSUlJT0rtK+cOmDEJ0Cr34ZNv7aqVkv7RtLN7a28V/v6MbS\nSqmBw+lAF5EgrB74g8aYqg6Hfw/80BjTbSIaY5YbY9KNMenR0dG9r9bVgqLhG+vhoptg4y/h/250\nalmA5KhA7ps3hnc+L2DT/gHwg0kppXAy0EXEjhXmK4wxqzppkg78TURygS8DT4vIdS6rsi/5BMD1\nz8A1v4PDG+HZy526UHrvvFGMigrk0X/qxtJKqYHBmVkuArwA5BhjnuisjTEm2RiTZIxJAlYC3zbG\n/NOllfYlEWt53a+vAdMGLyyGzFe6PcXX24v/vm4iR8vreOrfB/upUKWU6pozPfRLgduAK0Rkp+Pj\nahG5V0Tu7eP6+lfCNPjmZhg5G/71XXjrPmiu77L57DFRXD81nmc3H+JgcXU/FqqUUucSdy04lZ6e\nbjIyMtzy3j1qa4WNj8Pm31jLBNz4F4hI7rRpaU0jC363ifFxwfx92SysX2iUUqpviEimMSa9s2ND\n807Rnti84Iofwy2vQ2UeLL8c9q3utGlUkC8PX5XC9iPlrMzM7+dClVLqNA307oxbbA3BhI2E126C\nf/+31Xvv4Kb0RKaNDOeX7+VQpHPTlVJuooHek/AkuHstTL0VNv8WXr0BasvOamKzCb/60iSaWtq4\n7YVtVNQ2uadWpdSQpoHuDLs/XPsnWPpHyPsYnp0L+WeP/4+LDea5O9LJLavjjpe2U93Q7KZilVJD\nlQZ6b1x8O9y9Bmw2eHEJbH8OzrioPHt0FH/+2sVkn6ji7lcyqG/S+elKqf6jgd5bw6fCsk0wej68\n9//gzW9C0+n9RhekxvLETVP4NLecb63IpKnFuU00lFLqQmmgn4+ACLj57zD/x/D56/D8Qig7dOrw\n0snD+eX1k9i4r4Tv/X2nLrWrlOoXGujny2aDy/8Dbn0Dqgtg+TzIeefU4ZtnjOAn16Ty7q4CfrTq\nc9o01JVSfUwD/UKNWQDf3ASRY+DvX4NP/nTq0DcuG8V3F4zl9Yx8fvFuNu66iUspNTR4u7sAjxA2\nAr6+GlbdA2segYYqmPcwiPC9hWOpaWjhxY+OEOxn5/uLxrm7WqWUh9JAdxVvX7jhRfB5ADY9Do1V\nsPiXiAiPfiGVmsZmnvzgAMG+3twzd5S7q1VKeSANdFfy8rbmqvsGw9anrVD/4pOIzYtffekiahtb\neey9HIL8vLl5xgh3V6uU8jAa6K5ms8GSX4FfCGz6NTTWwJeew8vbh/+9aQq1TS088uYuAn29WTp5\nuLurVUp5EL0o2hdEYP4jcOVjkP1P+Nst0FSHj7eNZ26dxoykCL7/9518kFPk7kqVUh5EA70vzb4f\nvviktQn1qzdAQxV+di+evyOdCcND+NaKLD4+VOruKpVSHkIDva9NuwO+/ALkb4dXvgi1ZQT72Xn5\nrhkkRQbwjVcy2HG0wt1VKqU8gAZ6f5h4A3z1/6BkL7x8NVSdIDzQh1fvnkl0sC93vvQpOQUd991W\nSqne0UDvL+MWW3eVnsy3FvYqP0JMiB+v3j0Tf7sXt72wnSOlte6uUik1iGmg96ekOXD729Z0xpeu\nguK9JEYE8Oo3ZmKM4dbnt3G8sus9TJVSqjsa6P0tYRrc+R6YNivUT+xgTEwQr3x9BlUNzdz2/DZK\nqhvdXaVSahDSQHeH2DRrqQDfIHj5i5D7ERPjQ3npzukUnGzgthe2cbJON8hQSvWOBrq7RIyCu1ZD\nyDB49UtwYB3pSREsv30ah0tqufPl7VTW6VZ2SinnaaC7U2g83PU+RI2D126GPW9y2dho/njLVHYf\nP8k1T35IZl65u6tUSg0SGujuFhgFd74DCemw8uuQ9VcWT4hj5b2z8bIJNz67lT9tOKjrqSuleqSB\nPhD4hcKtq2DUfHj7fvjkaSYnhvHOd+dw1cQ4frtmH3e8tF0vliqluqWBPlD4BMDNr0HqUljzI9j4\nOCG+3vzx5qn86kuT2H6knKv+sIUtB0rcXalSaoDSQB9IvH3hyy/BlK/Bxl/BX69HKo5w84wRvHX/\npYQF2Ln9xe38ZvVeWlp182ml1Nk00AcaL29Y+hRc/T+QnwFPXwJbfkdKlB9v338pN05L5OmNh7hp\n+Va9CUkpdRYN9IHIZoMZ98D9n8LYK+GD/4Jn5xJQmMGvv3wRf/jqFPYWVHH1H7awdk+hu6tVSg0Q\nGugDWcgwuOmvcPPfoKkGXlwM/3qAa8cH8O53LyMxwp9lf83kZ2/vobGl1d3VKqXcTAN9MBh/FXx7\nK1xyP2T9BZ6aQVLB+7xx7yXcdWkSL3+cy5ee/lgX91JqiOsx0EUkUUQ2iEi2iOwRkQc6afM1Eflc\nRHaJyMciMrlvyh3CfINg8WNwzwbrhqQ37sb37zfx0zmBPHd7OvkV9XzhyS38c8dxd1eqlHITZ3ro\nLcAPjDFpwCzgPhFJ69DmCHC5MWYS8AtguWvLVKcMnwLf+ACW/BqOboU/zWJR+Wu8f/8sUoeF8ODf\nd/LQPz6jrqnF3ZUqpfpZj4FujCkwxmQ5nlcDOUB8hzYfG2Pat93ZCiS4ulB1BpsXzLoX7tsGYxbA\n+p8y/O9L+NvVXtw/fwwrs/JZ+tRH7C3UTTOUGkp6NYYuIknAVGBbN83uBt7v4vxlIpIhIhklJXqD\nzAULTYCvroCbVkB9Bd4vLeb/tSzn/25No7KumWuf+ogV2/IwRpcNUGooEGf/s4tIELAJeMwYs6qL\nNvOBp4E5xpiy7l4vPT3dZGRk9LJc1aXGavj3Y7D9WQiM4eT8x7h/RwJbDpZxzaRh/HRpGjHBfu6u\nUil1gUQk0xiT3tkxp3roImIH3gBWdBPmFwHPA9f2FOaqD/gGw1WPW+PrQTGE/utu/uL/BP81L4Q1\newq5/Dcb+c3qvZys13XWlfJUPfbQRUSAV4ByY8yDXbQZAfwbuN0Y87Ezb6w99D7U2gLbnoENjwFQ\nPv37/KZ4Bn/bXUOInzffmjeGO2cn4e/j5eZClVK91V0P3ZlAnwNsAXYB7QuIPAKMADDGPCMizwM3\nAHmO4y1dvWE7DfR+UHkU3nsI9q8GsVEXM5XVDZN4sXgMJYHjuX/heL46PRG7l96OoNRgcUGB3lc0\n0PuJMXA8Cw6sgQNr4cQOACptYXzQPIld/jOYvuAGrpqehs0mbi5WKdUTDXR1Wk0JHPoAc2AdLfvX\nY2+qpNUIe71T8EtdzKhLrkOGTbbWk1FKDTga6Kpzba205Wey/6NVcGAdKW0HAWj2i8Q+/koYsxBG\nXwEBEW4uVCnVTgNd9ai5tY23PtrJ7k2rmNqUyQL7LoLaqkFskDAdxiyCsQshTnvvSrmTBrpyWn1T\nK698ksuzG/aT1LiPe4YdYoH3Z/gWf2Y1CIyBlGsg7VpImgNedrfWq9RQo4Gueu1kfTPPbT7MCx8e\noam1jbumBHJ/Yh5h+Rtg/xporgX/CEi5GtKug+TLwdvH3WUr5fE00NV5K6lu5E8bDrJiWx4iwh2X\njOSeS4YRU/wxZL8F+96HxirwDXWE+7XWZtd2vStVqb6gga4u2LHyOn6//gCrduRjt9m4fmo898xN\nZkyEDxzeaIX73neg4ST4BMP4JVa4j1kIdn93l6+Ux9BAVy5zpLSW57ccZmVmPo0tbSxIiWHZ3FHM\nSI5AWpshd7MV7jnvQH052ANh3JVWuI+9EnwC3f0tKDWoaaArlyuraeSvW/P4yyd5lNc2MTkhlGVz\nR7N4QizeXjZr+YG8Dx3h/i+oLQFvf2umTNp1Vrj7hbj721Bq0NFAV32mvqmVlVn5vLDlMLlldSRG\n+PONOaP4SnoCAT7eVqO2Vjj6iRXu2W9DTSF4+cLo+RA/DWJSIToVIpKttd6VUl3SQFd9rrXNsC67\nkGc3H2bH0UpC/e3cNmskd8xOIjrY93TDtjbI326F+/7VUH749DEvX4geZ4V7TArEpEF0CoSN1Lnv\nSjlooKt+lZlXzrObDrMupwi7l40bLo7n7jmjGBMTdG7jxhoo3QfFe6EkB4pzrOdV+afb2AMgapwV\n8DEpjsBPtTb4EF1/Rg0tGujKLQ6X1PD8h0dYmZlPU0sbC1NjWDZ3NNOTwpGegrjhJJTsswK+ZC8U\nZ1tBX1N4uo1PMESPt8I9JhXiLoJhk3VsXnk0DXTlVqU1jfzlkzz++kkuFXXNTEkMY9ncUSyeEIdX\nb1d4rCt3BHzO6cfiHKgrPd0mYrS1mfbwqTBsCgy7CPxCXfo9KeUuGuhqQKhvamVl5jGe//AIeWV1\njIgI4KszEvnS1ATiQi/wRqSaYij4DE7shIKd1uOZwzbtIT9siuNxsoa8GpQ00NWA0tpmWLunkJc+\nymV7bjk2gcvGRvOV9AQWpsbiZ3fRTJeaEivkC3ZYAX9OyI9yBPxUDXk1aGigqwErt7SWlZn5vJGV\nT8HJBkL97SydPJyvpCcwKT6057H23qotdfTiHSFf8BmcPHb6eMQoK9gjx0J40umP4GE600YNCBro\nasBrbTN8fKiUf2Tks2ZPIY0tbYyPDeYr6QlcOyX+7KmPrlZbenqYpmAnFHxuhbxpO93GyxfCRpwO\n+Ijk08/DRoJvJzN4lOoDGuhqUDlZ38y/PjvBysx8dh6rxNsmzBsfw1fSE5g/PgYf737oKbc0WaFe\nkdv5R2PV2e0Do8/u0Z8V9sHWejZePjrNUl0wDXQ1aB0oqmZlZj6rdhynpLqRiEAfrpsSz1fSE0gd\n5qbpicZAfUUXYX8ETuaf3btvJzbw9rM+7P5dPPpZSySc9dh+TgD4h0NAJARGWo8BUeAT0K/fvnIv\nDXQ16LW0trH5QAn/yMhnfU4Rza2GifEhfPlia0gmPHAArcXe2ny6d195FJpqobkeWhrOeGyAlvpz\nH1saz23b0tD9+3n7nxvyAZGdfy0wyvqhoEssDFoa6MqjVNQ28dbO4/wjM589J6qwewkLU2NZOnk4\nl4+PPr2GjKdoa4NWR9DXV1hj/nVl1tz7ujLH5+WnP68rg9oyaKru4gUF/MMgJB5CEyEs0XoMTbCu\nE4QmWkNIehF4QNJAVx4r+0QVKzPzeWvnccpqm/Cz25g/PoarJg3jipQYgnw9LNx7o6XxjIA/I+zr\nyqzVL08et4aHTh4795qAly+Edgz8M56HxHv+DlXGWH9ulXnWhzHW8s/2gDMeA6wlon0CrN+U+uGH\noAa68ngtrW1szy3n/V2FrN5TSEl1Iz7eNuaOjebqSXEsSI0l1F/3P+1SfeXpcK88BiePWp9XHrO+\nVlPU4QSB4LjTPXv/cOvir28Q+IaAT5DjebC1REP7MR/H1wbKkE9zvTUsdtZ1kLzTz5tre/d69oCz\ng97u33n4j15gbQJzHjTQ1ZDS2mbIOlrBe7sKWL27kIKTDdi9hDljorhq4jAWpcUOrDH3waCl8XTg\nnxn07Z/XV0JTDbQ2Ofd69gBH2DsC/sznPgGng9Huf/ajT8evBXb43P/smURtbdb6P10F9plrA7XX\n1T476azZSiOsH0JNtdBcB011Vtg31Tk+r+3w2NVxx9dn3gvzHzmvvwoNdDVktbUZduZXsnp3Ie/t\nKiC/oh4vmzB7dCRLJsaxeEIcUUF9OMd9qGlptFbQbKqGxmrreWO14/P2547H9o9Tn9dYQz/N9Y6P\n2s5nC/WkPdy9/awhk9bGM1ruQlUAAA36SURBVA6K41pBh8AOd3weGD3gp5ZqoCsFGGPYfbyK93YX\n8P6uAnLL6rAJzEiO4KqJw1gyMY7YEN3cesAwxurxN9edEfJ1p3u97Z+feuz4tXprZs+pwE62wtx7\ncP8A10BXqgNjDHsLq3l/VwHv7S7kYHENIjBtRDhLJsaxZGIcCeE6v1sNPBroSvXgQFE17zuGZfYW\nWtP9JsWHngr30dF6a78aGDTQleqFI6W1rNlTyOrdhew8VgnA2JigU2PuE4aHuH7RMKWcpIGu1Hkq\nOFnP2j1FrN5dyLYjZbQZSAj3Z8kEq+d+8YhwbL3dpEOpC3BBgS4iicBfgFjAAMuNMX/o0EaAPwBX\nA3XAncaYrO5eVwNdDTZlNY18kFPM6j2FfHiglKbWNqKDfbkyLZYlE+OYNSoSu5feXan61oUG+jBg\nmDEmS0SCgUzgOmNM9hltrga+gxXoM4E/GGNmdve6GuhqMKtuaGbDvhLW7C5kw75i6ppaCfHzZmFa\nLEsmxDF3XLTrNupQ6gzdBXqP90UbYwqAAsfzahHJAeKB7DOaXQv8xVg/HbaKSJiIDHOcq5THCfaz\nNuJYOnk4Dc2tbDlQyurdhazPKWJV1nH87V7MT4lm8YQ45o2P0btUVb/o1UIXIpIETAW2dTgUD5yx\n7Qv5jq+dFegisgxYBjBixIjeVarUAOVn92JRWiyL0mJpbm1j2+FyVu8pYM2eIt7bVYi3TZg5KoKF\nqbEsTI0lMUKnQ6q+4fRFUREJAjYBjxljVnU49g7wuDHmQ8fnHwA/NMZ0OaaiQy7K07W1GXYcq2B9\nTjHrs4s4UFwDQEpcsBXuabFcFB+qF1VVr1zwLBcRsQPvAGuMMU90cvxZYKMx5jXH5/uAed0NuWig\nq6Emt7SW9TlFrM8p4tPcClrbDNHBvixIiWFhaiyXjonC30fH3VX3LvSiqACvAOXGmAe7aHMNcD+n\nL4o+aYyZ0d3raqCroayyromN+0pYl1PEpn0l1DS24Ge3MWdMNIvSYpifEkNMsC5DoM51oYE+B9gC\n7ALaV8p5BBgBYIx5xhH6TwFLsKYt3tXdcAtooCvVrqmljW1Hyvggp5h12UUcr6wHYEpiGIvSrHH3\ncbFBejOTAvTGIqUGjfY1ZtZnW0Mzn+WfBCAxwp+FqbHMGx/DzOQInRI5hGmgKzVIFVU18EFOMetz\nivjoYCmNLW34etuYOSqSuWOjmDc+mtHR2nsfSjTQlfIA9U2tbD1Sxub9JWzeX8KhEms3neGhfswd\nF83l46KZPSZK57x7OA10pTxQfkUdm/eXsnl/CR8dLKW6sQUvmzAlMYzLx0Uzd1w0k+JD8dJpkR5F\nA10pD9fc2sbOY5Vs3l/Cpv0l7Dp+EmMgPMDOnLHRzB0bxeXjoonRDTwGPQ10pYaY8tomthywwn3L\ngVJKqq1t2FLigk/13qeNDNeLq4OQBrpSQ5gxhpyCajY5xt4z8sppbjX4eNmYkhjGzFERzEyO5OKR\nYQT49Go1EOUGGuhKqVNqG1vYeriMbUfK2Xa4jN0nqmhtM3jbhEkJocxMjmTmqAjSR4YT7KcXWAca\nDXSlVJdqGlvIyC0/FfCf55+kpc1gE5gYH8rMZKsHPz0pgtAADXh300BXSjmtrqmFrLxKth8pY+uR\ncnYeraSptQ0RSIkLYWZyBLNGRTAjOZKIQB93lzvkaKArpc5bQ3MrO49Vsu1wOduOlJF1tIKGZmsV\nkHGxQcxIjmB6UgTTRoYTH+avNzn1MQ10pZTLNLW0set4JVsPW8M0mbnl1Da1AhAX4se0pHDSR4aT\nPjKC1GHBeOu2fC6lga6U6jMtrW3sLawmI7eczKOVZOaWc+JkAwD+di+mJIaRnhTOtJHhTB0Rrney\nXiANdKVUvzpRWU9GXgVZeRVk5JWTfaKKNgMiMC4m+KxefGKEDtP0hga6Usqtahtb2Hmsksy8CjLy\nKtiRV0F1YwsA0cG+TBsRfqoXP2F4KD7eOkzTlQvaJFoppS5UoK83l46J4tIxUQC0thn2F1WTmVfh\nCPlyVu8pBMDPbt3wND0pgvSkCC4eEabz4Z2kPXSl1IBQXNVARl4FGblWwO9x3PBkE0gdFuII+HCm\nJ0UQO4TXpNEhF6XUoFPT2MLOo5V8mltORl45WXmV1Ddbs2kSI/yZnhTh+AgfUmvC65CLUmrQCfL1\nZs7YKOaMtYZpmlvbyCmo4tPcCjJyy9m8v4RVWccBa1XJaSMjmJEcTnpSBBOH6Di89tCVUoOSMYbc\nsjqrB59bTkZuBYdLrU0/fL3PHIe3LrZ6yji8DrkopYaE0ppGaww+t5xPc8tPLTxmcyxbMD3J6sFP\nT4ogLnRwjsNroCulhqS6pvZxeOtCa2ZeBXWOu1oTws8dh7cNgt2ddAxdKTUkBfh4M3tMFLMd0yVb\nWtvIKag+daF1y4FS3txhjcOHBditm50cAT8xPhRf78G1AYj20JVSQ5YxhrxT4/AVfJpXzuGS0+Pw\nkxPDTg3TTIoPJSrI180V65CLUko5rbSmkcy8Cj49Us6neRXsOW6tDw8QFeRDSlwI4+OCGR8XTGpc\nCGNjg/p1Kz8dclFKKSdFBfmyeEIciyfEAY5x+GOV5BRUs7egin1F1azYlndqCWGbQFJkICnDghkf\na4V96rBgEsMD+n1MXgNdKaW6EeDjzezRUcweHXXqa61thryyWvYVVrO3sJq9hVVkn6ji/d2FtA96\nBPh4MTY2mFRHb358XDApcSF9uimIDrkopZSL1DW1sL+ohn2FVVbQF1Szr6ia8tqmU21ign2557JR\n3DN31Hm9hw65KKVUPwjw8WZKYhhTEsNOfc0YQ0lNoxXujh59TEjfXFzVQFdKqT4kIsQE+xET7Mfc\ncdF9+l5Db7EDpZTyUBroSinlIXoMdBF5UUSKRWR3F8dDReRfIvKZiOwRkbtcX6ZSSqmeONNDfxlY\n0s3x+4BsY8xkYB7wOxHpu3k5SimlOtVjoBtjNgPl3TUBgsVaXT7I0bbFNeUppZRylivG0J8CUoET\nwC7gAWNMW2cNRWSZiGSISEZJSYkL3loppVQ7VwT6YmAnMByYAjwlIiGdNTTGLDfGpBtj0qOj+3b6\njlJKDTWuCPS7gFXGchA4AqS44HWVUkr1gituLDoKLAC2iEgsMB443NNJmZmZpSKSd57vGQWUnue5\nfWmg1gUDtzatq3e0rt7xxLpGdnWgx7VcROQ1rNkrUUAR8FPADmCMeUZEhmPNhBkGCPC4MebV8yzU\nKSKS0dVaBu40UOuCgVub1tU7WlfvDLW6euyhG2Nu7uH4CeBKl1WklFLqvOidokop5SEGa6Avd3cB\nXRiodcHArU3r6h2tq3eGVF1uWw9dKaWUaw3WHrpSSqkONNCVUspDDLpAF5ElIrJPRA6KyMPurgdA\nRBJFZIOIZDtWnHzA3TWdSUS8RGSHiLzj7lraiUiYiKwUkb0ikiMil7i7JgAR+Z7j73C3iLwmIn5u\nquOcVU5FJEJE1onIAcdj+ACp67eOv8fPReRNEQnr7jX6s7Yzjv1ARIyIRHV2rjvqEpHvOP7c9ojI\nb1zxXoMq0EXEC/gTcBWQBtwsImnurQqwFiP7gTEmDZgF3DdA6mr3AJDj7iI6+AOw2hiTAkxmANQn\nIvHAd4F0Y8xEwAv4qpvKeZlzVzl9GPjAGDMW+MDxeX97mXPrWgdMNMZcBOwHftTfRTm8TCcrw4pI\nItbU6qP9XZDDy3SoS0TmA9cCk40xE4D/ccUbDapAB2YAB40xh40xTcDfsP5Q3MoYU2CMyXI8r8YK\np3j3VmURkQTgGuB5d9fSTkRCgbnACwDGmCZjTKV7qzrFG/AXEW8gAGvRuX7XxSqn1wKvOJ6/AlzX\nr0XReV3GmLXGmPYVVrcCCf1dl6OOrlaG/V/gP7BWhu13XdT1LaybMBsdbYpd8V6DLdDjgWNnfJ7P\nAAnOdiKSBEwFtrm3klN+j/WPudMVMN0kGSgBXnIMBT0vIoHuLsoYcxyrp3QUKABOGmPWureqs8Qa\nYwoczwuBWHcW04WvA++7u4h2InItcNwY85m7a+lgHHCZiGwTkU0iMt0VLzrYAn1AE5Eg4A3gQWNM\n1QCo5wtAsTEm0921dOANXAz82RgzFajFPcMHZ3GMSV+L9QNnOBAoIre6t6rOGWu+8YCacywiP8Ya\nflzh7loARCQAeAT4T3fX0glvIAJriPYh4HXHnhIXZLAF+nEg8YzPExxfczsRsWOF+QpjzCp31+Nw\nKbBURHKxhqeuEJE+XWfHSflAvjGm/beYlVgB724LgSPGmBJjTDOwCpjt5prOVCQiwwAcjy75Nd0V\nRORO4AvA18zAubllNNYP588c/wcSgCwRiXNrVZZ8Tq9Sux3rN+gLvmA72AL9U2CsiCQ7trn7KvC2\nm2vC8ZP1BSDHGPOEu+tpZ4z5kTEmwRiThPVn9W9jjNt7nMaYQuCYiIx3fGkBkO3GktodBWaJSIDj\n73QBA+Bi7RneBu5wPL8DeMuNtZwiIkuwhvWWGmPq3F1PO2PMLmNMjDEmyfF/IB+42PHvz93+CcwH\nEJFxgA8uWBVyUAW648LL/cAarP9orxtj9ri3KsDqCd+G1QPe6fi42t1FDXDfAVaIyOdYG6P80s31\n4PiNYSWQhbX7lg033TruWOX0E2C8iOSLyN3A48AiETmA9dvE4wOkrqeAYGCd49/+M/1dVze1uV0X\ndb0IjHJMZfwbcIcrfrPRW/+VUspDDKoeulJKqa5poCullIfQQFdKKQ+hga6UUh5CA10ppTyEBrpS\nSnkIDXSllPIQ/x8cBNEFD49gBwAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "s37I3HNtFiwS",
        "colab_type": "text"
      },
      "source": [
        "# From the plot, we can infer that validation loss has increased after epoch 17 for 2 successive epochs. Hence, training is stopped at epoch 19.\n",
        "\n",
        "Next, let’s build the dictionary to convert the index to word for target and source vocabulary:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jGI05Wcl-7zv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "reverse_target_word_index=y_tokenizer.index_word\n",
        "reverse_source_word_index=x_tokenizer.index_word\n",
        "target_word_index=y_tokenizer.word_index"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iz9izZXhFpNM",
        "colab_type": "text"
      },
      "source": [
        "# infrences"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VtItXvXS_Eej",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Encode the input sequence to get the feature vector\n",
        "encoder_model = Model(inputs=encoder_inputs,outputs=[encoder_outputs, state_h, state_c])\n",
        "\n",
        "# Decoder setup\n",
        "# Below tensors will hold the states of the previous time step\n",
        "decoder_state_input_h = Input(shape=(latent_dim,))\n",
        "decoder_state_input_c = Input(shape=(latent_dim,))\n",
        "decoder_hidden_state_input = Input(shape=(max_text_len,latent_dim))\n",
        "\n",
        "# Get the embeddings of the decoder sequence\n",
        "dec_emb2= dec_emb_layer(decoder_inputs) \n",
        "# To predict the next word in the sequence, set the initial states to the states from the previous time step\n",
        "decoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=[decoder_state_input_h, decoder_state_input_c])\n",
        "\n",
        "#attention inference\n",
        "attn_out_inf, attn_states_inf = attn_layer([decoder_hidden_state_input, decoder_outputs2])\n",
        "decoder_inf_concat = Concatenate(axis=-1, name='concat')([decoder_outputs2, attn_out_inf])\n",
        "\n",
        "# A dense softmax layer to generate prob dist. over the target vocabulary\n",
        "decoder_outputs2 = decoder_dense(decoder_inf_concat) \n",
        "\n",
        "# Final decoder model\n",
        "decoder_model = Model(\n",
        "    [decoder_inputs] + [decoder_hidden_state_input,decoder_state_input_h, decoder_state_input_c],\n",
        "    [decoder_outputs2] + [state_h2, state_c2])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PgNrhi1nFvOy",
        "colab_type": "text"
      },
      "source": [
        "# We are defining a function below which is the implementation of the inference process (which we covered here):"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "U2eEHBFm_KTu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def decode_sequence(input_seq):\n",
        "    # Encode the input as state vectors.\n",
        "    e_out, e_h, e_c = encoder_model.predict(input_seq)\n",
        "    \n",
        "    # Generate empty target sequence of length 1.\n",
        "    target_seq = np.zeros((1,1))\n",
        "    \n",
        "    # Populate the first word of target sequence with the start word.\n",
        "    target_seq[0, 0] = target_word_index['sostok']\n",
        "\n",
        "    stop_condition = False\n",
        "    decoded_sentence = ''\n",
        "    while not stop_condition:\n",
        "      \n",
        "        output_tokens, h, c = decoder_model.predict([target_seq] + [e_out, e_h, e_c])\n",
        "\n",
        "        # Sample a token\n",
        "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
        "        sampled_token = reverse_target_word_index[sampled_token_index]\n",
        "        \n",
        "        if(sampled_token!='eostok'):\n",
        "            decoded_sentence += ' '+sampled_token\n",
        "\n",
        "        # Exit condition: either hit max length or find stop word.\n",
        "        if (sampled_token == 'eostok'  or len(decoded_sentence.split()) >= (max_summary_len-1)):\n",
        "            stop_condition = True\n",
        "\n",
        "        # Update the target sequence (of length 1).\n",
        "        target_seq = np.zeros((1,1))\n",
        "        target_seq[0, 0] = sampled_token_index\n",
        "\n",
        "        # Update internal states\n",
        "        e_h, e_c = h, c\n",
        "\n",
        "    return decoded_sentence"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zkBBqHWzFzj_",
        "colab_type": "text"
      },
      "source": [
        "# Let us define the functions to convert an integer sequence to a word sequence for summary as well as the reviews:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "x8EP6Vt2_OOV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def seq2summary(input_seq):\n",
        "    newString=''\n",
        "    for i in input_seq:\n",
        "        if((i!=0 and i!=target_word_index['sostok']) and i!=target_word_index['eostok']):\n",
        "            newString=newString+reverse_target_word_index[i]+' '\n",
        "    return newString\n",
        "\n",
        "def seq2text(input_seq):\n",
        "    newString=''\n",
        "    for i in input_seq:\n",
        "        if(i!=0):\n",
        "            newString=newString+reverse_source_word_index[i]+' '\n",
        "    return newString"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OFBX2X1eF3Yz",
        "colab_type": "text"
      },
      "source": [
        "# Here are a few summaries generated by the model:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "j607j-be_S0z",
        "colab_type": "code",
        "outputId": "5ddc0b3d-942e-4975-85e1-471bcb9c3115",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        }
      },
      "source": [
        "for i in range(0,100):\n",
        "    print(\"Review:\",seq2text(x_tr[i]))\n",
        "    print(\"Original summary:\",seq2summary(y_tr[i]))\n",
        "    print(\"Predicted summary:\",decode_sequence(x_tr[i].reshape(1,max_text_len)))\n",
        "    print(\"\\n\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Review: ordering another brand years wanted switch bit see products brand appealed cuts back considerably packaging waste tea green tea suitable tastes pretty standard problem really mint flavor speak \n",
            "Original summary: okay for green not minty however \n",
            "Predicted summary:  not the best tea\n",
            "\n",
            "\n",
            "Review: never find chai last states lived order tried chai mixes find stores replace one best delicious \n",
            "Original summary: amazon has it \n",
            "Predicted summary:  best chai\n",
            "\n",
            "\n",
            "Review: yummy cannot believe lbs give away much personally would purchased retail price jump amazon chocolates sale \n",
            "Original summary: yummy \n",
            "Predicted summary:  delicious\n",
            "\n",
            "\n",
            "Review: exactly looking keep fruit turning brown price great \n",
            "Original summary: fruit \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: love pop chips available area happy find available amazon free shipping yummy also healthy choice cannot beat deal like \n",
            "Original summary: great snack \n",
            "Predicted summary:  great chips\n",
            "\n",
            "\n",
            "Review: taste incredible lacks creamy taste ingredients also could improved much potatoes \n",
            "Original summary: just alright soup \n",
            "Predicted summary:  not what expected\n",
            "\n",
            "\n",
            "Review: love hot chocolate excited find variety pack however gross horrible taste kids dont even like boxes stuff one drink disapointed \n",
            "Original summary: very disapointed \n",
            "Predicted summary:  not bad\n",
            "\n",
            "\n",
            "Review: great product thing would like last longer dog maybe making tougher would extend dogs time task \n",
            "Original summary: better than ears \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: bought students study food liked tasted looked exactly expected bigger \n",
            "Original summary: classic only bigger \n",
            "Predicted summary:  great\n",
            "\n",
            "\n",
            "Review: love green mountain coffee first one bought got kurig strong \n",
            "Original summary: good coffee \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: received lemon snaps faster expected one bags missing order cookies ok much lemon flavor give rating \n",
            "Original summary: so so \n",
            "Predicted summary:  not what expected\n",
            "\n",
            "\n",
            "Review: taste awesome esp salt pepper one love really healthy packaging really good \n",
            "Original summary: amazing \n",
            "Predicted summary:  best salt\n",
            "\n",
            "\n",
            "Review: tea lots flavor sweet without use sweeteners \n",
            "Original summary: lots of flavor \n",
            "Predicted summary:  great tea\n",
            "\n",
            "\n",
            "Review: drank try keep awake fell asleep minutes drinking feel anything \n",
            "Original summary: it made me fall \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: love using coffee chocolate mint stash tea purchasing via amazon actually cheaper vitacost also choice bought option \n",
            "Original summary: yum \n",
            "Predicted summary:  great tea\n",
            "\n",
            "\n",
            "Review: got bag christmas used sweeten coffee wonderful stuff \n",
            "Original summary: great in coffee \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: love vita coconut water wonderful use smoothies recipes everyday great stuff live far grocery store get fresh coconuts best alternative \n",
            "Original summary: favorite drink \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: never met dog didnt like good hips \n",
            "Original summary: dogs love them \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: excellent product use almost every night used available costco longer handle would recommend anyone \n",
            "Original summary: great salad dressing \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: love product suffer sinus problems energy level really helps boost \n",
            "Original summary: great energy \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: keep finding rotten beef steak carton quality control poor jack link \n",
            "Original summary: more beef steak in \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: hooked big train brand chai tea products like hot really buying sugar added vanilla chai need sugar product surprisingly lack flavor little caloric teas really quite good \n",
            "Original summary: on the train \n",
            "Predicted summary:  great tea\n",
            "\n",
            "\n",
            "Review: might cost bit store bought popcorn really worth works perfectly whirly pop maker great \n",
            "Original summary: tasty popcorn \n",
            "Predicted summary:  great popcorn\n",
            "\n",
            "\n",
            "Review: liked coffee much subscribing dark rich smooth \n",
            "Original summary: makes great cup of java \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: love larabars love pistachios pistachio larabars awful horrid fruitcake taste enough pistachios back cashew cookie cherry pie \n",
            "Original summary: ick \n",
            "Predicted summary:  yuck\n",
            "\n",
            "\n",
            "Review: love stuff makes great iced coffee mainly buy cereal price good since stores seem going got sister hooked usually get cal prime time went cal \n",
            "Original summary: better than milk \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: taste snack well price second time would certainly \n",
            "Original summary: excellent \n",
            "Predicted summary:  great snack\n",
            "\n",
            "\n",
            "Review: best raisins plump favorful keep extras big freezer thaw need cannot say anything negative \n",
            "Original summary: soft and yummy \n",
            "Predicted summary:  delicious\n",
            "\n",
            "\n",
            "Review: love product substitute sugar one tell difference \n",
            "Original summary: best substitute \n",
            "Predicted summary:  great taste\n",
            "\n",
            "\n",
            "Review: like every flavor cheez addicted really tasty everyone house liked problem half box crumbs assume tossed around shipping bummer \n",
            "Original summary: really tasty but too many crumbs \n",
            "Predicted summary:  not the best\n",
            "\n",
            "\n",
            "Review: price plain robbery number stores including target oz waste money want \n",
            "Original summary: overpriced \n",
            "Predicted summary:  good product\n",
            "\n",
            "\n",
            "Review: waste money disgusting product chocolate taste tastes like plastic lining paper carton using milk treated ultra high temperatures like fresh milk go get fresh milk hershey syrup want chocolate milk \n",
            "Original summary: please do not waste your money \n",
            "Predicted summary:  yuck\n",
            "\n",
            "\n",
            "Review: shih tzu favorite treats tried getting brands similar treats compare quality veggie life happy hips chicken banana treats even labrador loves lot order one time well sealed stay fresh \n",
            "Original summary: my toy dogs love these treats \n",
            "Predicted summary:  my dog loves these\n",
            "\n",
            "\n",
            "Review: coffee turned favorite cups best full bodied rich tasting great way start day \n",
            "Original summary: tully french roast \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: great buy price cannot get kind deal anywhere local buy cream way long get \n",
            "Original summary: flavored cream \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: zoe excellent cooking oil metal container promotes integrity long shelf life also useful processes calling olive oil \n",
            "Original summary: excellent oil \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: awesome tasting like authentic ranch dressing dont eat many one sitting flavour overpower \n",
            "Original summary: yum \n",
            "Predicted summary:  delicious\n",
            "\n",
            "\n",
            "Review: ok surprised rich twinings teas would go way buy boxes last strong might like could use little similar vanilla flavored teas shelf level hoped something different twinnings \n",
            "Original summary: more vanilla please \n",
            "Predicted summary:  not the best tea\n",
            "\n",
            "\n",
            "Review: due poor packing pouches ordered broke throw leaked everywhere mess disappointed waste money \n",
            "Original summary: very disappointed \n",
            "Predicted summary:  poor packaging\n",
            "\n",
            "\n",
            "Review: vermont maple syrup originally ny grew vermont maple syrup used syrup lacks little tang vermont syrup come second tho \n",
            "Original summary: this is good syrup but \n",
            "Predicted summary:  delicious\n",
            "\n",
            "\n",
            "Review: love cooking brands high quality coconut oil stable higher temperatures evoo add foods per day patients improvement functions could ask \n",
            "Original summary: healthy coconut oil \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: since used product previously knew getting product arrived timely manner packaged well fresh would order needed since longer find product locally care brands offered \n",
            "Original summary: cocoa \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: nom nom nom delicious filling mention delicious much better average protein bars high calorie could eating snickers bar yum \n",
            "Original summary: like healthy bar \n",
            "Predicted summary:  great snack\n",
            "\n",
            "\n",
            "Review: must low carb used baking making low carb ice cream sauces without use flour corn starch amazon best price \n",
            "Original summary: little goes long way \n",
            "Predicted summary:  great\n",
            "\n",
            "\n",
            "Review: absolutely love cereal slight sweetness satisfies sweetness want cereal also stays crunchy longer regular corn flakes \n",
            "Original summary: best gluten free cereal ever \n",
            "Predicted summary:  great cereal\n",
            "\n",
            "\n",
            "Review: bought bars grandson college munch studying said delicious linda \n",
            "Original summary: think thin brownie bars \n",
            "Predicted summary:  great\n",
            "\n",
            "\n",
            "Review: sauce really delivers says label bold creamy spicy enough make salads sandwiches interesting without inducing tears thick creamy enough ideal salad dressing complement stacked sandwich repeat buy item us \n",
            "Original summary: bold and delicious \n",
            "Predicted summary:  great sauce\n",
            "\n",
            "\n",
            "Review: found accidentally searching greenies dental chews dogs husky seems like smoke flavor quite know help joints wanted prevent future problems \n",
            "Original summary: my loves them \n",
            "Predicted summary:  greenies\n",
            "\n",
            "\n",
            "Review: pricey essentially small bag hard crumbs maybe dog spoiled treats like third class treats definitely bottom doggie treat often simply walk away glad people like buying \n",
            "Original summary: waste of money \n",
            "Predicted summary:  my dog loves these\n",
            "\n",
            "\n",
            "Review: would like able buy choc jello cook serve sugar free pudding site cannot seem find let know \n",
            "Original summary: more flavors in this size \n",
            "Predicted summary:  good\n",
            "\n",
            "\n",
            "Review: best decaf tasted cups perfect cup coffee would read package know decaf full rich flavor bitterness \n",
            "Original summary: the best decaf in cups \n",
            "Predicted summary:  great decaf\n",
            "\n",
            "\n",
            "Review: originally bought promotion best raisins recently took shipment amazon pack canisters good family four months discount subscribe save helps cancel want receive anymore without \n",
            "Original summary: great raisins \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: could find orange flower water used years took chance bought brand disappointed helped us make great christmas morning \n",
            "Original summary: good product \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: make son gluten free diet take lunch love \n",
            "Original summary: yummy \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: coffee really bold opinion falls along medium roast flavor decent smooth coffee think really bold like cups though seriously overpriced \n",
            "Original summary: good coffee \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: picky house every time open venison holiday go crazy \n",
            "Original summary: doggy delight \n",
            "Predicted summary:  great\n",
            "\n",
            "\n",
            "Review: buy tangerine juice spring available entire year tasty tart still sweet satisfying \n",
            "Original summary: the best ever \n",
            "Predicted summary:  great\n",
            "\n",
            "\n",
            "Review: two mo old twins way eat oatmeal combination eb pear convenient especially quickly mixing breakfast morning packaging amazon always secure never broken jars recommended \n",
            "Original summary: babies love this \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: great still mine wrapped never seem finish cant exactly suck since hurts stick entire thing mouth licked point tongue started little \n",
            "Original summary: multi \n",
            "Predicted summary:  not as good as other\n",
            "\n",
            "\n",
            "Review: sams purchased mine last forever cost cannot argue \n",
            "Original summary: stars for the item for the price \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: like tomatoes fresh flavorful also come carton welcome alternative metal cans impart flavor sometimes lined plastic containing \n",
            "Original summary: yummy tomatoes good packaging \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: herbs wonderful come grinder opened package smell package aroma wonderful tried herbs provence give try french know herbs purchase herbs amazon amazon makes easy fast \n",
            "Original summary: of \n",
            "Predicted summary:  great taste\n",
            "\n",
            "\n",
            "Review: far one top two favorite flavors boulder chips bought bag yet devoured family one sitting \n",
            "Original summary: eaten in one \n",
            "Predicted summary:  great chips\n",
            "\n",
            "\n",
            "Review: definitely minority opinion vanilla flavor strong tasted artificial someone looking tea vanilla using vanilla tea bag green tea bag makes mellow brew buy spiced chai decaf much better \n",
            "Original summary: not bigelow best \n",
            "Predicted summary:  not very good\n",
            "\n",
            "\n",
            "Review: absolute worst flavored coffee ever put mouth took several sips sure hated fourth sip back cup thought possible shipment tainted chemicals please buy large quantity pawn someone like enjoy \n",
            "Original summary: weak coffee with \n",
            "Predicted summary:  weak coffee\n",
            "\n",
            "\n",
            "Review: provide stay paper one paper always falls area would wind wind \n",
            "Original summary: bit disappointing \n",
            "Predicted summary:  not what expected\n",
            "\n",
            "\n",
            "Review: great company first ever seen offer low acid option absolutely taste difference coffee without bitterness makes possible enjoy purity bean without bitter taste also cool black mugs love \n",
            "Original summary: best coffee ever \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: needed juniper berries recipe could find store pleased find pleased transaction back \n",
            "Original summary: hard to find product \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: food sam loves stuff stinky cheaper brands canned catfood going order flavors try \n",
            "Original summary: my cat at me every morning \n",
            "Predicted summary:  great food\n",
            "\n",
            "\n",
            "Review: good coffee great price look like works well needs stored air tight container purchase \n",
            "Original summary: good coffee great price \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: dried peaches bad compared dried apricots love would say little dried chew long time could swallow actually bought use recipe peaches cream cookies saw food channel \n",
            "Original summary: chewy \n",
            "Predicted summary:  great taste\n",
            "\n",
            "\n",
            "Review: like taste light roast light find brewing smaller cups coffee hoping stronger taste next time go medium dark roast think medium really liked \n",
            "Original summary: good blend but roast was too light \n",
            "Predicted summary:  not my cup of coffee\n",
            "\n",
            "\n",
            "Review: product good goes long way quite good one dd good product less \n",
            "Original summary: very good \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: unless missing boxes something way overpriced buy directly keurig member people really fall amazon let get away \n",
            "Original summary: way overpriced \n",
            "Predicted summary:  not what expected\n",
            "\n",
            "\n",
            "Review: delicious special something center gr receive gift would never buy personally great value money \n",
            "Original summary: review \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: bought product purpose making yogurt worked well also convenient size added baked goods casseroles \n",
            "Original summary: good product good size \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: colombia looking coffee tasted like back home everyday make coffee aroma whole kitchen feels like home tastes like love coffee \n",
            "Original summary: like good coffee \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: great packaging great product cat definitely approved variety pack fish flavored cat food little hesitant first tasted happy would recommend anyone cat taste fish \n",
            "Original summary: cat food \n",
            "Predicted summary:  my cats love it\n",
            "\n",
            "\n",
            "Review: good fast snack worth money used make great price per bar wasting time anymore make always one purse \n",
            "Original summary: saves me time great snack \n",
            "Predicted summary:  great snack\n",
            "\n",
            "\n",
            "Review: almond product wonderful excited healthy options curb sweet tooth sugars ounce almonds \n",
            "Original summary: excellent low sugar sweet treat \n",
            "Predicted summary:  chocolate\n",
            "\n",
            "\n",
            "Review: one picky siamese eat anything newman organics chicken brown rice formula recommendation eats must good \n",
            "Original summary: cat food ever \n",
            "Predicted summary:  my cats love it\n",
            "\n",
            "\n",
            "Review: product comes natural let us forget people canola oil horrible oil stated non gmo \n",
            "Original summary: oil no good \n",
            "Predicted summary:  good product\n",
            "\n",
            "\n",
            "Review: sauce one best flavors ever tasted good balance spice garlic decent heat tastes good lot things big fan habanero sauces due extra bitterness pepper sauce blends everything perfectly \n",
            "Original summary: great sauce \n",
            "Predicted summary:  great sauce\n",
            "\n",
            "\n",
            "Review: since abundance popcorn eat ever pops evenly little waste getting supply \n",
            "Original summary: love popcorn \n",
            "Predicted summary:  popcorn\n",
            "\n",
            "\n",
            "Review: really like medium strength coffee little bold considered medium almost bitter compared medium coffees \n",
            "Original summary: almost bitter \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: love chips auto order every months taste great whole bag calories bag every day sure helped weight loss little bags eat huge amount \n",
            "Original summary: great purchase \n",
            "Predicted summary:  great chips\n",
            "\n",
            "\n",
            "Review: great cereal help taste great stays fresh crunchy last bit going try cereals worth money love comes small boxes keeps cereal fresh \n",
            "Original summary: taste so good and crunchy \n",
            "Predicted summary:  great cereal\n",
            "\n",
            "\n",
            "Review: best cup coffee able find rich flavors medium roast love one \n",
            "Original summary: best high quality cup \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: love honey stinger waffles great price deff order run love choices thanks \n",
            "Original summary: love waffles \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: tea pretty basic black tea exotic flavorings added tea decent amount strength full bodied overwhelming quite satisfied cup tea generally prefer drink flavored teas basic black tea variety quite good \n",
            "Original summary: basic black tea \n",
            "Predicted summary:  great tea\n",
            "\n",
            "\n",
            "Review: long coat chihuahua loves food eats right feel good giving highly nutritious food junk dog six pounds require lot food suspect would pricey food give large dog though \n",
            "Original summary: my dog loves it \n",
            "Predicted summary:  great food\n",
            "\n",
            "\n",
            "Review: exceptional coffee priced league totally price hike seen long anywhere anything \n",
            "Original summary: price \n",
            "Predicted summary:  great coffee\n",
            "\n",
            "\n",
            "Review: idea real pickling spice point purchased supermarket brands pale comparison minute see bottle open smell spices inside see mean corned beef never tasted good \n",
            "Original summary: wow \n",
            "Predicted summary:  great flavor\n",
            "\n",
            "\n",
            "Review: best coconut water tasted tried plenty brands must buy like coconut water \n",
            "Original summary: great product \n",
            "Predicted summary:  great taste\n",
            "\n",
            "\n",
            "Review: love sweet nutty natural flavor truly delish snack given one part gift basket fell love many wonderful flavors including cashew chai chocolate addition original like \n",
            "Original summary: snack or meal replacement on the go \n",
            "Predicted summary:  great product\n",
            "\n",
            "\n",
            "Review: also bought costco per box included bags oz kids fighting remaining bags good buying due price high price prevent product reaching mass distribution \n",
            "Original summary: very good but too pricey \n",
            "Predicted summary:  good product\n",
            "\n",
            "\n",
            "Review: love haribo candies color gummi candy great love licorice gummy bears ate whole package one day \n",
            "Original summary: delicious \n",
            "Predicted summary:  great\n",
            "\n",
            "\n",
            "Review: nuts plump fat salty seemed bit dry fresh would liked purchase amount feel fresh nuts uniquely flavorful signature taste seemed many nuts lovely one another \n",
            "Original summary: plump and fat \n",
            "Predicted summary:  not as good as other\n",
            "\n",
            "\n",
            "Review: superb green tea organic decaf bags really filled get lot bang bag came larger boxes \n",
            "Original summary: excellent green tea \n",
            "Predicted summary:  great tea\n",
            "\n",
            "\n",
            "Review: used almost never stock inventory improved making delicious flavored coffee much flavor since normally drink coffee black quite good give shot \n",
            "Original summary: delicious \n",
            "Predicted summary:  best coffee ever\n",
            "\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oSaMUD8jGA77",
        "colab_type": "text"
      },
      "source": [
        "## How can we Improve the Model’s Performance Even Further?\n",
        "### Your learning doesn’t stop here! There’s a lot more you can do to play around and experiment with the model:\n",
        "\n",
        "### I recommend you to increase the training dataset size and build the model. The generalization capability of a deep learning model enhances with an increase in the training dataset size\n",
        "\n",
        "### Try implementing Bi-Directional LSTM which is capable of capturing the context from both the directions and results in a better context vector\n",
        "\n",
        "### Use the beam search strategy for decoding the test sequence instead of using the greedy approach (argmax)\n",
        "\n",
        "### Evaluate the performance of your model based on the BLEU score\n",
        "\n",
        "### Implement pointer-generator networks and coverage mechanisms"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0oLM2Idc_XQ3",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}