Sentimental Analysis using RNN in Keras and Tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Algorithm for Sentimental Analysis using RNN \n",
    "\n",
    "### 1) First we need to convert the raw text-words into so-called tokens which are integer values.\n",
    "\n",
    "### 2) Then we convert these integer-tokens into so-called embeddings which are real-valued vectors, whose mapping will be trained along with the neural network, so as to map words with similar meanings to similar embedding-vectors. \n",
    "\n",
    "### 3) Then we input these embedding-vectors to a Recurrent Neural Network which can take sequences of arbitrary length as input and output a kind of summary of what it has seen in the input.\n",
    "\n",
    "### 4) Output from the RNN is squashed by an activation function (Sigmoid in this case)\n",
    "\n",
    "### 5) output is between 0 and 1 \n",
    " \n",
    "### { 0: highly negative, 1 : highly positive }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "#importing required Libraries\n",
    "\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import tensorflow as tf\n",
    "import numpy as np\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "from scipy.spatial.distance import cdist\n",
    "# from tf.keras.models import Sequential  # This does not work!\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, GRU, Embedding\n",
    "from keras.optimizers import Adam\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Some of the code and explaination here is taken from https://github.com/Hvass-Labs/ :)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import imdb # this is helper package to download and load the imdb dataset by https://github.com/Hvass-Labs/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Data has apparently already been downloaded and unpacked.\n"
     ]
    }
   ],
   "source": [
    "imdb.maybe_download_and_extract() #Downloading and Extracting the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_text, y_train = imdb.load_data(train=True) #loading train data\n",
    "x_test_text, y_test = imdb.load_data(train=False) # loading test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train-set size:  25000\n",
      "Test-set size:   25000\n"
     ]
    }
   ],
   "source": [
    "print(\"Train-set size: \", len(x_train_text))\n",
    "print(\"Test-set size:  \", len(x_test_text))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_text = x_train_text + x_test_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"Not to mention easily Pierce Brosnon's best performance. Of course Greg Kinnear is always great. Really, when has he really been bad? I think this film is incredibly underrated! The use of colors in this movie is something very different in today's film world where every other movie has the Payback blue filter. I also love the way they used the song by Asia. Proving that even what was once thought of as kinda cheesy can be really cool placed correctly.<br /><br />I was making my first feature when this came out. Being that my film was a hit-man movie, I had to check out anything in the genre that was released. After seeing it, I'm sure it had some effect on me through the process. It was pretty cool when my film got on the IMDb that it would recommend this film if you liked mine. How any of the others relate I have no idea, making an even more interesting coincidence.<br /><br />http://www.imdb.com/title/tt1337580/\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_text[100] # looking at an example text "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[100]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_words = 10000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = Tokenizer(num_words=num_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 12.3 s, sys: 36.6 ms, total: 12.3 s\n",
      "Wall time: 12.3 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "tokenizer.fit_on_texts(data_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'the': 1,\n",
       " 'and': 2,\n",
       " 'a': 3,\n",
       " 'of': 4,\n",
       " 'to': 5,\n",
       " 'is': 6,\n",
       " 'br': 7,\n",
       " 'in': 8,\n",
       " 'it': 9,\n",
       " 'i': 10,\n",
       " 'this': 11,\n",
       " 'that': 12,\n",
       " 'was': 13,\n",
       " 'as': 14,\n",
       " 'for': 15,\n",
       " 'with': 16,\n",
       " 'movie': 17,\n",
       " 'but': 18,\n",
       " 'film': 19,\n",
       " 'on': 20,\n",
       " 'not': 21,\n",
       " 'you': 22,\n",
       " 'are': 23,\n",
       " 'his': 24,\n",
       " 'have': 25,\n",
       " 'be': 26,\n",
       " 'one': 27,\n",
       " 'he': 28,\n",
       " 'all': 29,\n",
       " 'at': 30,\n",
       " 'by': 31,\n",
       " 'an': 32,\n",
       " 'they': 33,\n",
       " 'so': 34,\n",
       " 'who': 35,\n",
       " 'from': 36,\n",
       " 'like': 37,\n",
       " 'or': 38,\n",
       " 'just': 39,\n",
       " 'her': 40,\n",
       " 'out': 41,\n",
       " 'about': 42,\n",
       " 'if': 43,\n",
       " \"it's\": 44,\n",
       " 'has': 45,\n",
       " 'there': 46,\n",
       " 'some': 47,\n",
       " 'what': 48,\n",
       " 'good': 49,\n",
       " 'when': 50,\n",
       " 'more': 51,\n",
       " 'very': 52,\n",
       " 'up': 53,\n",
       " 'no': 54,\n",
       " 'time': 55,\n",
       " 'my': 56,\n",
       " 'even': 57,\n",
       " 'would': 58,\n",
       " 'she': 59,\n",
       " 'which': 60,\n",
       " 'only': 61,\n",
       " 'really': 62,\n",
       " 'see': 63,\n",
       " 'story': 64,\n",
       " 'their': 65,\n",
       " 'had': 66,\n",
       " 'can': 67,\n",
       " 'me': 68,\n",
       " 'well': 69,\n",
       " 'were': 70,\n",
       " 'than': 71,\n",
       " 'much': 72,\n",
       " 'we': 73,\n",
       " 'bad': 74,\n",
       " 'been': 75,\n",
       " 'get': 76,\n",
       " 'do': 77,\n",
       " 'great': 78,\n",
       " 'other': 79,\n",
       " 'will': 80,\n",
       " 'also': 81,\n",
       " 'into': 82,\n",
       " 'people': 83,\n",
       " 'because': 84,\n",
       " 'how': 85,\n",
       " 'first': 86,\n",
       " 'him': 87,\n",
       " 'most': 88,\n",
       " \"don't\": 89,\n",
       " 'made': 90,\n",
       " 'then': 91,\n",
       " 'its': 92,\n",
       " 'them': 93,\n",
       " 'make': 94,\n",
       " 'way': 95,\n",
       " 'too': 96,\n",
       " 'movies': 97,\n",
       " 'could': 98,\n",
       " 'any': 99,\n",
       " 'after': 100,\n",
       " 'think': 101,\n",
       " 'characters': 102,\n",
       " 'watch': 103,\n",
       " 'films': 104,\n",
       " 'two': 105,\n",
       " 'many': 106,\n",
       " 'seen': 107,\n",
       " 'character': 108,\n",
       " 'being': 109,\n",
       " 'never': 110,\n",
       " 'plot': 111,\n",
       " 'love': 112,\n",
       " 'acting': 113,\n",
       " 'life': 114,\n",
       " 'did': 115,\n",
       " 'best': 116,\n",
       " 'where': 117,\n",
       " 'know': 118,\n",
       " 'show': 119,\n",
       " 'little': 120,\n",
       " 'over': 121,\n",
       " 'off': 122,\n",
       " 'ever': 123,\n",
       " 'does': 124,\n",
       " 'your': 125,\n",
       " 'better': 126,\n",
       " 'end': 127,\n",
       " 'man': 128,\n",
       " 'scene': 129,\n",
       " 'still': 130,\n",
       " 'say': 131,\n",
       " 'these': 132,\n",
       " 'here': 133,\n",
       " 'scenes': 134,\n",
       " 'why': 135,\n",
       " 'while': 136,\n",
       " 'something': 137,\n",
       " 'such': 138,\n",
       " 'go': 139,\n",
       " 'through': 140,\n",
       " 'back': 141,\n",
       " 'should': 142,\n",
       " 'those': 143,\n",
       " 'real': 144,\n",
       " \"i'm\": 145,\n",
       " 'now': 146,\n",
       " 'watching': 147,\n",
       " 'thing': 148,\n",
       " \"doesn't\": 149,\n",
       " 'actors': 150,\n",
       " 'though': 151,\n",
       " 'funny': 152,\n",
       " 'years': 153,\n",
       " \"didn't\": 154,\n",
       " 'old': 155,\n",
       " 'another': 156,\n",
       " '10': 157,\n",
       " 'work': 158,\n",
       " 'before': 159,\n",
       " 'actually': 160,\n",
       " 'nothing': 161,\n",
       " 'makes': 162,\n",
       " 'look': 163,\n",
       " 'director': 164,\n",
       " 'find': 165,\n",
       " 'going': 166,\n",
       " 'same': 167,\n",
       " 'new': 168,\n",
       " 'lot': 169,\n",
       " 'every': 170,\n",
       " 'few': 171,\n",
       " 'again': 172,\n",
       " 'part': 173,\n",
       " 'cast': 174,\n",
       " 'down': 175,\n",
       " 'us': 176,\n",
       " 'things': 177,\n",
       " 'want': 178,\n",
       " 'quite': 179,\n",
       " 'pretty': 180,\n",
       " 'world': 181,\n",
       " 'horror': 182,\n",
       " 'around': 183,\n",
       " 'seems': 184,\n",
       " \"can't\": 185,\n",
       " 'young': 186,\n",
       " 'take': 187,\n",
       " 'however': 188,\n",
       " 'got': 189,\n",
       " 'thought': 190,\n",
       " 'big': 191,\n",
       " 'fact': 192,\n",
       " 'enough': 193,\n",
       " 'long': 194,\n",
       " 'both': 195,\n",
       " \"that's\": 196,\n",
       " 'give': 197,\n",
       " \"i've\": 198,\n",
       " 'own': 199,\n",
       " 'may': 200,\n",
       " 'between': 201,\n",
       " 'comedy': 202,\n",
       " 'right': 203,\n",
       " 'series': 204,\n",
       " 'action': 205,\n",
       " 'must': 206,\n",
       " 'music': 207,\n",
       " 'without': 208,\n",
       " 'times': 209,\n",
       " 'saw': 210,\n",
       " 'always': 211,\n",
       " 'original': 212,\n",
       " \"isn't\": 213,\n",
       " 'role': 214,\n",
       " 'come': 215,\n",
       " 'almost': 216,\n",
       " 'gets': 217,\n",
       " 'interesting': 218,\n",
       " 'guy': 219,\n",
       " 'point': 220,\n",
       " 'done': 221,\n",
       " \"there's\": 222,\n",
       " 'whole': 223,\n",
       " 'least': 224,\n",
       " 'far': 225,\n",
       " 'bit': 226,\n",
       " 'script': 227,\n",
       " 'minutes': 228,\n",
       " 'feel': 229,\n",
       " '2': 230,\n",
       " 'anything': 231,\n",
       " 'making': 232,\n",
       " 'might': 233,\n",
       " 'since': 234,\n",
       " 'am': 235,\n",
       " 'family': 236,\n",
       " \"he's\": 237,\n",
       " 'last': 238,\n",
       " 'probably': 239,\n",
       " 'tv': 240,\n",
       " 'performance': 241,\n",
       " 'kind': 242,\n",
       " 'away': 243,\n",
       " 'yet': 244,\n",
       " 'fun': 245,\n",
       " 'worst': 246,\n",
       " 'sure': 247,\n",
       " 'rather': 248,\n",
       " 'hard': 249,\n",
       " 'anyone': 250,\n",
       " 'girl': 251,\n",
       " 'each': 252,\n",
       " 'played': 253,\n",
       " 'day': 254,\n",
       " 'found': 255,\n",
       " 'looking': 256,\n",
       " 'woman': 257,\n",
       " 'screen': 258,\n",
       " 'although': 259,\n",
       " 'our': 260,\n",
       " 'especially': 261,\n",
       " 'believe': 262,\n",
       " 'having': 263,\n",
       " 'trying': 264,\n",
       " 'course': 265,\n",
       " 'dvd': 266,\n",
       " 'everything': 267,\n",
       " 'set': 268,\n",
       " 'goes': 269,\n",
       " 'comes': 270,\n",
       " 'put': 271,\n",
       " 'ending': 272,\n",
       " 'maybe': 273,\n",
       " 'place': 274,\n",
       " 'book': 275,\n",
       " 'shows': 276,\n",
       " 'three': 277,\n",
       " 'worth': 278,\n",
       " 'different': 279,\n",
       " 'main': 280,\n",
       " 'once': 281,\n",
       " 'sense': 282,\n",
       " 'american': 283,\n",
       " 'reason': 284,\n",
       " 'looks': 285,\n",
       " 'effects': 286,\n",
       " 'watched': 287,\n",
       " 'play': 288,\n",
       " 'true': 289,\n",
       " 'money': 290,\n",
       " 'actor': 291,\n",
       " \"wasn't\": 292,\n",
       " 'job': 293,\n",
       " 'together': 294,\n",
       " 'war': 295,\n",
       " 'someone': 296,\n",
       " 'plays': 297,\n",
       " 'instead': 298,\n",
       " 'high': 299,\n",
       " 'during': 300,\n",
       " 'said': 301,\n",
       " 'year': 302,\n",
       " 'half': 303,\n",
       " 'everyone': 304,\n",
       " 'later': 305,\n",
       " 'takes': 306,\n",
       " '1': 307,\n",
       " 'seem': 308,\n",
       " 'audience': 309,\n",
       " 'special': 310,\n",
       " 'beautiful': 311,\n",
       " 'left': 312,\n",
       " 'himself': 313,\n",
       " 'seeing': 314,\n",
       " 'john': 315,\n",
       " 'night': 316,\n",
       " 'black': 317,\n",
       " 'version': 318,\n",
       " 'shot': 319,\n",
       " 'excellent': 320,\n",
       " 'idea': 321,\n",
       " 'house': 322,\n",
       " 'mind': 323,\n",
       " 'star': 324,\n",
       " 'wife': 325,\n",
       " 'fan': 326,\n",
       " 'death': 327,\n",
       " 'used': 328,\n",
       " 'else': 329,\n",
       " 'simply': 330,\n",
       " 'nice': 331,\n",
       " 'budget': 332,\n",
       " 'poor': 333,\n",
       " 'short': 334,\n",
       " 'completely': 335,\n",
       " 'second': 336,\n",
       " \"you're\": 337,\n",
       " '3': 338,\n",
       " 'read': 339,\n",
       " 'along': 340,\n",
       " 'less': 341,\n",
       " 'top': 342,\n",
       " 'help': 343,\n",
       " 'home': 344,\n",
       " 'men': 345,\n",
       " 'either': 346,\n",
       " 'line': 347,\n",
       " 'boring': 348,\n",
       " 'dead': 349,\n",
       " 'friends': 350,\n",
       " 'kids': 351,\n",
       " 'try': 352,\n",
       " 'production': 353,\n",
       " 'enjoy': 354,\n",
       " 'camera': 355,\n",
       " 'use': 356,\n",
       " 'wrong': 357,\n",
       " 'given': 358,\n",
       " 'low': 359,\n",
       " 'classic': 360,\n",
       " 'father': 361,\n",
       " 'need': 362,\n",
       " 'full': 363,\n",
       " 'stupid': 364,\n",
       " 'until': 365,\n",
       " 'next': 366,\n",
       " 'performances': 367,\n",
       " 'school': 368,\n",
       " 'hollywood': 369,\n",
       " 'rest': 370,\n",
       " 'truly': 371,\n",
       " 'awful': 372,\n",
       " 'video': 373,\n",
       " 'couple': 374,\n",
       " 'start': 375,\n",
       " 'sex': 376,\n",
       " 'recommend': 377,\n",
       " 'women': 378,\n",
       " 'let': 379,\n",
       " 'tell': 380,\n",
       " 'terrible': 381,\n",
       " 'remember': 382,\n",
       " 'mean': 383,\n",
       " 'came': 384,\n",
       " 'understand': 385,\n",
       " 'getting': 386,\n",
       " 'perhaps': 387,\n",
       " 'moments': 388,\n",
       " 'name': 389,\n",
       " 'keep': 390,\n",
       " 'face': 391,\n",
       " 'itself': 392,\n",
       " 'wonderful': 393,\n",
       " 'playing': 394,\n",
       " 'human': 395,\n",
       " 'style': 396,\n",
       " 'small': 397,\n",
       " 'episode': 398,\n",
       " 'perfect': 399,\n",
       " 'others': 400,\n",
       " 'person': 401,\n",
       " 'doing': 402,\n",
       " 'often': 403,\n",
       " 'early': 404,\n",
       " 'stars': 405,\n",
       " 'definitely': 406,\n",
       " 'written': 407,\n",
       " 'head': 408,\n",
       " 'lines': 409,\n",
       " 'dialogue': 410,\n",
       " 'gives': 411,\n",
       " 'piece': 412,\n",
       " \"couldn't\": 413,\n",
       " 'went': 414,\n",
       " 'finally': 415,\n",
       " 'mother': 416,\n",
       " 'case': 417,\n",
       " 'title': 418,\n",
       " 'absolutely': 419,\n",
       " 'live': 420,\n",
       " 'boy': 421,\n",
       " 'yes': 422,\n",
       " 'laugh': 423,\n",
       " 'certainly': 424,\n",
       " 'liked': 425,\n",
       " 'become': 426,\n",
       " 'entertaining': 427,\n",
       " 'worse': 428,\n",
       " 'oh': 429,\n",
       " 'sort': 430,\n",
       " 'loved': 431,\n",
       " 'lost': 432,\n",
       " 'hope': 433,\n",
       " 'called': 434,\n",
       " 'picture': 435,\n",
       " 'felt': 436,\n",
       " 'overall': 437,\n",
       " 'entire': 438,\n",
       " 'several': 439,\n",
       " 'mr': 440,\n",
       " 'based': 441,\n",
       " 'supposed': 442,\n",
       " 'cinema': 443,\n",
       " 'friend': 444,\n",
       " 'guys': 445,\n",
       " 'sound': 446,\n",
       " '5': 447,\n",
       " 'problem': 448,\n",
       " 'drama': 449,\n",
       " 'against': 450,\n",
       " 'waste': 451,\n",
       " 'white': 452,\n",
       " 'beginning': 453,\n",
       " '4': 454,\n",
       " 'fans': 455,\n",
       " 'totally': 456,\n",
       " 'dark': 457,\n",
       " 'care': 458,\n",
       " 'direction': 459,\n",
       " 'humor': 460,\n",
       " 'wanted': 461,\n",
       " \"she's\": 462,\n",
       " 'seemed': 463,\n",
       " 'under': 464,\n",
       " 'game': 465,\n",
       " 'children': 466,\n",
       " 'despite': 467,\n",
       " 'lives': 468,\n",
       " 'lead': 469,\n",
       " 'guess': 470,\n",
       " 'example': 471,\n",
       " 'already': 472,\n",
       " 'final': 473,\n",
       " 'throughout': 474,\n",
       " \"you'll\": 475,\n",
       " 'turn': 476,\n",
       " 'evil': 477,\n",
       " 'becomes': 478,\n",
       " 'unfortunately': 479,\n",
       " 'able': 480,\n",
       " 'quality': 481,\n",
       " \"i'd\": 482,\n",
       " 'days': 483,\n",
       " 'history': 484,\n",
       " 'fine': 485,\n",
       " 'side': 486,\n",
       " 'wants': 487,\n",
       " 'heart': 488,\n",
       " 'horrible': 489,\n",
       " 'writing': 490,\n",
       " 'amazing': 491,\n",
       " 'b': 492,\n",
       " 'flick': 493,\n",
       " 'killer': 494,\n",
       " 'run': 495,\n",
       " 'son': 496,\n",
       " '\\x96': 497,\n",
       " 'michael': 498,\n",
       " 'works': 499,\n",
       " 'close': 500,\n",
       " \"they're\": 501,\n",
       " 'act': 502,\n",
       " 'art': 503,\n",
       " 'kill': 504,\n",
       " 'matter': 505,\n",
       " 'etc': 506,\n",
       " 'tries': 507,\n",
       " \"won't\": 508,\n",
       " 'past': 509,\n",
       " 'town': 510,\n",
       " 'turns': 511,\n",
       " 'enjoyed': 512,\n",
       " 'brilliant': 513,\n",
       " 'gave': 514,\n",
       " 'behind': 515,\n",
       " 'parts': 516,\n",
       " 'stuff': 517,\n",
       " 'genre': 518,\n",
       " 'eyes': 519,\n",
       " 'car': 520,\n",
       " 'favorite': 521,\n",
       " 'directed': 522,\n",
       " 'late': 523,\n",
       " 'hand': 524,\n",
       " 'expect': 525,\n",
       " 'soon': 526,\n",
       " 'hour': 527,\n",
       " 'obviously': 528,\n",
       " 'themselves': 529,\n",
       " 'sometimes': 530,\n",
       " 'killed': 531,\n",
       " 'thinking': 532,\n",
       " 'actress': 533,\n",
       " 'child': 534,\n",
       " 'girls': 535,\n",
       " 'viewer': 536,\n",
       " 'starts': 537,\n",
       " 'city': 538,\n",
       " 'myself': 539,\n",
       " 'decent': 540,\n",
       " 'highly': 541,\n",
       " 'stop': 542,\n",
       " 'type': 543,\n",
       " 'self': 544,\n",
       " 'god': 545,\n",
       " 'says': 546,\n",
       " 'group': 547,\n",
       " 'anyway': 548,\n",
       " 'voice': 549,\n",
       " 'took': 550,\n",
       " 'known': 551,\n",
       " 'blood': 552,\n",
       " 'kid': 553,\n",
       " 'heard': 554,\n",
       " 'happens': 555,\n",
       " 'except': 556,\n",
       " 'fight': 557,\n",
       " 'feeling': 558,\n",
       " 'experience': 559,\n",
       " 'coming': 560,\n",
       " 'slow': 561,\n",
       " 'daughter': 562,\n",
       " 'writer': 563,\n",
       " 'stories': 564,\n",
       " 'moment': 565,\n",
       " 'told': 566,\n",
       " 'leave': 567,\n",
       " 'extremely': 568,\n",
       " 'score': 569,\n",
       " 'violence': 570,\n",
       " 'involved': 571,\n",
       " 'police': 572,\n",
       " 'strong': 573,\n",
       " 'chance': 574,\n",
       " 'lack': 575,\n",
       " 'cannot': 576,\n",
       " 'hit': 577,\n",
       " 'hilarious': 578,\n",
       " 'roles': 579,\n",
       " 's': 580,\n",
       " 'wonder': 581,\n",
       " 'happen': 582,\n",
       " 'particularly': 583,\n",
       " 'ok': 584,\n",
       " 'including': 585,\n",
       " 'save': 586,\n",
       " 'living': 587,\n",
       " 'looked': 588,\n",
       " \"wouldn't\": 589,\n",
       " 'crap': 590,\n",
       " 'simple': 591,\n",
       " 'please': 592,\n",
       " 'murder': 593,\n",
       " 'cool': 594,\n",
       " 'obvious': 595,\n",
       " 'happened': 596,\n",
       " 'complete': 597,\n",
       " 'cut': 598,\n",
       " 'serious': 599,\n",
       " 'age': 600,\n",
       " 'gore': 601,\n",
       " 'attempt': 602,\n",
       " 'hell': 603,\n",
       " 'ago': 604,\n",
       " 'song': 605,\n",
       " 'shown': 606,\n",
       " 'taken': 607,\n",
       " 'english': 608,\n",
       " 'james': 609,\n",
       " 'robert': 610,\n",
       " 'david': 611,\n",
       " 'seriously': 612,\n",
       " 'released': 613,\n",
       " 'reality': 614,\n",
       " 'opening': 615,\n",
       " 'interest': 616,\n",
       " 'jokes': 617,\n",
       " 'across': 618,\n",
       " 'none': 619,\n",
       " 'hero': 620,\n",
       " 'today': 621,\n",
       " 'possible': 622,\n",
       " 'exactly': 623,\n",
       " 'alone': 624,\n",
       " 'sad': 625,\n",
       " 'brother': 626,\n",
       " 'number': 627,\n",
       " 'career': 628,\n",
       " 'saying': 629,\n",
       " \"film's\": 630,\n",
       " 'usually': 631,\n",
       " 'hours': 632,\n",
       " 'cinematography': 633,\n",
       " 'talent': 634,\n",
       " 'view': 635,\n",
       " 'yourself': 636,\n",
       " 'running': 637,\n",
       " 'annoying': 638,\n",
       " 'relationship': 639,\n",
       " 'documentary': 640,\n",
       " 'wish': 641,\n",
       " 'huge': 642,\n",
       " 'order': 643,\n",
       " 'whose': 644,\n",
       " 'shots': 645,\n",
       " 'ridiculous': 646,\n",
       " 'taking': 647,\n",
       " 'important': 648,\n",
       " 'light': 649,\n",
       " 'body': 650,\n",
       " 'middle': 651,\n",
       " 'level': 652,\n",
       " 'ends': 653,\n",
       " 'started': 654,\n",
       " 'female': 655,\n",
       " 'call': 656,\n",
       " \"i'll\": 657,\n",
       " 'husband': 658,\n",
       " 'four': 659,\n",
       " 'power': 660,\n",
       " 'turned': 661,\n",
       " 'major': 662,\n",
       " 'word': 663,\n",
       " 'opinion': 664,\n",
       " 'change': 665,\n",
       " 'mostly': 666,\n",
       " 'usual': 667,\n",
       " 'silly': 668,\n",
       " 'scary': 669,\n",
       " 'rating': 670,\n",
       " 'beyond': 671,\n",
       " 'somewhat': 672,\n",
       " 'ones': 673,\n",
       " 'happy': 674,\n",
       " 'words': 675,\n",
       " 'room': 676,\n",
       " 'knows': 677,\n",
       " 'knew': 678,\n",
       " 'country': 679,\n",
       " 'disappointed': 680,\n",
       " 'talking': 681,\n",
       " 'novel': 682,\n",
       " 'apparently': 683,\n",
       " 'non': 684,\n",
       " 'strange': 685,\n",
       " 'attention': 686,\n",
       " 'upon': 687,\n",
       " 'basically': 688,\n",
       " 'single': 689,\n",
       " 'finds': 690,\n",
       " 'cheap': 691,\n",
       " 'modern': 692,\n",
       " 'due': 693,\n",
       " 'jack': 694,\n",
       " 'musical': 695,\n",
       " 'television': 696,\n",
       " 'problems': 697,\n",
       " 'miss': 698,\n",
       " 'episodes': 699,\n",
       " 'clearly': 700,\n",
       " 'local': 701,\n",
       " '7': 702,\n",
       " 'british': 703,\n",
       " 'thriller': 704,\n",
       " 'talk': 705,\n",
       " 'events': 706,\n",
       " 'sequence': 707,\n",
       " 'five': 708,\n",
       " \"aren't\": 709,\n",
       " 'class': 710,\n",
       " 'french': 711,\n",
       " 'moving': 712,\n",
       " 'ten': 713,\n",
       " 'fast': 714,\n",
       " 'earth': 715,\n",
       " 'review': 716,\n",
       " 'tells': 717,\n",
       " 'predictable': 718,\n",
       " 'songs': 719,\n",
       " 'team': 720,\n",
       " 'comic': 721,\n",
       " 'straight': 722,\n",
       " '8': 723,\n",
       " 'whether': 724,\n",
       " 'die': 725,\n",
       " 'add': 726,\n",
       " 'dialog': 727,\n",
       " 'entertainment': 728,\n",
       " 'above': 729,\n",
       " 'sets': 730,\n",
       " 'future': 731,\n",
       " 'enjoyable': 732,\n",
       " 'appears': 733,\n",
       " 'near': 734,\n",
       " 'space': 735,\n",
       " 'easily': 736,\n",
       " 'hate': 737,\n",
       " 'soundtrack': 738,\n",
       " 'bring': 739,\n",
       " 'giving': 740,\n",
       " 'lots': 741,\n",
       " 'romantic': 742,\n",
       " 'similar': 743,\n",
       " 'george': 744,\n",
       " 'supporting': 745,\n",
       " 'release': 746,\n",
       " 'mention': 747,\n",
       " 'filmed': 748,\n",
       " 'within': 749,\n",
       " 'message': 750,\n",
       " 'sequel': 751,\n",
       " 'clear': 752,\n",
       " 'falls': 753,\n",
       " \"haven't\": 754,\n",
       " 'needs': 755,\n",
       " 'dull': 756,\n",
       " 'suspense': 757,\n",
       " 'bunch': 758,\n",
       " 'eye': 759,\n",
       " 'surprised': 760,\n",
       " 'showing': 761,\n",
       " 'sorry': 762,\n",
       " 'tried': 763,\n",
       " 'certain': 764,\n",
       " 'working': 765,\n",
       " 'easy': 766,\n",
       " 'ways': 767,\n",
       " 'theme': 768,\n",
       " 'theater': 769,\n",
       " 'named': 770,\n",
       " 'among': 771,\n",
       " \"what's\": 772,\n",
       " 'storyline': 773,\n",
       " 'monster': 774,\n",
       " 'king': 775,\n",
       " 'stay': 776,\n",
       " 'effort': 777,\n",
       " 'fall': 778,\n",
       " 'minute': 779,\n",
       " 'stand': 780,\n",
       " 'gone': 781,\n",
       " 'rock': 782,\n",
       " 'using': 783,\n",
       " '9': 784,\n",
       " 'feature': 785,\n",
       " 'buy': 786,\n",
       " 'comments': 787,\n",
       " \"'\": 788,\n",
       " 't': 789,\n",
       " 'typical': 790,\n",
       " 'sister': 791,\n",
       " 'editing': 792,\n",
       " 'tale': 793,\n",
       " 'avoid': 794,\n",
       " 'deal': 795,\n",
       " 'dr': 796,\n",
       " 'mystery': 797,\n",
       " 'doubt': 798,\n",
       " 'fantastic': 799,\n",
       " 'kept': 800,\n",
       " 'nearly': 801,\n",
       " 'feels': 802,\n",
       " 'subject': 803,\n",
       " 'okay': 804,\n",
       " 'viewing': 805,\n",
       " 'elements': 806,\n",
       " 'oscar': 807,\n",
       " 'check': 808,\n",
       " 'realistic': 809,\n",
       " 'points': 810,\n",
       " 'means': 811,\n",
       " 'greatest': 812,\n",
       " 'herself': 813,\n",
       " 'parents': 814,\n",
       " 'famous': 815,\n",
       " 'imagine': 816,\n",
       " 'rent': 817,\n",
       " 'viewers': 818,\n",
       " 'richard': 819,\n",
       " 'crime': 820,\n",
       " 'form': 821,\n",
       " 'peter': 822,\n",
       " 'actual': 823,\n",
       " 'lady': 824,\n",
       " 'general': 825,\n",
       " 'dog': 826,\n",
       " 'follow': 827,\n",
       " 'believable': 828,\n",
       " 'period': 829,\n",
       " 'red': 830,\n",
       " 'brought': 831,\n",
       " 'move': 832,\n",
       " 'material': 833,\n",
       " 'forget': 834,\n",
       " 'somehow': 835,\n",
       " 'begins': 836,\n",
       " 're': 837,\n",
       " 'reviews': 838,\n",
       " 'animation': 839,\n",
       " 'paul': 840,\n",
       " \"you've\": 841,\n",
       " 'leads': 842,\n",
       " 'weak': 843,\n",
       " 'figure': 844,\n",
       " 'surprise': 845,\n",
       " 'sit': 846,\n",
       " 'hear': 847,\n",
       " 'average': 848,\n",
       " 'open': 849,\n",
       " 'sequences': 850,\n",
       " 'atmosphere': 851,\n",
       " 'killing': 852,\n",
       " 'eventually': 853,\n",
       " 'learn': 854,\n",
       " 'tom': 855,\n",
       " 'premise': 856,\n",
       " 'wait': 857,\n",
       " '20': 858,\n",
       " 'sci': 859,\n",
       " 'deep': 860,\n",
       " 'fi': 861,\n",
       " 'expected': 862,\n",
       " 'whatever': 863,\n",
       " 'indeed': 864,\n",
       " 'particular': 865,\n",
       " 'note': 866,\n",
       " 'lame': 867,\n",
       " 'poorly': 868,\n",
       " 'imdb': 869,\n",
       " 'dance': 870,\n",
       " 'situation': 871,\n",
       " 'shame': 872,\n",
       " 'third': 873,\n",
       " 'york': 874,\n",
       " 'box': 875,\n",
       " 'truth': 876,\n",
       " 'decided': 877,\n",
       " 'free': 878,\n",
       " 'hot': 879,\n",
       " \"who's\": 880,\n",
       " 'difficult': 881,\n",
       " 'season': 882,\n",
       " 'needed': 883,\n",
       " 'acted': 884,\n",
       " 'leaves': 885,\n",
       " 'unless': 886,\n",
       " 'emotional': 887,\n",
       " 'romance': 888,\n",
       " 'possibly': 889,\n",
       " 'sexual': 890,\n",
       " 'gay': 891,\n",
       " 'boys': 892,\n",
       " 'footage': 893,\n",
       " 'write': 894,\n",
       " 'western': 895,\n",
       " 'forced': 896,\n",
       " 'credits': 897,\n",
       " 'doctor': 898,\n",
       " 'reading': 899,\n",
       " 'became': 900,\n",
       " 'memorable': 901,\n",
       " 'otherwise': 902,\n",
       " 'air': 903,\n",
       " 'de': 904,\n",
       " 'begin': 905,\n",
       " 'crew': 906,\n",
       " 'question': 907,\n",
       " 'meet': 908,\n",
       " 'society': 909,\n",
       " 'male': 910,\n",
       " 'meets': 911,\n",
       " \"let's\": 912,\n",
       " 'plus': 913,\n",
       " 'cheesy': 914,\n",
       " 'hands': 915,\n",
       " 'superb': 916,\n",
       " 'screenplay': 917,\n",
       " 'interested': 918,\n",
       " 'beauty': 919,\n",
       " 'street': 920,\n",
       " 'features': 921,\n",
       " 'masterpiece': 922,\n",
       " 'perfectly': 923,\n",
       " 'whom': 924,\n",
       " 'laughs': 925,\n",
       " 'nature': 926,\n",
       " 'stage': 927,\n",
       " 'effect': 928,\n",
       " 'comment': 929,\n",
       " 'forward': 930,\n",
       " 'nor': 931,\n",
       " 'previous': 932,\n",
       " 'badly': 933,\n",
       " 'sounds': 934,\n",
       " 'e': 935,\n",
       " 'japanese': 936,\n",
       " 'weird': 937,\n",
       " 'island': 938,\n",
       " 'personal': 939,\n",
       " 'inside': 940,\n",
       " 'quickly': 941,\n",
       " 'total': 942,\n",
       " 'keeps': 943,\n",
       " 'towards': 944,\n",
       " 'america': 945,\n",
       " 'result': 946,\n",
       " 'battle': 947,\n",
       " 'crazy': 948,\n",
       " 'worked': 949,\n",
       " 'incredibly': 950,\n",
       " 'setting': 951,\n",
       " 'background': 952,\n",
       " 'earlier': 953,\n",
       " 'mess': 954,\n",
       " 'cop': 955,\n",
       " 'writers': 956,\n",
       " 'fire': 957,\n",
       " 'copy': 958,\n",
       " 'unique': 959,\n",
       " 'realize': 960,\n",
       " 'dumb': 961,\n",
       " 'powerful': 962,\n",
       " 'lee': 963,\n",
       " 'mark': 964,\n",
       " 'business': 965,\n",
       " 'rate': 966,\n",
       " 'older': 967,\n",
       " 'dramatic': 968,\n",
       " 'pay': 969,\n",
       " 'following': 970,\n",
       " 'directors': 971,\n",
       " 'joke': 972,\n",
       " 'girlfriend': 973,\n",
       " 'plenty': 974,\n",
       " 'directing': 975,\n",
       " 'various': 976,\n",
       " 'creepy': 977,\n",
       " 'baby': 978,\n",
       " 'development': 979,\n",
       " 'appear': 980,\n",
       " 'brings': 981,\n",
       " 'front': 982,\n",
       " 'dream': 983,\n",
       " 'ask': 984,\n",
       " 'water': 985,\n",
       " 'bill': 986,\n",
       " 'rich': 987,\n",
       " 'admit': 988,\n",
       " 'apart': 989,\n",
       " 'joe': 990,\n",
       " 'fairly': 991,\n",
       " 'political': 992,\n",
       " 'leading': 993,\n",
       " 'reasons': 994,\n",
       " 'spent': 995,\n",
       " 'portrayed': 996,\n",
       " 'telling': 997,\n",
       " 'cover': 998,\n",
       " 'outside': 999,\n",
       " 'fighting': 1000,\n",
       " ...}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.word_index\n",
    "#. This is ordered by the number of occurrences of the words in the data-set.\n",
    "#These integer-numbers are called word indices or \"tokens\" because they uniquely \n",
    "#identify each word in the vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_tokens = tokenizer.texts_to_sequences(x_train_text) # converting all the text in training data to tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'DIG! is funny, fun, amusing, interesting, stylish, and very well done. Knowing that it was made on such a shoestring budget over 7 years it is amazing that such a story can be told, especially with such style and substance. If you are a music fan or documentary fan this is a must see.<br /><br />Focusing on The Brian Jonestown Masssacre and The Dandy Warhols over the years is a brilliant way to show the contrast between a decent band who meets with moderate success through perseverance and the ability to compromise and a genius megalomaniacal lead singer backed up by a varied cast of characters who sabotage their own success through drugs, alcohol, and insanity. If I did not know that this is footage of real people, I would swear it was an incredibly well written and imaginative scripted piece. The story is compelling, concise, and simply amazing.'"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_text[1] # actual text without tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3574,    6,  152,  245, 1143,  218, 3138,    2,   52,   69,  221,\n",
       "       1332,   12,    9,   13,   90,   20,  138,    3,  332,  121,  702,\n",
       "        153,    9,    6,  491,   12,  138,    3,   64,   67,   26,  566,\n",
       "        261,   16,  138,  396,    2, 2389,   43,   22,   23,    3,  207,\n",
       "        326,   38,  640,  326,   11,    6,    3,  206,   63,    7,    7,\n",
       "       4252,   20,    1, 1819,    2,    1, 8230,  121,    1,  153,    6,\n",
       "          3,  513,   95,    5,  119,    1, 2206,  201,    3,  540, 1144,\n",
       "         35,  911,   16, 1007,  140,    2,    1, 1239,    5, 9498,    2,\n",
       "          3, 1209,  469, 1843, 7720,   53,   31,    3, 7201,  174,    4,\n",
       "        102,   35, 9597,   65,  199, 1007,  140, 1479, 4805,    2, 5147,\n",
       "         43,   10,  115,   21,  118,   12,   11,    6,  893,    4,  144,\n",
       "         83,   10,   58, 3732,    9,   13,   32,  950,   69,  407,    2,\n",
       "       3233, 3677,  412,    1,   64,    6, 1488,    2,  330,  491])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(x_train_tokens[1]) # text after tokenizing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test_tokens = tokenizer.texts_to_sequences(x_test_text) # converting text data into tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Padding\n",
    "\n",
    "The Recurrent Neural Network can take sequences of arbitrary length as input, but in order to use a whole batch of data, the sequences need to have the same length. \n",
    "But we can't take the length of longest review and pad that many zeros to the shorter reviews because it will take lot of memory so we have to figure out a particular length that will be sufficent for most of our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_tokens = np.array([len(tokens) for tokens in x_train_tokens + x_test_tokens])\n",
    "#making a list to store the lengths of tokenized reviews in both training and test data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "221.27716000000001"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(num_tokens) #calculating average length "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2208"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.max(num_tokens) # maximum length of any tokenized review"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.min(num_tokens)# minimum"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing Token Lengths"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~royal240596/60.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import plotly.plotly as py\n",
    "import plotly.graph_objs as go\n",
    "\n",
    "trace0 = go.Box(\n",
    "    y=num_tokens\n",
    ")\n",
    "data=[trace0]\n",
    "py.iplot(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We can see in the above box plot that most of the token lengths are between 0 and 500 but also we have some outliers which go upto 2000+ "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "544"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_tokens = np.mean(num_tokens) + 2 * np.std(num_tokens)\n",
    "max_tokens = int(max_tokens)\n",
    "max_tokens"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#The max number of tokens we will allow is set to the average plus 2 standard deviations."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have already seen that most of our data is between 0 and 500 length but we can verify it "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'94.528 %'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str(np.sum(num_tokens < max_tokens) / len(num_tokens) * 100) +' %' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## When we pad data we need to decide where to pad the data,wether pad in the beginning or in the end\n",
    "\n",
    "#### - If we pad at the end then there might be a chance that RNN might get confused seeing lot of zeroes after processing some data \n",
    "\n",
    "#### - So we need to pad in beginning "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "pad = 'pre'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_pad = pad_sequences(x_train_tokens, maxlen=max_tokens,\n",
    "                            padding=pad, truncating=pad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_test_pad = pad_sequences(x_test_tokens, maxlen=max_tokens,\n",
    "                           padding=pad, truncating=pad)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 544)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_pad.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 544)"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_test_pad.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([3574,    6,  152,  245, 1143,  218, 3138,    2,   52,   69,  221,\n",
       "       1332,   12,    9,   13,   90,   20,  138,    3,  332,  121,  702,\n",
       "        153,    9,    6,  491,   12,  138,    3,   64,   67,   26,  566,\n",
       "        261,   16,  138,  396,    2, 2389,   43,   22,   23,    3,  207,\n",
       "        326,   38,  640,  326,   11,    6,    3,  206,   63,    7,    7,\n",
       "       4252,   20,    1, 1819,    2,    1, 8230,  121,    1,  153,    6,\n",
       "          3,  513,   95,    5,  119,    1, 2206,  201,    3,  540, 1144,\n",
       "         35,  911,   16, 1007,  140,    2,    1, 1239,    5, 9498,    2,\n",
       "          3, 1209,  469, 1843, 7720,   53,   31,    3, 7201,  174,    4,\n",
       "        102,   35, 9597,   65,  199, 1007,  140, 1479, 4805,    2, 5147,\n",
       "         43,   10,  115,   21,  118,   12,   11,    6,  893,    4,  144,\n",
       "         83,   10,   58, 3732,    9,   13,   32,  950,   69,  407,    2,\n",
       "       3233, 3677,  412,    1,   64,    6, 1488,    2,  330,  491])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(x_train_tokens[1]) # before padding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          0,    0,    0,    0,    0,    0, 3574,    6,  152,  245, 1143,\n",
       "        218, 3138,    2,   52,   69,  221, 1332,   12,    9,   13,   90,\n",
       "         20,  138,    3,  332,  121,  702,  153,    9,    6,  491,   12,\n",
       "        138,    3,   64,   67,   26,  566,  261,   16,  138,  396,    2,\n",
       "       2389,   43,   22,   23,    3,  207,  326,   38,  640,  326,   11,\n",
       "          6,    3,  206,   63,    7,    7, 4252,   20,    1, 1819,    2,\n",
       "          1, 8230,  121,    1,  153,    6,    3,  513,   95,    5,  119,\n",
       "          1, 2206,  201,    3,  540, 1144,   35,  911,   16, 1007,  140,\n",
       "          2,    1, 1239,    5, 9498,    2,    3, 1209,  469, 1843, 7720,\n",
       "         53,   31,    3, 7201,  174,    4,  102,   35, 9597,   65,  199,\n",
       "       1007,  140, 1479, 4805,    2, 5147,   43,   10,  115,   21,  118,\n",
       "         12,   11,    6,  893,    4,  144,   83,   10,   58, 3732,    9,\n",
       "         13,   32,  950,   69,  407,    2, 3233, 3677,  412,    1,   64,\n",
       "          6, 1488,    2,  330,  491], dtype=int32)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.array(x_train_pad[1]) # After Padding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_tokens_pad = np.array([len(tokens) for tokens in x_train_pad + x_test_pad])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~royal240596/62.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import plotly.plotly as py\n",
    "import plotly.graph_objs as go\n",
    "\n",
    "trace0 = go.Box(\n",
    "    y=num_tokens_pad\n",
    ")\n",
    "data=[trace0]\n",
    "py.iplot(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## We can see in the above plot that now all the data have same length (544)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Alternatives to padding\n",
    "\n",
    "### There are various options if you don't want to do padding:\n",
    "\n",
    "#### 1) Make your batch size equal to 1 i.e feed data one by one into RNN \n",
    "\n",
    "#### 2) Grouping sequences of same lengths i.e all the sequences of particular lengths like 100 or 500 together\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the Recurrent Neural Network using Keras \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first layer in the RNN is a so-called Embedding-layer which converts each integer-token into a vector of values\n",
    "Tokenized data is huge from 0 to vocaulary length (10000 in this case) and value of tokens(integer values) does not make any sense so tokens are converted into embedded vector  which is a vector that maps words with similar semantic meanings and of length usually 100-300\n",
    "\n",
    "Watch detailed explaination of Embeddings by Andrew Ng :https://www.youtube.com/watch?v=DDByc9LyMV8 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "embedding_size = 8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Embedding(input_dim=num_words,\n",
    "                   output_dim=embedding_size,\n",
    "                   input_length=max_tokens,\n",
    "                   ))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(GRU(units=16, return_sequences=True))#layer below will be processing sequences so thats why return_sequences=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(GRU(units=8, return_sequences=True))#layer below will be processing sequences so thats why return_sequences=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(GRU(units=4))# now we don't need sequences because in the next layer we will predict the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(1, activation='sigmoid'))\\\n",
    "# fully connected layer with output =1 since we will predict either positve or negative\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = Adam(lr=1e-3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "              optimizer=optimizer,\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 544, 8)            80000     \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (None, 544, 16)           1200      \n",
      "_________________________________________________________________\n",
      "gru_2 (GRU)                  (None, 544, 8)            600       \n",
      "_________________________________________________________________\n",
      "gru_3 (GRU)                  (None, 4)                 156       \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 5         \n",
      "=================================================================\n",
      "Total params: 81,961\n",
      "Trainable params: 81,961\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 23750 samples, validate on 1250 samples\n",
      "Epoch 1/3\n",
      "23750/23750 [==============================] - 421s 18ms/step - loss: 0.4581 - acc: 0.7688 - val_loss: 0.3666 - val_acc: 0.8376\n",
      "Epoch 2/3\n",
      "23750/23750 [==============================] - 409s 17ms/step - loss: 0.2644 - acc: 0.8986 - val_loss: 0.2749 - val_acc: 0.8864\n",
      "Epoch 3/3\n",
      "23750/23750 [==============================] - 376s 16ms/step - loss: 0.2031 - acc: 0.9281 - val_loss: 0.1821 - val_acc: 0.9328\n",
      "CPU times: user 59min 13s, sys: 18min 56s, total: 1h 18min 10s\n",
      "Wall time: 20min 8s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "history=model.fit(x_train_pad, y_train,\n",
    "          validation_split=0.05, epochs=3, batch_size=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~royal240596/64.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "acc = history.history['acc']\n",
    "val_acc = history.history['val_acc']\n",
    "loss = history.history['loss']\n",
    "val_loss = history.history['val_loss']\n",
    "epochs = range(1, len(acc) + 1)\n",
    "\n",
    "\n",
    "trace0 = go.Scatter(\n",
    "    x = list(epochs),\n",
    "    y = list(acc),\n",
    "    mode = 'lines',\n",
    "    name = 'training accuracy'\n",
    ")\n",
    "\n",
    "trace1= go.Scatter(\n",
    "    x = list(epochs),\n",
    "    y = list(val_acc),\n",
    "    mode = 'lines',\n",
    "    name = 'Validation accuracy'\n",
    ")\n",
    "\n",
    "layout = go.Layout( title='Training and Validation Accuracy')\n",
    "\n",
    "\n",
    "data = [trace0,trace1]\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "\n",
    "\n",
    "py.iplot(fig)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~royal240596/66.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trace0 = go.Scatter(\n",
    "    x = list(epochs),\n",
    "    y = list(loss),\n",
    "    mode = 'lines',\n",
    "    name = 'training loss'\n",
    ")\n",
    "\n",
    "trace1= go.Scatter(\n",
    "    x = list(epochs),\n",
    "    y = list(val_loss),\n",
    "    mode = 'lines',\n",
    "    name = 'Validation loss'\n",
    ")\n",
    "\n",
    "layout = go.Layout( title='Training and Validation loss')\n",
    "\n",
    "\n",
    "data = [trace0,trace1]\n",
    "fig = go.Figure(data=data, layout=layout)\n",
    "\n",
    "\n",
    "py.iplot(fig)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 86s 3ms/step\n",
      "CPU times: user 4min 39s, sys: 1min 43s, total: 6min 22s\n",
      "Wall time: 1min 25s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "result = model.evaluate(x_test_pad, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 84.70%\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy: {0:.2%}\".format(result[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Checking on unknown Real Data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### to check this I took two reviews from IMDB \n",
    "\n",
    "1) Positive review for Peaky Blinders(TV Series)\n",
    "\n",
    "2) Negative Review for  Race 3 (Indian Movie)\n",
    "\n",
    "Positive Review URL : https://www.imdb.com/review/rw2878383/?ref_=tt_urv\n",
    "\n",
    "Negative Review URL : https://www.imdb.com/title/tt7431594/reviews?ref_=tt_ql_3\n",
    "\n",
    "        \n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {},
   "outputs": [],
   "source": [
    "positive_review='''I was not expecting it to be this good,I really enjoyed all 4 episodes. \n",
    "The story is interesting,the acting is brilliant and the cinematography is just beautiful!\n",
    "I am eagerly waiting for the next episodes.When I compare Peaky Blinders to other popular TV shows that use\n",
    "sex,brutality and violence to shock the audiences and get high ratings(which they actually do)this sincere work is \n",
    "like needlework;fine,classy and detailed.The makers of this drama have not chosen the easy way,they have set off to\n",
    "make a first class period drama,that dares to be different.Cillian Murphy is at his best,I will even go as far as\n",
    "to say that this is one of the best performances I have seen of him.Sam Neil and Helen McCrory must be praised,all\n",
    "casting is perfect.Peaky Blinders sets high standards for other television dramas to follow.'''\n",
    "\n",
    "negative_review=''' I don't know what kind of mental conditions these people are suffering from, who are rating this movie\n",
    "10/10. Why couldn't they just make it simple why this whole addition of crap. Just another crappy amalgamation of\n",
    "the movies which had a better script. I just don't think Salman will make any sensible movies in which he just acts\n",
    "good and doesn't just say mindless dialogues.'''\n",
    "\n",
    "text=[positive_review,negative_review]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens = tokenizer.texts_to_sequences(text) # we need to tokenize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokens_pad = pad_sequences(tokens, maxlen=max_tokens,\n",
    "                           padding=pad, truncating='pre')\n",
    "# padding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2, 544)"
      ]
     },
     "execution_count": 143,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokens_pad.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [],
   "source": [
    "a=model.predict(tokens_pad)[0]\n",
    "b=model.predict(tokens_pad)[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Positive Review with a score of 97.52593040466309 %\n"
     ]
    }
   ],
   "source": [
    "if a > 0.60: # I am thresholding it at 60%\n",
    "    print('Positive Review with a score of {} %'.format(a[0]*100))\n",
    "else:\n",
    "    print('Negative Review ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Negative Review with a score of 2.249847538769245 %\n"
     ]
    }
   ],
   "source": [
    "if b > 0.50: # I am thresholding it at 50%\n",
    "    print('Positive Review ')\n",
    "else:\n",
    "    print('Negative Review with a score of {} %'.format(b[0]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# We can see that it is classifying pretty good "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}