lstm-sentiment-analysis-data-imbalance-keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "5b2497b3-60ee-7cd0-0625-f103214c0ed4",
    "_uuid": "b34dc51c4c60fc1cc8200129e74e7a025fd0cc42"
   },
   "source": [
    "**Created by Sanjay Raghuwanshi ** <br/>\n",
    "[Github](https://github.com/sanjay-raghu) <br/>\n",
    "[Linkedin](https://www.linkedin.com/in/sanjayiitg/) <br/>\n",
    "**Sentiment Analysis:** the process of computationally identifying and categorizing opinions expressed in a piece of text, especially in order to determine whether the writer's attitude towards a particular topic, product, etc. is positive, negative, or neutral.\n",
    "\n",
    "**What's New** I have added how to deal with data imbalance. Almost all classification task have this problem as number of data of every class if different. For current dataset number of data having positive sentiments is very low relative to data with negative sentiment.\n",
    "\n",
    "**Solving class imbalaned data**:\n",
    "- upsampling \n",
    "- using class weighted loss function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "39c44f0e-d62c-7e11-a542-4fcd58e21442",
    "_uuid": "4efef6a6c3143fbb6bca5903fc1a764bbbb861c4"
   },
   "source": [
    "Using LSTM to classify the movie reviews into positive and negative.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "6c53202d-5c34-4859-e7e9-8ef5c7068287",
    "_uuid": "717bb968c36b9325c7d4cae5724a3672e49ff243"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# This Python 3 environment comes with many helpful analytics libraries installed\n",
    "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
    "# For example, here's several helpful packages to load in \n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "from keras.preprocessing.sequence import pad_sequences\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Embedding, LSTM, SpatialDropout1D\n",
    "from sklearn.model_selection import train_test_split\n",
    "from keras.utils.np_utils import to_categorical\n",
    "from sklearn.utils import resample\n",
    "from sklearn.utils import shuffle\n",
    "from sklearn.metrics import confusion_matrix,classification_report\n",
    "import re\n",
    "\n",
    "# Input data files are available in the \"../input/\" directory.\n",
    "# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "2bc2702e-d6f4-df5f-b80e-50ab23a6d29e",
    "_uuid": "9b520acffb5cd85d0e1ada968ad0f12cee33a4b5"
   },
   "source": [
    "Only keeping the necessary columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "89c8c923-c0bf-7b35-9ab8-e63f00b74e5a",
    "_uuid": "d2bc3bbd2ea3961c49e6673145a0a7226c160e58"
   },
   "outputs": [],
   "source": [
    "data = pd.read_csv('../input/Sentiment.csv')\n",
    "# Keeping only the neccessary columns\n",
    "data = data[['text','sentiment']]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "68989f3bc936825f4425d2d08467ce17c4a2f092"
   },
   "source": [
    "Data preview"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "_uuid": "f6a102c71c8e281450f7e73a5678cc9d0bb99e99"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>RT @NancyLeeGrahn: How did everyone feel about...</td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>RT @ScottWalker: Didn't catch the full #GOPdeb...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>RT @TJMShow: No mention of Tamir Rice and the ...</td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>RT @RobGeorge: That Carly Fiorina is trending ...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>RT @DanScavino: #GOPDebate w/ @realDonaldTrump...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text sentiment\n",
       "0  RT @NancyLeeGrahn: How did everyone feel about...   Neutral\n",
       "1  RT @ScottWalker: Didn't catch the full #GOPdeb...  Positive\n",
       "2  RT @TJMShow: No mention of Tamir Rice and the ...   Neutral\n",
       "3  RT @RobGeorge: That Carly Fiorina is trending ...  Positive\n",
       "4  RT @DanScavino: #GOPDebate w/ @realDonaldTrump...  Positive"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "4c0ec63b-cdf8-8e29-812b-0fbbfcea2929",
    "_uuid": "ff12d183224670f9c4c96fd24581b9924d4dff20"
   },
   "source": [
    "Next, I am dropping the 'Neutral' sentiments as my goal was to only differentiate positive and negative tweets. After that, I am filtering the tweets so only valid texts and words remain.  Then, I define the number of max features as 2000 and use Tokenizer to vectorize and convert text into Sequences so the Network can deal with it as input."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "_cell_guid": "43632d2d-6160-12ce-48b0-e5eb1c207076",
    "_uuid": "d0f8b4542106a279f7398db7285ae5e370b2e813"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>rt scottwalker didnt catch the full gopdebate ...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>rt robgeorge that carly fiorina is trending  h...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>rt danscavino gopdebate w realdonaldtrump deli...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>rt gregabbott_tx tedcruz on my first day i wil...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>rt warriorwoman91 i liked her and was happy wh...</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text sentiment\n",
       "1  rt scottwalker didnt catch the full gopdebate ...  Positive\n",
       "3  rt robgeorge that carly fiorina is trending  h...  Positive\n",
       "4  rt danscavino gopdebate w realdonaldtrump deli...  Positive\n",
       "5  rt gregabbott_tx tedcruz on my first day i wil...  Positive\n",
       "6  rt warriorwoman91 i liked her and was happy wh...  Negative"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data[data.sentiment != \"Neutral\"]\n",
    "data['text'] = data['text'].apply(lambda x: x.lower())\n",
    "# removing special chars\n",
    "data['text'] = data['text'].apply((lambda x: re.sub('[^a-zA-z0-9\\s]','',x)))\n",
    "#\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "_cell_guid": "43632d2d-6160-12ce-48b0-e5eb1c207076",
    "_uuid": "d0f8b4542106a279f7398db7285ae5e370b2e813"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4472\n",
      "16986\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>scottwalker didnt catch the full gopdebate la...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>robgeorge that carly fiorina is trending  hou...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>danscavino gopdebate w realdonaldtrump delive...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>gregabbott_tx tedcruz on my first day i will ...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>warriorwoman91 i liked her and was happy when...</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text sentiment\n",
       "1   scottwalker didnt catch the full gopdebate la...  Positive\n",
       "3   robgeorge that carly fiorina is trending  hou...  Positive\n",
       "4   danscavino gopdebate w realdonaldtrump delive...  Positive\n",
       "5   gregabbott_tx tedcruz on my first day i will ...  Positive\n",
       "6   warriorwoman91 i liked her and was happy when...  Negative"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(data[ data['sentiment'] == 'Positive'].size)\n",
    "print(data[ data['sentiment'] == 'Negative'].size)\n",
    "\n",
    "for idx,row in data.iterrows():\n",
    "    row[0] = row[0].replace('rt','')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "_cell_guid": "43632d2d-6160-12ce-48b0-e5eb1c207076",
    "_uuid": "d0f8b4542106a279f7398db7285ae5e370b2e813"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "         359,  120,    1,  692,    2,   39,   58,  234,   37,  207,    6,\n",
       "         172, 1745,   12, 1308, 1394,  733],\n",
       "       [   0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,\n",
       "          16,  281,  249,    5,  809,  102,  170,   26,  134,    6,    1,\n",
       "         171,   12,    2,  231,  713,   17]], dtype=int32)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "    \n",
    "max_fatures = 2000\n",
    "tokenizer = Tokenizer(num_words=max_fatures, split=' ')\n",
    "tokenizer.fit_on_texts(data['text'].values)\n",
    "X = tokenizer.texts_to_sequences(data['text'].values)\n",
    "X = pad_sequences(X)\n",
    "X[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "9753421e-1303-77d5-b17f-5f25fa08c452",
    "_uuid": "aa7d103e946e631133d86ef3adc73e1a8b1a1e89"
   },
   "source": [
    "Next, I compose the LSTM Network. Note that **embed_dim**, **lstm_out**, **batch_size**, **droupout_x** variables are hyperparameters, their values are somehow intuitive, can be and must be played with in order to achieve good results. Please also note that I am using softmax as activation function. The reason is that our Network is using categorical crossentropy, and softmax is just the right activation method for that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "_cell_guid": "1ba3cf60-a83c-9c21-05e0-b14303027e93",
    "_uuid": "05cb9ef0ec9e0a4067e3ab7c1bda7b2c1211feda",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, 28, 128)           256000    \n",
      "_________________________________________________________________\n",
      "spatial_dropout1d_1 (Spatial (None, 28, 128)           0         \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 196)               254800    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2)                 394       \n",
      "=================================================================\n",
      "Total params: 511,194\n",
      "Trainable params: 511,194\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "embed_dim = 128\n",
    "lstm_out = 196\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_fatures, embed_dim,input_length = X.shape[1]))\n",
    "model.add(SpatialDropout1D(0.4))\n",
    "model.add(LSTM(lstm_out, dropout=0.2, recurrent_dropout=0.2))\n",
    "model.add(Dense(2,activation='softmax'))\n",
    "model.compile(loss = 'categorical_crossentropy', optimizer='adam',metrics = ['accuracy'])\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "15f4ee61-47e4-88c4-4b81-98a85237333f",
    "_uuid": "2dae0f3b95a4ba533453c512e573560a8358e162"
   },
   "source": [
    "Hereby I declare the train and test dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "_cell_guid": "b35748b8-2353-3db2-e571-5fd22bb93eb0",
    "_uuid": "a380bbfae2d098d407b138fc44622c9913a31c07"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(8583, 28) (8583, 2)\n",
      "(2146, 28) (2146, 2)\n"
     ]
    }
   ],
   "source": [
    "Y = pd.get_dummies(data['sentiment']).values\n",
    "X_train, X_test, Y_train, Y_test = train_test_split(X,Y, test_size = 0.20, random_state = 42)\n",
    "print(X_train.shape,Y_train.shape)\n",
    "print(X_test.shape,Y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "2a775979-a930-e627-2963-18557d7bf6e6",
    "_uuid": "8799a667a2c0254cb367c193d86e07ee36d91dd7"
   },
   "source": [
    "Here we train the Network. We should run much more than 7 epoch, but I would have to wait forever for kaggle, so it is 7 for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "_cell_guid": "d5e499ac-2eba-6ff7-8d9a-ff65eb04099b",
    "_uuid": "d0b239912cf67294a9f5af6883bb159c44318fc7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "8583/8583 [==============================] - 7s 837us/step - loss: 0.4851 - acc: 0.7949\n",
      "Epoch 2/15\n",
      "8583/8583 [==============================] - 5s 605us/step - loss: 0.3420 - acc: 0.8554\n",
      "Epoch 3/15\n",
      "8583/8583 [==============================] - 5s 604us/step - loss: 0.2960 - acc: 0.8743\n",
      "Epoch 4/15\n",
      "8583/8583 [==============================] - 5s 606us/step - loss: 0.2684 - acc: 0.8891\n",
      "Epoch 5/15\n",
      "8583/8583 [==============================] - 5s 603us/step - loss: 0.2516 - acc: 0.8974\n",
      "Epoch 6/15\n",
      "8583/8583 [==============================] - 5s 609us/step - loss: 0.2377 - acc: 0.8996\n",
      "Epoch 7/15\n",
      "8583/8583 [==============================] - 5s 614us/step - loss: 0.2260 - acc: 0.9060\n",
      "Epoch 8/15\n",
      "8583/8583 [==============================] - 5s 611us/step - loss: 0.2157 - acc: 0.9113\n",
      "Epoch 9/15\n",
      "8583/8583 [==============================] - 5s 600us/step - loss: 0.2033 - acc: 0.9179\n",
      "Epoch 10/15\n",
      "8583/8583 [==============================] - 5s 600us/step - loss: 0.1917 - acc: 0.9211\n",
      "Epoch 11/15\n",
      "8583/8583 [==============================] - 5s 601us/step - loss: 0.1880 - acc: 0.9216\n",
      "Epoch 12/15\n",
      "8583/8583 [==============================] - 5s 605us/step - loss: 0.1704 - acc: 0.9320\n",
      "Epoch 13/15\n",
      "8583/8583 [==============================] - 5s 593us/step - loss: 0.1640 - acc: 0.9331\n",
      "Epoch 14/15\n",
      "8583/8583 [==============================] - 5s 592us/step - loss: 0.1605 - acc: 0.9335\n",
      "Epoch 15/15\n",
      "8583/8583 [==============================] - 5s 597us/step - loss: 0.1544 - acc: 0.9335\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fa90c97fcf8>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "model.fit(X_train, Y_train, epochs = 15, batch_size=batch_size, verbose = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "4ebd7bc1-53c0-0e31-a0b0-b6d0a3017434",
    "_uuid": "47e99d7ed1f27a85eb01dbafc71b66b329fb1d12"
   },
   "source": [
    "Extracting a validation set, and measuring score and accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "_uuid": "4701961e44bd243e505fc2c1b53b323311ad2b80"
   },
   "outputs": [],
   "source": [
    "Y_pred = model.predict_classes(X_test,batch_size = batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "_uuid": "13d9f618b3b74f881d68aed864a26249d26e63de"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "confusion matrix [[1571  142]\n",
      " [ 215  218]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.92      0.90      1713\n",
      "           1       0.61      0.50      0.55       433\n",
      "\n",
      "   micro avg       0.83      0.83      0.83      2146\n",
      "   macro avg       0.74      0.71      0.72      2146\n",
      "weighted avg       0.82      0.83      0.83      2146\n",
      "\n"
     ]
    }
   ],
   "source": [
    "df_test = pd.DataFrame({'true': Y_test.tolist(), 'pred':Y_pred})\n",
    "df_test['true'] = df_test['true'].apply(lambda x: np.argmax(x))\n",
    "print(\"confusion matrix\",confusion_matrix(df_test.true, df_test.pred))\n",
    "print(classification_report(df_test.true, df_test.pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "018ebf39-9414-27d0-232c-a34de051feaf",
    "_uuid": "4b54f18bbf22a953c60f271c318cb076e684df9c"
   },
   "source": [
    "Finally measuring the number of correct guesses.  It is clear that finding negative tweets (**class 0**) goes very well (**recall 0.92**) for the Network but deciding whether is positive (**class 1**) is not really (**recall 0.52**). My educated guess here is that the positive training set is dramatically smaller than the negative, hence the \"bad\" results for positive tweets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "890a03c9-316e-4d55-98e1-ba29045eff6c",
    "_uuid": "cfcbefe939b72297019e221ca3f5a283974bffff"
   },
   "source": [
    "As expected accuracy for positive data is vary low compare to negative, Lets try to solve this problem.\n",
    "\n",
    "**1. Up-sample Minority Class**\n",
    "\n",
    "Up-sampling is the process of randomly duplicating observations from the minority class in order to reinforce its signal.\n",
    "There are several heuristics for doing so, but the most common way is to simply resample with replacement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "_uuid": "384ed509b57715fbc7cce5ad37802a8785603b52"
   },
   "outputs": [],
   "source": [
    "# Separate majority and minority classes\n",
    "data_majority = data[data['sentiment'] == 'Negative']\n",
    "data_minority = data[data['sentiment'] == 'Positive']\n",
    "\n",
    "bias = data_minority.shape[0]/data_majority.shape[0]\n",
    "# lets split train/test data first then \n",
    "train = pd.concat([data_majority.sample(frac=0.8,random_state=200),\n",
    "         data_minority.sample(frac=0.8,random_state=200)])\n",
    "test = pd.concat([data_majority.drop(data_majority.sample(frac=0.8,random_state=200).index),\n",
    "        data_minority.drop(data_minority.sample(frac=0.8,random_state=200).index)])\n",
    "\n",
    "train = shuffle(train)\n",
    "test = shuffle(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "_uuid": "ce10e582bd894ec271f2587ceb618a9cf8ab03c5",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "positive data in training: 1789\n",
      "negative data in training: 6794\n",
      "positive data in test: 447\n",
      "negative data in test: 1699\n"
     ]
    }
   ],
   "source": [
    "print('positive data in training:',(train.sentiment == 'Positive').sum())\n",
    "print('negative data in training:',(train.sentiment == 'Negative').sum())\n",
    "print('positive data in test:',(test.sentiment == 'Positive').sum())\n",
    "print('negative data in test:',(test.sentiment == 'Negative').sum())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "_uuid": "384ed509b57715fbc7cce5ad37802a8785603b52"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "majority class before upsample: (6794, 2)\n",
      "minority class before upsample: (1789, 2)\n",
      "After upsampling\n",
      "Positive    6794\n",
      "Negative    6794\n",
      "Name: sentiment, dtype: int64\n",
      "x_train shape: (13588, 29)\n",
      "x_test shape (2146, 29)\n"
     ]
    }
   ],
   "source": [
    "# Separate majority and minority classes in training data for upsampling \n",
    "data_majority = train[train['sentiment'] == 'Negative']\n",
    "data_minority = train[train['sentiment'] == 'Positive']\n",
    "\n",
    "print(\"majority class before upsample:\",data_majority.shape)\n",
    "print(\"minority class before upsample:\",data_minority.shape)\n",
    "\n",
    "# Upsample minority class\n",
    "data_minority_upsampled = resample(data_minority, \n",
    "                                 replace=True,     # sample with replacement\n",
    "                                 n_samples= data_majority.shape[0],    # to match majority class\n",
    "                                 random_state=123) # reproducible results\n",
    " \n",
    "# Combine majority class with upsampled minority class\n",
    "data_upsampled = pd.concat([data_majority, data_minority_upsampled])\n",
    " \n",
    "# Display new class counts\n",
    "print(\"After upsampling\\n\",data_upsampled.sentiment.value_counts(),sep = \"\")\n",
    "\n",
    "max_fatures = 2000\n",
    "tokenizer = Tokenizer(num_words=max_fatures, split=' ')\n",
    "tokenizer.fit_on_texts(data['text'].values) # training with whole data\n",
    "\n",
    "X_train = tokenizer.texts_to_sequences(data_upsampled['text'].values)\n",
    "X_train = pad_sequences(X_train,maxlen=29)\n",
    "Y_train = pd.get_dummies(data_upsampled['sentiment']).values\n",
    "print('x_train shape:',X_train.shape)\n",
    "\n",
    "X_test = tokenizer.texts_to_sequences(test['text'].values)\n",
    "X_test = pad_sequences(X_test,maxlen=29)\n",
    "Y_test = pd.get_dummies(test['sentiment']).values\n",
    "print(\"x_test shape\", X_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "_cell_guid": "1ba3cf60-a83c-9c21-05e0-b14303027e93",
    "_uuid": "05cb9ef0ec9e0a4067e3ab7c1bda7b2c1211feda",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, 29, 128)           256000    \n",
      "_________________________________________________________________\n",
      "spatial_dropout1d_2 (Spatial (None, 29, 128)           0         \n",
      "_________________________________________________________________\n",
      "lstm_2 (LSTM)                (None, 192)               246528    \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 2)                 386       \n",
      "=================================================================\n",
      "Total params: 502,914\n",
      "Trainable params: 502,914\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "# model\n",
    "embed_dim = 128\n",
    "lstm_out = 192\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_fatures, embed_dim,input_length = X_train.shape[1]))\n",
    "model.add(SpatialDropout1D(0.4))\n",
    "model.add(LSTM(lstm_out, dropout=0.4, recurrent_dropout=0.4))\n",
    "model.add(Dense(2,activation='softmax'))\n",
    "model.compile(loss = 'categorical_crossentropy', optimizer='adam',metrics = ['accuracy'])\n",
    "print(model.summary())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "2a775979-a930-e627-2963-18557d7bf6e6",
    "_uuid": "8799a667a2c0254cb367c193d86e07ee36d91dd7"
   },
   "source": [
    "Here we train the Network. We should run much more than 15 epoch, but I would have to wait forever for kaggle, so it is 15 for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "_cell_guid": "d5e499ac-2eba-6ff7-8d9a-ff65eb04099b",
    "_uuid": "d0b239912cf67294a9f5af6883bb159c44318fc7",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "13588/13588 [==============================] - 10s 700us/step - loss: 1.2697 - acc: 0.5703\n",
      "Epoch 2/15\n",
      "13588/13588 [==============================] - 9s 627us/step - loss: 0.7914 - acc: 0.7612\n",
      "Epoch 3/15\n",
      "13588/13588 [==============================] - 8s 621us/step - loss: 0.6597 - acc: 0.8159\n",
      "Epoch 4/15\n",
      "13588/13588 [==============================] - 8s 623us/step - loss: 0.5813 - acc: 0.8403\n",
      "Epoch 5/15\n",
      "13588/13588 [==============================] - 8s 621us/step - loss: 0.5450 - acc: 0.8534\n",
      "Epoch 6/15\n",
      "13588/13588 [==============================] - 8s 622us/step - loss: 0.4764 - acc: 0.8728\n",
      "Epoch 7/15\n",
      "13588/13588 [==============================] - 8s 620us/step - loss: 0.4493 - acc: 0.8817\n",
      "Epoch 8/15\n",
      "13588/13588 [==============================] - 8s 624us/step - loss: 0.4243 - acc: 0.8903\n",
      "Epoch 9/15\n",
      "13588/13588 [==============================] - 8s 624us/step - loss: 0.3913 - acc: 0.8970\n",
      "Epoch 10/15\n",
      "13588/13588 [==============================] - 8s 625us/step - loss: 0.3829 - acc: 0.9012\n",
      "Epoch 11/15\n",
      "13588/13588 [==============================] - 8s 622us/step - loss: 0.3653 - acc: 0.9062\n",
      "Epoch 12/15\n",
      "13588/13588 [==============================] - 8s 621us/step - loss: 0.3579 - acc: 0.9104\n",
      "Epoch 13/15\n",
      "13588/13588 [==============================] - 8s 619us/step - loss: 0.3393 - acc: 0.9152\n",
      "Epoch 14/15\n",
      "13588/13588 [==============================] - 8s 621us/step - loss: 0.3256 - acc: 0.9169\n",
      "Epoch 15/15\n",
      "13588/13588 [==============================] - 8s 620us/step - loss: 0.3225 - acc: 0.9185\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fa8d2663f60>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "batch_size = 128\n",
    "# also adding weights\n",
    "class_weights = {0: 1 ,\n",
    "                1: 1.6/bias }\n",
    "model.fit(X_train, Y_train, epochs = 15, batch_size=batch_size, verbose = 1,\n",
    "          class_weight=class_weights)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "_uuid": "2803c5699e0aec22463aadccd1255e63155c1b09"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "confusion matrix [[1379  320]\n",
      " [ 125  322]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.81      0.86      1699\n",
      "           1       0.50      0.72      0.59       447\n",
      "\n",
      "   micro avg       0.79      0.79      0.79      2146\n",
      "   macro avg       0.71      0.77      0.73      2146\n",
      "weighted avg       0.83      0.79      0.80      2146\n",
      "\n"
     ]
    }
   ],
   "source": [
    "Y_pred = model.predict_classes(X_test,batch_size = batch_size)\n",
    "df_test = pd.DataFrame({'true': Y_test.tolist(), 'pred':Y_pred})\n",
    "df_test['true'] = df_test['true'].apply(lambda x: np.argmax(x))\n",
    "print(\"confusion matrix\",confusion_matrix(df_test.true, df_test.pred))\n",
    "print(classification_report(df_test.true, df_test.pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "7c23e4d6b4cab5b46a58b4c2962abd7a0d29df42"
   },
   "source": [
    "So the class imbalance is reduced significantly recall value for positive tweets (Class 1) improved from 0.54 to 0.77. It is alwayes not possible to reduce it compleatly. \n",
    "\n",
    "You may also noticed that the recall value for Negative tweets also decreased from 0.90 to 0.78  but this can be improved using training model to more epocs and tuning the hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "_uuid": "085294a5409b58c2188019177041ceb1756f1826"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "13588/13588 [==============================] - 8s 621us/step - loss: 0.3102 - acc: 0.9200\n",
      "Epoch 2/15\n",
      "13588/13588 [==============================] - 8s 621us/step - loss: 0.2973 - acc: 0.9262\n",
      "Epoch 3/15\n",
      "13588/13588 [==============================] - 8s 622us/step - loss: 0.2899 - acc: 0.9290\n",
      "Epoch 4/15\n",
      "13588/13588 [==============================] - 8s 622us/step - loss: 0.2856 - acc: 0.9268\n",
      "Epoch 5/15\n",
      "13588/13588 [==============================] - 8s 622us/step - loss: 0.2863 - acc: 0.9297\n",
      "Epoch 6/15\n",
      "13588/13588 [==============================] - 8s 623us/step - loss: 0.2798 - acc: 0.9299\n",
      "Epoch 7/15\n",
      "13588/13588 [==============================] - 8s 621us/step - loss: 0.2769 - acc: 0.9310\n",
      "Epoch 8/15\n",
      "13588/13588 [==============================] - 8s 623us/step - loss: 0.2695 - acc: 0.9339\n",
      "Epoch 9/15\n",
      "13588/13588 [==============================] - 8s 619us/step - loss: 0.2647 - acc: 0.9340\n",
      "Epoch 10/15\n",
      "13588/13588 [==============================] - 8s 620us/step - loss: 0.2619 - acc: 0.9346\n",
      "Epoch 11/15\n",
      "13588/13588 [==============================] - 8s 619us/step - loss: 0.2495 - acc: 0.9374\n",
      "Epoch 12/15\n",
      "13588/13588 [==============================] - 8s 620us/step - loss: 0.2519 - acc: 0.9360\n",
      "Epoch 13/15\n",
      "13588/13588 [==============================] - 8s 621us/step - loss: 0.2479 - acc: 0.9404\n",
      "Epoch 14/15\n",
      "13588/13588 [==============================] - 8s 619us/step - loss: 0.2421 - acc: 0.9405\n",
      "Epoch 15/15\n",
      "13588/13588 [==============================] - 8s 619us/step - loss: 0.2406 - acc: 0.9417\n",
      "confusion matrix [[1372  327]\n",
      " [ 132  315]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.81      0.86      1699\n",
      "           1       0.49      0.70      0.58       447\n",
      "\n",
      "   micro avg       0.79      0.79      0.79      2146\n",
      "   macro avg       0.70      0.76      0.72      2146\n",
      "weighted avg       0.82      0.79      0.80      2146\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# running model to few more epochs\n",
    "model.fit(X_train, Y_train, epochs = 15, batch_size=batch_size, verbose = 1,\n",
    "          class_weight=class_weights)\n",
    "Y_pred = model.predict_classes(X_test,batch_size = batch_size)\n",
    "df_test = pd.DataFrame({'true': Y_test.tolist(), 'pred':Y_pred})\n",
    "df_test['true'] = df_test['true'].apply(lambda x: np.argmax(x))\n",
    "print(\"confusion matrix\",confusion_matrix(df_test.true, df_test.pred))\n",
    "print(classification_report(df_test.true, df_test.pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "_cell_guid": "24c64f46-edd1-8d0b-7c7c-ef50fd26b2fd",
    "_uuid": "d9aac68e2013b3beffb6a764cc5b85be83073e66"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0\n",
      "    0   0   0   0   0   0 381  47   1 137 464]]\n",
      "positive\n"
     ]
    }
   ],
   "source": [
    "twt = ['keep up the good work']\n",
    "#vectorizing the tweet by the pre-fitted tokenizer instance\n",
    "twt = tokenizer.texts_to_sequences(twt)\n",
    "#padding the tweet to have exactly the same shape as `embedding_2` input\n",
    "twt = pad_sequences(twt, maxlen=29, dtype='int32', value=0)\n",
    "print(twt)\n",
    "sentiment = model.predict(twt,batch_size=1,verbose = 2)[0]\n",
    "if(np.argmax(sentiment) == 0):\n",
    "    print(\"negative\")\n",
    "elif (np.argmax(sentiment) == 1):\n",
    "    print(\"positive\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "53b88b4e6b56cf0e00084e3a41506d9abb5127e0"
   },
   "source": [
    "1. ## Tuning the hyper-parameters using gridsearch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "_uuid": "ee81d1776849d65f0f4f472be81b53a497e77795"
   },
   "outputs": [],
   "source": [
    "# from sklearn.model_selection import GridSearchCV\n",
    "# from keras.models import Sequential\n",
    "# from keras.layers import Dense\n",
    "# from keras.wrappers.scikit_learn import KerasClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "_uuid": "f8258932d80c8cd64308c919b5830bdb71a2fe2a"
   },
   "outputs": [],
   "source": [
    "# # Function to create model, required for KerasClassifier\n",
    "# def create_model(dropout_rate = 0.0):\n",
    "#     # create model\n",
    "#     embed_dim = 128\n",
    "#     lstm_out = 192\n",
    "#     model = Sequential()\n",
    "#     model.add(Embedding(max_fatures, embed_dim,input_length = X_train.shape[1]))\n",
    "#     model.add(SpatialDropout1D(dropout_rate))\n",
    "#     model.add(LSTM(lstm_out, dropout=dropout_rate, recurrent_dropout=dropout_rate))\n",
    "#     model.add(Dense(2,activation='softmax'))\n",
    "#     model.compile(loss = 'categorical_crossentropy', optimizer='adam',metrics = ['accuracy'])\n",
    "# #     print(model.summary())\n",
    "#     return model\n",
    "# # fix random seed for reproducibility\n",
    "# seed = 7\n",
    "# np.random.seed(seed)\n",
    "\n",
    "# model = KerasClassifier(build_fn=create_model, verbose=2,epochs=30, batch_size=128)\n",
    "# # define the grid search parameters\n",
    "# # batch_size = [128]\n",
    "# # epochs = [5]\n",
    "# dropout_rate = [0.0, 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]\n",
    "# # class_weight = [{0: 1, 1: 1/bias},{0: 1, 1: 1.2/bias},{0: 1, 1: 1.5/bias},{0: 1, 1: 1.8/bias}]\n",
    "# param_grid = dict(dropout_rate = dropout_rate)\n",
    "# grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1)\n",
    "# grid_result = grid.fit(X_train, Y_train)\n",
    "# # summarize results\n",
    "# print(\"Best: %f using %s\" % (grid_result.best_score_, grid_result.best_params_))\n",
    "# means = grid_result.cv_results_['mean_test_score']\n",
    "# stds = grid_result.cv_results_['std_test_score']\n",
    "# params = grid_result.cv_results_['params']\n",
    "# for mean, stdev, param in zip(means, stds, params):\n",
    "#     print(\"%f (%f) with: %r\" % (mean, stdev, param))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "_uuid": "edf8e863a51e4c0d17df6add4c3f686c2a49742e"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "_uuid": "0899b20c13721be82de6f81a6882868808bb80bc"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "018ebf39-9414-27d0-232c-a34de051feaf",
    "_uuid": "4b54f18bbf22a953c60f271c318cb076e684df9c"
   },
   "source": [
    "Finally measuring the number of correct guesses.  It is clear that finding negative tweets goes very well for the Network but deciding whether is positive is not really. My educated guess here is that the positive training set is dramatically smaller than the negative, hence the \"bad\" results for positive tweets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_uuid": "f40eda29aeff51f865833c8f3d7a5d1fc52d29e5"
   },
   "source": [
    "its 10% increase in positive accuracy from 56% to 66%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "_cell_guid": "24c64f46-edd1-8d0b-7c7c-ef50fd26b2fd",
    "_uuid": "d9aac68e2013b3beffb6a764cc5b85be83073e66"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[   0    0    0    0    0    0    0    0    0    0    0    0    0    0\n",
      "     0    0    0    0    0    0    0    0    0    0 1288   43  200   14\n",
      "    74]]\n",
      "positive\n"
     ]
    }
   ],
   "source": [
    "twt = ['inaccurate facts, dont vote for him']\n",
    "#vectorizing the tweet by the pre-fitted tokenizer instance\n",
    "twt = tokenizer.texts_to_sequences(twt)\n",
    "#padding the tweet to have exactly the same shape as `embedding_2` input\n",
    "twt = pad_sequences(twt, maxlen=29, dtype='int32', value=0)\n",
    "print(twt)\n",
    "sentiment = model.predict(twt,batch_size=1,verbose = 2)[0]\n",
    "if(np.argmax(sentiment) == 0):\n",
    "    print(\"negative\")\n",
    "elif (np.argmax(sentiment) == 1):\n",
    "    print(\"positive\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "_cell_guid": "c611b55c-92e4-4a33-8e82-1812bef6193d",
    "_uuid": "8b10995b0832ec98ba0c75832186fcb09b1a2d5f",
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "_change_revision": 0,
  "_is_fork": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}