Assignment.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Logo](http://www.hva.nl/webfiles/1524744682263/img/logo.svg \"Hogeschool van Amsterdam\")\n",
    "# Data Mining & Data Analysis \n",
    "## Individual Assignment\n",
    "Student: Joost Buskermolen (500709241)\n",
    "\n",
    "Each individual student needs to show his/her data analysis and datamining skills by doing an individual assignment.This assignment is a follow-up of the assignment for the course Data Processing. For Data Mining and Data Analysis you need to build a more or less sophisticated classifier for movie reviews. The classifier should be able to classify the sentiment of the review (positive or negative).\n",
    "\n",
    "**It’s model is built with training data from both:**\n",
    "\n",
    "1. The dataset found at [Kaggle](https://www.kaggle.com/c/word2vec-nlp-tutorial/data \"Kaggle\") (In this case the dataset grabbed from VLO, converted to Excel for compatibility reasons.)\n",
    "2. An additional (large) set of reviews from another movie review website (also through Kaggle), Rottentomatoes in this case,  [which you can find by clicking this link](https://www.kaggle.com/abhipoo/sentiment-rotten-tomatoes/downloads/sentiment-rotten-tomatoes.zip/1 \"Download dataset\")\n",
    "\n",
    "**It's accuracy needs to be at least 75%**\n",
    "\n",
    "The accuracy is around ~85% and therefore above the required minimum. Run all the cells below, and the accuracy will be printed.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First, we need to import the necessary libraries into the project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import sklearn\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "from sklearn.naive_bayes import MultinomialNB\n",
    "from sklearn.metrics import accuracy_score, confusion_matrix"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. ```pandas```: I used Pandas (declared as pd in the code) as dataframe for the structured training data. With this I could easily import Excel sheets / CSV files and convert them to a usable dataframe.\n",
    "2. ```sklearn```: Short for scikit-learn, is used to analyse the training data and make predictions based on that.\n",
    "3. ```CountVectorizer```: The CountVectorizer provides a simple way to tokenize a collection of text documents and build a vocabulary of known words.\n",
    "4. ```MultinomialNB```: By using Naive Bayes, the concept of 'probability' is used to classify new entities; based on the tokenized training data.\n",
    "5. ```accuracy_score, confusion_matrix```: These functions from the metrics class are used to determine accuracy of the trained model later on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Next step, reading the Excel sheet and converting it to a dataframe, and showing the first 3 records:\n",
    "Down below I imported the by HvA supplied dataset and defined the column headers: id, sentiment & review. After that I called the head() function from pd to view the first (in this case 3) rows of the dataframe "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>5814_8</th>\n",
       "      <td>1</td>\n",
       "      <td>With all this stuff going down at the moment w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2381_9</th>\n",
       "      <td>1</td>\n",
       "      <td>\\The Classic War of the Worlds\\\" by Timothy Hi...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7759_3</th>\n",
       "      <td>0</td>\n",
       "      <td>The film starts with a manager (Nicholas Bell)...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        sentiment                                             review\n",
       "id                                                                  \n",
       "5814_8          1  With all this stuff going down at the moment w...\n",
       "2381_9          1  \\The Classic War of the Worlds\\\" by Timothy Hi...\n",
       "7759_3          0  The film starts with a manager (Nicholas Bell)..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hva_df = pd.read_excel(\"datasets/HvA_Traindata.xlsx\", header=0, index_col='id', columns=['id', 'sentiment', 'review'])\n",
    "hva_df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Check ratio between positive and negative sentiments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    12500\n",
       "0    12499\n",
       "Name: sentiment, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hva_df.sentiment.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## To meet the conditions for this assignment an extra training dataset is neccessary.\n",
    "For this I downloaded an additional (large) set of reviews and sentiment [from Kaggle](https://www.kaggle.com/abhipoo/sentiment-rotten-tomatoes/downloads/sentiment-rotten-tomatoes.zip/1 \"Download dataset\") gathered from [Rottentomatoes](https://rottentomatoes.com \"Rotten Tomatoes\"). \n",
    "\n",
    "This dataset was devided in two folders, positive (pos) and negative (neg) with in each seperate file one review. Because that's not a suitable format to import directly in a dataframe, I wrote a small script to enter each review in it's own row of a CSV file; because nobody got time to copy paste ~2000 reviews manually to a CSV. This script works as follows:\n",
    "\n",
    "1. Create a list of all filenames available in the folder, for both the positive as negative reviews.\n",
    "2. Create the file positive.csv and write the headers id, sentiment & review\n",
    "3. For each file in the list positives: assign an id, set sentiment to 1 and write it's contents to the review column\n",
    "4. When all files from the list are done, break.\n",
    "\n",
    "Follow the same steps for the negative reviews and you end up with two files: positive.csv and negative.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import os\n",
    "from os.path import isfile, join\n",
    "\n",
    "positives = [f for f in os.listdir('datasets/pos/') if isfile(join('datasets/pos/', f))]\n",
    "negatives = [f for f in os.listdir('datasets/neg/') if isfile(join('datasets/neg/', f))]\n",
    "\n",
    "id = 1\n",
    "with open('datasets/positive.csv', 'w', newline='') as f:\n",
    "    headers = ['id', 'sentiment', 'review']\n",
    "    writer = csv.DictWriter(f, fieldnames=headers)\n",
    "    \n",
    "    writer.writeheader()\n",
    "    \n",
    "    for file in positives:\n",
    "        if id != (len(positives)-1):\n",
    "            content = open('datasets/pos/' + file)\n",
    "            writer.writerow({'id' : id, 'sentiment' : 1, 'review' : content.read()})\n",
    "            id +=1\n",
    "        else:\n",
    "            break\n",
    "\n",
    "with open('datasets/negative.csv', 'w', newline='') as f:\n",
    "    headers = ['id', 'sentiment', 'review']\n",
    "    writer = csv.DictWriter(f, fieldnames=headers)\n",
    "    \n",
    "    id += 4 #dirty fix to get the right ID\n",
    "    \n",
    "    #writer.writeheader()\n",
    "    \n",
    "    for file in negatives:\n",
    "        if id != (len(negatives)+len(positives)):\n",
    "            content = open('datasets/neg/' + file)\n",
    "            writer.writerow({'id' : id, 'sentiment' : 0, 'review' : content.read()})\n",
    "            id +=1\n",
    "        else:\n",
    "            break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I ended up with two seperate files, which isn't ideal; so I merged its contents to one file: combined.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "positive = open('datasets/positive.csv', 'r')\n",
    "negative = open('datasets/negative.csv','r')\n",
    "merged = open('datasets/combined.csv', 'w')\n",
    "merged.write(positive.read() + negative.read())\n",
    "merged.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the pandas dataframe from the combined.csv file\n",
    "And display the last five rows with the tail function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentiment</th>\n",
       "      <th>review</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1998</th>\n",
       "      <td>0</td>\n",
       "      <td>movies like six days , seven nights make me ma...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1999</th>\n",
       "      <td>0</td>\n",
       "      <td>if anything , \" stigmata \" should be taken as ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2000</th>\n",
       "      <td>0</td>\n",
       "      <td>john boorman's \" zardoz \" is a goofy cinematic...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      sentiment                                             review\n",
       "id                                                                \n",
       "1998          0  movies like six days , seven nights make me ma...\n",
       "1999          0  if anything , \" stigmata \" should be taken as ...\n",
       "2000          0  john boorman's \" zardoz \" is a goofy cinematic..."
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "extra_df = pd.read_csv(\"datasets/combined.csv\", header=0, index_col='id')\n",
    "extra_df.tail(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combining the two dataframes hva_df & extra_df\n",
    "Because we want to train the classifier on both datasets, we need to 'append' one dataframe to the other. Below I appended the extra_df to hva_df and assigned it to df. For informative purposes I ran the value_counts() to get an insight about the total reviews and it's corresponding sentiments the dataframe contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    13499\n",
       "0    13496\n",
       "Name: sentiment, dtype: int64"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = hva_df.append(extra_df, ignore_index=True)\n",
    "df.sentiment.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenize dataframe (df) through CountVectorizer:\n",
    "The CountVectorizer provides a simple way to tokenize the dataframes and build a vocabulary of known words, and also predict new entries based on that vocabulary. This function from scikit-learn is so advanced, that all words are converted to lowercase and punctuation is removed automatically. \n",
    "\n",
    "(This step takes a while, depending on your hardware)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [1, 1, 0, ..., 0, 0, 0],\n",
       "       ..., \n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0],\n",
       "       [0, 0, 0, ..., 0, 0, 0]], dtype=int64)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vect = CountVectorizer(max_features=3000, binary=True)\n",
    "X = vect.fit_transform(df.review)\n",
    "X.toarray()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### To calculate accuracy, I'll split the dataframe in two unequally parts: 85% for training, 15% for testing.\n",
    "The package ```train_test_split``` from scikit-learn is used for this and the test_size is set to 0.15. This is just for accuracy calculation purposes. Later on we will undo this step to make our training set bigger again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X = df.review\n",
    "y = df.sentiment\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15)\n",
    "X_train_vect = vect.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```X_train_vect``` is now transformed into the right format to give to the Naive Bayes model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Next, we need to  instantiate a Naive Bayes model from sklearn and fit it to our training data:\n",
    "The following pieces of code will create this model and returns a fit percentage (not to be confused with accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitness: 0.851645238614\n"
     ]
    }
   ],
   "source": [
    "nb = MultinomialNB()\n",
    "nb.fit(X_train_vect, y_train)\n",
    "print(\"Fitness: \"+ str(nb.score(X_train_vect, y_train)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Now I'll vectorize the test set, and use that set to predict if each review is either positive or negative.\n",
    "This will give us the opportunity to calculate the accuracy using sklearn.metrics ```accuracy_score``` (as done at the last line of this block). I also used the ```math``` library to recalculate the value and round to one decimal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 84.5%\n"
     ]
    }
   ],
   "source": [
    "X_test_vect = vect.transform(X_test)\n",
    "y_pred = nb.predict(X_test_vect)\n",
    "import math\n",
    "acc = accuracy_score(y_test, y_pred) * 100\n",
    "print(\"Accuracy: \" + str(math.floor(acc*10)/10) + \"%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Up next, a confusion matrix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[1732,  305],\n",
       "       [ 321, 1692]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_test, y_pred, labels=[0, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Because we now know the accuracy, we don't need to split the data in train/test anymore. \n",
    "Therefore I will redefine the trainingset and build my Naive Bayes model again below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitness: 0.850009260974\n"
     ]
    }
   ],
   "source": [
    "X_vect = vect.fit_transform(X)\n",
    "nb = MultinomialNB()\n",
    "nb.fit(X_vect, y)\n",
    "print(\"Fitness: \"+ str(nb.score(X_vect, y)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last but not least, I made a function to input your own review and test the classifier:\n",
    "Enter your input in the box below and run the code till the end. You will get a result based on the trained classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter the sentence you would like to predict: Deadpool 2 was such an amazing movie, can't wait for number 3.\n",
      "Your input was positive!\n",
      "Would you like to try another review? Y/n: y\n",
      "Enter the sentence you would like to predict: Never seen such a bad movie as Movie 43. Never again.\n",
      "Your input was negative :(\n",
      "Would you like to try another review? Y/n: n\n",
      "Thank you for using Joost's sentiment analysis.\n"
     ]
    }
   ],
   "source": [
    "repeat = True\n",
    "while repeat:\n",
    "    userinput = input(\"Enter the sentence you would like to predict: \")\n",
    "    if userinput:\n",
    "        usersentiment = {'sentiment': [0], 'review' : [userinput]}\n",
    "        user_df = pd.DataFrame.from_dict(usersentiment)\n",
    "        user_df.index.name = 'id'\n",
    "        user_test_vect = vect.transform(user_df.review)\n",
    "        user_pred = nb.predict(user_test_vect)\n",
    "    else:\n",
    "        print(\"No user input was given :(\")\n",
    "    try:\n",
    "        if user_pred == 1:\n",
    "            print(\"Your input was positive!\")\n",
    "        else:\n",
    "            print(\"Your input was negative :(\")\n",
    "    except NameError:\n",
    "        print(\"No user input was given. Re-run codeblock above and give it some input.\")\n",
    "    except:\n",
    "        print(\"Something else went wrong.\")\n",
    "    repeat = str(input(\"Would you like to try another review? Y/n: \") or 'Y')\n",
    "    if repeat in ['N', 'n', 'No', 'no']:\n",
    "        print(\"Thank you for using Joost's sentiment analysis.\")\n",
    "        break\n",
    "    else:\n",
    "        pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}