MLP_with_without_keras_project.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mini Project_00: 1/0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from keras.utils import np_utils\n",
    "import tensorflow as tf\n",
    "'''\n",
    "from tensorflow.python.ops import control_flow_ops\n",
    "tensorflow.python.ops.control_flow_ops = tf #????????????????????????????????????????\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Set random seed\n",
    "np.random.seed(42)\n",
    "# Our data\n",
    "X = np.array([[0,0],[0,1],[1,0],[1,1]]).astype('float32')\n",
    "y = np.array([[0],[1],[1],[0]]).astype('float32') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# One-hot encoding the output\n",
    "y = np_utils.to_categorical(y); y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Initial Setup for Keras\n",
    "from keras.models import Sequential\n",
    "from keras.layers.core import Dense, Dropout, Activation, Flatten\n",
    "from keras.optimizers import SGD\n",
    "from keras.utils import np_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Building the model\n",
    "xor = Sequential()\n",
    "xor.add(Dense(8, input_dim=2))\n",
    "xor.add(Activation(\"sigmoid\"))\n",
    "xor.add(Dense(2))\n",
    "xor.add(Activation(\"sigmoid\"))\n",
    "\n",
    "xor.compile(loss=\"categorical_crossentropy\", optimizer=\"adam\", metrics = ['accuracy'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# print the model architecture\n",
    "xor.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Fitting the model\n",
    "history = xor.fit(X, y, nb_epoch=50, verbose=0)\n",
    "\n",
    "# Scoring the model\n",
    "score = xor.evaluate(X, y)\n",
    "print(\"\\nAccuracy: \", score[-1])\n",
    "\n",
    "# Checking the predictions\n",
    "print(\"\\nPredictions:\")\n",
    "print(xor.predict_proba(X))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "Out of 4 input points, we're correctly classifying only 3 of them? Let's try to change some parameters around to improve. For example, you can increase the number of epochs. Can you reach 100% ?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "---\n",
    "## Mini Project_01: Student Admissions\n",
    "Build a neural network which analyzes the dataset of student admissions at UCLA \n",
    "\n",
    "we predict student admissions to graduate school at UCLA based on three pieces of data:\n",
    "- GRE Scores (Test)\n",
    "- GPA Scores (Grades)\n",
    "- Class rank (1-4)\n",
    "\n",
    "The dataset originally came from here: http://www.ats.ucla.edu/\n",
    "\n",
    "## Loading the data\n",
    "To load the data and format it nicely, we will use two very useful packages called Pandas and Numpy. You can read on the documentation here:\n",
    "- https://pandas.pydata.org/pandas-docs/stable/\n",
    "- https://docs.scipy.org/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Importing pandas and numpy\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# Reading the csv file into a pandas DataFrame\n",
    "data = pd.read_csv('student_data.csv')\n",
    "\n",
    "# Printing out the first 10 rows of our data\n",
    "data[:10]\n",
    "# Here we can see that the first column is the label y, which corresponds to acceptance/rejection. Namely, a label of 1 means \n",
    "#the student got accepted, and a label of 0 means the student got rejected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# First let's make a plot of our data to see how it looks. In order to have a 2D plot, let's ingore the rank.\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "# Function to help us plot\n",
    "def plot_points(data):\n",
    "    X = np.array(data[[\"gre\",\"gpa\"]])\n",
    "    y = np.array(data[\"admit\"])\n",
    "    admitted = X[np.argwhere(y==1)]\n",
    "    rejected = X[np.argwhere(y==0)]\n",
    "    plt.scatter([s[0][0] for s in rejected], [s[0][1] for s in rejected], s = 25, color = 'red', edgecolor = 'k')\n",
    "    plt.scatter([s[0][0] for s in admitted], [s[0][1] for s in admitted], s = 25, color = 'cyan', edgecolor = 'k')\n",
    "    plt.xlabel('Test (GRE)')\n",
    "    plt.ylabel('Grades (GPA)')\n",
    "    \n",
    "# Plotting the points\n",
    "plot_points(data)\n",
    "plt.show()\n",
    "# the data is not as nicely separable as we'd hope..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Roughly, it looks like the students with high scores in the grades and test passed, while the ones with low scores didn't, \n",
    "#but the data is not as nicely separable as we hoped it would. Maybe it would help to take the rank into account? Let's make \n",
    "#4 plots, each one for each rank. So we make this one graph for each of the 4 ranks.\n",
    "\n",
    "# Separating the ranks\n",
    "data_rank1 = data[data[\"rank\"]==1]\n",
    "data_rank2 = data[data[\"rank\"]==2]\n",
    "data_rank3 = data[data[\"rank\"]==3]\n",
    "data_rank4 = data[data[\"rank\"]==4]\n",
    "\n",
    "# Plotting the graphs\n",
    "#plot_points(data_rank1)\n",
    "#plt.title(\"Rank 1\")\n",
    "#plt.show()\n",
    "#plot_points(data_rank2)\n",
    "#plt.title(\"Rank 2\")\n",
    "#plt.show()\n",
    "#plot_points(data_rank3)\n",
    "#plt.title(\"Rank 3\")\n",
    "#plt.show()\n",
    "#plot_points(data_rank4)\n",
    "#plt.title(\"Rank 4\")\n",
    "#plt.show()\n",
    "\n",
    "fig = plt.figure(figsize = (11,8))\n",
    "\n",
    "for i, rank in enumerate([data_rank1,data_rank2,data_rank3,data_rank4]):\n",
    "    ax = fig.add_subplot(2, 2, i+1)\n",
    "    plot_points(rank)\n",
    "    ax.set_title('Rank-%s'%(i+1), fontsize = 14)\n",
    "    \n",
    "fig.tight_layout()\n",
    "fig.show()\n",
    "\n",
    "\n",
    "# This looks more promising, as it seems that the lower the rank, the higher the acceptance rate. Let's use the rank as one of\n",
    "#our inputs. In order to do this, we should one-hot encode it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## TODO: One-hot encoding the rank\n",
    " It seems like the better grades and test the student has, the more likely they are to be accepted. And the rank has something to do with it. So what we'll do is, we'll one-hot encode the rank, and our 6 input variables will be: \n",
    " - Test (GPA)\n",
    " - Grades (GRE)\n",
    " - Rank 1\n",
    " - Rank 2\n",
    " - Rank 3\n",
    " - Rank 4\n",
    "#### The last 4 inputs will be binary variables that have a value of 1 if the student has that rank, or 0 otherwise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Use the get_dummies function in numpy in order to one-hot encode the data.\n",
    "\n",
    "# TODO:  Make dummy variables for rank\n",
    "#one_hot_data = pd.concat([data, pd.get_dummies(data['rank'], prefix='rank')], axis=1)\n",
    "one_hot_data = pd.get_dummies(data, columns=['rank'])\n",
    "\n",
    "# TODO: Drop the previous rank column\n",
    "#one_hot_data = one_hot_data.drop('rank', axis=1)\n",
    "\n",
    "# Print the first 5 rows of our data\n",
    "one_hot_data[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## TODO: Scaling the data\n",
    "The next step is to scale the data. So, first things first, let's notice that the **test scores** have a range of 800, while the **grades** have a range of 4. This is a huge discrepancy: We notice that the range for grades is 1.0-4.0, whereas the range for test scores is roughly 200-800, which is much larger. This means our data is skewed, and that makes it hard for a neural network to handle and it will affect our training. Normally, the best thing to do is to normalize the scores so **they are between 0 and 1**. Let's fit our two features into a range of 0-1, by dividing the grades by 4.0, and the test score by 800."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Making a copy of our data\n",
    "processed_data = one_hot_data[:]\n",
    "\n",
    "# TODO: Scale the columns\n",
    "processed_data['gre'] = processed_data['gre']/800\n",
    "processed_data['gpa'] = processed_data['gpa']/4\n",
    "\n",
    "# Printing the first 10 rows of our procesed data\n",
    "processed_data[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Splitting the data into Training and Testing\n",
    "In order to test our algorithm, we'll split the data into a Training and a Testing set. The size of the testing set will be 10% of the total data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "sample_90 = np.random.choice(processed_data.index, size=int(len(processed_data)*0.9), replace=False)\n",
    "train_data, test_data = processed_data.iloc[sample_90], processed_data.drop(sample_90) \n",
    "\n",
    "print(\"Number of training samples is\", len(train_data))\n",
    "print(\"Number of testing samples is\", len(test_data))\n",
    "print(train_data[:5])\n",
    "print(test_data[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Splitting the data into features and targets (labels)\n",
    "Now, we split our data input into X, and the labels y , and one-hot encode the output, so it appears as two classes (accepted and not accepted). As a final step before the training, we'll split the data into features (X) and targets (y). Also, in Keras, we need to one-hot encode the output. We'll do this with the \"`to_categorical function`\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#X = np.array(processed_data)[:,1:]\n",
    "#y = np_utils.to_categorical(np.array(processed_data[\"admit\"]))\n",
    "\n",
    "# Separate data and one-hot encode the output\n",
    "# Note: We're also turning the data into numpy arrays, in order to train the model in Keras\n",
    "train_X = np.array(train_data.drop('admit', axis=1))\n",
    "train_y = np.array(np_utils.to_categorical(train_data['admit'], 2))\n",
    "\n",
    "test_X = np.array(test_data.drop('admit', axis=1))\n",
    "test_y = np.array(np_utils.to_categorical(test_data['admit'], 2))\n",
    "\n",
    "#train_X = train_data.drop('admit', axis=1)\n",
    "#train_y = train_data['admit']\n",
    "\n",
    "#test_X = test_data.drop('admit', axis=1)\n",
    "#test_y = test_data['admit']\n",
    "\n",
    "print(test_X[:5])\n",
    "print(test_y[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Training the 2-layer Neural Network\n",
    "The following function trains the 2-layer neural network. First, we'll write some helper functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "\n",
    "'''\n",
    "model = Sequential()\n",
    "model.add(Dense(128, input_dim=6))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(32))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(2))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.compile(loss = 'categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model.summary()\n",
    "'''\n",
    "\n",
    "# Building the model\n",
    "model = Sequential()\n",
    "model.add(Dense(128, activation='relu', input_shape=(6,)))\n",
    "model.add(Dropout(.2))\n",
    "model.add(Dense(64, activation='relu'))\n",
    "model.add(Dropout(.1))\n",
    "model.add(Dense(2, activation='sigmoid'))  ## why not softmax ??\n",
    "\n",
    "# Compiling the model\n",
    "model.compile(loss = 'categorical_crossentropy', optimizer='adam', metrics=['accuracy'])\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Training and Scoring the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Training the model\n",
    "model.fit(train_X, train_y, nb_epoch=200, batch_size=100, verbose=0)\n",
    "\n",
    "# Evaluating the model on the training and testing set\n",
    "score = model.evaluate(train_X, train_y)\n",
    "print(\"\\n Training Accuracy:\", score[1])\n",
    "print(\"\\nPredictions:\")\n",
    "print(\"\\nProb:\", model.predict_proba(train_X)[:8])\n",
    "\n",
    "score = model.evaluate(test_X, test_y)\n",
    "print(\"\\n Testing Accuracy:\", score[1])\n",
    "print(\"\\nPredictions:\")\n",
    "print(\"\\nProb:\", model.predict_proba(test_X)[:8])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## Challenge: Play with the parameters!\n",
    "You can see that we made several decisions in our training. For instance, the number of layers, the sizes of the layers, the number of epochs, etc.\n",
    "It's your turn to play with parameters! Can you improve the accuracy? The following are other suggestions for these parameters. We'll learn the definitions later in the class:\n",
    "- Activation function: relu and sigmoid\n",
    "- Loss function: categorical_crossentropy, mean_squared_error\n",
    "- Optimizer: rmsprop, adam, ada"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# W/O keras\n",
    "## TODO: Backpropagate the error\n",
    "Now it's your turn to shine. Write the error term. Remember that this is given by the equation $$ -(y-\\hat{y}) \\sigma'(x) $$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Activation (sigmoid) function\n",
    "def sigmoid(x):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "def sigmoid_prime(x):\n",
    "    return sigmoid(x) * (1-sigmoid(x))\n",
    "def error_formula(y, output):\n",
    "    return - y*np.log(output) - (1 - y) * np.log(1-output)\n",
    "\n",
    "\n",
    "# TODO: Write the error term formula\n",
    "def error_term_formula(y, output):\n",
    "    pass\n",
    "\n",
    "# Neural Network hyperparameters\n",
    "epochs = 1000\n",
    "learnrate = 0.5\n",
    "\n",
    "# Training function\n",
    "def train_nn(features, targets, epochs, learnrate):\n",
    "    \n",
    "    # Use to same seed to make debugging easier\n",
    "    np.random.seed(42)\n",
    "\n",
    "    n_records, n_features = features.shape\n",
    "    last_loss = None\n",
    "\n",
    "    # Initialize weights\n",
    "    weights = np.random.normal(scale=1 / n_features**.5, size=n_features)\n",
    "\n",
    "    for e in range(epochs):\n",
    "        del_w = np.zeros(weights.shape)\n",
    "        for x, y in zip(features.values, targets):\n",
    "            # Loop through all records, x is the input, y is the target\n",
    "\n",
    "            # Activation of the output unit\n",
    "            #   Notice we multiply the inputs and the weights here \n",
    "            #   rather than storing h as a separate variable \n",
    "            output = sigmoid(np.dot(x, weights))\n",
    "\n",
    "            # The error, the target minus the network output\n",
    "            error = error_formula(y, output)\n",
    "\n",
    "            # The error term\n",
    "            #   Notice we calulate f'(h) here instead of defining a separate\n",
    "            #   sigmoid_prime function. This just makes it faster because we\n",
    "            #   can re-use the result of the sigmoid function stored in\n",
    "            #   the output variable\n",
    "            error_term = error_term_formula(y, output)\n",
    "\n",
    "            # The gradient descent step, the error times the gradient times the inputs\n",
    "            del_w += error_term * x\n",
    "\n",
    "        # Update the weights here. The learning rate times the \n",
    "        # change in weights, divided by the number of records to average\n",
    "        weights += learnrate * del_w / n_records\n",
    "\n",
    "        # Printing out the mean square error on the training set\n",
    "        if e % (epochs / 10) == 0:\n",
    "            out = sigmoid(np.dot(features, weights))\n",
    "            loss = np.mean((out - targets) ** 2)\n",
    "            print(\"Epoch:\", e)\n",
    "            if last_loss and last_loss < loss:\n",
    "                print(\"Train loss: \", loss, \"  WARNING - Loss Increasing\")\n",
    "            else:\n",
    "                print(\"Train loss: \", loss)\n",
    "            last_loss = loss\n",
    "            print(\"=========\")\n",
    "    print(\"Finished training!\")\n",
    "    return weights\n",
    "    \n",
    "weights = train_nn(features, targets, epochs, learnrate)\n",
    "\n",
    "\n",
    "\n",
    "# Calculate accuracy on test data\n",
    "tes_out = sigmoid(np.dot(features_test, weights))\n",
    "predictions = tes_out > 0.5\n",
    "accuracy = np.mean(predictions == targets_test)\n",
    "print(\"Prediction accuracy: {:.3f}\".format(accuracy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "---\n",
    "## Mini Project_02. Analyzing IMDB Data in Keras\n",
    "we will analyze a dataset from IMDB and use it to predict the sentiment analysis of a review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "import numpy as np\n",
    "import keras\n",
    "from keras.datasets import imdb\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Activation\n",
    "from keras.preprocessing.text import Tokenizer\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "np.random.seed(42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1. Loading the data\n",
    "This lab uses a dataset of 25,000 IMDB reviews. Each review, comes with a label. A label of 0 is given to a negative review, and a label of 1 is given to a positive review. The goal of this lab is to create a model that will predict the sentiment of a review, **based on the words** on it. \n",
    "\n",
    "Now, the input already comes preprocessed for us for convenience. Each review is encoded as a sequence of indexes, corresponding to the words in the review. **The words are ordered by frequency**, so the integer 1 corresponds to the most frequent word (\"the\"), the integer 2 to the second most frequent word, etc. By convention, the integer 0 corresponds to unknown words.\n",
    "\n",
    "Then, the sentence is turned into a vector by simply concatenating these integers. For instance, if the sentence is \"To be or not to be.\" and the indices of the words are as follows:\n",
    "- \"to\": 5\n",
    "- \"be\": 8\n",
    "- \"or\": 21\n",
    "- \"not\": 3\n",
    "\n",
    "Then the sentence gets encoded as the vector [5,8,21,3,5,8].\n",
    "\n",
    "This dataset comes preloaded with Keras, so one simple command will get us training and testing data. There is a parameter for how many words we want to look at. We've set it at 1000, but feel free to experiment.\n",
    "- **num_words**: Top most frequent words to consider. This is useful if you don't want to consider very obscure words such as \"Ultracrepidarian\"\n",
    "- **skip_top**: Top words to ignore. This is useful if you don't want to consider the most common words. For example, the word \"the\" would add no information to the review, so we can skip it by setting skip_top to 2 or higher."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "#(x_train, y_train), (x_test, y_test) = imdb.load_data(path=\"imdb.npz\",\n",
    "#                                                     num_words=None,\n",
    "#                                                     skip_top=0,\n",
    "#                                                     maxlen=None,\n",
    "#                                                     seed=113,\n",
    "#                                                     start_char=1,\n",
    "#                                                     oov_char=2,\n",
    "#                                                     index_from=3)\n",
    "\n",
    "# Loading the data (it's preloaded in Keras)\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=1000)\n",
    "\n",
    "print(x_train.shape)\n",
    "print(x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 2. Examining the data\n",
    "Notice that the data has been already pre-processed, where all the words have numbers, and the reviews come in as a vector with the words that the review contains. For example, if the word 'the' is the first one in our dictionary, and a review contains the word 'the', then there is a 1 in the corresponding vector.\n",
    "\n",
    "The output comes as a vector of 1's and 0's, where 1 is a positive sentiment for the review, and 0 is negative."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "print(x_test[:2])\n",
    "y_test[0:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "print(x_train[0])\n",
    "print(y_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 3. One-hot encoding the input / output\n",
    "Here, we'll turn the **input** vectors into (0,1)-vectors. For example, if the pre-processed vector contains the number 14, then in the processed vector, the 14th entry will be 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# One-hot encoding the output into vector mode, each of length 1000\n",
    "'''\n",
    "keras.preprocessing.text.Tokenizer(num_words=None, filters='!\"#$%&()*+,-./:;<=>?@[\\]^_`{|}~ \n",
    "', lower=True, split=' ', char_level=False, oov_token=None)\n",
    "\n",
    "turning each text into either a sequence of integers (each integer being the index of a token in a dictionary) or into a\n",
    "vector where the coefficient for each token could be binary, based on word count, based on tf,.... \n",
    "\n",
    "- num_words: the maximum number of words to keep, based on word frequency. Only the most common num_words words will be kept.\n",
    "- filters: a string where each element is a character that will be filtered from the texts. The default is all punctuation, \n",
    "plus tabs and line breaks, minus the ' character.\n",
    "- lower: boolean. Whether to convert the texts to lowercase.\n",
    "- split: str. Separator for word splitting.\n",
    "- char_level: if True, every character will be treated as a token.\n",
    "- oov_token: if given, it will be added to word_index and used to replace out-of-vocabulary words during text_to_sequence calls\n",
    "\n",
    "By default, all punctuation is removed, turning the texts into space-separated sequences of words (words maybe include the \n",
    "' character). These sequences are then split into lists of tokens. They will then be indexed or vectorized.\n",
    "\n",
    "0 is a reserved index that won't be assigned to any word.\n",
    "'''\n",
    "tokenizer = Tokenizer(num_words=1000)\n",
    "\n",
    "x_train = tokenizer.sequences_to_matrix(x_train, mode='binary')\n",
    "x_test = tokenizer.sequences_to_matrix(x_test, mode='binary')\n",
    "print(x_train[0])\n",
    "print(x_test[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "And we'll also one-hot encode the **output**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# One-hot encoding the output\n",
    "num_classes = 2\n",
    "\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "print(y_train.shape)\n",
    "print(y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 4. Building the  model architecture\n",
    "Build a model here using sequential. Feel free to experiment with different layers and sizes! Also, experiment adding dropout to reduce overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# TODO: Build the model architecture\n",
    "model = Sequential()\n",
    "model.add(Dense(512, activation='relu', input_dim=1000))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='sigmoid')) ## why not 'softmax'??\n",
    "model.summary()\n",
    "\n",
    "# TODO: Compile the model using a loss function and an optimizer.\n",
    "model.compile(loss='categorical_crossentropy',\n",
    "              optimizer='rmsprop',\n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 5. Training the model\n",
    "Run the model here. Experiment with different batch_size, and number of epochs!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "# TODO: Run the model. Feel free to experiment with different batch sizes and number of epochs.\n",
    "hist = model.fit(x_train, y_train,\n",
    "          batch_size=32,\n",
    "          epochs=10,\n",
    "          validation_data=(x_test, y_test), \n",
    "          verbose=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 6. Evaluating the model\n",
    "This will give you the accuracy of the model, as evaluated on the testing set. Can you get something over 85%?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "print(\"Accuracy: \", score[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}