SD_TSIA203_RNN_for_classification.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PBFqSEkKqpCN"
      },
      "source": [
        "# Lab Deep Learning/ Recurrent Neural Networks/ in pytorch\n",
        "\n",
        "## Using Many-to-One for movie rating predicton\n",
        "\n",
        "**Author: created by geoffroy.peeters@telecom-paris.fr** with the help of Stéphane Lathuilière\n",
        "\n",
        "For any remark or suggestion, please feel free to contact me.\n",
        "\n",
        "## Objective:\n",
        "You will implement two different networks to perform the automatic rating (0 or 1) of movies given the text of their reviews.\n",
        "You will use the ```imdb``` (internet movie database) dataset.\n",
        "\n",
        "The reviews are already available in the form of indexes that point to a word dictionary: each word is already encoded as an index in the dictionary."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QmkCSNaXLqjh"
      },
      "source": [
        "## Import packages"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "AOqjzDwioJj9"
      },
      "outputs": [],
      "source": [
        "from keras.datasets import imdb\n",
        "import numpy as np\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "import torch.optim as optim\n",
        "from torch.utils.data import DataLoader, TensorDataset\n",
        "from argparse import Namespace\n",
        "\n",
        "colab = True\n",
        "student = True"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5Yp4OQVvUtr"
      },
      "source": [
        "## Parameters of the model\n",
        "\n",
        "- We only consider the most used words in the word dictionary, we consider the top `param.n_word`\n",
        "- We truncate/zero-pad each review to a length `param.T_x`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4C_Pv7rYvRkM"
      },
      "outputs": [],
      "source": [
        "param = Namespace()\n",
        "\n",
        "param.n_word = 5000 # --- input dimension\n",
        "param.T_x = 100 # --- review length\n",
        "param.index_word_from = 3 # --- indicate where the index start from (first index are used to indicate `PAD` `START` `UNK` tokens)\n",
        "\n",
        "param.n_embedding = 32 # --- dimension of the embedding\n",
        "param.n_lstm = 100 # --- dimension of the LSTM (for a<t> and c<t>)\n",
        "param.n_out = 1 # --- binary classification problem\n",
        "\n",
        "param.batch_size = 64\n",
        "param.lr = 0.001\n",
        "param.n_epoch = 8\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZsNcRimyLzgP"
      },
      "source": [
        "## Import IMDB data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5Gfe1ex8oN8Q",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "3b7b1b2d-b293-485c-d07e-a435f6fb7b7e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb.npz\n",
            "17464789/17464789 [==============================] - 0s 0us/step\n"
          ]
        }
      ],
      "source": [
        "# --- Import the IMDB data and only consider the ``param.n_word``` most used words\n",
        "(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words=param.n_word, index_from=param.index_word_from )"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iSc5LmksOLyr"
      },
      "source": [
        "## Data content\n",
        "\n",
        "- ```X_train``` and ```X_test``` are each a numpy array, shape=(25000,), of lists.\n",
        "  - Each list represent a review; it is a sequence (represented as a list) of indexes (position of each word in the dictionary)\n",
        "\n",
        "- ```y_train``` and ```y_test``` are each a numpy array, shape=(25000,) of intergers.\n",
        "  - Each integer represent the values 0 (bad movie) or 1 (good movie)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 571
        },
        "id": "WouODCPrtiuu",
        "outputId": "2597c9ca-3551-43ad-d332-580accc4d5a0"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "type(X_train): <class 'numpy.ndarray'>\n",
            "number of training sequences: X_train.shape: (25000,)\n",
            "type(X_train[0]): <class 'list'>\n",
            "length of the first training sequence: len(X_train[0]): 218\n",
            "length of the second training sequence: len(X_train[1]): 189\n",
            "list of data of the first training sequence: X_train[0]: [1, 14, 22, 16, 43, 530, 973, 1622, 1385, 65, 458, 4468, 66, 3941, 4, 173, 36, 256, 5, 25, 100, 43, 838, 112, 50, 670, 2, 9, 35, 480, 284, 5, 150, 4, 172, 112, 167, 2, 336, 385, 39, 4, 172, 4536, 1111, 17, 546, 38, 13, 447, 4, 192, 50, 16, 6, 147, 2025, 19, 14, 22, 4, 1920, 4613, 469, 4, 22, 71, 87, 12, 16, 43, 530, 38, 76, 15, 13, 1247, 4, 22, 17, 515, 17, 12, 16, 626, 18, 2, 5, 62, 386, 12, 8, 316, 8, 106, 5, 4, 2223, 2, 16, 480, 66, 3785, 33, 4, 130, 12, 16, 38, 619, 5, 25, 124, 51, 36, 135, 48, 25, 1415, 33, 6, 22, 12, 215, 28, 77, 52, 5, 14, 407, 16, 82, 2, 8, 4, 107, 117, 2, 15, 256, 4, 2, 7, 3766, 5, 723, 36, 71, 43, 530, 476, 26, 400, 317, 46, 7, 4, 2, 1029, 13, 104, 88, 4, 381, 15, 297, 98, 32, 2071, 56, 26, 141, 6, 194, 2, 18, 4, 226, 22, 21, 134, 476, 26, 480, 5, 144, 30, 2, 18, 51, 36, 28, 224, 92, 25, 104, 4, 226, 65, 16, 38, 1334, 88, 12, 16, 283, 5, 16, 4472, 113, 103, 32, 15, 16, 2, 19, 178, 32]\n",
            "maximum length of a training sequence: 2494\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ],
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjAAAAGdCAYAAAAMm0nCAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAtHklEQVR4nO3de3BUZZ7/8U8IdEuE7hgg6WQIGESByE2ihl6VxSGTgNHRFatEGWAUoWCDtRCFmJUfIm5NWFgvjBfYKVfj1oKIW6IjGcAQDIwaUFJGbpISJmxwoRMGTBoQwiXP74/55fxsCEJCbk94v6pOVfo833P6OY+x8+E5lw4zxhgBAABYpENrdwAAAKChCDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOt0bO0ONJfa2lodPHhQXbt2VVhYWGt3BwAAXAZjjI4dO6a4uDh16HDxeZZ2G2AOHjyo+Pj41u4GAABohAMHDqhnz54XbW+3AaZr166S/jYAHo+nlXsDAAAuRzAYVHx8vPN3/GLabYCpO23k8XgIMAAAWOZSl39wES8AALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdTq2dgeuZtc/k3fBuv0L01uhJwAA2IUZGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGCdBgWYpUuXavDgwfJ4PPJ4PPL7/Vq7dq3TPnLkSIWFhYUs06ZNC9lHeXm50tPTFRERoejoaM2ePVtnz54NqSksLNSwYcPkdrvVt29f5ebmNv4IAQBAu9OxIcU9e/bUwoULdeONN8oYo3feeUf333+/vv76a918882SpClTpmjBggXONhEREc7P586dU3p6unw+n7744gsdOnRIEydOVKdOnfS73/1OklRWVqb09HRNmzZNy5cvV0FBgZ544gnFxsYqLS2tKY4ZAABYLswYY65kB1FRUVq8eLEmT56skSNHaujQoXrllVfqrV27dq3uvfdeHTx4UDExMZKkZcuWKSsrS4cPH5bL5VJWVpby8vK0c+dOZ7tx48apqqpK69atu+x+BYNBeb1eVVdXy+PxXMkhNpvrn8m7YN3+hemt0BMAANqGy/373ehrYM6dO6eVK1fqxIkT8vv9zvrly5ere/fuGjhwoLKzs/Xjjz86bUVFRRo0aJATXiQpLS1NwWBQu3btcmpSUlJC3istLU1FRUU/25+amhoFg8GQBQAAtE8NOoUkSTt27JDf79epU6fUpUsXrV69WomJiZKkRx99VL1791ZcXJy2b9+urKwslZaW6oMPPpAkBQKBkPAiyXkdCAR+tiYYDOrkyZPq3Llzvf3KycnR888/39DDAQAAFmpwgOnXr59KSkpUXV2t//7v/9akSZO0adMmJSYmaurUqU7doEGDFBsbq1GjRmnfvn264YYbmrTj58vOzlZmZqbzOhgMKj4+vlnfEwAAtI4Gn0JyuVzq27evkpKSlJOToyFDhmjJkiX11iYnJ0uS9u7dK0ny+XyqqKgIqal77fP5frbG4/FcdPZFktxut3N3VN0CAADapyt+Dkxtba1qamrqbSspKZEkxcbGSpL8fr927NihyspKpyY/P18ej8c5DeX3+1VQUBCyn/z8/JDrbAAAwNWtQaeQsrOzNWbMGPXq1UvHjh3TihUrVFhYqPXr12vfvn1asWKF7rnnHnXr1k3bt2/XrFmzNGLECA0ePFiSlJqaqsTERE2YMEGLFi1SIBDQ3LlzlZGRIbfbLUmaNm2aXnvtNc2ZM0ePP/64Nm7cqFWrVikv78I7dgAAwNWpQQGmsrJSEydO1KFDh+T1ejV48GCtX79ev/rVr3TgwAFt2LBBr7zyik6cOKH4+HiNHTtWc+fOdbYPDw/XmjVrNH36dPn9fl177bWaNGlSyHNjEhISlJeXp1mzZmnJkiXq2bOn3nzzTZ4BAwAAHFf8HJi2iufAAABgn2Z/DgwAAEBrIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKzT4G+jRvM6/+F2PNgOAIALMQMDAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrNCjALF26VIMHD5bH45HH45Hf79fatWud9lOnTikjI0PdunVTly5dNHbsWFVUVITso7y8XOnp6YqIiFB0dLRmz56ts2fPhtQUFhZq2LBhcrvd6tu3r3Jzcxt/hAAAoN1pUIDp2bOnFi5cqOLiYm3btk2//OUvdf/992vXrl2SpFmzZunjjz/W+++/r02bNungwYN68MEHne3PnTun9PR0nT59Wl988YXeeecd5ebmat68eU5NWVmZ0tPTdffdd6ukpEQzZ87UE088ofXr1zfRIQMAANuFGWPMlewgKipKixcv1kMPPaQePXpoxYoVeuihhyRJe/bs0YABA1RUVKThw4dr7dq1uvfee3Xw4EHFxMRIkpYtW6asrCwdPnxYLpdLWVlZysvL086dO533GDdunKqqqrRu3brL7lcwGJTX61V1dbU8Hs+VHGKzuf6ZvEvW7F+Y3gI9AQCgbbjcv9+Nvgbm3LlzWrlypU6cOCG/36/i4mKdOXNGKSkpTk3//v3Vq1cvFRUVSZKKioo0aNAgJ7xIUlpamoLBoDOLU1RUFLKPupq6fVxMTU2NgsFgyAIAANqnBgeYHTt2qEuXLnK73Zo2bZpWr16txMREBQIBuVwuRUZGhtTHxMQoEAhIkgKBQEh4qWuva/u5mmAwqJMnT160Xzk5OfJ6vc4SHx/f0EMDAACWaHCA6devn0pKSrR161ZNnz5dkyZN0u7du5ujbw2SnZ2t6upqZzlw4EBrdwkAADSTjg3dwOVyqW/fvpKkpKQkffXVV1qyZIkefvhhnT59WlVVVSGzMBUVFfL5fJIkn8+nL7/8MmR/dXcp/bTm/DuXKioq5PF41Llz54v2y+12y+12N/RwAACAha74OTC1tbWqqalRUlKSOnXqpIKCAqettLRU5eXl8vv9kiS/368dO3aosrLSqcnPz5fH41FiYqJT89N91NXU7QMAAKBBMzDZ2dkaM2aMevXqpWPHjmnFihUqLCzU+vXr5fV6NXnyZGVmZioqKkoej0dPPvmk/H6/hg8fLklKTU1VYmKiJkyYoEWLFikQCGju3LnKyMhwZk+mTZum1157TXPmzNHjjz+ujRs3atWqVcrLu/QdOwAA4OrQoABTWVmpiRMn6tChQ/J6vRo8eLDWr1+vX/3qV5Kkl19+WR06dNDYsWNVU1OjtLQ0vfHGG8724eHhWrNmjaZPny6/369rr71WkyZN0oIFC5yahIQE5eXladasWVqyZIl69uypN998U2lpaU10yAAAwHZX/ByYtornwAAAYJ9mfw4MAABAayHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6DQowOTk5uu2229S1a1dFR0frgQceUGlpaUjNyJEjFRYWFrJMmzYtpKa8vFzp6emKiIhQdHS0Zs+erbNnz4bUFBYWatiwYXK73erbt69yc3Mbd4QAAKDdaVCA2bRpkzIyMrRlyxbl5+frzJkzSk1N1YkTJ0LqpkyZokOHDjnLokWLnLZz584pPT1dp0+f1hdffKF33nlHubm5mjdvnlNTVlam9PR03X333SopKdHMmTP1xBNPaP369Vd4uAAAoD3o2JDidevWhbzOzc1VdHS0iouLNWLECGd9RESEfD5fvfv45JNPtHv3bm3YsEExMTEaOnSoXnjhBWVlZWn+/PlyuVxatmyZEhIS9OKLL0qSBgwYoM8++0wvv/yy0tLSGnqMAACgnbmia2Cqq6slSVFRUSHrly9fru7du2vgwIHKzs7Wjz/+6LQVFRVp0KBBiomJcdalpaUpGAxq165dTk1KSkrIPtPS0lRUVHTRvtTU1CgYDIYs7cH1z+RdsAAAcLVr0AzMT9XW1mrmzJm64447NHDgQGf9o48+qt69eysuLk7bt29XVlaWSktL9cEHH0iSAoFASHiR5LwOBAI/WxMMBnXy5El17tz5gv7k5OTo+eefb+zhAAAAizQ6wGRkZGjnzp367LPPQtZPnTrV+XnQoEGKjY3VqFGjtG/fPt1www2N7+klZGdnKzMz03kdDAYVHx/fbO8HAABaT6NOIc2YMUNr1qzRp59+qp49e/5sbXJysiRp7969kiSfz6eKioqQmrrXddfNXKzG4/HUO/siSW63Wx6PJ2QBAADtU4MCjDFGM2bM0OrVq7Vx40YlJCRccpuSkhJJUmxsrCTJ7/drx44dqqysdGry8/Pl8XiUmJjo1BQUFITsJz8/X36/vyHdBQAA7VSDAkxGRob+67/+SytWrFDXrl0VCAQUCAR08uRJSdK+ffv0wgsvqLi4WPv379cf//hHTZw4USNGjNDgwYMlSampqUpMTNSECRP0zTffaP369Zo7d64yMjLkdrslSdOmTdNf/vIXzZkzR3v27NEbb7yhVatWadasWU18+AAAwEYNCjBLly5VdXW1Ro4cqdjYWGd57733JEkul0sbNmxQamqq+vfvr6eeekpjx47Vxx9/7OwjPDxca9asUXh4uPx+v37zm99o4sSJWrBggVOTkJCgvLw85efna8iQIXrxxRf15ptvcgs1AACQJIUZY0xrd6I5BINBeb1eVVdXt9nrYRp7S/T+helN3BMAANqGy/37zXchAQAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGCdBgWYnJwc3Xbbberatauio6P1wAMPqLS0NKTm1KlTysjIULdu3dSlSxeNHTtWFRUVITXl5eVKT09XRESEoqOjNXv2bJ09ezakprCwUMOGDZPb7Vbfvn2Vm5vbuCMEAADtToMCzKZNm5SRkaEtW7YoPz9fZ86cUWpqqk6cOOHUzJo1Sx9//LHef/99bdq0SQcPHtSDDz7otJ87d07p6ek6ffq0vvjiC73zzjvKzc3VvHnznJqysjKlp6fr7rvvVklJiWbOnKknnnhC69evb4JDBgAAtgszxpjGbnz48GFFR0dr06ZNGjFihKqrq9WjRw+tWLFCDz30kCRpz549GjBggIqKijR8+HCtXbtW9957rw4ePKiYmBhJ0rJly5SVlaXDhw/L5XIpKytLeXl52rlzp/Ne48aNU1VVldatW3dZfQsGg/J6vaqurpbH42nsITar65/Ja9R2+xemN3FPAABoGy737/cVXQNTXV0tSYqKipIkFRcX68yZM0pJSXFq+vfvr169eqmoqEiSVFRUpEGDBjnhRZLS0tIUDAa1a9cup+an+6irqdsHAAC4unVs7Ia1tbWaOXOm7rjjDg0cOFCSFAgE5HK5FBkZGVIbExOjQCDg1Pw0vNS117X9XE0wGNTJkyfVuXPnC/pTU1Ojmpoa53UwGGzsoQEAgDau0TMwGRkZ2rlzp1auXNmU/Wm0nJwceb1eZ4mPj2/tLgEAgGbSqAAzY8YMrVmzRp9++ql69uzprPf5fDp9+rSqqqpC6isqKuTz+Zya8+9Kqnt9qRqPx1Pv7IskZWdnq7q62lkOHDjQmEMDAAAWaFCAMcZoxowZWr16tTZu3KiEhISQ9qSkJHXq1EkFBQXOutLSUpWXl8vv90uS/H6/duzYocrKSqcmPz9fHo9HiYmJTs1P91FXU7eP+rjdbnk8npAFAAC0Tw26BiYjI0MrVqzQRx99pK5duzrXrHi9XnXu3Fler1eTJ09WZmamoqKi5PF49OSTT8rv92v48OGSpNTUVCUmJmrChAlatGiRAoGA5s6dq4yMDLndbknStGnT9Nprr2nOnDl6/PHHtXHjRq1atUp5eY27awcAALQvDZqBWbp0qaqrqzVy5EjFxsY6y3vvvefUvPzyy7r33ns1duxYjRgxQj6fTx988IHTHh4erjVr1ig8PFx+v1+/+c1vNHHiRC1YsMCpSUhIUF5envLz8zVkyBC9+OKLevPNN5WWltYEhwwAAGx3Rc+Bact4DgwAAPZpkefAAAAAtAYCDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6zT6yxzRes6//ZrbqgEAVxtmYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArNPgALN582bdd999iouLU1hYmD788MOQ9t/+9rcKCwsLWUaPHh1Sc/ToUY0fP14ej0eRkZGaPHmyjh8/HlKzfft23XXXXbrmmmsUHx+vRYsWNfzoAABAu9TgAHPixAkNGTJEr7/++kVrRo8erUOHDjnLu+++G9I+fvx47dq1S/n5+VqzZo02b96sqVOnOu3BYFCpqanq3bu3iouLtXjxYs2fP19/+MMfGtpdAADQDnVs6AZjxozRmDFjfrbG7XbL5/PV2/btt99q3bp1+uqrr3TrrbdKkl599VXdc889+rd/+zfFxcVp+fLlOn36tN566y25XC7dfPPNKikp0UsvvRQSdAAAwNWpWa6BKSwsVHR0tPr166fp06fryJEjTltRUZEiIyOd8CJJKSkp6tChg7Zu3erUjBgxQi6Xy6lJS0tTaWmpfvjhh3rfs6amRsFgMGQBAADtU5MHmNGjR+s///M/VVBQoH/913/Vpk2bNGbMGJ07d06SFAgEFB0dHbJNx44dFRUVpUAg4NTExMSE1NS9rqs5X05Ojrxer7PEx8c39aEBAIA2osGnkC5l3Lhxzs+DBg3S4MGDdcMNN6iwsFCjRo1q6rdzZGdnKzMz03kdDAYJMQAAtFPNfht1nz591L17d+3du1eS5PP5VFlZGVJz9uxZHT161LluxufzqaKiIqSm7vXFrq1xu93yeDwhCwAAaJ+aPcB8//33OnLkiGJjYyVJfr9fVVVVKi4udmo2btyo2tpaJScnOzWbN2/WmTNnnJr8/Hz169dP1113XXN3GQAAtHENDjDHjx9XSUmJSkpKJEllZWUqKSlReXm5jh8/rtmzZ2vLli3av3+/CgoKdP/996tv375KS0uTJA0YMECjR4/WlClT9OWXX+rzzz/XjBkzNG7cOMXFxUmSHn30UblcLk2ePFm7du3Se++9pyVLloScIgIAAFevBgeYbdu26ZZbbtEtt9wiScrMzNQtt9yiefPmKTw8XNu3b9evf/1r3XTTTZo8ebKSkpL05z//WW6329nH8uXL1b9/f40aNUr33HOP7rzzzpBnvHi9Xn3yyScqKytTUlKSnnrqKc2bN49bqAEAgCQpzBhjWrsTzSEYDMrr9aq6urrNXg9z/TN5TbKf/QvTm2Q/AAC0tsv9+93kdyGh5dUXhAg1AID2jC9zBAAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDrdRt6Cmeu4LAABXO2ZgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6fJVAO3X+1xbsX5jeSj0BAKDpMQMDAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOz4FpJuc/hwUAADQdZmAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANYhwAAAAOsQYAAAgHUaHGA2b96s++67T3FxcQoLC9OHH34Y0m6M0bx58xQbG6vOnTsrJSVF3333XUjN0aNHNX78eHk8HkVGRmry5Mk6fvx4SM327dt111136ZprrlF8fLwWLVrU8KMDAADtUoMDzIkTJzRkyBC9/vrr9bYvWrRIv//977Vs2TJt3bpV1157rdLS0nTq1CmnZvz48dq1a5fy8/O1Zs0abd68WVOnTnXag8GgUlNT1bt3bxUXF2vx4sWaP3++/vCHPzTiEAEAQHsTZowxjd44LEyrV6/WAw88IOlvsy9xcXF66qmn9PTTT0uSqqurFRMTo9zcXI0bN07ffvutEhMT9dVXX+nWW2+VJK1bt0733HOPvv/+e8XFxWnp0qV69tlnFQgE5HK5JEnPPPOMPvzwQ+3Zs+ey+hYMBuX1elVdXS2Px9PYQ2y0tvZdSPsXprd2FwAAuKTL/fvdpNfAlJWVKRAIKCUlxVnn9XqVnJysoqIiSVJRUZEiIyOd8CJJKSkp6tChg7Zu3erUjBgxwgkvkpSWlqbS0lL98MMP9b53TU2NgsFgyAIAANqnJg0wgUBAkhQTExOyPiYmxmkLBAKKjo4Oae/YsaOioqJCaurbx0/f43w5OTnyer3OEh8ff+UHBAAA2qR2cxdSdna2qqurneXAgQOt3SUAANBMmjTA+Hw+SVJFRUXI+oqKCqfN5/OpsrIypP3s2bM6evRoSE19+/jpe5zP7XbL4/GELAAAoH3q2JQ7S0hIkM/nU0FBgYYOHSrpbxfjbN26VdOnT5ck+f1+VVVVqbi4WElJSZKkjRs3qra2VsnJyU7Ns88+qzNnzqhTp06SpPz8fPXr10/XXXddU3b5qlHfRcVc2AsAsFWDZ2COHz+ukpISlZSUSPrbhbslJSUqLy9XWFiYZs6cqX/5l3/RH//4R+3YsUMTJ05UXFycc6fSgAEDNHr0aE2ZMkVffvmlPv/8c82YMUPjxo1TXFycJOnRRx+Vy+XS5MmTtWvXLr333ntasmSJMjMzm+zAAQCAvRo8A7Nt2zbdfffdzuu6UDFp0iTl5uZqzpw5OnHihKZOnaqqqirdeeedWrduna655hpnm+XLl2vGjBkaNWqUOnTooLFjx+r3v/+90+71evXJJ58oIyNDSUlJ6t69u+bNmxfyrBgAAHD1uqLnwLRlPAfm0jiFBABoa1rlOTAAAAAtgQADAACs06R3IV3NbDhlBABAe8EMDAAAsA4BBgAAWIcAAwAArEOAAQAA1iHAAAAA6xBgAACAdQgwAADAOgQYAABgHQIMAACwDgEGAABYhwADAACsQ4ABAADW4cscr2L1fQHl/oXprdATAAAahhkYAABgHQIMAACwDgEGAABYh2tgEOL862K4JgYA0BYxAwMAAKxDgAEAANYhwAAAAOsQYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4BBgAAWIcAAwAArEOAAQAA1uGrBPCzzv9qAYmvFwAAtD5mYAAAgHUIMAAAwDoEGAAAYB2ugUGDnX9dDNfEAABaGjMwAADAOk0eYObPn6+wsLCQpX///k77qVOnlJGRoW7duqlLly4aO3asKioqQvZRXl6u9PR0RUREKDo6WrNnz9bZs2ebuqsAAMBSzXIK6eabb9aGDRv+/5t0/P9vM2vWLOXl5en999+X1+vVjBkz9OCDD+rzzz+XJJ07d07p6eny+Xz64osvdOjQIU2cOFGdOnXS7373u+boLgAAsEyzBJiOHTvK5/NdsL66ulr/8R//oRUrVuiXv/ylJOntt9/WgAEDtGXLFg0fPlyffPKJdu/erQ0bNigmJkZDhw7VCy+8oKysLM2fP18ul6s5ugwAACzSLNfAfPfdd4qLi1OfPn00fvx4lZeXS5KKi4t15swZpaSkOLX9+/dXr169VFRUJEkqKirSoEGDFBMT49SkpaUpGAxq165dF33PmpoaBYPBkAUAALRPTR5gkpOTlZubq3Xr1mnp0qUqKyvTXXfdpWPHjikQCMjlcikyMjJkm5iYGAUCAUlSIBAICS917XVtF5OTkyOv1+ss8fHxTXtgAACgzWjyU0hjxoxxfh48eLCSk5PVu3dvrVq1Sp07d27qt3NkZ2crMzPTeR0MBgkxAAC0U81+G3VkZKRuuukm7d27Vz6fT6dPn1ZVVVVITUVFhXPNjM/nu+CupLrX9V1XU8ftdsvj8YQsAACgfWr2AHP8+HHt27dPsbGxSkpKUqdOnVRQUOC0l5aWqry8XH6/X5Lk9/u1Y8cOVVZWOjX5+fnyeDxKTExs7u4CAAALNPkppKefflr33XefevfurYMHD+q5555TeHi4HnnkEXm9Xk2ePFmZmZmKioqSx+PRk08+Kb/fr+HDh0uSUlNTlZiYqAkTJmjRokUKBAKaO3euMjIy5Ha7m7q7AADAQk0eYL7//ns98sgjOnLkiHr06KE777xTW7ZsUY8ePSRJL7/8sjp06KCxY8eqpqZGaWlpeuONN5ztw8PDtWbNGk2fPl1+v1/XXnutJk2apAULFjR1VwEAgKXCjDGmtTvRHILBoLxer6qrq1vkepjzvx/oasJ3IQEAmsrl/v3mu5AAAIB1+DZqXLH6Zp+YlQEANCdmYAAAgHWYgUGzOH9WhhkZAEBTYgYGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1uAsJLYJnxQAAmhIzMAAAwDoEGAAAYB1OIaHV8LA7AEBjMQMDAACswwwM2gwu9AUAXC5mYAAAgHUIMAAAwDoEGAAAYB0CDAAAsA4X8aJN41ZrAEB9mIEBAADWYQYGVuFWawCAxAwMAACwEAEGAABYh1NIaHc4zQQA7R8BBtarL7AAANo3TiEBAADrMAODqwLPkwGA9oUZGAAAYB1mYHBV4kJfALAbAQb4fzjNBAD2IMAAF8EsDQC0XQSYRuC23asXszQA0DYQYIArcDmzNMzkAEDTI8AATexyZuja2kxOW+sPAFwKAQZoA1pyloZToADagzYdYF5//XUtXrxYgUBAQ4YM0auvvqrbb7+9tbsFtIjLmRVpTA0AtAdtNsC89957yszM1LJly5ScnKxXXnlFaWlpKi0tVXR0dGt3D2hxjTk1BQDtVZt9Eu9LL72kKVOm6LHHHlNiYqKWLVumiIgIvfXWW63dNQAA0Mra5AzM6dOnVVxcrOzsbGddhw4dlJKSoqKionq3qampUU1NjfO6urpakhQMBpu8f7U1Pzb5PoG2pNes9y9Yt/P5tFboCYCrTd3fbWPMz9a1yQDz17/+VefOnVNMTEzI+piYGO3Zs6febXJycvT8889fsD4+Pr5Z+ghcbbyvtHYPAFxNjh07Jq/Xe9H2NhlgGiM7O1uZmZnO69raWh09elTdunVTWFjYFe8/GAwqPj5eBw4ckMfjueL94eIY65bDWLccxrrlMNYto7nG2RijY8eOKS4u7mfr2mSA6d69u8LDw1VRURGyvqKiQj6fr95t3G633G53yLrIyMgm75vH4+F/iBbCWLccxrrlMNYth7FuGc0xzj8381KnTV7E63K5lJSUpIKCAmddbW2tCgoK5Pf7W7FnAACgLWiTMzCSlJmZqUmTJunWW2/V7bffrldeeUUnTpzQY4891tpdAwAArazNBpiHH35Yhw8f1rx58xQIBDR06FCtW7fuggt7W4rb7dZzzz13wWkqND3GuuUw1i2HsW45jHXLaO1xDjOXuk8JAACgjWmT18AAAAD8HAIMAACwDgEGAABYhwADAACsQ4C5TK+//rquv/56XXPNNUpOTtaXX37Z2l2yyvz58xUWFhay9O/f32k/deqUMjIy1K1bN3Xp0kVjx4694EGG5eXlSk9PV0REhKKjozV79mydPXu2pQ+lzdm8ebPuu+8+xcXFKSwsTB9++GFIuzFG8+bNU2xsrDp37qyUlBR99913ITVHjx7V+PHj5fF4FBkZqcmTJ+v48eMhNdu3b9ddd92la665RvHx8Vq0aFFzH1qbc6mx/u1vf3vB7/no0aNDahjrS8vJydFtt92mrl27Kjo6Wg888IBKS0tDaprqM6OwsFDDhg2T2+1W3759lZub29yH16ZczliPHDnygt/radOmhdS0ylgbXNLKlSuNy+Uyb731ltm1a5eZMmWKiYyMNBUVFa3dNWs899xz5uabbzaHDh1ylsOHDzvt06ZNM/Hx8aagoMBs27bNDB8+3Pzd3/2d03727FkzcOBAk5KSYr7++mvzpz/9yXTv3t1kZ2e3xuG0KX/605/Ms88+az744AMjyaxevTqkfeHChcbr9ZoPP/zQfPPNN+bXv/61SUhIMCdPnnRqRo8ebYYMGWK2bNli/vznP5u+ffuaRx55xGmvrq42MTExZvz48Wbnzp3m3XffNZ07dzb//u//3lKH2SZcaqwnTZpkRo8eHfJ7fvTo0ZAaxvrS0tLSzNtvv2127txpSkpKzD333GN69epljh8/7tQ0xWfGX/7yFxMREWEyMzPN7t27zauvvmrCw8PNunXrWvR4W9PljPXf//3fmylTpoT8XldXVzvtrTXWBJjLcPvtt5uMjAzn9blz50xcXJzJyclpxV7Z5bnnnjNDhgypt62qqsp06tTJvP/++866b7/91kgyRUVFxpi//eHo0KGDCQQCTs3SpUuNx+MxNTU1zdp3m5z/R7W2ttb4fD6zePFiZ11VVZVxu93m3XffNcYYs3v3biPJfPXVV07N2rVrTVhYmPnf//1fY4wxb7zxhrnuuutCxjorK8v069evmY+o7bpYgLn//vsvug1j3TiVlZVGktm0aZMxpuk+M+bMmWNuvvnmkPd6+OGHTVpaWnMfUpt1/lgb87cA80//9E8X3aa1xppTSJdw+vRpFRcXKyUlxVnXoUMHpaSkqKioqBV7Zp/vvvtOcXFx6tOnj8aPH6/y8nJJUnFxsc6cORMyxv3791evXr2cMS4qKtKgQYNCHmSYlpamYDCoXbt2teyBWKSsrEyBQCBkbL1er5KTk0PGNjIyUrfeeqtTk5KSog4dOmjr1q1OzYgRI+RyuZyatLQ0lZaW6ocffmiho7FDYWGhoqOj1a9fP02fPl1Hjhxx2hjrxqmurpYkRUVFSWq6z4yioqKQfdTVXM2f7eePdZ3ly5ere/fuGjhwoLKzs/Xjjz86ba011m32SbxtxV//+ledO3fugicAx8TEaM+ePa3UK/skJycrNzdX/fr106FDh/T888/rrrvu0s6dOxUIBORyuS748s2YmBgFAgFJUiAQqPe/QV0b6lc3NvWN3U/HNjo6OqS9Y8eOioqKCqlJSEi4YB91bdddd12z9N82o0eP1oMPPqiEhATt27dP//zP/6wxY8aoqKhI4eHhjHUj1NbWaubMmbrjjjs0cOBASWqyz4yL1QSDQZ08eVKdO3dujkNqs+oba0l69NFH1bt3b8XFxWn79u3KyspSaWmpPvjgA0mtN9YEGLSIMWPGOD8PHjxYycnJ6t27t1atWnXVfUig/Ro3bpzz86BBgzR48GDdcMMNKiws1KhRo1qxZ/bKyMjQzp079dlnn7V2V9q9i4311KlTnZ8HDRqk2NhYjRo1Svv27dMNN9zQ0t10cArpErp3767w8PALrm6vqKiQz+drpV7ZLzIyUjfddJP27t0rn8+n06dPq6qqKqTmp2Ps8/nq/W9Q14b61Y3Nz/3++nw+VVZWhrSfPXtWR48eZfyvUJ8+fdS9e3ft3btXEmPdUDNmzNCaNWv06aefqmfPns76pvrMuFiNx+O56v5hdbGxrk9ycrIkhfxet8ZYE2AuweVyKSkpSQUFBc662tpaFRQUyO/3t2LP7Hb8+HHt27dPsbGxSkpKUqdOnULGuLS0VOXl5c4Y+/1+7dixI+TDPz8/Xx6PR4mJiS3ef1skJCTI5/OFjG0wGNTWrVtDxraqqkrFxcVOzcaNG1VbW+t8UPn9fm3evFlnzpxxavLz89WvX7+r7pRGQ3z//fc6cuSIYmNjJTHWl8sYoxkzZmj16tXauHHjBafUmuozw+/3h+yjruZq+my/1FjXp6SkRJJCfq9bZawbffnvVWTlypXG7Xab3Nxcs3v3bjN16lQTGRkZcsU1ft5TTz1lCgsLTVlZmfn8889NSkqK6d69u6msrDTG/O2WyF69epmNGzeabdu2Gb/fb/x+v7N93W16qamppqSkxKxbt8706NGD26iNMceOHTNff/21+frrr40k89JLL5mvv/7a/M///I8x5m+3UUdGRpqPPvrIbN++3dx///313kZ9yy23mK1bt5rPPvvM3HjjjSG39lZVVZmYmBgzYcIEs3PnTrNy5UoTERFxVd3aa8zPj/WxY8fM008/bYqKikxZWZnZsGGDGTZsmLnxxhvNqVOnnH0w1pc2ffp04/V6TWFhYcituz/++KNT0xSfGXW39s6ePdt8++235vXXX7/qbqO+1Fjv3bvXLFiwwGzbts2UlZWZjz76yPTp08eMGDHC2UdrjTUB5jK9+uqrplevXsblcpnbb7/dbNmypbW7ZJWHH37YxMbGGpfLZX7xi1+Yhx9+2Ozdu9dpP3nypPnHf/xHc91115mIiAjzD//wD+bQoUMh+9i/f78ZM2aM6dy5s+nevbt56qmnzJkzZ1r6UNqcTz/91Ei6YJk0aZIx5m+3Uv+f//N/TExMjHG73WbUqFGmtLQ0ZB9HjhwxjzzyiOnSpYvxeDzmscceM8eOHQup+eabb8ydd95p3G63+cUvfmEWLlzYUofYZvzcWP/4448mNTXV9OjRw3Tq1Mn07t3bTJky5YJ/6DDWl1bfGEsyb7/9tlPTVJ8Zn376qRk6dKhxuVymT58+Ie9xNbjUWJeXl5sRI0aYqKgo43a7Td++fc3s2bNDngNjTOuMddj/OwAAAABrcA0MAACwDgEGAABYhwADAACsQ4ABAADWIcAAAADrEGAAAIB1CDAAAMA6BBgAAGAdAgwAALAOAQYAAFiHAAMAAKxDgAEAANb5v9sLDT+OFDwfAAAAAElFTkSuQmCC\n"
          },
          "metadata": {}
        }
      ],
      "source": [
        "print(\"type(X_train):\", type(X_train))\n",
        "print(\"number of training sequences: X_train.shape:\", X_train.shape)\n",
        "print(\"type(X_train[0]):\", type(X_train[0]))\n",
        "print(\"length of the first training sequence: len(X_train[0]):\",len(X_train[0]))\n",
        "print(\"length of the second training sequence: len(X_train[1]):\",len(X_train[1]))\n",
        "print(\"list of data of the first training sequence: X_train[0]:\", X_train[0] )\n",
        "len_list = [len(train) for train in X_train]\n",
        "print(\"maximum length of a training sequence:\", max(len_list))\n",
        "\n",
        "import matplotlib.pyplot as plt\n",
        "plt.hist(len_list, 100);"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2I-cEKUh_HM4"
      },
      "source": [
        "## Details of how the reviews are encoded"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "XcOwiMUT_HM5",
        "outputId": "96861e76-9b3b-4443-d1a0-7e1cd33454be"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/imdb_word_index.json\n",
            "1641221/1641221 [==============================] - 0s 0us/step\n",
            "<START> although i had seen <UNK> in a theater way back in <UNK> i couldn't remember anything of the plot except for vague images of kurt thomas running and fighting against a backdrop of stone walls and disappointment regarding the ending br br after reading some of the other reviews i picked up a copy of the newly released dvd to once again enter the world of <UNK> br br it turns out this is one of those films produced during the <UNK> that would go directly to video today the film stars <UNK> <UNK> kurt thomas as jonathan <UNK> <UNK> out of the blue to <UNK> the nation of <UNK> to enter and hopefully win the game a <UNK> <UNK> <UNK> by the khan who <UNK> his people by yelling what sounds like <UNK> power the goal of the mission involves the star wars defense system jonathan is trained in the martial arts by princess <UNK> who never speaks or leaves the house once trained tries to blend in with the <UNK> by wearing a bright red <UNK> with <UNK> of blue and white needless to say <UNK> finds himself running and fighting for his life along the stone streets of <UNK> on his way to a date with destiny and the game br br star kurt thomas was ill served by director robert <UNK> who it looks like was never on the set the so called script is just this side of incompetent see other reviews for the many <UNK> throughout the town of <UNK> has a few good moments but is ultimately ruined by bad editing the ending <UNK> still there's the <UNK> of a good action adventure here a hong kong version with more <UNK> action and faster pace might even be pretty good\n"
          ]
        }
      ],
      "source": [
        "word_to_id = imdb.get_word_index()\n",
        "word_to_id = {key:(value+param.index_word_from) for key,value in word_to_id.items()}\n",
        "word_to_id[\"<PAD>\"] = 0\n",
        "word_to_id[\"<START>\"] = 1\n",
        "word_to_id[\"<UNK>\"] = 2\n",
        "\n",
        "id_to_word = {value:key for key,value in word_to_id.items()}\n",
        "print(' '.join(id_to_word[id] for id in X_train[1000] ))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Hfl42LGCugWB",
        "outputId": "6b222a69-e01d-4db2-d2d6-2b484be9a794"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "type(y_train): <class 'numpy.ndarray'>\n",
            "y_train.shape: (25000,)\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "1"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "print(\"type(y_train):\", type(y_train))\n",
        "print(\"y_train.shape:\", y_train.shape)\n",
        "y_train[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iVw65PNNuobX",
        "outputId": "b4524587-5a28-420e-a925-352a9d8a0a5e"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "X_test.shape: (25000,)\n",
            "y_test.shape: (25000,)\n"
          ]
        }
      ],
      "source": [
        "print(\"X_test.shape:\", X_test.shape)\n",
        "print(\"y_test.shape:\", y_test.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "V18OA7oQNH3c"
      },
      "source": [
        "## Data processing\n",
        "\n",
        "Sequences (represented as a list of values) in `X_train` represent the reviews.\n",
        "They can have different length $T_x$.\n",
        "To train the network we should modify them so that they all have the same length `param.T_x`.\n",
        "\n",
        "We do this by:\n",
        "- **truncating** the ones that are too long,\n",
        "- **padding-with-zeros** the ones that are too short.\n",
        "\n",
        "This can be done at the start of the sequence (`pre`) or at the end (`post`).\n",
        "\n",
        "In our use-case (rating of reviews), the decision ($\\hat{y}$) is taken after reading the whole sentence/review ${x^{<t>}}$. Therefore we will truncate and pad-with-zeroes in `pre` mode (truncate the beginning of the sequence if too long, or add zeroes add the beggining of the sequence if too short)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8UIF_RBUkfQI"
      },
      "outputs": [],
      "source": [
        "def do_pad_sequences(sequences, required_len, truncating='pre', padding='pre'):\n",
        "    \"\"\"\n",
        "    Parameters\n",
        "    ----------\n",
        "        sequences:  numpy arrays of lists, shape=(25000,)\n",
        "        required_len:     required length of each sequence after truncating and padding\n",
        "        padding     'pre' or 'post' mode\n",
        "        truncating  'pre' or 'post' mode\n",
        "    Returns\n",
        "    -------\n",
        "        padded_sequences    numpy arrays of lists (each list has now length maxlen)\n",
        "    \"\"\"\n",
        "    padded_sequences = []\n",
        "\n",
        "    for seq in sequences:\n",
        "        if len(seq) > required_len:\n",
        "            # Truncate the sequence\n",
        "            if truncating == 'pre':\n",
        "                seq = seq[-required_len:]\n",
        "            elif truncating == 'post':\n",
        "                seq = seq[:required_len]\n",
        "        elif len(seq) < required_len:\n",
        "            # Pad the sequence\n",
        "            if padding == 'pre':\n",
        "                seq = [0] * (required_len - len(seq)) + seq\n",
        "            elif padding == 'post':\n",
        "                seq = seq + [0] * (required_len - len(seq))\n",
        "\n",
        "        padded_sequences.append(seq)\n",
        "\n",
        "    return np.array(padded_sequences)\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JhmiHsOGoRwT",
        "outputId": "aed2b22e-9a00-4a0f-fb82-ce884a96b830"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "len(X_train[0]): 100\n",
            "len(X_train[1]): 100\n",
            "X_train[0]: [1415   33    6   22   12  215   28   77   52    5   14  407   16   82\n",
            "    2    8    4  107  117    2   15  256    4    2    7 3766    5  723\n",
            "   36   71   43  530  476   26  400  317   46    7    4    2 1029   13\n",
            "  104   88    4  381   15  297   98   32 2071   56   26  141    6  194\n",
            "    2   18    4  226   22   21  134  476   26  480    5  144   30    2\n",
            "   18   51   36   28  224   92   25  104    4  226   65   16   38 1334\n",
            "   88   12   16  283    5   16 4472  113  103   32   15   16    2   19\n",
            "  178   32]\n"
          ]
        }
      ],
      "source": [
        "# --- truncate and pad input sequences\n",
        "X_train = do_pad_sequences(X_train, required_len=param.T_x, padding='pre', truncating='pre')\n",
        "X_test = do_pad_sequences(X_test, required_len=param.T_x, padding='pre', truncating='pre')\n",
        "\n",
        "print(\"len(X_train[0]):\", len(X_train[0]))\n",
        "print(\"len(X_train[1]):\", len(X_train[1]))\n",
        "print(\"X_train[0]:\", X_train[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Define training and testing functions"
      ],
      "metadata": {
        "id": "n8IktAODF8g9"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "def train_one_epoch(model, data_loader, criterion, optimizer):\n",
        "    \"\"\"\n",
        "    \"\"\"\n",
        "\n",
        "    model.train()\n",
        "    total_loss, total_acc = 0, 0\n",
        "    for X, y in data_loader:\n",
        "        optimizer.zero_grad()\n",
        "        hat_y = model(X)\n",
        "        loss = criterion(hat_y.squeeze(), y)\n",
        "        loss.backward() # --- SPECIFIC TO TRAINING\n",
        "        optimizer.step() # --- SPECIFIC TO TRAINING\n",
        "\n",
        "        total_loss += loss.item()\n",
        "        predicted = (hat_y.squeeze() > 0.5).float()\n",
        "        total_acc += (predicted == y).sum().item()/len(y)\n",
        "\n",
        "    return total_loss/len(data_loader), total_acc/len(data_loader)\n",
        "\n",
        "\n",
        "\n",
        "def test_one_epoch(model, data_loader, criterion):\n",
        "    \"\"\"\n",
        "    \"\"\"\n",
        "\n",
        "    model.eval()\n",
        "    total_loss, total_acc =  0, 0\n",
        "    with torch.no_grad():\n",
        "        for X, y in data_loader:\n",
        "            hat_y = model(X)\n",
        "            loss = criterion(hat_y.squeeze(), y)\n",
        "\n",
        "            total_loss += loss.item()\n",
        "            predicted = (hat_y.squeeze() > 0.5).float()\n",
        "            total_acc += (predicted == y).sum().item()/len(y)\n",
        "\n",
        "    return total_loss/len(data_loader), total_acc/len(data_loader)\n",
        "\n",
        "\n",
        "\n",
        "\n",
        "def train(model, train_loader, test_loader, criterion, optimizer, n_epoch):\n",
        "    \"\"\"\n",
        "    \"\"\"\n",
        "\n",
        "    for epoch in range(param.n_epoch):\n",
        "\n",
        "        train_loss, train_acc = train_one_epoch(model, train_loader, criterion, optimizer)\n",
        "        print(f\"Epoch {epoch + 1}, Loss: {train_loss }, Acc: {train_acc} \")\n",
        "\n",
        "        test_loss, test_acc = test_one_epoch(model, test_loader, criterion)\n",
        "        print(f\"\\tValidation Loss: {test_loss }, Acc: {test_acc} \")\n"
      ],
      "metadata": {
        "id": "g45DC3JEF72A"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# --- Convert numpy.array to torch.tensor\n",
        "X_train_tensor = torch.tensor(X_train, dtype=torch.long)\n",
        "y_train_tensor = torch.tensor(y_train, dtype=torch.float32)\n",
        "X_test_tensor = torch.tensor(X_test, dtype=torch.long)\n",
        "y_test_tensor = torch.tensor(y_test, dtype=torch.float32)\n",
        "\n",
        "# Create TensorDatasets for train and test data\n",
        "train_dataset = TensorDataset(X_train_tensor, y_train_tensor)\n",
        "test_dataset = TensorDataset(X_test_tensor, y_test_tensor)\n",
        "\n",
        "# Create DataLoaders for train and test data\n",
        "train_loader = DataLoader(train_dataset, batch_size=param.batch_size, shuffle=True)\n",
        "test_loader = DataLoader(test_dataset, batch_size=param.batch_size, shuffle=False)"
      ],
      "metadata": {
        "id": "j9k06bjfcypD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YlrDTuk5K65Q"
      },
      "source": [
        "# First model\n",
        "\n",
        "\n",
        "\n",
        "In the first model, you will successively\n",
        "- step-1) learn word embeddings $e^{<t>}$ of each item of the inut sequence $x^{(t)}$.\n",
        "    - This is done by learning an embedding matrix $E$. You will use the `nn.Embedding` layer in pytorch.\n",
        "    - In pytorch, the `nn.Embedding` layer does not really perform a matrix multiplication going from one-hot-encoding to embedding (it would be very costly to do that).\n",
        "    - In pytorch `nn.Embedding` is a special layer that goes directly from index-of-the-word-in-the-dictionary to the embedding $e^{(t)}$\n",
        "    - The embedding goes from `param.n_word` dimensions to  `param.n_embedding` dimensions\n",
        "- step-2) compute the average over time $t$ of the embedding $e^{(t)}$ obtained for each word $x^{(t)}$ of a sequence (you should use `torch.mean`)\n",
        "- step-3) apply a fully connected (`nn.linear` layer in pytorch) which output activation is a sigmoid (predicting the 0 or 1 rating)\n",
        "\n",
        "<img src=\"https://perso.telecom-paristech.fr/gpeeters/doc/Lab_DL_RNN_01.png\">\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "zspaUptgtW9l",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "5624d463-c58f-498d-cdab-a0a946b05e08"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([64, 100])\n",
            "torch.Size([64, 1])\n"
          ]
        }
      ],
      "source": [
        "if student:\n",
        "    # --- START CODE HERE (02)\n",
        "    class SimpleModel(nn.Module):\n",
        "        def __init__(self, param):\n",
        "            super(SimpleModel, self).__init__()\n",
        "            # Embedding layer\n",
        "            self.embedding = nn.Embedding(param.n_word, param.n_embedding)\n",
        "            # Fully connected layer\n",
        "            self.fc = nn.Linear(param.n_embedding, param.n_out)\n",
        "            # Sigmoid activation\n",
        "            self.sigmoid = nn.Sigmoid()\n",
        "        def forward(self, x):\n",
        "            # Step 1: Embed the input sequences\n",
        "            embeddings = self.embedding(x)\n",
        "            # Step 2: Compute the mean over time dimension\n",
        "            mean_embeddings = torch.mean(embeddings, dim=1)\n",
        "            # Step 3: Apply fully connected layer and sigmoid activation\n",
        "            out = self.fc(mean_embeddings)\n",
        "            out = self.sigmoid(out)\n",
        "            return out\n",
        "    # --- END CODE HERE\n",
        "\n",
        "\n",
        "\n",
        "# --- Test\n",
        "torch.manual_seed(0)\n",
        "model = SimpleModel(param)\n",
        "print(X_train_tensor[:param.batch_size, :].size())\n",
        "print(model(X_train_tensor[:param.batch_size, :]).size())"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Loss function and optimizer\n",
        "criterion = nn.BCELoss()\n",
        "optimizer = optim.Adam(model.parameters(),param.lr)"
      ],
      "metadata": {
        "id": "-RsBZDShCyag"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Train the model\n",
        "train(model, train_loader, test_loader, criterion, optimizer, param.n_epoch)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "kqQJLiu2c8_M",
        "outputId": "ecbdf089-a3b8-49fb-db4c-b35056a5eda3"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1, Loss: 0.6747531936601605, Acc: 0.618486253196931 \n",
            "\tValidation Loss: 0.6472773219618346, Acc: 0.6908967391304348 \n",
            "Epoch 2, Loss: 0.5944449584502394, Acc: 0.7352062020460358 \n",
            "\tValidation Loss: 0.5432630102042957, Acc: 0.7694213554987213 \n",
            "Epoch 3, Loss: 0.4897417963465766, Acc: 0.7975863171355498 \n",
            "\tValidation Loss: 0.4584000308800231, Acc: 0.8072250639386189 \n",
            "Epoch 4, Loss: 0.41904126911821876, Acc: 0.8285805626598466 \n",
            "\tValidation Loss: 0.41025567489206943, Acc: 0.8243286445012787 \n",
            "Epoch 5, Loss: 0.3767482465337914, Acc: 0.8445812020460358 \n",
            "\tValidation Loss: 0.3832709686378079, Acc: 0.8325207800511509 \n",
            "Epoch 6, Loss: 0.34965070213198357, Acc: 0.8558583759590793 \n",
            "\tValidation Loss: 0.3671487521789873, Acc: 0.8377557544757033 \n",
            "Epoch 7, Loss: 0.3303366067159511, Acc: 0.8615968670076726 \n",
            "\tValidation Loss: 0.3565715521269137, Acc: 0.842391304347826 \n",
            "Epoch 8, Loss: 0.3154794964415338, Acc: 0.8678628516624042 \n",
            "\tValidation Loss: 0.34926891612732197, Acc: 0.8453884271099744 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SBqyzLJRUIsC"
      },
      "source": [
        "## Results\n",
        "\n",
        "After only 8 epochs, you should obtain an accuracy \"around\" 86.7%/ 84.5% for the test data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uRP-h4Xr_HNJ"
      },
      "source": [
        "## Using the trained embedding to find equivalence between words\n",
        "\n",
        "Since the embedding is part of the models, we can look at the trained embedding matrix $E$ and use it to get the most similar words (according to the trained matrix $E$) in the dictionary.\n",
        "You will use the weights of the `nn.Embedding` layer to find the most similar words to `great`. We will use an Euclidean distance for that.\n",
        "\n",
        "- 1) Retrieve the weights of the `nn.Embedding` layer\n",
        "- 2) Get the position of `great` in the dictionary\n",
        "- 3) Knowing this position, get the word-embedding of `great`\n",
        "- 4) Find (using Euclidean distance), the closest embedded-words to `great`\n",
        "\n",
        "Remarks:\n",
        "- you can access a specific layer of the model by using the name you used to define `self.??? = nn.Embedding` in the `__init__`method of `SimpleModel`.\n",
        "- be careful about the order of the dimensions of the embedding matrix `E`"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7xMubRqJ_HNJ",
        "outputId": "f6c4f090-95cb-4665-b182-885e7be1cc6c"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([5000, 32])\n",
            "great\n",
            "fun\n",
            "masterpiece\n",
            "excellent\n",
            "friendship\n"
          ]
        }
      ],
      "source": [
        "if student:\n",
        "    # --- START CODE HERE (03)\n",
        "    E = model.embedding.weight.detach()\n",
        "    print(E.size())\n",
        "\n",
        "    great_embedding = E[word_to_id[\"great\"]]\n",
        "    distance = torch.norm(E - great_embedding, dim=1)\n",
        "    _, closest_id = torch.topk(distance, 5, largest=False)\n",
        "\n",
        "    for id in closest_id:\n",
        "      print(id_to_word[id.item()])\n",
        "    # --- END CODE HERE"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zK9e5Eo1Ks2a"
      },
      "source": [
        "# Second model\n",
        "\n",
        "In the second model, you will replace step-2 (which was \"compute the average over time $t$ of the embedding $e^{(t)}$\") by a RNN layer over time.\n",
        "More precisely you will use a LSTM (`nn.LSTM` layer in pytorch) with `param.n_lstm=100` units (or dimensions) in a Many-To-One configuration\n",
        "\n",
        "Don't forget that in ou data, the first dimension of `X_train/X_test` represents the batch (`batch_first=True` in pytorch).\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "7dl-CSMKoViX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "9a533b4e-0b8e-40bb-b535-0eac2e62c754"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "torch.Size([64, 100])\n",
            "torch.Size([64, 1])\n"
          ]
        }
      ],
      "source": [
        "if student:\n",
        "    # --- START CODE HERE (04)\n",
        "    class LstmModel(nn.Module):\n",
        "        def __init__(self, param):\n",
        "            super(LstmModel, self).__init__()\n",
        "            # Embedding layer\n",
        "            self.embedding = nn.Embedding(param.n_word, param.n_embedding)\n",
        "            # RNN layer\n",
        "            self.lstm = nn.LSTM(param.n_embedding, param.n_lstm, batch_first=True)\n",
        "            # Fully connected layer\n",
        "            self.fc = nn.Linear(param.n_lstm, param.n_out)\n",
        "        def forward(self, x):\n",
        "            # Step 1: Embed the input sequences\n",
        "            embeddings = self.embedding(x)\n",
        "            # Step 2: RNN Layer\n",
        "            embeddings, _ = self.lstm(embeddings)\n",
        "            embeddings = embeddings[:, -1, :]\n",
        "            # Step 3: Apply fully connected layer\n",
        "            out = self.fc(embeddings)\n",
        "            out = torch.sigmoid(out)\n",
        "            return out\n",
        "    # --- END CODE HERE\n",
        "\n",
        "\n",
        "# --- Test\n",
        "torch.manual_seed(0)\n",
        "model = LstmModel(param)\n",
        "print(X_train_tensor[:param.batch_size, :].size())\n",
        "print(model(X_train_tensor[:param.batch_size, :]).size())"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "-bp7PzX7oXtB",
        "scrolled": true
      },
      "outputs": [],
      "source": [
        "# Loss function and optimizer\n",
        "criterion = nn.BCELoss()\n",
        "optimizer = optim.Adam(model.parameters(), param.lr)"
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "# Train the model\n",
        "train(model, train_loader, test_loader, criterion, optimizer, param.n_epoch)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vP8TvdgNiUQd",
        "outputId": "0049adab-3e2b-4a9b-a6ff-9f69238ee9db"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Epoch 1, Loss: 0.6336799868384896, Acc: 0.6224904092071611 \n",
            "\tValidation Loss: 0.5547035950071671, Acc: 0.7137787723785166 \n",
            "Epoch 2, Loss: 0.48217614806826464, Acc: 0.7693254475703324 \n",
            "\tValidation Loss: 0.43780227409451816, Acc: 0.7974104859335039 \n",
            "Epoch 3, Loss: 0.4011488721879852, Acc: 0.8198769181585677 \n",
            "\tValidation Loss: 0.40805759092273614, Acc: 0.8147858056265985 \n",
            "Epoch 4, Loss: 0.3506053713581446, Acc: 0.8477461636828645 \n",
            "\tValidation Loss: 0.381709221721915, Acc: 0.8284127237851663 \n",
            "Epoch 5, Loss: 0.3105229107315278, Acc: 0.8692055626598465 \n",
            "\tValidation Loss: 0.39009654708683034, Acc: 0.8383791560102302 \n",
            "Epoch 6, Loss: 0.2784823154282692, Acc: 0.8848545396419437 \n",
            "\tValidation Loss: 0.36099135765181783, Acc: 0.8423593350383632 \n",
            "Epoch 7, Loss: 0.246628500025748, Acc: 0.9018542199488491 \n",
            "\tValidation Loss: 0.3870372835480039, Acc: 0.849056905370844 \n",
            "Epoch 8, Loss: 0.2231705136730543, Acc: 0.9104699488491048 \n",
            "\tValidation Loss: 0.3902951824809889, Acc: 0.8474984015345268 \n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "F1LN_fjMWBHJ"
      },
      "source": [
        "## Results\n",
        "\n",
        "After only 8 epochs, you should obtain an accuracy around 91.1%/ 84.7% for the test data."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2qBjoRpoqv_g"
      },
      "source": [
        "# Evaluation\n",
        "\n",
        "To evaluate the work, you should rate the code for\n",
        "- 1) Data Pre-Processing (01)\n",
        "- 2) Simple model  (02)\n",
        "- 3) Find equivalence between words (03)\n",
        "- 4) LSTM model (04)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Simple Model: Preferred for simplicity, computational efficiency, and tasks with simpler relationships or limited data.\n",
        "\n",
        "LSTM Model: Preferred for tasks requiring capturing intricate sequential dependencies, contextual understanding, and leveraging larger datasets."
      ],
      "metadata": {
        "id": "Wro6vUgWydZS"
      }
    }
  ],
  "metadata": {
    "accelerator": "TPU",
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3 (ipykernel)",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.16"
    },
    "toc": {
      "base_numbering": 1,
      "nav_menu": {},
      "number_sections": true,
      "sideBar": true,
      "skip_h1_title": false,
      "title_cell": "Table of Contents",
      "title_sidebar": "Contents",
      "toc_cell": false,
      "toc_position": {},
      "toc_section_display": true,
      "toc_window_display": true
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}