04-1. 用RNN做情意分析.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 主題 04-1. 用RNN做情意分析\n",
    "\n",
    "我們終於要介紹三大神經網路的最後一個, 也就是 RNN。RNN 有不少的變型, 例如 LSTM 和 GRU 等等, 不過我們都通稱叫 RNN。RNN 是一種「有記憶」的神經網路, 非常適合時間序列啦, 或是不定長度的輸入資料。\n",
    "\n",
    "我們來看看怎麼樣用 RNN 做電影評論的「情意分析」, 也就是知道一則評論究竟是「正評」還是「負評」。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 初始準備\n",
    "\n",
    "基本上和之前是一樣的, 我們就不再說明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: KERAS_BACKEND=tensorflow\n"
     ]
    }
   ],
   "source": [
    "%env KERAS_BACKEND=tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "nbpresent": {
     "id": "a73ce944-1f06-4b48-91eb-da5ff43936f9"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 讀入 IMDB 電影數據庫\n",
    "\n",
    "今天我們要評入 IMDB 電影數據庫影評的部份。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import imdb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要注意這裡我們限制只選「最常用」1 萬字, 也就是超過這範圍的就當不存在。這是文字分析常會做的事。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "訓練資料總筆數 = 25000\n",
      "測試資料總筆數 = 25000\n"
     ]
    }
   ],
   "source": [
    "print(\"訓練資料總筆數 =\", len(x_train))\n",
    "print(\"測試資料總筆數 =\", len(x_test))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 輸入資料部份\n",
    "\n",
    "我們來看一下輸入部份長什麼樣子?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[1,\n",
       " 1230,\n",
       " 3765,\n",
       " 566,\n",
       " 97,\n",
       " 189,\n",
       " 102,\n",
       " 86,\n",
       " 7,\n",
       " 32,\n",
       " 4,\n",
       " 973,\n",
       " 16,\n",
       " 55,\n",
       " 355,\n",
       " 18,\n",
       " 14,\n",
       " 20,\n",
       " 4,\n",
       " 64,\n",
       " 542,\n",
       " 173,\n",
       " 16,\n",
       " 4,\n",
       " 893,\n",
       " 2115,\n",
       " 5376,\n",
       " 250,\n",
       " 39,\n",
       " 8013,\n",
       " 4,\n",
       " 1362,\n",
       " 2,\n",
       " 14,\n",
       " 102,\n",
       " 47,\n",
       " 57,\n",
       " 599,\n",
       " 633,\n",
       " 6,\n",
       " 1317,\n",
       " 2,\n",
       " 8,\n",
       " 6,\n",
       " 189,\n",
       " 20,\n",
       " 57,\n",
       " 206,\n",
       " 57,\n",
       " 116,\n",
       " 5,\n",
       " 57,\n",
       " 836,\n",
       " 82,\n",
       " 6,\n",
       " 1317,\n",
       " 2,\n",
       " 3728,\n",
       " 2,\n",
       " 9,\n",
       " 6,\n",
       " 52,\n",
       " 284,\n",
       " 21,\n",
       " 29,\n",
       " 9,\n",
       " 38,\n",
       " 2245,\n",
       " 5,\n",
       " 1044,\n",
       " 11,\n",
       " 14,\n",
       " 15,\n",
       " 45,\n",
       " 619,\n",
       " 50,\n",
       " 71,\n",
       " 6,\n",
       " 171,\n",
       " 531,\n",
       " 15,\n",
       " 71,\n",
       " 424,\n",
       " 8,\n",
       " 30,\n",
       " 163,\n",
       " 6211,\n",
       " 4,\n",
       " 1629,\n",
       " 189,\n",
       " 212,\n",
       " 102,\n",
       " 5,\n",
       " 57,\n",
       " 31,\n",
       " 1498,\n",
       " 11,\n",
       " 4,\n",
       " 311,\n",
       " 13,\n",
       " 197,\n",
       " 15,\n",
       " 14,\n",
       " 20,\n",
       " 16,\n",
       " 1150,\n",
       " 1479,\n",
       " 5,\n",
       " 13,\n",
       " 161,\n",
       " 990,\n",
       " 692,\n",
       " 5,\n",
       " 1706,\n",
       " 12,\n",
       " 69,\n",
       " 77,\n",
       " 1194,\n",
       " 8,\n",
       " 3245,\n",
       " 2001,\n",
       " 553,\n",
       " 67,\n",
       " 14,\n",
       " 20,\n",
       " 48,\n",
       " 25,\n",
       " 423,\n",
       " 13,\n",
       " 131,\n",
       " 124,\n",
       " 51,\n",
       " 25,\n",
       " 122,\n",
       " 236,\n",
       " 1506,\n",
       " 198,\n",
       " 4,\n",
       " 64,\n",
       " 552,\n",
       " 7,\n",
       " 415,\n",
       " 37,\n",
       " 62,\n",
       " 169,\n",
       " 14,\n",
       " 20,\n",
       " 60,\n",
       " 2602,\n",
       " 629,\n",
       " 5,\n",
       " 615,\n",
       " 14,\n",
       " 9,\n",
       " 8,\n",
       " 25,\n",
       " 1230,\n",
       " 3765,\n",
       " 570,\n",
       " 231,\n",
       " 189,\n",
       " 102,\n",
       " 14,\n",
       " 20,\n",
       " 166,\n",
       " 2039,\n",
       " 168,\n",
       " 40,\n",
       " 2450,\n",
       " 5486,\n",
       " 3298]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[99]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "注意這其實是一個 list 而不是 array, 原因是每筆資料 (每段影評) 長度自然是不一樣的! 我們檢查一下前 10 筆的長度就可以知道。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(x_train[99])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "218, 189, 141, 550, 147, 43, 123, 562, 233, 130, "
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    print(len(x_train[i]), end=', ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "最後要說明的是, 在每筆輸入資料的數字都代表英文的一個單字。編號方式是在我們資料庫裡所有文字的排序: 也就是出現頻率越高, 代表的數字就越小。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 輸出資料部份\n",
    "\n",
    "輸出方面應該很容易想像, 我們來看看前 10 筆。結果自然就是 0 (負評) 或 1 (正評)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 0, 1, 0, 0, 1, 0, 1, 0])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 送入神經網路的輸入處理\n",
    "\n",
    "雖然 RNN 是可以處理不同長度的輸入, 在寫程式時我們還是要\n",
    "\n",
    "* 設輸入文字長度的上限\n",
    "* 把每段文字都弄成一樣長, 太短的後面補上 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.preprocessing import sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = sequence.pad_sequences(x_train, maxlen=100)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 100)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  14,   15,   45,  619,   50,   71,    6,  171,  531,   15,   71,\n",
       "        424,    8,   30,  163, 6211,    4, 1629,  189,  212,  102,    5,\n",
       "         57,   31, 1498,   11,    4,  311,   13,  197,   15,   14,   20,\n",
       "         16, 1150, 1479,    5,   13,  161,  990,  692,    5, 1706,   12,\n",
       "         69,   77, 1194,    8, 3245, 2001,  553,   67,   14,   20,   48,\n",
       "         25,  423,   13,  131,  124,   51,   25,  122,  236, 1506,  198,\n",
       "          4,   64,  552,    7,  415,   37,   62,  169,   14,   20,   60,\n",
       "       2602,  629,    5,  615,   14,    9,    8,   25, 1230, 3765,  570,\n",
       "        231,  189,  102,   14,   20,  166, 2039,  168,   40, 2450, 5486,\n",
       "       3298], dtype=int32)"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train[99]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此我們可以來寫我們的第一個 RNN 了!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 打造你的 RNN\n",
    "\n",
    "這裡我們選用 LSTM, 基本上用哪種 RNN 寫法都是差不多的!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 決定神經網路架構\n",
    "\n",
    "* 先將 10000 維的文字壓到 128 維\n",
    "* 然後用 128 個 LSTM\n",
    "* 最後一個 output, 直接用 sigmoid 送出"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 建構我們的神經網路\n",
    "\n",
    "文字我們用 1-hot 表示是很標準的方式, 不過要注意的是, 因為我們指定要 1 萬個字, 所以每個字是用 1 萬維的向量表示! 這一來很浪費記憶空間, 二來字和字間基本上是沒有關係的。我們可以用某種「合理」的方式, 把字壓到比較小的維度, 這些向量又代表某些意思 (比如說兩個字代表的向量角度小表相關程度大) 等等。\n",
    "\n",
    "這聽來很複雜的事叫 \"word embedding\", 而事實上 Keras 會幫我們做。我們只需告訴 Keras 原來最大的數字是多少 (10000), 還有我們打算壓到幾維 (128)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Embedding\n",
    "from keras.layers import LSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "nbpresent": {
     "id": "924cda44-78f6-4bbc-a669-68e5bdeb4704"
    }
   },
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "nbpresent": {
     "id": "654a314d-0144-4421-a293-3586e91cfedd"
    }
   },
   "outputs": [],
   "source": [
    "model.add(Embedding(10000, 128))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LSTM 層, 我們做 150 個 LSTM Cells。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "nbpresent": {
     "id": "b8e3a157-5933-4b63-b156-fe4bdf8905a4"
    }
   },
   "outputs": [],
   "source": [
    "model.add(LSTM(150))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "單純透過 sigmoid 輸出。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "nbpresent": {
     "id": "18b410f8-45b0-4188-95c2-c30cc76d07b8"
    }
   },
   "outputs": [],
   "source": [
    "model.add(Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, None, 128)         1280000   \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 150)               167400    \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 151       \n",
      "=================================================================\n",
      "Total params: 1,447,551\n",
      "Trainable params: 1,447,551\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "125550"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "3*(128+150+1)*150"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41850"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(128+150+1)*150"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "167400"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "41850 + 125550"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 組裝\n",
    "\n",
    "這次我們用 binary_crossentropy 做我們的 loss function, 另外用一個很潮的 Adam 學習法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "nbpresent": {
     "id": "9593a32d-9089-48fb-8e65-f39cb0137f08"
    }
   },
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy', \n",
    "              optimizer='adam', \n",
    "              metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 訓練"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我們用的 embedding 中, 會被 batch_size 影響輸入。輸入的 shape 會是\n",
    "\n",
    "    (batch_size, 每筆上限)\n",
    "    \n",
    "也就是 (32,100) 輸出是 (32,100,128), 其中 128 是我們決定要壓成幾維的向量。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "nbpresent": {
     "id": "29d4727a-4b5a-463c-9cc7-1a7df697e774"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "25000/25000 [==============================] - 57s 2ms/step - loss: 0.4171 - acc: 0.8065\n",
      "Epoch 2/15\n",
      "25000/25000 [==============================] - 56s 2ms/step - loss: 0.2665 - acc: 0.8954\n",
      "Epoch 3/15\n",
      "25000/25000 [==============================] - 56s 2ms/step - loss: 0.1960 - acc: 0.9234\n",
      "Epoch 4/15\n",
      "25000/25000 [==============================] - 56s 2ms/step - loss: 0.1394 - acc: 0.9486\n",
      "Epoch 5/15\n",
      "25000/25000 [==============================] - 56s 2ms/step - loss: 0.0951 - acc: 0.9657\n",
      "Epoch 6/15\n",
      "25000/25000 [==============================] - 56s 2ms/step - loss: 0.0793 - acc: 0.9726\n",
      "Epoch 7/15\n",
      "25000/25000 [==============================] - 56s 2ms/step - loss: 0.0558 - acc: 0.9814\n",
      "Epoch 8/15\n",
      "25000/25000 [==============================] - 56s 2ms/step - loss: 0.0359 - acc: 0.9880\n",
      "Epoch 9/15\n",
      "25000/25000 [==============================] - 56s 2ms/step - loss: 0.0245 - acc: 0.9926\n",
      "Epoch 10/15\n",
      "25000/25000 [==============================] - 55s 2ms/step - loss: 0.0263 - acc: 0.9919\n",
      "Epoch 11/15\n",
      "25000/25000 [==============================] - 55s 2ms/step - loss: 0.0348 - acc: 0.9884\n",
      "Epoch 12/15\n",
      "25000/25000 [==============================] - 55s 2ms/step - loss: 0.0271 - acc: 0.9910\n",
      "Epoch 13/15\n",
      "25000/25000 [==============================] - 55s 2ms/step - loss: 0.0134 - acc: 0.9958\n",
      "Epoch 14/15\n",
      "25000/25000 [==============================] - 55s 2ms/step - loss: 0.0084 - acc: 0.9978\n",
      "Epoch 15/15\n",
      "25000/25000 [==============================] - 55s 2ms/step - loss: 0.0121 - acc: 0.9967\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fbd63d364a8>"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train, \n",
    "          batch_size=32, \n",
    "          epochs=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 檢視結果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 分數\n",
    "\n",
    "我們照例來看看測試資料的分數。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000/25000 [==============================] - 11s 423us/step\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "測試資料的 loss: 0.8791450363755227\n",
      "測試資料正確率: 0.82544\n"
     ]
    }
   ],
   "source": [
    "print('測試資料的 loss:', score[0])\n",
    "print('測試資料正確率:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 儲存結果\n",
    "\n",
    "這裡有 8 成我們可以正確分辨, 看來還不差, 照例我們把結果存檔。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('imdb_rnn.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": true,
    "nbpresent": {
     "id": "dbc1a7a3-89db-4cbf-872c-b8ebcd035037"
    }
   },
   "outputs": [],
   "source": [
    "# model_json = model.to_json()\n",
    "# open('imdb_model_architecture.json', \n",
    "#      'w').write(model_json)\n",
    "# model.save_weights('imdb_model_weights.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "`"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
<<<<<<< HEAD
   "version": "3.6.5"
=======
   "version": "3.6.6"
>>>>>>> 54608a95a9c2e225243a1e873f0d9d03f08899d6
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}