04-1. 用RNN做情意分析.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 主題 04-1. 用RNN做情意分析\n",
    "\n",
    "我們終於要介紹三大神經網路的最後一個, 也就是 RNN。RNN 有不少的變型, 例如 LSTM 和 GRU 等等, 不過我們都通稱叫 RNN。RNN 是一種「有記憶」的神經網路, 非常適合時間序列啦, 或是不定長度的輸入資料。\n",
    "\n",
    "我們來看看怎麼樣用 RNN 做電影評論的「情意分析」, 也就是知道一則評論究竟是「正評」還是「負評」。\n",
    "\n",
    "【註】因 TensorFlow 2 已做了一些改變, 例如完全整合了 Keras。到 2021 年的今天, 有一些細節也做了調整。因此我們依新的規範修改了程式。最大的不同是, 以後大家直接安裝 tensorflow 即可, 不用再另外裝 keras。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 初始準備\n",
    "\n",
    "基本上和之前是一樣的, 我們就不再說明。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 讀入 IMDB 電影數據庫\n",
    "\n",
    "今天我們要評入 IMDB 電影數據庫影評的部份。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import imdb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(x_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25000"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(x_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 輸入資料部份\n",
    "\n",
    "我們來看一下輸入部份長什麼樣子?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "218, 189, 141, 550, 147, 43, 123, 562, 233, 130, "
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    print(len(x_train[i]), end=', ')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 輸出資料部份\n",
    "\n",
    "輸出方面應該很容易想像, 我們來看看前 10 筆。結果自然就是 0 (負評) 或 1 (正評)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 0, 0, 1, 0, 0, 1, 0, 1, 0])"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 送入神經網路的輸入處理\n",
    "\n",
    "雖然 RNN 是可以處理不同長度的輸入, 在寫程式時我們還是要\n",
    "\n",
    "* 設輸入文字長度的上限\n",
    "* 把每段文字都弄成一樣長, 太短的後面補上 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.preprocessing import sequence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = sequence.pad_sequences(x_train, maxlen=100)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(25000, 100)"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "至此我們可以來寫我們的第一個 RNN 了!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 打造你的 RNN\n",
    "\n",
    "這裡我們選用 LSTM, 基本上用哪種 RNN 寫法都是差不多的!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 決定神經網路架構\n",
    "\n",
    "* 先將 10000 維的文字壓到 128 維\n",
    "* 然後用 128 個 LSTM\n",
    "* 最後一個 output, 直接用 sigmoid 送出"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 建構我們的神經網路\n",
    "\n",
    "文字我們用 1-hot 表示是很標準的方式, 不過要注意的是, 因為我們指定要 1 萬個字, 所以每個字是用 1 萬維的向量表示! 這一來很浪費記憶空間, 二來字和字間基本上是沒有關係的。我們可以用某種「合理」的方式, 把字壓到比較小的維度, 這些向量又代表某些意思 (比如說兩個字代表的向量角度小表相關程度大) 等等。\n",
    "\n",
    "這聽來很複雜的事叫 \"word embedding\", 而事實上 Keras 會幫我們做。我們只需告訴 Keras 原來最大的數字是多少 (10000), 還有我們打算壓到幾維 (128)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, Embedding\n",
    "from tensorflow.keras.layers import LSTM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Embedding(10000, 128))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(LSTM(150))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(Dense(1, activation='sigmoid'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 組裝\n",
    "\n",
    "這次我們用 binary_crossentropy 做我們的 loss function, 另外用一個很潮的 Adam 學習法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, None, 128)         1280000   \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, 150)               167400    \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1)                 151       \n",
      "=================================================================\n",
      "Total params: 1,447,551\n",
      "Trainable params: 1,447,551\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "125550"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "3*(128+150+1)*150"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "41850"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(128+150+1)*150"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "167400"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "125550 + 41850"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss='binary_crossentropy',\n",
    "            optimizer='adam',\n",
    "            metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 訓練"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/15\n",
      "782/782 [==============================] - 10s 11ms/step - loss: 0.5123 - accuracy: 0.7313\n",
      "Epoch 2/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.2607 - accuracy: 0.8948\n",
      "Epoch 3/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.1821 - accuracy: 0.9326\n",
      "Epoch 4/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.1138 - accuracy: 0.9597\n",
      "Epoch 5/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0812 - accuracy: 0.9734\n",
      "Epoch 6/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0611 - accuracy: 0.9797\n",
      "Epoch 7/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0385 - accuracy: 0.9887\n",
      "Epoch 8/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0260 - accuracy: 0.9918\n",
      "Epoch 9/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0260 - accuracy: 0.9914\n",
      "Epoch 10/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0190 - accuracy: 0.9939\n",
      "Epoch 11/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0263 - accuracy: 0.9921\n",
      "Epoch 12/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0099 - accuracy: 0.9973\n",
      "Epoch 13/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0135 - accuracy: 0.9965\n",
      "Epoch 14/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0097 - accuracy: 0.9970\n",
      "Epoch 15/15\n",
      "782/782 [==============================] - 8s 11ms/step - loss: 0.0124 - accuracy: 0.9967\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fe9eb7591c0>"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(x_train, y_train,\n",
    "         batch_size=32,\n",
    "         epochs=15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 檢視結果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.1 分數\n",
    "\n",
    "我們照例來看看測試資料的分數。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 3s 3ms/step - loss: 0.9819 - accuracy: 0.8173\n"
     ]
    }
   ],
   "source": [
    "score = model.evaluate(x_test, y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "測試資料的 loss 0.9819172620773315\n",
      "測試資料的正確率 0.8172799944877625\n"
     ]
    }
   ],
   "source": [
    "print('測試資料的 loss', score[0])\n",
    "print('測試資料的正確率', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 儲存結果\n",
    "\n",
    "這裡有 8 成我們可以正確分辨, 看來還不差, 照例我們把結果存檔。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1965"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_json = model.to_json()\n",
    "open('imdb_model_architecture.json', 'w').write(model_json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save_weights('imdb_model_weights.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 另一種存的方式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.save('myrnn.h5')"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:tf2py38]",
   "language": "python",
   "name": "conda-env-tf2py38-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}