05-1. 建立神經網路的其他方式並學習第一個轉移學習模型.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 主題 05-1. 用不同方式使用 Sequential 及學習建立第一個轉移學習模型\n",
    "\n",
    "讓我們回顧一下生命中第一個做出來的神經網路..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. 初始準備"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: KERAS_BACKEND=tensorflow\n"
     ]
    }
   ],
   "source": [
    "%env KERAS_BACKEND=tensorflow"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\oroch\\Anaconda3\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    }
   ],
   "source": [
    "# Keras functions\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Activation\n",
    "from keras.optimizers import SGD\n",
    "\n",
    "# Keras dataset\n",
    "from keras.datasets import mnist\n",
    "\n",
    "# Keras utilis function\n",
    "from keras.utils import np_utils"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. 讀入 MNIST 數據庫"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 由 Keras 讀入 MNIST\n",
    "Keras 很貼心的幫我們準備好 MNIST 數據庫, 我們可以這樣讀進來 (第一周課程中已經讀過)。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/img-datasets/mnist.npz\n",
      "11493376/11490434 [==============================] - 10s 1us/step\n"
     ]
    }
   ],
   "source": [
    "(x_train, y_train), (x_test, y_test) = mnist.load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "資料的長相"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "總共有 60000 訓練資料，每筆資料的尺寸為 28 x 28\n",
      "總共有 10000 測試資料，每筆資料的尺寸為 28 x 28\n"
     ]
    }
   ],
   "source": [
    "print(\"總共有 %d 訓練資料，每筆資料的尺寸為 %d x %d\" %x_train.shape)\n",
    "print(\"總共有 %d 測試資料，每筆資料的尺寸為 %d x %d\" %x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 輸入格式整理\n",
    "\n",
    "我們現在要用標準神經網路學學手寫辨識。原來的每筆數據是個 28x28 的矩陣 (array), 但標準神經網路只吃「平平的」, 也就是每次要 28x28=784 長的向量。因此我們要用 `reshape` 調校一下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train = x_train.reshape(60000, 784)\n",
    "x_test = x_test.reshape(10000, 784)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "為了後面需要，我們把訓練/測試資料中的數字 0, 1 資料，複製一份出來"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_train_01 = x_train[y_train <= 1]\n",
    "x_test_01 = x_test[y_test <= 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "並將 label 轉換成 one-hot encoding 的形式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train_10 = np_utils.to_categorical(y_train, 10)\n",
    "y_test_10 = np_utils.to_categorical(y_test, 10)\n",
    "\n",
    "y_train_01 = y_train[y_train <= 1]\n",
    "y_train_01 = np_utils.to_categorical(y_train_01, 2)\n",
    "\n",
    "y_test_01 = y_test[y_test <= 1]\n",
    "y_test_01 = np_utils.to_categorical(y_test_01, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "養成良好的習慣，適時的確認資料的大小以確保資料的一致性"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((12665, 784), (2115, 784))"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_01.shape, x_test_01.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((12665, 2), (2115, 2))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train_01.shape, y_test_01.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. 回顧一下 Sequential API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在一開始的時候，我們以下列的方式建立了一個具有下列設定\n",
    "\n",
    "* 使用 <span style=\"color:red;\">2</span> 個 hidden layers\n",
    "* 每個 hidden layer 用 <span style=\"color:red;\">500</span> 個神經元\n",
    "* Activation Function 唯一指名 <span style=\"color:red;\">sigmoid</span>\n",
    "\n",
    "的神經網路，建立指令是透過建立 `Sequential()` 和 `.add` 的方式逐層建立，如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_1 (Dense)              (None, 500)               392500    \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 500)               250500    \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 10)                5010      \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 648,010\n",
      "Trainable params: 648,010\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "# Construct a sandbox to put layers inside\n",
    "model = Sequential()\n",
    "\n",
    "# Put fully-connected layers (Dense) inside \n",
    "model.add(Dense(500, input_dim=784))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(500))\n",
    "model.add(Activation('sigmoid'))\n",
    "model.add(Dense(10))\n",
    "model.add(Activation('softmax'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 觀察 model.layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "觀察 `model.layers`，可以發現 `model` 其實就是一堆神經網路層疊起來。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<keras.layers.core.Dense at 0x251a1e58668>,\n",
       " <keras.layers.core.Activation at 0x251a1221ef0>,\n",
       " <keras.layers.core.Dense at 0x251a1e58be0>,\n",
       " <keras.layers.core.Activation at 0x251a1e58ba8>,\n",
       " <keras.layers.core.Dense at 0x251a1e715f8>,\n",
       " <keras.layers.core.Activation at 0x251a1e71a90>]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.layers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "換言之，剛剛每一個 `.add` 其實在做的事情就是：\n",
    "\n",
    "`model.add(Dense(500, input_dim=784))` 是將 `<keras.layers.core.Activation at 0xe558ef0>` 加進 model.layers\n",
    "\n",
    "`model.add(Activation('sigmoid'))` 是將 `<keras.layers.core.Dense at 0xe58a278>` 加入 model.layers\n",
    "\n",
    "`model.add(Dense(500))` 是將 `<keras.layers.core.Dense at 0xe58a278>` 加入 model.layers\n",
    "\n",
    "`model.add(Activation('sigmoid'))` 是將 `<keras.layers.core.Activation at 0xe558d68>` 加入 model.layers\n",
    "\n",
    "`model.add(Dense(10))` 是將 `<keras.layers.core.Activation at 0xe558d68>` 加入 model.layers\n",
    "\n",
    "`model.add(Activation('softmax'))` 是將 `<keras.layers.core.Activation at 0xe58a898>` 加入 model.layers\n",
    "\n",
    "* 這邊的 at 0xe558ef0 代表的是記憶體位置，每次執行都會不一樣，所以和上面結果不同是正常的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 以 list 的形式使用 Sequential API"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "換言之，神經網路其實就是將隱藏層逐層堆疊在一起的 list，因此，我們也可以 list 的形式來建立相同的神經網路。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我們將兩個隱藏層及其 Activation Function 分別寫在 list 中，如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "first_layer = [Dense(500, input_dim=784), \n",
    "               Activation('sigmoid')]\n",
    "\n",
    "second_layer = [Dense(500), \n",
    "                Activation('sigmoid')]\n",
    "\n",
    "output_layer = [Dense(10), \n",
    "                Activation('softmax')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "從基本的 Python 資料結構中，我們知道 list 可以用 `+` 來進行合併，所以我們先來看看這三個 list 合併後的樣子。 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<keras.layers.core.Dense at 0x251a1e582e8>,\n",
       " <keras.layers.core.Activation at 0x251a1e58470>,\n",
       " <keras.layers.core.Dense at 0x251a1e58240>,\n",
       " <keras.layers.core.Activation at 0x251a1e58780>,\n",
       " <keras.layers.core.Dense at 0x251a1e584a8>,\n",
       " <keras.layers.core.Activation at 0x251a1ea48d0>]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "first_layer + second_layer + output_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "合併起來的 list 看起來就像是**某個** `model.layers` 一樣，我們接著要做的，就讓這個 list **真的**變成某個神經網路的 `.layers`\n",
    "\n",
    "很簡單，只要將寫成 list 的隱藏層 `+` 起來送進 `Sequential` 中即可。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "model = Sequential(first_layer + second_layer + output_layer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_4 (Dense)              (None, 500)               392500    \n",
      "_________________________________________________________________\n",
      "activation_4 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 500)               250500    \n",
      "_________________________________________________________________\n",
      "activation_5 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 10)                5010      \n",
      "_________________________________________________________________\n",
      "activation_6 (Activation)    (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 648,010\n",
      "Trainable params: 648,010\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Q: 用 `.add` 和用 list 寫法建立的神經網路之差異？\n",
    "\n",
    "A: 沒有任何差別，前者可以很直覺的建立神經網路，但後者則為使用轉移學習(transfer Learning)的手法之一，前者雖也可做，但較麻煩。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. 情境題:假設我們手上有一個好棒棒的 MNIST 手寫辨識模型，但我今天想建立可以辨識 0 或 1 的模型，除了最後一層，想沿用前兩層的網路設定及結構，我該怎麼做？"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先，我們準備一個上面一樣的神經網路手寫辨識模型，除了最後一層之外都被包在一起。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_7 (Dense)              (None, 500)               392500    \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 500)               250500    \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_9 (Dense)              (None, 10)                5010      \n",
      "_________________________________________________________________\n",
      "activation_9 (Activation)    (None, 10)                0         \n",
      "=================================================================\n",
      "Total params: 648,010\n",
      "Trainable params: 648,010\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "all_except_last = [Dense(500, input_dim=784), \n",
    "                   Activation('sigmoid'),\n",
    "                   Dense(500), \n",
    "                   Activation('sigmoid')]\n",
    "\n",
    "output_layer = [Dense(10), \n",
    "                Activation('softmax')]\n",
    "\n",
    "model_0_to_9 = Sequential(all_except_last + output_layer)\n",
    "model_0_to_9.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "建立完成後，我們讀取第一周已經訓練好的神經網路權重。\n",
    "\n",
    "若同學們沒有 `handwriting_model_weights.h5` 這份模型的權重檔，請至 https://github.com/yenlung/Deep-Learning-MOOC 下載"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_0_to_9.load_weights('handwriting_model_weights.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "由於我們沒有要真的使用這個手寫辨識模型，所以不需要 compile、fit、predict 或 evaluate。\n",
    "\n",
    "接著，我們定義新的 output layer。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_7 (Dense)              (None, 500)               392500    \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 500)               250500    \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 2)                 1002      \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 2)                 0         \n",
      "=================================================================\n",
      "Total params: 644,002\n",
      "Trainable params: 644,002\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "new_output_layer = [Dense(2), \n",
    "                    Activation('softmax')]\n",
    "\n",
    "model_0_to_1 = Sequential(all_except_last + new_output_layer)\n",
    "model_0_to_1.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "要注意的是，如果我們僅沿用而不訓練到前兩層，我們可以透過下面的方式將借過來的神經網路 **冷凍** 起來"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "for layer in all_except_last:\n",
    "    layer.trainable = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**冷凍**後的神經網路的 summary 會有些變化，你有發現嗎? ：)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "dense_7 (Dense)              (None, 500)               392500    \n",
      "_________________________________________________________________\n",
      "activation_7 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 500)               250500    \n",
      "_________________________________________________________________\n",
      "activation_8 (Activation)    (None, 500)               0         \n",
      "_________________________________________________________________\n",
      "dense_10 (Dense)             (None, 2)                 1002      \n",
      "_________________________________________________________________\n",
      "activation_10 (Activation)   (None, 2)                 0         \n",
      "=================================================================\n",
      "Total params: 644,002\n",
      "Trainable params: 1,002\n",
      "Non-trainable params: 643,000\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model_0_to_1.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "接著，我們來訓練這個(有一部分架構及權重跟別人借用的) 0 或 1 手寫辨識模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "model_0_to_1.compile(loss='mse', optimizer=SGD(lr=0.1), metrics=['accuracy'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. 訓練你的第一個透過轉移學習學到的神經網路\n",
    "\n",
    "恭喜! 我們完成了第一個 transfer leraning 的神經網路。這裡我們還有兩件事要決定:\n",
    "\n",
    "* 一次要訓練幾筆資料 (`batch_size`), 我們就 100 筆調一次參數好了\n",
    "* 這 ~~6 萬筆資料~~ 12665 筆資料一共要訓練幾次 (`epochs`), 我們訓練個 5 次試試 (因為只剩 0 或 1的資料了，訓練太多容易 over-fitting)\n",
    "\n",
    "於是最精彩的就來了。你要有比第一周快上100倍的心理準備... (這是因為訓練資料只剩下 1/5，且**可訓練**權重數量從 64萬變成 1千)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((12665, 784), (12665, 2))"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "x_train_01.shape, y_train_01.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/5\n",
      "12665/12665 [==============================] - 1s 79us/step - loss: 0.0183 - acc: 0.9806\n",
      "Epoch 2/5\n",
      "12665/12665 [==============================] - 0s 29us/step - loss: 0.0032 - acc: 0.9976\n",
      "Epoch 3/5\n",
      "12665/12665 [==============================] - 0s 29us/step - loss: 0.0026 - acc: 0.9978\n",
      "Epoch 4/5\n",
      "12665/12665 [==============================] - 0s 26us/step - loss: 0.0023 - acc: 0.9979\n",
      "Epoch 5/5\n",
      "12665/12665 [==============================] - 0s 26us/step - loss: 0.0021 - acc: 0.9979\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x251bf3dd518>"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_0_to_1.fit(x_train_01, y_train_01, batch_size=100, epochs=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2115/2115 [==============================] - 0s 46us/step\n"
     ]
    }
   ],
   "source": [
    "score = model_0_to_1.evaluate(x_test_01, y_test_01)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "測試資料的 loss: 0.0014183477335526931\n",
      "測試資料正確率: 0.9985815602836879\n"
     ]
    }
   ],
   "source": [
    "print('測試資料的 loss:', score[0])\n",
    "print('測試資料正確率:', score[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. 恭喜你完成了第一個透過轉移學習得到的神經網路模型！"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "雖然這個模型看起來很隨便，但轉移學習的模型**差不多**都是這樣建立的，實際上， Keras 亦提供許多被證實有良好表現且訓練好 (pre-trained) 的模型，如:\n",
    "\n",
    "* Xception\n",
    "* VGG16\n",
    "* VGG19\n",
    "* ResNet50\n",
    "* InceptionV3\n",
    "* InceptionResNetV2\n",
    "* MobileNet\n",
    "* DenseNet\n",
    "* NASNet\n",
    "\n",
    "詳細的使用方式可參考 Keras Documentation: https://keras.io/applications/\n",
    "\n",
    "但使用這些模型進行轉移學習，**可能**需要其他更彈性的神經網路寫法，更多神經網路的建構技巧，待下次課程繼續。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}