deep_learning/time_series_sequences/TF2Keras_IMDB_Classification.ipynb

{"nbformat":4,"nbformat_minor":0,"metadata":{"colab":{"name":"TF2Keras_IMDB_Classification.ipynb","provenance":[],"collapsed_sections":[]},"kernelspec":{"name":"python3","display_name":"Python 3"},"accelerator":"GPU"},"cells":[{"cell_type":"code","metadata":{"id":"InLTVsROQsxb","colab_type":"code","colab":{}},"source":["!pip install tensorflow-gpu==2.0.0\n","!pip install -q tensorflow-hub\n","!pip install -q tensorflow-datasets"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"9_q8hliuQ8Er","colab_type":"code","colab":{}},"source":["import tensorflow as tf\n","import tensorflow_hub as hub\n","import tensorflow_datasets as tfds\n","from tensorflow.python.client import device_lib\n","import numpy as np\n","import matplotlib.pyplot as plt\n","\n","tfds.disable_progress_bar()\n","\n","print(\"TF Version: {}\".format(tf.__version__))\n","print(\"Eager model: {}\".format(tf.executing_eagerly()))\n","print(\"TFHub Version: {}\".format(hub.__version__))\n","print(\"TFDS Version: {}\".format(tfds.__version__))\n","print(\"GPU {} available\".format(\"is\" if tf.config.experimental.list_physical_devices(\"GPU\") else \"is not\"))\n","print(\"Devices: {}\".format(device_lib.list_local_devices()))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"PWVVD7F5b8f7","colab_type":"text"},"source":["# Download IMDB Datasets"]},{"cell_type":"markdown","metadata":{"id":"RF1Uh2C-dHXp","colab_type":"text"},"source":["List all available datasets in Tensorflow datasets."]},{"cell_type":"code","metadata":{"id":"EFI2I0g-cY5e","colab_type":"code","colab":{}},"source":["tfds.list_builders()"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"JICIugAXc0iI","colab_type":"text"},"source":["Dwonload the IMDB datasets to the local (default: `$HOME/tensorflow_datasets/imdb_reviews`).\n","\n","Split the training dataset into two sub-datasets, training and validation, in a ratio of 6:4."]},{"cell_type":"code","metadata":{"id":"LpyIIzt8VwPk","colab_type":"code","colab":{}},"source":["train_validation_split = tfds.Split.TRAIN.subsplit([6, 4])"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"JM0IJgUZcQA_","colab_type":"code","colab":{}},"source":["(train_data, val_data), test_data = tfds.load(\n","    name=\"imdb_reviews\", split=(train_validation_split, tfds.Split.TEST), as_supervised=True)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"TCGrAisqdSV0","colab_type":"text"},"source":["## Explore the Dataset"]},{"cell_type":"markdown","metadata":{"id":"MvW8B70MSniP","colab_type":"text"},"source":["It is easy for you to take a batch of data via `batch()`."]},{"cell_type":"code","metadata":{"id":"t2qvlrTScxXy","colab_type":"code","colab":{}},"source":["train_sentences_batch, train_lables_batch = next(iter(train_data.batch(10)))"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"oxdF132GeD6Y","colab_type":"code","colab":{}},"source":["train_sentences_batch"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"D-Mf2xskeXvN","colab_type":"text"},"source":["The label represents which is a positive(1) or a negative(0) review."]},{"cell_type":"code","metadata":{"id":"ujdRKt7seSaS","colab_type":"code","colab":{}},"source":["train_lables_batch"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"A1_aUfYtetRb","colab_type":"text"},"source":["# Building the Model"]},{"cell_type":"markdown","metadata":{"id":"lK4gNKc8fDB_","colab_type":"text"},"source":["One of the best practices is to present the text as an embedding vector. There are several advantages, including no worrying about text preprocessing, benefiting from the transfer learning and the fixed size of the embedding vector.\n","\n","The following we are going to use a pre-trained model (https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1) from `TF.hub`."]},{"cell_type":"code","metadata":{"id":"6lgo9sFCeWUC","colab_type":"code","colab":{}},"source":["embedding = \"https://tfhub.dev/google/tf2-preview/gnews-swivel-20dim/1\""],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"--wTbHwVggBG","colab_type":"code","colab":{}},"source":["hub_layer = hub.KerasLayer(embedding, input_shape=[], dtype=tf.string, trainable=True)\n","hub_layer(train_sentences_batch[:3])"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"62Uu4vWbgyau","colab_type":"text"},"source":["After creating an embedding layer for the sentences, you can create a model for classifying them."]},{"cell_type":"code","metadata":{"id":"Wo6mfU7YgvK-","colab_type":"code","colab":{}},"source":["def build_model(inputs):\n","  x = hub_layer(inputs)\n","  x = tf.keras.layers.Dense(16, activation='relu')(x)\n","  x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n","  return x"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"n6tOGW60hYNV","colab_type":"code","colab":{}},"source":["inputs = tf.keras.Input(shape=[], dtype=tf.string)\n","outputs = build_model(inputs)\n","model = tf.keras.Model(inputs, outputs)\n","model.summary()"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"FaBSArfxi-Kd","colab_type":"text"},"source":["## Define the Loss and Metrics Functions"]},{"cell_type":"code","metadata":{"id":"TAdODOOIhlTs","colab_type":"code","colab":{}},"source":["model.compile(optimizer='adam', \n","              loss='binary_crossentropy', \n","              metrics=['accuracy'])"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"cHnvipX7mOEi","colab_type":"text"},"source":["## Training the Model"]},{"cell_type":"code","metadata":{"id":"H4Ye6tDkmMhs","colab_type":"code","colab":{}},"source":["model.fit(train_data.shuffle(10000).batch(512), \n","          epochs=20, \n","          validation_data=val_data.batch(512), \n","          verbose=1)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"0gUBO7ZxnQnL","colab_type":"text"},"source":["## Evaluation"]},{"cell_type":"code","metadata":{"id":"W1WFuA9TmmEi","colab_type":"code","colab":{}},"source":["test_loss, test_acc = model.evaluate(test_data.batch(512), verbose=2)\n","print(test_loss, test_acc)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"axYyfaj8Zm6F","colab_type":"text"},"source":["# Advacned Text Classification with Preprocessing Text"]},{"cell_type":"markdown","metadata":{"id":"4aHAYsghttnc","colab_type":"text"},"source":["## Preprocessing Text"]},{"cell_type":"markdown","metadata":{"id":"CySfHNZNsrdN","colab_type":"text"},"source":["You can also choose to load the IMDB datasets with ~8K words."]},{"cell_type":"code","metadata":{"id":"5MSsDBFwsqpM","colab_type":"code","colab":{}},"source":["(train_data_8k, test_data_8k), info = tfds.load(\n","    name=\"imdb_reviews/subwords8k\", \n","    split = (tfds.Split.TRAIN, tfds.Split.TEST),\n","    as_supervised = True,\n","    with_info = True    # also return the `info` structure\n",")"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qMezStNpOfZv","colab_type":"text"},"source":["The dataset `info` includes the text encoder (`tfds.features.text.SubwordTextEncoder`)."]},{"cell_type":"code","metadata":{"id":"yKhCxIIeOsUA","colab_type":"code","colab":{}},"source":["encoder = info.features[\"text\"].encoder"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"XXbHxzNMPSfF","colab_type":"code","colab":{}},"source":["print('Vocabulary Size in the Encoder: {}.'.format(encoder.vocab_size))\n","print('Subwords in the Encoder: {}.'.format(encoder.subwords))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"wZg2LJHXPu7U","colab_type":"text"},"source":["This text encoder would help to encode the string."]},{"cell_type":"code","metadata":{"id":"SFfxuwcuPnld","colab_type":"code","colab":{}},"source":["sample_str = \"Hello world, Tensorflow!\""],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"v9fkMfrKP-as","colab_type":"code","colab":{}},"source":["encoded = encoder.encode(sample_str)\n","encoded"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"1XOawWOpQvbl","colab_type":"code","colab":{}},"source":["decoded_str = encoder.decode(encoded)\n","decoded_str"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"Nov2fgVwQ13x","colab_type":"code","colab":{}},"source":["assert decoded_str == sample_str"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"hAieQSPPQ_bc","colab_type":"text"},"source":["You can easily decode the encoded code."]},{"cell_type":"code","metadata":{"id":"Sq7mNrJNQ-o5","colab_type":"code","colab":{}},"source":["for enc in encoded:\n","  print(\"{} -> {}\".format(enc, encoder.decode([enc])))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"IokUYruZSxfF","colab_type":"text"},"source":["## Explore Encoded Data"]},{"cell_type":"markdown","metadata":{"id":"IkohuECpIhvl","colab_type":"text"},"source":["A quick way to get a set of data via the `batch()` method."]},{"cell_type":"code","metadata":{"id":"FkMW6OVDIHz7","colab_type":"code","colab":{}},"source":["train_eg, train_lbl = next(iter(train_data_8k.batch(1)))"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"ymTNbD_hIdrJ","colab_type":"code","colab":{}},"source":["train_eg.numpy()[0][:10]"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"-iFKOr9WIMjk","colab_type":"text"},"source":["The above method can help to get a batch of data, however, it is only available for the data with the same shape, e.g. image datasets. In the text or sequence dataset, there is a better way to get a small set of data via the `take()` method.\n","\n","Another way to take a number of data is the method `take()`."]},{"cell_type":"code","metadata":{"id":"QdvfDcxTTSou","colab_type":"code","colab":{}},"source":["for train_example, train_label in train_data_8k.take(10):\n","  print('Encoded Text: {}'.format(train_example[:10].numpy()))\n","  print('Label: {}'.format(train_label.numpy()))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"SokwjXWMUQh5","colab_type":"text"},"source":["You can decode the encoded number into a string like the operation above."]},{"cell_type":"code","metadata":{"id":"fzInS9kzUQB9","colab_type":"code","colab":{}},"source":["encoder.decode(train_example)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"WRvs2AvuYL-G","colab_type":"text"},"source":["Notice `tfds.Datasets.output_shapes()` is deprecated(), you can access the output shapes via `tf.compat.v1.data.get_output_shapes(train_data_8k)`.\n","\n","Here this model doesn't use masking, but padding. The zero-padding is used as part of the input, so the padding length may affect the output."]},{"cell_type":"code","metadata":{"id":"YZyOdDnlXApR","colab_type":"code","colab":{}},"source":["train_batches = train_data_8k.shuffle(1000).padded_batch(32, tf.compat.v1.data.get_output_shapes(train_data_8k))"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"YGesx1oLYjdb","colab_type":"code","colab":{}},"source":["test_batches = test_data_8k.padded_batch(32, tf.compat.v1.data.get_output_shapes(test_data_8k))"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"i0CU--yqYxZ7","colab_type":"code","colab":{}},"source":["for example_batch, label_batch in train_batches.take(3):\n","  print(\"Batch shape {}\".format(example_batch.shape))\n","  print(\"Label shape {}\".format(label_batch.shape))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"ujyZNWffYzFi","colab_type":"text"},"source":["Each batches will have a shape of `(batch_size, sequence_length)`. The padding is dynamic on each batch."]},{"cell_type":"markdown","metadata":{"id":"Hr-uxX5eartO","colab_type":"text"},"source":["## Building the Model\n","\n","The output shape of the embedding layer is **(batch, sequence, embedding)**.\n","\n","The GlobalAveragePooling1D() layer helps to handle input of variable length in the simplest way possible.\n","\n","The last layer is a float between 0 and 1 representing the confidence or the probability."]},{"cell_type":"code","metadata":{"id":"wNoVbIWXneZw","colab_type":"code","colab":{}},"source":["def build_model_adv(inputs):\n","  x = tf.keras.layers.Embedding(input_dim=encoder.vocab_size, output_dim=32)(inputs)\n","  x = tf.keras.layers.GlobalAveragePooling1D()(x)\n","  x = tf.keras.layers.Dense(10)(x)\n","  x = tf.keras.layers.BatchNormalization()(x)\n","  x = tf.keras.layers.Activation('relu')(x)\n","  x = tf.keras.layers.Dense(1, activation='sigmoid')(x)\n","  return x"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"qBYUqej_L66h","colab_type":"text"},"source":["The input shape of the model was [None], the input length on each batch is variable."]},{"cell_type":"code","metadata":{"id":"mcee1xukbNhe","colab_type":"code","colab":{}},"source":["inputs = tf.keras.Input(shape=[None])\n","outputs = build_model_adv(inputs=inputs)\n","model_adv = tf.keras.Model(inputs, outputs)\n","model_adv.summary()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"dOgB3_LDbz1U","colab_type":"code","colab":{}},"source":["model_adv.compile(optimizer='Adam', \n","              loss='binary_crossentropy', \n","              metrics=['accuracy'])"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"Wu4S7ITLf8aR","colab_type":"text"},"source":["## Training the Model"]},{"cell_type":"code","metadata":{"id":"UtWMJ2Unf53F","colab_type":"code","colab":{}},"source":["his = model_adv.fit(train_batches, \n","                    epochs=10, \n","                    validation_data=test_batches, \n","                    validation_steps=30)"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"A-5hyHavgu2b","colab_type":"text"},"source":["Evaluate the model."]},{"cell_type":"code","metadata":{"id":"kfqEnRn8gRBs","colab_type":"code","colab":{}},"source":["loss, accuracy = model_adv.evaluate(test_batches)\n","print(\"Loss: {}, Accuracy: {}\".format(loss, accuracy))"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"hMIQnWOrNQSG","colab_type":"text"},"source":["## View the details in the training history"]},{"cell_type":"code","metadata":{"id":"6PZqdmkhg3s9","colab_type":"code","colab":{}},"source":["his_dict = his.history"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"YoyOzCGqNsWm","colab_type":"code","colab":{}},"source":["his_dict.keys()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"oPUXdoJFNxbV","colab_type":"code","colab":{}},"source":["acc = his_dict['accuracy']\n","loss = his_dict['loss']\n","val_loss = his_dict['val_loss']\n","val_acc = his_dict['val_accuracy']"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"PdMIRVYJOD0u","colab_type":"code","colab":{}},"source":["epochs = range(1, len(acc)+1)"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"xzRiGLMKOJge","colab_type":"code","colab":{}},"source":["plt.plot(epochs, loss, 'bo', label='Training Loss') # bo: blue dot\n","plt.plot(epochs, val_loss, 'b', label=\"Validation Loss\") # b: blue\n","plt.title('Training and Validation Loss')\n","plt.xlabel('Epochs')\n","plt.ylabel('Loss')\n","plt.ylim(0, 2)\n","plt.legend()\n","plt.show()"],"execution_count":0,"outputs":[]},{"cell_type":"code","metadata":{"id":"5EahC4-COxo3","colab_type":"code","colab":{}},"source":["plt.plot(epochs, acc, 'ro', label='Training Accuracy')\n","plt.plot(epochs, val_acc, 'r', label='Validation Accuracy')\n","plt.title('Training and Validation Accuracy')\n","plt.xlabel('Epochs')\n","plt.ylabel('Accuracy')\n","plt.ylim(0, 1)\n","plt.legend(loc='lower right')\n","plt.show()"],"execution_count":0,"outputs":[]},{"cell_type":"markdown","metadata":{"id":"eCa18rOuSMlw","colab_type":"text"},"source":["You can see the overfitting issue, one of the solutions to avoid it is to stop early."]},{"cell_type":"code","metadata":{"id":"RhYscsFxPc-Q","colab_type":"code","colab":{}},"source":[""],"execution_count":0,"outputs":[]}]}