11. Recurrent Neural Networks - Python.ipynb

{
  "cells": [
    {
      "metadata": {
        "collapsed": true
      },
      "cell_type": "markdown",
      "source": "Exercise 11 - Recurrent Neural Networks\n========\n\nA recurrent neural network (RNN) is a class of neural network that excels when your data can be treated as a sequence - such as text, music, speech recognition, connected handwriting, or data over a time period. \n\nRNN's can analyse or predict a word based on the previous words in a sentence - they allow a connection between previous information and current information.\n\nThis exercise looks at implementing a LSTM RNN to generate new characters after learning from a large sample of text. LSTMs are a special type of RNN which dramatically improves the model’s ability to connect previous data to current data where there is a long gap.\n\nWe will train an RNN model using a novel written by H. G. Wells - The Time Machine."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Step 1\n------\n\nLet's start by loading our libraries and text file. This might take a few minutes.\n\n#### Run the cell below to import the necessary libraries."
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "%%capture\n# Run this!\nfrom keras.models import load_model\nfrom keras.models import Sequential\nfrom keras.layers import Dense, Activation, LSTM\nfrom keras.callbacks import LambdaCallback, ModelCheckpoint\nimport numpy as np\nimport random, sys, io, string",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "#### Replace the `<addFileName>` with `The Time Machine`"
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "###\n# REPLACE THE <addFileName> BELOW WITH The Time Machine\n###\ntext = io.open('Data/<addFileName>.txt', encoding = 'UTF-8').read()\n###\n\n# Let's have a look at some of the text\nprint(text[0:198])\n\n# This cuts out punctuation and make all the characters lower case\ntext = text.lower().translate(str.maketrans(\"\", \"\", string.punctuation))\n\n# Character index dictionary\ncharset = sorted(list(set(text)))\nindex_from_char = dict((c, i) for i, c in enumerate(charset))\nchar_from_index = dict((i, c) for i, c in enumerate(charset))\n\nprint('text length: %s characters' %len(text))\nprint('unique characters: %s' %len(charset))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Expected output:  \n```The Time Traveller (for so it will be convenient to speak of him) was expounding a recondite matter to us. His pale grey eyes shone and twinkled, and his usually pale face was flushed and animated.\ntext length: 174201 characters\nunique characters: 39```\n\nStep 2\n-----\n\nNext we'll divide the text into sequences of 40 characters.\n\nThen for each sequence we'll make a training set - the following character will be the correct output for the test set.\n\n### In the cell below replace:\n#### 1. `<sequenceLength>` with `40`\n#### 2. `<step>` with `4`\n#### and then __run the code__. "
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "###\n# REPLACE <sequenceLength> WITH 40 AND <step> WITH 4\n###\nsequence_length = <sequenceLength>\nstep = <step>\n###\n\nsequences = []\ntarget_chars = []\nfor i in range(0, len(text) - sequence_length, step):\n    sequences.append([text[i: i + sequence_length]])\n    target_chars.append(text[i + sequence_length])\nprint('number of training sequences:', len(sequences))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Expected output:\n`number of training sequences: 43541`\n\n#### Replace `<addSequences>` with `sequences` and run the code."
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "# One-hot vectorise\n\nX = np.zeros((len(sequences), sequence_length, len(charset)), dtype=np.bool)\ny = np.zeros((len(sequences), len(charset)), dtype=np.bool)\n\n###\n# REPLACE THE <addSequences> BELOW WITH sequences\n###\nfor n, sequence in enumerate(<addSequences>):\n###\n    for m, character in enumerate(list(sequence[0])):\n        X[n, m, index_from_char[character]] = 1\n    y[n, index_from_char[target_chars[n]]] = 1",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Step 3\n------\n\nLet's build our model, using a single LSTM layer of 128 units. We'll keep the model simple for now, so that training does not take too long.\n\n### In the cell below replace:\n#### 1. `<addLSTM>` with `LSTM`\n#### 2. `<addLayerSize>` with `128`\n#### 3. `<addSoftmaxFunction>` with `'softmax`\n#### and then __run the code__."
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "model = Sequential()\n\n###\n# REPLACE THE <addLSTM> BELOW WITH LSTM (use uppercase) AND <addLayerSize> WITH 128\n###\nmodel.add(<addLSTM>(<addLayerSize>, input_shape = (X.shape[1], X.shape[2])))\n###\n\n###\n# REPLACE THE <addSoftmaxFunction> with 'softmax' (INCLUDING THE QUOTES)\n###\nmodel.add(Dense(y.shape[1], activation = <addSoftMaxFunction>))\n###\n\nmodel.compile(loss = 'categorical_crossentropy', optimizer = 'Adam')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The code below generates text at the end of an epoch (one training cycle). This allows us to see how the model is performing as it trains. If you're making a large neural network with a long training time it's useful to check in on the model as see if the text generating is legible as it trains, as overtraining may occur and the output of the model turn to nonsense.\n\nThe code below will also save a model if it is the best performing model, so we can use it later.\n\n#### Run the code below, but don't change it"
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Run this, but do not edit.\n# It helps generate the text and save the model epochs.\n\n# Generate new text\ndef on_epoch_end(epoch, _):\n    diversity = 0.5\n    print('\\n### Generating text with diversity %0.2f' %(diversity))\n\n    start = random.randint(0, len(text) - sequence_length - 1)\n    seed = text[start: start + sequence_length]\n    print('### Generating with seed: \"%s\"' %seed[:40])\n\n    output = seed[:40].lower().translate(str.maketrans(\"\", \"\", string.punctuation))\n    print(output, end = '')\n\n    for i in range(500):\n        x_pred = np.zeros((1, sequence_length, len(charset)))\n        for t, char in enumerate(output):\n            x_pred[0, t, index_from_char[char]] = 1.\n\n        predictions = model.predict(x_pred, verbose=0)[0]\n        exp_preds = np.exp(np.log(np.asarray(predictions).astype('float64')) / diversity)\n        next_index = np.argmax(np.random.multinomial(1, exp_preds / np.sum(exp_preds), 1))\n        next_char = char_from_index[next_index]\n\n        output = output[1:] + next_char\n\n        print(next_char, end = '')\n    print()\nprint_callback = LambdaCallback(on_epoch_end=on_epoch_end)\n\n# Save the model\ncheckpoint = ModelCheckpoint('Models/model-epoch-{epoch:02d}.hdf5', \n                             monitor = 'loss', verbose = 1, save_best_only = True, mode = 'min')",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The code below will start the model to train. This may take a long time. Feel free to stop the training with the `square stop button` to the right of the `Run button` in the toolbar.\n\nLater in the exercise, we will load a pretrained model.\n\n### In the cell below replace:\n#### 1. `<addPrintCallback>` with `print_callback`\n#### 2. `<addCheckpoint>` with `checkpoint`\n#### and then __run the code__."
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "###\n# REPLACE <addPrintCallback> WITH print_callback AND <addCheckpoint> WITH checkpoint\n###\nmodel.fit(X, y, batch_size = 128, epochs = 3, callbacks = [<addPrintCallback>, <addCheckpoint>])\n###",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The output won't appear to be very good. But then, this dataset is small, and we have trained it only for a short time using a rather small RNN. How might it look if we upscaled things?\n\nStep 5\n------\n\nWe could improve our model by:\n* Having a larger training set.\n* Increasing the number of LSTM units.\n* Training it for longer\n* Experimenting with difference activation functions, optimization functions etc\n\nTraining this would still take far too long on most computers to see good results - so we've trained a model already for you.\n\nThis model uses a different dataset - a few of the King Arthur tales pasted together. The model used:\n* sequences of 50 characters\n* Two LSTM layers (512 units each)\n* A dropout of 0.5 after each LSTM layer\n* Only 30 epochs (we'd recomend 100-200)\n\nLet's try importing this model that has already been trained.\n\n#### Replace `<addLoadModel>` with `load_model` and run the code."
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "from keras.models import load_model\nprint(\"loading model... \", end = '')\n\n###\n# REPLACE <addLoadModel> BELOW WITH load_model\n###\nmodel = <addLoadModel>('Models/arthur-model-epoch-30.hdf5')\n###\nmodel.compile(loss = 'categorical_crossentropy', optimizer = 'Adam')\n###\n\nprint(\"model loaded\")",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Step 6\n-------\n\nNow let's use this model to generate some new text!\n\n#### Replace `<addFilePath>` with `'Data/Arthur tales.txt'`"
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "###\n# REPLACE <addFilePath> BELOW WITH 'Data/Arthur tales.txt' (INCLUDING THE QUOTATION MARKS)\n###\ntext = io.open(<addFilePath>, encoding='UTF-8').read()\n###\n\n# Cut out punctuation and make lower case\ntext = text.lower().translate(str.maketrans(\"\", \"\", string.punctuation))\n\n# Character index dictionary\ncharset = sorted(list(set(text)))\nindex_from_char = dict((c, i) for i, c in enumerate(charset))\nchar_from_index = dict((i, c) for i, c in enumerate(charset))\n\nprint('text length: %s characters' %len(text))\nprint('unique characters: %s' %len(charset))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "### In the cell below replace:\n#### 1. `<sequenceLength>` with `50`\n#### 2. `<writeSentence>` with a sentence of your own, at least 50 characters long.\n#### 3. `<numCharsToGenerate>` with the number of characters you want to generate (choose a large number, like 1500)\n#### and then __run the code__."
    },
    {
      "metadata": {
        "trusted": false
      },
      "cell_type": "code",
      "source": "# Generate text\n\ndiversity = 0.5\nprint('\\n### Generating text with diversity %0.2f' %(diversity))\n\n###\n# REPLACE <sequenceLength> BELOW WITH 50\n###\nsequence_length = <sequenceLength>\n###\n\n# Next we'll make a starting point for our text generator\n\n###\n# REPLACE <writeSentence> WITH A SENTENCE OF AT LEAST 50 CHARACTERS\n###\nseed = \"<writeSentence>\"\n###\n\nseed = seed.lower().translate(str.maketrans(\"\", \"\", string.punctuation))\n\n###\n# OR, ALTERNATIVELY, UNCOMMENT THE FOLLOWING TWO LINES AND GRAB A RANDOM STRING FROM THE TEXT FILE\n###\n\n#start = random.randint(0, len(text) - sequence_length - 1)\n#seed = text[start: start + sequence_length]\n\n###\n\nprint('### Generating with seed: \"%s\"' %seed[:40])\n\noutput = seed[:sequence_length].lower().translate(str.maketrans(\"\", \"\", string.punctuation))\nprint(output, end = '')\n\n###\n# REPLACE THE <numCharsToGenerate> BELOW WITH THE NUMBER OF CHARACTERS WE WISH TO GENERATE, e.g. 1500\n###\nfor i in range(<numCharsToGenerate>):\n###\n    x_pred = np.zeros((1, sequence_length, len(charset)))\n    for t, char in enumerate(output):\n        x_pred[0, t, index_from_char[char]] = 1.\n\n    predictions = model.predict(x_pred, verbose=0)[0]\n    exp_preds = np.exp(np.log(np.asarray(predictions).astype('float64')) / diversity)\n    next_index = np.argmax(np.random.multinomial(1, exp_preds / np.sum(exp_preds), 1))\n    next_char = char_from_index[next_index]\n\n    output = output[1:] + next_char\n\n    print(next_char, end = '')\nprint()",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "How does it look? Does it seem intelligible?\n\nConclusion\n--------\n\nWe have trained an RNN that learns to predict characters based on a text sequence. We have trained a lightweight model from scratch, as well as imported a pre-trained model and generated new text from that."
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "python36",
      "display_name": "Python 3.6",
      "language": "python"
    },
    "language_info": {
      "mimetype": "text/x-python",
      "nbconvert_exporter": "python",
      "name": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6",
      "file_extension": ".py",
      "codemirror_mode": {
        "version": 3,
        "name": "ipython"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}