11. Recurrent Neural Networks - R.ipynb

{
  "cells": [
    {
      "metadata": {
        "collapsed": true
      },
      "cell_type": "markdown",
      "source": "Recurrent Neural Networks\n===\n\nA recurrent neural network (RNN) is a class of neural network that excels when your data can be treated as a sequence - such as text, music, speech recognition, connected handwriting, or data over a time period. \n\nRNNs can analyse or predict a word based on the previous words in a sentence - they allow a connection between previous information and current information.\n\nThis exercise looks at implementing a LSTM RNN to generate new characters after learning from a large sample of text. LSTMs are a special type of RNN which dramatically improves the model’s ability to connect previous data to current data where there is a long gap.\n\nWe will train an RNN model using a novel written by H. G. Wells - The Time Machine."
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Step 1\n------\n\nLet's start by loading our libraries looking at our text file. This might take a few minutes."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this!\n\nsuppressMessages(install.packages(\"keras\"))\nsuppressMessages(install.packages(\"tokenizers\"))\nsuppressMessages(install.packages(\"stringr\"))\nsuppressMessages(library(keras))\nsuppressMessages(library(readr))\nsuppressMessages(library(stringr))\nsuppressMessages(library(purrr))\nsuppressMessages(library(tokenizers))\nsuppressMessages(install_keras())",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "path <- file.path(\"Data/time-edit.txt\")\n# Let's have a look at the text\nread_lines(path)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Expected output:  \n```The Time Traveller (for so it will be convenient to speak of him) was expounding a recondite matter to us. His pale grey eyes shone and twinkled, and his usually pale face was flushed and animated.\ntext length: 174201 characters\nunique characters: 39```\n\nStep 2\n-----\n\nNext we'll divide the text into sequences of 35 characters.\n\nThen for each sequence we'll make a training set - the following character will be the correct output for the test set.\n\n### In the cell below replace:\n#### 1. `<textSequenceLength>` with `35`\n#### 2. `<pathToDataset>` with `path`\n#### then __run the code__."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "###\n# REPLACE <textSequenceLength> WITH 35\n###\nmaxlen <- <textSequenceLength>\n###\n\n# This makes all the characters lower case, and separates the individual characters from whole words.\n\n###\n# REPLACE <pathToDataset> WITH path\n###\ntext <- read_lines(<pathToDataset>) %>%\n###\n  str_to_lower() %>%\n  str_c(collapse = \"\\n\") %>%\n  tokenize_characters(strip_non_alphanum = FALSE, simplify = TRUE)\n\nprint(sprintf(\"Total length: %d\", length(text)))\n\nchars <- text %>%\n  unique() %>%\n  sort()\n\nprint(sprintf(\"Total chars: %d\", length(chars)))",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Expected output:  \n`\"Total length: 174666\"`  \n`\"Total chars: 29\"`\n\n#### Replace the 3 `<maximumLength>`'s with `maxlen`"
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "###\n# REPLACE ALL THE <maximumLength>'s WITH maxlen\n###\ndataset <- map(\n  seq(1, length(text) - <maximumLength> - 1, by = 6), \n  ~list(sentence = text[.x:(.x + <maximumLength> - 1)], next_char = text[.x + <maximumLength>])\n  )\n###\n\ndataset <- transpose(dataset)\n\nx <- array(0, dim = c(length(dataset$sentence), maxlen, length(chars)))\ny <- array(0, dim = c(length(dataset$sentence), length(chars)))\n\nfor(i in 1:length(dataset$sentence)){\n  \n  x[i,,] <- sapply(chars, function(x){\n    as.integer(x == dataset$sentence[[i]])\n  })\n  \n  y[i,] <- as.integer(chars == dataset$next_char[[i]])\n  \n}",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Step 3\n------\n\nLet's build our model, using a single LSTM layer of 64 units. We'll keep the model simple for now, so that training does not take too long.\n\n#### Replace the `<layerSize>` with 64, and run the cell."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "model <- keras_model_sequential()\n###\n# REPLACE <layerSize> WITH 64\n###\nmodel %>%\n  layer_lstm(<layerSize>, input_shape = c(maxlen, length(chars))) %>%\n###\n  layer_dense(length(chars)) %>%\n  layer_activation(\"softmax\")\n\nmodel %>% compile(\n  loss = \"categorical_crossentropy\", \n  optimizer = \"Adam\"\n)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "We'll just get a few helper functions ready, run the cell below to prepare them."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# Run this cell!\n\nsample_mod <- function(preds, temperature = 1){\n  preds <- log(preds)/temperature\n  exp_preds <- exp(preds)\n  preds <- exp_preds/sum(exp(preds))\n  \n  rmultinom(1, 1, preds) %>% \n    as.integer() %>%\n    which.max()\n}\n\non_epoch_end <- function(epoch, logs) {\n  \n  cat(sprintf(\"epoch: %02d ---------------\\n\\n\", epoch))\n    \n  diversity <- 0.5\n  generated <- \"\"\n    \n  cat(sprintf(\"diversity: %f ---------------\\n\\n\", diversity))\n    \n  start_index <- sample(1:(length(text) - maxlen), size = 1)\n  sentence <- text[start_index:(start_index + maxlen - 1)]\n    \n    for(i in 1:400){\n      \n      x <- sapply(chars, function(x){\n        as.integer(x == sentence)\n      })\n      x <- array_reshape(x, c(1, dim(x)))\n      \n      preds <- predict(model, x)\n      next_index <- sample_mod(preds, diversity)\n      next_char <- chars[next_index]\n      \n      generated <- str_c(generated, next_char, collapse = \"\")\n      sentence <- c(sentence[-1], next_char)\n      \n    }\n    \n    cat(generated)\n    cat(\"\\n\\n\")\n    \n  \n}",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "Ready to go. The next cell will train the model.\n\nTraining RNN's on low compute takes a long time. We'll only build a small one for now. If you want to leave this model training for longer change the number of epochs to a larger number.\n\n#### Replace the `<epochNumber>` with 3 and run the cell."
    },
    {
      "metadata": {
        "trusted": true
      },
      "cell_type": "code",
      "source": "# This will take a little while...\nprint_callback <- callback_lambda(on_epoch_end = on_epoch_end)\n\nhistory <- model %>% fit(\n  x, y,\n  batch_size = 1,\n###\n# REPLACE <epochNumber> WITH 3\n###\n  epochs = <epochNumber>,\n###\n  callbacks = print_callback\n)",
      "execution_count": null,
      "outputs": []
    },
    {
      "metadata": {},
      "cell_type": "markdown",
      "source": "The output won't appear to be very good. But then, this dataset is small, and we have trained it only for a short time using a rather small RNN. Feel free to increase the number of epochs and leave it training for a long time if you want to see better results.\n\nWe could improve our model by:\n* Having a larger training set.\n* Increasing the number of LSTM units.\n* Training it for longer\n* Experimenting with difference activation functions, optimization functions etc\n\n\nConclusion\n--------\n\nWe have trained an RNN that learns to predict characters based on a text sequence. We have trained a lightweight model from scratch."
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "r",
      "display_name": "R",
      "language": "R"
    },
    "language_info": {
      "mimetype": "text/x-r-source",
      "name": "R",
      "pygments_lexer": "r",
      "version": "3.4.1",
      "file_extension": ".r",
      "codemirror_mode": "r"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}