Chapter 5 - Word Embeddings.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "zKL6WYgQgMXM"
   },
   "source": [
    "# Chapter 5 Case Study\n",
    "\n",
    "We start off by a detailed look into the word2vec algorithm and examine a python implementation of the skip-gram model with negative sampling. Once the concepts underpinning word2vec are examined, we will use the Gensim package to speed up training time and investigate the translational properties of word embeddings. We will examine Glove embeddings as an alternative to word2vec. Both methods,  however, are unable to handle antonymy, polysemy and word-sense disambiguation. We will study how an embedding methods like sense2vec can better handle these issues. Lastly, we will examing document clustering by using an embeddings approach.\n",
    "\n",
    "## Software Tools and Libraries\n",
    "\n",
    "In this case study, we will be examining the inner operations of word2vec's skip-gram & negative sampling approach as well as GloVe embeddings with python. We will also leverage the popular nltk, gensim, glove_python, and spaCy libraries for our analysis. NLTK is a popular open-source toolkit for natural langauge processing and text analytics. The gensim library is an open-source toolkit for vector space modeling and topic modeling implemented in Python with Cython performance acceleration. The glove_python library (https://github.com/maciejkula/glove-python) is an efficient open-source implementation of GloVe in python. SpaCy is a fast open-source NLP library written in Python and Cython for part-of-speech tagging and named entity recognition.\n",
    "\n",
    "For our analysis, we will leverage the Open American National Corpus (http://www.anc.org/), which consists of roughly 15 million spoken and written words from a variety of sources. Specifically, we will be using the subcorpus which consists of 4531 Slate magazine articles from 1996 to 2000 (approximately 4.2 million words). \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 732,
     "status": "ok",
     "timestamp": 1536553356403,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "wldcLR05gMXN",
    "outputId": "46fd3ad1-3fe3-4244-9fd8-e39c479420bc"
   },
   "outputs": [],
   "source": [
    "# Prepping our environment\n",
    "\n",
    "# Basics\n",
    "import glob\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.manifold import TSNE\n",
    "from sklearn.feature_extraction import stop_words\n",
    "import math\n",
    "import re\n",
    "import string\n",
    "\n",
    "# NLTK\n",
    "import nltk\n",
    "from nltk import tokenize, sent_tokenize\n",
    "\n",
    "print(\"Packages imported\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "jWY6E3RYc7Cs"
   },
   "source": [
    "# 1. Exploratory Data Analysis\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1. Get Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 56045,
     "status": "ok",
     "timestamp": 1536553439295,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "yoB_h2EAhcYk",
    "outputId": "5684d5f0-14eb-4e26-f563-55a85f29f68d"
   },
   "outputs": [],
   "source": [
    "# Get slate data:  http://www.anc.org/data/oanc/contents/#slate\n",
    "\n",
    "!cd data; tar xf slate1.tgz\n",
    "!cd data; tar xf slate2.tgz\n",
    "!cd data; tar xf slate3.tgz\n",
    "!ls data/slate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DxF8oKWWdLq1"
   },
   "source": [
    "## 1.2. Load Data\n",
    "\n",
    "Next, we load the slate news articles into a pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 9330,
     "status": "ok",
     "timestamp": 1536553452366,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "A2mfyp6_gMXU",
    "outputId": "7a27f7d6-67f3-47e3-aa39-0aa4f8477fce"
   },
   "outputs": [],
   "source": [
    "#Load raw text files\n",
    "\n",
    "ndf = pd.DataFrame(columns=[\"filename\",'text'])\n",
    "\n",
    "for datafile in glob.glob( \"data/slate/*/*.txt\" ):\n",
    "    f=open(datafile,\"r\",encoding=\"utf-8\")\n",
    "    ndf = ndf.append( {'filename':datafile,'text':f.read()} ,ignore_index=True)\n",
    "    f.close()\n",
    "\n",
    "ndf.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Kz667UaNgqxl"
   },
   "source": [
    "To process the data with word2vec, we must first split each article into separate sentences and create a new dataframe to hold them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 36345,
     "status": "ok",
     "timestamp": 1536553492055,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "EjyzAmroqcRZ",
    "outputId": "3e0b4f65-ec03-4b15-f3dc-540fa7e43950"
   },
   "outputs": [],
   "source": [
    "# Convert into dataframe of sentences\n",
    "\n",
    "nltk.download('punkt')\n",
    "\n",
    "df = pd.DataFrame()\n",
    "for f in ndf['text']:\n",
    "  sentence = sent_tokenize(f)\n",
    "  df = df.append(sentence)\n",
    "df.columns = ['text']\n",
    "df.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "m_K5fXQegsti"
   },
   "source": [
    "## 1.3. Preprocessing\n",
    "\n",
    "Word2vec needs clean word tokens for **training**, so lets clean up the sentences by preprocessing to remove punctuation, hidden characters, and uppercase characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 11020,
     "status": "ok",
     "timestamp": 1536554098320,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "hva-Csg7gMXa",
    "outputId": "77700d40-2cc2-408c-f813-e18df660d0d1"
   },
   "outputs": [],
   "source": [
    "def process_text(text):\n",
    "    \n",
    "    text = re.sub(r\"\\\\n\", \" \", text)\n",
    "    text = re.sub(r\"\\n\", \" \", text)\n",
    "\n",
    "    ## Remove puncuation\n",
    "    text = text.translate(string.punctuation)\n",
    "    \n",
    "    ## Convert words to lower case and split them\n",
    "    text = text.lower().split()\n",
    "    \n",
    "    ## Remove stop words\n",
    "#    stops = stop_words.ENGLISH_STOP_WORDS\n",
    "#    text = [w for w in text if not w in stops and len(w) >= 3]\n",
    "    \n",
    "    text = \" \".join(text)\n",
    "    ## Clean the text\n",
    "    text = re.sub(r\"[^A-Za-z0-9^,!.\\/'+-=]\", \" \", text)\n",
    "    text = re.sub(r\"what's\", \"what is \", text)\n",
    "    text = re.sub(r\"\\'s\", \" \", text)\n",
    "    text = re.sub(r\"\\'ve\", \" have \", text)\n",
    "    text = re.sub(r\"n't\", \" not \", text)\n",
    "    text = re.sub(r\"i'm\", \"i am \", text)\n",
    "    text = re.sub(r\"\\'re\", \" are \", text)\n",
    "    text = re.sub(r\"\\'d\", \" would \", text)\n",
    "    text = re.sub(r\"\\'ll\", \" will \", text)\n",
    "    text = re.sub(r\",\", \" \", text)\n",
    "    text = re.sub(r\"\\.\", \" \", text)\n",
    "    text = re.sub(r\"!\", \" ! \", text)\n",
    "    text = re.sub(r\"\\/\", \" \", text)\n",
    "    text = re.sub(r\"\\^\", \" ^ \", text)\n",
    "    text = re.sub(r\"\\+\", \" + \", text)\n",
    "    text = re.sub(r\"\\-\", \" - \", text)\n",
    "    text = re.sub(r\"\\=\", \" = \", text)\n",
    "    text = re.sub(r\"'\", \" \", text)\n",
    "    text = re.sub(r\"(\\d+)(k)\", r\"\\g<1>000\", text)\n",
    "    text = re.sub(r\":\", \" : \", text)\n",
    "    text = re.sub(r\" e g \", \" eg \", text)\n",
    "    text = re.sub(r\" b g \", \" bg \", text)\n",
    "    text = re.sub(r\" u s \", \" american \", text)\n",
    "    text = re.sub(r\"\\0s\", \"0\", text)\n",
    "    text = re.sub(r\" 9 11 \", \"911\", text)\n",
    "    text = re.sub(r\"e - mail\", \"email\", text)\n",
    "    text = re.sub(r\"j k\", \"jk\", text)\n",
    "    text = re.sub(r\"\\s{2,}\", \" \", text)\n",
    "    # Stemming\n",
    "#     text = text.split()\n",
    "#     stemmer = SnowballStemmer('english')\n",
    "#     stemmed_words = [stemmer.stem(word) for word in text]\n",
    "#     text = \" \".join(stemmed_words)\n",
    "    return text\n",
    "\n",
    "# apply the above function to df['text']\n",
    "df['text'] = df['text'].map(lambda x: process_text(x))\n",
    "\n",
    "corpus = df['text'].tolist()\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9zKs7FuG_pnC"
   },
   "source": [
    "Let's calculate some basic statistics on this dataset, starting with document length and sentence length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 396
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 695,
     "status": "ok",
     "timestamp": 1536542501508,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "Y4UUeU_H8UeQ",
    "outputId": "7a5ac791-6862-4d53-d5e8-3c502daf45a9"
   },
   "outputs": [],
   "source": [
    "# document length histogram\n",
    "\n",
    "pd.DataFrame([len(x.split()) for x in ndf['text']],columns= ['document length']).hist(bins=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 396
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 914,
     "status": "ok",
     "timestamp": 1536542504689,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "beuSfefR-bpb",
    "outputId": "944ba2d3-587d-4179-f723-05a6b1999358"
   },
   "outputs": [],
   "source": [
    "pd.DataFrame([len(x.split()) for x in df['text']],columns= ['sentence length']).hist(bins=50,range=(0,100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xQHrm2dp_3KP"
   },
   "source": [
    "Let's examine word-frequency by looking at the top 1000 terms in this corpus. The top 100 terms are what we typically consider stop-words, since they are common across most sentences and do not capture much, if any, semantic meaning. As we move further down, we start to see words that play a more important role in conveying the meaning within a sentence or document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1105
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 39644,
     "status": "ok",
     "timestamp": 1536542762866,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "9XkgxgHKABbo",
    "outputId": "4300a6b2-ad26-45db-a3bf-bfc0780794b0"
   },
   "outputs": [],
   "source": [
    "words = [tokenize.word_tokenize(x) for x in corpus]\n",
    "words = [y for x in words for y in x]\n",
    "word_freq = pd.DataFrame(nltk.FreqDist(words).most_common(1000), columns=['word','frequency'])\n",
    "print(word_freq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "PlaoG8JnCuZ2"
   },
   "outputs": [],
   "source": [
    "word_freq.plot(xlim=(0,400))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2N9p1weIgMXb"
   },
   "source": [
    "# 2.  Learning Word Embeddings\n",
    "\n",
    "Our goal is to train a set of word embeddings for the corpus above. Let's build a skip-gram model with negative sampling, followed by a GloVe model. Before we train either model, let's see how many unique words exist in the preprocessed 4.86 million word corpus.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "L6hwjM2o1PPH"
   },
   "source": [
    "\n",
    "## 2.1. Create Dictionary of Vocabulary\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4164,
     "status": "ok",
     "timestamp": 1536475407260,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "IDC1lwN4gMXf",
    "outputId": "6210c66a-f13a-401b-b7b3-b138ac5e5fb9"
   },
   "outputs": [],
   "source": [
    "# Tokenize and create dictionary\n",
    "\n",
    "class VocabWord:\n",
    "    def __init__(self, word):\n",
    "        self.word = word\n",
    "        self.count = 0\n",
    "\n",
    "\n",
    "vocab_items = []\n",
    "vocab_hash = {}\n",
    "word_count = 0\n",
    "        \n",
    "for token in ['<bol>', '<eol>']:\n",
    "        vocab_hash[token] = len(vocab_items)\n",
    "        vocab_items.append(VocabWord(token))\n",
    "\n",
    "for line in df['text']:\n",
    "    tokens = line.split()\n",
    "    for token in tokens:\n",
    "        if token not in vocab_hash:\n",
    "            vocab_hash[token] = len(vocab_items)\n",
    "            vocab_items.append(VocabWord(token))\n",
    "\n",
    "        #assert vocab_items[vocab_hash[token]].word == token, 'Wrong vocab_hash index'\n",
    "        vocab_items[vocab_hash[token]].count += 1\n",
    "        word_count += 1\n",
    "\n",
    "        if word_count % 1000000 == 0:\n",
    "            print(\"\\rReading word %d\" % word_count)\n",
    "\n",
    "    # Add special tokens <bol> (beginning of line) and <eol> (end of line)\n",
    "    vocab_items[vocab_hash['<bol>']].count += 1\n",
    "    vocab_items[vocab_hash['<eol>']].count += 1\n",
    "    word_count += 2\n",
    "\n",
    "print('Total words in training file: %d' % word_count)\n",
    "print('Vocab size: %d' % len(vocab_items))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "SPkD92k5gMXv"
   },
   "source": [
    "## 2.2. Word2Vec\n",
    "\n",
    "We are now ready to train the neural network of the word2vec model. Let's define our model parameters:\n",
    "\n",
    "*  dim = dimension of the word vectors\n",
    "*  win = context window size (number of tokens)\n",
    "*  start_alpha = starting learning rate\n",
    "*  neg = number of samples for negative sampling\n",
    "*  min_count = minimum number of mentions for a word to be included in vocabulary\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "zbHevxE8gMXy"
   },
   "outputs": [],
   "source": [
    "# Initialize parameters of neural network\n",
    "\n",
    "dim = 100\n",
    "win = 10\n",
    "start_alpha = 0.05\n",
    "neg = 10\n",
    "min_count = 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n6RlCjOQiDu9"
   },
   "source": [
    "## 2.3. Handling Rare Words\n",
    "\n",
    "We want to filter out rare words that have few mentions than our min_count threshold. We will be mapping all of these words to a special out-of-vocabulary token. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 617,
     "status": "ok",
     "timestamp": 1536475408311,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "iS5lj0CWgMXh",
    "outputId": "e0b528a4-97e1-41ee-b4c3-fb973cde6e98"
   },
   "outputs": [],
   "source": [
    "# truncate dictionary and map rare words to <unk> token\n",
    "truncated = []\n",
    "truncated.append(VocabWord('<unk>'))\n",
    "unk_hash = 0\n",
    "\n",
    "count_unk = 0\n",
    "for token in vocab_items:\n",
    "    if token.count < min_count:\n",
    "        count_unk += 1\n",
    "        truncated[unk_hash].count += token.count\n",
    "    else:\n",
    "        truncated.append(token)\n",
    "\n",
    "truncated.sort(key=lambda token : token.count, reverse=True)\n",
    "\n",
    "vocab_hash = {}\n",
    "for i, token in enumerate(truncated):\n",
    "    vocab_hash[token.word] = i\n",
    "\n",
    "vocab_items = truncated\n",
    "vocab_hash = vocab_hash\n",
    "vocab_size = len(vocab_items)\n",
    "print('Unknown vocab size:', count_unk)\n",
    "print('Truncated vocab size: %d' % vocab_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2F-2JG7WiZxF"
   },
   "source": [
    "## 2.4. Negative Sampling\n",
    "\n",
    "To speed up training, let's create a negative sampling lookup table that we will use during training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "xstX1akpgMXk"
   },
   "outputs": [],
   "source": [
    "# Create table of probabilities for negative sampling\n",
    "\n",
    "power = 0.75\n",
    "norm = sum([math.pow(t.count, power) for t in vocab_items]) # Normalizing constant\n",
    "\n",
    "table_size = int(1e8) # Length of the unigram table\n",
    "table = np.zeros(table_size, dtype=np.int)\n",
    "\n",
    "p = 0 # Cumulative probability\n",
    "i = 0\n",
    "for j, unigram in enumerate(vocab_items):\n",
    "    p += float(math.pow(unigram.count, power))/norm\n",
    "    while i < table_size and float(i) / table_size < p:\n",
    "        table[i] = j\n",
    "        i += 1\n",
    "        \n",
    "def sample(table,count):\n",
    "    indices = np.random.randint(low=0, high=len(table), size=count)\n",
    "    return [table[i] for i in indices]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uaZPIf4rirUa"
   },
   "source": [
    "## 2.5. Training the Model\n",
    "\n",
    "We are now ready to train the word2vec model. The approach is to train a two-layer (syn0, syn1) neural network by iterating over the sentences in the corpus and adjusting lawyer weights to maximize the probabilities of context words given a target word (skip-gram) with negative sampling. After completion, the weights of the hidden layer syn0 are the word embeddings that we seek."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1870
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 5963379,
     "status": "ok",
     "timestamp": 1536481422883,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "m3j79pBUgMXp",
    "outputId": "7950a171-25b7-4f17-9574-ed4f7f85cf82"
   },
   "outputs": [],
   "source": [
    "# Train skip-gram with negative sampling\n",
    "import struct\n",
    "\n",
    "# Sigmoid Function\n",
    "def sigmoid(z):\n",
    "    if z > 6:\n",
    "        return 1.0\n",
    "    elif z < -6:\n",
    "        return 0.0\n",
    "    else:\n",
    "        return 1 / (1 + math.exp(-z))\n",
    "    \n",
    "# Init syn0 with uniform distribution on the interval [-0.5, 0.5]/dim\n",
    "tmp = np.random.uniform(low=-0.5/dim, high=0.5/dim, size=(vocab_size, dim))\n",
    "syn0 = np.ctypeslib.as_ctypes(tmp)\n",
    "syn0 = np.array(syn0)\n",
    "\n",
    "\n",
    "tmp = np.zeros(shape=(vocab_size, dim))\n",
    "syn1 = np.ctypeslib.as_ctypes(tmp)\n",
    "syn1 = np.array(syn1)\n",
    "    \n",
    "current_sent = 0\n",
    "truncated_vocabulary = [x.word for x in vocab_items]\n",
    "corpus = df['text'].tolist()\n",
    "\n",
    "while current_sent < df.count()[0]:\n",
    "    line = corpus[current_sent]\n",
    "    sent = [vocab_hash[token] if token in truncated_vocabulary else vocab_hash['<unk>'] \n",
    "            for token in [['<bol>'] + line.split() + ['<eol>']]]\n",
    "    for sent_pos, token in enumerate(sent):\n",
    "        \n",
    "        current_win = np.random.randint(low=1, high=win+1)\n",
    "        context_start = max(sent_pos - current_win, 0)\n",
    "        context_end = min(sent_pos + current_win + 1, len(sent))\n",
    "        context = sent[context_start:sent_pos] + sent[sent_pos+1:context_end]\n",
    "\n",
    "        for context_word in context:\n",
    "            embed = np.zeros(DIM)\n",
    "            classifiers = [(token, 1)] + [(target, 0) for target in table.sample(neg)]\n",
    "            for target, label in classifiers:\n",
    "                z = np.dot(syn0[context_word], syn1[target])\n",
    "                p = sigmoid(z)\n",
    "                g = alpha * (label - p)\n",
    "                embed += g * syn1[target] \n",
    "                syn1[target] += g * syn0[context_word] \n",
    "            syn0[context_word] += embed\n",
    "\n",
    "        word_count += 1\n",
    "    current_sent += 1\n",
    "    if current_sent % 2000 == 0:\n",
    "        print(\"\\rReading sentence %d\" % current_sent)\n",
    "\n",
    "\n",
    "embedding = dict(zip(truncated_vocabulary,syn0))\n",
    "print(\"Trained embeddings\")\n",
    "\n",
    "# Save embedding\n",
    "\n",
    "fo = open(\"word2vec\", 'w+')\n",
    "for token, vector in zip(truncated_vocabulary, syn0):\n",
    "    fo.write('%s ' % token)\n",
    "    for s in vector:\n",
    "        fo.write(('%f '% s))\n",
    "    fo.write('\\n')\n",
    "fo.close()\n",
    "\n",
    "print(syn0.shape)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "iT9pDHQiknqy"
   },
   "source": [
    "The semantic translation properties of these embeddings are noteworthy. Let's examine the cosine similarity between two similar words (man, woman) and two dissimilar words (candy, social). We would expect the similar words to exhibit higher similarity.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 618,
     "status": "ok",
     "timestamp": 1536481423540,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "J0gbRx-wDDGi",
    "outputId": "8a400709-8917-4ce4-d1aa-a9ba66880f88"
   },
   "outputs": [],
   "source": [
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "\n",
    "print(cosine_similarity([embedding['man']],[embedding['woman']]))\n",
    "print(cosine_similarity([embedding['candy']],[embedding['social']]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "YOmJILe9ojIC"
   },
   "source": [
    "## 2.6. Visualize Embedding space\n",
    "\n",
    "We can visualize the word embeddings using the T-SNE algorithm to map the embeddings to 2D space. Note that T-SNE is a dimensionality reduction technique that preserves notions of proximity within a vector space (points close together in 2D are close in proximity in higher dimensions). The figure below shows the relationships of a 300 word sample from the vocabulary.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 592
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 6121,
     "status": "ok",
     "timestamp": 1536481429792,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "tPqRWDUpDDGo",
    "outputId": "4147f052-fa08-449b-877c-a68c2e36a0bd"
   },
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "from matplotlib import pylab\n",
    "\n",
    "words = truncated_vocabulary[200:400]\n",
    "embeddings = syn0\n",
    "words_embedded = TSNE(n_components=2).fit_transform(embeddings[200:400])\n",
    "\n",
    "pylab.figure(figsize=(10, 10))\n",
    "for i, label in enumerate(words):\n",
    "  x, y = words_embedded[i, :]\n",
    "  pylab.scatter(x, y)\n",
    "  pylab.annotate(label, xy=(x, y), xytext=(5, 2), textcoords='offset points',\n",
    "                 ha='right', va='bottom')\n",
    "pylab.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "smA-v5wNLG7o"
   },
   "source": [
    "## 2.7. Using the Gensim package\n",
    "\n",
    "The python code above is useful for understanding principles, but is not the fastest to run. The original word2vec package was written in C++ to faciliate rapid training speed over multiple cores. The gensim package provides an API to the word2vec library, as well as several useful methods to examine vectors neighborhoods. Let's see how we can use gensim to train on the sample data corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "bmoPDLEtgMX0"
   },
   "outputs": [],
   "source": [
    "!pip install -q gensim==3.2.0\n",
    "\n",
    "import gensim\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "j24ysivxqrBD"
   },
   "source": [
    "Gensim expects us to provide a set of documents as a list of list of tokens. We will call the simple_preprocess() method of gensim to remove punctuation, special and uppercase characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "UQ-5KcjHDDGw"
   },
   "outputs": [],
   "source": [
    "documents = [gensim.utils.simple_preprocess(df['text'].iloc[i]) for i in range(len(df))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9gxa0Oheq8GR"
   },
   "source": [
    "With the wrapper api provided by the gensim package, training word2vec is as simple as defining a model and passing the set of training documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 107470,
     "status": "ok",
     "timestamp": 1536504281265,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "2xXRH4U6DDGz",
    "outputId": "49f9267f-9bf3-4d59-edf3-83e348cf3203"
   },
   "outputs": [],
   "source": [
    "model = gensim.models.Word2Vec(documents,\n",
    "                                size=100,\n",
    "                                window=10,\n",
    "                                min_count=2,\n",
    "                                workers=10)\n",
    "model.train(documents, total_examples=len(documents), epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "62tObRVfrNS-"
   },
   "source": [
    "## 2.8. Similarity\n",
    "\n",
    "Let's assess the quality of the learned word embeddings by examining word neighborhoods. If we look at the most similar words to \"man\" or \"book\", we find highly similar words in their neighborhoods. So far so good."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 358,
     "status": "ok",
     "timestamp": 1536506297815,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "5ddYx8kcUoOa",
    "outputId": "19e79ecb-b9ef-48f3-9767-6135672c28cf"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"man\",topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 334,
     "status": "ok",
     "timestamp": 1536506326068,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "pEGF6uyNVckS",
    "outputId": "4a864aab-e2d6-4c9f-ec03-9bd343c7ffdb"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"book\",topn=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MZ752ezJrrnl"
   },
   "source": [
    "Let's look at some polysemic words. The similar words to the word \"bass\" reflect the music definition of bass. That is, they only capture a single word sense (there are no words related to the aquatic definition of bass). Similarly, words similar to \"bank\" all reflect its financial word sense, but no seashores or riverbeds. This is one of the major shortcomings of word2vec. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 528,
     "status": "ok",
     "timestamp": 1536506335815,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "Yb0EuyP9M6go",
    "outputId": "41a5f728-51e7-4c13-86fe-c95cdb8a2155"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"bass\",topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 359,
     "status": "ok",
     "timestamp": 1536506340138,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "Co46Hkl5TNJZ",
    "outputId": "ae4b436d-4908-44b1-949b-5498e438e708"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"bank\",topn=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "nZ0HeCGSso0e"
   },
   "source": [
    "We can examine the semantic translation properties in more detail with some vector algebra. If we start with the word 'son' and subtract 'man' and add 'woman', we indeed find that 'daughter' is the closest word to the resulting sum. Similarly, if we invert the operation and start with the word 'daughter' and subtract 'woman' and add 'man', we find that 'son' is closest to the sum. Note that reciprocity is not guaranteed with word2vec."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 334,
     "status": "ok",
     "timestamp": 1536506346620,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "hC1VR5JfVHEM",
    "outputId": "9fc4c07d-1ae1-47ba-a0e5-a0fbb14bdb74"
   },
   "outputs": [],
   "source": [
    "model.wv.similar_by_vector(model.wv['son']-model.wv['man']+model.wv['woman'],topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 340,
     "status": "ok",
     "timestamp": 1536506348654,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "KfO0CU69VlFq",
    "outputId": "1da3b34c-125d-4198-d9d9-77a471fa6ffe"
   },
   "outputs": [],
   "source": [
    "model.wv.similar_by_vector(model.wv['daughter']-model.wv['woman']+model.wv['man'],topn=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "BcCw_1F-tdr2"
   },
   "source": [
    "We can also see that word2vec captures geographic similarities by taking the word 'paris', subtracting 'france' and adding 'russia'. There resulting sum is close to what we expect - 'moscow'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 438,
     "status": "ok",
     "timestamp": 1536506381949,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "ArVLsU5xX4XK",
    "outputId": "655018cb-283f-49bc-be8b-db8ca32c2c03"
   },
   "outputs": [],
   "source": [
    "model.wv.similar_by_vector(model.wv['paris']-model.wv['france']+model.wv['russia'],topn=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fk1er1qJt-WO"
   },
   "source": [
    "We have previously discussed that word embeddings generated by word2vec are unable to distinguish antonyms, as these words often share the same context words in normal usage and consequentially have learned embeddings close to each other. For instance, the most similar word to 'large' is 'small', or the most similar word to 'hard' is 'easy'. Antonymy is hard! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 471,
     "status": "ok",
     "timestamp": 1536481564014,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "Jiu4SeC6ZEUb",
    "outputId": "3f92c71d-a120-4957-cc68-c2f946684985"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"large\",topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 513,
     "status": "ok",
     "timestamp": 1536481564595,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "VQ7FAewwZ7US",
    "outputId": "b0d146be-4bbb-4c07-b830-b3188de3438b"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"hard\",topn=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "qVvab9UtvBTv"
   },
   "source": [
    "## 2.9. GloVe\n",
    "\n",
    "Whereas word2vec captures the local context of words within sentences, GloVe embeddings can additionally account for global context across the corpus. Let's take a deeper dive on how to calculate GloVe embeddings. We begin by building a vocabulary dictionary from the corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "eAWtd1AELwaf"
   },
   "outputs": [],
   "source": [
    "# Build vocab\n",
    "\n",
    "from collections import Counter\n",
    "\n",
    "vocab_count = Counter()\n",
    "\n",
    "for line in corpus:\n",
    "    tokens = line.strip().split()\n",
    "    vocab_count.update(tokens)\n",
    "\n",
    "vocab = {word: (i, freq) for i, (word, freq) in enumerate(vocab_count.items())}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "g6QUqK4U53cM"
   },
   "source": [
    "## 2.10. Co-occurence matrix\n",
    "\n",
    "Let's build the word co-occurence matrix from the corpus.  Note that word occurrences go both ways, from the main word to context, and vice versa. For smaller values of the context window, this matrix is expected to be sparse."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 408
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 319651,
     "status": "ok",
     "timestamp": 1536481886537,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "628M9WYhL0dv",
    "outputId": "928c4589-0f64-46cd-d13f-193c076a04fc"
   },
   "outputs": [],
   "source": [
    "# Build co-occurrence matrix\n",
    "\n",
    "from scipy import sparse\n",
    "\n",
    "min_count = 10\n",
    "window_size = 5\n",
    "\n",
    "vocab_size = len(vocab)\n",
    "id2word = dict((i, word) for word, (i, _) in vocab.items())\n",
    "occurrence = sparse.lil_matrix((vocab_size, vocab_size),dtype=np.float64)\n",
    "\n",
    "for i, line in enumerate(corpus):\n",
    "    tokens = line.split()\n",
    "    token_ids = [vocab[word][0] for word in tokens]\n",
    "\n",
    "    for center_i, center_id in enumerate(token_ids):\n",
    "        # Collect all word IDs in left window of center word\n",
    "        context_ids = token_ids[max(0, center_i - window_size) : center_i]\n",
    "        contexts_len = len(context_ids)\n",
    "\n",
    "        for left_i, left_id in enumerate(context_ids):\n",
    "            # Distance from center word\n",
    "            distance = contexts_len - left_i\n",
    "\n",
    "            # Weight by inverse of distance between words\n",
    "            increment = 1.0 / float(distance)\n",
    "\n",
    "            # Build co-occurrence matrix symmetrically (pretend we\n",
    "            # are calculating right contexts as well)\n",
    "            occurrence[center_id, left_id] += increment\n",
    "            occurrence[left_id, center_id] += increment\n",
    "    if i % 10000 == 0:\n",
    "        print(\"Processing sentence %d\" % i)\n",
    "\n",
    "# Create a generator for efficient calculation\n",
    "\n",
    "def occur_matrix(vocab, coccurrence, min_count):\n",
    "  for i, (row, data) in enumerate(zip(coccurrence.rows, coccurrence.data)):\n",
    "    if min_count is not None and vocab[id2word[i]][1] < min_count:\n",
    "        continue\n",
    "\n",
    "    for data_idx, j in enumerate(row):\n",
    "        if min_count is not None and vocab[id2word[j]][1] < min_count:\n",
    "            continue\n",
    "\n",
    "        yield i, j, data[data_idx]\n",
    "\n",
    "print(\"Creating co-occurrence matrix generator\")\n",
    "comatrix = occur_matrix(vocab, occurrence, min_count)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "2uoFehEQ71VC"
   },
   "source": [
    "## 2.11. GloVe training\n",
    "\n",
    "We can now train the embeddings by iterating over the documents (sentences) in the corpus.\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "gm8lvFKHNSjp"
   },
   "outputs": [],
   "source": [
    "# Train Glove Embeddings\n",
    "\n",
    "from random import shuffle\n",
    "from math import log\n",
    "import pickle\n",
    "\n",
    "iterations = 30\n",
    "dim = 100\n",
    "learning_rate = 0.05\n",
    "x_max = 100\n",
    "alpha = 0.75\n",
    "\n",
    "vocab_size = len(vocab)\n",
    "W = (np.random.rand(vocab_size * 2, dim) - 0.5) / float(dim + 1)\n",
    "biases = (np.random.rand(vocab_size * 2) - 0.5) / float(dim + 1)\n",
    "\n",
    "gradient_squared = np.ones((vocab_size * 2, dim), dtype=np.float64)\n",
    "gradient_squared_biases = np.ones(vocab_size * 2, dtype=np.float64)\n",
    "\n",
    "data = [(W[i_main], W[i_context + vocab_size],\n",
    "             biases[i_main : i_main + 1],\n",
    "             biases[i_context + vocab_size : i_context + vocab_size + 1],\n",
    "             gradient_squared[i_main], gradient_squared[i_context + vocab_size],\n",
    "             gradient_squared_biases[i_main : i_main + 1],\n",
    "             gradient_squared_biases[i_context + vocab_size\n",
    "                                     : i_context + vocab_size + 1],\n",
    "             cooccurrence)\n",
    "            for i_main, i_context, cooccurrence in comatrix]\n",
    "\n",
    "for i in range(iterations):\n",
    "    global_cost = 0\n",
    "    shuffle(data)\n",
    "    for (v_main, v_context, b_main, b_context, gradsq_W_main, gradsq_W_context,\n",
    "         gradsq_b_main, gradsq_b_context, cooccurrence) in data:\n",
    "\n",
    "        weight = (cooccurrence / x_max) ** alpha if cooccurrence < x_max else 1\n",
    "\n",
    "        cost_inner = (v_main.dot(v_context)\n",
    "                      + b_main[0] + b_context[0]\n",
    "                      - log(cooccurrence))\n",
    "        cost = weight * (cost_inner ** 2)\n",
    "        global_cost += 0.5 * cost\n",
    "\n",
    "        grad_main = weight * cost_inner * v_context\n",
    "        grad_context = weight * cost_inner * v_main\n",
    "        grad_bias_main = weight * cost_inner\n",
    "        grad_bias_context = weight * cost_inner\n",
    "\n",
    "        v_main -= (learning_rate * grad_main / np.sqrt(gradsq_W_main))\n",
    "        v_context -= (learning_rate * grad_context / np.sqrt(gradsq_W_context))\n",
    "\n",
    "        b_main -= (learning_rate * grad_bias_main / np.sqrt(gradsq_b_main))\n",
    "        b_context -= (learning_rate * grad_bias_context / np.sqrt(\n",
    "                gradsq_b_context))\n",
    "\n",
    "        gradsq_W_main += np.square(grad_main)\n",
    "        gradsq_W_context += np.square(grad_context)\n",
    "        gradsq_b_main += grad_bias_main ** 2\n",
    "        gradsq_b_context += grad_bias_context ** 2\n",
    "    print('iteration = ',i,' cost = ',global_cost)\n",
    "    \n",
    "# Save model\n",
    "\n",
    "with open(\"glove\",\"wb\") as f:\n",
    "    pickle.dump(W, f, protocol=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GkpcyAXI8kJN"
   },
   "source": [
    "The learned weight matrix consists of two sets of vectors, one if the word is in the main word position and one for the context word position. We will average them to generate the final GloVe embeddings for each word."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 2218,
     "status": "ok",
     "timestamp": 1536470812654,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "tpYpbW-wwqj6",
    "outputId": "a1924a6d-2ce7-4ecc-b5f8-114bff7e13c9"
   },
   "outputs": [],
   "source": [
    "# Merge main and context word vectors by taking average\n",
    "\n",
    "def merge_vectors(W, merge_fun=lambda m, c: np.mean([m, c], axis=0)):\n",
    "\n",
    "    vocab_size = int(len(W) / 2)\n",
    "    for i, row in enumerate(W[:vocab_size]):\n",
    "        merged = merge_fun(row, W[i + vocab_size])\n",
    "        merged /= np.linalg.norm(merged)\n",
    "        W[i, :] = merged\n",
    "\n",
    "    return W[:vocab_size]\n",
    "\n",
    "embedding = merge_vectors(W)\n",
    "len(embedding)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QBLLZ1Zv9DeZ"
   },
   "source": [
    "##2.12. GloVe Vector Similarity\n",
    "\n",
    "Let's examine the translational properties of these vectors. We define a simple function that returns the 5 most similar words to the word \"man\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 523,
     "status": "ok",
     "timestamp": 1536470954286,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "y9h3zY1COGn4",
    "outputId": "ab8a4ddb-5300-482c-92c0-7c991a25838e"
   },
   "outputs": [],
   "source": [
    "# Find most similar words\n",
    "\n",
    "def most_similar(W, vocab, id2word, word, n=5):\n",
    "    \"\"\"\n",
    "    Find the `n` words most similar to the given `word`. The provided\n",
    "    `W` must have unit vector rows, and must have merged main- and\n",
    "    context-word vectors (i.e., `len(W) == len(word2id)`).\n",
    "    Returns a list of word strings.\n",
    "    \"\"\"\n",
    "\n",
    "    assert len(W) == len(vocab)\n",
    "\n",
    "    word_id = vocab[word][0]\n",
    "\n",
    "    dists = np.dot(W, W[word_id])\n",
    "    top_ids = np.argsort(dists)[::-1][:n + 1]\n",
    "\n",
    "    return [(id2word[id],dists[id]) for id in top_ids if id != word_id][:n]\n",
    "\n",
    "for term in most_similar(embedding, vocab, id2word, 'man', 5): print(term)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "k01BvGFK9skq"
   },
   "source": [
    "Interestingly, the similarirty results fall into two categories. Whereas \"woman\" and \"girl\" have similar semantic meaning to 'man', the words \"dead\" and \"young\" do not. But these words do co-occur often, with phrases such as \"young man\" or \"dead man\". GloVe embeddings can capture both contexts. We can see this when we visualize the embeddings using T-SNE."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 592
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8424,
     "status": "ok",
     "timestamp": 1536471430016,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "86EcGLA2z4_8",
    "outputId": "bb7cdebe-ac5f-4d21-f692-f6520bcd1596"
   },
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "from matplotlib import pylab\n",
    "\n",
    "words = list(vocab.keys())[200:400]\n",
    "words_embedded = TSNE(n_components=2).fit_transform(embedding[200:400])\n",
    "\n",
    "pylab.figure(figsize=(10, 10))\n",
    "for i, label in enumerate(words):\n",
    "  x, y = words_embedded[i, :]\n",
    "  pylab.scatter(x, y)\n",
    "  pylab.annotate(label, xy=(x, y), xytext=(5, 2), textcoords='offset points',\n",
    "                 ha='right', va='bottom')\n",
    "pylab.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7tmAUjOJ-6nF"
   },
   "source": [
    "## 2.13. Using the Glove Package\n",
    "\n",
    "While useful, our python implemention is too slow to run with a large corpus. The 'glove' and 'glove_python' libraries are python packages that implement the GloVe algoritm efficiently. Let's retrain our embeddings using the 'glove_python' package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 16564,
     "status": "ok",
     "timestamp": 1536553550479,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "kCzBGy6vcZiW",
    "outputId": "a8bef9ca-6928-45d4-fae1-80f732144232"
   },
   "outputs": [],
   "source": [
    "!pip install glove_python\n",
    "\n",
    "from glove import Corpus, Glove"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "d2YlBSzNAnNW"
   },
   "source": [
    "To do so, we begin by defining a corpus of documents. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "9KkWgvd2wJSW"
   },
   "outputs": [],
   "source": [
    "corpus = Corpus()\n",
    "\n",
    "corpus.fit(documents, window=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ztbqTuarBD5M"
   },
   "source": [
    "We then learn embeddings from this corpus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 544
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 281419,
     "status": "ok",
     "timestamp": 1536547425742,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "O_m3t9Orwubn",
    "outputId": "d82bc898-0351-45b3-e7bc-07f62e4c5ea8"
   },
   "outputs": [],
   "source": [
    "glove = Glove(no_components=100, learning_rate=0.05)\n",
    "\n",
    "glove.fit(corpus.matrix, epochs=30, no_threads=4, verbose=True)\n",
    "glove.add_dictionary(corpus.dictionary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wrXur-BXBUM0"
   },
   "source": [
    "Let's asses the quality of these embeddings by examining a few words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 320,
     "status": "ok",
     "timestamp": 1536472433625,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "oopquK63xOuM",
    "outputId": "1f0feecb-b570-424a-dfca-b61bce7417eb"
   },
   "outputs": [],
   "source": [
    "glove.most_similar('man',number=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 467,
     "status": "ok",
     "timestamp": 1536474701475,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "xWfYIlLlB0uL",
    "outputId": "eb7c607f-936f-47e4-d972-5c434692a4fd"
   },
   "outputs": [],
   "source": [
    "glove.most_similar('nice',number=6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 328,
     "status": "ok",
     "timestamp": 1536475296675,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "Scgf7FK6wzp3",
    "outputId": "a6005d5f-4530-4fdb-be5d-af184540f142"
   },
   "outputs": [],
   "source": [
    "glove.most_similar('apple',number=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "D-JqH3yjBvWv"
   },
   "source": [
    "Once again, the most similar words exhibit both semantic similarity as well as high co-occurrence probability. Even with the additional context, GloVe embeddings still lack the ability to handle antonyms and word sense disambiguation. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0U_nKRMzT11H"
   },
   "source": [
    "# 3. Document Clustering\n",
    "\n",
    "The use of word embeddings provides a useful and efficient means for document clustering in comparison to traditional approaches such as LSA or LDA. The simplest approach is a bag-of-words method where a document vector is created by averaging the vectors of each of the words in the document. Let's take our Slate corpus and see what we can find with this approach.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tI5Tbms52_q7"
   },
   "source": [
    "##3.1. Document vectors\n",
    "\n",
    "We create a set of document vectors by adding the vectors of each word in the document and dividing by the total number of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 138126,
     "status": "ok",
     "timestamp": 1536553732576,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "PyMCx37IWSB1",
    "outputId": "a4d2cbac-56df-4248-8911-75149ae420f9"
   },
   "outputs": [],
   "source": [
    "documents = [gensim.utils.simple_preprocess(ndf['text'].iloc[i]) for i in range(len(ndf))]\n",
    "corpus = Corpus()\n",
    "corpus.fit(documents, window=5)\n",
    "glove = Glove(no_components=100, learning_rate=0.05)\n",
    "glove.fit(corpus.matrix, epochs=10, no_threads=4, verbose=True)\n",
    "glove.add_dictionary(corpus.dictionary)\n",
    "print(\"Glove embeddings trained.\")\n",
    "\n",
    "doc_vectors = []\n",
    "for doc in documents:\n",
    "    vec = np.zeros(glove.word_vectors[glove.dictionary['the']].shape)\n",
    "    for token in doc:\n",
    "      vec += glove.word_vectors[glove.dictionary[token]]\n",
    "    if len(doc) > 0:\n",
    "      vec = vec/len(doc)\n",
    "    doc_vectors.append(vec)\n",
    "    \n",
    "print(\"Processed documents = \",len(doc_vectors))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "GLdQHJqGeLe8"
   },
   "source": [
    "## 3.2. Cluster Analysis\n",
    "\n",
    "Let's visualize these embeddings and see if we can spot any clusters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 592
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 140175,
     "status": "ok",
     "timestamp": 1536554402026,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "VfWEtg6YZcH2",
    "outputId": "20ec88eb-9487-422c-fcac-342f01353b74"
   },
   "outputs": [],
   "source": [
    "from sklearn.manifold import TSNE\n",
    "from matplotlib import pylab\n",
    "\n",
    "docs_embedded = TSNE(n_components=2).fit_transform(doc_vectors)\n",
    "\n",
    "pylab.figure(figsize=(10, 10))\n",
    "for i, label in enumerate(docs_embedded):\n",
    "  x, y = docs_embedded[i, :]\n",
    "  pylab.scatter(x, y)\n",
    "pylab.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Ldq4R8hS5vzx"
   },
   "source": [
    "# 4. Word Sense Disambiguation\n",
    "\n",
    "Word sense disambiguation is an important task in computational linguistics. However, word2vec or GloVe embeddings map words to a single embedding vector, and therefore lack the ability to disambiguate between multiple senses of words. The sense2vec algorithm is an improved approach that can deal with polysemy or antonymy through supervised disambiguation. Moreover, sense2vec is computationally inexpensive and can be implemented as a preprocessing task prior to training a word2vec or GloVe model. To see this, let's apply the sense2vec algoritm to our corpus by leveraging the spaCy library to generate part-of-speech labels that will serve as our supervised disambiguation labels.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "LRsSoRoB3Sfd"
   },
   "source": [
    "## 4.1. Load spaCy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 564
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 4687,
     "status": "ok",
     "timestamp": 1536543605055,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "vV813p6g5OML",
    "outputId": "6b76d0ac-fd3f-4959-dee7-3347994e13cb"
   },
   "outputs": [],
   "source": [
    "!pip install spacy\n",
    "!python -m spacy download en\n",
    "\n",
    "import spacy\n",
    "nlp = spacy.load('en',disable=['parser', 'ner'])\n",
    "\n",
    "print('Model loaded')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "brerOdtO90MA"
   },
   "source": [
    "## 4.2. Supervised Disambiguation Annotations\n",
    "\n",
    "Let's process the sentences in our corpus using the spaCy NLP annotations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kNAsDAKU8vDE"
   },
   "outputs": [],
   "source": [
    "corpus = df['text'].tolist()\n",
    "print(\"Number of docs = \",len(corpus))\n",
    "\n",
    "docs = []\n",
    "count = 0\n",
    "for item in corpus:\n",
    "  docs.append(nlp(item))\n",
    "  count += 1\n",
    "  if count % 10000 == 0:\n",
    "    print(\"Processed document #\",count)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ShVj97wlJSPW"
   },
   "source": [
    "## 4.3. POS Label Annotation\n",
    "\n",
    "We will create a separate corpus where each word is augmented by its part-of-speech label. For instance, the word 'he' is mapped to 'he_PRON'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 71
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 8449,
     "status": "ok",
     "timestamp": 1536545829272,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "Bw23nc0FJQhS",
    "outputId": "b76b32cd-01a0-4ee0-f031-e773a21e8d7e"
   },
   "outputs": [],
   "source": [
    "sense_corpus = [[x.text+\"_\"+x.pos_ for x in y] for y in docs]\n",
    "print(corpus[0])\n",
    "print(' '.join(sense_corpus[0]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VP6Jgy6XLfGD"
   },
   "source": [
    "## 4.4. Train using word2vec\n",
    "\n",
    "With the new preprocessed corpus, we can proceed with training word2vec."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 115553,
     "status": "ok",
     "timestamp": 1536545950976,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "o5yymeNFC-c-",
    "outputId": "ce1a1973-e332-4ca5-ace7-b9d6a74fea61"
   },
   "outputs": [],
   "source": [
    "model = gensim.models.Word2Vec(sense_corpus,\n",
    "                                size=100,\n",
    "                                window=10,\n",
    "                                min_count=2,\n",
    "                                workers=10)\n",
    "model.train(sense_corpus, total_examples=len(sense_corpus), epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ubOCeDtNTinb"
   },
   "source": [
    "## 4.5. Evaluate Word Sense Disambiguation\n",
    "\n",
    "We can use this trained model to look at how words like 'run' or 'lie' can be disambiguated based on their part-of-speech."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 442,
     "status": "ok",
     "timestamp": 1536546251011,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "jlzdf6n_R8T_",
    "outputId": "b3edeb13-05c4-4774-8163-34bcab625d2e"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"run_NOUN\",topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 365,
     "status": "ok",
     "timestamp": 1536546204945,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "6ut4rjXGSc_i",
    "outputId": "67a25c9e-5ac8-41c7-c48b-fa36cda199c4"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"run_VERB\",topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 421,
     "status": "ok",
     "timestamp": 1536546404531,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "D9zEhF1-I9Sn",
    "outputId": "fe5b3eeb-8d59-401c-9a21-13ecd63bafd4"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"lie_NOUN\",topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 406,
     "status": "ok",
     "timestamp": 1536546409931,
     "user": {
      "displayName": "John Liu",
      "photoUrl": "//lh3.googleusercontent.com/-1qlmGEbXNp4/AAAAAAAAAAI/AAAAAAAAAI8/lRMkenerVqw/s50-c-k-no/photo.jpg",
      "userId": "111145929698637850127"
     },
     "user_tz": 300
    },
    "id": "AXYo0DzPR2DJ",
    "outputId": "382c12a6-d0a8-4ff7-9662-8b4b0fbc5d4c"
   },
   "outputs": [],
   "source": [
    "model.wv.most_similar(\"lie_VERB\",topn=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "kKwx2dEVSfqq"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Chapter_5_Case_Study_part1.ipynb",
   "provenance": [],
   "version": "0.3.2"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}