embeddings_intro.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# First steps with word embeddings\n",
    "Big Data and Automated Content Analysis\n",
    "\n",
    "Damian Trilling\n",
    "\n",
    "\n",
    "This notebook shows you some first steps of how to work with word embeddings in Gensim. \n",
    "\n",
    "\n",
    "**NB  Some of these things may take a lot of memory and/or computing power. **"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training word embeddings\n",
    "Training word embedings with gensim is very simple (see example code below - even though you wildo l need to specift some extra options). However, you need a massive dataset for that (think of millions of documents). We're therefore not going to do this in class.\n",
    "\n",
    "```\n",
    "model = gensim.models.Word2Vec()\n",
    "model.build_vocab(sentences)\n",
    "model.train(sentences)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using pre-trained models\n",
    "We can use pre-trained embeddings instead. You can either download them yourself and then read them from a file (which you probably want to do if you want, for instance, use our Dutch embeddings that we talked about). \n",
    "But we can also use some embeddings that come with gensim and that gensim downloads for us."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gensim.downloader as api\n",
    "\n",
    "model = api.load(\"glove-wiki-gigaword-100\")\n",
    "\n",
    "# alternative that is 16 times as big:\n",
    "# model = api.load(\"word2vec-google-news-300\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['fasttext-wiki-news-subwords-300', 'conceptnet-numberbatch-17-06-300', 'word2vec-ruscorpora-300', 'word2vec-google-news-300', 'glove-wiki-gigaword-50', 'glove-wiki-gigaword-100', 'glove-wiki-gigaword-200', 'glove-wiki-gigaword-300', 'glove-twitter-25', 'glove-twitter-50', 'glove-twitter-100', 'glove-twitter-200', '__testing_word2vec-matrix-synopsis'])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "api.info()['models'].keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# model = api.load(\"word2vec-google-news-300\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Word similarities\n",
    "Let's play around with word similarieties."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('dog', 0.8798074722290039),\n",
       " ('rabbit', 0.7424426674842834),\n",
       " ('cats', 0.7323004007339478),\n",
       " ('monkey', 0.7288709878921509),\n",
       " ('pet', 0.7190139889717102),\n",
       " ('dogs', 0.7163872718811035),\n",
       " ('mouse', 0.6915250420570374),\n",
       " ('puppy', 0.6800068020820618),\n",
       " ('rat', 0.6641027331352234),\n",
       " ('spider', 0.6501135230064392)]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar('cat')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A cat is almost the same as a dog.\n",
      "A dog is almost the same as a cat.\n",
      "A horse is almost the same as a horses.\n",
      "A goldfish is almost the same as a crackers.\n",
      "A lion is almost the same as a dragon.\n"
     ]
    }
   ],
   "source": [
    "animals = ['cat', 'dog', 'horse', 'goldfish', 'lion']\n",
    "for animal in animals:\n",
    "    print(\"A {} is almost the same as a {}.\".format(animal, model.most_similar(animal)[0][0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['woman']"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.closer_than('man','boy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.20851284265518188\n",
      "0.1676505208015442\n"
     ]
    }
   ],
   "source": [
    "print(model.distance('man','boy'))\n",
    "print(model.distance('man','woman'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, as we discussed in the lecture, we can literally calculate with the embeddings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('queen', 0.7698541283607483),\n",
       " ('monarch', 0.6843380928039551),\n",
       " ('throne', 0.6755735874176025),\n",
       " ('daughter', 0.6594556570053101),\n",
       " ('princess', 0.6520534753799438),\n",
       " ('prince', 0.6517034769058228),\n",
       " ('elizabeth', 0.6464517712593079),\n",
       " ('mother', 0.6311717629432678),\n",
       " ('emperor', 0.6106470823287964),\n",
       " ('wife', 0.6098655462265015)]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar(positive=['woman', 'king'], negative=['man'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('king', 0.8551837205886841),\n",
       " ('queen', 0.783441424369812),\n",
       " ('monarch', 0.6933802366256714),\n",
       " ('throne', 0.6833109855651855),\n",
       " ('daughter', 0.680908203125),\n",
       " ('prince', 0.6713142395019531),\n",
       " ('princess', 0.664408266544342),\n",
       " ('mother', 0.6579325199127197),\n",
       " ('elizabeth', 0.6563301086425781),\n",
       " ('father', 0.6392419338226318)]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# We can do the same by hand, but would need to do some manual cleanup afterwards\n",
    "# (e.g., removing 'king' itself from the results)\n",
    "model.similar_by_vector(model.get_vector('king') - \n",
    "                        model.get_vector('man') + \n",
    "                        model.get_vector('woman') )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.32307 , -0.87616 ,  0.21977 ,  0.25268 ,  0.22976 ,  0.7388  ,\n",
       "       -0.37954 , -0.35307 , -0.84369 , -1.1113  , -0.30266 ,  0.33178 ,\n",
       "       -0.25113 ,  0.30448 , -0.077491, -0.89815 ,  0.092496, -1.1407  ,\n",
       "       -0.58324 ,  0.66869 , -0.23122 , -0.95855 ,  0.28262 , -0.078848,\n",
       "        0.75315 ,  0.26584 ,  0.3422  , -0.33949 ,  0.95608 ,  0.065641,\n",
       "        0.45747 ,  0.39835 ,  0.57965 ,  0.39267 , -0.21851 ,  0.58795 ,\n",
       "       -0.55999 ,  0.63368 , -0.043983, -0.68731 , -0.37841 ,  0.38026 ,\n",
       "        0.61641 , -0.88269 , -0.12346 , -0.37928 , -0.38318 ,  0.23868 ,\n",
       "        0.6685  , -0.43321 , -0.11065 ,  0.081723,  1.1569  ,  0.78958 ,\n",
       "       -0.21223 , -2.3211  , -0.67806 ,  0.44561 ,  0.65707 ,  0.1045  ,\n",
       "        0.46217 ,  0.19912 ,  0.25802 ,  0.057194,  0.53443 , -0.43133 ,\n",
       "       -0.34311 ,  0.59789 , -0.58417 ,  0.068995,  0.23944 , -0.85181 ,\n",
       "        0.30379 , -0.34177 , -0.25746 , -0.031101, -0.16285 ,  0.45169 ,\n",
       "       -0.91627 ,  0.64521 ,  0.73281 , -0.22752 ,  0.30226 ,  0.044801,\n",
       "       -0.83741 ,  0.55006 , -0.52506 , -1.7357  ,  0.4751  , -0.70487 ,\n",
       "        0.056939, -0.7132  ,  0.089623,  0.41394 , -1.3363  , -0.61915 ,\n",
       "       -0.33089 , -0.52881 ,  0.16483 , -0.98878 ], dtype=float32)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.get_vector('king')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TRY YOURSELF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('iguanas', 0.6652647852897644),\n",
       " ('tortoises', 0.6609669923782349),\n",
       " ('rattlesnakes', 0.6575520038604736),\n",
       " ('crocodiles', 0.6308841109275818),\n",
       " ('snakes', 0.6242752075195312),\n",
       " ('toads', 0.6229562163352966),\n",
       " ('alligators', 0.6116349697113037),\n",
       " ('lizards', 0.6112721562385559),\n",
       " ('turtles', 0.582252025604248),\n",
       " ('salamanders', 0.5808340311050415)]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar('pythons')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('monty', 0.6886237263679504),\n",
       " ('php', 0.586538553237915),\n",
       " ('perl', 0.5784407258033752),\n",
       " ('cleese', 0.5446675419807434),\n",
       " ('flipper', 0.511298418045044),\n",
       " ('ruby', 0.5066928267478943),\n",
       " ('spamalot', 0.505638837814331),\n",
       " ('javascript', 0.5030568838119507),\n",
       " ('reticulated', 0.4983375668525696),\n",
       " ('monkey', 0.49764132499694824)]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.most_similar('python')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using word embeddings in supervised machine learning \n",
    "\n",
    "We need to `sudo pip3 install embeddingvectorizer` first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from glob import glob \n",
    "from collections import defaultdict\n",
    "import numpy as np\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "from sklearn.linear_model import SGDClassifier\n",
    "from sklearn import metrics\n",
    "from sklearn.pipeline import Pipeline\n",
    "import embeddingvectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "reviews=[]\n",
    "test=[]\n",
    "datadir = '/home/damian/Downloads/aclImdb'\n",
    "for sent in ['pos', 'neg']:\n",
    "    for file in glob (f\"{datadir}/train/{sent}/*.txt\"):\n",
    "        with open(file) as fi:\n",
    "            reviews.append((fi.read(),sent))\n",
    "\n",
    "    for file in glob (f\"{datadir}/test/{sent}/*.txt\"):\n",
    "        with open(file) as fi:\n",
    "            test.append((fi.read(),sent))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, let's look at our old model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision:\n",
      "0.8764847650197969\n",
      "Recall:\n",
      "0.81464\n"
     ]
    }
   ],
   "source": [
    "mypipe = Pipeline([('vectorizer', TfidfVectorizer()),\n",
    "                    ('svm', \n",
    "                     SGDClassifier(loss='hinge', penalty='l2', tol=1e-4, alpha=1e-6, max_iter=1000, random_state=42))])\n",
    "\n",
    "# Generate BOW representation of word counts\n",
    "mypipe.fit([e[0] for e in reviews], [e[1] for e in reviews])\n",
    "predictions = mypipe.predict([e[0] for e in test])\n",
    "\n",
    "print('Precision:')\n",
    "print(metrics.precision_score([e[1] for e in test],predictions,pos_label='pos', labels = ['pos','neg']))\n",
    "print('Recall:')\n",
    "print(metrics.recall_score([e[1] for e in test],predictions,pos_label='pos', labels = ['pos','neg']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then, let's compare with the new one.\n",
    "First, we need to convert our word embedding model to the so-called word2vec format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `syn0` (Attribute will be removed in 4.0.0, use self.vectors instead).\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "model = api.load(\"glove-wiki-gigaword-100\")\n",
    "w2vmodel = dict(zip(bigmodel.index2word, model.syn0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision:\n",
      "0.5051736125311849\n",
      "Recall:\n",
      "0.98816\n"
     ]
    }
   ],
   "source": [
    "mypipe = Pipeline([('vectorizer', embeddingvectorizer.EmbeddingTfidfVectorizer(w2vmodel, operator='mean')),\n",
    "                    ('svm', \n",
    "                     SGDClassifier(loss='hinge', penalty='l2', tol=1e-4, alpha=1e-6, max_iter=1000, random_state=42))])\n",
    "\n",
    "# Generate BOW representation of word counts\n",
    "mypipe.fit([e[0] for e in reviews], [e[1] for e in reviews])\n",
    "predictions = mypipe.predict([e[0] for e in test])\n",
    "\n",
    "print('Precision:')\n",
    "print(metrics.precision_score([e[1] for e in test],predictions,pos_label='pos', labels = ['neg','pos']))\n",
    "print('Recall:')\n",
    "print(metrics.recall_score([e[1] for e in test],predictions,pos_label='pos', labels = ['neg','pos']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, these embeddings seem to be crap. Let's use the new ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:2: DeprecationWarning: Call to deprecated `syn0` (Attribute will be removed in 4.0.0, use self.vectors instead).\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "bigmodel = api.load(\"word2vec-google-news-300\")\n",
    "w2vmodel = dict(zip(bigmodel.index2word, bigmodel.syn0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Precision:\n",
      "0.7773083475298126\n",
      "Recall:\n",
      "0.91256\n"
     ]
    }
   ],
   "source": [
    "mypipe = Pipeline([('vectorizer', embeddingvectorizer.EmbeddingTfidfVectorizer(w2vmodel, operator='mean')),\n",
    "                    ('svm', \n",
    "                     SGDClassifier(loss='hinge', penalty='l2', tol=1e-4, alpha=1e-6, max_iter=1000, random_state=42))])\n",
    "\n",
    "# Generate BOW representation of word counts\n",
    "mypipe.fit([e[0] for e in reviews], [e[1] for e in reviews])\n",
    "predictions = mypipe.predict([e[0] for e in test])\n",
    "\n",
    "print('Precision:')\n",
    "print(metrics.precision_score([e[1] for e in test],predictions,pos_label='pos', labels = ['neg','pos']))\n",
    "print('Recall:')\n",
    "print(metrics.recall_score([e[1] for e in test],predictions,pos_label='pos', labels = ['neg','pos']))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Soft cosine similarity\n",
    "\n",
    "Finally, let's explore soft cosine similarity by re-using our model `model` from above, and our movie reviews `reviews`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:5: DeprecationWarning: Call to deprecated `wv` (Attribute will be removed in 4.0.0, use self instead).\n",
      "  \"\"\"\n"
     ]
    }
   ],
   "source": [
    "from gensim.similarities import SoftCosineSimilarity, SparseTermSimilarityMatrix\n",
    "from gensim.corpora import Dictionary\n",
    "from gensim.models import WordEmbeddingSimilarityIndex\n",
    "\n",
    "termsim_index = WordEmbeddingSimilarityIndex(model.wv)\n",
    "documents = [e[0].lower().split() for e in reviews[:100]]\n",
    "\n",
    "id2word = Dictionary(documents)\n",
    "bow_corpus = [id2word.doc2bow(document) for document in documents]\n",
    "similarity_matrix = SparseTermSimilarityMatrix(termsim_index, id2word)  # construct similarity matrix\n",
    "docsim_index = SoftCosineSimilarity(bow_corpus, similarity_matrix, num_best=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = '''Pulp Fiction may be the single best film ever made, and quite appropriately it is by one of the most \n",
    "creative directors of all time, Quentin Tarantino. This movie is amazing from the beginning definition of pulp to\n",
    "the end credits and boasts one of the best casts ever assembled with the likes of Bruce Willis, Samuel L. Jackson, \n",
    "John Travolta, Uma Thurman, Harvey Keitel, Tim Roth and Christopher Walken. The dialog is surprisingly humorous for\n",
    "this type of film, and I think that's what has made it so successful. Wrongfully denied the many Oscars it was \n",
    "nominated for, Pulp Fiction is by far the best film of the 90s and no Tarantino film has surpassed the quality of\n",
    "this movie (although Kill Bill came close). As far as I'm concerned this is the top film of all-time and definitely \n",
    "deserves a watch if you haven't seen it.\n",
    "'''.lower().split()\n",
    "sims = docsim_index[id2word.doc2bow(query)]  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(20, 0.9642696380615234),\n",
       " (37, 0.9639226198196411),\n",
       " (41, 0.9627991318702698),\n",
       " (1, 0.9624872803688049),\n",
       " (31, 0.9612997770309448),\n",
       " (50, 0.9604517221450806),\n",
       " (85, 0.96035236120224),\n",
       " (13, 0.9603282809257507),\n",
       " (17, 0.9570714235305786),\n",
       " (71, 0.956860363483429)]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sims"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"i was blown away by the re-imagined battlestar galactica, a show that always kept me guessing and brought me to tears on more than one occasion. a hardened sci-fi fan, i like to think i can pick out the good stuff from the bs, and this was good stuff.<br /><br />as such, when i first heard about the prospect of a prequel series some months ago i got a sick feeling in my gut. i was afraid that the formula that made battlestar so successful would be reused in caprica, which wouldn't work at all. bsg's story, of a mournful ragged band of survivors, trapped aboard decaying star ships and guided by prophetic vision and a sequence of pseudo-miracles, was perfectly complimented by extraordinary music and a better cast of actors.<br /><br />caprica feels different. where bsg takes place after the fall of a great civilization, caprica portrays that civilization in it's cold and decadent heyday. the overall vibe i got from caprica was similar to that of minority report, minus excessive and counterproductive theatricality. in true bsg form, caprica has in it's first few hours of programming already tackled the issues of religious freedom, racism, the morality of playing god and the nature of the human soul.<br /><br />the casting for caprica is also excellent. each character is unique and deep, from the obsessive and distant scientist-turned-entrepreneur, to his troubled and willful daughter, each actor and actress throws themselves into their respective roles.<br /><br />music, which was used so powerfully in bsg, also plays a significant role in caprica. battlestar's powerful rolling drums and mournful duduks served it's themes very well. caprica uses a more orchestral sound, which gives the show it's own feeling quite distinct from either of it's predecessors. <br /><br />the new caprica is definitely it's own show, pulling from the battlestar franchise only as much as it needs. i look forward to the full series.\""
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\" \".join(documents[50])"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}