experiments.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Science in Culture\n",
    "This research project seeks to analyze the perception of science in culture. Techniques from NLP such as word embeddings (word2vec) and sentiment analysis are used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reference Code\n",
    "1. https://github.com/RaRe-Technologies/gensim/blob/develop/docs/notebooks/word2vec.ipynb\n",
    "2. https://github.com/RaRe-Technologies/gensim/blob/ba1ce894a5192fc493a865c535202695bb3c0424/docs/notebooks/Word2Vec_FastText_Comparison.ipynb\n",
    "\n",
    "## Other references\n",
    "1.   Cámara, M., & A., J. (2012). Political dimensions of scientific culture: Highlights from the Ibero-American survey on the social perception of science and scientific culture. Public Understanding of Science, 21(3), 369–384. https://doi.org/10.1177/0963662510373871\n",
    "2. Jones, M. (2014). Cultural Characters and Climate Change: How Heroes Shape Our Perception of Climate Science. Social Science Quarterly, 95(1), 1-39.\n",
    "3. Ruest, Nick, 2017, \"#climatemarch tweets April 19-May 3, 2017\", https://doi.org/10.5683/SP/KZZVZW, Scholars Portal Dataverse, V1\n",
    "4. Ruest, Nick, 2017, \"#MarchForScience tweets April 12-26, 2017\", https://doi.org/10.5683/SP/7BC9V1, Scholars Portal Dataverse, V1\n",
    "5. http://www.nltk.org/nltk_data/ id: brown; size: 3314357; author: W. N. Francis and H. Kucera; copyright: ; license: May be used for non-commercial purposes.\n",
    "6. Salehan, Kim, & Lee. (2018). Are there any relationships between technology and cultural values? A country-level trend study of the association between information communication technology and cultural values. Information & Management, 55(6), 725-745.\n",
    "7. Vishwanath, A., & Chen, H. (2008). Personal communication technologies as an extension of the self: A cross‐cultural comparison of people's associations with technology and their symbolic proximity with others. Journal of the American Society for Information Science and Technology, 59(11), 1761-1775.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk\n",
    "from gensim.models import Word2Vec\n",
    "from gensim.models.word2vec import Text8Corpus\n",
    "import numpy as np\n",
    "from smart_open import smart_open\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training the models\n",
    "The data for this project is not included in this repository due to restrictions on Twitter data. The two datasets used in this project are the widely available Brown corpus and Twitter data relating to #ClimateMarch and #MarchForScience hashtags. Roughly two million tweets were hydrated using twarc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Only train if you don't want to use the pretained models\n",
    "params = {\n",
    "    'alpha': 0.05,\n",
    "    'size': 100,\n",
    "    'window': 5,\n",
    "    'iter': 5,\n",
    "    'min_count': 5,\n",
    "    'sample': 1e-4,\n",
    "    'sg': 1,\n",
    "    'hs': 0,\n",
    "    'negative': 5\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "brown_model = Word2Vec(Text8Corpus('data/brown_corp.txt'), **params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "climate_model = Word2Vec(Text8Corpus('data/cleaned/climate_tweets_cleaned.txt'), **params)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfs_model = Word2Vec(Text8Corpus('data/cleaned/MarchForScience_tweets_cleaned.txt'), **params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Saving and loading the models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# File paths -> required by saving/loading methods\n",
    "brown_path = \"models/brown_model.bin\"\n",
    "brown_vec = \"models/vectors/brown_vec.kv\"\n",
    "brown_name = \"models/vectors/brown\"\n",
    "\n",
    "climate_path = \"models/climate_model.bin\"\n",
    "climate_vec = \"models/vectors/climate_vec.kv\"\n",
    "climate_name = \"models/vectors/climate\"\n",
    "\n",
    "mfs_path = \"models/mfs_model.bin\"\n",
    "mfs_vec = \"models/vectors/mfs_vec.kv\"\n",
    "mfs_name = \"models/vectors/mfs\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Saving models\n",
    "brown_model.save(brown_path)\n",
    "climate_model.save(climate_path)\n",
    "mfs_model.save(mfs_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 309,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Edited from gensim/scripts\n",
    "def word2vec2tensor(word2vec_model_path, tensor_filename, binary=False):\n",
    "    \"\"\"Convert file in Word2Vec format and writes two files 2D tensor TSV file.\n",
    "    File \"tensor_filename\"_tensor.tsv contains word-vectors, \"tensor_filename\"_metadata.tsv contains words.\n",
    "    Parameters\n",
    "    ----------\n",
    "    word2vec_model_path : str\n",
    "        Path to file in Word2Vec format.\n",
    "    tensor_filename : str\n",
    "        Prefix for output files.\n",
    "    binary : bool, optional\n",
    "        True if input file in binary format.\n",
    "    \"\"\"\n",
    "    model = gensim.models.KeyedVectors.load(word2vec_model_path, mmap='r')\n",
    "    outfiletsv = tensor_filename + '_tensor.tsv'\n",
    "    outfiletsvmeta = tensor_filename + '_metadata.tsv'\n",
    "\n",
    "    with smart_open(outfiletsv, 'wb') as file_vector, smart_open(outfiletsvmeta, 'wb') as file_metadata:\n",
    "        for word in model.index2word:\n",
    "            file_metadata.write(gensim.utils.to_utf8(word) + gensim.utils.to_utf8('\\n'))\n",
    "            vector_row = '\\t'.join(str(x) for x in model[word])\n",
    "            file_vector.write(gensim.utils.to_utf8(vector_row) + gensim.utils.to_utf8('\\n'))\n",
    "\n",
    "    print(\"2D tensor file saved to %s\", outfiletsv)\n",
    "    print(\"Tensor metadata file saved to %s\", outfiletsvmeta)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 311,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2D tensor file saved to %s models/vectors/brown_tensor.tsv\n",
      "Tensor metadata file saved to %s models/vectors/brown_metadata.tsv\n",
      "2D tensor file saved to %s models/vectors/climate_tensor.tsv\n",
      "Tensor metadata file saved to %s models/vectors/climate_metadata.tsv\n",
      "2D tensor file saved to %s models/vectors/mfs_tensor.tsv\n",
      "Tensor metadata file saved to %s models/vectors/mfs_metadata.tsv\n"
     ]
    }
   ],
   "source": [
    "# Saving vectors\n",
    "brown_model.wv.save(brown_vec)\n",
    "word2vec2tensor(brown_vec, brown_name)\n",
    "\n",
    "climate_model.wv.save(climate_vec)\n",
    "word2vec2tensor(climate_vec, climate_name)\n",
    "\n",
    "mfs_model.wv.save(mfs_vec)\n",
    "word2vec2tensor(mfs_vec, mfs_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loading models\n",
    "brown_model = Word2Vec.load(brown_path)\n",
    "climate_model = Word2Vec.load(climate_path)\n",
    "mfs_model = Word2Vec.load(mfs_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Evaluating the models\n",
    "Let's do some basic tests of cosine similarity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('dwell', 0.9033710956573486),\n",
       " ('humanity', 0.8990538716316223),\n",
       " ('divine', 0.8985900282859802),\n",
       " ('obedience', 0.8958655595779419),\n",
       " ('virtue', 0.8896442651748657),\n",
       " ('invention', 0.885993242263794),\n",
       " ('doctrine', 0.8842350244522095),\n",
       " ('judgments', 0.8829445838928223),\n",
       " ('Utopian', 0.8806315064430237),\n",
       " ('profound', 0.8802723288536072)]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brown_model.wv.most_similar(positive=['science', 'fear'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('Libs', 0.7004995346069336),\n",
       " ('oceans', 0.6837355494499207),\n",
       " ('might', 0.6310517191886902),\n",
       " ('Hiding', 0.6045321226119995),\n",
       " ('lab.', 0.5913923978805542),\n",
       " ('peacefully', 0.5868384838104248),\n",
       " ('backing', 0.5676073431968689),\n",
       " ('Muslims', 0.5611160397529602),\n",
       " ('NY', 0.5599805116653442),\n",
       " ('magically', 0.5589848756790161)]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "climate_model.wv.most_similar(positive=['science', 'fear'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('MC…', 0.6706839799880981),\n",
       " ('enterprise', 0.6391789317131042),\n",
       " ('ignorance…', 0.6318367123603821),\n",
       " ('triumph', 0.5998080968856812),\n",
       " ('open,', 0.5971928834915161),\n",
       " ('stay!', 0.5955246090888977),\n",
       " ('facets', 0.5903943777084351),\n",
       " ('protesting,', 0.5866920948028564),\n",
       " ('represses', 0.5861929655075073),\n",
       " ('less\"', 0.5806410312652588)]"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mfs_model.wv.most_similar(positive=['science', 'fear'], topn=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('science', 0.0006998557),\n",
       " ('philosophy', 0.0006726175),\n",
       " ('fear', 0.00059489015),\n",
       " ('mind', 0.0005511252),\n",
       " ('religion', 0.0005273468),\n",
       " ('pure', 0.00047090003),\n",
       " ('importance', 0.00046973577),\n",
       " ('feelings', 0.00046481914),\n",
       " ('poems', 0.00046039498),\n",
       " ('poetic', 0.0004552021)]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brown_model.predict_output_word(['fear', 'science'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('science', 0.014418488),\n",
       " ('rise.', 0.0102302795),\n",
       " ('might', 0.006852584),\n",
       " ('oceans', 0.0046408046),\n",
       " ('Muslims', 0.0039983448),\n",
       " ('declaring', 0.0037104057),\n",
       " (\"We'll\", 0.0035391354),\n",
       " ('fear', 0.0023272592),\n",
       " ('But', 0.0021787095),\n",
       " ('inaugurati…', 0.0019829252)]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "climate_model.predict_output_word(['fear', 'science'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('fear', 0.022886122),\n",
       " ('ignorance…', 0.002145881),\n",
       " ('less\"', 0.0015034123),\n",
       " ('Saturday', 0.0010085657),\n",
       " ('benefit', 0.0010045078),\n",
       " ('truth.', 0.00093975145),\n",
       " ('ignorance,', 0.00078472577),\n",
       " ('society', 0.0007797721),\n",
       " ('role', 0.0007566012),\n",
       " ('knowledge', 0.00072946213)]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mfs_model.predict_output_word(['science', 'fear'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "Let's analyze the models to see what we can discern about the differences in culture between these three models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualizing word vectors is done with Embedding Projector, a tool from the team at TensorFlow. Gephi is also used to generate network visualizations, and finally Scattertext is used for visualizing sentiment. The relevant figures can be found in the images folder.\n",
    "\n",
    "Here is the link for the embedding projection of climate data: https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/ndalton12/nlp-english-project/master/json/climate_config.json\n",
    "\n",
    "MFS data: https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/ndalton12/nlp-english-project/master/json/mfs_config.json\n",
    "\n",
    "Brown data: https://projector.tensorflow.org/?config=https://raw.githubusercontent.com/ndalton12/nlp-english-project/master/json/brown_config.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### KMeans Clustering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/niall/.local/lib/python3.6/site-packages/ipykernel_launcher.py:4: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).\n",
      "  after removing the cwd from sys.path.\n",
      "/home/niall/.local/lib/python3.6/site-packages/ipykernel_launcher.py:5: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).\n",
      "  \"\"\"\n",
      "/home/niall/.local/lib/python3.6/site-packages/ipykernel_launcher.py:6: DeprecationWarning: Call to deprecated `__getitem__` (Method will be removed in 4.0.0, use self.wv.__getitem__() instead).\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "from sklearn import cluster\n",
    "\n",
    "X_b = brown_model[brown_model.wv.vocab]\n",
    "X_c = climate_model[climate_model.wv.vocab]\n",
    "X_m = mfs_model[mfs_model.wv.vocab]\n",
    "NUM_CLUSTERS = 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "kmeans_b = cluster.KMeans(n_clusters=NUM_CLUSTERS)\n",
    "kmeans_c = cluster.KMeans(n_clusters=NUM_CLUSTERS)\n",
    "kmeans_m = cluster.KMeans(n_clusters=NUM_CLUSTERS)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "KMeans(algorithm='auto', copy_x=True, init='k-means++', max_iter=300,\n",
       "    n_clusters=100, n_init=10, n_jobs=None, precompute_distances='auto',\n",
       "    random_state=None, tol=0.0001, verbose=0)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "kmeans_b.fit(X_b)\n",
    "kmeans_c.fit(X_c)\n",
    "kmeans_m.fit(X_m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels_b = kmeans_b.labels_\n",
    "labels_c = kmeans_c.labels_\n",
    "labels_m = kmeans_m.labels_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# From Douglas Duhaime\n",
    "# https://douglasduhaime.com/posts/clustering-semantic-vectors.html\n",
    "\n",
    "class autovivify_list(dict):\n",
    "  '''A pickleable version of collections.defaultdict'''\n",
    "  def __missing__(self, key):\n",
    "    '''Given a missing key, set initial value to an empty list'''\n",
    "    value = self[key] = []\n",
    "    return value\n",
    "\n",
    "  def __add__(self, x):\n",
    "    '''Override addition for numeric types when self is empty'''\n",
    "    if not self and isinstance(x, Number):\n",
    "      return x\n",
    "    raise ValueError\n",
    "\n",
    "  def __sub__(self, x):\n",
    "    '''Also provide subtraction method'''\n",
    "    if not self and isinstance(x, Number):\n",
    "      return -1 * x\n",
    "    raise ValueError\n",
    "    \n",
    "def find_word_clusters(labels_array, cluster_labels):\n",
    "  cluster_to_words = autovivify_list()\n",
    "  for c, i in enumerate(cluster_labels):\n",
    "    cluster_to_words[ i ].append( labels_array[c] )\n",
    "  return cluster_to_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "clusters_b = find_word_clusters(list(brown_model.wv.vocab.keys()),labels_b);\n",
    "clusters_c = find_word_clusters(list(climate_model.wv.vocab.keys()),labels_c);\n",
    "clusters_m = find_word_clusters(list(mfs_model.wv.vocab.keys()),labels_m);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "brown:  ['purchasing', 'departments', 'personnel', 'provide', 'funds', 'services', 'assistance', 'program', 'management', 'medical']\n",
      "climate:  ['leave', 'behind', 'left', 'where', 'tons', 'cover', 'pile', 'garbage', 'remember', 'signs.']\n",
      "mfs:  ['Together', 'Side', 'FACT:', 'exonerated', 'post-conviction', 'DNA.', 'stop?', 'court!', 'soon,', 'builders.']\n",
      "\n",
      "brown:  ['many', 'other', 'among', 'servants', 'were', 'friendly', 'begin', 'most', 'While', 'sessions']\n",
      "climate:  ['ALREADY', 'US.', \"Solar's\", 'putting', 'trending', 'top', 'Why?', 'topic', 'Miami', 'vs']\n",
      "mfs:  [\"don't\", \"That's\", 'were', 'just', 'trying', 'know', 'disposing', 'shit', 'him', 'if']\n",
      "\n",
      "brown:  ['produced', 'number', 'items', ':', 'employed', '71', 'according', 'each', 'names', 'test']\n",
      "climate:  ['you', 'your', 'gonna', 'Which', 'How', 'or', 'Are', 'much', 'D.C.?', 'during']\n",
      "mfs:  ['United', 'States', 'even', 'advancements,', 'live.\"', 'Lawrence', 'Krauss,', 'theoretical', 'physicist', \"They're\"]\n",
      "\n",
      "brown:  ['Bar', 'Berry', 'resignation', 'Marvin', 'Jones', 'Schwartz', 'Paradise', 'La', 'Collins', 'Chapman']\n",
      "climate:  ['clean', 'water', 'again.', 'water.', 'food', 'dairy', 'assault', 'land.', 'ceremony!', 'requires']\n",
      "mfs:  ['my', 'poster', 'dog', 'speak.', 'Mae', 'My', 'dinosaurs', 'Bannon', 'representing', 'Beaker']\n",
      "\n",
      "brown:  ['often', 'have', 'well', 'best', 'so', 'all', 'they', 'very', 'worth', 'ones']\n",
      "climate:  ['can', 'When', 'stake', 'remain', 'Stand', 'do', 'woken', 'Up!', \"won't,\", 'it?']\n",
      "mfs:  ['air', 'lives', 'development', 'protecting', 'planet', 'create', 'join…', \"people's\", 'planet,', 'clean']\n",
      "\n",
      "brown:  [',', 'was', 'a', 'When', 'soon', 'missing', 'meager', 'fare', 'kept', \"boy's\"]\n",
      "climate:  ['AM', 'Center', 'Dept', 'website:', 'East', 'tonight', 'PM', 'du', 'Mont-Blanc,', 'Mont-Blanc.']\n",
      "mfs:  ['House', 'ignoring', 'Bill', 'Nye', 'world!’:', 'Nye’s', 'barn', 'burner', 'occupant', 'White']\n",
      "\n",
      "brown:  ['the', 'and', 'picking', 'pointed', 'pockets', 'laid', 'beating', 'glow', 'lip', 'filled']\n",
      "climate:  ['land', 'species', 'poles', 'ocean', 'Carbon', 'animal', 'ice', 'extinction', 'due', 'agriculture']\n",
      "mfs:  ['at', 'recorded', 'George', 'closed', 'booth', 'Bowlin', 'Woods', 'quad', 'of…', 'w']\n",
      "\n",
      "brown:  ['experienced', 'mental', 'strong', 'race', 'enthusiasm', 'become', 'violence', 'kind', 'learning', 'serious']\n",
      "climate:  ['legislation', 'stark', 'reminder', 'has', 'ask', 'launched', 'all-out', 'promises', 'millions', 'scientist']\n",
      "mfs:  ['to', 'are', 'is', 'will', 'up', 'in', 'this', 'the', 'and', 'on']\n",
      "\n",
      "brown:  ['investigate', 'Police', 'Grady', 'Hartsfield', 'filed', 'petition', '1913', 'Ala.', 'Pelham', '1937']\n",
      "climate:  ['but', 'cows', 'produces', 'New', 'partners', 'Oh,', 'ride', 'York', 'sold', 'send']\n",
      "mfs:  ['most', 'recent', 'Resolution,', 'But', 'protest', 'these', 'celebration', 'ever.', 'Most', 'perfect']\n",
      "\n",
      "brown:  ['should', 'take', 'future', 'will', 'continue', 'protect', 'work', 'make', 'offer', 'seek']\n",
      "climate:  ['scientists', 'Now', 'rest', 'world.', 'into', 'it’s', 'run', 'helped', 'there’s', 'come']\n",
      "mfs:  ['Water', 'Public', 'Protests', 'Repudiation', 'By', 'firing', 'Silent', 'Diseases', 'Life', 'Nuclear']\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(10):\n",
    "    print(\"brown: \", clusters_b[i][0:10])\n",
    "    print(\"climate: \", clusters_c[i][0:10])\n",
    "    print(\"mfs: \",clusters_m[i][0:10])\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictive text\n",
    "Using ULMFiT transfer learning model; need more compute/time to do this part properly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai import *\n",
    "from fastai.text import * "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = 'data/cleaned/'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_climate = TextLMDataBunch.from_csv(path, 'climate_csv.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_brown = TextLMDataBunch.from_csv(path, 'brown_csv.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 313,
   "metadata": {},
   "outputs": [],
   "source": [
    "date_mfs = TextLMDataBunch.from_csv(path, 'mfs_csv.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = language_model_learner(data_climate, pretrained_model=URLs.WT103, drop_mult=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total time: 3:22:57\n",
      "epoch  train_loss  valid_loss  accuracy\n",
      "1      1.991627    2.713907    0.520528  (3:22:57)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 1e-2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 266,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total time: 00:00\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'Technology is not interested in danger ,'"
      ]
     },
     "execution_count": 266,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "learn.predict(\"Technology is\", n_words=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save_encoder('climate_enc')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Similarity querying (top_n = 20)\n",
    "I am going to chose some words here that I think will be interesting to look at and graph in Gephi. Obviously, this method is biased as to what I think are interesting words, but so much is the point of any research in the social sciences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "TOP_N = 20 # this value is more or less arbitrary, just picking a small number so the visualization isn't too crowded\n",
    "\n",
    "words = ['science', 'fear', 'climate', 'liberty', 'justice', 'culture', 'money', 'environment', 'freedom', 'people', 'change', \n",
    "         'scientist', 'politics', 'hero', 'story', 'truth', 'lies', 'good', 'bad']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "top_dict_b = {}\n",
    "top_dict_c = {}\n",
    "top_dict_m = {}\n",
    "\n",
    "for word in words:\n",
    "    lst_b = [x[0] for x in brown_model.wv.most_similar(positive=[word], topn=TOP_N)]\n",
    "    lst_c = [x[0] for x in climate_model.wv.most_similar(positive=[word], topn=TOP_N)]\n",
    "    lst_m = [x[0] for x in mfs_model.wv.most_similar(positive=[word], topn=TOP_N)]\n",
    "    top_dict_b[word] = lst_b\n",
    "    top_dict_c[word] = lst_c\n",
    "    top_dict_m[word] = lst_m"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Graphing with networkx and Gephi\n",
    "Using the queries above, we can create a graph and visualize it. View images/graphs to see the visualizations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "import networkx as nx\n",
    "import matplotlib as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {},
   "outputs": [],
   "source": [
    "g_b = nx.Graph(top_dict_b)\n",
    "g_c = nx.Graph(top_dict_c)\n",
    "g_m = nx.Graph(top_dict_m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {},
   "outputs": [],
   "source": [
    "nx.write_gexf(g_b, \"graphs/brown.gexf\")\n",
    "nx.write_gexf(g_c, \"graphs/climate.gexf\")\n",
    "nx.write_gexf(g_m, \"graphs/mfs.gexf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "VADAR tool:\n",
    "\n",
    "Hutto, C.J. & Gilbert, E.E. (2014). VADER: A Parsimonious Rule-based Model for \n",
    "Sentiment Analysis of Social Media Text. Eighth International Conference on \n",
    "Weblogs and Social Media (ICWSM-14). Ann Arbor, MI, June 2014.\n",
    "'''\n",
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "sid = SentimentIntensityAnalyzer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [],
   "source": [
    "brown_file = pd.read_csv('data/cleaned/brown_csv.csv')\n",
    "climate_file = pd.read_csv('data/cleaned/climate_csv.csv')\n",
    "mfs_file = pd.read_csv('data/cleaned/mfs_csv.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Fulton County Grand Jury said Friday an investigation of Atlanta's recent primary election produced `` no evidence '' that any irregularities took place \n",
      "nan\n",
      "compound: 0.2023, \n",
      "neg: 0.081, \n",
      "neu: 0.809, \n",
      "pos: 0.11, \n",
      " The jury further said in term-end presentments that the City Executive Committee , which had over-all charge of the election , `` deserves the praise and thanks of the City of Atlanta '' for the manner in which the election was conducted \n",
      "nan\n",
      "compound: 0.7579, \n",
      "neg: 0.0, \n",
      "neu: 0.854, \n",
      "pos: 0.146, \n",
      " The September-October term jury had been charged by Fulton Superior Court Judge Durwood Pye to investigate reports of possible `` irregularities '' in the hard-fought primary which was won by Mayor-nominate Ivan Allen Jr\n",
      "nan\n",
      "compound: 0.7506, \n",
      "neg: 0.045, \n",
      "neu: 0.775, \n",
      "pos: 0.18, \n",
      " `` Only a relative handful of such reports was received '' , the jury said , `` considering the widespread interest in the election , the number of voters and the size of this city '' \n",
      "nan\n",
      "compound: 0.5106, \n",
      "neg: 0.0, \n",
      "neu: 0.875, \n",
      "pos: 0.125, \n",
      " The jury said it did find that many of Georgia's registration and election laws `` are outmoded or inadequate and often ambiguous '' \n",
      "nan\n",
      "compound: -0.4019, \n",
      "neg: 0.109, \n",
      "neu: 0.891, \n",
      "pos: 0.0, \n",
      " It recommended that Fulton legislators act `` to have these laws studied and revised to the end of modernizing and improving them '' \n",
      "nan\n",
      "compound: 0.5574, \n",
      "neg: 0.0, \n",
      "neu: 0.82, \n",
      "pos: 0.18, \n",
      " The grand jury commented on a number of other topics , among them the Atlanta and Fulton County purchasing departments which it said `` are well operated and follow generally accepted practices which inure to the best interest of both governments '' \n",
      "nan\n",
      "compound: 0.9287, \n",
      "neg: 0.0, \n",
      "neu: 0.684, \n",
      "pos: 0.316, \n",
      " Merger proposed However , the jury said it believes `` these two offices should be combined to achieve greater efficiency and reduce the cost of administration '' \n",
      "nan\n",
      "compound: 0.6124, \n",
      "neg: 0.0, \n",
      "neu: 0.828, \n",
      "pos: 0.172, \n",
      " The City Purchasing Department , the jury said , `` is lacking in experienced clerical personnel as a result of city personnel policies '' \n",
      "nan\n",
      "compound: 0.0, \n",
      "neg: 0.0, \n",
      "neu: 1.0, \n",
      "pos: 0.0, \n",
      " It urged that the city `` take steps to remedy '' this problem \n",
      "nan\n",
      "compound: -0.481, \n",
      "neg: 0.207, \n",
      "neu: 0.793, \n",
      "pos: 0.0, \n",
      " Implementation of Georgia's automobile title law was also recommended by the outgoing jury \n",
      "nan\n",
      "compound: 0.4588, \n",
      "neg: 0.0, \n",
      "neu: 0.733, \n",
      "pos: 0.267, \n",
      " It urged that the next Legislature `` provide enabling funds and re-set the effective date so that an orderly implementation of the law may be effected '' \n",
      "nan\n",
      "compound: 0.4767, \n",
      "neg: 0.0, \n",
      "neu: 0.893, \n",
      "pos: 0.107, \n"
     ]
    }
   ],
   "source": [
    "for i, line in enumerate(brown_file['text']):\n",
    "    print(line)\n",
    "    ss = sid.polarity_scores(line)\n",
    "    \n",
    "    for k in sorted(ss):\n",
    "        print('{0}: {1}, '.format(k, ss[k]))\n",
    "    \n",
    "    if i > 10:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Making a judgement call here that anything with pos/neg sentiment > 0.1 is significant, else label it neutral. This judgement is based on the fact that neutral sentiment tends to dominate VADER's classifications."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "5000\n",
      "10000\n",
      "15000\n",
      "20000\n",
      "25000\n",
      "30000\n",
      "35000\n",
      "40000\n",
      "45000\n",
      "50000\n",
      "done brown\n",
      "0\n",
      "10000\n",
      "20000\n",
      "30000\n",
      "40000\n",
      "50000\n",
      "60000\n",
      "70000\n",
      "80000\n",
      "90000\n",
      "100000\n",
      "110000\n",
      "120000\n",
      "130000\n",
      "140000\n",
      "150000\n",
      "160000\n",
      "done climate\n",
      "0\n",
      "10000\n",
      "20000\n",
      "30000\n",
      "40000\n",
      "50000\n",
      "60000\n",
      "70000\n",
      "80000\n",
      "90000\n",
      "100000\n",
      "110000\n",
      "120000\n",
      "130000\n",
      "140000\n",
      "150000\n",
      "160000\n",
      "170000\n",
      "180000\n",
      "190000\n",
      "200000\n",
      "210000\n",
      "220000\n",
      "done mfs\n"
     ]
    }
   ],
   "source": [
    "for i, line in enumerate(brown_file['text']):\n",
    "    ss = sid.polarity_scores(line)\n",
    "        \n",
    "    if ss['neg'] > 0.1 or ss['pos'] > 0.1:\n",
    "        if ss['neg'] >= ss['pos']:\n",
    "            result = 'neg'\n",
    "        else:\n",
    "            result = 'pos'\n",
    "    else:\n",
    "        result = 'neu'\n",
    "        \n",
    "    if i % 5000 == 0:\n",
    "        print(i)\n",
    "    \n",
    "    \n",
    "    brown_file['label'].loc[i] = result\n",
    "    \n",
    "print('done brown')\n",
    "    \n",
    "for i, line in enumerate(climate_file['text']):\n",
    "    ss = sid.polarity_scores(line)\n",
    "    \n",
    "    if ss['neg'] > 0.1 or ss['pos'] > 0.1:\n",
    "        if ss['neg'] >= ss['pos']:\n",
    "            result = 'neg'\n",
    "        else:\n",
    "            result = 'pos'\n",
    "    else:\n",
    "        result = 'neu'\n",
    "    \n",
    "    if i % 10000 == 0:\n",
    "        print(i)\n",
    "    \n",
    "    climate_file['label'].loc[i] = result\n",
    "    \n",
    "print('done climate')\n",
    "\n",
    "for i, line in enumerate(mfs_file['text']):\n",
    "    ss = sid.polarity_scores(line)\n",
    "    \n",
    "    if ss['neg'] > 0.1 or ss['pos'] > 0.1:\n",
    "        if ss['neg'] >= ss['pos']:\n",
    "            result = 'neg'\n",
    "        else:\n",
    "            result = 'pos'\n",
    "    else:\n",
    "        result = 'neu'\n",
    "        \n",
    "    if i % 10000 == 0:\n",
    "        print(i)\n",
    "    \n",
    "    \n",
    "    mfs_file['label'].loc[i] = result\n",
    "    \n",
    "print('done mfs')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pos</td>\n",
       "      <td>The Fulton County Grand Jury said Friday an in...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pos</td>\n",
       "      <td>The jury further said in term-end presentment...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pos</td>\n",
       "      <td>The September-October term jury had been char...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>pos</td>\n",
       "      <td>`` Only a relative handful of such reports wa...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>neg</td>\n",
       "      <td>The jury said it did find that many of Georgi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                               text\n",
       "0   pos  The Fulton County Grand Jury said Friday an in...\n",
       "1   pos   The jury further said in term-end presentment...\n",
       "2   pos   The September-October term jury had been char...\n",
       "3   pos   `` Only a relative handful of such reports wa...\n",
       "4   neg   The jury said it did find that many of Georgi..."
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "brown_file.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>pos</td>\n",
       "      <td>March for our right to clean air, water, land ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pos</td>\n",
       "      <td>Melania Trump a priority under new US immigra...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pos</td>\n",
       "      <td>USA Should Leave B'cuz It’s a Business Deal’ ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>neu</td>\n",
       "      <td>Our government is being run by climate denier...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>pos</td>\n",
       "      <td>Tomorrow, I march for climate justice and to ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                               text\n",
       "0   pos  March for our right to clean air, water, land ...\n",
       "1   pos   Melania Trump a priority under new US immigra...\n",
       "2   pos   USA Should Leave B'cuz It’s a Business Deal’ ...\n",
       "3   neu   Our government is being run by climate denier...\n",
       "4   pos   Tomorrow, I march for climate justice and to ..."
      ]
     },
     "execution_count": 128,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "climate_file.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>neg</td>\n",
       "      <td>Evansville, IN Trump would cut med research, d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>pos</td>\n",
       "      <td>is fighting science. I guarantee science will...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>pos</td>\n",
       "      <td>That's a bummer. is fighting science. I guara...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>neu</td>\n",
       "      <td>Authorities estimated that more than people w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>pos</td>\n",
       "      <td>This is what democracy looks like. President ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  label                                               text\n",
       "0   neg  Evansville, IN Trump would cut med research, d...\n",
       "1   pos   is fighting science. I guarantee science will...\n",
       "2   pos   That's a bummer. is fighting science. I guara...\n",
       "3   neu   Authorities estimated that more than people w...\n",
       "4   pos   This is what democracy looks like. President ..."
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mfs_file.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "metadata": {},
   "outputs": [],
   "source": [
    "brown_file.to_csv('data/csv/brown_sentiment.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "metadata": {},
   "outputs": [],
   "source": [
    "climate_file.to_csv('data/csv/climate_sentiment.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfs_file.to_csv('data/csv/mfs_sentiment.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use pd.read_csv to load back in"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scattertext\n",
    "Using scattertext to plot sentiment analysis trends."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {},
   "outputs": [],
   "source": [
    "import scattertext as st\n",
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "# run \"python3 -m spacy download en\" first\n",
    "nlp = spacy.load('en')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This block took a few hours on a 2.3 ghz cpu\n",
    "brown_corpus = st.CorpusFromPandas(brown_file, category_col='label', text_col='text', nlp=nlp).build()\n",
    "climate_corpus = st.CorpusFromPandas(climate_file, category_col='label', text_col='text', nlp=nlp).build()\n",
    "mfs_corpus = st.CorpusFromPandas(mfs_file, category_col='label', text_col='text', nlp=nlp).build()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 172,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To save scattertext corpora because they take forever to build\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 173,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('models/scattertext/brown.pkl', 'wb') as output:\n",
    "    pickle.dump(brown_corpus, output, pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 174,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('models/scattertext/climate.pkl', 'wb') as output:\n",
    "    pickle.dump(climate_corpus, output, pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('models/scattertext/mfs.pkl', 'wb') as output:\n",
    "    pickle.dump(mfs_corpus, output, pickle.HIGHEST_PROTOCOL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To load\n",
    "with open('models/scattertext/brown.pkl', 'rb') as input_f:\n",
    "    brown_corpus = pickle.load(input_f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['af', 'khrushchev', 'anode', 'negroes', 'prokofieff', 'negro', 'kohnstamm', 'helva', 'mrs', 'matsuo']\n"
     ]
    }
   ],
   "source": [
    "print(list(brown_corpus.get_scaled_f_scores_vs_background().index[:10]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['trump', 'marching', 'climate', 'retweet', 'streets', 'ppl', 'fight', 'badlyyyy', 'dc', 'protesting']\n"
     ]
    }
   ],
   "source": [
    "print(list(climate_corpus.get_scaled_f_scores_vs_background().index[:10]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['trump', 'marching', 'scientists', 'protest', 'retweet', 'science', 'lobbyists', 'sword', 'marches', 'phds']\n"
     ]
    }
   ],
   "source": [
    "print(list(mfs_corpus.get_scaled_f_scores_vs_background().index[:10]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 156,
   "metadata": {},
   "outputs": [],
   "source": [
    "term_freq_df_b = brown_corpus.get_term_freq_df()\n",
    "term_freq_df_c = climate_corpus.get_term_freq_df()\n",
    "term_freq_df_m = mfs_corpus.get_term_freq_df()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 157,
   "metadata": {},
   "outputs": [],
   "source": [
    "term_freq_df_b['neg score'] = brown_corpus.get_scaled_f_scores('neg')\n",
    "term_freq_df_c['neg score'] = climate_corpus.get_scaled_f_scores('neg')\n",
    "term_freq_df_m['neg score'] = mfs_corpus.get_scaled_f_scores('neg')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 158,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['hell',\n",
      " 'dead',\n",
      " 'enemy',\n",
      " 'death',\n",
      " 'pain',\n",
      " 'died',\n",
      " 'bad',\n",
      " 'killed',\n",
      " 'war',\n",
      " 'murder']\n",
      "['treason',\n",
      " 'racism',\n",
      " 'polluted',\n",
      " 'lies',\n",
      " 'fear ny',\n",
      " 'might rise',\n",
      " 'oceans might',\n",
      " 'fear in',\n",
      " 'declaring war',\n",
      " 'islamophobia']\n",
      "['and no',\n",
      " 'a funny',\n",
      " 'funny sign',\n",
      " 'are fighting',\n",
      " 'war in',\n",
      " 'marched to',\n",
      " 'think of',\n",
      " 'fighting to',\n",
      " 'w/ them',\n",
      " 'wonder scientists']\n"
     ]
    }
   ],
   "source": [
    "pprint(list(term_freq_df_b.sort_values(by='neg score', ascending=False).index[:10]))\n",
    "pprint(list(term_freq_df_c.sort_values(by='neg score', ascending=False).index[:10]))\n",
    "pprint(list(term_freq_df_m.sort_values(by='neg score', ascending=False).index[:10]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [],
   "source": [
    "term_freq_df_m['pos score'] = mfs_corpus.get_scaled_f_scores('pos')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 163,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['was chicago',\n",
      " 'show donald',\n",
      " 'retweet to',\n",
      " 'well this',\n",
      " 'cherish',\n",
      " 'cherish it',\n",
      " 'those today',\n",
      " 'embrace it',\n",
      " 'today who',\n",
      " 'a shout',\n",
      " 'who the',\n",
      " 'embrace',\n",
      " 'chicago today',\n",
      " 'good is',\n",
      " 'amazing signs',\n",
      " 'how good',\n",
      " 'signs up',\n",
      " 'grateful',\n",
      " 'many amazing',\n",
      " 'up here',\n",
      " 'in how',\n",
      " 'who support',\n",
      " 'grateful to',\n",
      " 'today all',\n",
      " 'support scien',\n",
      " 'so yes',\n",
      " 'relations so',\n",
      " 'scien',\n",
      " 'arts of',\n",
      " 'useful arts']\n"
     ]
    }
   ],
   "source": [
    "pprint(list(term_freq_df_m.sort_values(by='pos score', ascending=False).index[:30]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 164,
   "metadata": {},
   "outputs": [],
   "source": [
    "html = st.produce_scattertext_explorer(brown_corpus, category='neg', \n",
    "                                       category_name='Negative', \n",
    "                                       not_category_name='Pos or neutral', \n",
    "                                       width_in_pixels=1000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 165,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10034407"
      ]
     },
     "execution_count": 165,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open(\"./scattertext-html/test.html\", 'wb').write(html.encode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 167,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/niall/.local/lib/python3.6/site-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    }
   ],
   "source": [
    "html_b = st.word_similarity_explorer_gensim(brown_corpus,\n",
    "                                       category='neg',\n",
    "                                       category_name='Negative',\n",
    "                                       not_category_name='Pos/Neu',\n",
    "                                       target_term='science',\n",
    "                                       minimum_term_frequency=5,\n",
    "                                       pmi_threshold_coefficient=4,\n",
    "                                       width_in_pixels=1000,\n",
    "                                       word2vec=brown_model,\n",
    "                                       max_p_val=0.05,\n",
    "                                       save_svg_button=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 169,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11182897"
      ]
     },
     "execution_count": 169,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('./scattertext-html/brown_gensim_similarity.html', 'wb').write(html_b.encode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/niall/.local/lib/python3.6/site-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    }
   ],
   "source": [
    "html_c = st.word_similarity_explorer_gensim(climate_corpus,\n",
    "                                       category='neg',\n",
    "                                       category_name='Negative',\n",
    "                                       not_category_name='Pos/Neu',\n",
    "                                       target_term='science',\n",
    "                                       minimum_term_frequency=5,\n",
    "                                       pmi_threshold_coefficient=4,\n",
    "                                       width_in_pixels=1000,\n",
    "                                       word2vec=climate_model,\n",
    "                                       max_p_val=0.05,\n",
    "                                       save_svg_button=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "43715579"
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('./scattertext-html/climate_gensim_similarity.html', 'wb').write(html_c.encode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/niall/.local/lib/python3.6/site-packages/gensim/matutils.py:737: FutureWarning: Conversion of the second argument of issubdtype from `int` to `np.signedinteger` is deprecated. In future, it will be treated as `np.int64 == np.dtype(int).type`.\n",
      "  if np.issubdtype(vec.dtype, np.int):\n"
     ]
    }
   ],
   "source": [
    "html_m = st.word_similarity_explorer_gensim(mfs_corpus,\n",
    "                                       category='neg',\n",
    "                                       category_name='Negative',\n",
    "                                       not_category_name='Pos/Neu',\n",
    "                                       target_term='science',\n",
    "                                       minimum_term_frequency=5,\n",
    "                                       pmi_threshold_coefficient=4,\n",
    "                                       width_in_pixels=1000,\n",
    "                                       word2vec=mfs_model,\n",
    "                                       max_p_val=0.05,\n",
    "                                       save_svg_button=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "69292241"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "open('./scattertext-html/mfs_gensim_similarity.html', 'wb').write(html_m.encode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}