Sentiment Analysis Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A couple of reminders about how to use Jupyter Notebook: \n",
    "- Hit Shift+Enter to run code and finish typing in Markdown\n",
    "- You should not have to change much code here, but be careful to document carefully if you do "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import all the necessary packages into Python. If there are any you don't have, type \"!pip install package-name\" above the import statements and it should install that package. You should only have to do that once, so you can delete it right afterwards. To get access to the sentiment module package in this notebook, use this link and extract the resulting zip file into your .ipython folder in your user directory: https://www.csc2.ncsu.edu/faculty/healey/msa-18/text/sentiment_module.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sentiment_module import sentiment\n",
    "import pandas as pd\n",
    "import nltk\n",
    "import re\n",
    "import string"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This part is initializes the various lists that will be used to form the dataframes below. The preprocess() function in this case reads in the file, and takes the president and reelection values as parameters (these can be removed, it was relevant for this project but may not be for yours. It can also be used for any sort of categorization of the sentiment that you may want to do). Then a term vectors is created with stop words removed, and sentences are scored on their sentiment using the sentiment dictionary in the sentiment module, based on their arousal (how active/inactive the emotion is, for example \"content\" is low arousal and \"excited\" is high arousal) and their valence (how positive or negative a feeling is). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "punc = re.compile( '[%s]' % re.escape( string.punctuation ) )\n",
    "term_vec = []\n",
    "presidents = []\n",
    "reelections = []\n",
    "porter = nltk.stem.porter.PorterStemmer()\n",
    "stop_words = nltk.corpus.stopwords.words( 'english' )\n",
    "additional = ['weve','ive','were','would','well','here','there']\n",
    "arousal = []\n",
    "valence = []\n",
    "\n",
    "def preprocess(file, president,reelection):\n",
    "    term_vec_inner = []\n",
    "    president_list = []\n",
    "    reelection_list = []\n",
    "    arousal_list = []\n",
    "    valence_list = []\n",
    "    with open (file, \"r\",encoding=\"utf-8\") as myfile:\n",
    "        filename=myfile.readlines()\n",
    "        filename = [x.strip() for x in filename]\n",
    "    for d in filename:\n",
    "        d = d.lower()\n",
    "        d = punc.sub( '', d )\n",
    "        d = nltk.word_tokenize(d) \n",
    "        e = []\n",
    "        for i in d:\n",
    "            if i not in stop_words and i not in additional:\n",
    "                e.append(i)\n",
    "        president_list.append(president)\n",
    "        term_vec_inner.append(e)\n",
    "        reelection_list.append(reelection)\n",
    "        arousal_list.append(sentiment.sentiment(e)['arousal'])\n",
    "        valence_list.append(sentiment.sentiment(e)['valence'])\n",
    "    term_vec.append(term_vec_inner)\n",
    "    presidents.append(president_list)\n",
    "    reelections.append(reelection_list)\n",
    "    arousal.append(arousal_list)\n",
    "    valence.append(valence_list)\n",
    "        \n",
    "preprocess(\"Obama_town_hall_pre.txt\",\"Obama\",\"No\")\n",
    "preprocess(\"Obama_town_hall_post.txt\",\"Obama\",\"Yes\")\n",
    "preprocess(\"Bush_town_hall_pre.txt\",\"Bush\",\"No\")\n",
    "preprocess(\"Bush_town_hall_post.txt\",\"Bush\",\"Yes\")\n",
    "preprocess(\"Clinton_town_hall_pre.txt\",\"Clinton\",\"No\")\n",
    "preprocess(\"Clinton_town_hall_post.txt\",\"Clinton\",\"Yes\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code below builds a Pandas dataframe based on the lists we have: 'Sentences' is the list of term vectors for each sentence (in this case, synonymous with \"document\") we read in. Then for each sentence, we have the president who said it, whether or not they were running for reelection when they said it, the arousal for the sentence, and the valence for the sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dd = pd.DataFrame({'Sentence': term_vec[0], 'President':presidents[0],'Reelection':reelections[0],'Arousal':arousal[0],'Valence':valence[0]})\n",
    "for i in range(1,len(term_vec)):\n",
    "    df = pd.DataFrame({'Sentence': term_vec[i], 'President':presidents[i],'Reelection':reelections[i],'Arousal':arousal[i],'Valence':valence[i]})\n",
    "    dd = dd.append(df, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Finally, we have code to write the Pandas dataframe to a csv file, so that we can do visualizations or analysis in another software tool, or simply to continue that analysis by reading this csv file into another Jupyter notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dd.to_csv('sentiment.csv')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}