2. Sentiment Analysis in Python/Chapter 3_More on Numeric Vectors Transforming Tweets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Word cloud of tweets\n",
    "Your task in this exercise is to plot a word cloud using a sample of Twitter data, expressing customers' sentiments about airlines. A string text_tweet has been created for you and it contains the messages of a 1000 customers shared on Twitter.\n",
    "\n",
    "In the first step, your are asked to build the word cloud without removing the stop words, and in the second step to build the same cloud after you have removed the stop words.\n",
    "\n",
    "Feel free to familiarize yourself with the text_tweet list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the word cloud function and package.\n",
    "* Create and generate the word cloud, using the text_tweet vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the word cloud function \n",
    "from wordcloud import WordCloud \n",
    "\n",
    "# Create and generate a word cloud image\n",
    "my_cloud = WordCloud(background_color='white').generate(text_tweet)\n",
    "\n",
    "# Display the generated wordcloud image\n",
    "plt.imshow(my_cloud, interpolation='bilinear') \n",
    "plt.axis(\"off\")\n",
    "\n",
    "# Don't forget to show the final image\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Define the default list of stop words and update it.\n",
    "* Specify the stop words argument in the WordCloud function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the word cloud function and stop words list\n",
    "from wordcloud import WordCloud, STOPWORDS \n",
    "\n",
    "# Define and update the list of stopwords\n",
    "my_stop_words = STOPWORDS.update(['airline', 'airplane'])\n",
    "\n",
    "# Create and generate a word cloud image\n",
    "my_cloud = WordCloud(stopwords=my_stop_words).generate(text_tweet)\n",
    "\n",
    "# Display the generated wordcloud image\n",
    "plt.imshow(my_cloud, interpolation='bilinear') \n",
    "plt.axis(\"off\")\n",
    "# Don't forget to show the final image\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Airline sentiment with stop words\n",
    "You are given a dataset, called tweets, which contains customers' reviews and sentiments about airlines. It consists of two columns: airline_sentiment and text where the sentiment can be positive, negative or neutral, and the text is the text of the tweet.\n",
    "\n",
    "In this exercise, you will create a BOW representation but will account for the stop words. Remember that stop words are not informative and you might want to remove them. That will result in a smaller vocabulary and eventually, fewer features. Keep in mind that we can enrich a default list of stop words with ones that are specific to our context."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the default list of English stop words.\n",
    "* Update the default list of stop words with the given list ['airline', 'airlines', '@'] to create my_stop_words.\n",
    "* Specify the stop words argument in the vectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the stop words\n",
    "from sklearn.feature_extraction.text import CountVectorizer, ENGLISH_STOP_WORDS\n",
    "\n",
    "# Define the stop words\n",
    "my_stop_words = ENGLISH_STOP_WORDS.union(['airline', 'airlines', '@'])\n",
    "\n",
    "# Build and fit the vectorizer\n",
    "vect = CountVectorizer(stop_words=my_stop_words)\n",
    "vect.fit(tweets.text)\n",
    "\n",
    "# Create the bow representation\n",
    "X_review = vect.transform(tweets.text)\n",
    "# Create the data frame\n",
    "X_df = pd.DataFrame(X_review.toarray(), columns=vect.get_feature_names())\n",
    "print(X_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Multiple text columns\n",
    "In this exercise, you will continue working with the airline Twitter data. A dataset tweets has been imported for you.\n",
    "\n",
    "In some situations, you might have more than one text column in a dataset and you might want to create a numeric representation for each of the text columns. Here, besides the text column, which contains the body of the tweet, there is a second text column, called negativereason. It contains the reason the customer left a negative review.\n",
    "\n",
    "Your task is to build BOW representations for both columns and specify the required stop words."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the vectorizer package and the default list of English stop words.\n",
    "* Update the default list of English stop words and create the my_stop_words set.\n",
    "* Specify the stop words argument in the first vectorizer to the updated set, and in the second vectorizer - the default set of English stop words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the vectorizer and default English stop words list\n",
    "from sklearn.feature_extraction.text import CountVectorizer, ENGLISH_STOP_WORDS\n",
    "\n",
    "# Define the stop words\n",
    "my_stop_words = ENGLISH_STOP_WORDS.union(['airline', 'airlines', '@', 'am', 'pm'])\n",
    " \n",
    "# Build and fit the vectorizers\n",
    "vect1 = CountVectorizer(stop_words=my_stop_words)\n",
    "vect2 = CountVectorizer(stop_words=ENGLISH_STOP_WORDS) \n",
    "vect1.fit(tweets.text)\n",
    "vect2.fit(tweets.negative_reason)\n",
    "\n",
    "# Print the last 15 features from the first, and all from second vectorizer\n",
    "print(vect1.get_feature_names()[-15:])\n",
    "print(vect2.get_feature_names())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Specify the token pattern\n",
    "In this exercise, you will work with the text column of the tweets dataset. Your task is to vectorize the object column using CountVectorizer. You will apply different patterns of tokens in the vectorizer. Remember that by specifying the token pattern, you can filter out characters.\n",
    "\n",
    "The CountVectorizer has been imported for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Build a vectorizer from the text column, specifying the pattern of tokens to be equal to r'\\b[^\\d\\W][^\\d\\W]'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build and fit the vectorizer\n",
    "vect = CountVectorizer(token_pattern=r'\\b[^\\d\\W][^\\d\\W]+\\b').fit(tweets.text)\n",
    "vect_text = vect.transform(tweets.text)\n",
    "print('Length of vectorizer: ', len(vect.get_feature_names()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Build a vectorizer from the text column using the default values of the function's arguments.\n",
    "* Build a second vectorizer, specifying the pattern of tokens to be equal to r'\\b[^\\d\\W][^\\d\\W]'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build the first vectorizer\n",
    "vect1 = CountVectorizer().fit(tweets.text)\n",
    "vect1.transform(tweets.text)\n",
    "\n",
    "# Build the second vectorizer\n",
    "vect2 = CountVectorizer(token_pattern=r'\\b[^\\d\\W][^\\d\\W]').fit(tweets.text)\n",
    "vect2.transform(tweets.text)\n",
    "\n",
    "# Print out the length of each vectorizer\n",
    "print('Length of vectorizer 1: ', len(vect1.get_feature_names()))\n",
    "print('Length of vectorizer 2: ', len(vect2.get_feature_names()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### String operators with the Twitter data\n",
    "You continue working with the tweets data where the text column stores the content of each tweet.\n",
    "\n",
    "Your task is to turn the text column into a list of tokens. Then, using string operators, remove all non-alphabetic characters from the created list of tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the word tokenizing function.\n",
    "* Create word tokens from each tweet.\n",
    "* Filter out all non-alphabetic characters from the created list, i.e. retain only letters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the word tokenizing package\n",
    "from nltk import word_tokenize\n",
    "\n",
    "# Tokenize the text column\n",
    "word_tokens = [word_tokenize(review) for review in tweets.text]\n",
    "print('Original tokens: ', word_tokens[0])\n",
    "\n",
    "# Filter out non-letter characters\n",
    "cleaned_tokens = [[word for word in item if word.isalpha()] for item in word_tokens]\n",
    "print('Cleaned tokens: ', cleaned_tokens[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More string operators and Twitter\n",
    "In this exercise, you will apply different string operators to three strings, selected from the tweets dataset. A tweets_list has been created for you.\n",
    "\n",
    "You need to construct three new lists by applying different string operators:\n",
    "\n",
    "* a list retaining only letters\n",
    "* a list retaining only characters\n",
    "* a list retaining only digits\n",
    "\n",
    "The required functions have been imported for you from nltk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Create a list of the tokens from tweets_list.\n",
    "* In the list letters remove all digits and other characters, i.e. keep only letters.\n",
    "* Retain alphanumeric characters but remove all other characters in let_digits.\n",
    "* Create digits by removing letters and characters and keeping only numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a list of lists, containing the tokens from list_tweets\n",
    "tokens = [word_tokenize(item) for item in tweets_list]\n",
    "\n",
    "# Remove characters and digits , i.e. retain only letters\n",
    "letters = [[word for word in item if word.isalpha()] for item in tokens]\n",
    "# Remove characters, i.e. retain only letters and digits\n",
    "let_digits = [[word for word in item if word.isalnum()] for item in tokens]\n",
    "# Remove letters and characters, retain only digits\n",
    "digits = [[word for word in item if word.isdigit()] for item in tokens]\n",
    "\n",
    "# Print the last item in each list\n",
    "print('Last item in alphabetic list: ', letters[2])\n",
    "print('Last item in list of alphanumerics: ', let_digits[2])\n",
    "print('Last item in the list of digits: ', digits[2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stems and lemmas from GoT\n",
    "In this exercise, you are given a couple of sentences from George R.R. Martin's Game of Thrones. Your task is to create stems and lemmas from the given GoT string.\n",
    "\n",
    "Remember that stems reduce a word to its root whereas lemmas produce an actual word. However, speed can differ significantly between the methods with stemming being much faster. In Steps 2 and 3, pay attention to the total time it takes to perform each operation. We're making use of the time.time() method to measure the time it takes to perform stemming and lemmatization."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the stemming and lemmatization functions.\n",
    "* Build a list of tokens from the GoT string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the required packages from nltk\n",
    "from nltk.stem import PorterStemmer, WordNetLemmatizer\n",
    "from nltk import word_tokenize\n",
    "\n",
    "porter = PorterStemmer()\n",
    "WNlemmatizer = WordNetLemmatizer()\n",
    "\n",
    "# Tokenize the GoT string\n",
    "tokens = word_tokenize(GoT) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Using list comprehension and the porter stemmer you imported, create the stemmed_tokens list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# Log the start time\n",
    "start_time = time.time()\n",
    "\n",
    "# Build a stemmed list\n",
    "stemmed_tokens = [porter.stem(token) for token in tokens] \n",
    "\n",
    "# Log the end time\n",
    "end_time = time.time()\n",
    "\n",
    "print('Time taken for stemming in seconds: ', end_time - start_time)\n",
    "print('Stemmed tokens: ', stemmed_tokens) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Using list comprehension and the WNlemmatizer you imported, create the lem_tokens list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "\n",
    "# Log the start time\n",
    "start_time = time.time()\n",
    "\n",
    "# Build a lemmatized list\n",
    "lem_tokens = [WNlemmatizer.lemmatize(token) for token in tokens]\n",
    "\n",
    "# Log the end time\n",
    "end_time = time.time()\n",
    "\n",
    "print('Time taken for lemmatizing in seconds: ', end_time - start_time)\n",
    "print('Lemmatized tokens: ', lem_tokens) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stem Spanish reviews\n",
    "You will recall that in a previous chapter we used a language detection package to determine the language of different Amazon product reviews. In this exercise, you will first detect the languages in the non_english_reviews. The reviews are in multiple languages but you will select ONLY those in Spanish. Feel free to go back to the video discussing foreign language detection if you have forgotten some of the concepts.\n",
    "\n",
    "In the second step, you will create word tokens from the Spanish reviews and will stem them using a SnowBall stemmer for the Spanish language. The language detection package is not perfect, unfortunately. Therefore, it is possible that sometimes the detected language is not correct."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the langdetect package.\n",
    "* Iterate over the rows of the non_english_reviews using the len() method and range() function.\n",
    "* Use detect_langs() to detect the language of each review in the for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the language detection package\n",
    "import langdetect\n",
    "\n",
    "# Loop over the rows of the dataset and append  \n",
    "languages = [] \n",
    "for i in range(len(non_english_reviews)):\n",
    "    languages.append(langdetect.detect_langs(non_english_reviews.iloc[i, 1]))\n",
    "\n",
    "# Clean the list by splitting     \n",
    "languages = [str(lang).split(':')[0][1:] for lang in languages]\n",
    "# Assign the list to a new feature \n",
    "non_english_reviews['language'] = languages\n",
    "\n",
    "# Select the Spanish ones\n",
    "filtered_reviews = non_english_reviews[non_english_reviews.language == 'es']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the SnowballStemmer from the respective package.\n",
    "* Create word tokens from the review column of the filtered_reviews from the previous step.\n",
    "* Use the Spanish stemmer you imported to stem the created list of tokens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the required packages\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from nltk import word_tokenize\n",
    "\n",
    "# Import the Spanish SnowballStemmer\n",
    "SpanishStemmer = SnowballStemmer(\"spanish\")\n",
    "\n",
    "# Create a list of tokens\n",
    "tokens = [word_tokenize(review) for review in filtered_reviews.review]\n",
    "# Stem the list of tokens\n",
    "stemmed_tokens = [[SpanishStemmer.stem(word) for word in token] for token in tokens]\n",
    "\n",
    "# Print the first item of the stemmed tokenss\n",
    "print(stemmed_tokens[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stems from tweets\n",
    "In this exercise, you will work with an array called tweets. It contains the text of the airline sentiment data collected from Twitter.\n",
    "\n",
    "Your task is to work with this array and transform it into a list of tokens using list comprehension. After that, iterate over the list of tokens and create a stem out of each token. Remember that list comprehensions are a one-line alternative to for loops."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the function we used to transform strings into stems.\n",
    "* Call the Porter stemmer function you just imported.\n",
    "* Using a list comprehension, create the list tokens. It should contain all the word tokens from the tweets array.\n",
    "* Iterate over the tokens list and apply the stemming function to each item in the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the function to perform stemming\n",
    "from nltk.stem import PorterStemmer\n",
    "from nltk import word_tokenize\n",
    "\n",
    "# Call the stemmer\n",
    "porter = PorterStemmer()\n",
    "\n",
    "# Transform the array of tweets to tokens\n",
    "tokens = [word_tokenize(word) for word in tweets]\n",
    "# Stem the list of tokens\n",
    "stemmed_tokens = [[porter.stem(word) for word in tweet] for tweet in tokens] \n",
    "# Print the first element of the list\n",
    "print(stemmed_tokens[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Your first TfIdf\n",
    "In this exercise, you will apply the TfIdf method to the small annak dataset, containing the first sentence of Anna Karenina by Leo Tolstoy.\n",
    "\n",
    "Your task will be to work with this dataset and apply the TfidfVectorizer() function. Recall that performing a numeric transformation of text is your first step in being able to understand the sentiment of the text. The Tfidf vectorizer is another way to construct a vocabulary from our sentiment column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the function for building a TfIdf vectorizer from sklearn.feature_extraction.text.\n",
    "* Call the TfidfVectorizer() function and fit it on the annak dataset .\n",
    "* Transform the vectorizer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the required function\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer\n",
    "\n",
    "annak = ['Happy families are all alike;', 'every unhappy family is unhappy in its own way']\n",
    "\n",
    "# Call the vectorizer and fit it\n",
    "anna_vect = TfidfVectorizer().fit(annak)\n",
    "\n",
    "# Create the tfidf representation\n",
    "anna_tfidf = anna_vect.transform(annak)\n",
    "\n",
    "# Print the result \n",
    "print(anna_tfidf.toarray())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### TfIdf on Twitter airline sentiment data\n",
    "You will now build features using the TfIdf method. You will continue to work with the tweets dataset.\n",
    "\n",
    "In this exercise, you will utilize what you have learned in previous lessons and remove stop words, use a token pattern and specify the n-grams.\n",
    "\n",
    "The final output will be a DataFrame, of which the columns are created using the TfidfVectorizer(). Such a DataFrame can directly be passed to a supervised learning model, which is what we will tackle in the next chapter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the required package to build a TfidfVectorizer and the ENGLISH_STOP_WORDS.\n",
    "* Build a TfIdf vectorizer from the text column of the tweets dataset, specifying uni- and bi-grams as a choice of n-grams, tokens which include only alphanumeric characters using the given token pattern, and the stop words corresponding to the ENGLISH_STOP_WORDS.\n",
    "* Transform the vectorizer, specifying the same column that you fit.\n",
    "* Specify the column names in the DataFrame() function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the required vectorizer package and stop words list\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer,ENGLISH_STOP_WORDS\n",
    "\n",
    "# Define the vectorizer and specify the arguments\n",
    "my_pattern = r'\\b[^\\d\\W][^\\d\\W]+\\b'\n",
    "vect = TfidfVectorizer(ngram_range=(1, 2), max_features=100, token_pattern=my_pattern, stop_words=ENGLISH_STOP_WORDS).fit(tweets.text)\n",
    "\n",
    "# Transform the vectorizer\n",
    "X_txt = vect.transform(tweets.text)\n",
    "\n",
    "# Transform to a data frame and specify the column names\n",
    "X=pd.DataFrame(X_txt.toarray(), columns=vect.get_feature_names())\n",
    "print('Top 5 rows of the DataFrame: ', X.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Tfidf and a BOW on same data\n",
    "In this exercise, you will transform the review column of the Amazon product reviews using both a bag-of-words and a tfidf transformation.\n",
    "\n",
    "Build both vectorizers, specifying only the maximum number of features to be equal to 100. Create DataFrames after the transformation and print the top 5 rows of each.\n",
    "\n",
    "Be careful how you specify the maximum number of features in the vocabulary. A large vocabulary size can result in your session being disconnected."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Import the BOW and Tfidf vectorizers.\n",
    "* Build and fit a BOW and a Tfidf vectorizer from the review column and limit the number of created features to 100.\n",
    "* Create DataFrames from the transformed vector representations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import the required packages\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer\n",
    "\n",
    "# Build a BOW and tfidf vectorizers from the review column and with max of 100 features\n",
    "vect1 = CountVectorizer(max_features=100).fit(reviews.review)\n",
    "vect2 = TfidfVectorizer(max_features=100).fit(reviews.review) \n",
    "\n",
    "# Transform the vectorizers\n",
    "X1 = vect1.transform(reviews.review)\n",
    "X2 = vect2.transform(reviews.review)\n",
    "# Create DataFrames from the vectorizers \n",
    "X_df1 = pd.DataFrame(X1.toarray(), columns=vect1.get_feature_names())\n",
    "X_df2 = pd.DataFrame(X2.toarray(), columns=vect2.get_feature_names())\n",
    "print('Top 5 rows using BOW: \\n', X_df1.head())\n",
    "print('Top 5 rows using tfidf: \\n', X_df2.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.6 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.8.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "92adc0dc7608e96a50ce5fb9acbd8bea0a01f16a513a55f8bb8294402fce5e61"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}