news_NDiazDurana.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ead738d1",
   "metadata": {},
   "source": [
    "# Most Common Words per Classification in news.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e30c8d7",
   "metadata": {},
   "source": [
    "The file news.csv has data extracted from news on the internet with its corresponding classification depending on its content. They have five classifications: economy, sports, science, culture and entertainment.<br><br>\n",
    "We create a function that takes the data from the csv file and prints the five classifications with a list of the x most repeated words for each classification. The function has two parameters: the name of the file to be read and the number of words to show.<br><br>\n",
    "We use pandas, NLTK, gensim and collections."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "415b5c86",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: gensim in /home/nykolai/anaconda3/lib/python3.9/site-packages (4.1.2)\n",
      "Requirement already satisfied: scipy>=0.18.1 in /home/nykolai/anaconda3/lib/python3.9/site-packages (from gensim) (1.7.1)\n",
      "Requirement already satisfied: smart-open>=1.8.1 in /home/nykolai/anaconda3/lib/python3.9/site-packages (from gensim) (6.0.0)\n",
      "Requirement already satisfied: numpy>=1.17.0 in /home/nykolai/anaconda3/lib/python3.9/site-packages (from gensim) (1.20.3)\n"
     ]
    }
   ],
   "source": [
    "!pip install gensim"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b87df73d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import libraries\n",
    "\n",
    "import pandas as pd\n",
    "import nltk\n",
    "import gensim\n",
    "from gensim.utils import simple_preprocess\n",
    "from gensim.parsing.preprocessing import STOPWORDS\n",
    "from collections import Counter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b39cf1ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     /home/nykolai/nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Download stopwords\n",
    "\n",
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f3db1754",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Obtain additional stopwords from nltk\n",
    "\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = stopwords.words('english')\n",
    "stop_words.extend(['from', 'subject', 're', 'edu', 'use'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4b6aba3",
   "metadata": {},
   "source": [
    "## Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "70202879",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove stopwords and remove words with less than 3 characters\n",
    "\n",
    "def preprocess(text):\n",
    "    result = []\n",
    "    for token in gensim.utils.simple_preprocess(text):\n",
    "        if token not in gensim.parsing.preprocessing.STOPWORDS and len(token) > 3 and token not in stop_words:\n",
    "            result.append(token)            \n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "28a0051b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Identify the most common n words of a list\n",
    "\n",
    "def count(list, n):\n",
    "    total = Counter(list)\n",
    "    result = total.most_common(n)\n",
    "    words = []\n",
    "    for word in result:\n",
    "        words.append(word[0])\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "44d59ccd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open the news.csv file and return the most common x words per classification\n",
    "\n",
    "def most_common_words(file, x):\n",
    "    \n",
    "    # Read file into df dataframe\n",
    "    df = pd.read_csv(file)\n",
    "    df['clean'] = df['content'].apply(preprocess) # Apply the preprocess function to the dataframe\n",
    "    \n",
    "    # Create one list per category containing the whole of words corresponding to each classification\n",
    "    sports = df.loc[df['classification'] == 'sports']['clean'].sum()\n",
    "    economy = df.loc[df['classification'] == 'economy']['clean'].sum()\n",
    "    entertainment = df.loc[df['classification'] == 'entertainment']['clean'].sum()\n",
    "    culture = df.loc[df['classification'] == 'culture']['clean'].sum()\n",
    "    science = df.loc[df['classification'] == 'science']['clean'].sum()\n",
    "    \n",
    "    # Show results\n",
    "    print('The ' + str(x) + ' most common words per classification in the file news.csv are:' +\n",
    "        '\\n\\nSports: ' + str(count(sports, x)) + \n",
    "         '\\nEntertainment: ' + str(count(entertainment, x)) +\n",
    "         '\\nCulture: ' + str(count(culture, x)) +\n",
    "         '\\nScience: ' + str(count(science, x)) + \n",
    "         '\\nEconomy: ' + str(count(economy, x)))\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "2b2d3511",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The 8 most common words per classification in the file news.csv are:\n",
      "\n",
      "Sports: ['race', 'barty', 'time', 'liverpool', 'year', 'said', 'weekend', 'world']\n",
      "Entertainment: ['west', 'kanye', 'time', 'sophie', 'instagram', 'said', 'couple', 'madsen']\n",
      "Culture: ['said', 'hall', 'black', 'african', 'american', 'history', 'dallas', 'dance']\n",
      "Science: ['forests', 'says', 'water', 'spinosaurus', 'cooling', 'like', 'global', 'carbon']\n",
      "Economy: ['said', 'growth', 'china', 'year', 'russia', 'investment', 'economy', 'russian']\n"
     ]
    }
   ],
   "source": [
    "# Function\n",
    "most_common_words('news.csv', 8) # Number of shown words can be changed"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "023a4b79",
   "metadata": {},
   "source": [
    "### Appendix: Step-by-step with examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e5d2d3d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ex = pd.read_csv('news.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "d717b435",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>content</th>\n",
       "      <th>classification</th>\n",
       "      <th>webPage</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Is America’s Economy Entering a New Normal?</td>\n",
       "      <td>The pandemic and now the war in Ukraine have a...</td>\n",
       "      <td>economy</td>\n",
       "      <td>https://www.nytimes.com/2022/03/24/business/ec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mohamed Salah: New Liverpool deal likely to ha...</td>\n",
       "      <td>Mohamed Salah's options would be going \"sidewa...</td>\n",
       "      <td>sports</td>\n",
       "      <td>https://www.bbc.com/sport/football/60856636</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NASA’s exoplanet count surges past 5000</td>\n",
       "      <td>It’s official: The number of planets known bey...</td>\n",
       "      <td>science</td>\n",
       "      <td>https://www.sciencenews.org/article/exoplanet-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Moscow insiders describe panic frustration and...</td>\n",
       "      <td>Yuriy Shatalov (not his real name) an equity t...</td>\n",
       "      <td>economy</td>\n",
       "      <td>https://www.businessinsider.com/russian-trader...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Five 4000 Year Old Painted Tombs Discovered in...</td>\n",
       "      <td>Five painted tombs were recently unearthed in ...</td>\n",
       "      <td>culture</td>\n",
       "      <td>https://www.artnews.com/art-news/news/saqqara-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Julia Fox Backtracks After Calling Kanye West’...</td>\n",
       "      <td>She didn’t have all the facts. Julia Fox is ta...</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>https://www.usmagazine.com/celebrity-news/news...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Gifted and at the top of her game -- Ashleigh ...</td>\n",
       "      <td>(CNN)On her day Ashleigh Barty was an unstoppa...</td>\n",
       "      <td>sports</td>\n",
       "      <td>https://edition.cnn.com/2022/03/23/tennis/ashl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Spinosaurus’ dense bones fuel debate over whet...</td>\n",
       "      <td>A fierce group of predatory dinosaurs may have...</td>\n",
       "      <td>science</td>\n",
       "      <td>https://www.sciencenews.org/article/spinosauru...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Why did this cultural mecca for Dallas' Black ...</td>\n",
       "      <td>\"Dallas Rising: The Hall of Negro Life\" airs F...</td>\n",
       "      <td>culture</td>\n",
       "      <td>https://www.keranews.org/arts-culture/2022-03-...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Forests help reduce global warming in more way...</td>\n",
       "      <td>When it comes to cooling the planet forests ha...</td>\n",
       "      <td>science</td>\n",
       "      <td>https://www.sciencenews.org/article/forest-tre...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Saudi Arabian GP: When to watch practice quali...</td>\n",
       "      <td>Jeddah Corniche street race - which had a chao...</td>\n",
       "      <td>sports</td>\n",
       "      <td>https://www.skysports.com/f1/news/36788/125735...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>China’s economy posts healthy recovery solid m...</td>\n",
       "      <td>China’s economy started the year on a positive...</td>\n",
       "      <td>economy</td>\n",
       "      <td>https://www.hellenicshippingnews.com/chinas-ec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Sophie Turner and Joe Jonas expecting their se...</td>\n",
       "      <td>Actors Joe Jonas and Sophie Turner are expecti...</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>https://indianexpress.com/article/entertainmen...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Irish dancers perform in Ramona to raise money...</td>\n",
       "      <td>Irish dancers from the Clan Rince School of Ir...</td>\n",
       "      <td>culture</td>\n",
       "      <td>https://www.sandiegouniontribune.com/ramona-se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Michael Madsen Arrested in Malibu For Trespass...</td>\n",
       "      <td>The 64-year-old actor was apprehended at 9 p.m...</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>https://www.eonline.com/news/1320980/michael-m...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                title  \\\n",
       "0         Is America’s Economy Entering a New Normal?   \n",
       "1   Mohamed Salah: New Liverpool deal likely to ha...   \n",
       "2             NASA’s exoplanet count surges past 5000   \n",
       "3   Moscow insiders describe panic frustration and...   \n",
       "4   Five 4000 Year Old Painted Tombs Discovered in...   \n",
       "5   Julia Fox Backtracks After Calling Kanye West’...   \n",
       "6   Gifted and at the top of her game -- Ashleigh ...   \n",
       "7   Spinosaurus’ dense bones fuel debate over whet...   \n",
       "8   Why did this cultural mecca for Dallas' Black ...   \n",
       "9   Forests help reduce global warming in more way...   \n",
       "10  Saudi Arabian GP: When to watch practice quali...   \n",
       "11  China’s economy posts healthy recovery solid m...   \n",
       "12  Sophie Turner and Joe Jonas expecting their se...   \n",
       "13  Irish dancers perform in Ramona to raise money...   \n",
       "14  Michael Madsen Arrested in Malibu For Trespass...   \n",
       "\n",
       "                                              content classification  \\\n",
       "0   The pandemic and now the war in Ukraine have a...        economy   \n",
       "1   Mohamed Salah's options would be going \"sidewa...         sports   \n",
       "2   It’s official: The number of planets known bey...        science   \n",
       "3   Yuriy Shatalov (not his real name) an equity t...        economy   \n",
       "4   Five painted tombs were recently unearthed in ...        culture   \n",
       "5   She didn’t have all the facts. Julia Fox is ta...  entertainment   \n",
       "6   (CNN)On her day Ashleigh Barty was an unstoppa...         sports   \n",
       "7   A fierce group of predatory dinosaurs may have...        science   \n",
       "8   \"Dallas Rising: The Hall of Negro Life\" airs F...        culture   \n",
       "9   When it comes to cooling the planet forests ha...        science   \n",
       "10  Jeddah Corniche street race - which had a chao...         sports   \n",
       "11  China’s economy started the year on a positive...        economy   \n",
       "12  Actors Joe Jonas and Sophie Turner are expecti...  entertainment   \n",
       "13  Irish dancers from the Clan Rince School of Ir...        culture   \n",
       "14  The 64-year-old actor was apprehended at 9 p.m...  entertainment   \n",
       "\n",
       "                                              webPage  \n",
       "0   https://www.nytimes.com/2022/03/24/business/ec...  \n",
       "1         https://www.bbc.com/sport/football/60856636  \n",
       "2   https://www.sciencenews.org/article/exoplanet-...  \n",
       "3   https://www.businessinsider.com/russian-trader...  \n",
       "4   https://www.artnews.com/art-news/news/saqqara-...  \n",
       "5   https://www.usmagazine.com/celebrity-news/news...  \n",
       "6   https://edition.cnn.com/2022/03/23/tennis/ashl...  \n",
       "7   https://www.sciencenews.org/article/spinosauru...  \n",
       "8   https://www.keranews.org/arts-culture/2022-03-...  \n",
       "9   https://www.sciencenews.org/article/forest-tre...  \n",
       "10  https://www.skysports.com/f1/news/36788/125735...  \n",
       "11  https://www.hellenicshippingnews.com/chinas-ec...  \n",
       "12  https://indianexpress.com/article/entertainmen...  \n",
       "13  https://www.sandiegouniontribune.com/ramona-se...  \n",
       "14  https://www.eonline.com/news/1320980/michael-m...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# View original file\n",
    "df_ex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7a01f91d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'The pandemic and now the war in Ukraine have altered how America’s economy functions. While economists have spent months waiting for conditions to return to normal they are beginning to wonder what “normal” will mean. Some of the changes are noticeable in everyday life: Work from home is more popular burrito bowls and road trips cost more and buying a car or a couch made overseas is harder. But those are all symptoms of broader changes sweeping the economy — ones that could be a big deal for consumers businesses and policymakers alike if they linger. Consumer demand has been hot for months now workers are desperately wanted wages are climbing at a rapid clip and prices are rising at the fastest pace in four decades as vigorous buying clashes with roiled supply chains. Interest rates are expected to rise higher than they ever did in the 2010s as the Federal Reserve tries to rein in inflation. History is full of big moments that have changed America’s economic trajectory: The Great Depression of the 1930s the Great Inflation of the 1970s and the Great Recession of 2008 are examples. It’s too early to know for sure but the changes happening today could prove to be the next one. Economists have spent the past two years expecting many of the pandemic-era trends to prove temporary but that has not yet been the case. Forecasters predicted that rapid inflation would fade in 2021 only to have those expectations foiled as it accelerated instead. They thought workers would jump back into the labor market as schools reopened from pandemic shutdowns but many remain on its sidelines. And they thought consumer spending would taper off as government pandemic relief checks faded into the rearview mirror. Shoppers have kept at it. Now Russia’s invasion of Ukraine threatens the global geopolitical order yet another shock disrupting trade and the economic system. For Washington policymakers Wall Street investors and academic economists the surprises have added up to an economic mystery with potentially far-reaching consequences. The economy had spent decades churning out slow and steady growth clouded by weak demand interest rates that were chronically flirting with rock bottom and tepid inflation. Some are wondering if after repeated shocks that paradigm could change. “For the last quarter century we’ve had a perfect storm of disinflationary forces” Jerome H. Powell the Fed chair said in response to a question during a public appearance this week noting that the old regime had been disrupted by a pandemic a large spending and monetary policy response and a war that was generating “untold” economic uncertainty. “As we come out the other side of that the question is: What will be the nature of that economy?” he said.'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show example of news before removing stopwords\n",
    "df_ex['content'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "c90df012",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Apply the preprocess function to the dataframe\n",
    "df_ex['clean'] = df_ex['content'].apply(preprocess)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "d7c13422",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['pandemic', 'ukraine', 'altered', 'america', 'economy', 'functions', 'economists', 'spent', 'months', 'waiting', 'conditions', 'return', 'normal', 'beginning', 'wonder', 'normal', 'mean', 'changes', 'noticeable', 'everyday', 'life', 'work', 'home', 'popular', 'burrito', 'bowls', 'road', 'trips', 'cost', 'buying', 'couch', 'overseas', 'harder', 'symptoms', 'broader', 'changes', 'sweeping', 'economy', 'ones', 'deal', 'consumers', 'businesses', 'policymakers', 'alike', 'linger', 'consumer', 'demand', 'months', 'workers', 'desperately', 'wanted', 'wages', 'climbing', 'rapid', 'clip', 'prices', 'rising', 'fastest', 'pace', 'decades', 'vigorous', 'buying', 'clashes', 'roiled', 'supply', 'chains', 'rates', 'expected', 'rise', 'higher', 'federal', 'reserve', 'tries', 'rein', 'inflation', 'history', 'moments', 'changed', 'america', 'economic', 'trajectory', 'great', 'depression', 'great', 'inflation', 'great', 'recession', 'examples', 'early', 'know', 'sure', 'changes', 'happening', 'today', 'prove', 'economists', 'spent', 'past', 'years', 'expecting', 'pandemic', 'trends', 'prove', 'temporary', 'case', 'forecasters', 'predicted', 'rapid', 'inflation', 'fade', 'expectations', 'foiled', 'accelerated', 'instead', 'thought', 'workers', 'jump', 'labor', 'market', 'schools', 'reopened', 'pandemic', 'shutdowns', 'remain', 'sidelines', 'thought', 'consumer', 'spending', 'taper', 'government', 'pandemic', 'relief', 'checks', 'faded', 'rearview', 'mirror', 'shoppers', 'kept', 'russia', 'invasion', 'ukraine', 'threatens', 'global', 'geopolitical', 'order', 'shock', 'disrupting', 'trade', 'economic', 'washington', 'policymakers', 'wall', 'street', 'investors', 'academic', 'economists', 'surprises', 'added', 'economic', 'mystery', 'potentially', 'reaching', 'consequences', 'economy', 'spent', 'decades', 'churning', 'slow', 'steady', 'growth', 'clouded', 'weak', 'demand', 'rates', 'chronically', 'flirting', 'rock', 'tepid', 'inflation', 'wondering', 'repeated', 'shocks', 'paradigm', 'change', 'quarter', 'century', 'perfect', 'storm', 'disinflationary', 'forces', 'jerome', 'powell', 'chair', 'said', 'response', 'question', 'public', 'appearance', 'week', 'noting', 'regime', 'disrupted', 'pandemic', 'large', 'spending', 'monetary', 'policy', 'response', 'generating', 'untold', 'economic', 'uncertainty', 'come', 'question', 'nature', 'economy', 'said']\n"
     ]
    }
   ],
   "source": [
    "# Show example of cleaned up news after removing stopwords\n",
    "print(df_ex['clean'][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "26944fb0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>content</th>\n",
       "      <th>classification</th>\n",
       "      <th>webPage</th>\n",
       "      <th>clean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Is America’s Economy Entering a New Normal?</td>\n",
       "      <td>The pandemic and now the war in Ukraine have a...</td>\n",
       "      <td>economy</td>\n",
       "      <td>https://www.nytimes.com/2022/03/24/business/ec...</td>\n",
       "      <td>[pandemic, ukraine, altered, america, economy,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Mohamed Salah: New Liverpool deal likely to ha...</td>\n",
       "      <td>Mohamed Salah's options would be going \"sidewa...</td>\n",
       "      <td>sports</td>\n",
       "      <td>https://www.bbc.com/sport/football/60856636</td>\n",
       "      <td>[mohamed, salah, options, going, sideways, man...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>NASA’s exoplanet count surges past 5000</td>\n",
       "      <td>It’s official: The number of planets known bey...</td>\n",
       "      <td>science</td>\n",
       "      <td>https://www.sciencenews.org/article/exoplanet-...</td>\n",
       "      <td>[official, number, planets, known, solar, pass...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Moscow insiders describe panic frustration and...</td>\n",
       "      <td>Yuriy Shatalov (not his real name) an equity t...</td>\n",
       "      <td>economy</td>\n",
       "      <td>https://www.businessinsider.com/russian-trader...</td>\n",
       "      <td>[yuriy, shatalov, real, equity, trader, moscow...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Five 4000 Year Old Painted Tombs Discovered in...</td>\n",
       "      <td>Five painted tombs were recently unearthed in ...</td>\n",
       "      <td>culture</td>\n",
       "      <td>https://www.artnews.com/art-news/news/saqqara-...</td>\n",
       "      <td>[painted, tombs, recently, unearthed, saqqara,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Julia Fox Backtracks After Calling Kanye West’...</td>\n",
       "      <td>She didn’t have all the facts. Julia Fox is ta...</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>https://www.usmagazine.com/celebrity-news/news...</td>\n",
       "      <td>[facts, julia, taking, claims, kanye, west, ha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Gifted and at the top of her game -- Ashleigh ...</td>\n",
       "      <td>(CNN)On her day Ashleigh Barty was an unstoppa...</td>\n",
       "      <td>sports</td>\n",
       "      <td>https://edition.cnn.com/2022/03/23/tennis/ashl...</td>\n",
       "      <td>[ashleigh, barty, unstoppable, force, sudden, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Spinosaurus’ dense bones fuel debate over whet...</td>\n",
       "      <td>A fierce group of predatory dinosaurs may have...</td>\n",
       "      <td>science</td>\n",
       "      <td>https://www.sciencenews.org/article/spinosauru...</td>\n",
       "      <td>[fierce, group, predatory, dinosaurs, hunting,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Why did this cultural mecca for Dallas' Black ...</td>\n",
       "      <td>\"Dallas Rising: The Hall of Negro Life\" airs F...</td>\n",
       "      <td>culture</td>\n",
       "      <td>https://www.keranews.org/arts-culture/2022-03-...</td>\n",
       "      <td>[dallas, rising, hall, negro, life, airs, frid...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Forests help reduce global warming in more way...</td>\n",
       "      <td>When it comes to cooling the planet forests ha...</td>\n",
       "      <td>science</td>\n",
       "      <td>https://www.sciencenews.org/article/forest-tre...</td>\n",
       "      <td>[comes, cooling, planet, forests, trick, trees...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Saudi Arabian GP: When to watch practice quali...</td>\n",
       "      <td>Jeddah Corniche street race - which had a chao...</td>\n",
       "      <td>sports</td>\n",
       "      <td>https://www.skysports.com/f1/news/36788/125735...</td>\n",
       "      <td>[jeddah, corniche, street, race, chaotic, debu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>China’s economy posts healthy recovery solid m...</td>\n",
       "      <td>China’s economy started the year on a positive...</td>\n",
       "      <td>economy</td>\n",
       "      <td>https://www.hellenicshippingnews.com/chinas-ec...</td>\n",
       "      <td>[china, economy, started, year, positive, note...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Sophie Turner and Joe Jonas expecting their se...</td>\n",
       "      <td>Actors Joe Jonas and Sophie Turner are expecti...</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>https://indianexpress.com/article/entertainmen...</td>\n",
       "      <td>[actors, jonas, sophie, turner, expecting, sec...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Irish dancers perform in Ramona to raise money...</td>\n",
       "      <td>Irish dancers from the Clan Rince School of Ir...</td>\n",
       "      <td>culture</td>\n",
       "      <td>https://www.sandiegouniontribune.com/ramona-se...</td>\n",
       "      <td>[irish, dancers, clan, rince, school, irish, d...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Michael Madsen Arrested in Malibu For Trespass...</td>\n",
       "      <td>The 64-year-old actor was apprehended at 9 p.m...</td>\n",
       "      <td>entertainment</td>\n",
       "      <td>https://www.eonline.com/news/1320980/michael-m...</td>\n",
       "      <td>[year, actor, apprehended, later, booked, lost...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                title  \\\n",
       "0         Is America’s Economy Entering a New Normal?   \n",
       "1   Mohamed Salah: New Liverpool deal likely to ha...   \n",
       "2             NASA’s exoplanet count surges past 5000   \n",
       "3   Moscow insiders describe panic frustration and...   \n",
       "4   Five 4000 Year Old Painted Tombs Discovered in...   \n",
       "5   Julia Fox Backtracks After Calling Kanye West’...   \n",
       "6   Gifted and at the top of her game -- Ashleigh ...   \n",
       "7   Spinosaurus’ dense bones fuel debate over whet...   \n",
       "8   Why did this cultural mecca for Dallas' Black ...   \n",
       "9   Forests help reduce global warming in more way...   \n",
       "10  Saudi Arabian GP: When to watch practice quali...   \n",
       "11  China’s economy posts healthy recovery solid m...   \n",
       "12  Sophie Turner and Joe Jonas expecting their se...   \n",
       "13  Irish dancers perform in Ramona to raise money...   \n",
       "14  Michael Madsen Arrested in Malibu For Trespass...   \n",
       "\n",
       "                                              content classification  \\\n",
       "0   The pandemic and now the war in Ukraine have a...        economy   \n",
       "1   Mohamed Salah's options would be going \"sidewa...         sports   \n",
       "2   It’s official: The number of planets known bey...        science   \n",
       "3   Yuriy Shatalov (not his real name) an equity t...        economy   \n",
       "4   Five painted tombs were recently unearthed in ...        culture   \n",
       "5   She didn’t have all the facts. Julia Fox is ta...  entertainment   \n",
       "6   (CNN)On her day Ashleigh Barty was an unstoppa...         sports   \n",
       "7   A fierce group of predatory dinosaurs may have...        science   \n",
       "8   \"Dallas Rising: The Hall of Negro Life\" airs F...        culture   \n",
       "9   When it comes to cooling the planet forests ha...        science   \n",
       "10  Jeddah Corniche street race - which had a chao...         sports   \n",
       "11  China’s economy started the year on a positive...        economy   \n",
       "12  Actors Joe Jonas and Sophie Turner are expecti...  entertainment   \n",
       "13  Irish dancers from the Clan Rince School of Ir...        culture   \n",
       "14  The 64-year-old actor was apprehended at 9 p.m...  entertainment   \n",
       "\n",
       "                                              webPage  \\\n",
       "0   https://www.nytimes.com/2022/03/24/business/ec...   \n",
       "1         https://www.bbc.com/sport/football/60856636   \n",
       "2   https://www.sciencenews.org/article/exoplanet-...   \n",
       "3   https://www.businessinsider.com/russian-trader...   \n",
       "4   https://www.artnews.com/art-news/news/saqqara-...   \n",
       "5   https://www.usmagazine.com/celebrity-news/news...   \n",
       "6   https://edition.cnn.com/2022/03/23/tennis/ashl...   \n",
       "7   https://www.sciencenews.org/article/spinosauru...   \n",
       "8   https://www.keranews.org/arts-culture/2022-03-...   \n",
       "9   https://www.sciencenews.org/article/forest-tre...   \n",
       "10  https://www.skysports.com/f1/news/36788/125735...   \n",
       "11  https://www.hellenicshippingnews.com/chinas-ec...   \n",
       "12  https://indianexpress.com/article/entertainmen...   \n",
       "13  https://www.sandiegouniontribune.com/ramona-se...   \n",
       "14  https://www.eonline.com/news/1320980/michael-m...   \n",
       "\n",
       "                                                clean  \n",
       "0   [pandemic, ukraine, altered, america, economy,...  \n",
       "1   [mohamed, salah, options, going, sideways, man...  \n",
       "2   [official, number, planets, known, solar, pass...  \n",
       "3   [yuriy, shatalov, real, equity, trader, moscow...  \n",
       "4   [painted, tombs, recently, unearthed, saqqara,...  \n",
       "5   [facts, julia, taking, claims, kanye, west, ha...  \n",
       "6   [ashleigh, barty, unstoppable, force, sudden, ...  \n",
       "7   [fierce, group, predatory, dinosaurs, hunting,...  \n",
       "8   [dallas, rising, hall, negro, life, airs, frid...  \n",
       "9   [comes, cooling, planet, forests, trick, trees...  \n",
       "10  [jeddah, corniche, street, race, chaotic, debu...  \n",
       "11  [china, economy, started, year, positive, note...  \n",
       "12  [actors, jonas, sophie, turner, expecting, sec...  \n",
       "13  [irish, dancers, clan, rince, school, irish, d...  \n",
       "14  [year, actor, apprehended, later, booked, lost...  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show dataframe with 'clean' column\n",
    "df_ex"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "406e36d9",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add all words from sports category into a 'sports' list\n",
    "sports = df_ex.loc[df_ex['classification'] == 'sports']['clean'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "8a652095",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['mohamed', 'salah', 'options', 'going', 'sideways', 'manchester', 'city', 'left', 'liverpool', 'says', 'reds', 'striker', 'michael', 'owen', 'year', 'egypt', 'forward', 'deal', 'runs', 'summer', 'previously', 'said', 'wanted', 'stay', 'future', 'liverpool', 'hands', 'reds', 'manager', 'jurgen', 'klopp', 'said', 'deal', 'decision', 'surprised', 'salah', 'sign', 'liverpool', 'options', 'moment', 'owen', 'said', 'going', 'sideways', 'manchester', 'city', 'good', 'teams', 'world', 'moment', 'time', 'things', 'happen', 'overnight', 'talking', 'biggest', 'clubs', 'world', 'biggest', 'players', 'world', 'player', 'sign', 'biggest', 'contract', 'life', 'think', 'entirely', 'normal', 'negotiations', 'drag', 'little', 'owen', 'told', 'sport', 'salah', 'joined', 'liverpool', 'italian', 'club', 'roma', 'summer', 'scored', 'goals', 'appearances', 'helped', 'club', 'premier', 'league', 'champions', 'league', 'league', 'fifa', 'club', 'world', 'uefa', 'super', 'time', 'anfield', 'owen', 'scored', 'goals', 'england', 'started', 'career', 'liverpool', 'spells', 'real', 'madrid', 'newcastle', 'united', 'manchester', 'united', 'stoke', 'city', 'speaking', 'launch', 'soccer', 'tournament', 'help', 'british', 'asians', 'showcase', 'talents', 'believes', 'liverpool', 'break', 'wage', 'structure', 'salah', 'players', 'want', 'game', 'disaster', 'case', 'said', 'liverpool', 'business', 'course', 'player', 'want', 'lots', 'players', 'squad', 'knocking', 'jurgen', 'klopp', 'door', 'saying', 'hang', 'minute', 'structure', 'built', 'years', 'fall', 'apart', 'sure', 'liverpool', 'sensible', 'sure', 'salah', 'representatives', 'sensible', 'understand', 'case', 'finding', 'middle', 'ground', 'hopefully', 'everybody', 'soon', 'salah', 'thinking', 'chances', 'success', 'future', 'thinking', 'lines', 'liverpool', 'ticks', 'question', 'tick', 'financially', 'soccer', 'tournament', 'place', 'derby', 'county', 'pride', 'park', 'stadium', 'feature', 'teams', 'london', 'derby', 'birmingham', 'city', 'leicester', 'city', 'british', 'asian', 'players', 'selected', 'regional', 'trials', 'networks', 'premier', 'league', 'player', 'team', 'arsenal', 'liverpool', 'winger', 'jermaine', 'pennant', 'liverpool', 'leicester', 'city', 'striker', 'emile', 'heskey', 'aston', 'villa', 'midfielder', 'hendrie', 'aston', 'villa', 'striker', 'gabby', 'agbonlahor', 'involved', 'scouts', 'number', 'professional', 'clubs', 'invited', 'tournament', 'plans', 'expand', 'event', 'include', 'women', 'tournament', 'ashleigh', 'barty', 'unstoppable', 'force', 'sudden', 'retirement', 'comes', 'shock', 'looked', 'dominate', 'women', 'tennis', 'foreseeable', 'future', 'barty', 'mixture', 'crushing', 'groundstrokes', 'slice', 'backhand', 'near', 'perfect', 'shot', 'times', 'irrepressible', 'apparent', 'january', 'australian', 'open', 'site', 'barty', 'grand', 'slam', 'titles', 'drop', 'tournament', 'home', 'champion', 'time', 'question', 'remained', 'immediately', 'australian', 'open', 'grand', 'slams', 'despite', 'success', 'barty', 'said', 'thinking', 'retirement', 'long', 'time', 'fact', 'admitted', 'decision', 'retire', 'solidified', 'australian', 'open', 'wimbledon', 'victories', 'year', 'wimbledon', 'year', 'changed', 'person', 'athlete', 'barty', 'said', 'retirement', 'video', 'work', 'hard', 'life', 'goal', 'able', 'wimbledon', 'dream', 'true', 'dream', 'wanted', 'tennis', 'changed', 'perspective', 'barty', 'second', 'woman', 'history', 'retire', 'ranked', 'world', 'following', 'justine', 'henin', 'time', 'walked', 'away', 'sport', 'years', 'starting', 'tennis', 'career', 'took', 'month', 'break', 'aged', 'saying', 'time', 'quickly', 'australian', 'showed', 'prodigious', 'talents', 'limited', 'court', 'played', 'cricket', 'professionally', 'brisbane', 'heat', 'women', 'bash', 'league', 'keen', 'golfer', 'apparently', 'good', 'barty', 'women', 'championship', 'brookwater', 'golf', 'club', 'near', 'brisbane', 'time', 'major', 'winner', 'tiger', 'woods', 'reportedly', 'described', 'great', 'swing', 'grand', 'slam', 'champion', 'golf', 'tennis', 'world', 'simona', 'halep', 'asked', 'twitter', 'coronavirus', 'pandemic', 'tennis', 'season', 'presented', 'previously', 'unseen', 'challenges', 'travel', 'restrictions', 'australia', 'meant', 'barty', 'unable', 'return', 'home', 'year', 'barty', 'wimbledon', 'time', 'realized', 'achieving', 'lifelong', 'goal', 'missing', 'feeling', 'wimbledon', 'spoken', 'team', 'said', 'little', 'satisfied', 'fulfilled', 'perspective', 'shift', 'second', 'phase', 'career', 'happiness', 'dependent', 'results', 'doubt', 'presence', 'tennis', 'sorely', 'missed', 'time', 'careers', 'time', 'greats', 'williams', 'sisters', 'roger', 'federer', 'rafael', 'nadal', 'drawing', 'close', 'barty', 'forefront', 'current', 'group', 'stars', 'looking', 'carry', 'sport', 'forward', 'departure', 'opens', 'door', 'number', 'players', 'women', 'game', 'stake', 'claim', 'best', 'world', 'swiatek', 'recent', 'indian', 'wells', 'champion', 'best', 'placed', 'bumped', 'world', 'barty', 'departure', 'year', 'major', 'champion', 'french', 'open', 'polish', 'player', 'history', 'grand', 'slam', 'title', 'barty', 'leaves', 'tennis', 'dominated', 'surface', 'winning', 'wimbledon', 'french', 'open', 'australian', 'open', 'career', 'titles', 'undoubtedly', 'remembered', 'sport', 'gifted', 'players', 'know', 'work', 'takes', 'bring', 'best', 'barty', 'said', 'said', 'team', 'multiple', 'times', 'anymore', 'physical', 'drive', 'emotional', 'want', 'takes', 'challenge', 'level', 'anymore', 'know', 'spent', 'know', 'physically', 'success', 'jeddah', 'corniche', 'street', 'race', 'chaotic', 'debut', 'year', 'hosts', 'second', 'race', 'season', 'ferrari', 'look', 'build', 'strong', 'start', 'watch', 'action', 'live', 'sports', 'qualifying', 'saturday', 'race', 'sunday', 'scintillating', 'opener', 'formula', 'continues', 'pace', 'weekend', 'saudi', 'arabian', 'sport', 'fastest', 'street', 'circuit', 'jeddah', 'corniche', 'race', 'enjoyed', 'dramatic', 'debut', 'calendar', 'year', 'whet', 'appetite', 'weekend', 'charles', 'leclerc', 'ferrari', 'flying', 'start', 'season', 'ahead', 'mercedes', 'bull', 'increases', 'anticipation', 'action', 'weekend', 'friday', 'practice', 'sunday', 'night', 'race', 'live', 'sports', 'looks', 'thrilling', 'team', 'battle', 'year', 'mercedes', 'bull', 'joined', 'usurped', 'ferrari', 'italian', 'giants', 'appear', 'aced', 'rules', 'overhaul', 'leclerc', 'ferrari', 'ahead', 'carlos', 'sainz', 'weekend', 'season', 'opener', 'ending', 'second', 'longest', 'drought', 'scuderia', 'illustrious', 'history', 'races', 'feat', 'enforced', 'ferrari', 'credentials', 'title', 'contenders', 'main', 'rivals', 'plenty', 'motivation', 'bounce', 'weekend', 'bull', 'case', 'pace', 'issue', 'world', 'champion', 'verstappen', 'second', 'running', 'late', 'stages', 'sergio', 'perez', 'contending', 'reliability', 'concerns', 'thanks', 'agonising', 'double', 'closing', 'stages', 'weekend', 'tough', 'lose', 'team', 'come', 'stronger', 'week', 'said', 'verstappen', 'good', 'package', 'competitive', 'bahrain', 'positives', 'weekend', 'long', 'season', 'ahead', 'race', 'mercedes', 'came', 'away', 'strong', 'points', 'haul', 'opener', 'arguably', 'bigger', 'issues', 'fight', 'champions', 'years', 'adrift', 'ferrari', 'bull', 'opener', 'lewis', 'hamilton', 'george', 'russell', 'finished', 'fourth', 'capitalising', 'bull', 'woes', 'mercedes', 'hope', 'unlocked', 'speed', 'admitting', 'time', 'step', 'forward', 'terms', 'performance', 'compared', 'season', 'tests', 'clearly', 'quickest', 'ferrari', 'bull', 'stated', 'team', 'boss', 'toto', 'wolff', 'opening', 'weekend', 'head', 'jeddah', 'different', 'track', 'fast', 'twisty', 'taken', 'throttle', 'months', 'formula', 'race', 'saudi', 'arabia', 'tyres', 'learning', 'steep', 'learning', 'curve', 'challenge', 'looking', 'forward', 'going', 'plenty', 'anticipation', 'second', 'race', 'season', 'drivers', 'teams', 'cars', 'testing', 'opening', 'race', 'barcelona', 'bahrain', 'fact', 'second', 'race', 'jeddah', 'makes', 'exciting', 'specially', 'built', 'layout', 'designed', 'encourage', 'fast', 'wheel', 'wheel', 'racing', 'average', 'speed', 'falls', 'monza', 'terms', 'fastest', 'calendar', 'surrounded', 'barriers', 'little', 'think', 'challenges', 'tight', 'walls', 'monaco', 'singapore', 'baku', 'speed', 'flowing', 'corners', 'track', 'located', 'corniche', 'coastal', 'resort', 'area', 'jeddah', 'alongside', 'second', 'longest', 'corners', 'contend', 'high', 'speed', 'turns', 'year', 'hosted', 'penultimate', 'race', 'verstappen', 'hamilton', 'title', 'battle', 'fair', 'thriller', 'qualifying', 'verstappen', 'crashed', 'bull', 'final', 'race', 'played', 'host', 'flags', 'carnage', 'aplenty', 'hamilton', 'race', 'controversy', 'verstappen', 'came', 'racing', 'bizarre', 'brake', 'test', 'incident', 'dutchman', 'punished', 'previewing', 'weekend', 'verstappen', 'said', 'looking', 'ahead', 'saudi', 'track', 'year', 'humidity', 'challenging', 'small', 'adjustments', 'visibility', 'sector', 'curious', 'going', 'time', 'quick', 'track', 'high', 'speed', 'straights', 'year', 'cars', 'slightly', 'heavier', 'going', 'interesting', 'perform', 'looking', 'forward', 'enjoyable', 'sports', 'home', 'race', 'season', 'continues', 'saudi', 'arabian', 'grand', 'prix', 'weekend', 'action', 'friday', 'practice', 'sessions', 'saturday', 'final', 'practice', 'qualifying', 'sunday', 'grand', 'prix', 'live', 'channel', 'follow', 'action', 'online', 'sports', 'jenson', 'button', 'paul', 'resta', 'anthony', 'davidson', 'johnny', 'herbert', 'david', 'croft', 'simon', 'lazenby', 'natalie', 'pinkham', 'jeddah', 'bringing', 'expert', 'insight', 'analysis']\n"
     ]
    }
   ],
   "source": [
    "# View the 'sports' list before counting its most common words\n",
    "print(sports)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0372e5fc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['race', 'barty', 'time', 'liverpool', 'year']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply count function to list 'sports'\n",
    "count(sports,5)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}