web scarabing1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Task 1\n",
    "\n",
    "---\n",
    "\n",
    "## Web scraping and analysis\n",
    "\n",
    "This Jupyter notebook includes some code to get you started with web scraping. We will use a package called `BeautifulSoup` to collect the data from the web. Once you've collected your data and saved it into a local `.csv` file you should start with your analysis.\n",
    "\n",
    "### Scraping data from Skytrax\n",
    "\n",
    "If you visit [https://www.airlinequality.com] you can see that there is a lot of data there. For this task, we are only interested in reviews related to British Airways and the Airline itself.\n",
    "\n",
    "If you navigate to this link: [https://www.airlinequality.com/airline-reviews/british-airways] you will see this data. Now, we can use `Python` and `BeautifulSoup` to collect all the links to the reviews and then to collect the text data on each of the individual review links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping page 1\n",
      "   ---> 200 total reviews\n",
      "Scraping page 2\n",
      "   ---> 400 total reviews\n",
      "Scraping page 3\n",
      "   ---> 600 total reviews\n",
      "Scraping page 4\n",
      "   ---> 800 total reviews\n",
      "Scraping page 5\n",
      "   ---> 1000 total reviews\n",
      "Scraping page 6\n",
      "   ---> 1200 total reviews\n",
      "Scraping page 7\n",
      "   ---> 1400 total reviews\n",
      "Scraping page 8\n",
      "   ---> 1600 total reviews\n",
      "Scraping page 9\n",
      "   ---> 1800 total reviews\n",
      "Scraping page 10\n",
      "   ---> 2000 total reviews\n"
     ]
    }
   ],
   "source": [
    "base_url = \"https://www.airlinequality.com/airline-reviews/british-airways\"\n",
    "pages = 10\n",
    "page_size = 200\n",
    "\n",
    "reviews = []\n",
    "\n",
    "\n",
    "# for i in range(1, pages + 1):\n",
    "for i in range(1, pages + 1):\n",
    "\n",
    "    print(f\"Scraping page {i}\")\n",
    "\n",
    "    # Create URL to collect links from paginated data\n",
    "    url = f\"{base_url}/page/{i}/?sortby=post_date%3ADesc&pagesize={page_size}\"\n",
    "\n",
    "    # Collect HTML data from this page\n",
    "    response = requests.get(url)\n",
    "\n",
    "    # Parse content\n",
    "    content = response.content\n",
    "    parsed_content = BeautifulSoup(content, 'html.parser')\n",
    "    for para in parsed_content.find_all(\"div\", {\"class\": \"text_content\"}):\n",
    "        reviews.append(para.get_text())\n",
    "    \n",
    "    print(f\"   ---> {len(reviews)} total reviews\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reviews</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>✅ Trip Verified |  Very competent check in sta...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>✅ Trip Verified |  Check in was so slow, no se...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>✅ Trip Verified |  My review relates to the ap...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>✅ Trip Verified | This was my first time flyin...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>✅ Trip Verified |  Lots of cancellations and d...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             reviews\n",
       "0  ✅ Trip Verified |  Very competent check in sta...\n",
       "1  ✅ Trip Verified |  Check in was so slow, no se...\n",
       "2  ✅ Trip Verified |  My review relates to the ap...\n",
       "3  ✅ Trip Verified | This was my first time flyin...\n",
       "4  ✅ Trip Verified |  Lots of cancellations and d..."
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame()\n",
    "df[\"reviews\"] = reviews\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"data/BA_reviews.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! Now you have your dataset for this task! The loops above collected 1000 reviews by iterating through the paginated pages on the website. However, if you want to collect more data, try increasing the number of pages!\n",
    "\n",
    " The next thing that you should do is clean this data to remove any unnecessary text from each of the rows. For example, \"✅ Trip Verified\" can be removed from each row if it exists, as it's not relevant to what we want to investigate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Cleaning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0       ✅ Trip Verified |  Very competent check in sta...\n",
      "1       ✅ Trip Verified |  Check in was so slow, no se...\n",
      "2       ✅ Trip Verified |  My review relates to the ap...\n",
      "3       ✅ Trip Verified | This was my first time flyin...\n",
      "4       ✅ Trip Verified |  Lots of cancellations and d...\n",
      "                              ...                        \n",
      "1995    ✅ Verified Review |  For those who have allude...\n",
      "1996    British Airways have randomly cancelled a flig...\n",
      "1997    ✅ Verified Review |  Domestic BA from London a...\n",
      "1998    Las Vegas to London Heathrow return, and we di...\n",
      "1999    My wife and I flew to Dublin from London Heath...\n",
      "Name: reviews, Length: 2000, dtype: object\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "csv_path = \"data/BA_reviews.csv\"\n",
    "df1 = pd.read_csv(csv_path)\n",
    "df1.reset_index(drop=True, inplace=True)\n",
    "\n",
    "print(df1['reviews'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 2000 entries, 0 to 1999\n",
      "Data columns (total 2 columns):\n",
      " #   Column      Non-Null Count  Dtype \n",
      "---  ------      --------------  ----- \n",
      " 0   Unnamed: 0  2000 non-null   int64 \n",
      " 1   reviews     2000 non-null   object\n",
      "dtypes: int64(1), object(1)\n",
      "memory usage: 31.4+ KB\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>2000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>999.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>577.494589</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>499.750000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>999.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1499.250000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1999.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Unnamed: 0\n",
       "count  2000.000000\n",
       "mean    999.500000\n",
       "std     577.494589\n",
       "min       0.000000\n",
       "25%     499.750000\n",
       "50%     999.500000\n",
       "75%    1499.250000\n",
       "max    1999.000000"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df1.info()\n",
    "df1.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "      Unnamed: 0                                            reviews\n",
      "0              0  y competent check in staff, saw had a problem ...\n",
      "1              1  check in was so slow, no self check in and bag...\n",
      "2              2  my review relates to the appalling experiences...\n",
      "3              3  his was my first time flying with ba & i was p...\n",
      "4              4  lots of cancellations and delays and no one ap...\n",
      "...          ...                                                ...\n",
      "1995        1995  review |  for those who have alluded to there ...\n",
      "1996        1996  british airways have randomly cancelled a flig...\n",
      "1997        1997  review |  domestic ba from london and edinburg...\n",
      "1998        1998  las vegas to london heathrow return, and we di...\n",
      "1999        1999  my wife and i flew to dublin from london heath...\n",
      "\n",
      "[2000 rows x 2 columns]\n"
     ]
    }
   ],
   "source": [
    "df1['reviews'] = df1['reviews'].str.strip()\n",
    "df1['reviews']=df1['reviews'].str.lstrip('✅ Trip Verified |')\n",
    "df1['reviews']=df1['reviews'].str.lstrip('Not Verified |')\n",
    "df1['reviews']= df1['reviews'].str.lower()\n",
    "print(df1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0       y competent check in staff saw had a problem w...\n",
      "1       check in was so slow no self check in and bag ...\n",
      "2       my review relates to the appalling experiences...\n",
      "3       his was my first time flying with ba  i was pl...\n",
      "4       lots of cancellations and delays and no one ap...\n",
      "                              ...                        \n",
      "1995    review   for those who have alluded to there b...\n",
      "1996    british airways have randomly cancelled a flig...\n",
      "1997    review   domestic ba from london and edinburgh...\n",
      "1998    las vegas to london heathrow return and we did...\n",
      "1999    my wife and i flew to dublin from london heath...\n",
      "Name: reviews, Length: 2000, dtype: object\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "<ipython-input-4-05b35ac04ac2>:2: FutureWarning: The default value of regex will change from True to False in a future version.\n",
      "  df1['reviews'] = df1['reviews'].str.replace('[^\\w\\s]','')\n"
     ]
    }
   ],
   "source": [
    "#remove punctuation\n",
    "df1['reviews'] = df1['reviews'].str.replace('[^\\w\\s]','')\n",
    "print(df1['reviews'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>reviews</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>y competent check in staff saw had a problem w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>check in was so slow no self check in and bag ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>my review relates to the appalling experiences...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>his was my first time flying with ba  i was pl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>lots of cancellations and delays and no one ap...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                             reviews\n",
       "0  y competent check in staff saw had a problem w...\n",
       "1  check in was so slow no self check in and bag ...\n",
       "2  my review relates to the appalling experiences...\n",
       "3  his was my first time flying with ba  i was pl...\n",
       "4  lots of cancellations and delays and no one ap..."
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "del df1['Unnamed: 0']\n",
    "df1.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "df1.to_csv(\"data/BA_reviews_cleaning.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Text preprocessing functions\n",
    "1. U have to make sure there are:\n",
    "2. No useless text data.\n",
    "3. No Uppercase letters (turn all letters to lowercase).\n",
    "4. No Punctuations.\n",
    "4. Tokenization and stop words handling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import string   # we need it for Punctuations removal\n",
    "from stop_words import get_stop_words   # or we can use from nltk.corpus import stopwords \n",
    "from nltk.tokenize import word_tokenize # it turn text to list but more faster \n",
    "\n",
    "# preprocessing function for sentiment analysis\n",
    "def sentiment_clean_text(text):\n",
    "    '''\n",
    "    this function take text and clean it  \n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    text : string before preprocessing.\n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    text : string after preprocessing.\n",
    "\n",
    "    '''\n",
    "\n",
    "\n",
    "    # A. first step remove useless text data (if there are any) note: i don't need data before | so I will remove it \n",
    "    if '|' in text:\n",
    "        text =  text.split('|')[1]   \n",
    "       \n",
    "    # B. second turn letters into lowercase \n",
    "    text = text.lower()\n",
    "        \n",
    "    # C. third remove all Punctuations.\n",
    "    text = text.translate(str.maketrans('','',string.punctuation))\n",
    "    \n",
    "    return text\n",
    "\n",
    "\n",
    "# preprocessing function for emotion analysis\n",
    "def emotion_clean_text(text):\n",
    "    '''\n",
    "    this function take text and clean it then turn it to list of words \n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    text : string \n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    text_list : list of text words after cleaning.\n",
    "\n",
    "    '''\n",
    "        \n",
    "    # D. forth step Tokenization and stop words \n",
    "        \n",
    "    # Tokenizaiton: turning string into list of words.\n",
    "    # Stop words: words without meaning for sentiment analysis.\n",
    "\n",
    "        \n",
    "    # Tokenizaiton\n",
    "    text = word_tokenize(text,\"english\")\n",
    "    \n",
    "    # handeling the stop words but what are the stop words \n",
    "    stop_words = get_stop_words('english') #or we can use stop_words = stopwords.words('english')\n",
    "\n",
    "    # Removing stop words from the tokenized words list\n",
    "    text_list = []\n",
    "    \n",
    "    for word in text:\n",
    "        if word not in stop_words:\n",
    "            text_list.append(word)\n",
    "\n",
    "       \n",
    "    # return the list of words\n",
    "    return text_list\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Emotion text dectionary function\n",
    "- now it's time to make dictionary function for emotions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def emotion_maping (file,di): \n",
    "    '''\n",
    "    this function take emotions file and store emotions in dictionary \n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    file : emotions file  \n",
    "\n",
    "    Returns\n",
    "    -------\n",
    "    di : emotions dictionary.\n",
    "    \n",
    "    '''\n",
    "    for line in file:\n",
    "        clear_line = line.replace(\"\\n\", '').replace(\",\", '').replace(\"'\", '').strip()\n",
    "        word, emotion = clear_line.split(':')\n",
    "        di[word] = emotion\n",
    "            \n",
    "    return di"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vader Sentiment analysis function\n",
    "- Now we have cleaned data so we are ready to do sentiment analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's import the needed packages \n",
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "\n",
    "# function to carry out the     \n",
    "def sentiment_analyze(text):\n",
    "    \n",
    "    scores = SentimentIntensityAnalyzer().polarity_scores(text) # return dictionary of scores\n",
    "    \n",
    "    if (scores['neg'] > scores['pos']):\n",
    "        \n",
    "        return 0\n",
    "    \n",
    "    else:\n",
    "        return 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Main Section\n",
    "- apply functions to the dataset texts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package vader_lexicon to\n",
      "[nltk_data]     C:\\Users\\agusa\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package vader_lexicon is already up-to-date!\n"
     ]
    },
    {
     "ename": "NameError",
     "evalue": "name 'sentiment_clean_text' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-9-203c21795b29>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m     31\u001b[0m     \u001b[1;31m# step 1: let's clean the text and assign cleaned list to dataFrame\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     32\u001b[0m     \u001b[1;31m# simple clean\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m---> 33\u001b[1;33m     \u001b[0mcleaned_text\u001b[0m\u001b[1;33m=\u001b[0m \u001b[0msentiment_clean_text\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mtext\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m     34\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m     35\u001b[0m     \u001b[1;31m#Step 2: sentiment Analysis\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mNameError\u001b[0m: name 'sentiment_clean_text' is not defined"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "nltk.download('vader_lexicon')\n",
    "import opendatasets as od\n",
    "# define needed data structures\n",
    "\n",
    "cleaned_text = \"\"\n",
    "temp_emotion_list = []\n",
    "score = 0\n",
    "emotion_dict = {}\n",
    "words_score_dict = {}\n",
    "moods_list_st = []\n",
    "moods_list_tp = []\n",
    "\n",
    "# create category list for better understanding \n",
    "airline_main_categories = ['flight','service','seat','food','crew','time','good','class','cabin','seats','staff','business']\n",
    "temp_category_list = []\n",
    "\n",
    "# get the emotion dictionary ready\n",
    "emotion_file = open('data/emotion.txt','r',encoding='utf-8') \n",
    "emotion_dict = emotion_maping(emotion_file,emotion_dict)\n",
    "emotion_file.close()\n",
    "\n",
    "# ------------------------ loop for the skytrx Dataframe ------------------------------\n",
    "\n",
    "# loop for all reviews in Skytrax dataFrame  \n",
    "for i in range(len(df1)):\n",
    "    \n",
    "    # get the review of index i\n",
    "    text = str(df1['reviews'][i])\n",
    "    \n",
    "    # step 1: let's clean the text and assign cleaned list to dataFrame \n",
    "    # simple clean \n",
    "    cleaned_text= sentiment_clean_text(text)\n",
    "    \n",
    "    #Step 2: sentiment Analysis\n",
    "    score = sentiment_analyze(cleaned_text)\n",
    "    moods_list_st.append(score)\n",
    "\n",
    "    \n",
    "    # Step 3: advanced clean for emotions\n",
    "    cleaned_text_list = emotion_clean_text(cleaned_text)\n",
    "    df_st['reviews'][i] = cleaned_text_list\n",
    "    \n",
    "\n",
    "    # Step 4: emotion list builder\n",
    "    for word in emotion_dict.keys():\n",
    "        if word in cleaned_text_list:\n",
    "            temp_emotion_list.append(emotion_dict[word])   \n",
    "    \n",
    "    # Step 5: category list builder\n",
    "    for cat in airline_main_categories:\n",
    "        if cat in cleaned_text_list:\n",
    "            temp_category_list.append(cat)  \n",
    "\n",
    "# now let's create new column for moods for skytrax\n",
    "df_st['mood'] = moods_list_st\n",
    "\n",
    "\n",
    "# ------------------------ loop for the trustpilot Dataframe ------------------------------\n",
    "\n",
    "# loop for all reviews in trustpilot dataFrame  \n",
    "for i in range(len(df_tp)):\n",
    "    \n",
    "    # get the review of index i\n",
    "    text = str(df_tp['reviews'][i])\n",
    "    \n",
    "    # step 1: let's clean the text and assign cleaned list to dataFrame \n",
    "    # simple clean \n",
    "    cleaned_text= sentiment_clean_text(text)\n",
    "    \n",
    "    #Step 2: sentiment Analysis\n",
    "    score = sentiment_analyze(cleaned_text)\n",
    "    moods_list_tp.append(score)\n",
    "\n",
    "    \n",
    "    # Step 3: advanced clean for emotions\n",
    "    cleaned_text_list = emotion_clean_text(cleaned_text)\n",
    "    df_tp['reviews'][i] = cleaned_text_list\n",
    "    \n",
    "\n",
    "    # Step 4: emotion list builder\n",
    "    for word in emotion_dict.keys():\n",
    "        if word in cleaned_text_list:\n",
    "            temp_emotion_list.append(emotion_dict[word])  \n",
    "\n",
    "    # Step 5: category list builder\n",
    "    for cat in airline_main_categories:\n",
    "        if cat in cleaned_text_list:\n",
    "            temp_category_list.append(cat)  \n",
    "            \n",
    "# now let's create new column for moods trust pilot dataframe\n",
    "df_tp['mood'] = moods_list_tp\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "vscode": {
   "interpreter": {
    "hash": "4f7924c4c56b083e0e50eadfe7ef592a7a8ef70df33a0047f82280e6be1afe15"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}