notebook_btc.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7f931490",
   "metadata": {
    "id": "8031b7fc"
   },
   "source": [
    "## Web-Crawling Twitter Data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "048ebb4b",
   "metadata": {
    "id": "6ed56d7b"
   },
   "source": [
    "### Instructions\n",
    "----------------\n",
    "**Please Do not Reapeat the Steps In This Section**\n",
    "\n",
    "we select the data we will use for the analysis later on.\n",
    "We have chosen the following datasets for our analysis:\n",
    "- Bitcoin Twitter chatter dataset\n",
    "  > We webcrawl this data Ourselves and use it only for the purposes of attempting to predict bitcoin price according to the sentiment of the tweets.\n",
    "  \n",
    "  \n",
    "### Research Question \n",
    "We chose to investigate how the price of Bitcoin may be affected by twitter sentiments about the currency based on a sentiment analysis model trained on the UCC corpus and a final prediction model based on the sentiment model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dce7793b",
   "metadata": {
    "id": "87a8f8ed"
   },
   "source": [
    "## Preprocessing \n",
    "-------------------------\n",
    "\n",
    "> For our project, we perform a sentiment analysis on tweets related to crypto currencies and use this analysis to predict how the currencies will varry depending on the sentiment. \n",
    "\n",
    "> Since we are only interested in tweets that are related to Bitcoin, we will specify a filter then filter out the tweets that do not contain the words in the filter.\n",
    "\n",
    ">After that, we perform a sentiment analysis using pre trained models to see whether we can accurately predict what the sentiment of the tweets are.\n",
    "\n",
    ">The models used will be trained on the UCC(The Unhealthy Comments Corpus) Coprus that was mentioned before , which contains over 40,000 online comments which have been tagged with sentiment values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "c14abcb5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "P9TJkifveqT1",
    "outputId": "d4d579b9-be3f-4454-cddd-45f79efbdc24"
   },
   "outputs": [],
   "source": [
    "#import sys\n",
    "#!{sys.executable} -m pip install tweepy==4.10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "cfe77c5a",
   "metadata": {
    "id": "45e7b924"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'/home/jelke/Documents/UD/teach/TextMining/ngare'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import tweepy as tw \n",
    "from tqdm import tqdm\n",
    "from IPython.display import clear_output\n",
    "import time\n",
    "import os\n",
    "from dateutil import parser\n",
    "\n",
    "os.getcwd()\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "bca088fb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "b8d5234a",
    "outputId": "ffa18fc9-7a27-4be6-df68-1e3540c51e28"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['conf.conf']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import configparser\n",
    "config = configparser.ConfigParser(interpolation=None)\n",
    "config.read(\"conf.conf\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "69e1800e",
   "metadata": {
    "id": "128cdd73"
   },
   "outputs": [],
   "source": [
    "client = tw.Client(config['twitter']['bearer_token'], wait_on_rate_limit=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "13dcb6a9",
   "metadata": {
    "id": "67e5754c"
   },
   "outputs": [],
   "source": [
    "date_since = parser.parse(\"2021-08-31T23:00:00.00Z\")\n",
    "date_until= parser.parse(\"2021-09-01T00:00:00Z\")\n",
    "search_words= (\"Bitcoin lang:en -is:retweet\"or\"bitcoin lang:en -is:retweet\"or\n",
    "               \"Btc lang:en -is:retweet\"or\"btc lang:en -is:retweet\"or\n",
    "               \"#bitcoin lang:en -is:retweet\"or\"#Btc lang:en -is:retweet\"or\n",
    "               \"#btc lang:en -is:retweet\")\n",
    "fields=['created_at','text']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "5f1d6b92-ff55-4f15-bc0b-c68883da82db",
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets_df=pd.DataFrame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "f7a88d78",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c593fbe7",
    "outputId": "ca12294a-065b-49af-f5b6-cefd7f174ba1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "20160\n"
     ]
    }
   ],
   "source": [
    "# Collect tweets\n",
    "import datetime as dt\n",
    "i=0\n",
    "tweets_copy = []\n",
    "\n",
    "while i in range(2016):\n",
    "    date_since = date_since + dt.timedelta(hours=1)\n",
    "    date_until = date_until + dt.timedelta(hours=1)\n",
    "    tweets = tw.Paginator(client.search_all_tweets,  \n",
    "                      tweet_fields=fields,\n",
    "                      query=search_words,\n",
    "                      start_time=date_since,\n",
    "                      end_time=date_until,\n",
    "                      max_results=10).flatten(limit=10) #We instruct the Paginator to return maximum of 20,160 tweets. 10 each hour\n",
    "    #Tweet retrival\n",
    "    for tweet in tqdm(tweets):\n",
    "        time.sleep(1)\n",
    "        tweets_copy.append(tweet)\n",
    "        tweets_df=tweets_df.append(pd.DataFrame({'date': tweet.created_at,\n",
    "                                               'text': tweet.text}, index=[0]))\n",
    "    clear_output()  \n",
    "    i=i+1\n",
    "    print(len(tweets_df))\n",
    "    if i % 100 == 0:\n",
    "        pd.DataFrame(tweets_df).to_csv(\"data/Btc_tweets_1-31_unprocessed\" + str(i) + \".csv\", line_terminator='\\n')\n",
    "\n",
    "pd.DataFrame(tweets_df).to_csv(\"data/Btc_tweets_1-31_unprocessed.csv\", line_terminator='\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "756b0459",
   "metadata": {
    "id": "27ec12ea"
   },
   "source": [
    "### Checking we have received the desired number of Tweets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "bc4687ff",
   "metadata": {
    "id": "abb4fde5",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New tweets retrieved: 20160\n"
     ]
    }
   ],
   "source": [
    "print(f\"New tweets retrieved: {len(tweets_copy)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "e09ae3da",
   "metadata": {
    "id": "56f51d36"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:58+00:00</td>\n",
       "      <td>Cyber-thieves used malware to swipe 16.4 Bitco...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:53+00:00</td>\n",
       "      <td>Our CEO and Co-Founder @raypaxful believes tha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:49+00:00</td>\n",
       "      <td>@Dennis_Porter_ I agree. Although loss of hope...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:46+00:00</td>\n",
       "      <td>What a fast growing ecosystem the XRP ledger h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:37+00:00</td>\n",
       "      <td>@finance_keep I have participated. I believe t...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       date                                               text\n",
       "0 2021-09-01 00:59:58+00:00  Cyber-thieves used malware to swipe 16.4 Bitco...\n",
       "0 2021-09-01 00:59:53+00:00  Our CEO and Co-Founder @raypaxful believes tha...\n",
       "0 2021-09-01 00:59:49+00:00  @Dennis_Porter_ I agree. Although loss of hope...\n",
       "0 2021-09-01 00:59:46+00:00  What a fast growing ecosystem the XRP ledger h...\n",
       "0 2021-09-01 00:59:37+00:00  @finance_keep I have participated. I believe t..."
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tweets_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a038ed68",
   "metadata": {
    "id": "61179c22"
   },
   "source": [
    "###  Filltering and Lemmatizing Tweets\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "3e77caf2",
   "metadata": {
    "id": "244324be"
   },
   "outputs": [],
   "source": [
    "import re \n",
    "\n",
    "filtered_btc = tweets_df.dropna()\n",
    "\n",
    "# provide case insensitive data\n",
    "filtered_btc[\"text\"]=filtered_btc[\"text\"].str.lower().astype(str)\n",
    "\n",
    "# Take out links with or without www\n",
    "filtered_btc[\"text\"] = filtered_btc[\"text\"].apply(lambda x: re.sub(r'https?:\\/\\/\\S+', '', x))\n",
    "filtered_btc[\"text\"].apply(lambda x: re.sub(r\"www\\.[a-z]?\\.?(com)+|[a-z]+\\.(com)\", '', x))\n",
    "\n",
    "#Take out possible HTML character references \n",
    "filtered_btc[\"text\"] = filtered_btc[\"text\"].apply(lambda x: re.sub(r'&[a-z]+;', '', x))\n",
    "\n",
    "#Take out nonletter characters except for spaces and sentence delimitators\n",
    "filtered_btc[\"text\"] = filtered_btc[\"text\"].apply(lambda x: re.sub(r\"[^a-z\\s.!?]\", '', x))\n",
    "\n",
    "#Sometimes twitter data has links preprocessed into a reference such as {link}\n",
    "filtered_btc[\"text\"] = filtered_btc[\"text\"].apply(lambda x: re.sub(r'{link}', '', x))\n",
    "\n",
    "# I noticed the dataset contains at user and url references so we can remove them\n",
    "\n",
    "filtered_btc[\"text\"]= filtered_btc[\"text\"].str.replace('url', '')\n",
    "filtered_btc[\"text\"]= filtered_btc[\"text\"].str.replace('atuser', '')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "3e4da88a",
   "metadata": {
    "id": "5f231e8c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:58+00:00</td>\n",
       "      <td>cyberthieves used malware to swipe . bitcoin f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:53+00:00</td>\n",
       "      <td>our ceo and cofounder raypaxful believes that ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:49+00:00</td>\n",
       "      <td>dennisporter i agree. although loss of hope ca...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:46+00:00</td>\n",
       "      <td>what a fast growing ecosystem the xrp ledger h...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2021-09-01 00:59:37+00:00</td>\n",
       "      <td>financekeep i have participated. i believe thi...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       date                                               text\n",
       "0 2021-09-01 00:59:58+00:00  cyberthieves used malware to swipe . bitcoin f...\n",
       "0 2021-09-01 00:59:53+00:00  our ceo and cofounder raypaxful believes that ...\n",
       "0 2021-09-01 00:59:49+00:00  dennisporter i agree. although loss of hope ca...\n",
       "0 2021-09-01 00:59:46+00:00  what a fast growing ecosystem the xrp ledger h...\n",
       "0 2021-09-01 00:59:37+00:00  financekeep i have participated. i believe thi..."
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "filtered_btc.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "51f4a895-0c75-4019-825f-67bad8d5da4f",
   "metadata": {},
   "outputs": [],
   "source": [
    "#import nltk\n",
    "#nltk.download('averaged_perceptron_tagger')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "b16add4e",
   "metadata": {
    "id": "4ad26ad6"
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.tokenize import TweetTokenizer\n",
    "\n",
    "tweets = list(zip(filtered_btc[\"text\"], filtered_btc[\"date\"]))\n",
    "\n",
    "tweet_tokenizer = TweetTokenizer(preserve_case=True, reduce_len=False, strip_handles=False)\n",
    "\n",
    "tokens = [(tweet_tokenizer.tokenize(tweet), date) for (tweet, date) in tweets if type(tweet) == str]\n",
    "\n",
    "filtered = []\n",
    "for tweet in tokens:\n",
    "    new = []\n",
    "    for tok in tweet[0]:\n",
    "        if tok != \"AT_USER\" and tok != \"URL\":\n",
    "            new.append(tok)\n",
    "            \n",
    "    filtered.append((new, tweet[1]))\n",
    "\n",
    "tagged = [(nltk.pos_tag(tweet), date) for tweet, date in filtered]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "82237d24",
   "metadata": {
    "id": "0c0be26c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20160"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import string\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.corpus import wordnet as wn\n",
    "\n",
    "def wn_pos(tag):\n",
    "    \"converts treebank tags into wordbank tags for lemmatization\"\n",
    "    if tag.startswith('J'):\n",
    "        return wn.ADJ\n",
    "    if tag.startswith('V'):\n",
    "        return wn.VERB\n",
    "    if tag.startswith('N'):\n",
    "        return wn.NOUN\n",
    "    if tag.startswith('R'):\n",
    "        return wn.ADV\n",
    "    return None\n",
    "\n",
    "lem_tweets = []\n",
    "lem = WordNetLemmatizer()\n",
    "\n",
    "for tweet in tagged:\n",
    "    lemmas = []\n",
    "    \n",
    "    for word, tag in tweet[0]:\n",
    "        wn_tag = wn_pos(tag)\n",
    "        \n",
    "        if word[-1] in string.punctuation:\n",
    "                word = word[:-1]\n",
    "\n",
    "        if wn_pos(tag) is not None:\n",
    "            lemmas.append(lem.lemmatize(word, wn_tag))\n",
    "        else:\n",
    "            lemmas.append(lem.lemmatize(word))\n",
    "                \n",
    "    lem_tweets.append((lemmas, tweet[1]))\n",
    "\n",
    "lemmas = [lem for tweet in lem_tweets for lem in tweet]\n",
    "\n",
    "len(lem_tweets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "8eaa01d0",
   "metadata": {
    "id": "37d1a2a7"
   },
   "outputs": [],
   "source": [
    "lem_tweets = pd.DataFrame(lem_tweets, columns =['tweet', 'date'])  \n",
    "pd.DataFrame(lem_tweets).to_csv(\"data/Btc_tweets_1-31.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f086c6b6",
   "metadata": {
    "id": "e12ca06b"
   },
   "source": [
    "## Bitcoin Daily Price Data Pre-Processing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4cf903d1",
   "metadata": {
    "id": "8017475a"
   },
   "source": [
    "## Analysis\n",
    "------------------\n",
    "\n",
    "We begin the Analysis by splitting the data by day which will will then run through the model and create a daily report on based on the results.\n",
    "- First, we combine the data that will be used in the model to a dataframe with the data from the past 7 days.\n",
    "- Then we will append to the same data frame the average change in the price of Bitcoin per day, caluctaed as a difference between the opening price at midnight and the closing price at midnight 24 hours later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "5304b8a4",
   "metadata": {
    "id": "2a0f655f"
   },
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "cfc3ade2",
   "metadata": {
    "id": "8e2b4613"
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'data/Btc_tweets_1-31'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m/tmp/ipykernel_22300/2236066990.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mdf\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconcat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_csv\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mf'data/Btc_tweets_1-31/{f}'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mf\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mos\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlistdir\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'data/Btc_tweets_1-31'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mendswith\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'.csv'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'data/Btc_tweets_1-31'"
     ]
    }
   ],
   "source": [
    "df = pd.concat([pd.read_csv(f'data/Btc_tweets_1-31/{f}') for f in os.listdir('data/Btc_tweets_1-31') if f.endswith('.csv')])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bff0b6f",
   "metadata": {
    "id": "7ac2a56e"
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f06354df",
   "metadata": {
    "id": "32dabcaa"
   },
   "outputs": [],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3bdc5533",
   "metadata": {
    "id": "ca89cfe5"
   },
   "outputs": [],
   "source": [
    "pd.DataFrame(df).to_csv(\"Btc_tweets_1-31.csv\")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "notebook-ngare.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}