notebooks/Reddit Data Collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reddit Flair Detector\n",
    "\n",
    "---\n",
    "\n",
    "## Part I - Reddit Data Collection\n",
    "\n",
    "### 1) Import required modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import pandas as pd\n",
    "import praw"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2) Scrape data from r/india\n",
    "\n",
    "* We use the Python Reddit API Wrapper (PRAW) to scrape data\n",
    "* ```client_id``` , ```client_secret``` , ```user_agent``` can be obtained from the Reddit Apps website detailed in References [[1]](../README.md/#References)\n",
    "* Initialize required variables and list of flairs on r/india (As of 10th April 2020)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "reddit = praw.Reddit(client_id='#', client_secret='#', user_agent='#')\n",
    "\n",
    "flairs = [\"Scheduled\", \"Politics\", \"Photography\", \"Coronavirus\", \"Policy/Economy\", \"AskIndia\", \"Sports\", \"Non-Political\",  \"Science/Technology\", \"Food\", \"Business/Finance\"]\n",
    "\n",
    "subreddit = reddit.subreddit('India')\n",
    "topics_dict = { \"flair\":[],\n",
    "                \"title\":[], \n",
    "                \"score\":[], \n",
    "                \"id\":[], \"url\":[], \n",
    "                \"comms_num\": [], \n",
    "                \"author\": [], \n",
    "                \"body\":[]}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3) Get data\n",
    "\n",
    "* We'll scrape atmost 300 top posts for each flair"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "for flair in flairs:\n",
    "    individual_flairs = subreddit.search(flair, limit=300, sort='top')\n",
    "    for subs in individual_flairs:\n",
    "        topics_dict[\"flair\"].append(flair)\n",
    "        topics_dict[\"title\"].append(subs.title)\n",
    "        topics_dict[\"score\"].append(subs.score)\n",
    "        topics_dict[\"id\"].append(subs.id)\n",
    "        topics_dict[\"url\"].append(subs.url)\n",
    "        topics_dict[\"comms_num\"].append(subs.num_comments)\n",
    "        topics_dict[\"author\"].append(subs.author)\n",
    "        topics_dict[\"body\"].append(subs.selftext)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4) Save raw data to CSV file\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "topics_data = pd.DataFrame(topics_dict,columns=[\"flair\",\"title\",\"score\",\"id\",\"author\",\"comms_num\",\"body\",\"url\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>flair</th>\n",
       "      <th>title</th>\n",
       "      <th>score</th>\n",
       "      <th>id</th>\n",
       "      <th>author</th>\n",
       "      <th>comms_num</th>\n",
       "      <th>body</th>\n",
       "      <th>url</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Scheduled</td>\n",
       "      <td>Untouchability, even in quarantine. 'We have n...</td>\n",
       "      <td>52</td>\n",
       "      <td>fzvwz8</td>\n",
       "      <td>sam-sepiol</td>\n",
       "      <td>7</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://www.telegraphindia.com/india/coronavir...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Scheduled</td>\n",
       "      <td>Delhi Govt Sources: Names of CM Arvind Kejriwa...</td>\n",
       "      <td>301</td>\n",
       "      <td>f7ogd8</td>\n",
       "      <td>The_andh_bhakth</td>\n",
       "      <td>30</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://twitter.com/ANI/status/123109390051893...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Scheduled</td>\n",
       "      <td>Delhi: AP Singh, advocate of 2012 Delhi gang-r...</td>\n",
       "      <td>17</td>\n",
       "      <td>flgvah</td>\n",
       "      <td>devCR7</td>\n",
       "      <td>21</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://twitter.com/ani/status/124073128907587...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Scheduled</td>\n",
       "      <td>Why the Supreme Court’s verdict on SC/ST quota...</td>\n",
       "      <td>110</td>\n",
       "      <td>f1o839</td>\n",
       "      <td>rahulthewall</td>\n",
       "      <td>47</td>\n",
       "      <td>NaN</td>\n",
       "      <td>https://scroll.in/article/952687/why-the-supre...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Scheduled</td>\n",
       "      <td>What about the entrance exams scheduled in May?</td>\n",
       "      <td>8</td>\n",
       "      <td>fvcvo1</td>\n",
       "      <td>plsacceptmyusername</td>\n",
       "      <td>3</td>\n",
       "      <td>CLAT, AILET, NEET, JEE have been postponed bu...</td>\n",
       "      <td>https://www.reddit.com/r/india/comments/fvcvo1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       flair                                              title  score  \\\n",
       "0  Scheduled  Untouchability, even in quarantine. 'We have n...     52   \n",
       "1  Scheduled  Delhi Govt Sources: Names of CM Arvind Kejriwa...    301   \n",
       "2  Scheduled  Delhi: AP Singh, advocate of 2012 Delhi gang-r...     17   \n",
       "3  Scheduled  Why the Supreme Court’s verdict on SC/ST quota...    110   \n",
       "4  Scheduled    What about the entrance exams scheduled in May?      8   \n",
       "\n",
       "       id               author  comms_num  \\\n",
       "0  fzvwz8           sam-sepiol          7   \n",
       "1  f7ogd8      The_andh_bhakth         30   \n",
       "2  flgvah               devCR7         21   \n",
       "3  f1o839         rahulthewall         47   \n",
       "4  fvcvo1  plsacceptmyusername          3   \n",
       "\n",
       "                                                body  \\\n",
       "0                                                NaN   \n",
       "1                                                NaN   \n",
       "2                                                NaN   \n",
       "3                                                NaN   \n",
       "4   CLAT, AILET, NEET, JEE have been postponed bu...   \n",
       "\n",
       "                                                 url  \n",
       "0  https://www.telegraphindia.com/india/coronavir...  \n",
       "1  https://twitter.com/ANI/status/123109390051893...  \n",
       "2  https://twitter.com/ani/status/124073128907587...  \n",
       "3  https://scroll.in/article/952687/why-the-supre...  \n",
       "4  https://www.reddit.com/r/india/comments/fvcvo1...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "topics_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Coronavirus           244\n",
       "Business/Finance      233\n",
       "AskIndia              226\n",
       "Sports                226\n",
       "Photography           225\n",
       "Politics              225\n",
       "Food                  225\n",
       "Policy/Economy        222\n",
       "Science/Technology    221\n",
       "Scheduled             221\n",
       "Non-Political         218\n",
       "Name: flair, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Almost balanced dataset\n",
    "topics_data.flair.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "topics_data.to_csv('../data/reddit_raw_data.csv', index=False)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}