Scraping_Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "9618310d",
   "metadata": {
    "id": "9618310d"
   },
   "source": [
    "# Web Scraping Tutorial\n",
    "\n",
    "This tutorial will teach you how Python to scrap and extract data from a web page. We will use two packages, `requests` to scrap the webpage and `BeautifulSoup` to extract the data.\n",
    "\n",
    "Many good references on web scraping are available online. I would recommend the following resources:\n",
    "1. Automate Boring Stuff with Python by Al Sweigart (2020) has a chapter on Web Scraping tutorial, which can be read [online](https://automatetheboringstuff.com/2e/chapter12/).\n",
    "2. Web Scraping With Python by Ryan Mitchell (2018) is a bit old book but provides a comprehensive guide to the topic.\n",
    "\n",
    "**Goal:** We will extract the cryptocurrency market price from Etherscan website: https://etherscan.io/tokens\n",
    "\n",
    "Your first step should always be to familiarize yourself with the website you want to scrape. Take a look at the website and try to inspect the HTML elements on the webpage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d17dfa0c",
   "metadata": {
    "id": "d17dfa0c"
   },
   "source": [
    "## Step 1: Scrap a web page\n",
    "\n",
    "Now, we are ready to scrap a webpage we want to get the data from with the `requests` package. We will use the following functions:\n",
    "\n",
    "* `requests.get('URL')` - make a request to the specified URL\n",
    "* `r.status_code` - get the status code of the request\n",
    "* `r.content` - get the binary content of the page\n",
    "\n",
    "More functions in the `requests` package are available in [its documentation](https://requests.readthedocs.io/en/latest/)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0d4756eb",
   "metadata": {
    "id": "0d4756eb"
   },
   "outputs": [],
   "source": [
    "# First, we will import the requests package\n",
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c15921a",
   "metadata": {
    "id": "8c15921a"
   },
   "outputs": [],
   "source": [
    "# Request the webpage\n",
    "headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "04ba2a22",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "04ba2a22",
    "outputId": "96c3dea0-e5ce-4d3e-ac37-e5e1fead486e"
   },
   "outputs": [],
   "source": [
    "# Type of the request object we've got\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cb0b645",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3cb0b645",
    "outputId": "0cfbda06-534f-4963-a159-40aac2ad318a"
   },
   "outputs": [],
   "source": [
    "# Check if the request is success\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74151d99",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "74151d99",
    "outputId": "d809fd17-e515-4cdb-eaeb-6577faf0e85b"
   },
   "outputs": [],
   "source": [
    "# Get the header of the web page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "63b8bf8d",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "63b8bf8d",
    "outputId": "682e708a-0309-4814-cae5-f151b0490e37"
   },
   "outputs": [],
   "source": [
    "# Get the content of the web page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9341389",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "d9341389",
    "outputId": "42318c8d-ddae-4de9-b3bc-6dc1daf71e01"
   },
   "outputs": [],
   "source": [
    "# Get the text in the web page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5938e1df",
   "metadata": {
    "id": "5938e1df"
   },
   "outputs": [],
   "source": [
    "# Save the content of web page\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "130d01f8",
   "metadata": {
    "id": "130d01f8"
   },
   "source": [
    "## Step 2: Load the web page as BeautifulSoup object\n",
    "\n",
    "After we crawled the web page and download it to the local disk, we will use `BeautifulSoup` package to parse HTML file and access the content. We will use the following functions:\n",
    "\n",
    "**1. Load the web page to BeautifulSoup**\n",
    "* `soup = BeautifulSoup(html_doc, 'html.parser')` - parse the HTML content to BeautifulSoup object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b2eef79",
   "metadata": {
    "id": "8b2eef79"
   },
   "outputs": [],
   "source": [
    "# First, we will import the BeautifulSoup from bs4 package\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29460bba",
   "metadata": {
    "id": "29460bba"
   },
   "outputs": [],
   "source": [
    "# Load the web page and parse it to BeautifulSoup\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9f7ab9a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c9f7ab9a",
    "outputId": "f56c9df6-ea85-4932-bb1a-ce4c6debe496"
   },
   "outputs": [],
   "source": [
    "# Check the type of our soup object\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a89243d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the content of the web page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ee1f8e4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 125
    },
    "id": "9ee1f8e4",
    "outputId": "660728bf-2794-4bc3-ab42-b7cbfaf359c7"
   },
   "outputs": [],
   "source": [
    "# Print all the text in the webpage\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "LjNxOJUEyuPY",
   "metadata": {
    "id": "LjNxOJUEyuPY"
   },
   "source": [
    "**2. Get the content of the element**\n",
    "* `soup.title` - get the title of the page\n",
    "* `soup.title.string` - get the string in the title element\n",
    "* `soup.h1` - get the H1 element in the web page\n",
    "* `soup.h1.attrs` - get all attributes in the H1 element\n",
    "* `soup.h1['class']` - get the class attribute in the H1 element"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae45ff28",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "ae45ff28",
    "outputId": "c0d11b0e-b2d2-45b4-8de5-c6e5ed7d55bd"
   },
   "outputs": [],
   "source": [
    "# Get the title of the page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ecb4ef3b",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "ecb4ef3b",
    "outputId": "ec8c6bd3-e2f0-4d06-dd6e-d481eccdf1de"
   },
   "outputs": [],
   "source": [
    "# Other HTML elements also work too\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4107df52",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the class attribute of an element\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "wIsB9X-Jyz5b",
   "metadata": {
    "id": "wIsB9X-Jyz5b"
   },
   "source": [
    "**3. Look for the element in the web page**\n",
    "* `soup.find('HTML_tag')` - get the element from an HTML tag\n",
    "* `soup.find_all('HTML_tag')` - get the list of elelemts that has the specified HTML tag\n",
    "* `soup.select('CSS_selector')` - get the list of elements with the specified [CSS selector](https://www.w3schools.com/cssref/css_selectors.asp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d370ecc0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "d370ecc0",
    "outputId": "5463fecc-622d-42d7-d29c-b5f24e65a2f1"
   },
   "outputs": [],
   "source": [
    "# We can also get the page title using soup.find() function\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9reBVSAuzRkK",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9reBVSAuzRkK",
    "outputId": "955d41c0-1590-4365-b241-5300c5517d4b"
   },
   "outputs": [],
   "source": [
    "# Get all the elements with image tag\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fiu6CYDGocEi",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "fiu6CYDGocEi",
    "outputId": "2c0eb893-aafc-4b96-94de-5f9b55751aac"
   },
   "outputs": [],
   "source": [
    "# Get all the token names on the web page\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "u4qcpEWJy9mp",
   "metadata": {
    "id": "u4qcpEWJy9mp"
   },
   "source": [
    "## Step 3: Extract the data from the table\n",
    "\n",
    "Now, we will extract the cryptocurrencies market price from the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e54b736",
   "metadata": {
    "id": "8e54b736"
   },
   "outputs": [],
   "source": [
    "# Get the table element in the web page\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69ee70a8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "69ee70a8",
    "outputId": "f32fdbe6-87eb-4a88-ac4b-2d879b88791d"
   },
   "outputs": [],
   "source": [
    "# Get the table headers\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6qb-NIfV1kxL",
   "metadata": {
    "id": "6qb-NIfV1kxL"
   },
   "source": [
    "For loop over each row in the table and extract the data for each column in the row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "rHg-Ri_S3_SN",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rHg-Ri_S3_SN",
    "outputId": "fc4a7b87-adc1-4e2a-ac2c-2cb2b9277d98"
   },
   "outputs": [],
   "source": [
    "# For loop over each row in the table\n",
    "\n",
    "\n",
    "    # Get all the columns in the row\n",
    "    \n",
    "    \n",
    "    # For loop over each column and extract the string\n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "CYIamv6e-lOH",
   "metadata": {
    "id": "CYIamv6e-lOH"
   },
   "source": [
    "## Step 4: Create a DataFrame table and write to a CSV file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "BvFNIO2x_M2P",
   "metadata": {
    "id": "BvFNIO2x_M2P"
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "YoPXwFOb9rvE",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "YoPXwFOb9rvE",
    "outputId": "46ed6c17-544e-499e-a26c-3b8defcce2ca"
   },
   "outputs": [],
   "source": [
    "# How many rows in the extracted data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "y4xoW-fQ9xnT",
   "metadata": {
    "id": "y4xoW-fQ9xnT"
   },
   "outputs": [],
   "source": [
    "# Convert the data list to DataFrame object\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "vuGWTQ4LCg0l",
   "metadata": {
    "id": "vuGWTQ4LCg0l"
   },
   "source": [
    "Split the columns with \"\\n\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "CxSweamlCf8O",
   "metadata": {
    "id": "CxSweamlCf8O"
   },
   "outputs": [],
   "source": [
    "# Split between token name and token symbol\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "s5n2YlKA_yUd",
   "metadata": {
    "id": "s5n2YlKA_yUd"
   },
   "outputs": [],
   "source": [
    "# Split between the USD and ETH prices\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "pdIUxN4yACgS",
   "metadata": {
    "id": "pdIUxN4yACgS"
   },
   "outputs": [],
   "source": [
    "# Split the number of holders and percent changes\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "YplcV-TmEvF7",
   "metadata": {
    "id": "YplcV-TmEvF7"
   },
   "source": [
    "Convert string into numerical columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "BkIsQ83uPjcR",
   "metadata": {
    "id": "BkIsQ83uPjcR"
   },
   "outputs": [],
   "source": [
    "# Regular expression pattern to match numbers\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37WDkw6LGWtu",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "37WDkw6LGWtu",
    "outputId": "efe93297-3901-4630-fb9d-ec0984afc252"
   },
   "outputs": [],
   "source": [
    "# For each numerical column, convert the string to float numbers\n",
    "\n",
    "\n",
    "    # Use df[col_name].str.extract() to extract the numbers and\n",
    "    # .astype(float) to convert the string to float numbers\n",
    "    \n",
    "    \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "u4MSGKsjHl3o",
   "metadata": {
    "id": "u4MSGKsjHl3o"
   },
   "source": [
    "Last but not least, remove the bracket in token symbol column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01EdKFSLGk80",
   "metadata": {
    "id": "01EdKFSLGk80"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "LcGeD1xOQFIb",
   "metadata": {
    "id": "LcGeD1xOQFIb"
   },
   "source": [
    "Write the DataFrame table to CSV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "DavpYCJHIvCG",
   "metadata": {
    "id": "DavpYCJHIvCG"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}