X02E. Web Scraping Exercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# X02E: Web Scraping Exercises"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DfT Data Science Strategy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Challenge 1: For the URL below see if you can scrape the text from each of the content bubbles and print this.\n",
    "\n",
    "Challenge 2: Try storing the data in a list!\n",
    "\n",
    "Challenge 3: Try splitting the data into separate lists for the heading, grey italic text and the paragraph text "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "url = 'https://s3-eu-west-1.amazonaws.com/dft-innovation4/Innovation-timeline.html'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html lang=\"en\">\n",
      " <head>\n",
      "  <meta charset=\"utf-8\"/>\n",
      "  <title>\n",
      "   DfT Data Science · Timeline\n",
      "  </title>\n",
      "  <meta content=\"width=device-width, initial-scale=1.0\" name=\"viewport\"/>\n",
      "  <meta content=\"\" name=\"description\"/>\n",
      "  <meta content=\"\" name=\"author\"/>\n",
      "  <!-- Le styles -->\n",
      "  <link href=\"css/Bootstrap.css\" rel=\"stylesheet\"/>\n",
      "  <link href=\"css/Bootstrap-resp.css\" rel=\"stylesheet\"/>\n",
      "  <link href=\"css/Custom.css\" rel=\"stylesheet\"/>\n",
      "  <link href=\"css/timeline.css\" rel=\"stylesheet\"/>\n",
      "  <link href=\"css/reset.css\" rel=\"stylesheet\"/>\n",
      "  <link href=\"css/styles.css\" rel=\"stylesheet\" type=\"text/css\"/>\n",
      "  <link href=\"https://ajax.googleapis.com/ajax/libs/jqueryui/1.11.4/themes/smoothness/jquery-ui.css\" rel=\"stylesheet\"/>\n",
      "  <script src=\"https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js\">\n",
      "  </script>\n",
      "  <script src=\"https://ajax.googleapis.com/ajax/libs/jqueryui/1.11.4/jquery-ui.min.js\">\n",
      "  </script>\n",
      "  <script src=\"js/main.js\">\n",
      "  </script>\n",
      "  <script src=\"js/modernizr.js\">\n",
      "  \n"
     ]
    }
   ],
   "source": [
    "r = requests.get(url)                     # Making the request to the page\n",
    "r.status_code                             # Checking that the request has been successful\n",
    "soup = BeautifulSoup(r.content,'lxml')           # Creating the beautiful soup object                  \n",
    "print(soup.prettify()[0:1000])     \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "data = soup.findAll('div',class_=\"cd-timeline-content\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "March 2016: Planning\n",
      "\"Because plans are useless, but planning is indispensible\"\n",
      "Mix two parts ideas, with one part strategy and a shot of organisation. Add a generous glug of enthusiasm and two litres of coffee and voila! The cocktail for success in                     innovation!\n",
      "We'll be setting up two groups... The Innovation Group who will set the direction and The Action Group who will help out with the legwork and make things happen.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "April 2016: DfT HackClub\n",
      "\"The first rule of Hackclub is that you don't talk about Hackclub...\"\n",
      "16 people from across Dft tackle a variety of business challenges over 2 days using nothing but open source software, code and enthusiasm. And maybe some pizza. Their                       solutions will be pitched to the assembled judges and the winning team will be crowned DfT Hackclub champions!\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "May 2016: Skills\n",
      "\"Investing in our people with digital skills\"\n",
      "We're currently developing training for Python; an open source (and free) software package which is a one stop shop for analysis, data visualisation, programming, websites, app development, web scraping etc.\n",
      "For each event we hold, we're planning to deliver some Python training to capitalise on the energy and enthusiasm created! Trainees will also take on a follow up project and continue to participate in cath-up sessions in order to consolidate their skills and continue to learn, and more importantly, share and collaborate with the rest of the training group.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "June 2016: IMCreate\n",
      "\"Intelligent Mobility\"\n",
      "Our good friends at Transport Systems Catapult in Milton Keynes are holding their second IMCreate Hackathon and if it's  even half as good as the first one, it'll knock everyones socks off.\n",
      "Coders, business thinkers, technical experts and designers from academia and industry will compete for 24hrs to come up with the next big idea for transport, using a huge array of data sets and cutting edge hardware. The three best ideas from the event will go on to a showcase event at the Imagine Festival in June having had further time to develop their idea.\n",
      "You can expect DfT to get even more involved this year as we'll be both setting a challenge, and sending some more of our people along to revel in the creative and innovative atmosphere!\n",
      "\n",
      "\n",
      "\n",
      "  \n",
      "June 2016: DfT Vishack!\n",
      "\"Visualising our Impact\"\n",
      "June is a busy month as it also sees our inaugural DfT VisHack. We'll be spending two days deep in all kinds of data with a view to creating innovative, beautiful and insightful data visualisations.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "July 2016: The DfT DataJam\n",
      "\"Insert Jam related pun here\"\n",
      "A DataJam is basically a cooler name for a Data Dive! We're bringing together 40-50 analysts, stat geeks and data scientists from DfT, Dft's friends (TfL, Highways England, HS2, DVLA etc.) and wider government (GDS, BIS, MOJ, DWP etc.) for an two day project fueled by energy, enthusiasm and innovation to solve transport related problems and issues using data and black belt level analysis and visualisation skills.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "August 2016: Skills\n",
      "\"Investing in our people with digital skills\"\n",
      "The DataJam will  generate energy and enthusiasm which we intend to capitalise on by running a second skills event. As before this is likely to focus on Pythton and will follow our model of a training course followed by a project.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "October 2016: Hacktrain!\n",
      "\"All aboard!!\"\n",
      "Our inspiration for DfT Hacks comes from none other than Alejandro and River, the dynamic duo who run the Hacktrain. DfT was involved in this last year but you can expect us to play an even bigger part this year in what's fast becoming the most innovative event in Transport! \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "November 2016: Skills\n",
      "\"Investing in our people with digital skills\"\n",
      "Our third skills event will capitalise on the energy created by Hacktrain. As before this is likely to focus on Pythton and will follow our model of a training course followed by a project.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "January 2017: DfT HackClub\n",
      "\"Hopefully a lot of people have been breaking the first two rules of Hackclub...\"\n",
      "By this time, DfT should be well and truly buzzing as a result of the awesome events we've organised and not even the murky, cold and wet weather of January is going to dampen our spirits. As such we expect the 2nd DfT HackClub to be even more awesome than the first with 30+ people from across Dft particiapting putting the wide range of skills they've developed to the test on a series of challenges more fiendish and complex than ever before!\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "February 2017: DfT Hacks!\n",
      "\"There is a better way. Find it.\"\n",
      "We're gathering 60 of the most talented designers, developers and entrepreneurs from across the world to collaborate and work day and night to tackle a series of challenges. Our hackers will incept, imagine, innovate and iterate their solutions and pitch them to our board of judges who will digest, discuss and decide which team will        be elected as DfT's Hack Champions 2017!\n",
      "This will all be supported by 'Team DfT': mentors, experts and enablers who'll be there every step of the way to support our hackers and ensure the event goes smoothly.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "March 2017: Review\n",
      "\"Review how things went and plan to do it all bigger and better next year!\"\n",
      "If we can pull even half of the things we want off, then we'll have done a massive amount of work and made a massive amount of friends in the process and we should be justly                 proud...\n",
      "However! There's always opportunities to improve and we'll be asking ourselves some pretty candid questions... What went well? What didn't go well? What can we do better?                    How can we make DfT even more awesome for next year?\n",
      "Probably the most important thing to come out of the review is the 'DfT Hackbook'. People come and go, tech moves on and changes, but the Hackbook will provide a constant                    and consistent framework for future events, empowering others to take the mantle of innovation from us and continue to run with it.\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for x in data:\n",
    "    print (x.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}