01_rezepte_download.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Chefkoch.de](http://www.chefkoch.de/)\n",
    "------\n",
    "\n",
    "## Ziel: \n",
    "### Scraping der Hauptrezeptesammlung von über 300'000 verschiedenen Rezepten (1.Teil)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import datetime as dt\n",
    "from datetime import datetime\n",
    "from multiprocessing import Pool\n",
    "\n",
    "from time import sleep, time\n",
    "from random import randint, choice\n",
    "import requests\n",
    "import urllib.request\n",
    "from bs4 import BeautifulSoup\n",
    "import csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Daten wie Rezeptname, Bewertung, Datum vom Upload des Rezeptes, etc. werden in eine csv Datei gespeichert.\n",
    "Falls das Rezept ein Bild hat, wird das Thumbnail im Ordner **search_thumbnails** abgelegt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# OS\n",
    "NOW           = dt.datetime.now()\n",
    "FILE_NAME     = 'chefkoch_rezepte_' + NOW.strftime('%d-%m-%Y') + '.csv'\n",
    "DATASET_FOLDER = 'input/csv_files/'\n",
    "IMGS_FOLDER  = 'input/images/search_thumbnails/'\n",
    "\n",
    "# Chefkoch.de Seite\n",
    "CHEFKOCH_URL  = 'http://www.chefkoch.de'\n",
    "START_URL     = 'http://www.chefkoch.de/rs/s'\n",
    "CATEGORY      = '/Rezepte.html'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alle 300k Rezepte sortiert nach Datum: http://www.chefkoch.de/rs/s30o3/Rezepte.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wenn man Website Scrapping durchführt, ist es wichtig die robots.txt Datei zu respektieren. Manche Administratoren möchten nicht das bestimmte Directories von Bots besucht werden. https://www.chefkoch.de/robots.txt liefert:\n",
    "\n",
    "- User-agent: *  # directed to all spiders, not just Scooter\n",
    "- Disallow: /cgi-bin\n",
    "- Disallow: /stats\n",
    "- Disallow: /pictures/fotoalben/\n",
    "- Disallow: /forumuploads/\n",
    "- Disallow: /pictures/user/\n",
    "- Disallow: /user/\n",
    "- Disallow: /avatar/\n",
    "- Disallow: /cms/\n",
    "- Disallow: /produkte/\n",
    "- Disallow: /how2videos/\n",
    "\n",
    "Aufgeführt sind Directories die uns gar nicht interessieren, weshalb man getrost weiter machen kann. Nichtsdestotrotz  sind Massnahmen, wie zufällige Headers und genügend grosse Pausen zwischen den einzelnen Requests empfehlenswert, um einen möglichen Ban der Website zu vermeiden."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "desktop_agents = ['Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36',\n",
    "                 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36',\n",
    "                 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36',\n",
    "                 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/602.2.14 (KHTML, like Gecko) Version/10.0.1 Safari/602.2.14',\n",
    "                 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36',\n",
    "                 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',\n",
    "                 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.98 Safari/537.36',\n",
    "                 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36',\n",
    "                 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36',\n",
    "                 'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0']\n",
    "\n",
    "def random_headers():\n",
    "    return {'User-Agent': choice(desktop_agents),'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total pages:  10560\n"
     ]
    }
   ],
   "source": [
    "category_url = START_URL + '0o3' + CATEGORY\n",
    "\n",
    "def _get_html(url):\n",
    "    page = ''\n",
    "    while page == '':\n",
    "        try:\n",
    "            page = requests.get(url, headers=random_headers())\n",
    "        except:\n",
    "            print('Connection refused')\n",
    "            time.sleep(10)\n",
    "            continue\n",
    "    return page.text\n",
    "\n",
    "def _get_total_pages(html):\n",
    "    soup = BeautifulSoup(html, 'lxml')\n",
    "    total_pages = soup.find('div', class_='ck-pagination qa-pagination').find('a', class_='qa-pagination-pagelink-last').text\n",
    "    return int(total_pages)\n",
    "\n",
    "html_text_total_pages = _get_html(category_url)\n",
    "total_pages = _get_total_pages(html_text_total_pages)\n",
    "print('Total pages: ', total_pages)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Liste von allen einzelnen Rezepteurls bei Chefkoch im folgenden Format:\n",
    "1. Seite: http&#58;//www.chefkoch.de/rs/s**0**o3/Rezepte.html\n",
    "2. Seite: http&#58;//www.chefkoch.de/rs/s**30**o3/Rezepte.html\n",
    "3. Seite: http&#58;//www.chefkoch.de/rs/s**60**o3/Rezepte.html\n",
    "4. Seite: ...\n",
    "\n",
    "Auf einer Seite erhält man 30 Rezepte. Um jede Seite aufrufen zu können, muss man nur die Zahl zwischen **s** und **o3** um 30 erhöhen."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "url_list = []\n",
    "\n",
    "for i in range(0, total_pages + 1):\n",
    "    url_to_scrap = START_URL + str(i * 30) + 'o3' + CATEGORY\n",
    "    url_list.append(url_to_scrap)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['http://www.chefkoch.de/rs/s0o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s30o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s60o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s90o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s120o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s150o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s180o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s210o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s240o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s270o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s300o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s330o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s360o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s390o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s420o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s450o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s480o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s510o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s540o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s570o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s600o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s630o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s660o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s690o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s720o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s750o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s780o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s810o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s840o3/Rezepte.html',\n",
      " 'http://www.chefkoch.de/rs/s870o3/Rezepte.html']\n"
     ]
    }
   ],
   "source": [
    "from pprint import pprint\n",
    "# Die ersten 30 Seiten:\n",
    "pprint(url_list[:30])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _write_to_recipes(data):\n",
    "    path = DATASET_FOLDER + FILE_NAME\n",
    "    with open(path, 'a', newline='') as f:\n",
    "        writer = csv.writer(f)\n",
    "        writer.writerow((data['recipe_id'],\n",
    "                        data['recipe_name'],\n",
    "                        data['average_rating'],\n",
    "                        data['stars_shown'],\n",
    "                        data['votes'],\n",
    "                        data['difficulty'],\n",
    "                        data['preparation_time'],\n",
    "                        data['date'],\n",
    "                        data['link'],\n",
    "                        data['has_picture']))\n",
    "\n",
    "def _get_picture_link(item):\n",
    "    item_class = item.find('picture').find('img').get('class')\n",
    "    if item_class == ['lazyload']:\n",
    "        img_link = item.find('picture').find('img').get('data-srcset')\n",
    "    else: \n",
    "        img_link = item.find('picture').find('source').get('srcset')\n",
    "    return(img_link)\n",
    "\n",
    "def _get_front_page(html):\n",
    "    soup = BeautifulSoup(html, 'lxml')\n",
    "    lis = soup.find_all('li', class_=\"search-list-item\")\n",
    "    \n",
    "    for index, li in enumerate(lis):\n",
    "\n",
    "        # get rezept ID\n",
    "        try:\n",
    "            id_r = li.get('id')\n",
    "        except:\n",
    "            id_r = ''\n",
    "\n",
    "        # bild speichern falls eins verügbar\n",
    "        try: \n",
    "            if li.find('picture') is not None:\n",
    "                img_link = _get_picture_link(li)\n",
    "                img_name = IMGS_FOLDER + str(id_r) + '.jpg'\n",
    "                urllib.request.urlretrieve(img_link, img_name)\n",
    "                has_pic = 'yes'\n",
    "            else: \n",
    "                has_pic = 'no'\n",
    "        except:\n",
    "            has_pic = ''\n",
    "\n",
    "        # link\n",
    "        try:\n",
    "            link = CHEFKOCH_URL + li.find('a').get('href')\n",
    "        except:\n",
    "            link = ''\n",
    "\n",
    "        # name des rezeptes\n",
    "        try:\n",
    "            name = li.find('div', class_='search-list-item-title').text.strip()\n",
    "        except:\n",
    "            name = ''\n",
    "\n",
    "        # durchschnitts bewertung von nutzern\n",
    "        try:\n",
    "            stars = li.find('span', class_='search-list-item-uservotes-stars').get('title')\n",
    "        except:\n",
    "            stars = ''\n",
    "\n",
    "        # anzahl sterne\n",
    "        try:\n",
    "            stars_shown = li.find('span', class_='search-list-item-uservotes-stars').get('class')[1]\n",
    "        except:\n",
    "            stars_shown = ''\n",
    "\n",
    "        # anzahl votes\n",
    "        try:\n",
    "            votes = li.find('span', class_='search-list-item-uservotes-count').text.strip()\n",
    "        except:\n",
    "            votes = ''\n",
    "\n",
    "        # schwierigkeitsgrad des rezeptes => simpel, normal oder pfiffig\n",
    "        try:\n",
    "            difficulty = li.find('span', class_='search-list-item-difficulty').text.strip()\n",
    "\n",
    "        except:\n",
    "            difficulty = ''\n",
    "\n",
    "        # zubereitungs zeit\n",
    "        try:\n",
    "            preptime = li.find('span', class_='search-list-item-preptime').text.strip()\n",
    "        except:\n",
    "            preptime = ''\n",
    "\n",
    "        # datum\n",
    "        try:\n",
    "            date = li.find('span', class_='search-list-item-activationdate').text.strip()\n",
    "        except:\n",
    "            date = ''\n",
    "\n",
    "        # write dictionary\n",
    "        data = {'recipe_id' : id_r,\n",
    "                'recipe_name' : name,\n",
    "                'average_rating': stars,\n",
    "                'stars_shown' : stars_shown,\n",
    "                'votes' : votes,\n",
    "                'difficulty' : difficulty,\n",
    "                'preparation_time' : preptime,\n",
    "                'has_picture' : has_pic,\n",
    "                'date' : date,\n",
    "                'link' : link}\n",
    "        \n",
    "        # append file\n",
    "        _write_to_recipes(data)\n",
    "        \n",
    "def scrap_main(url):\n",
    "    print('Current url: ', url)\n",
    "    html = _get_html(url)\n",
    "    _get_front_page(html)\n",
    "    #sleep(randint(1, 2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Current url:  http://www.chefkoch.de/rs/s234390o3/Rezepte.html\n",
      "...\n",
      "Current url:  http://www.chefkoch.de/rs/s316800o3/Rezepte.html\n",
      "--- 7346.419415235519 seconds ---\n"
     ]
    }
   ],
   "source": [
    "start_time = time()\n",
    "with Pool(1) as p:\n",
    "    p.map(scrap_main, url_list[7813:])\n",
    "print(\"--- %s seconds ---\" % (time() - start_time))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CSV Datei lesen und korrigieren\n",
    "\n",
    "Unter input/csv_files/ findet man die erstellte CSV Datei.\n",
    "Grösse: 62.1 MB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "chefkoch_rezepte_26-12-2017.csv\r\n"
     ]
    }
   ],
   "source": [
    "!ls input/csv_files/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>recipe-3449791514066439</td>\n",
       "      <td>Bratapfel-Rotkohl</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>normal</td>\n",
       "      <td>45 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>http://www.chefkoch.de/rezepte/344979151406643...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>recipe-3449721514056932</td>\n",
       "      <td>„Frühstückswolke 7“ aus Fiefhusen</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>pfiffig</td>\n",
       "      <td>15 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>http://www.chefkoch.de/rezepte/344972151405693...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>recipe-3449701514054063</td>\n",
       "      <td>Deichgrafensalat von der Westküste</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>normal</td>\n",
       "      <td>30 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>http://www.chefkoch.de/rezepte/344970151405406...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>recipe-3449651514047893</td>\n",
       "      <td>Geschichteter Kohl</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>normal</td>\n",
       "      <td>40 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>http://www.chefkoch.de/rezepte/344965151404789...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>recipe-3449681514052511</td>\n",
       "      <td>Rinderlendenragout</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>normal</td>\n",
       "      <td>15 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>http://www.chefkoch.de/rezepte/344968151405251...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         0                                   1  \\\n",
       "0  recipe-3449791514066439                   Bratapfel-Rotkohl   \n",
       "1  recipe-3449721514056932   „Frühstückswolke 7“ aus Fiefhusen   \n",
       "2  recipe-3449701514054063  Deichgrafensalat von der Westküste   \n",
       "3  recipe-3449651514047893                  Geschichteter Kohl   \n",
       "4  recipe-3449681514052511                  Rinderlendenragout   \n",
       "\n",
       "                       2         3    4        5        6           7  \\\n",
       "0  0 Bewertungen - Ø 2.5  star-unb  (0)   normal  45 min.  24.12.2017   \n",
       "1  0 Bewertungen - Ø 2.5  star-unb  (0)  pfiffig  15 min.  24.12.2017   \n",
       "2  0 Bewertungen - Ø 2.5  star-unb  (0)   normal  30 min.  24.12.2017   \n",
       "3  0 Bewertungen - Ø 2.5  star-unb  (0)   normal  40 min.  24.12.2017   \n",
       "4  0 Bewertungen - Ø 2.5  star-unb  (0)   normal  15 min.  24.12.2017   \n",
       "\n",
       "                                                   8   9  \n",
       "0  http://www.chefkoch.de/rezepte/344979151406643...  no  \n",
       "1  http://www.chefkoch.de/rezepte/344972151405693...  no  \n",
       "2  http://www.chefkoch.de/rezepte/344970151405406...  no  \n",
       "3  http://www.chefkoch.de/rezepte/344965151404789...  no  \n",
       "4  http://www.chefkoch.de/rezepte/344968151405251...  no  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "chef_rezepte = pd.read_csv('input/csv_files/chefkoch_rezepte_26-12-2017.csv', header=None)\n",
    "chef_rezepte.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344979151406643...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344972151405693...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344970151405406...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344965151404789...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344968151405251...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344948151403464...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344964151404709...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344957151404240...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344947151403342...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>http://www.chefkoch.de/rezepte/344924151395310...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   8\n",
       "0  http://www.chefkoch.de/rezepte/344979151406643...\n",
       "1  http://www.chefkoch.de/rezepte/344972151405693...\n",
       "2  http://www.chefkoch.de/rezepte/344970151405406...\n",
       "3  http://www.chefkoch.de/rezepte/344965151404789...\n",
       "4  http://www.chefkoch.de/rezepte/344968151405251...\n",
       "5  http://www.chefkoch.de/rezepte/344948151403464...\n",
       "6  http://www.chefkoch.de/rezepte/344964151404709...\n",
       "7  http://www.chefkoch.de/rezepte/344957151404240...\n",
       "8  http://www.chefkoch.de/rezepte/344947151403342...\n",
       "9  http://www.chefkoch.de/rezepte/344924151395310..."
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chef_rezepte[[8]][:10] # erste 10 zeilen der 8. spalte"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Beim Scraping ist ein Fehler unterlaufen. Die Links müssen als https Links und nicht als http Links gespeichert werden.\n",
    "### Korrektur:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "chef_rezepte[8] = chef_rezepte[8].str.replace('http', 'https')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>8</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34497915140664...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34497215140569...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34497015140540...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34496515140478...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34496815140525...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34494815140346...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34496415140470...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34495715140424...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34494715140334...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>https://www.chefkoch.de/rezepte/34492415139531...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   8\n",
       "0  https://www.chefkoch.de/rezepte/34497915140664...\n",
       "1  https://www.chefkoch.de/rezepte/34497215140569...\n",
       "2  https://www.chefkoch.de/rezepte/34497015140540...\n",
       "3  https://www.chefkoch.de/rezepte/34496515140478...\n",
       "4  https://www.chefkoch.de/rezepte/34496815140525...\n",
       "5  https://www.chefkoch.de/rezepte/34494815140346...\n",
       "6  https://www.chefkoch.de/rezepte/34496415140470...\n",
       "7  https://www.chefkoch.de/rezepte/34495715140424...\n",
       "8  https://www.chefkoch.de/rezepte/34494715140334...\n",
       "9  https://www.chefkoch.de/rezepte/34492415139531..."
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chef_rezepte[[8]][:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "chef_rezepte.columns = ['recipe_id', 'recipe_name', 'average_rating',\n",
    "                        'stars_shown', 'votes', 'difficulty', 'preparation_time',\n",
    "                        'date', 'link', 'has_picture']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>recipe_id</th>\n",
       "      <th>recipe_name</th>\n",
       "      <th>average_rating</th>\n",
       "      <th>stars_shown</th>\n",
       "      <th>votes</th>\n",
       "      <th>difficulty</th>\n",
       "      <th>preparation_time</th>\n",
       "      <th>date</th>\n",
       "      <th>link</th>\n",
       "      <th>has_picture</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>recipe-3449791514066439</td>\n",
       "      <td>Bratapfel-Rotkohl</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>normal</td>\n",
       "      <td>45 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>https://www.chefkoch.de/rezepte/34497915140664...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>recipe-3449721514056932</td>\n",
       "      <td>„Frühstückswolke 7“ aus Fiefhusen</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>pfiffig</td>\n",
       "      <td>15 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>https://www.chefkoch.de/rezepte/34497215140569...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>recipe-3449701514054063</td>\n",
       "      <td>Deichgrafensalat von der Westküste</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>normal</td>\n",
       "      <td>30 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>https://www.chefkoch.de/rezepte/34497015140540...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>recipe-3449651514047893</td>\n",
       "      <td>Geschichteter Kohl</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>normal</td>\n",
       "      <td>40 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>https://www.chefkoch.de/rezepte/34496515140478...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>recipe-3449681514052511</td>\n",
       "      <td>Rinderlendenragout</td>\n",
       "      <td>0 Bewertungen - Ø 2.5</td>\n",
       "      <td>star-unb</td>\n",
       "      <td>(0)</td>\n",
       "      <td>normal</td>\n",
       "      <td>15 min.</td>\n",
       "      <td>24.12.2017</td>\n",
       "      <td>https://www.chefkoch.de/rezepte/34496815140525...</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 recipe_id                         recipe_name  \\\n",
       "0  recipe-3449791514066439                   Bratapfel-Rotkohl   \n",
       "1  recipe-3449721514056932   „Frühstückswolke 7“ aus Fiefhusen   \n",
       "2  recipe-3449701514054063  Deichgrafensalat von der Westküste   \n",
       "3  recipe-3449651514047893                  Geschichteter Kohl   \n",
       "4  recipe-3449681514052511                  Rinderlendenragout   \n",
       "\n",
       "          average_rating stars_shown votes difficulty preparation_time  \\\n",
       "0  0 Bewertungen - Ø 2.5    star-unb   (0)     normal          45 min.   \n",
       "1  0 Bewertungen - Ø 2.5    star-unb   (0)    pfiffig          15 min.   \n",
       "2  0 Bewertungen - Ø 2.5    star-unb   (0)     normal          30 min.   \n",
       "3  0 Bewertungen - Ø 2.5    star-unb   (0)     normal          40 min.   \n",
       "4  0 Bewertungen - Ø 2.5    star-unb   (0)     normal          15 min.   \n",
       "\n",
       "         date                                               link has_picture  \n",
       "0  24.12.2017  https://www.chefkoch.de/rezepte/34497915140664...          no  \n",
       "1  24.12.2017  https://www.chefkoch.de/rezepte/34497215140569...          no  \n",
       "2  24.12.2017  https://www.chefkoch.de/rezepte/34497015140540...          no  \n",
       "3  24.12.2017  https://www.chefkoch.de/rezepte/34496515140478...          no  \n",
       "4  24.12.2017  https://www.chefkoch.de/rezepte/34496815140525...          no  "
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chef_rezepte.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "chef_rezepte.to_csv('input/csv_files/chefkoch_rezepte_27-12-2017.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Umbennenung der Thumbnails: Hänge ein 0 hintendran -> Thumbnail ist das erste Bild des Rezeptes.\n",
    "#### 1, 2, 3, ..., n Bilder kommen im Teil 2 dazu."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "path = 'input/images/search_thumbnails/'\n",
    "files = os.listdir(path)\n",
    "i = 1\n",
    "for file in files:\n",
    "    filename, file_extension = os.path.splitext(file)\n",
    "    os.rename(os.path.join(path, file), os.path.join(path, filename + '-0' + file_extension))\n",
    "    print('renamed: ', i)\n",
    "    i = i+1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Weiter gehts mit Teil 2: 02_rezepte_details.ipynb"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}