Coursera_Capstone.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Coursera Capstone\n",
    "# The Battle of the Neighborhoods\n",
    "This document will be used for the Coursera Capstone project, part of the IBM Data Science Professional Certificate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import geopandas as gpd\n",
    "import numpy as np\n",
    "import seaborn as sns\n",
    "from shapely.geometry import Point, Polygon\n",
    "\n",
    "import shapely.speedups #For point in polygon calculations\n",
    "shapely.speedups.enable()\n",
    "\n",
    "import json # library to handle JSON files\n",
    "\n",
    "# Matplotlib and associated plotting modules\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.cm as cm\n",
    "import matplotlib.colors as colors\n",
    "\n",
    "# libraries for displaying images\n",
    "from IPython.display import Image \n",
    "from IPython.core.display import HTML \n",
    "\n",
    "#!conda install -c conda-forge geopy --yes # uncomment this line if you haven't completed the Foursquare API lab\n",
    "from geopy.geocoders import Nominatim # convert an address into latitude and longitude values\n",
    "\n",
    "# import K Nearest Neightbors\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "# import k-means from clustering stage\n",
    "from sklearn.cluster import KMeans\n",
    "\n",
    "#!conda install -c conda-forge folium=0.5.0 --yes # uncomment this line if you haven't completed the Foursquare API lab\n",
    "import folium # map rendering library\n",
    "from folium import plugins\n",
    "\n",
    "import requests # library to handle requests\n",
    "\n",
    "from pandas.io.json import json_normalize # tranform JSON file into a pandas dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a dataframe of neighborhoods in Minneapolis and St. Paul\n",
    "As far as I could tell, the data for Minneapolis neighborhoods did not come attached with latitude and longitude values. The following section creates a dataframe with neighborhood names and attached lat/long values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Data retrieved from\n",
    "# https://opendata.minneapolismn.gov/datasets/minneapolis-neighborhoods\n",
    "# Data downloaded and placed in local directory.\n",
    "\n",
    "with open('Minneapolis_Neighborhoods.geojson.json') as jsondata:\n",
    "    mpls_data = json.load(jsondata)\n",
    "    \n",
    "#with open('Communities.geojson') as jsondata:\n",
    " #   mpls_comm_data = json.load(jsondata)\n",
    "    \n",
    "with open('stp_neighborhoods.geojson') as jsondata:\n",
    "    stp_nbhd_data = json.load(jsondata)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We will refer back to the larger mpls_data dataframe later.\n",
    "mpls_nbhds = mpls_data['features']\n",
    "#mpls_comms = mpls_comm_data['features']\n",
    "stp_nbhds = stp_nbhd_data['features']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "source": [
    "Look at the first entry. Note the neighborhood name has key 'BDNAME' for the Minneapolis neighborhoods. The neighborhood name has key 'name2' for the St. Paul neighborhoods. Using these keys, we will build geo dataframes with the columns `Neighborhood`, `Latitude`, and `Longitude`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Populate the latitude and longitude using Geopy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "current nbhd is Phillips West\n",
      "current nbhd is Downtown West\n",
      "Downtown West, Minneapolis, MN is not found on geopy.\n",
      "current nbhd is Downtown East\n",
      "current nbhd is Ventura Village\n",
      "Ventura Village, Minneapolis, MN is not found on geopy.\n",
      "current nbhd is Sumner - Glenwood\n",
      "current nbhd is Shingle Creek\n",
      "current nbhd is Lind - Bohanon\n",
      "current nbhd is Webber - Camden\n",
      "current nbhd is Bottineau\n",
      "current nbhd is Victory\n",
      "current nbhd is Humboldt Industrial Area\n",
      "current nbhd is Howe\n",
      "current nbhd is Hiawatha\n",
      "current nbhd is Windom\n",
      "current nbhd is Ericsson\n",
      "current nbhd is Field\n",
      "current nbhd is Page\n",
      "current nbhd is Como\n",
      "current nbhd is Cedar Riverside\n",
      "current nbhd is Prospect Park - East River Road\n",
      "current nbhd is University of Minnesota\n",
      "current nbhd is Jordan\n",
      "current nbhd is Hawthorne\n",
      "current nbhd is Willard - Hay\n",
      "current nbhd is Near - North\n",
      "current nbhd is Harrison\n",
      "current nbhd is Loring Park\n",
      "current nbhd is Steven's Square - Loring Heights\n",
      "current nbhd is Minnehaha\n",
      "current nbhd is Diamond Lake\n",
      "current nbhd is Cooper\n",
      "current nbhd is Tangletown\n",
      "current nbhd is North Loop\n",
      "current nbhd is Hale\n",
      "current nbhd is Keewaydin\n",
      "current nbhd is Fulton\n",
      "current nbhd is Lynnhurst\n",
      "current nbhd is Bancroft\n",
      "current nbhd is Standish\n",
      "current nbhd is Seward\n",
      "current nbhd is Longfellow\n",
      "current nbhd is Lowry Hill East\n",
      "current nbhd is ECCO\n",
      "current nbhd is South Uptown\n",
      "South Uptown, Minneapolis, MN is not found on geopy.\n",
      "current nbhd is East Phillips\n",
      "current nbhd is Cleveland\n",
      "current nbhd is Wenonah\n",
      "current nbhd is Holland\n",
      "current nbhd is Audubon Park\n",
      "current nbhd is Regina\n",
      "current nbhd is Northrop\n",
      "current nbhd is Sheridan\n",
      "current nbhd is Logan Park\n",
      "current nbhd is Windom Park\n",
      "current nbhd is Linden Hills\n",
      "current nbhd is East Harriet\n",
      "current nbhd is King Field\n",
      "current nbhd is Morris Park\n",
      "current nbhd is Mid - City Industrial\n",
      "Mid - City Industrial, Minneapolis, MN is not found on geopy.\n",
      "current nbhd is Midtown Phillips\n",
      "current nbhd is West Calhoun\n",
      "current nbhd is Bryn - Mawr\n",
      "current nbhd is Northeast Park\n",
      "current nbhd is St. Anthony West\n",
      "current nbhd is St. Anthony East\n",
      "current nbhd is Central\n",
      "current nbhd is Powderhorn Park\n",
      "current nbhd is Corcoran\n",
      "current nbhd is Bryant\n",
      "current nbhd is Kenny\n",
      "current nbhd is Armatage\n",
      "current nbhd is Beltrami\n",
      "current nbhd is Cedar - Isles - Dean\n",
      "current nbhd is Kenwood\n",
      "current nbhd is Lowry Hill\n",
      "current nbhd is East Isles\n",
      "current nbhd is Elliot Park\n",
      "current nbhd is Nicollet Island - East Bank\n",
      "Nicollet Island - East Bank, Minneapolis, MN is not found on geopy.\n",
      "current nbhd is Marcy Holmes\n",
      "current nbhd is Folwell\n",
      "current nbhd is McKinley\n",
      "current nbhd is Camden Industrial\n",
      "current nbhd is Columbia Park\n",
      "current nbhd is Waite Park\n",
      "current nbhd is Marshall Terrace\n",
      "current nbhd is Whittier\n",
      "current nbhd is Lyndale\n",
      "['Downtown West', 'Ventura Village', 'South Uptown', 'Mid - City Industrial', 'Nicollet Island - East Bank']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Phillips West</td>\n",
       "      <td>44.9539</td>\n",
       "      <td>-93.2663</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Downtown West</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Downtown East</td>\n",
       "      <td>44.975</td>\n",
       "      <td>-93.2599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Ventura Village</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Sumner - Glenwood</td>\n",
       "      <td>44.9837</td>\n",
       "      <td>-93.2914</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        Neighborhood Latitude Longitude\n",
       "0      Phillips West  44.9539  -93.2663\n",
       "1      Downtown West      NaN       NaN\n",
       "2      Downtown East   44.975  -93.2599\n",
       "3    Ventura Village      NaN       NaN\n",
       "4  Sumner - Glenwood  44.9837  -93.2914"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Initalize the dataframe\n",
    "column_names = ['Neighborhood','Latitude', 'Longitude']\n",
    "df_nbhds_mpls = pd.DataFrame(columns=column_names)\n",
    "\n",
    "# Fill in the neighborhood names. The latitude and longitude are filled in the next step.\n",
    "for data in mpls_nbhds:\n",
    "    nbhd_name = data['properties']['BDNAME']\n",
    "    df_nbhds_mpls = df_nbhds_mpls.append({'Neighborhood':nbhd_name}, ignore_index=True)\n",
    "\n",
    "# Populate lat/lon values for neighborhoods. Gather values not found by Geopy.\n",
    "missing_latlons = []\n",
    "for nbhd in df_nbhds_mpls['Neighborhood']:\n",
    "    print('current nbhd is', nbhd)\n",
    "    address = nbhd + ', Minneapolis, MN'\n",
    "    geolocator = Nominatim(user_agent=\"mn_explorer\")\n",
    "    location = geolocator.geocode(address, timeout=10)\n",
    "    if location is None:\n",
    "        print(address+' is not found on geopy.')\n",
    "        missing_latlons = missing_latlons +[nbhd]\n",
    "    else:\n",
    "        df_nbhds_mpls.loc[df_nbhds_mpls.index[df_nbhds_mpls['Neighborhood'] == nbhd].to_list(),'Latitude']= location.latitude\n",
    "        df_nbhds_mpls.loc[df_nbhds_mpls.index[df_nbhds_mpls['Neighborhood'] == nbhd].to_list(),'Longitude']= location.longitude\n",
    "    \n",
    "print(missing_latlons)\n",
    "df_nbhds_mpls.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are 5 neighborhoods that Geopy could not identify. We can manually input these by pulling the location data from a google search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Manually look up missing lat/lon values for these neighborhoods.\n",
    "Downtown_West =[44.9742, -93.2733]\n",
    "Ventura_Village = [44.9618, -93.2582]\n",
    "# This neighborhood is now found on geopy...\n",
    "#Humboldt_Industrial_Area = [45.0421, -93.3077]\n",
    "South_Uptown = [44.9411, -93.2911]\n",
    "Mid_City_Industrial = [44.9989, -93.2178]\n",
    "Nicollet_Island = [44.9879, -93.2629]\n",
    "\n",
    "missing_mpls = [Downtown_West, Ventura_Village, South_Uptown, Mid_City_Industrial, Nicollet_Island]\n",
    "\n",
    "# Populate dictionary of neighborhoods and missing lat/long values\n",
    "missing_dict={}\n",
    "for j in range(0,len(missing_mpls)):\n",
    "    entry ={missing_latlons[j]:missing_mpls[j]}\n",
    "    missing_dict.update(entry)\n",
    "\n",
    "\n",
    "# Fill in missing lat/long values into our dataframe\n",
    "for j in range(0,len(missing_mpls)):\n",
    "    value=df_nbhds_mpls.index[df_nbhds_mpls['Neighborhood']==missing_latlons[j]]\n",
    "    df_nbhds_mpls.loc[value[0], 'Latitude'] = missing_dict[missing_latlons[j]][0]\n",
    "    df_nbhds_mpls.loc[value[0], 'Longitude'] = missing_dict[missing_latlons[j]][1]\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## St. Paul Neighborhoods\n",
    "Repeat the above steps to create a dataframe of the St. Paul neighborhoods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Macalester-Groveland, St. Paul, MN is not found on geopy.\n",
      "West Side Community Organization, St. Paul, MN is not found on geopy.\n",
      "Summit Hill Association, St. Paul, MN is not found on geopy.\n",
      "Summit-University, St. Paul, MN is not found on geopy.\n",
      "Thomas-Dale/Frogtown, St. Paul, MN is not found on geopy.\n",
      "West 7th Federation/Fort Road, St. Paul, MN is not found on geopy.\n",
      "CapitolRiver Council, St. Paul, MN is not found on geopy.\n",
      "Payne-Phalen, St. Paul, MN is not found on geopy.\n",
      "The Greater East Side, St. Paul, MN is not found on geopy.\n",
      "Eastview-Conway-Battle Creek-Highwood Hills, St. Paul, MN is not found on geopy.\n",
      "Union Park, St. Paul, MN is not found on geopy.\n",
      "['Macalester-Groveland', 'West Side Community Organization', 'Summit Hill Association', 'Summit-University', 'Thomas-Dale/Frogtown', 'West 7th Federation/Fort Road', 'CapitolRiver Council', 'Payne-Phalen', 'The Greater East Side', 'Eastview-Conway-Battle Creek-Highwood Hills', 'Union Park']\n"
     ]
    }
   ],
   "source": [
    "# Initalize the dataframe\n",
    "column_names = ['Neighborhood','Latitude', 'Longitude']\n",
    "df_nbhds_stp = pd.DataFrame(columns=column_names)\n",
    "\n",
    "\n",
    "# Fill in the neighborhood names. The latitude and longitude are filled in the next step.\n",
    "for data in stp_nbhds:\n",
    "    nbhd_name = data['properties']['name2']\n",
    "    df_nbhds_stp = df_nbhds_stp.append({'Neighborhood':nbhd_name}, ignore_index=True)\n",
    "    \n",
    "# Populate Lat/Long using Geopy\n",
    "missing_latlons = []\n",
    "for nbhd in df_nbhds_stp['Neighborhood']:\n",
    "    address = nbhd + ', St. Paul, MN'\n",
    "    geolocator = Nominatim(user_agent=\"geo_explorer\")\n",
    "    location = geolocator.geocode(address)\n",
    "    if location is None:\n",
    "        print(address+' is not found on geopy.')\n",
    "        missing_latlons = missing_latlons +[nbhd]\n",
    "    else:\n",
    "        df_nbhds_stp.loc[df_nbhds_stp.index[df_nbhds_stp['Neighborhood'] == nbhd].to_list(),'Latitude']= location.latitude\n",
    "        df_nbhds_stp.loc[df_nbhds_stp.index[df_nbhds_stp['Neighborhood'] == nbhd].to_list(),'Longitude']= location.longitude\n",
    "\n",
    "print(missing_latlons)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We manually insert the data for the missing St. Paul neighborhoods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Neighborhood</th>\n",
       "      <th>Latitude</th>\n",
       "      <th>Longitude</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>St Anthony Park</td>\n",
       "      <td>44.9792</td>\n",
       "      <td>-93.1935</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Como</td>\n",
       "      <td>44.9759</td>\n",
       "      <td>-93.1313</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>North End</td>\n",
       "      <td>53.7432</td>\n",
       "      <td>-2.39092</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Hamline-Midway</td>\n",
       "      <td>44.9629</td>\n",
       "      <td>-93.1669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Macalester-Groveland</td>\n",
       "      <td>44.9343</td>\n",
       "      <td>-93.167</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Neighborhood Latitude Longitude\n",
       "0       St Anthony Park  44.9792  -93.1935\n",
       "1                  Como  44.9759  -93.1313\n",
       "2             North End  53.7432  -2.39092\n",
       "3        Hamline-Midway  44.9629  -93.1669\n",
       "4  Macalester-Groveland  44.9343   -93.167"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "mac_grove=[44.9343,-93.1670]\n",
    "west_side_comm_org=[44.9298, -93.0780]\n",
    "summit_hill = [44.9399, -93.1364]\n",
    "summit_uni = [44.9518, -93.1263]\n",
    "thomas_dale_frogtown = [44.9594, -93.1212]\n",
    "west_seventh = [44.9277, -93.1270]\n",
    "capitol_river_council = [44.9512, -93.0952]\n",
    "payne_phalen = [44.9775, -93.0660]\n",
    "greater_east_side = [44.9741, -93.0240]\n",
    "eastview_conway = [44.9525, -93.0117]\n",
    "union_park = [44.9484, -93.1746]\n",
    "\n",
    "missing_stp = [mac_grove, west_side_comm_org, summit_hill, summit_uni, thomas_dale_frogtown, west_seventh,\n",
    "             capitol_river_council, payne_phalen, greater_east_side, eastview_conway, union_park ]\n",
    "\n",
    "# Populate dictionary of neighborhoods and missing lat/long values\n",
    "missing_dict={}\n",
    "for j in range(0,len(missing_stp)):\n",
    "    entry ={missing_latlons[j]:missing_stp[j]}\n",
    "    missing_dict.update(entry)\n",
    "\n",
    "# Fill in missing lat/long values into our dataframe\n",
    "for j in range(0,len(missing_stp)):\n",
    "    value=df_nbhds_stp.index[df_nbhds_stp['Neighborhood']==missing_latlons[j]]\n",
    "    df_nbhds_stp.loc[value[0], 'Latitude'] = missing_dict[missing_latlons[j]][0]\n",
    "    df_nbhds_stp.loc[value[0], 'Longitude'] = missing_dict[missing_latlons[j]][1]\n",
    "\n",
    "df_nbhds_stp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Put the Minneapolis and St. Paul neighborhoods into one dataframe which we will call `nbhds`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "nbhds = pd.concat([df_nbhds_mpls,df_nbhds_stp]).reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use FourSquare API to gather data about breweries in Minneapolis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(99, 6)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>index</th>\n",
       "      <th>venue.id</th>\n",
       "      <th>venue.name</th>\n",
       "      <th>venue.location.address</th>\n",
       "      <th>venue.location.lat</th>\n",
       "      <th>venue.location.lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>538bf392498e382b6fa108b4</td>\n",
       "      <td>Eastlake Craft Brewery</td>\n",
       "      <td>920 E Lake St #123</td>\n",
       "      <td>44.948491</td>\n",
       "      <td>-93.260416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>56242508498e6aeb80142c2c</td>\n",
       "      <td>Lakes &amp; Legends Brewing Company</td>\n",
       "      <td>1368 Lasalle Ave</td>\n",
       "      <td>44.968908</td>\n",
       "      <td>-93.279479</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>52e6ad8c498efd5a184d1105</td>\n",
       "      <td>LynLake Brewery</td>\n",
       "      <td>2934 Lyndale Ave S</td>\n",
       "      <td>44.948919</td>\n",
       "      <td>-93.288345</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>5aa9baebff03062a4b1dccba</td>\n",
       "      <td>Finnegans House</td>\n",
       "      <td>817 5th Ave S</td>\n",
       "      <td>44.972301</td>\n",
       "      <td>-93.266410</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>52d4225b498e07070c180a62</td>\n",
       "      <td>Sisyphus Brewing</td>\n",
       "      <td>712 Ontario Ave W</td>\n",
       "      <td>44.973214</td>\n",
       "      <td>-93.289040</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   index                  venue.id                       venue.name  \\\n",
       "0      0  538bf392498e382b6fa108b4           Eastlake Craft Brewery   \n",
       "1      1  56242508498e6aeb80142c2c  Lakes & Legends Brewing Company   \n",
       "2      2  52e6ad8c498efd5a184d1105                  LynLake Brewery   \n",
       "3      3  5aa9baebff03062a4b1dccba                  Finnegans House   \n",
       "4      4  52d4225b498e07070c180a62                 Sisyphus Brewing   \n",
       "\n",
       "  venue.location.address  venue.location.lat  venue.location.lng  \n",
       "0     920 E Lake St #123           44.948491          -93.260416  \n",
       "1       1368 Lasalle Ave           44.968908          -93.279479  \n",
       "2     2934 Lyndale Ave S           44.948919          -93.288345  \n",
       "3          817 5th Ave S           44.972301          -93.266410  \n",
       "4      712 Ontario Ave W           44.973214          -93.289040  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Will drop these columns from the search results.\n",
    "drop_columns = ['referralId', 'reasons.count', 'reasons.items', 'venue.location.labeledLatLngs',\n",
    "                'venue.photos.count', 'venue.photos.groups', 'venue.location.postalCode',\n",
    "                'venue.location.cc', 'venue.location.city', 'venue.location.state', 'venue.location.country',\n",
    "                'venue.location.formattedAddress', 'venue.venuePage.id', 'venue.delivery.id', 'venue.delivery.url',\n",
    "                'venue.delivery.provider.name', 'venue.delivery.provider.icon.prefix', 'venue.delivery.provider.icon.sizes',\n",
    "                'venue.delivery.provider.icon.name', 'venue.location.neighborhood', 'venue.categories',\n",
    "                'venue.location.distance', 'venue.location.crossStreet']\n",
    "\n",
    "\n",
    "# Define FourSquare credentials\n",
    "CLIENT_ID = 'JSN5MX1DKF5XI3CXVZADJMU5LZE5FMLT2COF00LRJDFMFWIK' # your Foursquare ID\n",
    "CLIENT_SECRET = 'ZST1WYPJCG2J2LGQGUER23BPAC1OMF1BKYC4WQKSSRD3WC1T' # your Foursquare Secret\n",
    "VERSION = '20200601' # Foursquare API version\n",
    "\n",
    "category = '50327c8591d4c4b30a586d5d' # Foursquare category for brewery\n",
    "radius = 3500 # in meters\n",
    "LIMIT = 200\n",
    "\n",
    "# Initialize df_brew\n",
    "df_brew =pd.DataFrame(columns=['venue.id', 'venue.name', 'venue.location.address',\n",
    "       'venue.location.lat', 'venue.location.lng'])\n",
    "\n",
    "# Look at a 5 km radius area centered at each of the neighbhorhood lat/lon coordinates.\n",
    "# Do FourSquare API call for each neighborhood, locating breweries. Store results in df_brew.\n",
    "for j in range(0,len(nbhds)):\n",
    "    lat=nbhds.loc[j,'Latitude']\n",
    "    lon=nbhds.loc[j,'Longitude']\n",
    "    url='https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&v={}&ll={},{}&categoryId={}&radius={}&limit={}'\\\n",
    "        .format(CLIENT_ID, CLIENT_SECRET, VERSION, lat, lon, category, radius, LIMIT)\n",
    "    results=requests.get(url).json()\n",
    "    venues = results['response']['groups'][0]['items']\n",
    "    df_temp = pd.json_normalize(venues)\n",
    "    df_brew = pd.concat([df_brew, df_temp])\n",
    "\n",
    "# Drop unwanted columns, eliminate duplicates, reset index to arrive at final df_brew dataframe.    \n",
    "df_brew.drop(columns=drop_columns, inplace=True)\n",
    "df_brew.drop_duplicates(inplace=True)\n",
    "df_brew.reset_index(inplace=True)\n",
    "print(df_brew.shape)\n",
    "df_brew.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By looking at the names of the breweries, we identify those that are clearly coffee shops, and remove them from the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'12welve Eyes Brewing',\n",
       " '56 Brewing',\n",
       " '612Brew',\n",
       " 'Able Seedhouse + Brewery',\n",
       " 'Bad Weather Brewing Company',\n",
       " 'Bang Brewing Company',\n",
       " \"Barley John's Brewpub\",\n",
       " 'Barrel Theory Beer Company',\n",
       " 'Bauhaus Brew Labs',\n",
       " \"Bennett's Chop and Railhouse\",\n",
       " 'Big Clock Brewery',\n",
       " \"Birch's Lowertown\",\n",
       " 'Black & Tan Brewery',\n",
       " 'BlackStack Brewing',\n",
       " 'Brewmenn Brewing',\n",
       " 'Broken Clock Brewing Cooperative',\n",
       " 'Burning Brothers Brewing',\n",
       " 'Bus Stop Burgers and Brewhouse',\n",
       " 'Chatterbox Pub',\n",
       " 'Clockwerks Brewing',\n",
       " 'Clutch Brewing',\n",
       " 'Craft Beer Corner',\n",
       " 'Dangerous Man Brewing Co',\n",
       " 'Dangerous Man Growler Shop',\n",
       " 'Day Block Brewing Company',\n",
       " 'Dual Citizen Brewing Company',\n",
       " \"Dusty's Bar\",\n",
       " 'Eastlake Craft Brewery',\n",
       " 'FINNEGANS Headquarters',\n",
       " 'Fair State Brewing Cooperative',\n",
       " 'Fair State Co-op Production Brewery',\n",
       " 'Falling Knife Brewing',\n",
       " 'Finnegans House',\n",
       " 'Flat Earth Brewing Company',\n",
       " \"Fordham's Beer Company\",\n",
       " 'Forgotten Star Brewing',\n",
       " 'Fulton Brewing Company',\n",
       " 'Headflyer Brewing',\n",
       " 'HopCat',\n",
       " 'Inbound BrewCo',\n",
       " 'Indeed Brewing Company',\n",
       " 'Insight Brewing',\n",
       " \"Jackson's Hole\",\n",
       " 'La Doña Cervecería',\n",
       " 'Lake Monster Brewing',\n",
       " 'Lakes & Legends Brewing Company',\n",
       " 'LynLake Brewery',\n",
       " 'Metro Warehouse Liquor',\n",
       " 'Minneapolis Cider Company',\n",
       " 'Minneapolis Town Hall Brewery',\n",
       " 'Modist Brewing Co',\n",
       " 'Norseman Distillery',\n",
       " 'Northbound Smokehouse and Brewpub',\n",
       " 'Northern Brewer',\n",
       " 'Northgate Brewing',\n",
       " 'Number 12 Cider',\n",
       " 'ONE Fermentary & Taproom',\n",
       " 'Payne Brewing',\n",
       " 'Pig Ate My Pizza',\n",
       " 'Pizza Lucé',\n",
       " 'Pryes Brewing Company',\n",
       " 'Republic',\n",
       " 'Richfield Liquors',\n",
       " 'Rock Bottom Brewery Minneapolis-St. Paul International Airport',\n",
       " 'Rock Bottom Restaurant & Brewery',\n",
       " 'Saint Paul Brewing',\n",
       " \"Schmidt Organic Water - Schmidt's Brewery\",\n",
       " 'SiP Coffee Bar',\n",
       " 'Sisyphus Brewing',\n",
       " 'Sociable Cider Werks',\n",
       " 'Spyhouse Coffee',\n",
       " 'Stacked Deck Brewing Co.',\n",
       " 'Steel Toe Brewing',\n",
       " \"Stout's Pub\",\n",
       " 'Summit Brewing Company',\n",
       " 'Surly Brewing Co',\n",
       " 'Surly Brewing Company',\n",
       " 'Surlyfest',\n",
       " 'The Ball Park Cafe',\n",
       " 'The Beamer Bar',\n",
       " 'The Fixer Upper',\n",
       " 'The Freehouse',\n",
       " 'The Herkimer Pub & Brewery',\n",
       " 'The Lab - Brewery, Pilot Facility & Taproom',\n",
       " 'The Summit',\n",
       " 'Tin Whiskers Brewing Co.',\n",
       " 'Town Hall Lanes',\n",
       " 'Twin Spirits Distillery',\n",
       " 'Urban Growler Brewing Company',\n",
       " 'Utepils Brewing Co.',\n",
       " 'Venn Brewing',\n",
       " 'Vine Park Brewing Co.',\n",
       " 'Wabasha Brewing Company',\n",
       " 'Waldmann Brewery & Wurstery',\n",
       " 'White Bull',\n",
       " 'Wicked Wort',\n",
       " 'Wild Mind Artisan Ales',\n",
       " 'moto-i'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "set(df_brew['venue.name'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the following venues from df_brew.\n",
    "venue_brew_drop = {'Metro Warehouse Liquor', 'Pizza Lucé', 'SiP Coffee Bar', 'Spyhouse Coffee'}\n",
    "for j in venue_brew_drop:\n",
    "    df_brew.drop(df_brew.index[df_brew['venue.name']==j], inplace=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use FourSquare API to gather data about restaurants in Twin Cities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(1570, 5)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>venue.id</th>\n",
       "      <th>venue.name</th>\n",
       "      <th>venue.location.address</th>\n",
       "      <th>venue.location.lat</th>\n",
       "      <th>venue.location.lng</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>44b492d9f964a52057351fe3</td>\n",
       "      <td>Midtown Global Market</td>\n",
       "      <td>920 E Lake St</td>\n",
       "      <td>44.948589</td>\n",
       "      <td>-93.260502</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>49fb95f5f964a5205d6e1fe3</td>\n",
       "      <td>Taqueria La Hacienda</td>\n",
       "      <td>334 E Lake St #101</td>\n",
       "      <td>44.948605</td>\n",
       "      <td>-93.271181</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>44f1f813f964a5200a381fe3</td>\n",
       "      <td>Quang Restaurant</td>\n",
       "      <td>2719 Nicollet Ave</td>\n",
       "      <td>44.953136</td>\n",
       "      <td>-93.277745</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>50255babe4b0bc802b45360b</td>\n",
       "      <td>FIKA</td>\n",
       "      <td>2600 Park Ave</td>\n",
       "      <td>44.954555</td>\n",
       "      <td>-93.265717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>49e4c13ef964a52034631fe3</td>\n",
       "      <td>Manny's Tortas</td>\n",
       "      <td>920 E Lake St</td>\n",
       "      <td>44.948679</td>\n",
       "      <td>-93.260649</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   venue.id             venue.name venue.location.address  \\\n",
       "0  44b492d9f964a52057351fe3  Midtown Global Market          920 E Lake St   \n",
       "1  49fb95f5f964a5205d6e1fe3   Taqueria La Hacienda     334 E Lake St #101   \n",
       "2  44f1f813f964a5200a381fe3       Quang Restaurant      2719 Nicollet Ave   \n",
       "3  50255babe4b0bc802b45360b                   FIKA          2600 Park Ave   \n",
       "4  49e4c13ef964a52034631fe3         Manny's Tortas          920 E Lake St   \n",
       "\n",
       "   venue.location.lat  venue.location.lng  \n",
       "0           44.948589          -93.260502  \n",
       "1           44.948605          -93.271181  \n",
       "2           44.953136          -93.277745  \n",
       "3           44.954555          -93.265717  \n",
       "4           44.948679          -93.260649  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Will drop these columns from the search results.\n",
    "drop_columns = ['referralId', 'reasons.count', 'reasons.items', 'venue.location.labeledLatLngs',\n",
    "                'venue.photos.count', 'venue.photos.groups', 'venue.location.postalCode',\n",
    "                'venue.location.cc', 'venue.location.city', 'venue.location.state', 'venue.location.country',\n",
    "                'venue.location.formattedAddress', 'venue.venuePage.id', 'venue.delivery.id', 'venue.delivery.url',\n",
    "                'venue.delivery.provider.name', 'venue.delivery.provider.icon.prefix', 'venue.delivery.provider.icon.sizes',\n",
    "                'venue.delivery.provider.icon.name', 'venue.location.neighborhood', 'venue.categories',\n",
    "                'venue.location.distance', 'venue.location.crossStreet']\n",
    "\n",
    "category = '4d4b7105d754a06374d81259' # Foursquare category for food (restaurants)\n",
    "radius = 2000 # in meters\n",
    "LIMIT = 200\n",
    "\n",
    "# Initialize df_food\n",
    "df_food =pd.DataFrame(columns=['venue.id', 'venue.name', 'venue.location.address',\n",
    "       'venue.location.lat', 'venue.location.lng'])\n",
    "\n",
    "# Look at a 2 km radius area centered at each of the Minneapolis neighbhorhoods.\n",
    "# Do FourSquare API call for each neighborhood, locating restaurants. Store results in df_food.\n",
    "counter = 0\n",
    "for j in range(0,len(df_nbhds_mpls)):\n",
    "    lat=df_nbhds_mpls.loc[j,'Latitude']\n",
    "    lon=df_nbhds_mpls.loc[j,'Longitude']\n",
    "    url='https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&v={}&ll={},{}&categoryId={}&radius={}&limit={}'\\\n",
    "        .format(CLIENT_ID, CLIENT_SECRET, VERSION, lat, lon, category, radius, LIMIT)\n",
    "    results=requests.get(url).json()\n",
    "    venues = results['response']['groups'][0]['items']\n",
    "    df_temp = pd.json_normalize(venues)\n",
    "#    print('df_temp shape is', df_temp.shape, '. Counter is ', counter)\n",
    "    df_food = pd.concat([df_food, df_temp])\n",
    "    #print(counter)\n",
    "    #counter+=1\n",
    "    \n",
    "# Repeat above call but with St. Paul neighborhoods. Increase the radius size.\n",
    "radius = 3500\n",
    "counter = 0\n",
    "for j in range(0,len(df_nbhds_stp)):\n",
    "    lat=df_nbhds_stp.loc[j,'Latitude']\n",
    "    lon=df_nbhds_stp.loc[j,'Longitude']\n",
    "    url='https://api.foursquare.com/v2/venues/explore?client_id={}&client_secret={}&v={}&ll={},{}&categoryId={}&radius={}&limit={}'\\\n",
    "        .format(CLIENT_ID, CLIENT_SECRET, VERSION, lat, lon, category, radius, LIMIT)\n",
    "    results=requests.get(url).json()\n",
    "    venues = results['response']['groups'][0]['items']\n",
    "    df_temp = pd.json_normalize(venues)\n",
    "    #print('df_temp shape is', df_temp.shape, '. Counter is ', counter)\n",
    "    df_food = pd.concat([df_food, df_temp])\n",
    "    #print(counter)\n",
    "    counter+=1\n",
    "\n",
    "# Drop unwanted columns, eliminate duplicates, reset index to arrive at final df_food dataframe.    \n",
    "df_food.drop(columns=drop_columns, inplace=True)\n",
    "df_food.drop_duplicates(inplace=True)\n",
    "df_food.reset_index(drop=True, inplace=True)\n",
    "print(df_food.shape)\n",
    "df_food.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Make geodataframes of the neighborhoods\n",
    "Minneapolis has neighborhoods and communities. The communities encompass several neighborhoods in general. The area size of the communities is generally closer to the size of the St. Paul neighborhoods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(104, 3)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>geometry</th>\n",
       "      <th>BDNAME</th>\n",
       "      <th>city</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MULTIPOLYGON (((-93.26258 44.96091, -93.26258 ...</td>\n",
       "      <td>Phillips West</td>\n",
       "      <td>Minneapolis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MULTIPOLYGON (((-93.26011 44.98300, -93.26010 ...</td>\n",
       "      <td>Downtown West</td>\n",
       "      <td>Minneapolis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MULTIPOLYGON (((-93.24499 44.97893, -93.24499 ...</td>\n",
       "      <td>Downtown East</td>\n",
       "      <td>Minneapolis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MULTIPOLYGON (((-93.24958 44.96630, -93.24951 ...</td>\n",
       "      <td>Ventura Village</td>\n",
       "      <td>Minneapolis</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MULTIPOLYGON (((-93.28830 44.98904, -93.28830 ...</td>\n",
       "      <td>Sumner - Glenwood</td>\n",
       "      <td>Minneapolis</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            geometry             BDNAME  \\\n",
       "0  MULTIPOLYGON (((-93.26258 44.96091, -93.26258 ...      Phillips West   \n",
       "1  MULTIPOLYGON (((-93.26011 44.98300, -93.26010 ...      Downtown West   \n",
       "2  MULTIPOLYGON (((-93.24499 44.97893, -93.24499 ...      Downtown East   \n",
       "3  MULTIPOLYGON (((-93.24958 44.96630, -93.24951 ...    Ventura Village   \n",
       "4  MULTIPOLYGON (((-93.28830 44.98904, -93.28830 ...  Sumner - Glenwood   \n",
       "\n",
       "          city  \n",
       "0  Minneapolis  \n",
       "1  Minneapolis  \n",
       "2  Minneapolis  \n",
       "3  Minneapolis  \n",
       "4  Minneapolis  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Make Minneapolis neighborhoods\n",
    "mpls_nbhd_geo = gpd.read_file('Minneapolis_Neighborhoods.geojson.json')\n",
    "mpls_nbhd_geo = mpls_nbhd_geo[['geometry', 'BDNAME']]\n",
    "mpls_nbhd_geo['city']=pd.Series(['Minneapolis']*len(mpls_nbhd_geo))\n",
    "\n",
    "# Minneapolis communities are closer in size to the St. Paul neighborhoods.\n",
    "# If we prefer to consider Minneapolis communities and St. Paul neighborhoods, comment the above code and\n",
    "# run the following instead.\n",
    "\n",
    "#mpls_comm_geo = gpd.read_file('Communities.geojson')\n",
    "#mpls_comm_geo = mpls_comm_geo[['geometry', 'CommName']]\n",
    "#mpls_comm_geo['city']=pd.Series(['Minneapolis']*len(mpls_comm_geo))\n",
    "#mpls_comm_geo.rename(columns={'CommName':'BDNAME'}, inplace=True) # Rename so columns match\n",
    "\n",
    "# Make St. Paul neighborhoods\n",
    "stp_geo = gpd.read_file('stp_neighborhoods.geojson')\n",
    "stp_geo = stp_geo[['geometry', 'name2']]\n",
    "stp_geo.rename(columns={'name2':'BDNAME'}, inplace=True) # Rename so columns match\n",
    "stp_geo['city']=pd.Series(['St. Paul']*len(stp_geo))\n",
    "\n",
    "# Combine to twin cities neighborhoods\n",
    "tc_geo=pd.concat([mpls_nbhd_geo, stp_geo]).drop_duplicates().reset_index(drop=True)\n",
    "print(tc_geo.shape)\n",
    "tc_geo.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Assign neighborhoods and communities to breweries and neighborhoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn df_brew into geodatafram\n",
    "df_brew_geo = gpd.GeoDataFrame(df_brew, geometry=gpd.points_from_xy(df_brew['venue.location.lng'], df_brew['venue.location.lat']))\n",
    "#df_brew_geo.reset_index(inplace=True)\n",
    "\n",
    "brewery_count=[]\n",
    "for j in range(0,len(tc_geo)):\n",
    "    nbhd = tc_geo.loc[j,'BDNAME']  # nbhd is jth neighborhood name, string\n",
    "    nbhd_geo = tc_geo[tc_geo['BDNAME'] == nbhd]  # nbhd_geo is jth row\n",
    "    nbhd_geo.reset_index(drop=True, inplace=True)  # reset index of dataframe\n",
    "    nbhd_mask = df_brew_geo['geometry'].within(nbhd_geo.loc[0,'geometry'])  # Boolean mask of breweries in nbhd.\n",
    "    brewery_count.append(sum(nbhd_mask))  # create list of brewery counts per neighborhood.\n",
    "\n",
    "# Add column to tc_geo    \n",
    "tc_geo['Brewery Count'] = pd.Series(brewery_count) # add brewery count column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "79\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "95"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(tc_geo['Brewery Count'].sum())\n",
    "len(df_brew)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn df_food into geodataframe\n",
    "df_food_geo = gpd.GeoDataFrame(df_food, geometry=gpd.points_from_xy(df_food['venue.location.lng'], df_food['venue.location.lat']))\n",
    "\n",
    "food_count=[]\n",
    "for j in range(0,len(tc_geo)):\n",
    "    nbhd = tc_geo.loc[j,'BDNAME']  # nbhd is jth neighborhood name, string\n",
    "    nbhd_geo = tc_geo[tc_geo['BDNAME'] == nbhd]  # nbhd_geo is jth row\n",
    "    nbhd_geo.reset_index(drop=True, inplace=True)  # reset index of dataframe\n",
    "    nbhd_mask = df_food_geo['geometry'].within(nbhd_geo.loc[0,'geometry'])  # Boolean mask of breweries in nbhd.\n",
    "    food_count.append(sum(nbhd_mask))  # create list of brewery counts per neighborhood.\n",
    "\n",
    "# Add column to tc_geo    \n",
    "tc_geo['Restaurant Count'] = pd.Series(food_count) # add brewery count column\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>geometry</th>\n",
       "      <th>BDNAME</th>\n",
       "      <th>city</th>\n",
       "      <th>Brewery Count</th>\n",
       "      <th>Restaurant Count</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>MULTIPOLYGON (((-93.26258 44.96091, -93.26258 ...</td>\n",
       "      <td>Phillips West</td>\n",
       "      <td>Minneapolis</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>MULTIPOLYGON (((-93.26011 44.98300, -93.26010 ...</td>\n",
       "      <td>Downtown West</td>\n",
       "      <td>Minneapolis</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>MULTIPOLYGON (((-93.24499 44.97893, -93.24499 ...</td>\n",
       "      <td>Downtown East</td>\n",
       "      <td>Minneapolis</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>MULTIPOLYGON (((-93.24958 44.96630, -93.24951 ...</td>\n",
       "      <td>Ventura Village</td>\n",
       "      <td>Minneapolis</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>MULTIPOLYGON (((-93.28830 44.98904, -93.28830 ...</td>\n",
       "      <td>Sumner - Glenwood</td>\n",
       "      <td>Minneapolis</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            geometry             BDNAME  \\\n",
       "0  MULTIPOLYGON (((-93.26258 44.96091, -93.26258 ...      Phillips West   \n",
       "1  MULTIPOLYGON (((-93.26011 44.98300, -93.26010 ...      Downtown West   \n",
       "2  MULTIPOLYGON (((-93.24499 44.97893, -93.24499 ...      Downtown East   \n",
       "3  MULTIPOLYGON (((-93.24958 44.96630, -93.24951 ...    Ventura Village   \n",
       "4  MULTIPOLYGON (((-93.28830 44.98904, -93.28830 ...  Sumner - Glenwood   \n",
       "\n",
       "          city  Brewery Count  Restaurant Count  \n",
       "0  Minneapolis              0                 9  \n",
       "1  Minneapolis              4                77  \n",
       "2  Minneapolis              2                13  \n",
       "3  Minneapolis              0                 4  \n",
       "4  Minneapolis              0                 1  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tc_geo.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "current neighborhood is                                             geometry         BDNAME  \\\n",
      "0  MULTIPOLYGON (((-93.26258 44.96091, -93.26258 ...  Phillips West   \n",
      "\n",
      "          city  Brewery Count  Restaurant Count  \n",
      "0  Minneapolis              0                 9  \n"
     ]
    },
    {
     "ename": "KeyError",
     "evalue": "33",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_loc\u001b[0;34m(self, key, method, tolerance)\u001b[0m\n\u001b[1;32m   2645\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2646\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2647\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.Int64HashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.Int64HashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 33",
      "\nDuring handling of the above exception, another exception occurred:\n",
      "\u001b[0;31mKeyError\u001b[0m                                  Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-18-6c8b6e4bd5ec>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     12\u001b[0m \u001b[0;31m#        print('index is', index)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m \u001b[0;31m#        print(\"df_brew_geo['geometry'][index] is\", df_brew_geo['geometry'][index])\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 14\u001b[0;31m         \u001b[0mnbhd_temp_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnbhd_temp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'geometry'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontains\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf_brew_geo\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'geometry'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     15\u001b[0m \u001b[0;31m#        print('here is',nbhd_temp_mask)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m         \u001b[0mnbhd_temp_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf_brew_geo\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'geometry'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwithin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mnbhd_temp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mloc\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m'geometry'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/geopandas/geoseries.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m    301\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    302\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__getitem__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 303\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_wrapped_pandas_method\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"__getitem__\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    304\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    305\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0minherit_doc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSeries\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/geopandas/geoseries.py\u001b[0m in \u001b[0;36m_wrapped_pandas_method\u001b[0;34m(self, mtd, *args, **kwargs)\u001b[0m\n\u001b[1;32m    293\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_wrapped_pandas_method\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmtd\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    294\u001b[0m         \u001b[0;34m\"\"\"Wrap a generic pandas method to ensure it returns a GeoSeries\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 295\u001b[0;31m         \u001b[0mval\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mgetattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mGeoSeries\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmtd\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    296\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mtype\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mval\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0mSeries\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    297\u001b[0m             \u001b[0mval\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__class__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mGeoSeries\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/series.py\u001b[0m in \u001b[0;36m__getitem__\u001b[0;34m(self, key)\u001b[0m\n\u001b[1;32m    869\u001b[0m         \u001b[0mkey\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcom\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mapply_if_callable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    870\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 871\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mindex\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_value\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    872\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    873\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mis_scalar\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_value\u001b[0;34m(self, series, key)\u001b[0m\n\u001b[1;32m   4386\u001b[0m                 \u001b[0;31m# try that\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4387\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 4388\u001b[0;31m                     \u001b[0miloc\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   4389\u001b[0m                     \u001b[0;32mreturn\u001b[0m \u001b[0ms\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0miloc\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   4390\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/lib/python3.7/site-packages/pandas/core/indexes/base.py\u001b[0m in \u001b[0;36mget_loc\u001b[0;34m(self, key, method, tolerance)\u001b[0m\n\u001b[1;32m   2646\u001b[0m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2647\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mKeyError\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2648\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_engine\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_loc\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_maybe_cast_indexer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2649\u001b[0m         \u001b[0mindexer\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_indexer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0mkey\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtolerance\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtolerance\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2650\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mindexer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mndim\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m1\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mindexer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/index.pyx\u001b[0m in \u001b[0;36mpandas._libs.index.IndexEngine.get_loc\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.Int64HashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;32mpandas/_libs/hashtable_class_helper.pxi\u001b[0m in \u001b[0;36mpandas._libs.hashtable.Int64HashTable.get_item\u001b[0;34m()\u001b[0m\n",
      "\u001b[0;31mKeyError\u001b[0m: 33"
     ]
    }
   ],
   "source": [
    "def get_nbhd(index):\n",
    "    return nbhds.loc[index, 'Neighborhood']\n",
    "\n",
    "brew_dict={}\n",
    "# Identify brewery neighborhoods\n",
    "# Return dictionary with brewery names as keys, neighborhoods as values.\n",
    "for j in range(0,len(nbhds)):\n",
    "    nbhd_temp = tc_geo[tc_geo['BDNAME']==get_nbhd(j)] # neighborhood row\n",
    "    nbhd_temp.reset_index(drop=True, inplace=True)\n",
    "    print('current neighborhood is', nbhd_temp)\n",
    "    for index in range(0, len(df_brew)):\n",
    "#        print('index is', index)\n",
    "#        print(\"df_brew_geo['geometry'][index] is\", df_brew_geo['geometry'][index])\n",
    "        nbhd_temp_mask = nbhd_temp.loc[0,'geometry'].contains(df_brew_geo['geometry'][index])\n",
    "#        print('here is',nbhd_temp_mask)\n",
    "        nbhd_temp_mask = df_brew_geo['geometry'][index].within(nbhd_temp.loc[0,'geometry'])\n",
    "#        print(nbhd_temp_mask)\n",
    "        if nbhd_temp_mask == True:           \n",
    "#            print('Brewery {} is in {}'.format(df_brew_geo['venue.name'][index], nbhd_temp.loc[0,'BDNAME'] ))\n",
    "            brew_dict[df_brew_geo['venue.name'][index]]=get_nbhd(j)\n",
    "\n",
    "#brew_dict    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "food_dict={}\n",
    "# Identify restaurant neighborhoods\n",
    "# Return dictionary with restaurant names as keys, neighborhoods as values.\n",
    "for j in range(0,len(nbhds)):\n",
    "    nbhd_temp = tc_geo[tc_geo['BDNAME']==get_nbhd(j)] # neighborhood row\n",
    "    nbhd_temp.reset_index(drop=True, inplace=True)\n",
    "#    print('current neighborhood is', nbhd_temp)\n",
    "    for index in range(0, len(df_food)):\n",
    "#        print('index is', index)\n",
    "#        print(\"df_brew_geo['geometry'][index] is\", df_brew_geo['geometry'][index])\n",
    "        nbhd_temp_mask = nbhd_temp.loc[0,'geometry'].contains(df_food_geo['geometry'][index])\n",
    "#        print('here is',nbhd_temp_mask)\n",
    "        nbhd_temp_mask = df_food_geo['geometry'][index].within(nbhd_temp.loc[0,'geometry'])\n",
    "#        print(nbhd_temp_mask)\n",
    "        if nbhd_temp_mask == True:           \n",
    "#            print('Brewery {} is in {}'.format(df_brew_geo['venue.name'][index], nbhd_temp.loc[0,'BDNAME'] ))\n",
    "            food_dict[df_food_geo['venue.name'][index]]=get_nbhd(j)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choropleth map of breweries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize = (15,15))\n",
    "tc_geo.plot(column='Brewery Count', ax=ax, legend=True, cmap='BuPu', \n",
    "            legend_kwds={'shrink': 0.3}, linewidth=0.95, edgecolor='.3')\n",
    "plt.title('Choropleth map of breweries in Twin Cities neighborhoods')\n",
    "plt.savefig('images/brew_choropleth_gpd', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choropleth map of restaurants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize = (15,15))\n",
    "tc_geo.plot(column='Restaurant Count', ax=ax, legend=True, cmap='BuPu', \n",
    "            legend_kwds={'shrink': 0.3}, linewidth=0.95, edgecolor='.3')\n",
    "plt.title('Choropleth map of restaurants in Twin Cities neighborhoods')\n",
    "plt.savefig('images/food_choropleth_gpd', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing breweries and neighborhoods\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "address = 'Minneapolis, MN'\n",
    "geolocator = Nominatim(user_agent=\"mn_explorer\")\n",
    "location = geolocator.geocode(address)\n",
    "mpls_latitude=location.latitude\n",
    "mpls_longitude=location.longitude\n",
    "\n",
    "address = 'St. Paul, MN'\n",
    "geolocator = Nominatim(user_agent=\"mn_explorer\")\n",
    "location = geolocator.geocode(address)\n",
    "stp_latitude=location.latitude\n",
    "stp_longitude=location.longitude\n",
    "\n",
    "# Center the map between the two cities.\n",
    "avg_lat = (mpls_latitude + stp_latitude)/2\n",
    "avg_lon = (mpls_longitude + stp_longitude)/2\n",
    " \n",
    "tc_map = folium.Map(location=[avg_lat, avg_lon], zoom_start=12, tiles='Stamen Terrain')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add neighborhoods to map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Minneapolis communities\n",
    "#folium.GeoJson(mpls_comm_data).add_to(tc_map)\n",
    "\n",
    "# Minneapolis neighborhoods\n",
    "folium.GeoJson(mpls_data).add_to(tc_map)\n",
    "\n",
    "# St. Paul neighborhoods\n",
    "folium.GeoJson(stp_nbhd_data).add_to(tc_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add breweries to map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create brewery instances\n",
    "breweries = folium.map.FeatureGroup()\n",
    "\n",
    "# add breweries to map\n",
    "for lat, lon in zip(df_brew['venue.location.lat'], df_brew['venue.location.lng']):\n",
    "    folium.features.CircleMarker(\n",
    "        [lat, lon],\n",
    "        radius=5,\n",
    "        color='blue',\n",
    "        fill=True,\n",
    "#        popup=label,\n",
    "        fill_color='magenta',\n",
    "        fill_opacity=0.6\n",
    "    ).add_to(tc_map)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Add restaurants to map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create restaurant instances\n",
    "foods = folium.map.FeatureGroup()\n",
    "\n",
    "# add breweries to map\n",
    "for lat, lon in zip(df_food['venue.location.lat'], df_food['venue.location.lng']):\n",
    "    folium.features.CircleMarker(\n",
    "        [lat, lon],\n",
    "        radius=5,\n",
    "        color='orange',\n",
    "        fill=True,\n",
    "#        popup=label,\n",
    "        fill_color='yellow',\n",
    "        fill_opacity=0.6\n",
    "    ).add_to(tc_map)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tc_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Cluster restaurants\n",
    "This code may be skipped when running. At the moment, not sure it adds too much other than clutter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "restaurant_cluster = plugins.MarkerCluster().add_to(tc_map)\n",
    "\n",
    "# loop through the dataframe and add each data point to the mark cluster\n",
    "for lat, lng in zip(df_food['venue.location.lat'], df_food['venue.location.lng']):\n",
    "    folium.Marker(\n",
    "        location=[lat, lng],\n",
    "        icon=None\n",
    "#        popup=label,\n",
    "    ).add_to(restaurant_cluster)\n",
    "tc_map"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CLuster restaurant regions vs brewery regions\n",
    "\n",
    "Supervised: restaurant vs brewery\n",
    "high density brewery vs high density restaurant\n",
    "Want to build brewery in region lots of restaurants, low brewery density.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# type variable set to 0 for restaurants\n",
    "df_food['type'] = pd.Series(np.zeros(len(df_food)))\n",
    "\n",
    "# type variable set to 1 for breweries\n",
    "df_brew['type'] = pd.Series(np.ones(len(df_brew)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_brew.drop(columns=['level_0','index'],inplace=True)\n",
    "df_brew.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_brew.drop(columns=['index'], inplace=True)\n",
    "df_brew.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([df_food, df_brew]).reset_index(drop=True)\n",
    "print(XX.head(2))\n",
    "print(XX.tail(2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = XX.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = X[:,0:2]\n",
    "y = XX['type'].values\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn import metrics\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.model_selection import cross_val_score\n",
    "from sklearn.impute import SimpleImputer\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=0)\n",
    "\n",
    "k=15\n",
    "my_pipeline = Pipeline(steps=[('preprocessor', SimpleImputer()),\n",
    "                              ('model', KNeighborsClassifier(n_neighbors=k))])\n",
    "\n",
    "scores = -1 * cross_val_score(my_pipeline, X, y,\n",
    "                              cv=5,\n",
    "                              scoring='neg_mean_absolute_error')\n",
    "\n",
    "print(\"MAE scores:\\n\", scores)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k=10\n",
    "neigh = KNeighborsClassifier(n_neighbors=k)\n",
    "neigh.fit(X_train, y_train)\n",
    "\n",
    "yhat = neigh.predict(X_test)\n",
    "\n",
    "print(\"Train set Accuracy: \", metrics.accuracy_score(y_train, neigh.predict(X_train)))\n",
    "print(\"Test set Accuracy: \", metrics.accuracy_score(y_test, yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    " my_pipeline = Pipeline(steps=[('preprocessor', SimpleImputer()),\n",
    "                              ('model', KNeighborsClassifier(n_neighbors=k))])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Ks = 20\n",
    "mean_acc = np.zeros((Ks-1))\n",
    "std_acc = np.zeros((Ks-1))\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=2)\n",
    "\n",
    "mean_scores=[]\n",
    "\n",
    "for n in range(1,Ks):\n",
    "    #Train Model and Predict  \n",
    "    my_pipeline = Pipeline(steps=[('preprocessor', SimpleImputer()),\n",
    "                              ('model', KNeighborsClassifier(n_neighbors=n))])\n",
    "    scores = -1 * cross_val_score(my_pipeline, X, y,\n",
    "                              cv=5,\n",
    "                              scoring='neg_mean_absolute_error')\n",
    "    mean_scores.append(scores.mean())\n",
    "\n",
    "#    neigh = KNeighborsClassifier(n_neighbors = n).fit(X_train,y_train)\n",
    "#    yhat=neigh.predict(X_test)\n",
    "#    mean_acc[n-1] = metrics.accuracy_score(y_test, yhat)\n",
    "\n",
    "    \n",
    "#    std_acc[n-1]=np.std(yhat==y_test)/np.sqrt(yhat.shape[0])\n",
    "\n",
    "#mean_acc\n",
    "\n",
    "plt.plot(range(1,Ks),mean_scores,'g')\n",
    "#plt.fill_between(range(1,Ks),mean_acc - 1 * std_acc,mean_acc + 1 * std_acc, alpha=0.10)\n",
    "#plt.legend(('Accuracy ', '+/- 3xstd'))\n",
    "#plt.ylabel('Accuracy ')\n",
    "plt.ylabel('Mean CV scores')\n",
    "plt.xlabel('Number of Neighbors (K)')\n",
    "plt.xticks(range(1,Ks))\n",
    "plt.grid()\n",
    "#plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(mean_scores[9])\n",
    "print(mean_scores[13])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use 10 neighbors or 14 neighbors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib.colors import ListedColormap\n",
    "from sklearn import neighbors, datasets\n",
    "\n",
    "#n_neighbors = \n",
    "k=2\n",
    "neigh = KNeighborsClassifier(n_neighbors=k)\n",
    "neigh.fit(X_train, y_train)\n",
    "yhat = neigh.predict(X_test)\n",
    "\n",
    "# import some data to play with\n",
    "#iris = datasets.load_iris()\n",
    "\n",
    "# we only take the first two features. We could avoid this ugly\n",
    "# slicing by using a two-dim dataset\n",
    "#X = iris.data[:, :2]\n",
    "#y = iris.target\n",
    "\n",
    "h = 0.0001  # step size in the mesh\n",
    "\n",
    "# Create color maps\n",
    "cmap_light = ListedColormap(['cyan', 'cornflowerblue'])\n",
    "cmap_bold = ListedColormap(['darkorange', 'darkblue'])\n",
    "\n",
    "#for weights in ['uniform', 'distance']:\n",
    "    # we create an instance of Neighbours Classifier and fit the data.\n",
    "    #clf = neighbors.KNeighborsClassifier(n_neighbors, weights=weights)\n",
    "    #clf.fit(X_train, y_train)\n",
    "\n",
    "    # Plot the decision boundary. For that, we will assign a color to each\n",
    "    # point in the mesh [x_min, x_max]x[y_min, y_max].\n",
    "x_min, x_max = 44.9, 45.02\n",
    "y_min, y_max = -93.35, -93.19\n",
    "xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "yhat = neigh.predict(np.c_[xx.ravel(), yy.ravel()])\n",
    "\n",
    "    # Put the result into a color plot\n",
    "yhat = yhat.reshape(xx.shape)\n",
    "plt.figure()\n",
    "plt.pcolormesh(xx, yy, yhat, cmap=cmap_light)\n",
    "\n",
    "    # Plot also the training points\n",
    "plt.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cmap_bold, edgecolor='k', s=20)\n",
    "plt.xlim(xx.min(), xx.max())\n",
    "plt.ylim(yy.min(), yy.max())\n",
    "#plt.title(\"3-Class classification (k = %i, weights = '%s')\"\n",
    "#              % (n_neighbors, weights))\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "[(XX['venue.location.lat'].max()-XX['venue.location.lat'].min())/100, (XX['venue.location.lng'].max()-XX['venue.location.lng'].min())/100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Clustering based off lat, long, and type\n",
    "# Amend! categorical variable no good for cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# type variable set to 0 for restaurants\n",
    "df_food['type'] = pd.Series(np.zeros(len(df_food)))\n",
    "\n",
    "# type variable set to 1 for breweries\n",
    "df_brew['type'] = pd.Series(np.ones(len(df_brew)))\n",
    "\n",
    "print(df_food.head(2))\n",
    "print(df_brew.head(2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.concat([df_food, df_brew]).reset_index(drop=True)\n",
    "print(df.shape)\n",
    "df.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=df[df['venue.location.lng']<-70]\n",
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(df.shape)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.cluster import DBSCAN\n",
    "import sklearn.utils\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "sklearn.utils.check_random_state(1000)\n",
    "Clus_dataSet = df[['venue.location.lng','venue.location.lat','type']]\n",
    "Clus_dataSet = np.nan_to_num(Clus_dataSet)\n",
    "Clus_dataSet = StandardScaler().fit_transform(Clus_dataSet)\n",
    "\n",
    "# Compute DBSCAN\n",
    "db = DBSCAN(eps=0.01, min_samples=6).fit(Clus_dataSet)\n",
    "core_samples_mask = np.zeros_like(db.labels_, dtype=bool)\n",
    "core_samples_mask[db.core_sample_indices_] = True\n",
    "labels = db.labels_\n",
    "df[\"Clus_Db\"]=labels\n",
    "\n",
    "realClusterNum=len(set(labels)) - (1 if -1 in labels else 0)\n",
    "clusterNum = len(set(labels)) \n",
    "\n",
    "\n",
    "# A sample of clusters\n",
    "print(realClusterNum)\n",
    "df[['venue.location.lng','venue.location.lat','type','Clus_Db']].head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.cm as cm\n",
    "\n",
    "fig, ax = plt.subplots(figsize = (30,30))\n",
    "tc_geo.plot(column='Restaurant Count', ax=ax, legend=True, cmap='BuPu', \n",
    "            legend_kwds={'shrink': 0.3}, linewidth=0.95, edgecolor='.3')\n",
    "plt.title('Choropleth map of restaurants in Twin Cities neighborhoods')\n",
    "\n",
    "\n",
    "\n",
    "colors = cm.gnuplot(np.linspace(0, 1, clusterNum))\n",
    "\n",
    "\n",
    "for x, y, c in zip(df['venue.location.lng'], df['venue.location.lat'], colors[df['Clus_Db']]):\n",
    "    plt.scatter(x, y, 9, color=c)\n",
    "    \n",
    "\n",
    "#plt.scatter(df['venue.location.lng'], df['venue.location.lat'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "colors = cm.rainbow(np.linspace(0, 1, clusterNum))\n",
    "colors[[1,2,1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(colors[df['Clus_Db']])\n",
    "len(df['Clus_Db'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(colors[df['Clus_Db']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['color']=colors[df['Clus_Db']+1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cluster with distance to brewery."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute distance to nearest brewery\n",
    "df_brew.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Run k-means"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualize Clusters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choropleth map of brewery density using Geopandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make Minneapolis neighborhoods\n",
    "#mpls_geo = gpd.read_file('Minneapolis_Neighborhoods.geojson.json')\n",
    "#mpls_geo = mpls_geo[['geometry', 'BDNAME']]\n",
    "#mpls_geo['city']=pd.Series(['Minneapolis']*len(mpls_geo))\n",
    "\n",
    "# Minneapolis communities are closer in size to the St. Paul neighborhoods.\n",
    "# If we prefer to consider Minneapolis communities and St. Paul neighborhoods, comment the above code and\n",
    "# run the following instead.\n",
    "\n",
    "mpls_geo = gpd.read_file('Communities.geojson')\n",
    "mpls_geo = mpls_geo[['geometry', 'CommName']]\n",
    "mpls_geo['city']=pd.Series(['Minneapolis']*len(mpls_geo))\n",
    "mpls_geo.rename(columns={'CommName':'BDNAME'}, inplace=True) # Rename so columns match\n",
    "\n",
    "# Make St. Paul neighborhoods\n",
    "stp_geo = gpd.read_file('stp_neighborhoods.geojson')\n",
    "stp_geo = stp_geo[['geometry', 'name2']]\n",
    "stp_geo.rename(columns={'name2':'BDNAME'}, inplace=True) # Rename so columns match\n",
    "stp_geo['city']=pd.Series(['St. Paul']*len(stp_geo))\n",
    "\n",
    "# Combine to twin cities neighborhoods\n",
    "tc_geo=pd.concat([mpls_geo, stp_geo]).drop_duplicates().reset_index(drop=True)\n",
    "print(tc_geo.shape)\n",
    "tc_geo.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize = (15,15))\n",
    "#tc_geo.plot(ax=ax, facecolor='black')\n",
    "mpls_geo.plot(ax=ax, facecolor='.45') # color between 0 and 1 is greyscale. 0 is darker. 1 is lighter.\n",
    "stp_geo.plot(ax=ax, facecolor='.55')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Turn df_brew in geodataframe and count breweries in neighborhoods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn df_brew into geodatafram\n",
    "df_brew_geo = gpd.GeoDataFrame(df_brew, geometry=gpd.points_from_xy(df_brew['venue.location.lng'], df_brew['venue.location.lat']))\n",
    "\n",
    "\n",
    "brewery_count=[]\n",
    "for j in range(0,len(tc_geo)):\n",
    "    nbhd = tc_geo.loc[j,'BDNAME']  # nbhd is jth neighborhood name, string\n",
    "    nbhd_geo = tc_geo[tc_geo['BDNAME'] == nbhd]  # nbhd_geo is jth row\n",
    "    nbhd_geo.reset_index(drop=True, inplace=True)  # reset index of dataframe\n",
    "    nbhd_mask = df_brew_geo['geometry'].within(nbhd_geo.loc[0,'geometry'])  # Boolean mask of breweries in nbhd.\n",
    "    brewery_count.append(sum(nbhd_mask))  # create list of brewery counts per neighborhood.\n",
    "\n",
    "# Add column to tc_geo    \n",
    "tc_geo['Brewery Count'] = pd.Series(brewery_count) # add brewery count column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize = (15,15))\n",
    "#tc_geo.plot(ax=ax, facecolor='black')\n",
    "#mpls_geo.plot(ax=ax, facecolor='.45') # color between 0 and 1 is greyscale. 0 is darker. 1 is lighter.\n",
    "#stp_geo.plot(ax=ax, facecolor='.55')\n",
    "tc_geo.plot(column='Brewery Count', ax=ax, legend=True)\n",
    "plt.title('Choropleth map of breweries in Minneapolis communities and St. Paul neighborhoods')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choropleth map of restaurant density"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make Minneapolis neighborhoods\n",
    "mpls_geo = gpd.read_file('Minneapolis_Neighborhoods.geojson.json')\n",
    "mpls_geo = mpls_geo[['geometry', 'BDNAME']]\n",
    "mpls_geo['city']=pd.Series(['Minneapolis']*len(mpls_geo))\n",
    "\n",
    "# Minneapolis communities are closer in size to the St. Paul neighborhoods.\n",
    "# If we prefer to consider Minneapolis communities and St. Paul neighborhoods, comment the above code and\n",
    "# run the following instead.\n",
    "\n",
    "#mpls_geo = gpd.read_file('Communities.geojson')\n",
    "#mpls_geo = mpls_geo[['geometry', 'CommName']]\n",
    "#mpls_geo['city']=pd.Series(['Minneapolis']*len(mpls_geo))\n",
    "#mpls_geo.rename(columns={'CommName':'BDNAME'}, inplace=True) # Rename so columns match\n",
    "\n",
    "# Make St. Paul neighborhoods\n",
    "stp_geo = gpd.read_file('stp_neighborhoods.geojson')\n",
    "stp_geo = stp_geo[['geometry', 'name2']]\n",
    "stp_geo.rename(columns={'name2':'BDNAME'}, inplace=True) # Rename so columns match\n",
    "stp_geo['city']=pd.Series(['St. Paul']*len(stp_geo))\n",
    "\n",
    "# Combine to twin cities neighborhoods\n",
    "tc_geo=pd.concat([mpls_geo, stp_geo]).drop_duplicates().reset_index(drop=True)\n",
    "\n",
    "# Plot twin city neighborhoods\n",
    "fig, ax = plt.subplots(figsize = (15,15))\n",
    "tc_geo.plot(ax=ax, facecolor='.45')\n",
    "#mpls_geo.plot(ax=ax, facecolor='.45') # color between 0 and 1 is greyscale. 0 is darker. 1 is lighter.\n",
    "#stp_geo.plot(ax=ax, facecolor='.55')\n",
    "\n",
    "# save file\n",
    "plt.savefig('tc_map_gpd.png', bbox_inches='tight')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Turn df_food into geodataframe\n",
    "df_food_geo = gpd.GeoDataFrame(df_food, geometry=gpd.points_from_xy(df_food['venue.location.lng'], df_food['venue.location.lat']))\n",
    "\n",
    "food_count=[]\n",
    "for j in range(0,len(tc_geo)):\n",
    "    nbhd = tc_geo.loc[j,'BDNAME']  # nbhd is jth neighborhood name, string\n",
    "    nbhd_geo = tc_geo[tc_geo['BDNAME'] == nbhd]  # nbhd_geo is jth row\n",
    "    nbhd_geo.reset_index(drop=True, inplace=True)  # reset index of dataframe\n",
    "    nbhd_mask = df_food_geo['geometry'].within(nbhd_geo.loc[0,'geometry'])  # Boolean mask of breweries in nbhd.\n",
    "    food_count.append(sum(nbhd_mask))  # create list of brewery counts per neighborhood.\n",
    "\n",
    "# Add column to tc_geo    \n",
    "tc_geo['Restaurant Count'] = pd.Series(food_count) # add brewery count column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "fig, ax = plt.subplots(figsize = (15,15))\n",
    "tc_geo.plot(column='Restaurant Count', ax=ax, legend=True, cmap='Oranges')\n",
    "plt.title('Choropleth map of restaurants in Minneapolis communities and St. Paul neighborhoods')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Put breweries into their neighborhoods/communities "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nbhds.loc[0,'Neighborhood']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tc_geo[tc_geo['BDNAME']==get_nbhd(0)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(nbhds))\n",
    "print(len(df_brew))\n",
    "print(len(df_brew_geo))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_nbhd(index):\n",
    "    return nbhds.loc[index, 'Neighborhood']\n",
    "\n",
    "brew_dict={}\n",
    "# Identify brewery neighborhoods\n",
    "# Return dictionary with brewery names as keys, neighborhoods as values.\n",
    "for j in range(0,len(nbhds)):\n",
    "    nbhd_temp = tc_geo[tc_geo['BDNAME']==get_nbhd(j)]\n",
    "    nbhd_temp.reset_index(drop=True, inplace=True)\n",
    "#    print(nbhd_temp)\n",
    "    for index in range(0, len(df_brew)):\n",
    "        nbhd_temp_mask = nbhd_temp.loc[0,'geometry'].contains(df_brew_geo['geometry'][index])\n",
    "#        print(nbhd_temp_mask)\n",
    "#        nbhd_temp_mask = df_brew_geo['geometry'][index].within(nbhd_temp.loc[0,'geometry'])\n",
    "        if nbhd_temp_mask == True:           \n",
    "            print('Brewery {} is in {}'.format(df_brew_geo['venue.name'][index], nbhd_temp.loc[0,'BDNAME'] ))\n",
    "            brew_dict[df_brew_geo['venue.name'][index]]=get_nbhd(j)\n",
    "\n",
    "brew_dict        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Take our brew_dict and turn it into a DataFrame.\n",
    "brew_dict=pd.DataFrame.from_dict(brew_dict, orient='index', columns=['Neighborhood'])\n",
    "brew_dict.reset_index(inplace=True)\n",
    "brew_dict.rename(columns={'index':'venue.name'}, inplace=True)\n",
    "brew_dict.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_brew = pd.merge(df_brew, brew_dict, on='venue.name')\n",
    "df_brew.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_brew.set_index('venue.name')\n",
    "df_brew.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_brew.drop(columns=['lonlat', 'Neighborhood_x', 'Neighborhood_y'], inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_brew.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Choropleth"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_tc_brew=pd.DataFrame(df_brew['Neighborhood'].value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_tc_brew = df_tc_brew.reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_tc_brew.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_tc_brew.rename(columns={'index':'Neighborhood','Neighborhood':'Brewery Count'}, inplace=True)\n",
    "df_tc_brew.head()"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "# create a numpy array of length 6 and has linear spacing from the minium total\n",
    "# breweries to the maximum total breweries\n",
    "threshold_scale = np.linspace(0,\n",
    "                              df_tc_brew['Brewery Count'].max(),\n",
    "                              6, dtype=int)\n",
    "threshold_scale = threshold_scale.tolist() # change the numpy array to a list\n",
    "threshold_scale[-1] = threshold_scale[-1] + 1 # make sure that the last value of the list is greater than the maximum\n",
    "\n",
    "mpls_map = folium.Map(location=[mpls_latitude, mpls_longitude], zoom_start=13, tiles='Stamen Terrain')\n",
    "\n",
    "mpls_map.choropleth(\n",
    "    geo_data=mpls_data,\n",
    "    data=df_tc_brew,\n",
    "    columns=['Neighborhood','Brewery Count'],\n",
    "    key_on='feature.properties.BDNAME',\n",
    "    threshold_scale=threshold_scale,\n",
    "    fill_color='BuPu', \n",
    "    fill_opacity=0.7, \n",
    "    line_opacity=0.2,\n",
    "    legend_name= 'Minneapolis Breweries',\n",
    "    reset=True\n",
    ")\n",
    "mpls_map"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nbhds.to_csv('nbhds_larger.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(df_nbhds_mpls['Neighborhood']))\n",
    "for j in df_nbhds_mpls['Neighborhood'].sort_values():\n",
    "    print(j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minne = pd.read_csv('df_nbhds.csv')\n",
    "minne = minne[minne['City'] == 'Minneapolis']\n",
    "minne.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "minne.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "set(minne['Neighborhood']).symmetric_difference(df_nbhds_mpls['Neighborhood'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for j in minne['Neighborhood']:\n",
    "    print(j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test  = pd.read_csv('df_nbhds.csv')\n",
    "minne = test[test['City']=='Minneapolis']\n",
    "paulie = test[test['City'] == 'St. Paul']\n",
    "print(minne.shape)\n",
    "print(paulie.shape)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for j in paulie['Neighborhood']:\n",
    "    print(j)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}