san-francisco-building-permits.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "adb081b7",
   "metadata": {
    "papermill": {
     "duration": 0.007055,
     "end_time": "2022-10-10T11:16:43.159767",
     "exception": false,
     "start_time": "2022-10-10T11:16:43.152712",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "In this exercise, you'll apply what you learned in the **Handling missing values** tutorial.\n",
    "\n",
    "# Setup\n",
    "\n",
    "The questions below will give you feedback on your work. Run the following cell to set up the feedback system."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ea7f1124",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:43.174241Z",
     "iopub.status.busy": "2022-10-10T11:16:43.173221Z",
     "iopub.status.idle": "2022-10-10T11:16:47.645299Z",
     "shell.execute_reply": "2022-10-10T11:16:47.644019Z"
    },
    "papermill": {
     "duration": 4.482361,
     "end_time": "2022-10-10T11:16:47.648120",
     "exception": false,
     "start_time": "2022-10-10T11:16:43.165759",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/ipykernel_launcher.py:3: DtypeWarning: Columns (22,32) have mixed types.Specify dtype option on import or set low_memory=False.\n",
      "  This is separate from the ipykernel package so we can avoid doing imports until\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Setup Complete\n"
     ]
    }
   ],
   "source": [
    "from learntools.core import binder\n",
    "binder.bind(globals())\n",
    "from learntools.data_cleaning.ex1 import *\n",
    "print(\"Setup Complete\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5dddb14f",
   "metadata": {
    "papermill": {
     "duration": 0.005287,
     "end_time": "2022-10-10T11:16:47.659176",
     "exception": false,
     "start_time": "2022-10-10T11:16:47.653889",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 1) Take a first look at the data\n",
    "\n",
    "Run the next code cell to load in the libraries and dataset you'll use to complete the exercise."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e48727cd",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:47.672066Z",
     "iopub.status.busy": "2022-10-10T11:16:47.671639Z",
     "iopub.status.idle": "2022-10-10T11:16:49.411530Z",
     "shell.execute_reply": "2022-10-10T11:16:49.410278Z"
    },
    "papermill": {
     "duration": 1.749575,
     "end_time": "2022-10-10T11:16:49.414368",
     "exception": false,
     "start_time": "2022-10-10T11:16:47.664793",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.7/site-packages/IPython/core/interactiveshell.py:3552: DtypeWarning: Columns (22,32) have mixed types.Specify dtype option on import or set low_memory=False.\n",
      "  exec(code_obj, self.user_global_ns, self.user_ns)\n"
     ]
    }
   ],
   "source": [
    "# modules we'll use\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# read in all our data\n",
    "sf_permits = pd.read_csv(\"../input/building-permit-applications-data/Building_Permits.csv\")\n",
    "\n",
    "# set seed for reproducibility\n",
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "95ba48e1",
   "metadata": {
    "papermill": {
     "duration": 0.00538,
     "end_time": "2022-10-10T11:16:49.425572",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.420192",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Use the code cell below to print the first five rows of the `sf_permits` DataFrame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "fb8e8859",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:49.439433Z",
     "iopub.status.busy": "2022-10-10T11:16:49.438476Z",
     "iopub.status.idle": "2022-10-10T11:16:49.478890Z",
     "shell.execute_reply": "2022-10-10T11:16:49.477924Z"
    },
    "papermill": {
     "duration": 0.050076,
     "end_time": "2022-10-10T11:16:49.481439",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.431363",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Permit Number</th>\n",
       "      <th>Permit Type</th>\n",
       "      <th>Permit Type Definition</th>\n",
       "      <th>Permit Creation Date</th>\n",
       "      <th>Block</th>\n",
       "      <th>Lot</th>\n",
       "      <th>Street Number</th>\n",
       "      <th>Street Number Suffix</th>\n",
       "      <th>Street Name</th>\n",
       "      <th>Street Suffix</th>\n",
       "      <th>...</th>\n",
       "      <th>Existing Construction Type</th>\n",
       "      <th>Existing Construction Type Description</th>\n",
       "      <th>Proposed Construction Type</th>\n",
       "      <th>Proposed Construction Type Description</th>\n",
       "      <th>Site Permit</th>\n",
       "      <th>Supervisor District</th>\n",
       "      <th>Neighborhoods - Analysis Boundaries</th>\n",
       "      <th>Zipcode</th>\n",
       "      <th>Location</th>\n",
       "      <th>Record ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>201505065519</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>05/06/2015</td>\n",
       "      <td>0326</td>\n",
       "      <td>023</td>\n",
       "      <td>140</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Ellis</td>\n",
       "      <td>St</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>constr type 3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>94102.0</td>\n",
       "      <td>(37.785719256680785, -122.40852313194863)</td>\n",
       "      <td>1380611233945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>201604195146</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>04/19/2016</td>\n",
       "      <td>0306</td>\n",
       "      <td>007</td>\n",
       "      <td>440</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Geary</td>\n",
       "      <td>St</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>constr type 3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>94102.0</td>\n",
       "      <td>(37.78733980600732, -122.41063199757738)</td>\n",
       "      <td>1420164406718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>201605278609</td>\n",
       "      <td>3</td>\n",
       "      <td>additions alterations or repairs</td>\n",
       "      <td>05/27/2016</td>\n",
       "      <td>0595</td>\n",
       "      <td>203</td>\n",
       "      <td>1647</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>Av</td>\n",
       "      <td>...</td>\n",
       "      <td>1.0</td>\n",
       "      <td>constr type 1</td>\n",
       "      <td>1.0</td>\n",
       "      <td>constr type 1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Russian Hill</td>\n",
       "      <td>94109.0</td>\n",
       "      <td>(37.7946573324287, -122.42232562979227)</td>\n",
       "      <td>1424856504716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>201611072166</td>\n",
       "      <td>8</td>\n",
       "      <td>otc alterations permit</td>\n",
       "      <td>11/07/2016</td>\n",
       "      <td>0156</td>\n",
       "      <td>011</td>\n",
       "      <td>1230</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>Av</td>\n",
       "      <td>...</td>\n",
       "      <td>5.0</td>\n",
       "      <td>wood frame (5)</td>\n",
       "      <td>5.0</td>\n",
       "      <td>wood frame (5)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3.0</td>\n",
       "      <td>Nob Hill</td>\n",
       "      <td>94109.0</td>\n",
       "      <td>(37.79595867909168, -122.41557405519474)</td>\n",
       "      <td>1443574295566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>201611283529</td>\n",
       "      <td>6</td>\n",
       "      <td>demolitions</td>\n",
       "      <td>11/28/2016</td>\n",
       "      <td>0342</td>\n",
       "      <td>001</td>\n",
       "      <td>950</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Market</td>\n",
       "      <td>St</td>\n",
       "      <td>...</td>\n",
       "      <td>3.0</td>\n",
       "      <td>constr type 3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Tenderloin</td>\n",
       "      <td>94102.0</td>\n",
       "      <td>(37.78315261897309, -122.40950883997789)</td>\n",
       "      <td>144548169992</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  Permit Number  Permit Type            Permit Type Definition  \\\n",
       "0  201505065519            4                      sign - erect   \n",
       "1  201604195146            4                      sign - erect   \n",
       "2  201605278609            3  additions alterations or repairs   \n",
       "3  201611072166            8            otc alterations permit   \n",
       "4  201611283529            6                       demolitions   \n",
       "\n",
       "  Permit Creation Date Block  Lot  Street Number Street Number Suffix  \\\n",
       "0           05/06/2015  0326  023            140                  NaN   \n",
       "1           04/19/2016  0306  007            440                  NaN   \n",
       "2           05/27/2016  0595  203           1647                  NaN   \n",
       "3           11/07/2016  0156  011           1230                  NaN   \n",
       "4           11/28/2016  0342  001            950                  NaN   \n",
       "\n",
       "  Street Name Street Suffix  ...  Existing Construction Type  \\\n",
       "0       Ellis            St  ...                         3.0   \n",
       "1       Geary            St  ...                         3.0   \n",
       "2     Pacific            Av  ...                         1.0   \n",
       "3     Pacific            Av  ...                         5.0   \n",
       "4      Market            St  ...                         3.0   \n",
       "\n",
       "  Existing Construction Type Description Proposed Construction Type  \\\n",
       "0                          constr type 3                        NaN   \n",
       "1                          constr type 3                        NaN   \n",
       "2                          constr type 1                        1.0   \n",
       "3                         wood frame (5)                        5.0   \n",
       "4                          constr type 3                        NaN   \n",
       "\n",
       "  Proposed Construction Type Description Site Permit Supervisor District  \\\n",
       "0                                    NaN         NaN                 3.0   \n",
       "1                                    NaN         NaN                 3.0   \n",
       "2                          constr type 1         NaN                 3.0   \n",
       "3                         wood frame (5)         NaN                 3.0   \n",
       "4                                    NaN         NaN                 6.0   \n",
       "\n",
       "  Neighborhoods - Analysis Boundaries  Zipcode  \\\n",
       "0                          Tenderloin  94102.0   \n",
       "1                          Tenderloin  94102.0   \n",
       "2                        Russian Hill  94109.0   \n",
       "3                            Nob Hill  94109.0   \n",
       "4                          Tenderloin  94102.0   \n",
       "\n",
       "                                    Location      Record ID  \n",
       "0  (37.785719256680785, -122.40852313194863)  1380611233945  \n",
       "1   (37.78733980600732, -122.41063199757738)  1420164406718  \n",
       "2    (37.7946573324287, -122.42232562979227)  1424856504716  \n",
       "3   (37.79595867909168, -122.41557405519474)  1443574295566  \n",
       "4   (37.78315261897309, -122.40950883997789)   144548169992  \n",
       "\n",
       "[5 rows x 43 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the first five rows of the sf_permits_data file. \n",
    "# I can see a handful of missing data already!Your code here!\n",
    "sf_permits.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3eef916b",
   "metadata": {
    "papermill": {
     "duration": 0.00578,
     "end_time": "2022-10-10T11:16:49.493464",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.487684",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Does the dataset have any missing values?  Once you have an answer, run the code cell below to get credit for your work."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d441faed",
   "metadata": {
    "papermill": {
     "duration": 0.005698,
     "end_time": "2022-10-10T11:16:49.505208",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.499510",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 2) How many missing data points do we have?\n",
    "\n",
    "What percentage of the values in the dataset are missing?  Your answer should be a number between 0 and 100.  (If 1/4 of the values in the dataset are missing, the answer is 25.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f467597c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:49.519697Z",
     "iopub.status.busy": "2022-10-10T11:16:49.518902Z",
     "iopub.status.idle": "2022-10-10T11:16:49.756430Z",
     "shell.execute_reply": "2022-10-10T11:16:49.755257Z"
    },
    "papermill": {
     "duration": 0.247568,
     "end_time": "2022-10-10T11:16:49.758855",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.511287",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Permit Number                  0\n",
       "Permit Type                    0\n",
       "Permit Type Definition         0\n",
       "Permit Creation Date           0\n",
       "Block                          0\n",
       "Lot                            0\n",
       "Street Number                  0\n",
       "Street Number Suffix      196684\n",
       "Street Name                    0\n",
       "Street Suffix               2768\n",
       "dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the number of missing data points per column\n",
    "missing_values_count = sf_permits.isnull().sum()\n",
    "\n",
    "# look at the # of missing points in the first hundred columns\n",
    "missing_values_count[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7644db0c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:49.773454Z",
     "iopub.status.busy": "2022-10-10T11:16:49.772749Z",
     "iopub.status.idle": "2022-10-10T11:16:49.777339Z",
     "shell.execute_reply": "2022-10-10T11:16:49.775993Z"
    },
    "papermill": {
     "duration": 0.014924,
     "end_time": "2022-10-10T11:16:49.780015",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.765091",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q2.hint()\n",
    "#q2.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be30dcf3",
   "metadata": {
    "papermill": {
     "duration": 0.006712,
     "end_time": "2022-10-10T11:16:49.793405",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.786693",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 3) Figure out why the data is missing\n",
    "\n",
    "Look at the columns **\"Street Number Suffix\"** and **\"Zipcode\"** from the [San Francisco Building Permits dataset](https://www.kaggle.com/aparnashastry/building-permit-applications-data). Both of these contain missing values. \n",
    "- Which, if either, are missing because they don't exist? \n",
    "- Which, if either, are missing because they weren't recorded?  \n",
    "\n",
    "Once you have an answer, run the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9fc8e550",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:49.808736Z",
     "iopub.status.busy": "2022-10-10T11:16:49.808319Z",
     "iopub.status.idle": "2022-10-10T11:16:49.816793Z",
     "shell.execute_reply": "2022-10-10T11:16:49.815808Z"
    },
    "papermill": {
     "duration": 0.019458,
     "end_time": "2022-10-10T11:16:49.819119",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.799661",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Permit Number                  0\n",
       "Permit Type                    0\n",
       "Permit Type Definition         0\n",
       "Permit Creation Date           0\n",
       "Block                          0\n",
       "Lot                            0\n",
       "Street Number                  0\n",
       "Street Number Suffix      196684\n",
       "Street Name                    0\n",
       "Street Suffix               2768\n",
       "dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# look at the # of missing points in the first ten columns\n",
    "missing_values_count[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "985d37e4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:49.833381Z",
     "iopub.status.busy": "2022-10-10T11:16:49.832966Z",
     "iopub.status.idle": "2022-10-10T11:16:49.837785Z",
     "shell.execute_reply": "2022-10-10T11:16:49.836505Z"
    },
    "papermill": {
     "duration": 0.014697,
     "end_time": "2022-10-10T11:16:49.840083",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.825386",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Line below will give you a hint\n",
    "#q3.hint()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4fe4c80",
   "metadata": {
    "papermill": {
     "duration": 0.006116,
     "end_time": "2022-10-10T11:16:49.852614",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.846498",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 4) Drop missing values: rows\n",
    "\n",
    "If you removed all of the rows of `sf_permits` with missing values, how many rows are left?\n",
    "\n",
    "**Note**: Do not change the value of `sf_permits` when checking this.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "69245c3f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:49.868152Z",
     "iopub.status.busy": "2022-10-10T11:16:49.867414Z",
     "iopub.status.idle": "2022-10-10T11:16:50.122591Z",
     "shell.execute_reply": "2022-10-10T11:16:50.121387Z"
    },
    "papermill": {
     "duration": 0.26595,
     "end_time": "2022-10-10T11:16:50.125152",
     "exception": false,
     "start_time": "2022-10-10T11:16:49.859202",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Permit Number</th>\n",
       "      <th>Permit Type</th>\n",
       "      <th>Permit Type Definition</th>\n",
       "      <th>Permit Creation Date</th>\n",
       "      <th>Block</th>\n",
       "      <th>Lot</th>\n",
       "      <th>Street Number</th>\n",
       "      <th>Street Number Suffix</th>\n",
       "      <th>Street Name</th>\n",
       "      <th>Street Suffix</th>\n",
       "      <th>...</th>\n",
       "      <th>Existing Construction Type</th>\n",
       "      <th>Existing Construction Type Description</th>\n",
       "      <th>Proposed Construction Type</th>\n",
       "      <th>Proposed Construction Type Description</th>\n",
       "      <th>Site Permit</th>\n",
       "      <th>Supervisor District</th>\n",
       "      <th>Neighborhoods - Analysis Boundaries</th>\n",
       "      <th>Zipcode</th>\n",
       "      <th>Location</th>\n",
       "      <th>Record ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>0 rows × 43 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [Permit Number, Permit Type, Permit Type Definition, Permit Creation Date, Block, Lot, Street Number, Street Number Suffix, Street Name, Street Suffix, Unit, Unit Suffix, Description, Current Status, Current Status Date, Filed Date, Issued Date, Completed Date, First Construction Document Date, Structural Notification, Number of Existing Stories, Number of Proposed Stories, Voluntary Soft-Story Retrofit, Fire Only Permit, Permit Expiration Date, Estimated Cost, Revised Cost, Existing Use, Existing Units, Proposed Use, Proposed Units, Plansets, TIDF Compliance, Existing Construction Type, Existing Construction Type Description, Proposed Construction Type, Proposed Construction Type Description, Site Permit, Supervisor District, Neighborhoods - Analysis Boundaries, Zipcode, Location, Record ID]\n",
       "Index: []\n",
       "\n",
       "[0 rows x 43 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove all the rows that contain a missing value\n",
    "sf_permits.dropna()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d856aef4",
   "metadata": {
    "papermill": {
     "duration": 0.006183,
     "end_time": "2022-10-10T11:16:50.137886",
     "exception": false,
     "start_time": "2022-10-10T11:16:50.131703",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "Once you have an answer, run the code cell below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "063cff6e",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:50.152688Z",
     "iopub.status.busy": "2022-10-10T11:16:50.152281Z",
     "iopub.status.idle": "2022-10-10T11:16:50.415530Z",
     "shell.execute_reply": "2022-10-10T11:16:50.414344Z"
    },
    "papermill": {
     "duration": 0.273557,
     "end_time": "2022-10-10T11:16:50.417947",
     "exception": false,
     "start_time": "2022-10-10T11:16:50.144390",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Permit Number</th>\n",
       "      <th>Permit Type</th>\n",
       "      <th>Permit Type Definition</th>\n",
       "      <th>Permit Creation Date</th>\n",
       "      <th>Block</th>\n",
       "      <th>Lot</th>\n",
       "      <th>Street Number</th>\n",
       "      <th>Street Name</th>\n",
       "      <th>Current Status</th>\n",
       "      <th>Current Status Date</th>\n",
       "      <th>Filed Date</th>\n",
       "      <th>Record ID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>201505065519</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>05/06/2015</td>\n",
       "      <td>0326</td>\n",
       "      <td>023</td>\n",
       "      <td>140</td>\n",
       "      <td>Ellis</td>\n",
       "      <td>expired</td>\n",
       "      <td>12/21/2017</td>\n",
       "      <td>05/06/2015</td>\n",
       "      <td>1380611233945</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>201604195146</td>\n",
       "      <td>4</td>\n",
       "      <td>sign - erect</td>\n",
       "      <td>04/19/2016</td>\n",
       "      <td>0306</td>\n",
       "      <td>007</td>\n",
       "      <td>440</td>\n",
       "      <td>Geary</td>\n",
       "      <td>issued</td>\n",
       "      <td>08/03/2017</td>\n",
       "      <td>04/19/2016</td>\n",
       "      <td>1420164406718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>201605278609</td>\n",
       "      <td>3</td>\n",
       "      <td>additions alterations or repairs</td>\n",
       "      <td>05/27/2016</td>\n",
       "      <td>0595</td>\n",
       "      <td>203</td>\n",
       "      <td>1647</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>withdrawn</td>\n",
       "      <td>09/26/2017</td>\n",
       "      <td>05/27/2016</td>\n",
       "      <td>1424856504716</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>201611072166</td>\n",
       "      <td>8</td>\n",
       "      <td>otc alterations permit</td>\n",
       "      <td>11/07/2016</td>\n",
       "      <td>0156</td>\n",
       "      <td>011</td>\n",
       "      <td>1230</td>\n",
       "      <td>Pacific</td>\n",
       "      <td>complete</td>\n",
       "      <td>07/24/2017</td>\n",
       "      <td>11/07/2016</td>\n",
       "      <td>1443574295566</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>201611283529</td>\n",
       "      <td>6</td>\n",
       "      <td>demolitions</td>\n",
       "      <td>11/28/2016</td>\n",
       "      <td>0342</td>\n",
       "      <td>001</td>\n",
       "      <td>950</td>\n",
       "      <td>Market</td>\n",
       "      <td>issued</td>\n",
       "      <td>12/01/2017</td>\n",
       "      <td>11/28/2016</td>\n",
       "      <td>144548169992</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Permit Number  Permit Type            Permit Type Definition  \\\n",
       "0  201505065519            4                      sign - erect   \n",
       "1  201604195146            4                      sign - erect   \n",
       "2  201605278609            3  additions alterations or repairs   \n",
       "3  201611072166            8            otc alterations permit   \n",
       "4  201611283529            6                       demolitions   \n",
       "\n",
       "  Permit Creation Date Block  Lot  Street Number Street Name Current Status  \\\n",
       "0           05/06/2015  0326  023            140       Ellis        expired   \n",
       "1           04/19/2016  0306  007            440       Geary         issued   \n",
       "2           05/27/2016  0595  203           1647     Pacific      withdrawn   \n",
       "3           11/07/2016  0156  011           1230     Pacific       complete   \n",
       "4           11/28/2016  0342  001            950      Market         issued   \n",
       "\n",
       "  Current Status Date  Filed Date      Record ID  \n",
       "0          12/21/2017  05/06/2015  1380611233945  \n",
       "1          08/03/2017  04/19/2016  1420164406718  \n",
       "2          09/26/2017  05/27/2016  1424856504716  \n",
       "3          07/24/2017  11/07/2016  1443574295566  \n",
       "4          12/01/2017  11/28/2016   144548169992  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove all columns with at least one missing value\n",
    "columns_with_na_dropped = sf_permits.dropna(axis=1)\n",
    "columns_with_na_dropped.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43d9c87f",
   "metadata": {
    "papermill": {
     "duration": 0.006349,
     "end_time": "2022-10-10T11:16:50.430956",
     "exception": false,
     "start_time": "2022-10-10T11:16:50.424607",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 5) Drop missing values: columns\n",
    "\n",
    "Now try removing all the columns with empty values.  \n",
    "- Create a new DataFrame called `sf_permits_with_na_dropped` that has all of the columns with empty values removed.  \n",
    "- How many columns were removed from the original `sf_permits` DataFrame? Use this number to set the value of the `dropped_columns` variable below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f1fbf952",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:50.446344Z",
     "iopub.status.busy": "2022-10-10T11:16:50.445439Z",
     "iopub.status.idle": "2022-10-10T11:16:50.452507Z",
     "shell.execute_reply": "2022-10-10T11:16:50.451018Z"
    },
    "papermill": {
     "duration": 0.017378,
     "end_time": "2022-10-10T11:16:50.454981",
     "exception": false,
     "start_time": "2022-10-10T11:16:50.437603",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Columns in original dataset: 43 \n",
      "\n",
      "Columns with na's dropped: 12\n"
     ]
    }
   ],
   "source": [
    "# just how much data did we lose?\n",
    "print(\"Columns in original dataset: %d \\n\" % sf_permits.shape[1])\n",
    "print(\"Columns with na's dropped: %d\" % columns_with_na_dropped.shape[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5fc9db9",
   "metadata": {
    "papermill": {
     "duration": 0.006444,
     "end_time": "2022-10-10T11:16:50.468556",
     "exception": false,
     "start_time": "2022-10-10T11:16:50.462112",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# 6) Fill in missing values automatically\n",
    "\n",
    "Try replacing all the NaN's in the `sf_permits` data with the one that comes directly after it and then replacing any remaining NaN's with 0.  Set the result to a new DataFrame `sf_permits_with_na_imputed`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "9aaa93f4",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:50.484587Z",
     "iopub.status.busy": "2022-10-10T11:16:50.483542Z",
     "iopub.status.idle": "2022-10-10T11:16:51.402934Z",
     "shell.execute_reply": "2022-10-10T11:16:51.401667Z"
    },
    "papermill": {
     "duration": 0.930102,
     "end_time": "2022-10-10T11:16:51.405663",
     "exception": false,
     "start_time": "2022-10-10T11:16:50.475561",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "sf_permits_fill = sf_permits.fillna(method='bfill')\n",
    "sf_permits_with_na_imputed = sf_permits_fill.fillna(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "064d2d8c",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2022-10-10T11:16:51.421824Z",
     "iopub.status.busy": "2022-10-10T11:16:51.420667Z",
     "iopub.status.idle": "2022-10-10T11:16:51.425189Z",
     "shell.execute_reply": "2022-10-10T11:16:51.424210Z"
    },
    "papermill": {
     "duration": 0.01496,
     "end_time": "2022-10-10T11:16:51.427495",
     "exception": false,
     "start_time": "2022-10-10T11:16:51.412535",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Lines below will give you a hint or solution code\n",
    "#q6.hint()\n",
    "#q6.solution()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4e4b8195",
   "metadata": {
    "papermill": {
     "duration": 0.006415,
     "end_time": "2022-10-10T11:16:51.440693",
     "exception": false,
     "start_time": "2022-10-10T11:16:51.434278",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# More practice\n",
    "\n",
    "If you're looking for more practice handling missing values:\n",
    "\n",
    "* Check out [this noteboook](https://www.kaggle.com/alexisbcook/missing-values) on handling missing values using scikit-learn's imputer. \n",
    "* Look back at the \"Zipcode\" column in the `sf_permits` dataset, which has some missing values. How would you go about figuring out what the actual zipcode of each address should be? (You might try using another dataset. You can search for datasets about San Fransisco on the [Datasets listing](https://www.kaggle.com/datasets).) \n",
    "\n",
    "# Keep going\n",
    "\n",
    "In the next lesson, learn how to [**apply scaling and normalization**](https://www.kaggle.com/alexisbcook/scaling-and-normalization) to transform your data."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "238d57f6",
   "metadata": {
    "papermill": {
     "duration": 0.006332,
     "end_time": "2022-10-10T11:16:51.453712",
     "exception": false,
     "start_time": "2022-10-10T11:16:51.447380",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "---\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "*Have questions or comments? Visit the [course discussion forum](https://www.kaggle.com/learn/data-cleaning/discussion) to chat with other learners.*"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 18.628159,
   "end_time": "2022-10-10T11:16:52.283547",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2022-10-10T11:16:33.655388",
   "version": "2.3.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}