pandas-best-practices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "597d4814",
   "metadata": {
    "lines_to_next_cell": 0,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Pandas Best Practices\n",
    "## 5 Tips for Better Pandas Code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fe77712e",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "6188e883",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## About Matt  Harrison @\\_\\_mharrison\\_\\_\n",
    "\n",
    "* Author of Effective Pandas, Machine Learning Pocket Reference, and Illustrated Guide to Python 3.\n",
    "* Advisor at Ponder (creators of Modin)\n",
    "* Corporate trainer at MetaSnake. Taught Pandas to 1000's of students.\n",
    "* Use coupon LIVE for 10% off Effective Pandas book or bundle ( https://store.metasnake.com )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8bf569d6",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68b20887",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3fd6901",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "352e081c",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61a77bda",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7a1b91f",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5c67d9ed",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Practice this on your data with your team!\n",
    "* Contact me matt@metasnake.com\n",
    "* Follow on Twitter @\\_\\_mharrison\\_\\_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b56b65e9",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c579886",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c38061e7",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Outline\n",
    "\n",
    "* Load Data\n",
    "* Types\n",
    "* Chaining\n",
    "* Mutation\n",
    "* Apply\n",
    "* Aggregation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc9d13b9",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ec04b162",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from IPython.display import display\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pyarrow\n",
    "\n",
    "import io\n",
    "import zipfile"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29ef6997",
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae401f97",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyarrow.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1dea3558",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1d7ee1c0",
   "metadata": {},
   "source": [
    "## Data Preprocessing\n",
    "\n",
    "Don't run this code. I'm providing it here to show you where the data came from.\n",
    "(If you really want to run this download the ZIP file and update the path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a5723a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://gss.norc.org/get-the-data/spss\n",
    "# https://gss.norc.org/Documents/spss/gss_spss_with_codebook.zip\n",
    "# takes a few minutes on my computer to load\n",
    "path = '/mnt/c/Users/matt/Downloads/gss_spss_with_codebook.zip'\n",
    "with zipfile.ZipFile(path) as z:\n",
    "    print(z.namelist())\n",
    "    with open('gss.sav', mode='bw') as fout:\n",
    "        fout.write(z.open('GSS7218_R3.sav').read())\n",
    "    gss = pd.read_spss('gss.sav')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f8587141",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install pyreadstat"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4b686fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "import pyreadstat\n",
    "gss, meta = pyreadstat.read_sav('gss.sav')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c73cd05b",
   "metadata": {},
   "outputs": [],
   "source": [
    "gss.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1e2e1777",
   "metadata": {},
   "outputs": [],
   "source": [
    "gss.to_feather('gss.fth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "46ec0b7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "raw = pd.read_feather('~/Dropbox/work/jupyter/gss.fth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f029dc45",
   "metadata": {
    "lines_to_next_cell": 0,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "raw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cad5ba6",
   "metadata": {},
   "outputs": [],
   "source": [
    "# 6000 columns!\n",
    "raw.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d08680b5",
   "metadata": {
    "lines_to_next_cell": 0
   },
   "outputs": [],
   "source": [
    "cols = ['YEAR','ID','AGE', 'HRS1','OCC','MAJOR1','SEX','RACE','BORN','INCOME',\n",
    "        'INCOME06','HONEST','TICKET']\n",
    "\n",
    "raw[cols].to_feather('honest.fth')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "506f2f1d",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5ab74806",
   "metadata": {},
   "source": [
    "## Loading Data\n",
    "\n",
    "This is the data we will be using. Run this code!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07444860",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "raw = pd.read_feather('honest.fth', dtype_backend='pyarrow')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "582255a1",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "db1c15f3",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## My Cleanup\n",
    "See GSS_Codebook.pdf for explanation\n",
    "\n",
    "Columns:\n",
    "\n",
    "* YEAR\n",
    "* ID - RESPONDENT ID NUMBER\n",
    "* AGE - AGE OF RESPONENT\n",
    "* HRS1 - NUMBER OF HOURS WORKED LAST WEEK\n",
    "* OCC - R'S CENSUS OCCUPATION CODE (1970) - Page 126 (VAR: OCC) see page 125 for notes APPENDIX F,G,H\n",
    "   Appendix F - Page 3286\n",
    "* MAJOR1 - COLLEGE MAJOR 1\n",
    "* SEX - RESPONDENTS SEX\n",
    "* RACE - RACE OF RESPONDENT\n",
    "* BORN -  WAS R BORN IN THIS COUNTRY\n",
    "* INCOME - TOTAL FAMILY INCOME 1970\n",
    "* INCOME06 - TOTAL FAMILY INCOME 2006\n",
    "* HONEST - HONEST\n",
    "* TICKET - EVER RECEIVED A TRAFFIC TICKET\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "65089c43",
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = ['YEAR','ID','AGE', 'HRS1','OCC','MAJOR1','SEX','RACE','BORN','INCOME',\n",
    "        'INCOME06','HONEST','TICKET']\n",
    "\n",
    "raw[cols].isna().mean()*100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "67f1d8f4",
   "metadata": {},
   "outputs": [],
   "source": [
    "(raw\n",
    " [cols]\n",
    " .isna()\n",
    " .mean()*100\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "df146d91",
   "metadata": {},
   "outputs": [],
   "source": [
    "MAJOR= '''RESPONSE PUNCH 1972-82 1982B 1983-87 1987B 1988-91 1993-98 2000-04 2006 2008 2010 2012 2014 2016 2018 ALL\n",
    "Accounting/bookkeeping 1 0 0 0 0 0 0 0 0 0 0 28 32 30 29 119\n",
    "Advertising 2 0 0 0 0 0 0 0 0 0 0 3 2 0 0 5\n",
    "Agriculture/horticulture 3 0 0 0 0 0 0 0 0 0 0 8 2 7 5 22\n",
    "Allied health 4 0 0 0 0 0 0 0 0 0 0 0 2 1 0 3\n",
    "Anthropology 5 0 0 0 0 0 0 0 0 0 0 3 5 1 1 10\n",
    "Architecture 6 0 0 0 0 0 0 0 0 0 0 2 3 5 3 13\n",
    "Art 7 0 0 0 0 0 0 0 0 0 0 6 7 11 10 34\n",
    "Biology 8 0 0 0 0 0 0 0 0 0 0 16 22 33 26 97\n",
    "Business administration 9 0 0 0 0 0 0 0 0 0 0 90 142 172 138 542\n",
    "Chemistry 11 0 0 0 0 0 0 0 0 0 0 5 8 10 4 27\n",
    "Communications/speech 12 0 0 0 0 0 0 0 0 0 0 20 18 26 18 82\n",
    "Comm. disorders 13 0 0 0 0 0 0 0 0 0 0 4 6 2 2 14\n",
    "Computer science 14 0 0 0 0 0 0 0 0 0 0 25 24 33 17 99\n",
    "Dentistry 15 0 0 0 0 0 0 0 0 0 0 2 4 3 5 14\n",
    "Education 16 0 0 0 0 0 0 0 0 0 0 73 91 97 79 340\n",
    "Economics 17 0 0 0 0 0 0 0 0 0 0 11 19 13 19 62\n",
    "Engineering 18 0 0 0 0 0 0 0 0 0 0 47 49 47 54 197\n",
    "English 19 0 0 0 0 0 0 0 0 0 0 23 26 27 24 100\n",
    "Finance 20 0 0 0 0 0 0 0 0 0 0 7 15 14 16 52\n",
    "Foreign language 21 0 0 0 0 0 0 0 0 0 0 4 8 6 5 23\n",
    "Forestry 22 0 0 0 0 0 0 0 0 0 0 1 0 3 0 4\n",
    "Geography 23 0 0 0 0 0 0 0 0 0 0 0 2 2 4 8\n",
    "Geology 24 0 0 0 0 0 0 0 0 0 0 1 3 4 2 10\n",
    "History 25 0 0 0 0 0 0 0 0 0 0 10 19 14 19 62\n",
    "Home economics 26 0 0 0 0 0 0 0 0 0 0 0 0 3 2 5\n",
    "Industry & techn 27 0 0 0 0 0 0 0 0 0 0 3 4 6 0 13\n",
    "Journalism 28 0 0 0 0 0 0 0 0 0 0 5 6 6 4 21\n",
    "Law 29 0 0 0 0 0 0 0 0 0 0 13 18 23 14 68\n",
    "Law enforcement 30 0 0 0 0 0 0 0 0 0 0 3 5 4 2 14\n",
    "Library science 31 0 0 0 0 0 0 0 0 0 0 4 5 2 3 14\n",
    "Marketing 32 0 0 0 0 0 0 0 0 0 0 11 15 13 12 51\n",
    "Mathematics 33 0 0 0 0 0 0 0 0 0 0 5 10 12 5 32\n",
    "Medicine 34 0 0 0 0 0 0 0 0 0 0 9 25 12 11 57\n",
    "Music 35 0 0 0 0 0 0 0 0 0 0 4 2 10 2 18\n",
    "Nursing 36 0 0 0 0 0 0 0 0 0 0 36 39 60 51 186\n",
    "Optometry 37 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\n",
    "Pharmacy 38 0 0 0 0 0 0 0 0 0 0 2 5 4 4 15\n",
    "Philosophy 39 0 0 0 0 0 0 0 0 0 0 2 0 2 2 6\n",
    "Physical education 40 0 0 0 0 0 0 0 0 0 0 9 6 16 6 37\n",
    "Physics 41 0 0 0 0 0 0 0 0 0 0 3 6 7 4 20\n",
    "Psychology 42 0 0 0 0 0 0 0 0 0 0 32 32 34 29 127\n",
    "Political science/international relations 43 0 0 0 0 0 0 0 0 0 0 16 22 19 14 71\n",
    "Sociology 44 0 0 0 0 0 0 0 0 0 0 9 15 10 12 46\n",
    "Special education 45 0 0 0 0 0 0 0 0 0 0 5 3 5 2 15\n",
    "Theater arts 46 0 0 0 0 0 0 0 0 0 0 6 2 3 1 12\n",
    "Theology 47 0 0 0 0 0 0 0 0 0 0 6 6 13 8 33\n",
    "Veterinary medicine 48 0 0 0 0 0 0 0 0 0 0 1 5 3 4 13\n",
    "Liberal arts 49 0 0 0 0 0 0 0 0 0 0 8 16 12 10 46\n",
    "Other 50 0 0 0 0 0 0 0 0 0 0 8 10 21 27 66\n",
    "General sciences 51 0 0 0 0 0 0 0 0 0 0 10 13 15 14 52\n",
    "Social work 52 0 0 0 0 0 0 0 0 0 0 7 17 24 7 55\n",
    "General studies 53 0 0 0 0 0 0 0 0 0 0 2 5 7 7 21\n",
    "Other vocational 54 0 0 0 0 0 0 0 0 0 0 5 11 6 5 27\n",
    "Health 55 0 0 0 0 0 0 0 0 0 0 23 31 31 42 127\n",
    "Industrial Relations 56 0 0 0 0 0 0 0 0 0 0 1 0 0 3 4\n",
    "Child/Human/Family Development 57 0 0 0 0 0 0 0 0 0 0 11 3 7 7 28\n",
    "Food Science/Nutrition/Culinary Arts 58 0 0 0 0 0 0 0 0 0 0 3 6 9 9 27\n",
    "Environmental Science/Ecology 59 0 0 0 0 0 0 0 0 0 0 5 5 6 8 24\n",
    "Social Sciences 60 0 0 0 0 0 0 0 0 0 0 4 2 7 5 18\n",
    "Human Services/Human Resources 61 0 0 0 0 0 0 0 0 0 0 3 7 7 5 22\n",
    "Visual Arts/Graphic Design/Design and Drafting 62 0 0 0 0 0 0 0 0 0 0 3 8 9 10 30\n",
    "Fine Arts 63 0 0 0 0 0 0 0 0 0 0 4 5 5 6 20\n",
    "Humanities 64 0 0 0 0 0 0 0 0 0 0 0 2 0 1 3\n",
    "Ethnic studies 65 0 0 0 0 0 0 0 0 0 0 3 1 0 0 4\n",
    "Educational administration 66 0 0 0 0 0 0 0 0 0 0 3 4 8 9 24\n",
    "Television/Film 67 0 0 0 0 0 0 0 0 0 0 0 2 6 1 9\n",
    "Aviation/Aeronatics 68 0 0 0 0 0 0 0 0 0 0 2 1 1 3 7\n",
    "Statistics/Biostatistics 69 0 0 0 0 0 0 0 0 0 0 0 0 2 2 4\n",
    "Criminology/Criminal Justice 70 0 0 0 0 0 0 0 0 0 0 13 17 17 13 60\n",
    "Administrative Science/Public Administration 71 0 0 0 0 0 0 0 0 0 0 2 11 3 5 21\n",
    "Electronics 72 0 0 0 0 0 0 0 0 0 0 6 6 5 9 26\n",
    "Urban and Regional Planning 73 0 0 0 0 0 0 0 0 0 0 1 1 3 2 7\n",
    "Mechanics/Machine Trade 74 0 0 0 0 0 0 0 0 0 0 0 1 1 4 6\n",
    "Dance 75 0 0 0 0 0 0 0 0 0 0 1 0 1 1 3\n",
    "Gerontology 76 0 0 0 0 0 0 0 0 0 0 1 0 1 1 3\n",
    "Public Relations 77 0 0 0 0 0 0 0 0 0 0 3 1 2 1 7\n",
    "Textiles/Cloth 78 0 0 0 0 0 0 0 0 0 0 3 4 0 0 7\n",
    "Parks and Recreation 79 0 0 0 0 0 0 0 0 0 0 1 2 1 0 4\n",
    "Information Technology 80 0 0 0 0 0 0 0 0 0 0 0 5 8 11 24\n",
    "Fashion 81 0 0 0 0 0 0 0 0 0 0 0 0 3 1 4\n",
    "Counseling 82 0 0 0 0 0 0 0 0 0 0 0 0 11 9 20\n",
    "Don't know/UNCODED 98 0 0 0 0 0 0 0 0 0 0 2 3 0 0 5\n",
    "No answer 99 0 0 0 0 0 0 0 0 0 0 0 1 5 3 9\n",
    "Not applicable 0 13626 354 7542 353 5907 10334 8394 4510 2023 2044 1263 1597 1795 1435 61177'''\n",
    "\n",
    "# copy paste slight tweak from page 186\n",
    "major_dict = {int(row.split()[-16]): ' '.join(row.split()[:-16])  for row in MAJOR.split('\\n')[1:]}\n",
    "major_dict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9d6c34d",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw.MAJOR1.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74652b6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "(raw\n",
    " [cols]\n",
    " .assign(\n",
    "     MAJOR1=raw.MAJOR1.fillna(99).astype('int').replace(major_dict),\n",
    "     SEX=raw.SEX#\n",
    "           \n",
    "           .astype(int)\n",
    "           .replace({1:'Male', 2:'Female'}),\n",
    "     RACE=raw.RACE.astype(int).replace({1:'White', 2:'Black', 3:'Other'}),\n",
    "     OCC=raw.OCC.fillna(9999).astype(int),\n",
    "     BORN=raw.BORN.fillna(4).astype(int).replace({1:'Yes', 2:'No', 3:'Don\\'t Know',\n",
    "                                                    4:'No answer', 5:'Not applicable'}),\n",
    "     INCOME=raw.INCOME.fillna(99).astype(int).replace({99:'No answer', **dict(enumerate(['Not applicable',\n",
    "                                                                                  0,1000,3000,4000,5000,6000,\n",
    "                                                                                  7000,8000,10000,15000,20000,25000,]))}),\n",
    "     INCOME06=raw.INCOME06.fillna(26).astype(int).replace({26:'Refused', **dict(enumerate(['Not applicable',\n",
    "                                                                                  0,1000,3000,4000,5000,6000,\n",
    "                                                                                  7000,8000,10000,12500,15000,\n",
    "                                                                                  17500,20000,22500,25000,30_000,\n",
    "                                                                                  35_000, 40_000, 50_000, 60_000,\n",
    "                                                                                 75_000, 90_000, 110_000, 130_000,\n",
    "                                                                                 150_000]))}),\n",
    "     HONEST=raw.HONEST.fillna(9).astype(int).replace({1:'Most desirable', 2:'3 most desireable',\n",
    "                                                                   3:'Not mentioned', 4:  '3 least desireable',\n",
    "                                                                   5: 'One least desireable',\n",
    "                                                                    9:'No answer'}),\n",
    "     TICKET=raw.TICKET.fillna(9).astype(int).replace({1:'Yes', 2:'No', 3:'Refused', 9: 'No answer'}),\n",
    "     )\n",
    " .astype({'YEAR':int, 'ID': 'uint16[pyarrow]'})\n",
    " .to_csv('GSS.csv')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a14afd45",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ce8f0020",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "043a0085",
   "metadata": {},
   "source": [
    "## Types\n",
    "Getting the right types will enable analysis and correctness.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5d6c6cd5",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "gss = pd.read_csv('GSS.csv', index_col=0, dtype_backend='pyarrow', engine='pyarrow')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d26b8f6c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "gss.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "19890585",
   "metadata": {},
   "outputs": [],
   "source": [
    "gss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "852115fe",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "gss.memory_usage(deep=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5cfc13b",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# 36 M (pandas 1)\n",
    "# 8.6 M (Pandas 2)\n",
    "gss.memory_usage(deep=True).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1d1b51d",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5fcab8c0",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Ints"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ad4eddc7",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "gss.select_dtypes(int).describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac323e5e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# chaining\n",
    "(gss\n",
    " .select_dtypes(int)\n",
    " .describe()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2351d051",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# can comb08 be an int8?\n",
    "# Do completion on int\n",
    "np.iinfo(np.int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "323df8fb",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "np.iinfo(np.uint8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb063be4",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "np.iinfo(np.uint16)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0fab927",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# chaining\n",
    "(gss\n",
    " .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]' })\n",
    " .select_dtypes(['uint16'])\n",
    " .describe()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1d0ed15",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# chaining\n",
    "# use 'integer' so see all int-like columns\n",
    "(gss\n",
    " .astype({#'YEAR': 'uint16[pyarrow]',\n",
    "          'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]' }) \n",
    " .select_dtypes(['integer'])  # see https://numpy.org/doc/stable/reference/arrays.scalars.html\n",
    " .describe()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5d4c3e1",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Inspect memory usage\n",
    "(gss\n",
    " .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]' }) \n",
    " .memory_usage(deep=True)\n",
    " .sum()  # was 36M\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ad6e733",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f339194e",
   "metadata": {},
   "source": [
    "## Int Exercise\n",
    "* Try converting *YEAR* to `'int8'`. What do the values look like?\n",
    "* Try converting *YEAR* to `'int8[pyarrow]'`. What do the values look like?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "908545d1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18a3bf52",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b09f89c6",
   "metadata": {},
   "source": [
    "## Floats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7fed87e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss\n",
    ".select_dtypes('float'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "49265726",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# surprise! age and hours worked looks int-like\n",
    "gss.HRS1.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cd39df3c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# opps! missing values\n",
    "gss.HRS1.value_counts(dropna=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "31a67da2",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# where are they missing?\n",
    "(gss\n",
    "  .query('HRS1.isna()')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e697d070",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# where are they missing?\n",
    "(gss\n",
    "  .query('AGE.isna()')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a9166e1a",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# where are they missing?\n",
    "# It turns out that ID is not consistent across years\n",
    "(gss\n",
    "  .query('ID == 229')\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81a8c902",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Convert to integers\n",
    "(gss\n",
    "  .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]',\n",
    "         'HRS1': 'uint8[pyarrow]', 'AGE': 'uint8[pyarrow]'})\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d3388e2c",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss\n",
    "  .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]',\n",
    "         'HRS1': 'uint8[pyarrow]', 'AGE': 'uint8[pyarrow]'})\n",
    " .memory_usage(deep=True)\n",
    " .sum()  # was 36M  \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9bb70ac2",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "75bfd716",
   "metadata": {},
   "source": [
    "## Float Exercise\n",
    "\n",
    "* What is the mean of the numeric columns?\n",
    "* How many values are missing in the numeric columns?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e3e30c4e",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a136fe09",
   "metadata": {},
   "source": [
    "## Objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f8b0477",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# pandas 1.x\n",
    "(gss\n",
    " .select_dtypes(object)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f80da8d2",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# pandas 2\n",
    "(gss\n",
    " .select_dtypes('string') # str doesn't work\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f762143",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# My goto method - .value_counts\n",
    "# looks categorical\n",
    "(gss.MAJOR1.value_counts(dropna=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55c21c7a",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss\n",
    "  .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]',\n",
    "         'HRS1': 'uint8[pyarrow]', 'AGE': 'uint8[pyarrow]',\n",
    "         'MAJOR1': 'category'})\n",
    " .memory_usage(deep=True)\n",
    " .sum()  # was 36M  \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "69969c1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "(gss\n",
    " .select_dtypes(object)\n",
    " .columns\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5d51601",
   "metadata": {
    "lines_to_next_cell": 0,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# wow!\n",
    "(gss\n",
    "  .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]',\n",
    "         'HRS1': 'uint8[pyarrow]', 'AGE': 'uint8[pyarrow]',\n",
    "         'MAJOR1': 'category',\n",
    "          **{col: 'category' for col in ['SEX', 'RACE', 'BORN', \n",
    "                'INCOME', 'INCOME06', 'HONEST','TICKET']}})           \n",
    " .memory_usage(deep=True)\n",
    " .sum()  # was 36M  \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17206364",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "50bf3fa6",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "246041ae",
   "metadata": {},
   "source": [
    "## Category Exercises\n",
    "* There is a `.cat` attribute on the category columns. What can you do with this attribute? (Use `dir` or tab completion to inspect).\n",
    "* Categories can be ordered. How do you order *INCOME*?\n",
    "* Order the *HONEST* column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f543c52c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "338e5ba3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fb313d6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1d75a84",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c87e18d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85aaccbb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8a321513",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8af7a3d4",
   "metadata": {},
   "source": [
    "## Make a Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cb9a32b3",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# a glorious function\n",
    "# add ordered categories to this\n",
    "def tweak_gss(gss):\n",
    "    return (gss\n",
    "      .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]',\n",
    "             'HRS1': 'uint8[pyarrow]', 'AGE': 'uint8[pyarrow]',\n",
    "             'MAJOR1': 'category',\n",
    "              **{col: 'category' for col in ['SEX', 'RACE', 'BORN', \n",
    "                    'INCOME', 'INCOME06', 'HONEST','TICKET']}})\n",
    "               )\n",
    "\n",
    "tweak_gss(gss)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c615739",
   "metadata": {},
   "source": [
    "## Function Exercise\n",
    "* Rearrange your notebook. Put the imports, code to load raw data, and tweak function at the top of the notebook. Restart the kernel and validate that your code works."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c61b9f0a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6589902c",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b350e12e",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "source": [
    "## Fix Column Names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99b39238",
   "metadata": {
    "lines_to_next_cell": 0,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# a glorious function\n",
    "def tweak_gss(gss):\n",
    "    return (gss\n",
    "      .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]',\n",
    "             'HRS1': 'uint8[pyarrow]', 'AGE': 'uint8[pyarrow]',\n",
    "             'MAJOR1': 'category',\n",
    "              **{col: 'category' for col in ['SEX', 'RACE', 'BORN', \n",
    "                    'INCOME', 'INCOME06', 'HONEST','TICKET']}})\n",
    "     .rename(columns={'YEAR': 'year', 'ID': 'year_id', 'AGE':'age', \n",
    "          'HRS1': 'hours_worked', 'OCC': 'occupation', \n",
    "          'MAJOR1': 'college_major', 'SEX':'sex', \n",
    "          'RACE':'race', 'BORN':'born_in_US',\n",
    "          'INCOME':'income_1970', 'INCOME06': 'income_2006',\n",
    "          'HONEST':'honesty_rank',\n",
    "          'TICKET':'traffic_ticket'})\n",
    "    )\n",
    "\n",
    "tweak_gss(gss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "84ecc0de",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf14ec3f",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "003b96b9",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Chain\n",
    "\n",
    "Chaining is also called \"flow\" programming. Rather than making intermediate variables, just leverage the fact that most operations return a new object and work on that.\n",
    "\n",
    "The chain should read like a recipe of ordered steps.\n",
    "\n",
    "(BTW, this is actually what we did above.)\n",
    "\n",
    "<div class='alert alert-warning'>\n",
    "    Hint: Leverage <tt>.pipe</tt> if you can't find a way to chain 😉🐼💪\n",
    "</div>\n",
    "    \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a74cd1a9",
   "metadata": {
    "lines_to_next_cell": 0,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# a glorious function\n",
    "def tweak_gss(gss):\n",
    "    return (gss\n",
    "      .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]',\n",
    "             'HRS1': 'uint8[pyarrow]', 'AGE': 'uint8[pyarrow]',\n",
    "             'MAJOR1': 'category',\n",
    "              **{col: 'category' for col in ['SEX', 'RACE', 'BORN', \n",
    "                    'INCOME', 'INCOME06', 'HONEST','TICKET']}})\n",
    "     .rename(columns={'YEAR': 'year', 'ID': 'year_id', 'AGE':'age', \n",
    "          'HRS1': 'hours_worked', 'OCC': 'occupation', \n",
    "          'MAJOR1': 'college_major', 'SEX':'sex', \n",
    "          'RACE':'race', 'BORN':'born_in_US',\n",
    "          'INCOME':'income_1970', 'INCOME06': 'income_2006',\n",
    "          'HONEST':'honesty_rank',\n",
    "          'TICKET':'traffic_ticket'})\n",
    "    )\n",
    "\n",
    "tweak_gss(gss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "efc594da",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# compare chain to this mess\n",
    "gss2 = gss.copy()\n",
    "year = gss.YEAR\n",
    "year_int = year.astype('uint16')\n",
    "gss2['year'] = year_int\n",
    "id = gss.ID\n",
    "id_int = id.astype('uint16')\n",
    "gss2['year_id'] = id_int\n",
    "occ = gss.OCC\n",
    "occ_int = occ.astype('uint16')\n",
    "gss2['occupation'] = occ_int\n",
    "\n",
    "# more of this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32411eaf",
   "metadata": {
    "lines_to_next_cell": 0,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# easy to debug\n",
    "#  - assign to var (df3)\n",
    "#  - comment out\n",
    "#  - pipe to display\n",
    "\n",
    "\n",
    "from IPython.display import display\n",
    "\n",
    "def get_var(df, var_name):\n",
    "    globals()[var_name] = df\n",
    "    return df\n",
    "\n",
    "def tweak_gss(gss):\n",
    "    return (gss\n",
    "      .pipe(get_var, 'df3')   \n",
    "     .pipe(lambda df: print(df.shape) or df)                \n",
    "      .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]',\n",
    "             'HRS1': 'uint8[pyarrow]', 'AGE': 'uint8[pyarrow]',\n",
    "             'MAJOR1': 'category',\n",
    "              **{col: 'category' for col in ['SEX', 'RACE', 'BORN', \n",
    "                    'INCOME', 'INCOME06', 'HONEST','TICKET']}})\n",
    "     .pipe(lambda df: print(df.shape) or df)                            \n",
    "     .rename(columns={'YEAR': 'year', 'ID': 'year_id', 'AGE':'age', \n",
    "          'HRS1': 'hours_worked', 'OCC': 'occupation', \n",
    "          'MAJOR1': 'college_major', 'SEX':'sex', \n",
    "          'RACE':'race', 'BORN':'born_in_US',\n",
    "          'INCOME':'income_1970', 'INCOME06': 'income_2006',\n",
    "          'HONEST':'honesty_rank',\n",
    "          'TICKET':'traffic_ticket'})\n",
    "     .pipe(lambda df: print(df.shape) or df)                            \n",
    "    )\n",
    "\n",
    "tweak_gss(gss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdc2894e",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# inspect intermediate data frame\n",
    "df3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1842701c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "source": [
    "## Chain Exercise\n",
    "* Write a function that acccepts a dataframe and an index value. It should print any rows that match the index and return the dataframe that was passed in.\n",
    "* Use the function with pipe after each step of the chain. Show the rows for index 2 and 64,813.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "## Don't Mutate\n",
    "\n",
    "> \"you are missing the point, inplace rarely actually does something inplace, you are thinking that you are saving memory but you are not.\"\n",
    ">\n",
    "> **jreback** - Pandas core dev\n",
    "\n",
    "\n",
    "\n",
    "https://github.com/pandas-dev/pandas/issues/16529#issuecomment-676518136\n",
    "\n",
    "* In general, no performance benefits\n",
    "* Prohibits chaining\n",
    "* ``SettingWithCopyWarning`` fun\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9b1955ed",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "pd.read_csv??"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bce7abe3",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e6a8e2f",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "2a263d38",
   "metadata": {
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "## Don't Apply (if you can)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e68b584",
   "metadata": {
    "lines_to_next_cell": 0,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# a glorious function\n",
    "def tweak_gss(gss):\n",
    "    return (gss\n",
    "      .astype({'YEAR': 'uint16[pyarrow]', 'ID': 'uint16[pyarrow]', 'OCC': 'uint16[pyarrow]',\n",
    "             'HRS1': 'uint8[pyarrow]', 'AGE': 'uint8[pyarrow]',\n",
    "             'MAJOR1': 'category',\n",
    "              **{col: 'category' for col in ['SEX', 'RACE', 'BORN', \n",
    "                    'INCOME', 'INCOME06', 'HONEST','TICKET']}})\n",
    "     .rename(columns={'YEAR': 'year', 'ID': 'year_id', 'AGE':'age', \n",
    "          'HRS1': 'hours_worked', 'OCC': 'occupation', \n",
    "          'MAJOR1': 'college_major', 'SEX':'sex', \n",
    "          'RACE':'race', 'BORN':'born_in_US',\n",
    "          'INCOME':'income_1970', 'INCOME06': 'income_2006',\n",
    "          'HONEST':'honesty_rank',\n",
    "          'TICKET':'traffic_ticket'})\n",
    "    )\n",
    "\n",
    "gss2 = tweak_gss(gss)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a82332f",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# convert age to months\n",
    "def to_months(val):\n",
    "    return val * 12\n",
    "\n",
    "gss2.age.apply(to_months)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a221e972",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# this gives the sames results\n",
    "gss2.age * 12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9cb2b9d2",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "gss2.age.apply(to_months)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51bcc862",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "gss2.age * 12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72a01657",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# ~42x slower!\n",
    "4_590 / 110"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "619094f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "gss.MAJOR1.value_counts()[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4817aee",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "def is_science(val):\n",
    "    return val in {'Engineering', 'Computer science', 'Biology'}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f00a069c",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "# string\n",
    "gss.MAJOR1.apply(is_science)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e13ae10",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "gss.MAJOR1.isin({'Engineering', 'Computer science', 'Biology'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc933ec1",
   "metadata": {
    "lines_to_next_cell": 0,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "%%timeit\n",
    "# categorical\n",
    "gss2.college_major.isin({'Engineering', 'Computer science', 'Biology'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42a822c2",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d56720b4",
   "metadata": {},
   "source": [
    "## Apply Exercise\n",
    "* Make a new column called *minutes_worked* derived the *hours_worked* column.\n",
    "* Make a new column called *income_ratio*.\n",
    "  * Convert the income columns to numbers (replace `'No answer'` and `'Refused'` with `np.nan`).\n",
    "  * Fill in the missing values with the median\n",
    "  * Divide the 2006 value by 1970 value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b818eb6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "59ebffeb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9d2a4e10",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "132efb76",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f3631607",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1f627b9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eee8ef40",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c19d66ca",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85c49f0f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "537487a7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f62adb56",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c59b615",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "55ce9070",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "15e221af",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ed68b41",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae90e79b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96420545",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba8633ef",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8993c970",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ca9ac23",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "139432bb",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "816e9c31",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f71cd5b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f4975a6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "44aa5162",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5702992",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0dead63",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e0faa823",
   "metadata": {},
   "source": [
    "## Master Aggregation\n",
    "\n",
    "Let's compare age by sex by year...🤔"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d444c6b8",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss2\n",
    "   .groupby('year')\n",
    "   .mean()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85441b08",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss2\n",
    "   .groupby('year')\n",
    "   .mean(numeric_only=True)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "eadbc6cd",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss2\n",
    "   .groupby('year')\n",
    "   [['age', 'hours_worked']]\n",
    "   .mean()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6d008ae",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "#plt.style.use('pandas1book') \n",
    "sns.set_context('talk')\n",
    "plt.plot(range(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffc36b52",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss2\n",
    "   .groupby('year')\n",
    "   [['age', 'hours_worked']]\n",
    "   .median()\n",
    "   .plot()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bdcd3e8",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss2\n",
    "   .groupby('year')\n",
    "   [['age', 'hours_worked']]\n",
    "   #.mean()\n",
    "   #.median()\n",
    "   #.std()\n",
    "   .max()\n",
    "   .plot()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54ebb97d",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# add sex\n",
    "(gss2\n",
    "   .groupby(['year', 'sex'])\n",
    "   [['age', 'hours_worked']]\n",
    "   .mean()\n",
    "   #.median()\n",
    "   #.std()\n",
    "   #.max()\n",
    "   #.plot()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "266a53da",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# add sex\n",
    "(gss2\n",
    "   .groupby(['year', 'sex'])\n",
    "   [['age', 'hours_worked']]\n",
    "   .mean()\n",
    "   #.median()\n",
    "   #.std()\n",
    "   #.max()\n",
    "   .plot()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e20f409",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# unstack\n",
    "(gss2\n",
    "   .groupby(['year', 'sex'])\n",
    "   [['age', 'hours_worked']]\n",
    "   .mean()\n",
    "   #.median()\n",
    "   #.std()\n",
    "   #.max()\n",
    "   .unstack() \n",
    "   .plot()\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5481e10",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss2\n",
    "   .groupby(['year', 'sex'])\n",
    "   [['age', 'hours_worked']]\n",
    "   .mean()\n",
    "   .unstack()\n",
    "   .age\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e01d055",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "(gss2\n",
    "   .groupby(['year', 'sex'])\n",
    "   [['age', 'hours_worked']]\n",
    "   .mean()\n",
    "   .unstack()\n",
    "   .age\n",
    "   .plot()\n",
    "   .legend(bbox_to_anchor=(1,1))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d1528728",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Let's try looking at hours worked\n",
    "(gss2\n",
    "   .groupby(['year', 'sex'])\n",
    "   [['age', 'hours_worked']]\n",
    "   .mean()\n",
    "   .unstack()\n",
    "   .hours_worked\n",
    "   .plot()\n",
    "   .legend(bbox_to_anchor=(1,1))\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a52537a5",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "# Multiple aggregates\n",
    "def second(group):\n",
    "    return group.iloc[1]\n",
    "(gss2\n",
    "   .groupby(['year', 'sex'])\n",
    "   [['age', 'hours_worked']]\n",
    "  .agg(['min', 'max', 'mean', second])\n",
    "   \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b780beb4",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "9aca44a7",
   "metadata": {},
   "source": [
    "## Aggregation Exercise\n",
    "* Which occupation has the highest median hours worked?\n",
    "* Which occupation has the lowest age?\n",
    "* What is the breakdown of respondents by race for each year?\n",
    "* Convert the previous to a percentage.\n",
    "* How many unique occupations are there for each year?\n",
    "* What is the most popular college_major for each year?\n",
    "* What is the second most popular college_major for each year?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9e5477d4",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bcc93724",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3a7f368e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "697919b8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d93f8db",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3907736f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18186540",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f1089e32",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "116593ba",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaa6e44e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "648b7e1e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7ee8d6cc",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4f252bd0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6090240",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a37bb2df",
   "metadata": {
    "lines_to_next_cell": 2
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7cf8f182",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "* Correct types save space and enable convenient math, string, and date functionality\n",
    "* Chaining operations will:\n",
    "   * Make code readable\n",
    "   * Remove bugs\n",
    "   * Easier to debug\n",
    "* Don't mutate (there's no point). Embrace chaining.\n",
    "* ``.apply`` is slow for math\n",
    "* Aggregations are powerful. Play with them until they make sense\n",
    "\n",
    "Follow on Twitter ``@__mharrison__``\n",
    "\n",
    "Book giveaway!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89444b02",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import random\n",
    "random.randrange(1,13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35931834",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53ab759b",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "063efccd",
   "metadata": {
    "lines_to_next_cell": 2,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "26386f48",
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}