uber-lyft.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f8eafed2-77a1-4691-8e8b-aeb1187ce8f5",
   "metadata": {
    "tags": []
   },
   "source": [
    "\n",
    "NYC Uber/Lyft Rides\n",
    "===================\n",
    "\n",
    "<img src=\"https://docs.dask.org/en/stable/_images/dask-dataframe.svg\"\n",
    "     align=\"right\"\n",
    "     width=\"40%\"/>\n",
    "\n",
    "The NYC Taxi dataset is a timeless classic.  \n",
    "\n",
    "Interestingly there is a new variant.  The NYC Taxi and Livery Commission requires data from all ride-share services in the city of New York.  This includes private limosine services, van services, and a new category \"High Volume For Hire Vehicle\" services, those that dispatch 10,000 rides per day or more.  This is a special category defined for Uber and Lyft.  \n",
    "\n",
    "This data is available here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "09728a96-0c84-4198-ab52-4dcdfd704606",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We can read a small piece of data with pandas\n",
    "# but this is slow and not scalable\n",
    "\n",
    "import pandas as pd\n",
    "\n",
    "df = pd.read_parquet(\n",
    "    \"s3://coiled-data/uber/part.0.parquet\",\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a9e6076-c8b3-4282-90a4-0fe3ab49440d",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import coiled\n",
    "\n",
    "cluster = coiled.Cluster(\n",
    "    n_workers=30,\n",
    "    worker_memory=\"16 GiB\",\n",
    "    region=\"us-east-2\",\n",
    ")\n",
    "\n",
    "client = cluster.get_client()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33b598a4-fe0a-43c5-8007-0e955ac193f9",
   "metadata": {},
   "outputs": [],
   "source": [
    "import dask\n",
    "import dask.dataframe as dd\n",
    "\n",
    "df = dd.read_parquet(\n",
    "    \"s3://coiled-data/uber/\",\n",
    ")\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9d012d8-6055-4cdc-a37e-d6a01b36c7db",
   "metadata": {},
   "source": [
    "Play time\n",
    "---------\n",
    "\n",
    "We actually don't know what to expect from this dataset.  No one in our team has spent much time inspecting it.  We'd like to solicit help from you, new Dask user, to uncover some interesting insights.  \n",
    "\n",
    "Care to explore and report your findings?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8c589e9f-f3e9-41d3-b34c-ca42fee44729",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = df.persist()\n",
    "\n",
    "df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "062fe739-a3f1-4a9a-8e62-34da95b63982",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4d906ce-605b-4128-890d-830334986974",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e622064f-0e1c-48bc-a639-8d181fc92dea",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d264b39e-2dee-49a1-bbbf-2537265a3630",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "98d43c82-9a03-4311-8c17-2ef3b3519aa7",
   "metadata": {},
   "source": [
    "## Tipping Practices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23af7df8-cf98-4e04-8cd0-829a33e65840",
   "metadata": {},
   "outputs": [],
   "source": [
    "# How often do New Yorkers tip?\n",
    "\n",
    "(df.tips != 0).mean().compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ac9e09e-1b8f-4e79-9929-fc26a753e23e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18236b33-c27d-4d96-8a61-98d73c224744",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "10f285be-6839-4866-9535-43adbcc965d0",
   "metadata": {},
   "source": [
    "## Broken down by carrier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b71729eb-f841-433a-b020-0f2b1c425355",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Uber / Lyft / Via / ... different carriers\n",
    "df.hvfhs_license_num.value_counts().compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da79527d-8db9-4f7b-96fe-c87159241103",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "df[\"tipped\"] = df.tips != 0\n",
    "\n",
    "df.groupby(\"hvfhs_license_num\").tipped.mean().compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5a6e07be-4694-496c-a323-161518ebed74",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74cb092c-11ea-40ef-8bcb-127b540681f3",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0cac5bd4-8f32-43b4-bfe0-e20767cf1db2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6ae5e64d-4bf3-4f99-b54f-aebedfdfd98a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7c962f9f-7308-465a-98fb-ac622a967a34",
   "metadata": {
    "jp-MarkdownHeadingCollapsed": true,
    "tags": []
   },
   "source": [
    "## Dask TV\n",
    "\n",
    "We use this in conference events just to make the dashboard go and bring in a crowd.  Colloquially we call this \"Dask TV\".  Enjoy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "829de2bc-ed09-4e10-b06f-268aa79ead59",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import dask\n",
    "import dask.dataframe as dd\n",
    "dask.config.set({\"dataframe.convert-string\": True})  # use PyArrow strings by default\n",
    "\n",
    "while True:\n",
    "    client.restart()\n",
    "\n",
    "    df = dd.read_parquet(\n",
    "        \"s3://coiled-datasets/uber-lyft-tlc/\",\n",
    "        storage_options={\"anon\": True},\n",
    "    ).persist()\n",
    "\n",
    "    for _ in range(10):\n",
    "        df[\"tipped\"] = df.tips != 0\n",
    "\n",
    "        df.groupby(\"hvfhs_license_num\").tipped.mean().compute()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f4b6b9f9-7ef3-4ca0-b769-0dd7e4ce6b0b",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:coiled]",
   "language": "python",
   "name": "conda-env-coiled-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}