covid19_county_level_EDA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Covid19 County-Level EDA Notebook\n",
    "---\n",
    "> ### [County-Level EDA Notebook](https://github.com/speediedan/covid19/blob/master/covid19_county_level_EDA.ipynb)\n",
    "> * **This notebook can be used for manual EDA of county-level hotspot data**\n",
    "\n",
    "> ### [The \"Real-Time\" County-Level Dashboard](county_covid_explorer.html):\n",
    "> * **A \"real-time\"<sup>[1](#daily-onset-estimation)</sup> county-level dashboard w/ a focus on estimated effective reproduction number (R<sub>t</sub>)<sup>[2](#effective-reproduction-number-estimation)</sup>, 2nd order growth rates and confirmed infection density for most US counties (counties w/ > 0.03% confirmed infection density and > 1000 cases)**\n",
    "\n",
    "> ### [The \"Real-Time\" Choropleth Dashboard](choropleth_covid_county_explorer.html):\n",
    "> * **State and national choropleths for exploring the geographic distribution of \"real-time\"<sup>[1](#daily-onset-estimation)</sup> county-level R<sub>t</sub><sup>[2](#effective-reproduction-number-estimation)</sup> along with other relevant epidemiological statistics. Due to resource constraints, the national choropleth represents exclusively R<sub>t</sub> data while the state choropleths include additional county-level metrics. The national choropleth can currently be temporally evolved over a 14-day horizon.**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Daily Onset Estimation\n",
    "* It's important to be clear that these county-level R<sub>t</sub> estimates are \"real-time\" in the sense that the approach outlined in [(Bettencourt & Ribeiro, 2008)](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0002185) is used while convolving the latest [onset-confirmed latency distribution](https://github.com/beoutbreakprepared/nCoV2019/tree/master/latest_data) onto daily reported cases (then adjusting for right-censoring) to obtain the estimated daily onset values. The latency between case onset and confirmation/reporting means that significant changes in local conditions still require some time (days) to be fully reflected in the R<sub>t</sub> estimates, but the estimate for a given point in time should improve with each passing day to a degree roughly correlated with the aforementioned onset-delay distribution.\n",
    "\n",
    "### Effective Reproduction Number Estimation\n",
    "   * I've extended [this great notebook](https://github.com/k-sys/covid-19/blob/master/Realtime%20R0.ipynb) to a county-level. \n",
    "   * Importantly, it should be noted that (as of 2020.05.12) access to testing is continuing to increase and test positivity rates are therefore changing at a [substantial rate](https://www.cdc.gov/coronavirus/2019-ncov/covid-data/covidview/index.html). As the testing bias continues to evolve in the near-term, one should recognize that point R<sub>t</sub> estimates will be biased to be higher than ground truth R<sub>t</sub>. There are approaches that can [mitigate this bias to a limited extent](http://freerangestats.info/blog/2020/05/09/covid-population-incidence) but fundamentally, we don't have sufficient data to eliminate the bias at this point so I've deprioritized making those model adjustments at the moment (I may make testing-related adjustments in the future though and PRs are welcome!). Fortunately, as testing access and bias stabilize at a level that increases validity of confirmed case counts, these R<sub>t</sub> estimates should become increasingly accurate. I think we can expect hotspot monitoring tools such as this to have utility for a number of months, so this initial period of testing volatility does not nullify their value.\n",
    "   * The most salient change I've made in the process of the extension is that rather than using a prior of gamma-distributed generation intervals to estimate R (which seems totally reasonable), I'm experimenting with incorporating more locally-relevant information by calculating an R<sub>0</sub> using initial incidence data from each locality.\n",
    "   * For execution environments that are compute-constrained, I've also provided (but left disabled) some performance enhancing functions that cut execution time by about 50% at the cost of ~5% accuracy.\n",
    "   \n",
    "### SEIR Model Notes\n",
    "* #### At the time the SEIR model component of this notebook was written (2020.03.30) there remained significant uncertainty regarding some sars-cov-2 parameters. The data fit varied substantially by county so I used what I perceived (N.B.: w/ no personal epidemiological expertise!!) to be the consensus values, documented below:\n",
    "\n",
    "| Parameter   | Source  | Reference Value     |\n",
    "| :---        | :----:  |     ---:            |\n",
    "| Latent Period   | [Lin et al., 2020](https://www.ijidonline.com/article/S1201-9712(20)30117-X/fulltext) | 3   |\n",
    "| Latent Period   | [Wu et al., 2020](https://www.sciencedirect.com/science/article/pii/S0140673620302609) | 3     |\n",
    "| Latent Period   | [Li et al., 2020](https://www.medrxiv.org/content/10.1101/2020.03.06.20031880v1.full.pdf) | 2 |\n",
    "| Serial Interval | [Nishura et al. 2020](https://www.ijidonline.com/article/S1201-9712(20)30119-3/pdf) | 4.6 |\n",
    "| Serial Interval | [Li et al., 2020](https://www.nejm.org/doi/pdf/10.1056/NEJMoa2001316?articleTools=true) | 7.5 |\n",
    "| Incubation Period | [Li et al., 2020](https://www.nejm.org/doi/pdf/10.1056/NEJMoa2001316?articleTools=true) | 5.2 |\n",
    "| Infectious Period | [Li et al., 2020](https://www.nejm.org/doi/pdf/10.1056/NEJMoa2001316?articleTools=true) | 2.3 |\n",
    "| Infectious Period | [Zhou et al., 2020](https://www.medrxiv.org/content/10.1101/2020.02.24.20026773v1.full.pdf) | 6 |\n",
    "| Infectious Period | [Bi et al., 2020](https://www.medrxiv.org/content/10.1101/2020.03.03.20028423v3) | 1.5\n",
    "| Infectious Period | [Kucharski et al., 2020](https://cmmid.github.io/topics/covid19/current-patterns-transmission/wuhan-early-dynamics.html) | 2.9\n",
    "| Time to Hospitalization | [Huang et al., 2020](https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30183-5/fulltext) | 8\n",
    "| Mean Hospitalization Period | [Wang et al., 2020](https://jamanetwork.com/journals/jama/fullarticle/2761044?guestAccessKey=f61bd430-07d8-4b86-a749-bec05bfffb65) | 12\n",
    "| Hospitalization Rate | [Ferguson et al., 2020](https://spiral.imperial.ac.uk/bitstream/10044/1/77482/5/Imperial%20College%20COVID19%20NPI%20modelling%2016-03-2020.pdf) (weighted by us demo by [Covid Act Now](https://covidactnow.org/model)) | 0.073"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from IPython.core.debugger import set_trace\n",
    "import datetime\n",
    "import pandas as pd\n",
    "import c19_analysis.dataprep_utils as covid_utils\n",
    "from c19_analysis.dataprep_flow import build_latest_case_data\n",
    "import config\n",
    "import c19_analysis.dataprep_utils as covid_utils\n",
    "import c19_analysis.bayesian_rt_est as bayes_rt\n",
    "# import c19_analysis.cust_seir_model as cust_seir\n",
    "import dashboard.rt_explorer as rt_explorer\n",
    "import dashboard.choropleth_explorer as choropleth_explorer\n",
    "import dashboard.static_mpl_viz as static_mpl_viz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build/Update Core Case Data\n",
    "covid_delta_df, updated = build_latest_case_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# explore specific counties in more detail...\n",
    "target_counties = ['New York County, NY', 'King County, WA']\n",
    "time_delta = 8\n",
    "for c in target_counties:\n",
    "    s = covid_delta_df[(covid_delta_df.index.get_level_values('name') == c) & (covid_delta_df.index.get_level_values('Date') >  (datetime.datetime.today() - datetime.timedelta(time_delta)))]\n",
    "    s = s.style.apply(covid_utils.color_mask, subset=['2nd_order_growth'], thresh=0.0)\n",
    "    display(s)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update Effective R Estimates\n",
    "# Bayesian Rt Estimation\n",
    "rt_df = bayes_rt.gen_rt_df(covid_delta_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The applicable scope of this particular SEIR model was only the initial, exponential stage of the outbreak and remains here for reference only.\n",
    "# cust_seir.gen_seir_viz(rt_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Prepare dataframes necessary for downstream dashboard generation\n",
    "# rt_df = core rt_explorer dataframe source\n",
    "# viz_df_instances = used to generate temporal evolution of national choropleth\n",
    "# status_df = rt_explorer table dataframe source\n",
    "# county_date_instances: list of dates for which to generate national choropleth\n",
    "rt_df, viz_df_instances, status_df, county_date_instances = covid_utils.prep_dashboard_dfs(rt_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# feel free to explore the data using the dataframes generated above\n",
    "# e.g.:\n",
    "# view 5 largest Rt\n",
    "s = status_df.nlargest(5, 'Rt')\n",
    "s = s.style.apply(covid_utils.color_mask, subset=['Rt'], thresh=1.0)\n",
    "display(s)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}