D06. Summary Statistics & GroupBy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# D06: Summary Statistics & GroupBy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Here we'll be exploring some basic statistical functions we can perform on dataframes. We'll start by importing pandas and numpy and importing some data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "url = 'https://raw.github.com/pydata/pandas/master/pandas/tests/data/tips.csv'\n",
    "df = pd.read_csv(url)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Pandas can produce some excellent summary statistics for us by calling a variety of methods on the dataframe object we've just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "''' Some Basics '''\n",
    "\n",
    "sum_col = df['total_bill'].sum()      # Summing columns (ignores NaN values)\n",
    "cou_col = df['total_bill'].count()    # Counting values in a column (ignores NaN values)\n",
    "min_col = df['tip'].min()             # Returns the minimum value in a column\n",
    "min_idx = df['size'].idxmin()         # Returns the index of the minimum value in a column\n",
    "max_col = df['size'].max()            # Returns the maximum value in a column\n",
    "max_idx = df['size'].idxmax()         # Returns the index of the maximum value in a column\n",
    "\n",
    "''' Averages + Stats '''\n",
    "\n",
    "med_col = df['size'].median()                       # Returns the median  \n",
    "mean_col = df['size'].mean()                        # Returns the mean\n",
    "mode_col = df['size'].mode()                        # Returns the mode\n",
    "std_col = df['tip'].std()                           # Returns the standard deviation\n",
    "var_col = df['tip'].var()                           # Returns the variance\n",
    "qua_col = df['total_bill'].quantile([.25,.5,.75,1]) # Returns the quantile values (note you can use any % values you like!)\n",
    "stats_col = df['total_bill'].describe()             # Returns summary statistics (count, mean, std, min, 25%, 50%, 75%, max)\n",
    "\n",
    "''' Cumulative Values'''\n",
    "\n",
    "df['tip cumulative'] = df['tip'].cumsum()           # Creating a column for the Cumulative Sum\n",
    "df['tip cumulative max'] = df['tip'].cummax()       # Creating a column for the Cumulative Max\n",
    "df['tip cumulative min'] = df['tip'].cummin()       # Creating a column for the Cumulative Min"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we can also specify keyword arguements to the methods. For example controlling how we deal with missing values with the skipna keyword argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sum_col = df['total_bill'].sum(skipna = False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also can be more versatile with how we call methods. They don't have to be on a single column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sum = df.sum()     # Sums the values in the dataset\n",
    "sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "sum2 = df[['size','tip','total_bill']].sum()    # sums the values in the selected columns\n",
    "sum2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Groupby"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whilst this is nice, you'll probably be wanting to create summary tables of statistics rather than storing them as individual objects and this is where <b>groupby</b> can help us.\n",
    "\n",
    "Groupby does three things for us:\n",
    "\n",
    "* <b>Split</b> Splits the data out based upon the input parameters\n",
    "* <b>Apply</b> Applies a function/method/transformation to each group separately \n",
    "* <b>Combine</b> Combines the resulting data into a single data structure\n",
    "\n",
    "You'll hear this referred to as 'Split, Apply, Combine' but this makes things sound a little complicated! In actual fact it's easy to understand once you've seen an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gp1 = df.groupby('sex').sum()         # Simple Groupby statment\n",
    "gp1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we've split out the sex variable from the tips dataset, applied a sum() method and combined the results into a new dataframe.\n",
    "\n",
    "We can group my more than one variable and use different methods as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gp2 = df.groupby(['sex','time']).mean() # Grouping by two variables\n",
    "gp2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You'll notice that the sex and time columns are in bold. This is because groupby automatically creates an index for them - you can create multiple indexes on a dataframe and this is called multiindexing and where things start to get complicated! We could reset the index to a basic numeric index as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gp2.reset_index(inplace=True)        # Reseting the index inplace without creating a new object\n",
    "gp2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the inplace=True keyword argument to the reset_index() method. This allows us to modify the index inplace without specifying to create or overwrite a new object like so:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "gp2 = gp2.reset_index()   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that we could just keep the first level of the index (sex because that was specified first in the groupby function) as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gp2 = df.groupby(['sex','time']).mean()     # Grouping by the sex and time variables\n",
    "gp2 = gp2.reset_index(level=0)                # Reseting the index to level 0\n",
    "gp2.sort_index(inplace=True)                  # Sorting by the index\n",
    "gp2.index.name = None                         # Renaming the index so it displays nicely =)\n",
    "gp2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Often using the groupby function will be the final step before you visualise your data with a chart or graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further Reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href = \"http://pandas.pydata.org/pandas-docs/stable/groupby.html\">GroupBy</a>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}