D01. Advanced Data Structures.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# D01: Advanced Data Structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson we'll be meeting some data structures as part of Numpy and Pandas, namely arrays, series and dataframes. As analysts, you'll have likely used such data structures before and be quite comfortable with their use already, but learning how to deal with them in Python is the basis for almost every piece of work that you do and is important!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing and using Pandas and Numpy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The conventions for importing Pandas and Numpy are as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Structures"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As I mentioned previously, we're going to be meeting and working with some new data structures in this section of the course as follows:\n",
    "\n",
    "* Numpy arrays\n",
    "* Pandas series\n",
    "* Pandas dataframes\n",
    "\n",
    "As I've mentioned previously knowling how these data structures work and how to convert between them is crucial to performing efficient and accurate data analysis using Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Numpy Arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NumPy’s main object is the multidimensional array. It is a table of elements (usually numbers), all of the same type. In Numpy dimensions are called axes. The number of axes is by rank.\n",
    "\n",
    "For example, the coordinates of a point in 3D space [1, 2, 1] is an array of rank 1, because it has one axis. That axis has a length of 3. \n",
    "\n",
    "In example pictured below, the array has rank 2 (it is 2-dimensional). The first dimension (axis) has a length of 2, the second dimension has a length of 3:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "[[ 1., 0., 0.],\n",
    " [ 0., 1., 2.]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can read more about arrays <a href = \"https://docs.scipy.org/doc/numpy-dev/user/quickstart.html\">here</a>.\n",
    "\n",
    "We can create arrays as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr1 = np.random.random(10)  # Creating an array of 10 random numbers\n",
    "print(type(arr1))            # Printing the object type\n",
    "arr1                         # Calling the array"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the axis and length described above, we can create multi-dimensional arrays as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr2 = np.random.random((6,4))     # Creating a 6 (axis) x 4 (length) array with random numbers\n",
    "arr2                               "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can use indexing on them just as we can with other data structures..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr2[0]  # Array indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr2[0][0] # Array nested indexing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As with previously, data structures are dynamic and you can store arrays in other data strucutres..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr3 = np.random.random((2,2))\n",
    "arr4 = np.random.random((2,5))\n",
    "list = [arr3,arr4]               # Creating a list of arrays\n",
    "list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And vice versa by using traditional Python data structures to create arrays:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "list1 = [1,2,3,4,5]\n",
    "list2 = [6,7,8,9,0]\n",
    "arr5 = np.array([list1,list2])\n",
    "arr5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also perform mathematical calculations on arrays quickly and simply:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.add(arr5,5)                # Adds a value to the array\n",
    "arr6 = np.multiply(arr5,arr4) # Multiplys an array with the values from another array\n",
    "arr6 = np.round(arr6,3)       # Rounds an array to a given number of decimals\n",
    "arr6                          "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can store character data in arrays..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "list3 = ['A','B','C','D','E']\n",
    "arr7 = np.array(list3)\n",
    "arr7"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However (rather unsurprisingly) you can't perform mathematical functions on character data, making arrays a poor choice for storing it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "np.multiply(arr7,2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However, if we want to store non numeric data , Pandas is a much better choice!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas Series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Pandas series is very similar to an Numpy array..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ser1 = pd.Series(list1)    # Creating a pandas series using the Series class\n",
    "print(type(ser1))\n",
    "ser1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However there are some differences! Firstly Pandas will create a formal index for our series and display this when we call our series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "print(type(ser1.index))  # Printing the index object details\n",
    "print(ser1.index)        # Printing the index of a series"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see our series also has a separate object for an index. We can actually create this ourselves and it doesn't have to be numeric either..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ser1 = pd.Series(data=list1,index=['A','B','C','D','E'])   # Creating a series with a custom index\n",
    "print(ser1)                                                # Printing the series\n",
    "ser1['A']                                                  # Calling a value with a custom index value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike numpy arrays, series can contiain a variety of data types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "list3 = [1,2.1,'Three',True,None] # List of different data types\n",
    "ser2 = pd.Series(list3)           # Creating a series containing different data types\n",
    "ser2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also create a series from a numpy array using the Series class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr8 = np.random.random(5)  # Creatign a 1D array with 5 records\n",
    "ser3 = pd.Series(arr8)      # Creating a series from a np array\n",
    "ser3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However series are one dimensional so the array must be one dimensional also for it to work!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ser3 = pd.Series(arr3)   # Failing to create a Series from a multidimensional array!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also convert series to arrays using the values method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "arr9 = ser3.values    # Converting a pandas series to a numpy array\n",
    "arr9"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pandas Dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dataframes are something which we should already be familiar with as analysts as they conform to the traditional 'dataset' structure that we're used to dealing with. Creating dataframes is slightly more involving than creating a series or array but still quite simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "index_data = np.arange(3)           # Index data\n",
    "data1 = np.random.random(3)         # Row 1 data\n",
    "data2 = np.random.random(3)         # Row 2 data\n",
    "data3 = np.random.random(3)         # Row 3 data\n",
    "cols_data = ['col1','col2','col3']\n",
    "\n",
    "# Creating a basic dataframe\n",
    "\n",
    "df1 = pd.DataFrame(data=[data1,data2,data3],  # Specifying the data\n",
    "                  index=index_data,           # Specifying the index\n",
    "                  columns=cols_data)          # Specifying the column headers\n",
    "df1                                           # Calling the dataframe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also create a dataframe from a dictionary, and like a series, the columns can contain different data types:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Creating a dataframe from a dictionary\n",
    "\n",
    "df2 = pd.DataFrame({'col1':['A','B','C'],   # Setting column1 data\n",
    "                    'col2':[1,2,3]})        # Setting column2 data\n",
    "\n",
    "df2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also call individual columns from a dataframe..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "df2['col1']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And, as it turns out, dataframe columns are actually series!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "type(df2['col1'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we could actually think of a dataframe as a collection of series, one for each column. This makes understanding the link between series and dataframes a bit simpler and easier.\n",
    "\n",
    "We'll be looking at dataframes in more depth in the upcoming sections."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Converting to lists and dicts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There will doubtless be times when you'll want to convert data you have stored in arrays, series and dataframes to the standard data structures in base Python. Often when installing smaller libraries for specific functions, you'll find that they don't recognise the Numpy and Pandas data structures.\n",
    "\n",
    "Fortunately converting is very simple:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "list1 = arr1.tolist()         # Converting an array to a list\n",
    "list2 = df2['col1'].tolist()  # Converting a series to a list\n",
    "\n",
    "dict1 = df2['col1'].to_dict() # Converting a series to a dict \n",
    "dict2 = df2.to_dict()         # Converting a dataframe to a dict\n",
    "\n",
    "print(list1)\n",
    "print(dict1)\n",
    "\n",
    "print(list2)\n",
    "print(dict2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we've seen, Arrays, Series and Dataframes are all essentially different ways of storing data.\n",
    "\n",
    "Each structure has it's own strengths:\n",
    "\n",
    "<strong>Arrays:</strong>\n",
    "\n",
    "* Excellent for performing mathematical functions\n",
    "* Multidimensional\n",
    "\n",
    "<strong>Series:</strong>\n",
    "\n",
    "* Great for storing a set of one dimensional values of mixed types\n",
    "* Formal indexing\n",
    "* Integrate very well with Pandas dataframes\n",
    "\n",
    "<strong>Dataframes:</strong>\n",
    "\n",
    "* Traditional 'dataset' structure\n",
    "* Formal indexing\n",
    "* Good for a wide variety of data of different types\n",
    "\n",
    "Speaking from experience, I spend most of my time working with dataframes. However it is useful to be able to convert a dataframe column (aka a series!) to a numpy array to perform a function or to transform the data before merging it back to the original dataframe. This is a vital part of using Python for data analysis!\n",
    "\n",
    "Many of the smaller libraries that perform niche functions will be built to be compatible with numpy (or even just the basic data structures such as lists, tuples and dictionaries), but not pandas, as pandas is a newer package. The same can be said for many of the data visualisation libraries. This makes understanding, transforming and converting between these data structures especially important."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One other thing to note is the both pd.DataFrame and pd.Series are something called 'Classes' which is an Object Orientated Programming (OOP) concept. We won't be delving into how these work behind the scenes but it important to know that whilst classes are different to functions that we've met previously, for now you can just think of them as a special kind of function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Further Reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href = \"http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html\">Numpy Array Reference</a><br/>\n",
    "<a href = \"http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.html#pandas.DataFrame\">Pandas DataFrame reference</a><br/>\n",
    "<a href = \"http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.html\">Pandas Series reference</a><br/>\n",
    "<a href = \"http://www.jesshamrick.com/2011/05/18/an-introduction-to-classes-and-inheritance-in-python/\">Introduction to Classes</a><br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}