7C) PCA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Principal Component Analysis "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data import and preparation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sklearn\n",
    "from sklearn.decomposition import PCA\n",
    "import pandas as pandas\n",
    "import numpy as np\n",
    "#load data and merge both tables to one, ignore_index to reindex\n",
    "redwinedata = pandas.read_csv('data/winequality-red.csv', sep =';')\n",
    "whitewinedata = pandas.read_csv('data/winequality-white.csv', sep =';')\n",
    "concat_data = redwinedata.append(whitewinedata, ignore_index=True)\n",
    "# drop the quality label and normalize the data\n",
    "concat_data = concat_data.drop('quality', axis=1)\n",
    "winearray = concat_data.values\n",
    "winearray_norm = sklearn.preprocessing.scale(winearray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### How many of the Principal Components should be used?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "First three PC:\n",
      "[[-0.2387989  -0.3807575   0.15238844  0.34591993 -0.29011259  0.43091401\n",
      "   0.48741806 -0.04493664 -0.21868644 -0.29413517 -0.10643712]\n",
      " [ 0.33635454  0.11754972  0.1832994   0.32991418  0.31525799  0.0719326\n",
      "   0.08726628  0.58403734 -0.155869    0.19171577 -0.46505769]\n",
      " [ 0.4343013  -0.30725942  0.59056967 -0.16468843 -0.0166791  -0.13422395\n",
      "  -0.1074623  -0.17560555 -0.45532412  0.07004248  0.26110053]]\n",
      "Percent of Variance each PC accounts for:\n",
      "[0.2754426  0.22671146 0.14148609 0.08823201 0.06544317 0.05521016\n",
      " 0.04755989 0.04559184 0.03063855 0.02069961 0.00298462]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe id=\"igraph\" scrolling=\"no\" style=\"border:none;\" seamless=\"seamless\" src=\"https://plot.ly/~timohe/14.embed\" height=\"525px\" width=\"100%\"></iframe>"
      ],
      "text/plain": [
       "<plotly.tools.PlotlyDisplay object>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pca = PCA()\n",
    "pca.fit(winearray_norm)\n",
    "print(\"First three PC:\")\n",
    "print(pca.components_[:3]) \n",
    "print(\"Percent of Variance each PC accounts for:\")\n",
    "print(pca.explained_variance_ratio_) \n",
    "\n",
    "\n",
    "import plotly.plotly as py\n",
    "import plotly.graph_objs as go\n",
    "\n",
    "trace1 = go.Scatter(\n",
    "    x=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10],\n",
    "    y=np.cumsum(pca.explained_variance_ratio_),\n",
    "    fill='tozeroy'\n",
    ")\n",
    "layout = go.Layout(\n",
    "    title='Plot Title',\n",
    "    xaxis=dict(\n",
    "        title='x Axis',\n",
    "        titlefont=dict(\n",
    "            family='Courier New, monospace',\n",
    "            size=18,\n",
    "            color='#7f7f7f'\n",
    "        )\n",
    "    ),\n",
    "    yaxis=dict(\n",
    "        title='y Axis',\n",
    "        titlefont=dict(\n",
    "            family='Courier New, monospace',\n",
    "            size=18,\n",
    "            color='#7f7f7f'\n",
    "        )\n",
    "    )\n",
    ")\n",
    "\n",
    "data = [trace1]\n",
    "py.iplot({'data': data, 'layout': {'title': 'Cumultative explained variance ratio', 'font': dict(size=16)}}, filename='basic-area')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on this Graph, my desicion would be to include 4 PC, because it explains 73% of the variance while drastically reducing the number of components."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature-Composition of the most important PC\n",
    "We now look at the correlation between the features and the principal components to see which of the features play into the principal components. This gives us a hint, which features we could drop for our analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# see https://stackoverflow.com/questions/23294616/how-to-use-scikit-learn-pca-for-features-reduction-and-know-which-features-are-d\n",
    "pca = PCA(n_components=4)\n",
    "pca.fit(winearray_norm)\n",
    "#print(pca.components_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# first take absolute values and then normalize this data to get a better overview\n",
    "comp1 = sklearn.preprocessing.scale(np.absolute(pca.components_[0]))\n",
    "comp2 = sklearn.preprocessing.scale(np.absolute(pca.components_[1]))\n",
    "comp3 = sklearn.preprocessing.scale(np.absolute(pca.components_[2]))\n",
    "comp4 = sklearn.preprocessing.scale(np.absolute(pca.components_[3]))\n",
    "\n",
    "df = pandas.DataFrame([comp1,comp2, comp3, comp4])\n",
    "print(df.describe().drop(['count', '25%', '50%', '75%']).round(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having a look at the dataset above, the features (columns) with the highest means means over all 4 components they have the biggest impact.\n",
    "In this case the ranking would be: 9, 8, 0, 2, 5, 1, 3, 6, 10, 4, 7\n",
    "Additionally one could put a weight according to their cumultative explained variance ratio because the PC which explain more of the variance should have a bigger influence in deciding which feature is important.\n",
    "So let's do the same as before with the variance ratio.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#take the explained variance ratio array and use it to weight the component arrays\n",
    "comp1_weighted = pca.explained_variance_ratio_[0] * comp1\n",
    "comp2_weighted = pca.explained_variance_ratio_[1] * comp2\n",
    "comp3_weighted = pca.explained_variance_ratio_[2] * comp3\n",
    "comp4_weighted = pca.explained_variance_ratio_[3] * comp4\n",
    "df_weighted = pandas.DataFrame([comp1_weighted,comp2_weighted, comp3_weighted, comp4_weighted])\n",
    "print(df_weighted.describe().drop(['count', '25%', '50%', '75%']).round(4))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Altough this ranking is quite different, I will compare those two to make a decision:    \n",
    "0, 3, 6, 1, 5, 9, 8, 2, 4, 10, 7 - weighted     \n",
    "9, 8, 0, 2, 5, 1, 3, 6, 10, 4, 7 - unweighted    \n",
    "\n",
    "It can therefore be argued, that the components 4, 7 and 10 can be dropped (index begins at 0).\n",
    "A look at the head of the data shows us which feature those represent:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fixed acidity</th>\n",
       "      <th>volatile acidity</th>\n",
       "      <th>citric acid</th>\n",
       "      <th>residual sugar</th>\n",
       "      <th>chlorides</th>\n",
       "      <th>free sulfur dioxide</th>\n",
       "      <th>total sulfur dioxide</th>\n",
       "      <th>density</th>\n",
       "      <th>pH</th>\n",
       "      <th>sulphates</th>\n",
       "      <th>alcohol</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7.4</td>\n",
       "      <td>0.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.9</td>\n",
       "      <td>0.076</td>\n",
       "      <td>11.0</td>\n",
       "      <td>34.0</td>\n",
       "      <td>0.9978</td>\n",
       "      <td>3.51</td>\n",
       "      <td>0.56</td>\n",
       "      <td>9.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fixed acidity  volatile acidity  citric acid  residual sugar  chlorides  \\\n",
       "0            7.4               0.7          0.0             1.9      0.076   \n",
       "\n",
       "   free sulfur dioxide  total sulfur dioxide  density    pH  sulphates  \\\n",
       "0                 11.0                  34.0   0.9978  3.51       0.56   \n",
       "\n",
       "   alcohol  \n",
       "0      9.4  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "concat_data.head(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore chlorides, density, alcohol can be dropped!\n",
    "Comparing with excercise a), it does only partly reflect the importance of those features in difference of white and red wines, so it must also have found other \"underlying\" factors."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}