s3761977_FortnightlyTask1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# COSC2669 Case Studies in Data Science\n",
    "# Fortnightly Task 1\n",
    "## Name: Bhargav Rele (s3761977)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 175,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Packages required for this notebook\n",
    "from IPython.display import Image\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import altair as alt\n",
    "\n",
    "from sklearn import preprocessing \n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn import metrics\n",
    "\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "from sklearn.model_selection import RepeatedStratifiedKFold, GridSearchCV\n",
    "from sklearn.feature_selection import SelectKBest, f_classif, mutual_info_classif\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Job Role"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The description of the job role is as follows:\n",
    "[Note: the job description has been copied and pasted from the original link on the company application portal: https://omnicommediagroup.springboard.com.au/jobs/OMG-1470758?in_site=LINKEDIN]\n",
    "\n",
    "\n",
    "### Job Description\n",
    "##### Senior Data Scientist\n",
    "\n",
    "Annalect is a global data, technology, and analytics consultancy, with offices based in Sydney, Melbourne, Brisbane and major global hubs. We create solutions across data, measurement, analytics, visualisation, and marketing technology to maximise the effectiveness of media and our client's media ROI.\n",
    "\n",
    "When we launched Annalect globally in 2011, we dedicated ourselves to transforming the effects of data and analytics on media. Now we have learned to transform the entire marketing process.\n",
    "\n",
    "Annalect is part of the Omnicom Media Group, and services hundreds of clients across OMD, PHD, Hearts & Science, Resolution Media, Foundation, and OMG United, in addition to working on our own portfolio of direct clients\n",
    "\n",
    "Our work with PHD won B&T's 2019 award for Best Media Campaign for Children's Panadol, and our work with OMD won B&T's 2019 award for Data Driven Marketing for McDonald's Monopoly.\n",
    "\n",
    "##### The role\n",
    "\n",
    "As a Senior Data Scientist in the marketing science team, you will be responsible for helping some of Omnicom's leading clients make informed data driven marketing decisions.\n",
    "\n",
    "Specifically, you will analyse large amounts of data to determine how customers interact with our clients' advertising across different channels and devices.\n",
    "\n",
    "Your focus will be to develop custom attribution and predictive models for both online and offline media. Using these models, you will deliver actionable insights that will drive value to our clients and improve their ROI.\n",
    "\n",
    "This role will allow you refine your work across a number of clients internally and externally. You will be responsible for evangelising the product across the agencies within Omnicom and really own the process that you will be responsible for creating.\n",
    "\n",
    "##### Responsibilities\n",
    "\n",
    "* Your role will be to apply your broad skillset across data and analytics to understand business problems, create insights and envisage practical solutions in areas such as media measurement and customer insights\n",
    "* Build bespoke digital attribution models and improve our digital attribution product offering\n",
    "* Provide business solutions and optimisation through various statistical and quantitative methods.\n",
    "* Provide statistical models to support predictive analytics and deliver non-technical presentations to all levels of the business as well as technical documentation to the wider team.\n",
    "* Derive insights from data and communicate those insights to a non-technical audience through presentations and documentation\n",
    "* Build Annalect's experimentation/predictive modelling ecosystem\n",
    "* Build scalable backend solutions for automation of data processing\n",
    "* Analyse and mash-up massive amounts of data to mine useful business insights\n",
    "\n",
    "##### What you'll need\n",
    "\n",
    "* A strong background in data science, analytics, or data engineering\n",
    "* Proficiency and experience in statistical modelling and machine learning techniques (feature engineering, regression, classification, segmentation, cross validation, bootstrapping, Bayesian techniques etc.)\n",
    "* A high level of proficiency with at least one programming languages used in data science (R/Python/SAS/Scala/MATLAB).\n",
    "* Excellent communication skills - being able to both interpret and convey information in a clear, concise way with people from technical and non-technical background and use data to tell a story\n",
    "* Strong understanding of media, and media data\n",
    "* Ability to work in a fast-paced growth environment.\n",
    "\n",
    "### Domain\n",
    "\n",
    "Annalect is a part of Omnicom Media Group, an American based media company that services clients across several domains and; their primary purpose is to use advanced analytical techniques to enhance marketing decisions and strategies of said clients. Given that Omnicom deals with clients across a diverse range of domains, Annalect uses data and analytics to optimise marketing efforts for companies across healthcare and business (primarily). \n",
    "\n",
    "### Insights or Predictions?\n",
    "\n",
    "As a senior data scientist at Annalect, my primary purpose will be to analyse offline and online customer activity using customer datasets provided by clients. The analysis will utilise machine learning models (such as classification supervised machine learning techniques) to gain an insight into the effectiveness of current marketing strategies by examing the way in which customers interact with advertising. On gaining an insight into such interactions, Annalect may be able to predict whether or not a new customer is likely to make purchases online. Based on the results of the analysis we can make recommendations to clients on how they can maximise return on investment of marketing strategies. Hence, Annalect relies on analytics to not only gain insights but also make predictions. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Data Set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The purpose of this analysis is to create a model that is capable (with a certain degree of accuracy) of predicting whether or not a customer will contribute to the sales revenue of an online retailer. To be able to do so, we shall train and test Classifiers on an existing customer dataset that has been sourced from the UCI Machine Learning Repository (Web Link: https://archive.ics.uci.edu/ml/datasets/Online+Shoppers+Purchasing+Intention+Dataset).\n",
    "\n",
    "The dataset consists of 12,330 online customers of an undisclosed online retailer and, comprises of 10 numerical features and 8 categorical features. The description of each feature can be accessed via the web link provided above. Given that the role requires the applicant to analyse customer interactions with online retailers, the dataset was deemed appropriate.\n",
    "\n",
    "The feature termed 'Revenue' takes up the value, 'TRUE' for those customers that had made purchases online and 'FALSE' for those customers that had not. The dataset consists of 10,422 customers where 'Revenue' = 'FALSE' and 1908 customers where 'Revenue' = 'TRUE'. On being able to predict the 'Revenue' identification of new customers, we may be able to gain an insight into the effectiveness of targetted-marketing strategies while also gaining an insight into how the strategies can be optimised. For instance, a greater portion of marketing efforts can be focused on new customers who are predicted to make purchases online. Alternatively, a greater portion of marketing efforts can also be focused on those customers that are predicted to not make purchases online, thus enabling the client to capture a larger market share ('long tail theory')."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 3: Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the purpose of creating a model that is capable of classifying customers into those that contribute to online sales and those that do not, we use a supervised machine learning technique (i.e., Classification). The variable we are trying to predict is the 'Revenue' variable. Hence, we have a binary classification problem. Furthermore, given that we have 10,422 customers where 'Revenue' = 'FALSE' and 1908 customers where 'Revenue' = 'TRUE', we have a class imbalance problem. However, we assume that the dataset is a statistically accurate representation of customer behaviour online. Hence, we do not need to carry out stratified sampling to balance out the class labels in 'Revenue'. \n",
    "\n",
    "The two classification techniques we will be utilising, is the K-Nearest Neighbour[s] classifier and the Gaussian-Naive Bayes Classifer. For both classifiers, we shall train and test the model on the same dataset using the K-Fold Cross Validation method. We will then utilise Pipelines to construct several models at once (based on several hyperparameter values and feature selection methods). From all the models that are created, we shall pick the one with the highest performance metric. \n",
    "\n",
    "The dataset does not consist of any missing values and is clean for the most part. Nonetheless, we make several adjustments to the dataset in order to prepare it for the pipeline we wish to create. The next section goes over the transformation and preparation of the dataset. The modelling phase is commenced after."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation and Transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Data Importation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 176,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales = pd.read_csv('online_shoppers_intention.csv', error_bad_lines=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 177,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Administrative</th>\n",
       "      <th>Administrative_Duration</th>\n",
       "      <th>Informational</th>\n",
       "      <th>Informational_Duration</th>\n",
       "      <th>ProductRelated</th>\n",
       "      <th>ProductRelated_Duration</th>\n",
       "      <th>BounceRates</th>\n",
       "      <th>ExitRates</th>\n",
       "      <th>PageValues</th>\n",
       "      <th>SpecialDay</th>\n",
       "      <th>Month</th>\n",
       "      <th>OperatingSystems</th>\n",
       "      <th>Browser</th>\n",
       "      <th>Region</th>\n",
       "      <th>TrafficType</th>\n",
       "      <th>VisitorType</th>\n",
       "      <th>Weekend</th>\n",
       "      <th>Revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Feb</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Returning_Visitor</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Feb</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>Returning_Visitor</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Feb</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>3</td>\n",
       "      <td>Returning_Visitor</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2.666667</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Feb</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>Returning_Visitor</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10</td>\n",
       "      <td>627.500000</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Feb</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>Returning_Visitor</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Administrative  Administrative_Duration  Informational  \\\n",
       "0               0                      0.0              0   \n",
       "1               0                      0.0              0   \n",
       "2               0                      0.0              0   \n",
       "3               0                      0.0              0   \n",
       "4               0                      0.0              0   \n",
       "\n",
       "   Informational_Duration  ProductRelated  ProductRelated_Duration  \\\n",
       "0                     0.0               1                 0.000000   \n",
       "1                     0.0               2                64.000000   \n",
       "2                     0.0               1                 0.000000   \n",
       "3                     0.0               2                 2.666667   \n",
       "4                     0.0              10               627.500000   \n",
       "\n",
       "   BounceRates  ExitRates  PageValues  SpecialDay Month  OperatingSystems  \\\n",
       "0         0.20       0.20         0.0         0.0   Feb                 1   \n",
       "1         0.00       0.10         0.0         0.0   Feb                 2   \n",
       "2         0.20       0.20         0.0         0.0   Feb                 4   \n",
       "3         0.05       0.14         0.0         0.0   Feb                 3   \n",
       "4         0.02       0.05         0.0         0.0   Feb                 3   \n",
       "\n",
       "   Browser  Region  TrafficType        VisitorType  Weekend  Revenue  \n",
       "0        1       1            1  Returning_Visitor    False    False  \n",
       "1        2       1            2  Returning_Visitor    False    False  \n",
       "2        1       9            3  Returning_Visitor    False    False  \n",
       "3        2       2            4  Returning_Visitor    False    False  \n",
       "4        3       1            4  Returning_Visitor     True    False  "
      ]
     },
     "execution_count": 177,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 178,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Administrative</th>\n",
       "      <th>Administrative_Duration</th>\n",
       "      <th>Informational</th>\n",
       "      <th>Informational_Duration</th>\n",
       "      <th>ProductRelated</th>\n",
       "      <th>ProductRelated_Duration</th>\n",
       "      <th>BounceRates</th>\n",
       "      <th>ExitRates</th>\n",
       "      <th>PageValues</th>\n",
       "      <th>SpecialDay</th>\n",
       "      <th>Month</th>\n",
       "      <th>OperatingSystems</th>\n",
       "      <th>Browser</th>\n",
       "      <th>Region</th>\n",
       "      <th>TrafficType</th>\n",
       "      <th>VisitorType</th>\n",
       "      <th>Weekend</th>\n",
       "      <th>Revenue</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>12325</td>\n",
       "      <td>3</td>\n",
       "      <td>145.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>53</td>\n",
       "      <td>1783.791667</td>\n",
       "      <td>0.007143</td>\n",
       "      <td>0.029031</td>\n",
       "      <td>12.241717</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Dec</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>Returning_Visitor</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12326</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>5</td>\n",
       "      <td>465.750000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.021333</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Nov</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>Returning_Visitor</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12327</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6</td>\n",
       "      <td>184.250000</td>\n",
       "      <td>0.083333</td>\n",
       "      <td>0.086667</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Nov</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>Returning_Visitor</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12328</td>\n",
       "      <td>4</td>\n",
       "      <td>75.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>15</td>\n",
       "      <td>346.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.021053</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Nov</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>Returning_Visitor</td>\n",
       "      <td>False</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12329</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3</td>\n",
       "      <td>21.250000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.066667</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Nov</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>New_Visitor</td>\n",
       "      <td>True</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Administrative  Administrative_Duration  Informational  \\\n",
       "12325               3                    145.0              0   \n",
       "12326               0                      0.0              0   \n",
       "12327               0                      0.0              0   \n",
       "12328               4                     75.0              0   \n",
       "12329               0                      0.0              0   \n",
       "\n",
       "       Informational_Duration  ProductRelated  ProductRelated_Duration  \\\n",
       "12325                     0.0              53              1783.791667   \n",
       "12326                     0.0               5               465.750000   \n",
       "12327                     0.0               6               184.250000   \n",
       "12328                     0.0              15               346.000000   \n",
       "12329                     0.0               3                21.250000   \n",
       "\n",
       "       BounceRates  ExitRates  PageValues  SpecialDay Month  OperatingSystems  \\\n",
       "12325     0.007143   0.029031   12.241717         0.0   Dec                 4   \n",
       "12326     0.000000   0.021333    0.000000         0.0   Nov                 3   \n",
       "12327     0.083333   0.086667    0.000000         0.0   Nov                 3   \n",
       "12328     0.000000   0.021053    0.000000         0.0   Nov                 2   \n",
       "12329     0.000000   0.066667    0.000000         0.0   Nov                 3   \n",
       "\n",
       "       Browser  Region  TrafficType        VisitorType  Weekend  Revenue  \n",
       "12325        6       1            1  Returning_Visitor     True    False  \n",
       "12326        2       1            8  Returning_Visitor     True    False  \n",
       "12327        2       1           13  Returning_Visitor     True    False  \n",
       "12328        2       3           11  Returning_Visitor    False    False  \n",
       "12329        2       1            2        New_Visitor     True    False  "
      ]
     },
     "execution_count": 178,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Administrative               int64\n",
       "Administrative_Duration    float64\n",
       "Informational                int64\n",
       "Informational_Duration     float64\n",
       "ProductRelated               int64\n",
       "ProductRelated_Duration    float64\n",
       "BounceRates                float64\n",
       "ExitRates                  float64\n",
       "PageValues                 float64\n",
       "SpecialDay                 float64\n",
       "Month                       object\n",
       "OperatingSystems             int64\n",
       "Browser                      int64\n",
       "Region                       int64\n",
       "TrafficType                  int64\n",
       "VisitorType                 object\n",
       "Weekend                       bool\n",
       "Revenue                       bool\n",
       "dtype: object"
      ]
     },
     "execution_count": 179,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Data type specifications\n",
    "\n",
    "The first 10 columns of the dataset are the numeric features. The 8 columns that follow after are the categorical features. It can be observed from the previous output that a few of our categorical features consist of numeric data types. Hence, we convert them to object data types."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales.iloc[:, 10:] = sales.iloc[:, 10:].astype(np.object)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Administrative               int64\n",
       "Administrative_Duration    float64\n",
       "Informational                int64\n",
       "Informational_Duration     float64\n",
       "ProductRelated               int64\n",
       "ProductRelated_Duration    float64\n",
       "BounceRates                float64\n",
       "ExitRates                  float64\n",
       "PageValues                 float64\n",
       "SpecialDay                 float64\n",
       "Month                       object\n",
       "OperatingSystems            object\n",
       "Browser                     object\n",
       "Region                      object\n",
       "TrafficType                 object\n",
       "VisitorType                 object\n",
       "Weekend                     object\n",
       "Revenue                     object\n",
       "dtype: object"
      ]
     },
     "execution_count": 181,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Feature Encoding\n",
    "\n",
    "Prior to encoding, we seperate the descriptive features from the target feature of our dataset. Followed by which, we begin to use one-hot encoding techniques to encode the categorical features of our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 182,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_descrip = sales.iloc[:,0:17]\n",
    "sales_target = sales.iloc[:,17]\n",
    "\n",
    "sales_descrip_onehot = sales_descrip.copy()\n",
    "sales_descrip_onehot['Month'] = sales_descrip_onehot['Month'].replace({'Jan':1,'Feb':2,'Mar':3,'Apr':4,\n",
    "                                                                       'May':5,'Jun':6,'Jul':7,'Aug':8,\n",
    "                                                                       'Sep':9,'Oct':10,'Nov':11,'Dec':12})\n",
    "sales_descrip_onehot['Month'] = sales_descrip_onehot['Month'].astype(np.object)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 183,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_cols = sales_descrip_onehot.columns[sales_descrip_onehot.dtypes==np.object].tolist()\n",
    "\n",
    "for i in cat_cols:\n",
    "    n = len(sales_descrip_onehot[i].unique())\n",
    "    if (n==2):\n",
    "        sales_descrip_onehot[i]=pd.get_dummies(sales_descrip_onehot[i],drop_first=True)\n",
    "\n",
    "sales_descrip_onehot = pd.get_dummies(sales_descrip_onehot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 184,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Administrative</th>\n",
       "      <th>Administrative_Duration</th>\n",
       "      <th>Informational</th>\n",
       "      <th>Informational_Duration</th>\n",
       "      <th>ProductRelated</th>\n",
       "      <th>ProductRelated_Duration</th>\n",
       "      <th>BounceRates</th>\n",
       "      <th>ExitRates</th>\n",
       "      <th>PageValues</th>\n",
       "      <th>SpecialDay</th>\n",
       "      <th>...</th>\n",
       "      <th>TrafficType_14</th>\n",
       "      <th>TrafficType_15</th>\n",
       "      <th>TrafficType_16</th>\n",
       "      <th>TrafficType_17</th>\n",
       "      <th>TrafficType_18</th>\n",
       "      <th>TrafficType_19</th>\n",
       "      <th>TrafficType_20</th>\n",
       "      <th>VisitorType_New_Visitor</th>\n",
       "      <th>VisitorType_Other</th>\n",
       "      <th>VisitorType_Returning_Visitor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>64.000000</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.10</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.20</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>2.666667</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>10</td>\n",
       "      <td>627.500000</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 74 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Administrative  Administrative_Duration  Informational  \\\n",
       "0               0                      0.0              0   \n",
       "1               0                      0.0              0   \n",
       "2               0                      0.0              0   \n",
       "3               0                      0.0              0   \n",
       "4               0                      0.0              0   \n",
       "\n",
       "   Informational_Duration  ProductRelated  ProductRelated_Duration  \\\n",
       "0                     0.0               1                 0.000000   \n",
       "1                     0.0               2                64.000000   \n",
       "2                     0.0               1                 0.000000   \n",
       "3                     0.0               2                 2.666667   \n",
       "4                     0.0              10               627.500000   \n",
       "\n",
       "   BounceRates  ExitRates  PageValues  SpecialDay  ...  TrafficType_14  \\\n",
       "0         0.20       0.20         0.0         0.0  ...               0   \n",
       "1         0.00       0.10         0.0         0.0  ...               0   \n",
       "2         0.20       0.20         0.0         0.0  ...               0   \n",
       "3         0.05       0.14         0.0         0.0  ...               0   \n",
       "4         0.02       0.05         0.0         0.0  ...               0   \n",
       "\n",
       "   TrafficType_15  TrafficType_16  TrafficType_17  TrafficType_18  \\\n",
       "0               0               0               0               0   \n",
       "1               0               0               0               0   \n",
       "2               0               0               0               0   \n",
       "3               0               0               0               0   \n",
       "4               0               0               0               0   \n",
       "\n",
       "   TrafficType_19  TrafficType_20  VisitorType_New_Visitor  VisitorType_Other  \\\n",
       "0               0               0                        0                  0   \n",
       "1               0               0                        0                  0   \n",
       "2               0               0                        0                  0   \n",
       "3               0               0                        0                  0   \n",
       "4               0               0                        0                  0   \n",
       "\n",
       "   VisitorType_Returning_Visitor  \n",
       "0                              1  \n",
       "1                              1  \n",
       "2                              1  \n",
       "3                              1  \n",
       "4                              1  \n",
       "\n",
       "[5 rows x 74 columns]"
      ]
     },
     "execution_count": 184,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_descrip_onehot.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Target Encoding\n",
    "\n",
    "We can then proceed with encoding our target feature such that 'False' = 0 (negative class) and 'True' = 1 (positive class)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_target_encoded = sales_target.copy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False    10422\n",
       "True      1908\n",
       "Name: Revenue, dtype: int64"
      ]
     },
     "execution_count": 186,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_target.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_target_encoded= sales_target_encoded.replace({False:0,True:1})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    10422\n",
       "1     1908\n",
       "Name: Revenue, dtype: int64"
      ]
     },
     "execution_count": 188,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_target_encoded.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Normalisation/Scaling\n",
    "\n",
    "The dataset is then normalised using a robust scaler in order to;\n",
    "1) Normalise the distributions of numeric features.\n",
    "2) Prevent outliers from influencing the normalisation procedure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 189,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_descrip_onehot_norm = sales_descrip_onehot.copy()\n",
    "\n",
    "sales_descrip_onehot_norm = preprocessing.RobustScaler().fit_transform(sales_descrip_onehot)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 190,
   "metadata": {},
   "outputs": [],
   "source": [
    "sales_descrip_onehot_norm = pd.DataFrame(sales_descrip_onehot_norm,\n",
    "                                         columns=sales_descrip_onehot.columns.values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 191,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Administrative</th>\n",
       "      <th>Administrative_Duration</th>\n",
       "      <th>Informational</th>\n",
       "      <th>Informational_Duration</th>\n",
       "      <th>ProductRelated</th>\n",
       "      <th>ProductRelated_Duration</th>\n",
       "      <th>BounceRates</th>\n",
       "      <th>ExitRates</th>\n",
       "      <th>PageValues</th>\n",
       "      <th>SpecialDay</th>\n",
       "      <th>...</th>\n",
       "      <th>TrafficType_14</th>\n",
       "      <th>TrafficType_15</th>\n",
       "      <th>TrafficType_16</th>\n",
       "      <th>TrafficType_17</th>\n",
       "      <th>TrafficType_18</th>\n",
       "      <th>TrafficType_19</th>\n",
       "      <th>TrafficType_20</th>\n",
       "      <th>VisitorType_New_Visitor</th>\n",
       "      <th>VisitorType_Other</th>\n",
       "      <th>VisitorType_Returning_Visitor</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.548387</td>\n",
       "      <td>-0.467912</td>\n",
       "      <td>11.710742</td>\n",
       "      <td>4.895621</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.516129</td>\n",
       "      <td>-0.417913</td>\n",
       "      <td>-0.185128</td>\n",
       "      <td>2.095621</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.548387</td>\n",
       "      <td>-0.467912</td>\n",
       "      <td>11.710742</td>\n",
       "      <td>4.895621</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.516129</td>\n",
       "      <td>-0.465829</td>\n",
       "      <td>2.788840</td>\n",
       "      <td>3.215621</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.258065</td>\n",
       "      <td>0.022315</td>\n",
       "      <td>1.004459</td>\n",
       "      <td>0.695621</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12325</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.474432</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.129032</td>\n",
       "      <td>0.925654</td>\n",
       "      <td>0.239725</td>\n",
       "      <td>0.108478</td>\n",
       "      <td>12.241717</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12326</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.419355</td>\n",
       "      <td>-0.104051</td>\n",
       "      <td>-0.185128</td>\n",
       "      <td>-0.107046</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12327</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.387097</td>\n",
       "      <td>-0.323969</td>\n",
       "      <td>4.771485</td>\n",
       "      <td>1.722287</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12328</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.723812</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.096774</td>\n",
       "      <td>-0.197604</td>\n",
       "      <td>-0.185128</td>\n",
       "      <td>-0.114906</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>12329</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.483871</td>\n",
       "      <td>-0.451311</td>\n",
       "      <td>-0.185128</td>\n",
       "      <td>1.162287</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>12330 rows × 74 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Administrative  Administrative_Duration  Informational  \\\n",
       "0               -0.25                -0.080424            0.0   \n",
       "1               -0.25                -0.080424            0.0   \n",
       "2               -0.25                -0.080424            0.0   \n",
       "3               -0.25                -0.080424            0.0   \n",
       "4               -0.25                -0.080424            0.0   \n",
       "...               ...                      ...            ...   \n",
       "12325            0.50                 1.474432            0.0   \n",
       "12326           -0.25                -0.080424            0.0   \n",
       "12327           -0.25                -0.080424            0.0   \n",
       "12328            0.75                 0.723812            0.0   \n",
       "12329           -0.25                -0.080424            0.0   \n",
       "\n",
       "       Informational_Duration  ProductRelated  ProductRelated_Duration  \\\n",
       "0                         0.0       -0.548387                -0.467912   \n",
       "1                         0.0       -0.516129                -0.417913   \n",
       "2                         0.0       -0.548387                -0.467912   \n",
       "3                         0.0       -0.516129                -0.465829   \n",
       "4                         0.0       -0.258065                 0.022315   \n",
       "...                       ...             ...                      ...   \n",
       "12325                     0.0        1.129032                 0.925654   \n",
       "12326                     0.0       -0.419355                -0.104051   \n",
       "12327                     0.0       -0.387097                -0.323969   \n",
       "12328                     0.0       -0.096774                -0.197604   \n",
       "12329                     0.0       -0.483871                -0.451311   \n",
       "\n",
       "       BounceRates  ExitRates  PageValues  SpecialDay  ...  TrafficType_14  \\\n",
       "0        11.710742   4.895621    0.000000         0.0  ...             0.0   \n",
       "1        -0.185128   2.095621    0.000000         0.0  ...             0.0   \n",
       "2        11.710742   4.895621    0.000000         0.0  ...             0.0   \n",
       "3         2.788840   3.215621    0.000000         0.0  ...             0.0   \n",
       "4         1.004459   0.695621    0.000000         0.0  ...             0.0   \n",
       "...            ...        ...         ...         ...  ...             ...   \n",
       "12325     0.239725   0.108478   12.241717         0.0  ...             0.0   \n",
       "12326    -0.185128  -0.107046    0.000000         0.0  ...             0.0   \n",
       "12327     4.771485   1.722287    0.000000         0.0  ...             0.0   \n",
       "12328    -0.185128  -0.114906    0.000000         0.0  ...             0.0   \n",
       "12329    -0.185128   1.162287    0.000000         0.0  ...             0.0   \n",
       "\n",
       "       TrafficType_15  TrafficType_16  TrafficType_17  TrafficType_18  \\\n",
       "0                 0.0             0.0             0.0             0.0   \n",
       "1                 0.0             0.0             0.0             0.0   \n",
       "2                 0.0             0.0             0.0             0.0   \n",
       "3                 0.0             0.0             0.0             0.0   \n",
       "4                 0.0             0.0             0.0             0.0   \n",
       "...               ...             ...             ...             ...   \n",
       "12325             0.0             0.0             0.0             0.0   \n",
       "12326             0.0             0.0             0.0             0.0   \n",
       "12327             0.0             0.0             0.0             0.0   \n",
       "12328             0.0             0.0             0.0             0.0   \n",
       "12329             0.0             0.0             0.0             0.0   \n",
       "\n",
       "       TrafficType_19  TrafficType_20  VisitorType_New_Visitor  \\\n",
       "0                 0.0             0.0                      0.0   \n",
       "1                 0.0             0.0                      0.0   \n",
       "2                 0.0             0.0                      0.0   \n",
       "3                 0.0             0.0                      0.0   \n",
       "4                 0.0             0.0                      0.0   \n",
       "...               ...             ...                      ...   \n",
       "12325             0.0             0.0                      0.0   \n",
       "12326             0.0             0.0                      0.0   \n",
       "12327             0.0             0.0                      0.0   \n",
       "12328             0.0             0.0                      0.0   \n",
       "12329             0.0             0.0                      1.0   \n",
       "\n",
       "       VisitorType_Other  VisitorType_Returning_Visitor  \n",
       "0                    0.0                            0.0  \n",
       "1                    0.0                            0.0  \n",
       "2                    0.0                            0.0  \n",
       "3                    0.0                            0.0  \n",
       "4                    0.0                            0.0  \n",
       "...                  ...                            ...  \n",
       "12325                0.0                            0.0  \n",
       "12326                0.0                            0.0  \n",
       "12327                0.0                            0.0  \n",
       "12328                0.0                            0.0  \n",
       "12329                0.0                           -1.0  \n",
       "\n",
       "[12330 rows x 74 columns]"
      ]
     },
     "execution_count": 191,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sales_descrip_onehot_norm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 192,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Administrative</th>\n",
       "      <th>Administrative_Duration</th>\n",
       "      <th>Informational</th>\n",
       "      <th>Informational_Duration</th>\n",
       "      <th>ProductRelated</th>\n",
       "      <th>ProductRelated_Duration</th>\n",
       "      <th>BounceRates</th>\n",
       "      <th>ExitRates</th>\n",
       "      <th>PageValues</th>\n",
       "      <th>SpecialDay</th>\n",
       "      <th>...</th>\n",
       "      <th>TrafficType_15</th>\n",
       "      <th>TrafficType_16</th>\n",
       "      <th>TrafficType_17</th>\n",
       "      <th>TrafficType_18</th>\n",
       "      <th>TrafficType_19</th>\n",
       "      <th>TrafficType_20</th>\n",
       "      <th>VisitorType_New_Visitor</th>\n",
       "      <th>VisitorType_Other</th>\n",
       "      <th>VisitorType_Returning_Visitor</th>\n",
       "      <th>Target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>332</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.144762</td>\n",
       "      <td>3.0</td>\n",
       "      <td>116.0</td>\n",
       "      <td>0.096774</td>\n",
       "      <td>0.011507</td>\n",
       "      <td>0.310534</td>\n",
       "      <td>-0.043268</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8274</td>\n",
       "      <td>2.50</td>\n",
       "      <td>2.423162</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.225806</td>\n",
       "      <td>2.853293</td>\n",
       "      <td>-0.185128</td>\n",
       "      <td>-0.349910</td>\n",
       "      <td>14.671986</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>10754</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.486116</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.258065</td>\n",
       "      <td>0.908071</td>\n",
       "      <td>0.225075</td>\n",
       "      <td>-0.082195</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4817</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>-0.548387</td>\n",
       "      <td>-0.467912</td>\n",
       "      <td>11.710742</td>\n",
       "      <td>4.895621</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>436</td>\n",
       "      <td>-0.25</td>\n",
       "      <td>-0.080424</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.096774</td>\n",
       "      <td>-0.063681</td>\n",
       "      <td>0.947812</td>\n",
       "      <td>-0.004379</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 75 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       Administrative  Administrative_Duration  Informational  \\\n",
       "332              0.00                 0.144762            3.0   \n",
       "8274             2.50                 2.423162            0.0   \n",
       "10754            0.75                 0.486116            0.0   \n",
       "4817            -0.25                -0.080424            0.0   \n",
       "436             -0.25                -0.080424            0.0   \n",
       "\n",
       "       Informational_Duration  ProductRelated  ProductRelated_Duration  \\\n",
       "332                     116.0        0.096774                 0.011507   \n",
       "8274                      0.0        3.225806                 2.853293   \n",
       "10754                     0.0        1.258065                 0.908071   \n",
       "4817                      0.0       -0.548387                -0.467912   \n",
       "436                       0.0        0.096774                -0.063681   \n",
       "\n",
       "       BounceRates  ExitRates  PageValues  SpecialDay  ...  TrafficType_15  \\\n",
       "332       0.310534  -0.043268    0.000000         0.0  ...             0.0   \n",
       "8274     -0.185128  -0.349910   14.671986         0.0  ...             0.0   \n",
       "10754     0.225075  -0.082195    0.000000         0.0  ...             0.0   \n",
       "4817     11.710742   4.895621    0.000000         0.0  ...             0.0   \n",
       "436       0.947812  -0.004379    0.000000         0.0  ...             0.0   \n",
       "\n",
       "       TrafficType_16  TrafficType_17  TrafficType_18  TrafficType_19  \\\n",
       "332               0.0             0.0             0.0             0.0   \n",
       "8274              0.0             0.0             0.0             0.0   \n",
       "10754             0.0             0.0             0.0             0.0   \n",
       "4817              0.0             0.0             0.0             0.0   \n",
       "436               0.0             0.0             0.0             0.0   \n",
       "\n",
       "       TrafficType_20  VisitorType_New_Visitor  VisitorType_Other  \\\n",
       "332               0.0                      0.0                0.0   \n",
       "8274              0.0                      0.0                0.0   \n",
       "10754             0.0                      0.0                0.0   \n",
       "4817              0.0                      0.0                0.0   \n",
       "436               0.0                      0.0                0.0   \n",
       "\n",
       "       VisitorType_Returning_Visitor  Target  \n",
       "332                              0.0       0  \n",
       "8274                             0.0       1  \n",
       "10754                            0.0       0  \n",
       "4817                             0.0       0  \n",
       "436                              0.0       0  \n",
       "\n",
       "[5 rows x 75 columns]"
      ]
     },
     "execution_count": 192,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_combo = sales_descrip_onehot_norm\n",
    "df_combo['Target'] = sales_target_encoded\n",
    "df_final = df_combo.sample(n=5000, random_state=999)\n",
    "df_final.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Modelling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to fit the model, we first specify the the training and testing descriptive and target arrays. The datasets are assigned rows at random. Specifying the stratify argument ensures us that each cross validation experiment will have the same imbalance in target features, as the original dataset.\n",
    "\n",
    "[Note: We randomly sample 5000 observations from the dataset. This is to ensure we do not encounter any computational issues while running the models.]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 193,
   "metadata": {},
   "outputs": [],
   "source": [
    "Data = df_final.iloc[:,0:74]\n",
    "target=df_final.iloc[:,74].astype('category')\n",
    "\n",
    "D_train, D_test, t_train, t_test = train_test_split(Data, target, test_size = 0.3, \n",
    "                                                    stratify=target, shuffle=True, \n",
    "                                                    random_state=999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then proceed with specifying the cross validation experiment to be used. We want the sample to be split into 5 equal parts (at random). In each experiment of the cross validation, 1/5th of the class labels will be predicted using 4/5th of the dataset. This will occur until every split has participated in the testing and training procedure. The cross validation will be carried out thrice. Note that we do not run the cross validation experiment outside of the pipeline. We are simply specifying the parameters of the experiment in the immediate code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 194,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_method = RepeatedStratifiedKFold(n_splits=5,\n",
    "                                    n_repeats=3,\n",
    "                                    random_state=999)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### K-Nearest Neighbours "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first classification model we shall use is the K-Nearest Neighbours Algorithm. Pipelines enable us to specify various parameters over which we would like to construct our model. These parameter specifications are as follows:\n",
    "\n",
    "* The F-Score method and the Mutual Information method are specified as feature selection methods. \n",
    "\n",
    "* The model will utilise several combinations of features (30,50,70 or 74 features at a time), within these methods.\n",
    "\n",
    "* The appropriate number of neighbours that determine the prediction for any given observation, can be either; 71, 81, 91 or 101. According to popular literature (Subramanian, 2019), a good approximation of k is given as the square root of the number of observation in the training set. Hence, a good benchmark for k in our model is approximately 92 [=√(12330*0.7)]. Furthermore, given that we have a binary classification task at hand, it is advisable for us to use an odd number of neighbours (Subramanian, 2019). This would prevent the algorithm from being unable to classify observations that are equi-distant from its neighbours.\n",
    "\n",
    "* The appropriate value of 'p' in our distance metric can be either; 1, 2, or 3.\n",
    "\n",
    "* According to Raschka (2018), the scoring metric we should aim to optimise is the F-1 Score (macro/micro/sample/weighted) or the Receiver Operating Charecteristic (ROC-AUC score), given that we have a class imbalance problem in our target feature (i.e., the number of non-churners is almost 4 times the number of churners in our dataset). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 195,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 15 folds for each of 96 candidates, totalling 1440 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done 1440 out of 1440 | elapsed: 24.1min finished\n"
     ]
    }
   ],
   "source": [
    "pipe_KNN = Pipeline( [ ('fselector',SelectKBest()),\n",
    "                     ('knn',KNeighborsClassifier())] )\n",
    "\n",
    "params_pipe_KNN = {'fselector__score_func':[f_classif,mutual_info_classif],\n",
    "                  'fselector__k':[30,50,70,74],\n",
    "                  'knn__n_neighbors':[71,81,91,101],\n",
    "                  'knn__p':[1,2,3]}\n",
    "\n",
    "gs_pipe_KNN = GridSearchCV(estimator=pipe_KNN,\n",
    "                          param_grid = params_pipe_KNN,\n",
    "                          cv=cv_method,\n",
    "                          verbose=1,\n",
    "                          scoring='f1_weighted')\n",
    "\n",
    "gs_pipe_KNN.fit(D_train,t_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 203,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('fselector',\n",
       "                 SelectKBest(k=30,\n",
       "                             score_func=<function mutual_info_classif at 0x1a20dc94d0>)),\n",
       "                ('knn',\n",
       "                 KNeighborsClassifier(algorithm='auto', leaf_size=30,\n",
       "                                      metric='minkowski', metric_params=None,\n",
       "                                      n_jobs=None, n_neighbors=91, p=3,\n",
       "                                      weights='uniform'))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 203,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_pipe_KNN.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence, from 1,440 tentative models that the algorithm has created within the grid search, the model that yields an optimal F1 Weighted Score, has the following parameter specifications:\n",
    "\n",
    "* 30 descriptive features are being used to predict the target feature. (mutual information feature selection)\n",
    "* The optimal value for the number of neighbors is 91\n",
    "* The selected distance metric is the Minkowski distance metric where p=3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 197,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8864922822765989"
      ]
     },
     "execution_count": 197,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_pipe_KNN.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence, the best score in our model is 88.6492%. Now that our model has been trained, we can utilise it to make predictions on the testing data. In particular we can predict the target variable (t_test), by allowing the algorithm to examine the descriptive features of our testing dataset (D_test). \n",
    "\n",
    "We then find the various scoring metrics in order to evaluate the accuracy of our predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 198,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the roc auc score is: 0.7582228255194737\n",
      "the confusion matrix is: [[1184   72]\n",
      " [ 104  140]]\n",
      "the accuracy score is: 0.8826666666666667\n",
      "the classification report is: \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.94      0.93      1256\n",
      "           1       0.66      0.57      0.61       244\n",
      "\n",
      "    accuracy                           0.88      1500\n",
      "   macro avg       0.79      0.76      0.77      1500\n",
      "weighted avg       0.88      0.88      0.88      1500\n",
      "\n",
      "the F-1 score is: 0.6140350877192982\n"
     ]
    }
   ],
   "source": [
    "knn_model=gs_pipe_KNN.best_estimator_\n",
    "\n",
    "t_pred = knn_model.predict(D_test)\n",
    "\n",
    "print('the roc auc score is:',metrics.roc_auc_score(t_test, t_pred))\n",
    "print('the confusion matrix is:',metrics.confusion_matrix(t_test,t_pred))\n",
    "print('the accuracy score is:',metrics.accuracy_score(t_test,t_pred))\n",
    "print('the classification report is: ')\n",
    "print(metrics.classification_report(t_test, t_pred))\n",
    "print('the F-1 score is:',metrics.f1_score(t_test,t_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy of the model is 88.2667%. A non-purchaser is missclassified as a purchaser 72 times while a purchaser is missclassified as a non-purchaser 104 times. Depending on the profit and loss matrix (to be provided by the client), we can determine the significance of these results.\n",
    "\n",
    "The classification report indicates an ideal precision and recall for non-purchasers but the same cannot be said for purchasers. \n",
    "\n",
    "The ROC-AUC metric (0.7582) and the F1-Score for our model (0.6140) is however less than ideal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Random Forest Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next classification model we shall use is the Random Forest Classifier. The parameters over which we would like to construct our model are as follows:\n",
    "\n",
    "* The F-Score method and the Mutual Information method are specified as the appropriate feature selection methods, again. \n",
    "\n",
    "* The model will utilise several combinations of features (20,30,50 or 74 features at a time), within these methods.\n",
    "\n",
    "* The appropriate number of estimators (n_estimators), according to Jan (2013), shouldn't be overestimated in order to prevent overfitting and limit excess strain on computational power. Although the model accuracy may increase by increasing the number of trees being constructed, the cost associated with computation time and overfitting may far outweigh the benefits. Hence, the random forest model will construct 100, 150, 200 or 250 trees at a time, within our grid search.\n",
    "\n",
    "* For the same reasons stated in the K-Nearest Neighbours parameter specifications, we shall choose to optimise the F-1 Score or the ROC-AUC Score in our grid search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 199,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 15 folds for each of 24 candidates, totalling 360 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.\n",
      "[Parallel(n_jobs=1)]: Done 360 out of 360 | elapsed:  7.7min finished\n"
     ]
    }
   ],
   "source": [
    "pipe_RFC = Pipeline( [ ('fselector',SelectKBest()),\n",
    "                     ('rfc',RandomForestClassifier())] )\n",
    "\n",
    "params_pipe_RFC = {'fselector__score_func':[f_classif,mutual_info_classif],\n",
    "                  'fselector__k':[20,30,50,74],\n",
    "                  'rfc__n_estimators':[100,150,200]}\n",
    "\n",
    "gs_pipe_RFC = GridSearchCV(estimator=pipe_RFC,\n",
    "                          param_grid = params_pipe_RFC,\n",
    "                          cv=cv_method,\n",
    "                          verbose=1,\n",
    "                          scoring='f1_weighted')\n",
    "\n",
    "gs_pipe_RFC.fit(D_train,t_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 204,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('fselector',\n",
       "                 SelectKBest(k=20,\n",
       "                             score_func=<function f_classif at 0x1a2017ab90>)),\n",
       "                ('rfc',\n",
       "                 RandomForestClassifier(bootstrap=True, class_weight=None,\n",
       "                                        criterion='gini', max_depth=None,\n",
       "                                        max_features='auto',\n",
       "                                        max_leaf_nodes=None,\n",
       "                                        min_impurity_decrease=0.0,\n",
       "                                        min_impurity_split=None,\n",
       "                                        min_samples_leaf=1, min_samples_split=2,\n",
       "                                        min_weight_fraction_leaf=0.0,\n",
       "                                        n_estimators=100, n_jobs=None,\n",
       "                                        oob_score=False, random_state=None,\n",
       "                                        verbose=0, warm_start=False))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 204,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_pipe_RFC.best_estimator_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence, from 360 tentative models that the algorithm has created within the grid search, the model that yields the highest F1 Weighted, has the following parameter specifications:\n",
    "\n",
    "* the forest consists of 100 decision trees\n",
    "* the min_sample_split is specified as 2\n",
    "* the criterion for splitting is specified as 'gini'\n",
    "* the min_samples_leaf is specified as 1\n",
    "* 20 descriptive features have been utilised in the model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 201,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8928982939240057"
      ]
     },
     "execution_count": 201,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gs_pipe_RFC.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hence, the best score in our model is 89.2898%. Now that our model has been trained, we can utilise it to make predictions on the testing data. In particular we can predict the target variable (t_test), by allowing the algorithm to examine the descriptive features of our testing dataset (D_test).\n",
    "\n",
    "We then find the various scoring metrics in order to evaluate the accuracy of our predictions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 202,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the roc auc score is: 0.7876487939855905\n",
      "the confusion matrix is: [[1191   65]\n",
      " [  91  153]]\n",
      "the accuracy score is: 0.896\n",
      "the classification report is: \n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.95      0.94      1256\n",
      "           1       0.70      0.63      0.66       244\n",
      "\n",
      "    accuracy                           0.90      1500\n",
      "   macro avg       0.82      0.79      0.80      1500\n",
      "weighted avg       0.89      0.90      0.89      1500\n",
      "\n",
      "the F-1 score is: 0.6623376623376623\n"
     ]
    }
   ],
   "source": [
    "rfc_model=gs_pipe_RFC.best_estimator_\n",
    "\n",
    "t_pred = rfc_model.predict(D_test)\n",
    "\n",
    "print('the roc auc score is:',metrics.roc_auc_score(t_test, t_pred))\n",
    "print('the confusion matrix is:',metrics.confusion_matrix(t_test,t_pred))\n",
    "print('the accuracy score is:',metrics.accuracy_score(t_test,t_pred))\n",
    "print('the classification report is: ')\n",
    "print(metrics.classification_report(t_test, t_pred))\n",
    "print('the F-1 score is:',metrics.f1_score(t_test,t_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The accuracy of the model is 89.6%. A non-purchaser is missclassified as a purchaser 65 times while a purchaser is missclassified as a non-purchaser 91 times. Depending on the profit and loss matrix (to be provided by the client), we can determine the significance of these results.\n",
    "\n",
    "The classification report indicates an ideal precision and recall for non-purchasers but the same cannot be said for purchasers. \n",
    "\n",
    "The ROC-AUC metric (0.78) and the F1-Score for our model (0.6623) is however less than ideal."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The performance evaluation metrics indicate that the random forest classifier has greater accuracy in predicting whether or not a customer is going to make purchases online. The random forest classifier has a higher f1-score and roc-auc score than the KNN model; a result that is of greater significance to us given than we have a class imbalance problem. It should also be noted that classification techniques had been utilised to predict whether or not new customers are going to make purchases online, based on the data of previous customers. If the purpose of the analysis is to gain insights into the online behaviour of consumers, we recommend using clustering machine learning algorithms."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "Sebastien, R. Machine Learning FAQ: How can the F1-Score help with dealing with class imbalance?. Sebastianraschka.com. https://sebastianraschka.com/faq/docs/computing-the-f1-score.html\n",
    "\n",
    "Dhilip, S. (2019). A simple introduction to K-Nearest Neighbours algorithm. Towards Data Science. https://towardsdatascience.com/a-simple-introduction-to-k-nearest-neighbors-algorithm-b3519ed98e\n",
    "\n",
    "Varul, A. (2019).  Feature Selection and Ranking in Machine Learning. www.featureranking.com\n",
    "\n",
    "Sakar, C.O., Polat, S.O., Katircioglu, M. et al. Neural Comput & Applic (2018)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}