AmExpert2019_Final_Submission.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](amex.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# AmExpert 2019 – Machine Learning Hackathon\n",
    "\n",
    "\n",
    "## Introduction\n",
    "American Express and Analytics Vidhya presents “AmExpert 2019 – Machine Learning Hackathon”. An amazing opportunity to showcase your analytical abilities and talent.\n",
    "\n",
    "Get a taste of the kind of challenges we face here at American Express on day to day basis.\n",
    "\n",
    "Exciting prizes up for grabs! Best performers also get a chance to get interviewed by American Express for analytics roles!\n",
    "\n",
    "Don’t hold on to the buzzing ideas in your mind. Just wear your thinking hats and display the spectrum of your creativity!\n",
    "\n",
    "\n",
    "# Problem Statement\n",
    "\n",
    "## Predicting Coupon Redemption\n",
    "\n",
    "XYZ Credit Card company regularly helps it’s merchants understand their data better and take key business decisions accurately by providing machine learning and analytics consulting. ABC is an established Brick & Mortar retailer that frequently conducts marketing campaigns for its diverse product range. As a merchant of XYZ, they have sought XYZ to assist them in their discount marketing process using the power of machine learning. Can you wear the AmExpert hat and help out ABC?\n",
    "\n",
    " \n",
    "Discount marketing and coupon usage are very widely used promotional techniques to attract new customers and to retain & reinforce loyalty of existing customers. The measurement of a consumer’s propensity towards coupon usage and the prediction of the redemption behaviour are crucial parameters in assessing the effectiveness of a marketing campaign.\n",
    "\n",
    " \n",
    "ABC’s promotions are shared across various channels including email, notifications, etc. A number of these campaigns include coupon discounts that are offered for a specific product/range of products. The retailer would like the ability to predict whether customers redeem the coupons received across channels, which will enable the retailer’s marketing team to accurately design coupon construct, and develop more precise and targeted marketing strategies.\n",
    "\n",
    " \n",
    "The data available in this problem contains the following information, including the details of a sample of campaigns and coupons used in previous campaigns -\n",
    "\n",
    "**User Demographic Details**\n",
    "\n",
    "**Campaign and coupon Details**\n",
    "\n",
    "**Product details**\n",
    "\n",
    "**Previous transactions**\n",
    "\n",
    "\n",
    "Based on previous transaction & performance data from the last 18 campaigns, predict the probability for the next 10 campaigns in the test set for each coupon and customer combination, whether the customer will redeem the coupon or not?\n",
    "\n",
    " \n",
    "\n",
    "### Dataset Description\n",
    "\n",
    "Here is the schema for the different data tables available. The detailed data dictionary is provided next.\n",
    "\n",
    "![title](amex19.png)\n",
    "\n",
    "You are provided with the following files in train.zip:\n",
    "\n",
    "**train.csv:** Train data containing the coupons offered to the given customers under the 18 campaigns\n",
    "\n",
    "|Variable |Definition |\n",
    "|----|----|\n",
    "|id\t|Unique id for coupon customer impression|\n",
    "|campaign_id\t|Unique id for a discount campaign|\n",
    "|coupon_id\t|Unique id for a discount coupon|\n",
    "|customer_id\t|Unique id for a customer|\n",
    "|redemption_status|\t(target) (0 - Coupon not redeemed, 1 - Coupon redeemed) |\n",
    "\n",
    "\n",
    "**campaign_data.csv:** Campaign information for each of the 28 campaigns\n",
    "\n",
    "|Variable|\tDefinition|\n",
    "|----|----|\n",
    "|campaign_id\t|Unique id for a discount campaign|\n",
    "|campaign_type\t|Anonymised Campaign Type (X/Y)|\n",
    "|start_date\t|Campaign Start Date|\n",
    "|end_date\t|Campaign End Date |\n",
    "\n",
    "\n",
    "**coupon_item_mapping.csv:** Mapping of coupon and items valid for discount under that coupon\n",
    "\n",
    "|Variable|\tDefinition|\n",
    "|----|----|\n",
    "|coupon_id\t|Unique id for a discount coupon (no order)|\n",
    "|item_id\t|Unique id for items for which given coupon is valid (no order) |\n",
    "\n",
    "\n",
    "**customer_demographics.csv:** Customer demographic information for some customers\n",
    "\n",
    "|Variable\t|Definition|\n",
    "|----|----|\n",
    "|customer_id\t|Unique id for a customer|\n",
    "|age_range\t|Age range of customer family in years|\n",
    "|marital_status|\tMarried/Single|\n",
    "|rented|\t0 - not rented accommodation, 1 - rented accommodation|\n",
    "|family_size|\tNumber of family members|\n",
    "|no_of_children\t|Number of children in the family|\n",
    "|income_bracket|\tLabel Encoded Income Bracket (Higher income corresponds to higher number) |\n",
    "\n",
    "\n",
    "**customer_transaction_data.csv:** Transaction data for all customers for duration of campaigns in the train data\n",
    "\n",
    "|Variable\t|Definition|\n",
    "|----|----|\n",
    "|date\t|Date of Transaction|\n",
    "|customer_id|Unique id for a customer|\n",
    "|item_id|\tUnique id for item|\n",
    "|quantity\t|quantity of item bought|\n",
    "|selling_price\t|Sales value of the transaction|\n",
    "|other_discount|\tDiscount from other sources such as manufacturer coupon/loyalty card|\n",
    "|coupon_discount|\tDiscount availed from retailer coupon |\n",
    "\n",
    "\n",
    "**item_data.csv:** Item information for each item sold by the retailer\n",
    "\n",
    "|Variable|\tDefinition|\n",
    "|----|----|\n",
    "|item_id|\tUnique id for itemv\n",
    "|brand\t|Unique id for item brand|\n",
    "|brand_type\t|Brand Type (local/Established)|\n",
    "|category\t|Item Category |\n",
    "\n",
    "\n",
    "**test.csv:** Contains the coupon customer combination for which redemption status is to be predicted\n",
    "\n",
    "|Variable|\tDefinition|\n",
    "|----|----|\n",
    "|id\t|Unique id for coupon customer impression|\n",
    "|campaign_id\t|Unique id for a discount campaign|\n",
    "|coupon_id\t|Unique id for a discount coupon|\n",
    "|customer_id\t|Unique id for a customer |\n",
    "\n",
    "*Campaign, coupon and customer data for test set is also contained in train.zip *\n",
    "\n",
    "**sample_submission.csv:** This file contains the format in which you have to submit your predictions.\n",
    "\n",
    "To summarise the entire process:\n",
    "\n",
    "* Customers receive coupons under various campaigns and may choose to redeem it.\n",
    "* They can redeem the given coupon for any valid product for that coupon as per coupon item mapping within the duration between campaign start date and end date\n",
    "* Next, the customer will redeem the coupon for an item at the retailer store and that will reflect in the transaction table in the column coupon_discount.\n",
    " \n",
    "### Evaluation Metric\n",
    "Submissions are evaluated on area under the ROC curve between the predicted probability and the observed target.\n",
    " \n",
    "\n",
    "### Public and Private Split\n",
    "* Test data is further randomly divided into Public (40%) and Private data (60%)\n",
    "* Your initial responses will be checked and scored on the Public data.\n",
    "* The final rankings would be based on your private score which will be published once the competition is over."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Importing Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/kaggle/input/amexpert/campaign_data.csv\n",
      "/kaggle/input/amexpert/test_QyjYwdj.csv\n",
      "/kaggle/input/amexpert/coupon_item_mapping.csv\n",
      "/kaggle/input/amexpert/item_data.csv\n",
      "/kaggle/input/amexpert/train.csv\n",
      "/kaggle/input/amexpert/customer_transaction_data.csv\n",
      "/kaggle/input/amexpert/customer_demographics.csv\n",
      "/kaggle/input/amexpert/sample_submission_Byiv0dS.csv\n"
     ]
    }
   ],
   "source": [
    "# This Python 3 environment comes with many helpful analytics libraries installed\n",
    "# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n",
    "# For example, here's several helpful packages to load in \n",
    "\n",
    "import numpy as np # linear algebra\n",
    "import pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n",
    "\n",
    "# Input data files are available in the \"../input/\" directory.\n",
    "# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n",
    "\n",
    "import os\n",
    "for dirname, _, filenames in os.walk('/kaggle/input'):\n",
    "    for filename in filenames:\n",
    "        print(os.path.join(dirname, filename))\n",
    "\n",
    "# Any results you write to the current directory are saved as output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(78369, 5) (50226, 4) (92663, 2) (28, 4) (1324566, 7) (760, 7) (74066, 4)\n"
     ]
    }
   ],
   "source": [
    "train=pd.read_csv('/kaggle/input/amexpert/train.csv')\n",
    "s=pd.read_csv('/kaggle/input/amexpert/sample_submission_Byiv0dS.csv')\n",
    "coup_item=pd.read_csv('/kaggle/input/amexpert/coupon_item_mapping.csv')\n",
    "test=pd.read_csv('/kaggle/input/amexpert/test_QyjYwdj.csv')\n",
    "comp=pd.read_csv('/kaggle/input/amexpert/campaign_data.csv')\n",
    "tran=pd.read_csv('/kaggle/input/amexpert/customer_transaction_data.csv')\n",
    "demo=pd.read_csv('/kaggle/input/amexpert/customer_demographics.csv')  # Didnt use due to so many null values\n",
    "item=pd.read_csv('/kaggle/input/amexpert/item_data.csv')\n",
    "print(train.shape,test.shape,coup_item.shape,comp.shape,tran.shape,demo.shape,item.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>redemption_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>27</td>\n",
       "      <td>1053</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>13</td>\n",
       "      <td>116</td>\n",
       "      <td>48</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>9</td>\n",
       "      <td>635</td>\n",
       "      <td>205</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>13</td>\n",
       "      <td>644</td>\n",
       "      <td>1050</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>8</td>\n",
       "      <td>1017</td>\n",
       "      <td>1489</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  campaign_id  coupon_id  customer_id  redemption_status\n",
       "0   1           13         27         1053                  0\n",
       "1   2           13        116           48                  0\n",
       "2   6            9        635          205                  0\n",
       "3   7           13        644         1050                  0\n",
       "4   9            8       1017         1489                  0"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(78369, 5)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0    77640\n",
       "1      729\n",
       "Name: redemption_status, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(train.shape)\n",
    "train.redemption_status.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(28, 4)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>campaign_type</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>Y</td>\n",
       "      <td>21/10/13</td>\n",
       "      <td>20/12/13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>Y</td>\n",
       "      <td>21/10/13</td>\n",
       "      <td>22/11/13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>Y</td>\n",
       "      <td>07/09/13</td>\n",
       "      <td>16/11/13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>23</td>\n",
       "      <td>Y</td>\n",
       "      <td>08/10/13</td>\n",
       "      <td>15/11/13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>21</td>\n",
       "      <td>Y</td>\n",
       "      <td>16/09/13</td>\n",
       "      <td>18/10/13</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   campaign_id campaign_type start_date  end_date\n",
       "0           24             Y   21/10/13  20/12/13\n",
       "1           25             Y   21/10/13  22/11/13\n",
       "2           20             Y   07/09/13  16/11/13\n",
       "3           23             Y   08/10/13  15/11/13\n",
       "4           21             Y   16/09/13  18/10/13"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(comp.shape)\n",
    "comp.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train + Test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.6/site-packages/pandas/core/frame.py:7123: FutureWarning: Sorting because non-concatenation axis is not aligned. A future version\n",
      "of pandas will change to not sort by default.\n",
      "\n",
      "To accept the future behavior, pass 'sort=False'.\n",
      "\n",
      "To retain the current behavior and silence the warning, pass 'sort=True'.\n",
      "\n",
      "  sort=sort,\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>27</td>\n",
       "      <td>1053</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>116</td>\n",
       "      <td>48</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>635</td>\n",
       "      <td>205</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>644</td>\n",
       "      <td>1050</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>1017</td>\n",
       "      <td>1489</td>\n",
       "      <td>9</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   campaign_id  coupon_id  customer_id  id  redemption_status\n",
       "0           13         27         1053   1                0.0\n",
       "1           13        116           48   2                0.0\n",
       "2            9        635          205   6                0.0\n",
       "3           13        644         1050   7                0.0\n",
       "4            8       1017         1489   9                0.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=train.append(test,ignore_index=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Campaign Data - Feat Engg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "comp['start_date']=pd.to_datetime(comp['start_date'],format='%d/%m/%y',dayfirst=True)\n",
    "comp['end_date']=pd.to_datetime(comp['end_date'],format='%d/%m/%y',dayfirst=True)\n",
    "\n",
    "# comp['start_date_d']=comp['start_date'].dt.day.astype('category')\n",
    "# comp['start_date_m']=comp['start_date'].dt.month.astype('category')\n",
    "# comp['start_date_y']=comp['start_date'].dt.year.astype('category')\n",
    "# comp['start_date_w']=comp['start_date'].dt.week.astype('category')\n",
    "\n",
    "\n",
    "# comp['end_date_d']=comp['end_date'].dt.day.astype('category')\n",
    "# comp['end_date_m']=comp['end_date'].dt.month.astype('category')\n",
    "# comp['end_date_y']=comp['end_date'].dt.year.astype('category')\n",
    "# comp['end_date_w']=comp['end_date'].dt.week.astype('category')\n",
    "\n",
    "\n",
    "comp['diff_d']=(comp['end_date']-comp['start_date'])/np.timedelta64(1,'D')\n",
    "comp['diff_m']=(comp['end_date']-comp['start_date'])/np.timedelta64(1,'M')\n",
    "comp['diff_w']=(comp['end_date']-comp['start_date'])/np.timedelta64(1,'W')\n",
    "\n",
    "# comp.drop(['start_date','end_date'],axis=1,inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>unique</th>\n",
       "      <th>top</th>\n",
       "      <th>freq</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>campaign_id</td>\n",
       "      <td>28</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaT</td>\n",
       "      <td>15.5714</td>\n",
       "      <td>9.11827</td>\n",
       "      <td>1</td>\n",
       "      <td>7.75</td>\n",
       "      <td>16.5</td>\n",
       "      <td>23.25</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>campaign_type</td>\n",
       "      <td>28</td>\n",
       "      <td>2</td>\n",
       "      <td>Y</td>\n",
       "      <td>22</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>start_date</td>\n",
       "      <td>28</td>\n",
       "      <td>25</td>\n",
       "      <td>2013-04-22 00:00:00</td>\n",
       "      <td>2</td>\n",
       "      <td>2012-08-12</td>\n",
       "      <td>2013-10-21</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>end_date</td>\n",
       "      <td>28</td>\n",
       "      <td>26</td>\n",
       "      <td>2013-10-18 00:00:00</td>\n",
       "      <td>2</td>\n",
       "      <td>2012-09-21</td>\n",
       "      <td>2013-12-20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>diff_d</td>\n",
       "      <td>28</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaT</td>\n",
       "      <td>41.8571</td>\n",
       "      <td>11.9589</td>\n",
       "      <td>32</td>\n",
       "      <td>32</td>\n",
       "      <td>35.5</td>\n",
       "      <td>49.25</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>diff_m</td>\n",
       "      <td>28</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaT</td>\n",
       "      <td>1.37521</td>\n",
       "      <td>0.392909</td>\n",
       "      <td>1.05136</td>\n",
       "      <td>1.05136</td>\n",
       "      <td>1.16635</td>\n",
       "      <td>1.6181</td>\n",
       "      <td>2.29984</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>diff_w</td>\n",
       "      <td>28</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaT</td>\n",
       "      <td>NaT</td>\n",
       "      <td>5.97959</td>\n",
       "      <td>1.70842</td>\n",
       "      <td>4.57143</td>\n",
       "      <td>4.57143</td>\n",
       "      <td>5.07143</td>\n",
       "      <td>7.03571</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              count unique                  top freq      first       last  \\\n",
       "campaign_id      28    NaN                  NaN  NaN        NaT        NaT   \n",
       "campaign_type    28      2                    Y   22        NaT        NaT   \n",
       "start_date       28     25  2013-04-22 00:00:00    2 2012-08-12 2013-10-21   \n",
       "end_date         28     26  2013-10-18 00:00:00    2 2012-09-21 2013-12-20   \n",
       "diff_d           28    NaN                  NaN  NaN        NaT        NaT   \n",
       "diff_m           28    NaN                  NaN  NaN        NaT        NaT   \n",
       "diff_w           28    NaN                  NaN  NaN        NaT        NaT   \n",
       "\n",
       "                  mean       std      min      25%      50%      75%      max  \n",
       "campaign_id    15.5714   9.11827        1     7.75     16.5    23.25       30  \n",
       "campaign_type      NaN       NaN      NaN      NaN      NaN      NaN      NaN  \n",
       "start_date         NaN       NaN      NaN      NaN      NaN      NaN      NaN  \n",
       "end_date           NaN       NaN      NaN      NaN      NaN      NaN      NaN  \n",
       "diff_d         41.8571   11.9589       32       32     35.5    49.25       70  \n",
       "diff_m         1.37521  0.392909  1.05136  1.05136  1.16635   1.6181  2.29984  \n",
       "diff_w         5.97959   1.70842  4.57143  4.57143  5.07143  7.03571       10  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comp.describe(include='all').T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>campaign_type</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>diff_d</th>\n",
       "      <th>diff_m</th>\n",
       "      <th>diff_w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>24</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-10-21</td>\n",
       "      <td>2013-12-20</td>\n",
       "      <td>60.0</td>\n",
       "      <td>1.971293</td>\n",
       "      <td>8.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>25</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-10-21</td>\n",
       "      <td>2013-11-22</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1.051356</td>\n",
       "      <td>4.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>20</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-09-07</td>\n",
       "      <td>2013-11-16</td>\n",
       "      <td>70.0</td>\n",
       "      <td>2.299842</td>\n",
       "      <td>10.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>23</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-10-08</td>\n",
       "      <td>2013-11-15</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1.248486</td>\n",
       "      <td>5.428571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>21</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-09-16</td>\n",
       "      <td>2013-10-18</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1.051356</td>\n",
       "      <td>4.571429</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   campaign_id campaign_type start_date   end_date  diff_d    diff_m  \\\n",
       "0           24             Y 2013-10-21 2013-12-20    60.0  1.971293   \n",
       "1           25             Y 2013-10-21 2013-11-22    32.0  1.051356   \n",
       "2           20             Y 2013-09-07 2013-11-16    70.0  2.299842   \n",
       "3           23             Y 2013-10-08 2013-11-15    38.0  1.248486   \n",
       "4           21             Y 2013-09-16 2013-10-18    32.0  1.051356   \n",
       "\n",
       "      diff_w  \n",
       "0   8.571429  \n",
       "1   4.571429  \n",
       "2  10.000000  \n",
       "3   5.428571  \n",
       "4   4.571429  "
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "comp.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "      <th>campaign_type</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>diff_d</th>\n",
       "      <th>diff_m</th>\n",
       "      <th>diff_w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>27</td>\n",
       "      <td>1053</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>116</td>\n",
       "      <td>48</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>635</td>\n",
       "      <td>205</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-03-11</td>\n",
       "      <td>2013-04-12</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1.051356</td>\n",
       "      <td>4.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>644</td>\n",
       "      <td>1050</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>1017</td>\n",
       "      <td>1489</td>\n",
       "      <td>9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-02-16</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1.577034</td>\n",
       "      <td>6.857143</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   campaign_id  coupon_id  customer_id  id  redemption_status campaign_type  \\\n",
       "0           13         27         1053   1                0.0             X   \n",
       "1           13        116           48   2                0.0             X   \n",
       "2            9        635          205   6                0.0             Y   \n",
       "3           13        644         1050   7                0.0             X   \n",
       "4            8       1017         1489   9                0.0             X   \n",
       "\n",
       "  start_date   end_date  diff_d    diff_m    diff_w  \n",
       "0 2013-05-19 2013-07-05    47.0  1.544180  6.714286  \n",
       "1 2013-05-19 2013-07-05    47.0  1.544180  6.714286  \n",
       "2 2013-03-11 2013-04-12    32.0  1.051356  4.571429  \n",
       "3 2013-05-19 2013-07-05    47.0  1.544180  6.714286  \n",
       "4 2013-02-16 2013-04-05    48.0  1.577034  6.857143  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df=df.merge(comp,on='campaign_id',how='left')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Item Data and Coupon Item Mapping"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "brand 5528\n",
      "brand_type 2\n",
      "category 19\n"
     ]
    }
   ],
   "source": [
    "for j in ['brand', 'brand_type', 'category']:\n",
    "    print(j,item[j].nunique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "for j in ['brand', 'brand_type', 'category']:\n",
    "    item[j]=item[j].astype('category')\n",
    "    \n",
    "coup_item=coup_item.merge(item,on='item_id',how='left')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1116"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coup_item.coupon_id.nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(   coupon_id  item_id brand   brand_type category\n",
       " 0        105       37    56        Local  Grocery\n",
       " 1        107       75    56        Local  Grocery\n",
       " 2        494       76   209  Established  Grocery\n",
       " 3        522       77   278  Established  Grocery\n",
       " 4        518       77   278  Established  Grocery, (92663, 5))"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "coup_item.head(),coup_item.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customer Transaction - Feat Engg"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>quantity</th>\n",
       "      <th>selling_price</th>\n",
       "      <th>other_discount</th>\n",
       "      <th>coupon_discount</th>\n",
       "      <th>date_d</th>\n",
       "      <th>date_m</th>\n",
       "      <th>date_w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>54253</td>\n",
       "      <td>1</td>\n",
       "      <td>53.43</td>\n",
       "      <td>-13.89</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>31962</td>\n",
       "      <td>1</td>\n",
       "      <td>106.50</td>\n",
       "      <td>-14.25</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>33647</td>\n",
       "      <td>1</td>\n",
       "      <td>67.32</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>48199</td>\n",
       "      <td>1</td>\n",
       "      <td>71.24</td>\n",
       "      <td>-28.14</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date  customer_id  item_id  quantity  selling_price  other_discount  \\\n",
       "0 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "1 2012-01-02         1501    54253         1          53.43          -13.89   \n",
       "2 2012-01-02         1501    31962         1         106.50          -14.25   \n",
       "3 2012-01-02         1501    33647         1          67.32            0.00   \n",
       "4 2012-01-02         1501    48199         1          71.24          -28.14   \n",
       "\n",
       "   coupon_discount date_d date_m date_w  \n",
       "0              0.0      2      1      1  \n",
       "1              0.0      2      1      1  \n",
       "2              0.0      2      1      1  \n",
       "3              0.0      2      1      1  \n",
       "4              0.0      2      1      1  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tran=pd.read_csv('/kaggle/input/amexpert/customer_transaction_data.csv')\n",
    "tran['date']=pd.to_datetime(tran['date'],format='%Y-%m-%d')\n",
    "tran['date_d']=tran['date'].dt.day.astype('category')\n",
    "tran['date_m']=tran['date'].dt.month.astype('category')\n",
    "tran['date_w']=tran['date'].dt.week.astype('category')\n",
    "\n",
    "# tran.drop('date',axis=1,inplace=True)\n",
    "tran.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>quantity</th>\n",
       "      <th>selling_price</th>\n",
       "      <th>other_discount</th>\n",
       "      <th>coupon_discount</th>\n",
       "      <th>date_d</th>\n",
       "      <th>date_m</th>\n",
       "      <th>date_w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>2257</td>\n",
       "      <td>2012-01-11</td>\n",
       "      <td>632</td>\n",
       "      <td>36682</td>\n",
       "      <td>20</td>\n",
       "      <td>156.73</td>\n",
       "      <td>-21.37</td>\n",
       "      <td>0.0</td>\n",
       "      <td>11</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>30413</td>\n",
       "      <td>2012-02-22</td>\n",
       "      <td>1324</td>\n",
       "      <td>9561</td>\n",
       "      <td>20</td>\n",
       "      <td>626.91</td>\n",
       "      <td>-505.80</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>35980</td>\n",
       "      <td>2012-02-28</td>\n",
       "      <td>757</td>\n",
       "      <td>25460</td>\n",
       "      <td>20</td>\n",
       "      <td>118.61</td>\n",
       "      <td>-2.49</td>\n",
       "      <td>0.0</td>\n",
       "      <td>28</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>60726</td>\n",
       "      <td>2012-03-17</td>\n",
       "      <td>1239</td>\n",
       "      <td>18151</td>\n",
       "      <td>20</td>\n",
       "      <td>142.48</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>82841</td>\n",
       "      <td>2012-03-31</td>\n",
       "      <td>767</td>\n",
       "      <td>37397</td>\n",
       "      <td>20</td>\n",
       "      <td>277.84</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>31</td>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1280892</td>\n",
       "      <td>2013-06-16</td>\n",
       "      <td>1483</td>\n",
       "      <td>20360</td>\n",
       "      <td>20</td>\n",
       "      <td>142.48</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16</td>\n",
       "      <td>6</td>\n",
       "      <td>24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1281796</td>\n",
       "      <td>2013-06-17</td>\n",
       "      <td>811</td>\n",
       "      <td>5873</td>\n",
       "      <td>20</td>\n",
       "      <td>121.11</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>17</td>\n",
       "      <td>6</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1285318</td>\n",
       "      <td>2013-06-18</td>\n",
       "      <td>132</td>\n",
       "      <td>20360</td>\n",
       "      <td>20</td>\n",
       "      <td>142.48</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>18</td>\n",
       "      <td>6</td>\n",
       "      <td>25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1309218</td>\n",
       "      <td>2013-06-28</td>\n",
       "      <td>362</td>\n",
       "      <td>72255</td>\n",
       "      <td>20</td>\n",
       "      <td>71.24</td>\n",
       "      <td>-35.62</td>\n",
       "      <td>0.0</td>\n",
       "      <td>28</td>\n",
       "      <td>6</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1317988</td>\n",
       "      <td>2013-06-30</td>\n",
       "      <td>1320</td>\n",
       "      <td>55744</td>\n",
       "      <td>20</td>\n",
       "      <td>237.59</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30</td>\n",
       "      <td>6</td>\n",
       "      <td>26</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>98 rows × 10 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              date  customer_id  item_id  quantity  selling_price  \\\n",
       "2257    2012-01-11          632    36682        20         156.73   \n",
       "30413   2012-02-22         1324     9561        20         626.91   \n",
       "35980   2012-02-28          757    25460        20         118.61   \n",
       "60726   2012-03-17         1239    18151        20         142.48   \n",
       "82841   2012-03-31          767    37397        20         277.84   \n",
       "...            ...          ...      ...       ...            ...   \n",
       "1280892 2013-06-16         1483    20360        20         142.48   \n",
       "1281796 2013-06-17          811     5873        20         121.11   \n",
       "1285318 2013-06-18          132    20360        20         142.48   \n",
       "1309218 2013-06-28          362    72255        20          71.24   \n",
       "1317988 2013-06-30         1320    55744        20         237.59   \n",
       "\n",
       "         other_discount  coupon_discount date_d date_m date_w  \n",
       "2257             -21.37              0.0     11      1      2  \n",
       "30413           -505.80              0.0     22      2      8  \n",
       "35980             -2.49              0.0     28      2      9  \n",
       "60726              0.00              0.0     17      3     11  \n",
       "82841              0.00              0.0     31      3     13  \n",
       "...                 ...              ...    ...    ...    ...  \n",
       "1280892            0.00              0.0     16      6     24  \n",
       "1281796            0.00              0.0     17      6     25  \n",
       "1285318            0.00              0.0     18      6     25  \n",
       "1309218          -35.62              0.0     28      6     26  \n",
       "1317988            0.00              0.0     30      6     26  \n",
       "\n",
       "[98 rows x 10 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tran[tran['quantity']==20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "tran['discount_bin']=tran['coupon_discount'].apply(lambda x: 0 if x>=0 else 1)\n",
    "tran['marked_price']=tran['selling_price']-tran['other_discount']-tran['coupon_discount']\n",
    "tran['disc_percent']=(tran['marked_price']-tran['selling_price'])/tran['selling_price']\n",
    "tran['price_per_quan']=tran['marked_price']/tran['quantity']\n",
    "tran['marked_by_sale']=tran['marked_price']/tran['selling_price']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['date', 'customer_id', 'item_id', 'quantity', 'selling_price',\n",
       "       'other_discount', 'coupon_discount', 'date_d', 'date_m', 'date_w',\n",
       "       'discount_bin', 'marked_price', 'disc_percent', 'price_per_quan',\n",
       "       'marked_by_sale'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tran.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>quantity</th>\n",
       "      <th>selling_price</th>\n",
       "      <th>other_discount</th>\n",
       "      <th>coupon_discount</th>\n",
       "      <th>date_d</th>\n",
       "      <th>date_m</th>\n",
       "      <th>date_w</th>\n",
       "      <th>discount_bin</th>\n",
       "      <th>marked_price</th>\n",
       "      <th>disc_percent</th>\n",
       "      <th>price_per_quan</th>\n",
       "      <th>marked_by_sale</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>brand</th>\n",
       "      <th>brand_type</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>45.95</td>\n",
       "      <td>0.303176</td>\n",
       "      <td>45.95</td>\n",
       "      <td>1.303176</td>\n",
       "      <td>7.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>45.95</td>\n",
       "      <td>0.303176</td>\n",
       "      <td>45.95</td>\n",
       "      <td>1.303176</td>\n",
       "      <td>20.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>45.95</td>\n",
       "      <td>0.303176</td>\n",
       "      <td>45.95</td>\n",
       "      <td>1.303176</td>\n",
       "      <td>29.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>54253</td>\n",
       "      <td>1</td>\n",
       "      <td>53.43</td>\n",
       "      <td>-13.89</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>67.32</td>\n",
       "      <td>0.259966</td>\n",
       "      <td>67.32</td>\n",
       "      <td>1.259966</td>\n",
       "      <td>7.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>54253</td>\n",
       "      <td>1</td>\n",
       "      <td>53.43</td>\n",
       "      <td>-13.89</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>67.32</td>\n",
       "      <td>0.259966</td>\n",
       "      <td>67.32</td>\n",
       "      <td>1.259966</td>\n",
       "      <td>20.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date  customer_id  item_id  quantity  selling_price  other_discount  \\\n",
       "0 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "1 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "2 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "3 2012-01-02         1501    54253         1          53.43          -13.89   \n",
       "4 2012-01-02         1501    54253         1          53.43          -13.89   \n",
       "\n",
       "   coupon_discount date_d date_m date_w  discount_bin  marked_price  \\\n",
       "0              0.0      2      1      1             0         45.95   \n",
       "1              0.0      2      1      1             0         45.95   \n",
       "2              0.0      2      1      1             0         45.95   \n",
       "3              0.0      2      1      1             0         67.32   \n",
       "4              0.0      2      1      1             0         67.32   \n",
       "\n",
       "   disc_percent  price_per_quan  marked_by_sale  coupon_id brand brand_type  \\\n",
       "0      0.303176           45.95        1.303176        7.0    56      Local   \n",
       "1      0.303176           45.95        1.303176       20.0    56      Local   \n",
       "2      0.303176           45.95        1.303176       29.0    56      Local   \n",
       "3      0.259966           67.32        1.259966        7.0    56      Local   \n",
       "4      0.259966           67.32        1.259966       20.0    56      Local   \n",
       "\n",
       "           category  \n",
       "0  Natural Products  \n",
       "1  Natural Products  \n",
       "2  Natural Products  \n",
       "3  Natural Products  \n",
       "4  Natural Products  "
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tran=tran.merge(coup_item,on='item_id',how='left')\n",
    "tran.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(2657495, 19)\n",
      "(2650887, 19) (78369, 5)\n"
     ]
    }
   ],
   "source": [
    "print(tran.shape)\n",
    "tran=tran[tran.duplicated()==False]\n",
    "print(tran.shape,train.shape)\n",
    "# --drop it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>quantity</th>\n",
       "      <th>selling_price</th>\n",
       "      <th>other_discount</th>\n",
       "      <th>coupon_discount</th>\n",
       "      <th>date_d</th>\n",
       "      <th>date_m</th>\n",
       "      <th>date_w</th>\n",
       "      <th>discount_bin</th>\n",
       "      <th>marked_price</th>\n",
       "      <th>disc_percent</th>\n",
       "      <th>price_per_quan</th>\n",
       "      <th>marked_by_sale</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>brand</th>\n",
       "      <th>brand_type</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>45.95</td>\n",
       "      <td>0.303176</td>\n",
       "      <td>45.95</td>\n",
       "      <td>1.303176</td>\n",
       "      <td>7.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>45.95</td>\n",
       "      <td>0.303176</td>\n",
       "      <td>45.95</td>\n",
       "      <td>1.303176</td>\n",
       "      <td>20.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>45.95</td>\n",
       "      <td>0.303176</td>\n",
       "      <td>45.95</td>\n",
       "      <td>1.303176</td>\n",
       "      <td>29.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>54253</td>\n",
       "      <td>1</td>\n",
       "      <td>53.43</td>\n",
       "      <td>-13.89</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>67.32</td>\n",
       "      <td>0.259966</td>\n",
       "      <td>67.32</td>\n",
       "      <td>1.259966</td>\n",
       "      <td>7.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>54253</td>\n",
       "      <td>1</td>\n",
       "      <td>53.43</td>\n",
       "      <td>-13.89</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>67.32</td>\n",
       "      <td>0.259966</td>\n",
       "      <td>67.32</td>\n",
       "      <td>1.259966</td>\n",
       "      <td>20.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        date  customer_id  item_id  quantity  selling_price  other_discount  \\\n",
       "0 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "1 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "2 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "3 2012-01-02         1501    54253         1          53.43          -13.89   \n",
       "4 2012-01-02         1501    54253         1          53.43          -13.89   \n",
       "\n",
       "   coupon_discount date_d date_m date_w  discount_bin  marked_price  \\\n",
       "0              0.0      2      1      1             0         45.95   \n",
       "1              0.0      2      1      1             0         45.95   \n",
       "2              0.0      2      1      1             0         45.95   \n",
       "3              0.0      2      1      1             0         67.32   \n",
       "4              0.0      2      1      1             0         67.32   \n",
       "\n",
       "   disc_percent  price_per_quan  marked_by_sale  coupon_id brand brand_type  \\\n",
       "0      0.303176           45.95        1.303176        7.0    56      Local   \n",
       "1      0.303176           45.95        1.303176       20.0    56      Local   \n",
       "2      0.303176           45.95        1.303176       29.0    56      Local   \n",
       "3      0.259966           67.32        1.259966        7.0    56      Local   \n",
       "4      0.259966           67.32        1.259966       20.0    56      Local   \n",
       "\n",
       "           category  \n",
       "0  Natural Products  \n",
       "1  Natural Products  \n",
       "2  Natural Products  \n",
       "3  Natural Products  \n",
       "4  Natural Products  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tran.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "tran=tran.merge(tran.groupby(['customer_id','date']).agg({'coupon_id':'count','item_id':'count','disc_percent':sum}).reset_index().rename(columns={'coupon_id':'coupon_aquired','item_id':'item_bought','disc_percent':'tot_disc'}),on=['customer_id','date'],how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>quantity</th>\n",
       "      <th>selling_price</th>\n",
       "      <th>other_discount</th>\n",
       "      <th>coupon_discount</th>\n",
       "      <th>date_d</th>\n",
       "      <th>date_m</th>\n",
       "      <th>date_w</th>\n",
       "      <th>...</th>\n",
       "      <th>disc_percent</th>\n",
       "      <th>price_per_quan</th>\n",
       "      <th>marked_by_sale</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>brand</th>\n",
       "      <th>brand_type</th>\n",
       "      <th>category</th>\n",
       "      <th>coupon_aquired</th>\n",
       "      <th>item_bought</th>\n",
       "      <th>tot_disc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>184957</td>\n",
       "      <td>2012-04-05</td>\n",
       "      <td>1052</td>\n",
       "      <td>13774</td>\n",
       "      <td>1</td>\n",
       "      <td>35.62</td>\n",
       "      <td>-31.70</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.889949</td>\n",
       "      <td>67.32</td>\n",
       "      <td>1.889949</td>\n",
       "      <td>21.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>26</td>\n",
       "      <td>32</td>\n",
       "      <td>8.928818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>184966</td>\n",
       "      <td>2012-04-05</td>\n",
       "      <td>1052</td>\n",
       "      <td>30293</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>35.26</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>21.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>26</td>\n",
       "      <td>32</td>\n",
       "      <td>8.928818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>184970</td>\n",
       "      <td>2012-04-05</td>\n",
       "      <td>1052</td>\n",
       "      <td>5207</td>\n",
       "      <td>1</td>\n",
       "      <td>106.86</td>\n",
       "      <td>-21.02</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.196706</td>\n",
       "      <td>127.88</td>\n",
       "      <td>1.196706</td>\n",
       "      <td>21.0</td>\n",
       "      <td>681</td>\n",
       "      <td>Established</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>26</td>\n",
       "      <td>32</td>\n",
       "      <td>8.928818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>184973</td>\n",
       "      <td>2012-04-05</td>\n",
       "      <td>1052</td>\n",
       "      <td>9571</td>\n",
       "      <td>1</td>\n",
       "      <td>106.86</td>\n",
       "      <td>-21.02</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.196706</td>\n",
       "      <td>127.88</td>\n",
       "      <td>1.196706</td>\n",
       "      <td>21.0</td>\n",
       "      <td>681</td>\n",
       "      <td>Established</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>26</td>\n",
       "      <td>32</td>\n",
       "      <td>8.928818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>184978</td>\n",
       "      <td>2012-04-05</td>\n",
       "      <td>1052</td>\n",
       "      <td>47434</td>\n",
       "      <td>1</td>\n",
       "      <td>106.86</td>\n",
       "      <td>-21.02</td>\n",
       "      <td>0.00</td>\n",
       "      <td>5</td>\n",
       "      <td>4</td>\n",
       "      <td>14</td>\n",
       "      <td>...</td>\n",
       "      <td>0.196706</td>\n",
       "      <td>127.88</td>\n",
       "      <td>1.196706</td>\n",
       "      <td>21.0</td>\n",
       "      <td>681</td>\n",
       "      <td>Established</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>26</td>\n",
       "      <td>32</td>\n",
       "      <td>8.928818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2594456</td>\n",
       "      <td>2013-06-22</td>\n",
       "      <td>1052</td>\n",
       "      <td>27522</td>\n",
       "      <td>1</td>\n",
       "      <td>110.07</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>22</td>\n",
       "      <td>6</td>\n",
       "      <td>25</td>\n",
       "      <td>...</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>110.07</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>21.0</td>\n",
       "      <td>634</td>\n",
       "      <td>Established</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>33</td>\n",
       "      <td>34</td>\n",
       "      <td>14.583416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2594461</td>\n",
       "      <td>2013-06-22</td>\n",
       "      <td>1052</td>\n",
       "      <td>36578</td>\n",
       "      <td>1</td>\n",
       "      <td>17.81</td>\n",
       "      <td>-10.33</td>\n",
       "      <td>0.00</td>\n",
       "      <td>22</td>\n",
       "      <td>6</td>\n",
       "      <td>25</td>\n",
       "      <td>...</td>\n",
       "      <td>0.580011</td>\n",
       "      <td>28.14</td>\n",
       "      <td>1.580011</td>\n",
       "      <td>21.0</td>\n",
       "      <td>1337</td>\n",
       "      <td>Established</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>33</td>\n",
       "      <td>34</td>\n",
       "      <td>14.583416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2594464</td>\n",
       "      <td>2013-06-22</td>\n",
       "      <td>1052</td>\n",
       "      <td>47434</td>\n",
       "      <td>1</td>\n",
       "      <td>99.38</td>\n",
       "      <td>-28.50</td>\n",
       "      <td>0.00</td>\n",
       "      <td>22</td>\n",
       "      <td>6</td>\n",
       "      <td>25</td>\n",
       "      <td>...</td>\n",
       "      <td>0.286778</td>\n",
       "      <td>127.88</td>\n",
       "      <td>1.286778</td>\n",
       "      <td>21.0</td>\n",
       "      <td>681</td>\n",
       "      <td>Established</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>33</td>\n",
       "      <td>34</td>\n",
       "      <td>14.583416</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2623916</td>\n",
       "      <td>2013-06-28</td>\n",
       "      <td>1052</td>\n",
       "      <td>5859</td>\n",
       "      <td>3</td>\n",
       "      <td>39.18</td>\n",
       "      <td>-30.99</td>\n",
       "      <td>-14.25</td>\n",
       "      <td>28</td>\n",
       "      <td>6</td>\n",
       "      <td>26</td>\n",
       "      <td>...</td>\n",
       "      <td>1.154671</td>\n",
       "      <td>28.14</td>\n",
       "      <td>2.154671</td>\n",
       "      <td>21.0</td>\n",
       "      <td>1337</td>\n",
       "      <td>Established</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>10.206436</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2623931</td>\n",
       "      <td>2013-06-28</td>\n",
       "      <td>1052</td>\n",
       "      <td>18168</td>\n",
       "      <td>3</td>\n",
       "      <td>53.43</td>\n",
       "      <td>-30.99</td>\n",
       "      <td>0.00</td>\n",
       "      <td>28</td>\n",
       "      <td>6</td>\n",
       "      <td>26</td>\n",
       "      <td>...</td>\n",
       "      <td>0.580011</td>\n",
       "      <td>28.14</td>\n",
       "      <td>1.580011</td>\n",
       "      <td>21.0</td>\n",
       "      <td>1337</td>\n",
       "      <td>Established</td>\n",
       "      <td>Grocery</td>\n",
       "      <td>24</td>\n",
       "      <td>30</td>\n",
       "      <td>10.206436</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>233 rows × 22 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              date  customer_id  item_id  quantity  selling_price  \\\n",
       "184957  2012-04-05         1052    13774         1          35.62   \n",
       "184966  2012-04-05         1052    30293         1          35.26   \n",
       "184970  2012-04-05         1052     5207         1         106.86   \n",
       "184973  2012-04-05         1052     9571         1         106.86   \n",
       "184978  2012-04-05         1052    47434         1         106.86   \n",
       "...            ...          ...      ...       ...            ...   \n",
       "2594456 2013-06-22         1052    27522         1         110.07   \n",
       "2594461 2013-06-22         1052    36578         1          17.81   \n",
       "2594464 2013-06-22         1052    47434         1          99.38   \n",
       "2623916 2013-06-28         1052     5859         3          39.18   \n",
       "2623931 2013-06-28         1052    18168         3          53.43   \n",
       "\n",
       "         other_discount  coupon_discount date_d date_m date_w  ...  \\\n",
       "184957           -31.70             0.00      5      4     14  ...   \n",
       "184966             0.00             0.00      5      4     14  ...   \n",
       "184970           -21.02             0.00      5      4     14  ...   \n",
       "184973           -21.02             0.00      5      4     14  ...   \n",
       "184978           -21.02             0.00      5      4     14  ...   \n",
       "...                 ...              ...    ...    ...    ...  ...   \n",
       "2594456            0.00             0.00     22      6     25  ...   \n",
       "2594461          -10.33             0.00     22      6     25  ...   \n",
       "2594464          -28.50             0.00     22      6     25  ...   \n",
       "2623916          -30.99           -14.25     28      6     26  ...   \n",
       "2623931          -30.99             0.00     28      6     26  ...   \n",
       "\n",
       "         disc_percent  price_per_quan  marked_by_sale  coupon_id  brand  \\\n",
       "184957       0.889949           67.32        1.889949       21.0     56   \n",
       "184966       0.000000           35.26        1.000000       21.0     56   \n",
       "184970       0.196706          127.88        1.196706       21.0    681   \n",
       "184973       0.196706          127.88        1.196706       21.0    681   \n",
       "184978       0.196706          127.88        1.196706       21.0    681   \n",
       "...               ...             ...             ...        ...    ...   \n",
       "2594456      0.000000          110.07        1.000000       21.0    634   \n",
       "2594461      0.580011           28.14        1.580011       21.0   1337   \n",
       "2594464      0.286778          127.88        1.286778       21.0    681   \n",
       "2623916      1.154671           28.14        2.154671       21.0   1337   \n",
       "2623931      0.580011           28.14        1.580011       21.0   1337   \n",
       "\n",
       "          brand_type category coupon_aquired item_bought   tot_disc  \n",
       "184957         Local  Grocery             26          32   8.928818  \n",
       "184966         Local  Grocery             26          32   8.928818  \n",
       "184970   Established  Grocery             26          32   8.928818  \n",
       "184973   Established  Grocery             26          32   8.928818  \n",
       "184978   Established  Grocery             26          32   8.928818  \n",
       "...              ...      ...            ...         ...        ...  \n",
       "2594456  Established  Grocery             33          34  14.583416  \n",
       "2594461  Established  Grocery             33          34  14.583416  \n",
       "2594464  Established  Grocery             33          34  14.583416  \n",
       "2623916  Established  Grocery             24          30  10.206436  \n",
       "2623931  Established  Grocery             24          30  10.206436  \n",
       "\n",
       "[233 rows x 22 columns]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tran[(tran['customer_id']==1052) & (tran['coupon_id']==21)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "tran['coupon_to_item']=tran['item_bought']-tran['coupon_aquired']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>quantity</th>\n",
       "      <th>selling_price</th>\n",
       "      <th>other_discount</th>\n",
       "      <th>coupon_discount</th>\n",
       "      <th>date_d</th>\n",
       "      <th>date_m</th>\n",
       "      <th>date_w</th>\n",
       "      <th>...</th>\n",
       "      <th>price_per_quan</th>\n",
       "      <th>marked_by_sale</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>brand</th>\n",
       "      <th>brand_type</th>\n",
       "      <th>category</th>\n",
       "      <th>coupon_aquired</th>\n",
       "      <th>item_bought</th>\n",
       "      <th>tot_disc</th>\n",
       "      <th>coupon_to_item</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>859832</td>\n",
       "      <td>2012-08-16</td>\n",
       "      <td>413</td>\n",
       "      <td>16887</td>\n",
       "      <td>1</td>\n",
       "      <td>110.07</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>16</td>\n",
       "      <td>8</td>\n",
       "      <td>33</td>\n",
       "      <td>...</td>\n",
       "      <td>110.07</td>\n",
       "      <td>1.0</td>\n",
       "      <td>577.0</td>\n",
       "      <td>408</td>\n",
       "      <td>Established</td>\n",
       "      <td>Pharmaceutical</td>\n",
       "      <td>13</td>\n",
       "      <td>22</td>\n",
       "      <td>2.389521</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "             date  customer_id  item_id  quantity  selling_price  \\\n",
       "859832 2012-08-16          413    16887         1         110.07   \n",
       "\n",
       "        other_discount  coupon_discount date_d date_m date_w  ...  \\\n",
       "859832             0.0              0.0     16      8     33  ...   \n",
       "\n",
       "        price_per_quan  marked_by_sale  coupon_id  brand   brand_type  \\\n",
       "859832          110.07             1.0      577.0    408  Established   \n",
       "\n",
       "              category coupon_aquired item_bought  tot_disc  coupon_to_item  \n",
       "859832  Pharmaceutical             13          22  2.389521               9  \n",
       "\n",
       "[1 rows x 23 columns]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tran[(tran['customer_id']==413) & (tran['coupon_id']==577)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "      <th>campaign_type</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>diff_d</th>\n",
       "      <th>diff_m</th>\n",
       "      <th>diff_w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>121186</td>\n",
       "      <td>18</td>\n",
       "      <td>577</td>\n",
       "      <td>413</td>\n",
       "      <td>109551</td>\n",
       "      <td>NaN</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-08-10</td>\n",
       "      <td>2013-10-04</td>\n",
       "      <td>55.0</td>\n",
       "      <td>1.807019</td>\n",
       "      <td>7.857143</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        campaign_id  coupon_id  customer_id      id  redemption_status  \\\n",
       "121186           18        577          413  109551                NaN   \n",
       "\n",
       "       campaign_type start_date   end_date  diff_d    diff_m    diff_w  \n",
       "121186             X 2013-08-10 2013-10-04    55.0  1.807019  7.857143  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[(df['customer_id']==413) & (df['coupon_id']==577)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>date</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>item_id</th>\n",
       "      <th>quantity</th>\n",
       "      <th>selling_price</th>\n",
       "      <th>other_discount</th>\n",
       "      <th>coupon_discount</th>\n",
       "      <th>date_d</th>\n",
       "      <th>date_m</th>\n",
       "      <th>date_w</th>\n",
       "      <th>...</th>\n",
       "      <th>price_per_quan</th>\n",
       "      <th>marked_by_sale</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>brand</th>\n",
       "      <th>brand_type</th>\n",
       "      <th>category</th>\n",
       "      <th>coupon_aquired</th>\n",
       "      <th>item_bought</th>\n",
       "      <th>tot_disc</th>\n",
       "      <th>coupon_to_item</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>45.95</td>\n",
       "      <td>1.303176</td>\n",
       "      <td>7.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>3.403242</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>45.95</td>\n",
       "      <td>1.303176</td>\n",
       "      <td>20.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>3.403242</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>26830</td>\n",
       "      <td>1</td>\n",
       "      <td>35.26</td>\n",
       "      <td>-10.69</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>45.95</td>\n",
       "      <td>1.303176</td>\n",
       "      <td>29.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>3.403242</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>54253</td>\n",
       "      <td>1</td>\n",
       "      <td>53.43</td>\n",
       "      <td>-13.89</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>67.32</td>\n",
       "      <td>1.259966</td>\n",
       "      <td>7.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>3.403242</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>2012-01-02</td>\n",
       "      <td>1501</td>\n",
       "      <td>54253</td>\n",
       "      <td>1</td>\n",
       "      <td>53.43</td>\n",
       "      <td>-13.89</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>67.32</td>\n",
       "      <td>1.259966</td>\n",
       "      <td>20.0</td>\n",
       "      <td>56</td>\n",
       "      <td>Local</td>\n",
       "      <td>Natural Products</td>\n",
       "      <td>11</td>\n",
       "      <td>12</td>\n",
       "      <td>3.403242</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        date  customer_id  item_id  quantity  selling_price  other_discount  \\\n",
       "0 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "1 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "2 2012-01-02         1501    26830         1          35.26          -10.69   \n",
       "3 2012-01-02         1501    54253         1          53.43          -13.89   \n",
       "4 2012-01-02         1501    54253         1          53.43          -13.89   \n",
       "\n",
       "   coupon_discount date_d date_m date_w  ...  price_per_quan  marked_by_sale  \\\n",
       "0              0.0      2      1      1  ...           45.95        1.303176   \n",
       "1              0.0      2      1      1  ...           45.95        1.303176   \n",
       "2              0.0      2      1      1  ...           45.95        1.303176   \n",
       "3              0.0      2      1      1  ...           67.32        1.259966   \n",
       "4              0.0      2      1      1  ...           67.32        1.259966   \n",
       "\n",
       "   coupon_id  brand  brand_type          category coupon_aquired item_bought  \\\n",
       "0        7.0     56       Local  Natural Products             11          12   \n",
       "1       20.0     56       Local  Natural Products             11          12   \n",
       "2       29.0     56       Local  Natural Products             11          12   \n",
       "3        7.0     56       Local  Natural Products             11          12   \n",
       "4       20.0     56       Local  Natural Products             11          12   \n",
       "\n",
       "   tot_disc  coupon_to_item  \n",
       "0  3.403242               1  \n",
       "1  3.403242               1  \n",
       "2  3.403242               1  \n",
       "3  3.403242               1  \n",
       "4  3.403242               1  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# tran.groupby(['customer_id','coupon_id']).agg({'date':set}).reset_index()\n",
    "tran.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def func(a,b,c):\n",
    "    if c!=0:\n",
    "        c=list(c)\n",
    "        v=0\n",
    "        for k in c:\n",
    "            if a<=k and b>k:\n",
    "                v+=1\n",
    "        return v\n",
    "    else:\n",
    "        return 0\n",
    "# cc['within']=cc.apply(lambda x: func(x['start_date'],x['end_date'],x['date']),axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Magic features\n",
    "# tran.groupby(['customer_id','date']).agg({'coupon_id':'count','discount_bin':sum,'quantity':sum,'item_id':'count'}).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "      <th>campaign_type</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>diff_d</th>\n",
       "      <th>diff_m</th>\n",
       "      <th>diff_w</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>27</td>\n",
       "      <td>1053</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>116</td>\n",
       "      <td>48</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>635</td>\n",
       "      <td>205</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-03-11</td>\n",
       "      <td>2013-04-12</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1.051356</td>\n",
       "      <td>4.571429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>644</td>\n",
       "      <td>1050</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>1017</td>\n",
       "      <td>1489</td>\n",
       "      <td>9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-02-16</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1.577034</td>\n",
       "      <td>6.857143</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   campaign_id  coupon_id  customer_id  id  redemption_status campaign_type  \\\n",
       "0           13         27         1053   1                0.0             X   \n",
       "1           13        116           48   2                0.0             X   \n",
       "2            9        635          205   6                0.0             Y   \n",
       "3           13        644         1050   7                0.0             X   \n",
       "4            8       1017         1489   9                0.0             X   \n",
       "\n",
       "  start_date   end_date  diff_d    diff_m    diff_w  \n",
       "0 2013-05-19 2013-07-05    47.0  1.544180  6.714286  \n",
       "1 2013-05-19 2013-07-05    47.0  1.544180  6.714286  \n",
       "2 2013-03-11 2013-04-12    32.0  1.051356  4.571429  \n",
       "3 2013-05-19 2013-07-05    47.0  1.544180  6.714286  \n",
       "4 2013-02-16 2013-04-05    48.0  1.577034  6.857143  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['date', 'customer_id', 'item_id', 'quantity', 'selling_price',\n",
       "       'other_discount', 'coupon_discount', 'date_d', 'date_m', 'date_w',\n",
       "       'discount_bin', 'marked_price', 'disc_percent', 'price_per_quan',\n",
       "       'marked_by_sale', 'coupon_id', 'brand', 'brand_type', 'category',\n",
       "       'coupon_aquired', 'item_bought', 'tot_disc', 'coupon_to_item'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# cc=df.merge(tran.groupby(['customer_id','date']).agg({'coupon_id':'count','discount_bin':sum,'quantity':sum,'item_id':'count'}).reset_index(),on=['customer_id','date'],how='left')\n",
    "# cc.sample(10)\n",
    "tran.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Best Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "      <th>campaign_type</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>diff_d</th>\n",
       "      <th>diff_m</th>\n",
       "      <th>diff_w</th>\n",
       "      <th>date</th>\n",
       "      <th>discount_bin</th>\n",
       "      <th>quantity</th>\n",
       "      <th>item_id</th>\n",
       "      <th>coupon_aquired</th>\n",
       "      <th>item_bought</th>\n",
       "      <th>tot_disc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>84326</td>\n",
       "      <td>18</td>\n",
       "      <td>690</td>\n",
       "      <td>1324</td>\n",
       "      <td>15101</td>\n",
       "      <td>NaN</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-08-10</td>\n",
       "      <td>2013-10-04</td>\n",
       "      <td>55.0</td>\n",
       "      <td>1.807019</td>\n",
       "      <td>7.857143</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>28507</td>\n",
       "      <td>13</td>\n",
       "      <td>511</td>\n",
       "      <td>101</td>\n",
       "      <td>46731</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>73553</td>\n",
       "      <td>13</td>\n",
       "      <td>418</td>\n",
       "      <td>733</td>\n",
       "      <td>120807</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>27146</td>\n",
       "      <td>13</td>\n",
       "      <td>886</td>\n",
       "      <td>1320</td>\n",
       "      <td>44528</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>74144</td>\n",
       "      <td>8</td>\n",
       "      <td>10</td>\n",
       "      <td>1158</td>\n",
       "      <td>121758</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-02-16</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1.577034</td>\n",
       "      <td>6.857143</td>\n",
       "      <td>{2012-05-27 00:00:00, 2012-03-05 00:00:00}</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>54.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>9.498264</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>65266</td>\n",
       "      <td>29</td>\n",
       "      <td>873</td>\n",
       "      <td>659</td>\n",
       "      <td>107145</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Y</td>\n",
       "      <td>2012-10-08</td>\n",
       "      <td>2012-11-30</td>\n",
       "      <td>53.0</td>\n",
       "      <td>1.741309</td>\n",
       "      <td>7.571429</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>86956</td>\n",
       "      <td>23</td>\n",
       "      <td>1113</td>\n",
       "      <td>911</td>\n",
       "      <td>21767</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-10-08</td>\n",
       "      <td>2013-11-15</td>\n",
       "      <td>38.0</td>\n",
       "      <td>1.248486</td>\n",
       "      <td>5.428571</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>101305</td>\n",
       "      <td>18</td>\n",
       "      <td>730</td>\n",
       "      <td>1133</td>\n",
       "      <td>58775</td>\n",
       "      <td>NaN</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-08-10</td>\n",
       "      <td>2013-10-04</td>\n",
       "      <td>55.0</td>\n",
       "      <td>1.807019</td>\n",
       "      <td>7.857143</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>33676</td>\n",
       "      <td>8</td>\n",
       "      <td>86</td>\n",
       "      <td>809</td>\n",
       "      <td>55184</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-02-16</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1.577034</td>\n",
       "      <td>6.857143</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>100142</td>\n",
       "      <td>18</td>\n",
       "      <td>777</td>\n",
       "      <td>1492</td>\n",
       "      <td>55861</td>\n",
       "      <td>NaN</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-08-10</td>\n",
       "      <td>2013-10-04</td>\n",
       "      <td>55.0</td>\n",
       "      <td>1.807019</td>\n",
       "      <td>7.857143</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        campaign_id  coupon_id  customer_id      id  redemption_status  \\\n",
       "84326            18        690         1324   15101                NaN   \n",
       "28507            13        511          101   46731                0.0   \n",
       "73553            13        418          733  120807                0.0   \n",
       "27146            13        886         1320   44528                0.0   \n",
       "74144             8         10         1158  121758                0.0   \n",
       "65266            29        873          659  107145                0.0   \n",
       "86956            23       1113          911   21767                NaN   \n",
       "101305           18        730         1133   58775                NaN   \n",
       "33676             8         86          809   55184                0.0   \n",
       "100142           18        777         1492   55861                NaN   \n",
       "\n",
       "       campaign_type start_date   end_date  diff_d    diff_m    diff_w  \\\n",
       "84326              X 2013-08-10 2013-10-04    55.0  1.807019  7.857143   \n",
       "28507              X 2013-05-19 2013-07-05    47.0  1.544180  6.714286   \n",
       "73553              X 2013-05-19 2013-07-05    47.0  1.544180  6.714286   \n",
       "27146              X 2013-05-19 2013-07-05    47.0  1.544180  6.714286   \n",
       "74144              X 2013-02-16 2013-04-05    48.0  1.577034  6.857143   \n",
       "65266              Y 2012-10-08 2012-11-30    53.0  1.741309  7.571429   \n",
       "86956              Y 2013-10-08 2013-11-15    38.0  1.248486  5.428571   \n",
       "101305             X 2013-08-10 2013-10-04    55.0  1.807019  7.857143   \n",
       "33676              X 2013-02-16 2013-04-05    48.0  1.577034  6.857143   \n",
       "100142             X 2013-08-10 2013-10-04    55.0  1.807019  7.857143   \n",
       "\n",
       "                                              date  discount_bin  quantity  \\\n",
       "84326                                          NaN           NaN       NaN   \n",
       "28507                                          NaN           NaN       NaN   \n",
       "73553                                          NaN           NaN       NaN   \n",
       "27146                                          NaN           NaN       NaN   \n",
       "74144   {2012-05-27 00:00:00, 2012-03-05 00:00:00}           0.0       3.0   \n",
       "65266                                          NaN           NaN       NaN   \n",
       "86956                                          NaN           NaN       NaN   \n",
       "101305                                         NaN           NaN       NaN   \n",
       "33676                                          NaN           NaN       NaN   \n",
       "100142                                         NaN           NaN       NaN   \n",
       "\n",
       "        item_id  coupon_aquired  item_bought  tot_disc  \n",
       "84326       NaN             NaN          NaN       NaN  \n",
       "28507       NaN             NaN          NaN       NaN  \n",
       "73553       NaN             NaN          NaN       NaN  \n",
       "27146       NaN             NaN          NaN       NaN  \n",
       "74144       2.0            54.0         30.0  9.498264  \n",
       "65266       NaN             NaN          NaN       NaN  \n",
       "86956       NaN             NaN          NaN       NaN  \n",
       "101305      NaN             NaN          NaN       NaN  \n",
       "33676       NaN             NaN          NaN       NaN  \n",
       "100142      NaN             NaN          NaN       NaN  "
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ddf=df.merge(tran.groupby(['customer_id','coupon_id']).agg({'date':set,'discount_bin':sum,'quantity':sum,'item_id':'count',\n",
    "                                                            'coupon_aquired':sum,'item_bought':'mean','tot_disc':sum}).reset_index(),on=['customer_id','coupon_id'],how='left')\n",
    "ddf.sample(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0           0.0\n",
       "1           0.0\n",
       "2           0.0\n",
       "3           0.0\n",
       "4           0.0\n",
       "          ...  \n",
       "128590      0.0\n",
       "128591    216.0\n",
       "128592    391.0\n",
       "128593     62.0\n",
       "128594      0.0\n",
       "Name: coupon_aquired, Length: 128595, dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ddf['coupon_aquired'].fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Features tried but everyone gave 99CV and overfitted LB but only 'within_date' helped me improve**\n",
    "\n",
    "* Single feature that boosted my score from 86.XX to 93.XX"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(128595, 18)\n"
     ]
    }
   ],
   "source": [
    "# def new_df(df):\n",
    "\n",
    "print(ddf.shape)\n",
    "ddf['date'].replace(np.nan,0,inplace=True)\n",
    "ddf['discount_bin'].replace(np.nan,-1,inplace=True)\n",
    "# ddf['quantity'].replace(np.nan,0,inplace=True)\n",
    "# ddf['item_id'].replace(np.nan,0,inplace=True)\n",
    "# df['camp_date_within_count']=ddf.apply(lambda x: func(x['start_date'],x['end_date'],x['date']),axis=1)\n",
    "\n",
    "\n",
    "# df['bin']=ddf['discount_bin'].apply(lambda x: 1 if x!=-1 else 0)\n",
    "df['within_date']=ddf['date'].apply(lambda x: len(x) if x !=0 else 0)\n",
    "# df['C1']=ddf['coupon_aquired'].fillna(0)\n",
    "# df['C2']=ddf['item_bought'].fillna(0)\n",
    "# df['C3']=ddf['tot_disc'].fillna(0)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# df['within_date_discount']=ddf['discount_bin'].apply(lambda x: x if x >=0 else 0)\n",
    "#     df['quantity_date']=ddf['quantity']\n",
    "# df['item_count']=ddf['item_id']\n",
    "\n",
    "# -- worked good\n",
    "    # df['quantity_date']=ddf['quantity']\n",
    "    # df['item_count']=ddf['item_id']\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>customer_id</th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "      <th>campaign_type</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>diff_d</th>\n",
       "      <th>diff_m</th>\n",
       "      <th>diff_w</th>\n",
       "      <th>date</th>\n",
       "      <th>discount_bin</th>\n",
       "      <th>quantity</th>\n",
       "      <th>item_id</th>\n",
       "      <th>coupon_aquired</th>\n",
       "      <th>item_bought</th>\n",
       "      <th>tot_disc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>27</td>\n",
       "      <td>1053</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>116</td>\n",
       "      <td>48</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>635</td>\n",
       "      <td>205</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-03-11</td>\n",
       "      <td>2013-04-12</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1.051356</td>\n",
       "      <td>4.571429</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>644</td>\n",
       "      <td>1050</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>6.714286</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>1017</td>\n",
       "      <td>1489</td>\n",
       "      <td>9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-02-16</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1.577034</td>\n",
       "      <td>6.857143</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.0</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   campaign_id  coupon_id  customer_id  id  redemption_status campaign_type  \\\n",
       "0           13         27         1053   1                0.0             X   \n",
       "1           13        116           48   2                0.0             X   \n",
       "2            9        635          205   6                0.0             Y   \n",
       "3           13        644         1050   7                0.0             X   \n",
       "4            8       1017         1489   9                0.0             X   \n",
       "\n",
       "  start_date   end_date  diff_d    diff_m    diff_w date  discount_bin  \\\n",
       "0 2013-05-19 2013-07-05    47.0  1.544180  6.714286    0          -1.0   \n",
       "1 2013-05-19 2013-07-05    47.0  1.544180  6.714286    0          -1.0   \n",
       "2 2013-03-11 2013-04-12    32.0  1.051356  4.571429    0          -1.0   \n",
       "3 2013-05-19 2013-07-05    47.0  1.544180  6.714286    0          -1.0   \n",
       "4 2013-02-16 2013-04-05    48.0  1.577034  6.857143    0          -1.0   \n",
       "\n",
       "   quantity  item_id  coupon_aquired  item_bought  tot_disc  \n",
       "0       NaN      NaN             NaN          NaN       NaN  \n",
       "1       NaN      NaN             NaN          NaN       NaN  \n",
       "2       NaN      NaN             NaN          NaN       NaN  \n",
       "3       NaN      NaN             NaN          NaN       NaN  \n",
       "4       NaN      NaN             NaN          NaN       NaN  "
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ddf.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr:last-of-type th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"2\" halign=\"left\">date_d</th>\n",
       "      <th colspan=\"2\" halign=\"left\">date_m</th>\n",
       "      <th colspan=\"2\" halign=\"left\">date_w</th>\n",
       "      <th colspan=\"4\" halign=\"left\">quantity</th>\n",
       "      <th>...</th>\n",
       "      <th colspan=\"4\" halign=\"left\">tot_disc</th>\n",
       "      <th colspan=\"6\" halign=\"left\">coupon_to_item</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>nunique</th>\n",
       "      <th>count</th>\n",
       "      <th>nunique</th>\n",
       "      <th>count</th>\n",
       "      <th>nunique</th>\n",
       "      <th>mean</th>\n",
       "      <th>max</th>\n",
       "      <th>min</th>\n",
       "      <th>sum</th>\n",
       "      <th>...</th>\n",
       "      <th>max</th>\n",
       "      <th>min</th>\n",
       "      <th>sum</th>\n",
       "      <th>std</th>\n",
       "      <th>mean</th>\n",
       "      <th>max</th>\n",
       "      <th>min</th>\n",
       "      <th>sum</th>\n",
       "      <th>std</th>\n",
       "      <th>quantile</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>customer_id</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2206</td>\n",
       "      <td>28</td>\n",
       "      <td>2206</td>\n",
       "      <td>12</td>\n",
       "      <td>2206</td>\n",
       "      <td>38</td>\n",
       "      <td>1.133726</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>2501</td>\n",
       "      <td>...</td>\n",
       "      <td>30.803886</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29544.403002</td>\n",
       "      <td>6.798600</td>\n",
       "      <td>13.276065</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "      <td>29287</td>\n",
       "      <td>5.671542</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>823</td>\n",
       "      <td>19</td>\n",
       "      <td>823</td>\n",
       "      <td>11</td>\n",
       "      <td>823</td>\n",
       "      <td>27</td>\n",
       "      <td>1.117861</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>920</td>\n",
       "      <td>...</td>\n",
       "      <td>16.836933</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6242.841276</td>\n",
       "      <td>5.037921</td>\n",
       "      <td>7.814095</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>6431</td>\n",
       "      <td>6.097201</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>1351</td>\n",
       "      <td>23</td>\n",
       "      <td>1351</td>\n",
       "      <td>12</td>\n",
       "      <td>1351</td>\n",
       "      <td>30</td>\n",
       "      <td>6.861584</td>\n",
       "      <td>6949</td>\n",
       "      <td>1</td>\n",
       "      <td>9270</td>\n",
       "      <td>...</td>\n",
       "      <td>46.132935</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22281.617402</td>\n",
       "      <td>12.410100</td>\n",
       "      <td>13.960030</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>18860</td>\n",
       "      <td>9.525554</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>451</td>\n",
       "      <td>23</td>\n",
       "      <td>451</td>\n",
       "      <td>11</td>\n",
       "      <td>451</td>\n",
       "      <td>21</td>\n",
       "      <td>1.283814</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>579</td>\n",
       "      <td>...</td>\n",
       "      <td>13.875339</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1927.087224</td>\n",
       "      <td>4.159218</td>\n",
       "      <td>6.374723</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>2875</td>\n",
       "      <td>4.604267</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>1594</td>\n",
       "      <td>31</td>\n",
       "      <td>1594</td>\n",
       "      <td>12</td>\n",
       "      <td>1594</td>\n",
       "      <td>51</td>\n",
       "      <td>59.345671</td>\n",
       "      <td>14638</td>\n",
       "      <td>1</td>\n",
       "      <td>94597</td>\n",
       "      <td>...</td>\n",
       "      <td>14.588230</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3806.955362</td>\n",
       "      <td>3.228779</td>\n",
       "      <td>2.474279</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>3944</td>\n",
       "      <td>2.351349</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 85 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "            date_d         date_m         date_w           quantity         \\\n",
       "             count nunique  count nunique  count nunique       mean    max   \n",
       "customer_id                                                                  \n",
       "1             2206      28   2206      12   2206      38   1.133726      5   \n",
       "2              823      19    823      11    823      27   1.117861      5   \n",
       "3             1351      23   1351      12   1351      30   6.861584   6949   \n",
       "4              451      23    451      11    451      21   1.283814      5   \n",
       "5             1594      31   1594      12   1594      51  59.345671  14638   \n",
       "\n",
       "                        ...   tot_disc                                \\\n",
       "            min    sum  ...        max  min           sum        std   \n",
       "customer_id             ...                                            \n",
       "1             1   2501  ...  30.803886  0.0  29544.403002   6.798600   \n",
       "2             1    920  ...  16.836933  0.0   6242.841276   5.037921   \n",
       "3             1   9270  ...  46.132935  0.0  22281.617402  12.410100   \n",
       "4             1    579  ...  13.875339  0.0   1927.087224   4.159218   \n",
       "5             1  94597  ...  14.588230  0.0   3806.955362   3.228779   \n",
       "\n",
       "            coupon_to_item                                    \n",
       "                      mean max min    sum       std quantile  \n",
       "customer_id                                                   \n",
       "1                13.276065  23   0  29287  5.671542     14.0  \n",
       "2                 7.814095  20   0   6431  6.097201      6.0  \n",
       "3                13.960030  35   0  18860  9.525554     12.0  \n",
       "4                 6.374723  13   0   2875  4.604267      4.0  \n",
       "5                 2.474279  11   0   3944  2.351349      2.0  \n",
       "\n",
       "[5 rows x 85 columns]"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "c=['count','nunique']\n",
    "n=['mean','max','min','sum','std']\n",
    "nn=['mean','max','min','sum','std','quantile']\n",
    "agg_c={'date_d':c,'date_m':c,'date_w':c,'quantity':n,'selling_price':n,'other_discount':n,'coupon_discount':n,'item_id':c,'brand':c,\n",
    "       'category':c,'coupon_id':c,'discount_bin':nn,'marked_price':n,'disc_percent':n,'price_per_quan':n,'brand_type':c,'marked_by_sale':n,\n",
    "       'coupon_aquired':nn, 'item_bought':nn, 'tot_disc':n, 'coupon_to_item':nn}\n",
    "trans=tran.groupby(['customer_id']).agg(agg_c)\n",
    "trans.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>customer_id</th>\n",
       "      <th>F_date_d_count</th>\n",
       "      <th>F_date_d_nunique</th>\n",
       "      <th>F_date_m_count</th>\n",
       "      <th>F_date_m_nunique</th>\n",
       "      <th>F_date_w_count</th>\n",
       "      <th>F_date_w_nunique</th>\n",
       "      <th>F_quantity_mean</th>\n",
       "      <th>F_quantity_max</th>\n",
       "      <th>F_quantity_min</th>\n",
       "      <th>...</th>\n",
       "      <th>F_tot_disc_max</th>\n",
       "      <th>F_tot_disc_min</th>\n",
       "      <th>F_tot_disc_sum</th>\n",
       "      <th>F_tot_disc_std</th>\n",
       "      <th>F_coupon_to_item_mean</th>\n",
       "      <th>F_coupon_to_item_max</th>\n",
       "      <th>F_coupon_to_item_min</th>\n",
       "      <th>F_coupon_to_item_sum</th>\n",
       "      <th>F_coupon_to_item_std</th>\n",
       "      <th>F_coupon_to_item_quantile</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2206</td>\n",
       "      <td>28</td>\n",
       "      <td>2206</td>\n",
       "      <td>12</td>\n",
       "      <td>2206</td>\n",
       "      <td>38</td>\n",
       "      <td>1.133726</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>30.803886</td>\n",
       "      <td>0.0</td>\n",
       "      <td>29544.403002</td>\n",
       "      <td>6.798600</td>\n",
       "      <td>13.276065</td>\n",
       "      <td>23</td>\n",
       "      <td>0</td>\n",
       "      <td>29287</td>\n",
       "      <td>5.671542</td>\n",
       "      <td>14.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>823</td>\n",
       "      <td>19</td>\n",
       "      <td>823</td>\n",
       "      <td>11</td>\n",
       "      <td>823</td>\n",
       "      <td>27</td>\n",
       "      <td>1.117861</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>16.836933</td>\n",
       "      <td>0.0</td>\n",
       "      <td>6242.841276</td>\n",
       "      <td>5.037921</td>\n",
       "      <td>7.814095</td>\n",
       "      <td>20</td>\n",
       "      <td>0</td>\n",
       "      <td>6431</td>\n",
       "      <td>6.097201</td>\n",
       "      <td>6.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>1351</td>\n",
       "      <td>23</td>\n",
       "      <td>1351</td>\n",
       "      <td>12</td>\n",
       "      <td>1351</td>\n",
       "      <td>30</td>\n",
       "      <td>6.861584</td>\n",
       "      <td>6949</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>46.132935</td>\n",
       "      <td>0.0</td>\n",
       "      <td>22281.617402</td>\n",
       "      <td>12.410100</td>\n",
       "      <td>13.960030</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>18860</td>\n",
       "      <td>9.525554</td>\n",
       "      <td>12.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>451</td>\n",
       "      <td>23</td>\n",
       "      <td>451</td>\n",
       "      <td>11</td>\n",
       "      <td>451</td>\n",
       "      <td>21</td>\n",
       "      <td>1.283814</td>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>13.875339</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1927.087224</td>\n",
       "      <td>4.159218</td>\n",
       "      <td>6.374723</td>\n",
       "      <td>13</td>\n",
       "      <td>0</td>\n",
       "      <td>2875</td>\n",
       "      <td>4.604267</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>1594</td>\n",
       "      <td>31</td>\n",
       "      <td>1594</td>\n",
       "      <td>12</td>\n",
       "      <td>1594</td>\n",
       "      <td>51</td>\n",
       "      <td>59.345671</td>\n",
       "      <td>14638</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>14.588230</td>\n",
       "      <td>0.0</td>\n",
       "      <td>3806.955362</td>\n",
       "      <td>3.228779</td>\n",
       "      <td>2.474279</td>\n",
       "      <td>11</td>\n",
       "      <td>0</td>\n",
       "      <td>3944</td>\n",
       "      <td>2.351349</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 86 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   customer_id  F_date_d_count  F_date_d_nunique  F_date_m_count  \\\n",
       "0            1            2206                28            2206   \n",
       "1            2             823                19             823   \n",
       "2            3            1351                23            1351   \n",
       "3            4             451                23             451   \n",
       "4            5            1594                31            1594   \n",
       "\n",
       "   F_date_m_nunique  F_date_w_count  F_date_w_nunique  F_quantity_mean  \\\n",
       "0                12            2206                38         1.133726   \n",
       "1                11             823                27         1.117861   \n",
       "2                12            1351                30         6.861584   \n",
       "3                11             451                21         1.283814   \n",
       "4                12            1594                51        59.345671   \n",
       "\n",
       "   F_quantity_max  F_quantity_min  ...  F_tot_disc_max  F_tot_disc_min  \\\n",
       "0               5               1  ...       30.803886             0.0   \n",
       "1               5               1  ...       16.836933             0.0   \n",
       "2            6949               1  ...       46.132935             0.0   \n",
       "3               5               1  ...       13.875339             0.0   \n",
       "4           14638               1  ...       14.588230             0.0   \n",
       "\n",
       "   F_tot_disc_sum  F_tot_disc_std  F_coupon_to_item_mean  \\\n",
       "0    29544.403002        6.798600              13.276065   \n",
       "1     6242.841276        5.037921               7.814095   \n",
       "2    22281.617402       12.410100              13.960030   \n",
       "3     1927.087224        4.159218               6.374723   \n",
       "4     3806.955362        3.228779               2.474279   \n",
       "\n",
       "   F_coupon_to_item_max  F_coupon_to_item_min  F_coupon_to_item_sum  \\\n",
       "0                    23                     0                 29287   \n",
       "1                    20                     0                  6431   \n",
       "2                    35                     0                 18860   \n",
       "3                    13                     0                  2875   \n",
       "4                    11                     0                  3944   \n",
       "\n",
       "   F_coupon_to_item_std  F_coupon_to_item_quantile  \n",
       "0              5.671542                       14.0  \n",
       "1              6.097201                        6.0  \n",
       "2              9.525554                       12.0  \n",
       "3              4.604267                        4.0  \n",
       "4              2.351349                        2.0  \n",
       "\n",
       "[5 rows x 86 columns]"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trans.columns=['F_' + '_'.join(col).strip() for col in trans.columns.values]\n",
    "trans.reset_index(inplace=True)\n",
    "trans.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1582, 86)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trans.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "df=df.merge(trans,on=['customer_id'],how='left')\n",
    "\n",
    "\n",
    "# -------to uncomment\n",
    "\n",
    "# df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['campaign_type']=df['campaign_type'].astype('category')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "# df['campaign_id']=df['campaign_id'].astype('category')\n",
    "# df['coupon_id']=df['coupon_id'].astype('category')\n",
    "# df['customer_id']=df['customer_id'].astype('category')\n",
    "# df['campaign_type']=df['campaign_type'].astype('category')\n",
    "\n",
    "# df['within_date_discount'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 128595 entries, 0 to 128594\n",
      "Data columns (total 97 columns):\n",
      "campaign_id                  128595 non-null int64\n",
      "coupon_id                    128595 non-null int64\n",
      "customer_id                  128595 non-null int64\n",
      "id                           128595 non-null int64\n",
      "redemption_status            78369 non-null float64\n",
      "campaign_type                128595 non-null category\n",
      "start_date                   128595 non-null datetime64[ns]\n",
      "end_date                     128595 non-null datetime64[ns]\n",
      "diff_d                       128595 non-null float64\n",
      "diff_m                       128595 non-null float64\n",
      "diff_w                       128595 non-null float64\n",
      "within_date                  128595 non-null int64\n",
      "F_date_d_count               128595 non-null int64\n",
      "F_date_d_nunique             128595 non-null int64\n",
      "F_date_m_count               128595 non-null int64\n",
      "F_date_m_nunique             128595 non-null int64\n",
      "F_date_w_count               128595 non-null int64\n",
      "F_date_w_nunique             128595 non-null int64\n",
      "F_quantity_mean              128595 non-null float64\n",
      "F_quantity_max               128595 non-null int64\n",
      "F_quantity_min               128595 non-null int64\n",
      "F_quantity_sum               128595 non-null int64\n",
      "F_quantity_std               128595 non-null float64\n",
      "F_selling_price_mean         128595 non-null float64\n",
      "F_selling_price_max          128595 non-null float64\n",
      "F_selling_price_min          128595 non-null float64\n",
      "F_selling_price_sum          128595 non-null float64\n",
      "F_selling_price_std          128595 non-null float64\n",
      "F_other_discount_mean        128595 non-null float64\n",
      "F_other_discount_max         128595 non-null float64\n",
      "F_other_discount_min         128595 non-null float64\n",
      "F_other_discount_sum         128595 non-null float64\n",
      "F_other_discount_std         128595 non-null float64\n",
      "F_coupon_discount_mean       128595 non-null float64\n",
      "F_coupon_discount_max        128595 non-null float64\n",
      "F_coupon_discount_min        128595 non-null float64\n",
      "F_coupon_discount_sum        128595 non-null float64\n",
      "F_coupon_discount_std        128595 non-null float64\n",
      "F_item_id_count              128595 non-null int64\n",
      "F_item_id_nunique            128595 non-null int64\n",
      "F_brand_count                128595 non-null int64\n",
      "F_brand_nunique              128595 non-null int64\n",
      "F_category_count             128595 non-null int64\n",
      "F_category_nunique           128595 non-null int64\n",
      "F_coupon_id_count            128595 non-null int64\n",
      "F_coupon_id_nunique          128595 non-null int64\n",
      "F_discount_bin_mean          128595 non-null float64\n",
      "F_discount_bin_max           128595 non-null int64\n",
      "F_discount_bin_min           128595 non-null int64\n",
      "F_discount_bin_sum           128595 non-null int64\n",
      "F_discount_bin_std           128595 non-null float64\n",
      "F_discount_bin_quantile      128595 non-null float64\n",
      "F_marked_price_mean          128595 non-null float64\n",
      "F_marked_price_max           128595 non-null float64\n",
      "F_marked_price_min           128595 non-null float64\n",
      "F_marked_price_sum           128595 non-null float64\n",
      "F_marked_price_std           128595 non-null float64\n",
      "F_disc_percent_mean          128595 non-null float64\n",
      "F_disc_percent_max           128595 non-null float64\n",
      "F_disc_percent_min           128595 non-null float64\n",
      "F_disc_percent_sum           128595 non-null float64\n",
      "F_disc_percent_std           128595 non-null float64\n",
      "F_price_per_quan_mean        128595 non-null float64\n",
      "F_price_per_quan_max         128595 non-null float64\n",
      "F_price_per_quan_min         128595 non-null float64\n",
      "F_price_per_quan_sum         128595 non-null float64\n",
      "F_price_per_quan_std         128595 non-null float64\n",
      "F_brand_type_count           128595 non-null int64\n",
      "F_brand_type_nunique         128595 non-null int64\n",
      "F_marked_by_sale_mean        128595 non-null float64\n",
      "F_marked_by_sale_max         128595 non-null float64\n",
      "F_marked_by_sale_min         128595 non-null float64\n",
      "F_marked_by_sale_sum         128595 non-null float64\n",
      "F_marked_by_sale_std         128595 non-null float64\n",
      "F_coupon_aquired_mean        128595 non-null float64\n",
      "F_coupon_aquired_max         128595 non-null int64\n",
      "F_coupon_aquired_min         128595 non-null int64\n",
      "F_coupon_aquired_sum         128595 non-null int64\n",
      "F_coupon_aquired_std         128595 non-null float64\n",
      "F_coupon_aquired_quantile    128595 non-null float64\n",
      "F_item_bought_mean           128595 non-null float64\n",
      "F_item_bought_max            128595 non-null int64\n",
      "F_item_bought_min            128595 non-null int64\n",
      "F_item_bought_sum            128595 non-null int64\n",
      "F_item_bought_std            128595 non-null float64\n",
      "F_item_bought_quantile       128595 non-null float64\n",
      "F_tot_disc_mean              128595 non-null float64\n",
      "F_tot_disc_max               128595 non-null float64\n",
      "F_tot_disc_min               128595 non-null float64\n",
      "F_tot_disc_sum               128595 non-null float64\n",
      "F_tot_disc_std               128595 non-null float64\n",
      "F_coupon_to_item_mean        128595 non-null float64\n",
      "F_coupon_to_item_max         128595 non-null int64\n",
      "F_coupon_to_item_min         128595 non-null int64\n",
      "F_coupon_to_item_sum         128595 non-null int64\n",
      "F_coupon_to_item_std         128595 non-null float64\n",
      "F_coupon_to_item_quantile    128595 non-null float64\n",
      "dtypes: category(1), datetime64[ns](2), float64(58), int64(36)\n",
      "memory usage: 95.3 MB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(78369, 97) (50226, 97)\n"
     ]
    }
   ],
   "source": [
    "df_train=df[df['redemption_status'].isnull()==False].copy()\n",
    "df_test=df[df['redemption_status'].isnull()==True].copy()\n",
    "\n",
    "print(df_train.shape,df_test.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id</th>\n",
       "      <th>coupon_id_x</th>\n",
       "      <th>customer_id_x</th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "      <th>campaign_type</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>diff_d_x</th>\n",
       "      <th>diff_m_x</th>\n",
       "      <th>...</th>\n",
       "      <th>F_tot_disc_max_y</th>\n",
       "      <th>F_tot_disc_min_y</th>\n",
       "      <th>F_tot_disc_sum_y</th>\n",
       "      <th>F_tot_disc_std_y</th>\n",
       "      <th>F_coupon_to_item_mean_y</th>\n",
       "      <th>F_coupon_to_item_max_y</th>\n",
       "      <th>F_coupon_to_item_min_y</th>\n",
       "      <th>F_coupon_to_item_sum_y</th>\n",
       "      <th>F_coupon_to_item_std_y</th>\n",
       "      <th>F_coupon_to_item_quantile_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>13</td>\n",
       "      <td>27</td>\n",
       "      <td>1053</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>33.552230</td>\n",
       "      <td>0.003844</td>\n",
       "      <td>22777.589335</td>\n",
       "      <td>8.323239</td>\n",
       "      <td>7.638486</td>\n",
       "      <td>21.502875</td>\n",
       "      <td>0.021233</td>\n",
       "      <td>17797.572591</td>\n",
       "      <td>5.573441</td>\n",
       "      <td>6.509931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>13</td>\n",
       "      <td>116</td>\n",
       "      <td>48</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>33.552230</td>\n",
       "      <td>0.003844</td>\n",
       "      <td>22777.589335</td>\n",
       "      <td>8.323239</td>\n",
       "      <td>7.638486</td>\n",
       "      <td>21.502875</td>\n",
       "      <td>0.021233</td>\n",
       "      <td>17797.572591</td>\n",
       "      <td>5.573441</td>\n",
       "      <td>6.509931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>635</td>\n",
       "      <td>205</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-03-11</td>\n",
       "      <td>2013-04-12</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1.051356</td>\n",
       "      <td>...</td>\n",
       "      <td>33.359071</td>\n",
       "      <td>0.009611</td>\n",
       "      <td>23196.832997</td>\n",
       "      <td>8.582874</td>\n",
       "      <td>7.262253</td>\n",
       "      <td>19.761364</td>\n",
       "      <td>0.028409</td>\n",
       "      <td>16053.545455</td>\n",
       "      <td>5.015611</td>\n",
       "      <td>6.281250</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>13</td>\n",
       "      <td>644</td>\n",
       "      <td>1050</td>\n",
       "      <td>7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>33.552230</td>\n",
       "      <td>0.003844</td>\n",
       "      <td>22777.589335</td>\n",
       "      <td>8.323239</td>\n",
       "      <td>7.638486</td>\n",
       "      <td>21.502875</td>\n",
       "      <td>0.021233</td>\n",
       "      <td>17797.572591</td>\n",
       "      <td>5.573441</td>\n",
       "      <td>6.509931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>8</td>\n",
       "      <td>1017</td>\n",
       "      <td>1489</td>\n",
       "      <td>9</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-02-16</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1.577034</td>\n",
       "      <td>...</td>\n",
       "      <td>32.011696</td>\n",
       "      <td>0.004153</td>\n",
       "      <td>21276.398491</td>\n",
       "      <td>8.122024</td>\n",
       "      <td>7.581794</td>\n",
       "      <td>20.875460</td>\n",
       "      <td>0.021459</td>\n",
       "      <td>16666.729905</td>\n",
       "      <td>5.472313</td>\n",
       "      <td>6.557127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>78364</td>\n",
       "      <td>8</td>\n",
       "      <td>71</td>\n",
       "      <td>1523</td>\n",
       "      <td>128587</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-02-16</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1.577034</td>\n",
       "      <td>...</td>\n",
       "      <td>32.011696</td>\n",
       "      <td>0.004153</td>\n",
       "      <td>21276.398491</td>\n",
       "      <td>8.122024</td>\n",
       "      <td>7.581794</td>\n",
       "      <td>20.875460</td>\n",
       "      <td>0.021459</td>\n",
       "      <td>16666.729905</td>\n",
       "      <td>5.472313</td>\n",
       "      <td>6.557127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>78365</td>\n",
       "      <td>30</td>\n",
       "      <td>547</td>\n",
       "      <td>937</td>\n",
       "      <td>128589</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2012-11-19</td>\n",
       "      <td>2013-01-04</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1.511325</td>\n",
       "      <td>...</td>\n",
       "      <td>35.776467</td>\n",
       "      <td>0.003922</td>\n",
       "      <td>24342.696046</td>\n",
       "      <td>8.920721</td>\n",
       "      <td>7.579016</td>\n",
       "      <td>20.785527</td>\n",
       "      <td>0.018710</td>\n",
       "      <td>17979.403433</td>\n",
       "      <td>5.513836</td>\n",
       "      <td>6.445493</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>78366</td>\n",
       "      <td>8</td>\n",
       "      <td>754</td>\n",
       "      <td>1004</td>\n",
       "      <td>128590</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-02-16</td>\n",
       "      <td>2013-04-05</td>\n",
       "      <td>48.0</td>\n",
       "      <td>1.577034</td>\n",
       "      <td>...</td>\n",
       "      <td>32.011696</td>\n",
       "      <td>0.004153</td>\n",
       "      <td>21276.398491</td>\n",
       "      <td>8.122024</td>\n",
       "      <td>7.581794</td>\n",
       "      <td>20.875460</td>\n",
       "      <td>0.021459</td>\n",
       "      <td>16666.729905</td>\n",
       "      <td>5.472313</td>\n",
       "      <td>6.557127</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>78367</td>\n",
       "      <td>13</td>\n",
       "      <td>134</td>\n",
       "      <td>71</td>\n",
       "      <td>128592</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>33.552230</td>\n",
       "      <td>0.003844</td>\n",
       "      <td>22777.589335</td>\n",
       "      <td>8.323239</td>\n",
       "      <td>7.638486</td>\n",
       "      <td>21.502875</td>\n",
       "      <td>0.021233</td>\n",
       "      <td>17797.572591</td>\n",
       "      <td>5.573441</td>\n",
       "      <td>6.509931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>78368</td>\n",
       "      <td>13</td>\n",
       "      <td>681</td>\n",
       "      <td>623</td>\n",
       "      <td>128595</td>\n",
       "      <td>0.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>33.552230</td>\n",
       "      <td>0.003844</td>\n",
       "      <td>22777.589335</td>\n",
       "      <td>8.323239</td>\n",
       "      <td>7.638486</td>\n",
       "      <td>21.502875</td>\n",
       "      <td>0.021233</td>\n",
       "      <td>17797.572591</td>\n",
       "      <td>5.573441</td>\n",
       "      <td>6.509931</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>78369 rows × 188 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       campaign_id  coupon_id_x  customer_id_x      id  redemption_status  \\\n",
       "0               13           27           1053       1                0.0   \n",
       "1               13          116             48       2                0.0   \n",
       "2                9          635            205       6                0.0   \n",
       "3               13          644           1050       7                0.0   \n",
       "4                8         1017           1489       9                0.0   \n",
       "...            ...          ...            ...     ...                ...   \n",
       "78364            8           71           1523  128587                0.0   \n",
       "78365           30          547            937  128589                0.0   \n",
       "78366            8          754           1004  128590                0.0   \n",
       "78367           13          134             71  128592                0.0   \n",
       "78368           13          681            623  128595                0.0   \n",
       "\n",
       "      campaign_type start_date   end_date  diff_d_x  diff_m_x  ...  \\\n",
       "0                 X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "1                 X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "2                 Y 2013-03-11 2013-04-12      32.0  1.051356  ...   \n",
       "3                 X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "4                 X 2013-02-16 2013-04-05      48.0  1.577034  ...   \n",
       "...             ...        ...        ...       ...       ...  ...   \n",
       "78364             X 2013-02-16 2013-04-05      48.0  1.577034  ...   \n",
       "78365             X 2012-11-19 2013-01-04      46.0  1.511325  ...   \n",
       "78366             X 2013-02-16 2013-04-05      48.0  1.577034  ...   \n",
       "78367             X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "78368             X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "\n",
       "       F_tot_disc_max_y  F_tot_disc_min_y  F_tot_disc_sum_y  F_tot_disc_std_y  \\\n",
       "0             33.552230          0.003844      22777.589335          8.323239   \n",
       "1             33.552230          0.003844      22777.589335          8.323239   \n",
       "2             33.359071          0.009611      23196.832997          8.582874   \n",
       "3             33.552230          0.003844      22777.589335          8.323239   \n",
       "4             32.011696          0.004153      21276.398491          8.122024   \n",
       "...                 ...               ...               ...               ...   \n",
       "78364         32.011696          0.004153      21276.398491          8.122024   \n",
       "78365         35.776467          0.003922      24342.696046          8.920721   \n",
       "78366         32.011696          0.004153      21276.398491          8.122024   \n",
       "78367         33.552230          0.003844      22777.589335          8.323239   \n",
       "78368         33.552230          0.003844      22777.589335          8.323239   \n",
       "\n",
       "       F_coupon_to_item_mean_y  F_coupon_to_item_max_y  \\\n",
       "0                     7.638486               21.502875   \n",
       "1                     7.638486               21.502875   \n",
       "2                     7.262253               19.761364   \n",
       "3                     7.638486               21.502875   \n",
       "4                     7.581794               20.875460   \n",
       "...                        ...                     ...   \n",
       "78364                 7.581794               20.875460   \n",
       "78365                 7.579016               20.785527   \n",
       "78366                 7.581794               20.875460   \n",
       "78367                 7.638486               21.502875   \n",
       "78368                 7.638486               21.502875   \n",
       "\n",
       "       F_coupon_to_item_min_y  F_coupon_to_item_sum_y  F_coupon_to_item_std_y  \\\n",
       "0                    0.021233            17797.572591                5.573441   \n",
       "1                    0.021233            17797.572591                5.573441   \n",
       "2                    0.028409            16053.545455                5.015611   \n",
       "3                    0.021233            17797.572591                5.573441   \n",
       "4                    0.021459            16666.729905                5.472313   \n",
       "...                       ...                     ...                     ...   \n",
       "78364                0.021459            16666.729905                5.472313   \n",
       "78365                0.018710            17979.403433                5.513836   \n",
       "78366                0.021459            16666.729905                5.472313   \n",
       "78367                0.021233            17797.572591                5.573441   \n",
       "78368                0.021233            17797.572591                5.573441   \n",
       "\n",
       "       F_coupon_to_item_quantile_y  \n",
       "0                         6.509931  \n",
       "1                         6.509931  \n",
       "2                         6.281250  \n",
       "3                         6.509931  \n",
       "4                         6.557127  \n",
       "...                            ...  \n",
       "78364                     6.557127  \n",
       "78365                     6.445493  \n",
       "78366                     6.557127  \n",
       "78367                     6.509931  \n",
       "78368                     6.509931  \n",
       "\n",
       "[78369 rows x 188 columns]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.merge(df_train.drop(['id','redemption_status'],axis=1).groupby('campaign_id').mean().reset_index(),on='campaign_id',how='left')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train=df_train.merge(df_train.drop(['id','redemption_status'],axis=1).groupby('coupon_id').mean().reset_index(),on='coupon_id',how='left')\n",
    "df_test=df_test.merge(df_test.drop(['id','redemption_status'],axis=1).groupby('coupon_id').mean().reset_index(),on='coupon_id',how='left')\n",
    "\n",
    "# df_train=df_train.merge(df_train.drop(['id','redemption_status'],axis=1).groupby('coupon_id_x').mean().reset_index(),on='coupon_id_x',how='left')\n",
    "# df_test=df_test.merge(df_test.drop(['id','redemption_status'],axis=1).groupby('coupon_id_x').mean().reset_index(),on='coupon_id_x',how='left')\n",
    "\n",
    "\n",
    "\n",
    "# df_train=new_df(df_train)\n",
    "# print(df_train.shape)\n",
    "\n",
    "# df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>campaign_id_x</th>\n",
       "      <th>coupon_id</th>\n",
       "      <th>customer_id_x</th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "      <th>campaign_type</th>\n",
       "      <th>start_date</th>\n",
       "      <th>end_date</th>\n",
       "      <th>diff_d_x</th>\n",
       "      <th>diff_m_x</th>\n",
       "      <th>...</th>\n",
       "      <th>F_tot_disc_max_y</th>\n",
       "      <th>F_tot_disc_min_y</th>\n",
       "      <th>F_tot_disc_sum_y</th>\n",
       "      <th>F_tot_disc_std_y</th>\n",
       "      <th>F_coupon_to_item_mean_y</th>\n",
       "      <th>F_coupon_to_item_max_y</th>\n",
       "      <th>F_coupon_to_item_min_y</th>\n",
       "      <th>F_coupon_to_item_sum_y</th>\n",
       "      <th>F_coupon_to_item_std_y</th>\n",
       "      <th>F_coupon_to_item_quantile_y</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>44</td>\n",
       "      <td>13</td>\n",
       "      <td>413</td>\n",
       "      <td>276</td>\n",
       "      <td>72</td>\n",
       "      <td>1.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>33.993859</td>\n",
       "      <td>0.000709</td>\n",
       "      <td>26329.668196</td>\n",
       "      <td>8.588316</td>\n",
       "      <td>8.431821</td>\n",
       "      <td>22.890000</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>21165.840000</td>\n",
       "      <td>5.953643</td>\n",
       "      <td>7.230000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>181</td>\n",
       "      <td>13</td>\n",
       "      <td>960</td>\n",
       "      <td>467</td>\n",
       "      <td>299</td>\n",
       "      <td>1.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>41.552949</td>\n",
       "      <td>0.002928</td>\n",
       "      <td>24843.989722</td>\n",
       "      <td>10.051581</td>\n",
       "      <td>7.761234</td>\n",
       "      <td>22.821705</td>\n",
       "      <td>0.023256</td>\n",
       "      <td>17768.968992</td>\n",
       "      <td>5.963228</td>\n",
       "      <td>6.348837</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>217</td>\n",
       "      <td>26</td>\n",
       "      <td>797</td>\n",
       "      <td>214</td>\n",
       "      <td>359</td>\n",
       "      <td>1.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2012-08-12</td>\n",
       "      <td>2012-09-21</td>\n",
       "      <td>40.0</td>\n",
       "      <td>1.314195</td>\n",
       "      <td>...</td>\n",
       "      <td>43.636938</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>39157.851565</td>\n",
       "      <td>10.153123</td>\n",
       "      <td>8.264332</td>\n",
       "      <td>22.096774</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>25078.548387</td>\n",
       "      <td>5.607778</td>\n",
       "      <td>7.419355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>237</td>\n",
       "      <td>13</td>\n",
       "      <td>754</td>\n",
       "      <td>1103</td>\n",
       "      <td>387</td>\n",
       "      <td>1.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>36.048456</td>\n",
       "      <td>0.009408</td>\n",
       "      <td>25913.440478</td>\n",
       "      <td>8.897671</td>\n",
       "      <td>7.772261</td>\n",
       "      <td>22.079470</td>\n",
       "      <td>0.019868</td>\n",
       "      <td>19612.453642</td>\n",
       "      <td>5.682630</td>\n",
       "      <td>6.632450</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>607</td>\n",
       "      <td>13</td>\n",
       "      <td>21</td>\n",
       "      <td>1457</td>\n",
       "      <td>998</td>\n",
       "      <td>1.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>37.188686</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>24724.593362</td>\n",
       "      <td>9.156290</td>\n",
       "      <td>7.731749</td>\n",
       "      <td>21.625000</td>\n",
       "      <td>0.022059</td>\n",
       "      <td>18004.727941</td>\n",
       "      <td>5.569303</td>\n",
       "      <td>6.632353</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>77617</td>\n",
       "      <td>13</td>\n",
       "      <td>982</td>\n",
       "      <td>1408</td>\n",
       "      <td>127381</td>\n",
       "      <td>1.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>31.569600</td>\n",
       "      <td>0.000507</td>\n",
       "      <td>22105.532576</td>\n",
       "      <td>7.829935</td>\n",
       "      <td>7.356067</td>\n",
       "      <td>19.925110</td>\n",
       "      <td>0.039648</td>\n",
       "      <td>16883.255507</td>\n",
       "      <td>5.067273</td>\n",
       "      <td>6.414097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>77729</td>\n",
       "      <td>13</td>\n",
       "      <td>885</td>\n",
       "      <td>781</td>\n",
       "      <td>127565</td>\n",
       "      <td>1.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>30.375460</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20081.389206</td>\n",
       "      <td>7.711301</td>\n",
       "      <td>7.621228</td>\n",
       "      <td>20.983402</td>\n",
       "      <td>0.020747</td>\n",
       "      <td>17294.406639</td>\n",
       "      <td>5.379502</td>\n",
       "      <td>6.821577</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>77830</td>\n",
       "      <td>13</td>\n",
       "      <td>726</td>\n",
       "      <td>407</td>\n",
       "      <td>127701</td>\n",
       "      <td>1.0</td>\n",
       "      <td>X</td>\n",
       "      <td>2013-05-19</td>\n",
       "      <td>2013-07-05</td>\n",
       "      <td>47.0</td>\n",
       "      <td>1.544180</td>\n",
       "      <td>...</td>\n",
       "      <td>32.871010</td>\n",
       "      <td>0.007154</td>\n",
       "      <td>26021.574803</td>\n",
       "      <td>8.497170</td>\n",
       "      <td>7.739586</td>\n",
       "      <td>21.406667</td>\n",
       "      <td>0.020000</td>\n",
       "      <td>18749.953333</td>\n",
       "      <td>5.629960</td>\n",
       "      <td>6.590000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>78060</td>\n",
       "      <td>9</td>\n",
       "      <td>705</td>\n",
       "      <td>999</td>\n",
       "      <td>128076</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-03-11</td>\n",
       "      <td>2013-04-12</td>\n",
       "      <td>32.0</td>\n",
       "      <td>1.051356</td>\n",
       "      <td>...</td>\n",
       "      <td>34.178960</td>\n",
       "      <td>0.007673</td>\n",
       "      <td>25406.924841</td>\n",
       "      <td>8.652287</td>\n",
       "      <td>8.002850</td>\n",
       "      <td>21.817544</td>\n",
       "      <td>0.026316</td>\n",
       "      <td>19098.743860</td>\n",
       "      <td>5.628999</td>\n",
       "      <td>6.964912</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>78102</td>\n",
       "      <td>7</td>\n",
       "      <td>392</td>\n",
       "      <td>63</td>\n",
       "      <td>128154</td>\n",
       "      <td>1.0</td>\n",
       "      <td>Y</td>\n",
       "      <td>2013-02-02</td>\n",
       "      <td>2013-03-08</td>\n",
       "      <td>34.0</td>\n",
       "      <td>1.117066</td>\n",
       "      <td>...</td>\n",
       "      <td>32.697383</td>\n",
       "      <td>0.005104</td>\n",
       "      <td>25566.425479</td>\n",
       "      <td>7.962299</td>\n",
       "      <td>7.904796</td>\n",
       "      <td>22.005051</td>\n",
       "      <td>0.020202</td>\n",
       "      <td>22025.297980</td>\n",
       "      <td>5.513581</td>\n",
       "      <td>6.909091</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>729 rows × 188 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       campaign_id_x  coupon_id  customer_id_x      id  redemption_status  \\\n",
       "44                13        413            276      72                1.0   \n",
       "181               13        960            467     299                1.0   \n",
       "217               26        797            214     359                1.0   \n",
       "237               13        754           1103     387                1.0   \n",
       "607               13         21           1457     998                1.0   \n",
       "...              ...        ...            ...     ...                ...   \n",
       "77617             13        982           1408  127381                1.0   \n",
       "77729             13        885            781  127565                1.0   \n",
       "77830             13        726            407  127701                1.0   \n",
       "78060              9        705            999  128076                1.0   \n",
       "78102              7        392             63  128154                1.0   \n",
       "\n",
       "      campaign_type start_date   end_date  diff_d_x  diff_m_x  ...  \\\n",
       "44                X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "181               X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "217               X 2012-08-12 2012-09-21      40.0  1.314195  ...   \n",
       "237               X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "607               X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "...             ...        ...        ...       ...       ...  ...   \n",
       "77617             X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "77729             X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "77830             X 2013-05-19 2013-07-05      47.0  1.544180  ...   \n",
       "78060             Y 2013-03-11 2013-04-12      32.0  1.051356  ...   \n",
       "78102             Y 2013-02-02 2013-03-08      34.0  1.117066  ...   \n",
       "\n",
       "       F_tot_disc_max_y  F_tot_disc_min_y  F_tot_disc_sum_y  F_tot_disc_std_y  \\\n",
       "44            33.993859          0.000709      26329.668196          8.588316   \n",
       "181           41.552949          0.002928      24843.989722         10.051581   \n",
       "217           43.636938          0.000000      39157.851565         10.153123   \n",
       "237           36.048456          0.009408      25913.440478          8.897671   \n",
       "607           37.188686          0.000000      24724.593362          9.156290   \n",
       "...                 ...               ...               ...               ...   \n",
       "77617         31.569600          0.000507      22105.532576          7.829935   \n",
       "77729         30.375460          0.000000      20081.389206          7.711301   \n",
       "77830         32.871010          0.007154      26021.574803          8.497170   \n",
       "78060         34.178960          0.007673      25406.924841          8.652287   \n",
       "78102         32.697383          0.005104      25566.425479          7.962299   \n",
       "\n",
       "       F_coupon_to_item_mean_y  F_coupon_to_item_max_y  \\\n",
       "44                    8.431821               22.890000   \n",
       "181                   7.761234               22.821705   \n",
       "217                   8.264332               22.096774   \n",
       "237                   7.772261               22.079470   \n",
       "607                   7.731749               21.625000   \n",
       "...                        ...                     ...   \n",
       "77617                 7.356067               19.925110   \n",
       "77729                 7.621228               20.983402   \n",
       "77830                 7.739586               21.406667   \n",
       "78060                 8.002850               21.817544   \n",
       "78102                 7.904796               22.005051   \n",
       "\n",
       "       F_coupon_to_item_min_y  F_coupon_to_item_sum_y  F_coupon_to_item_std_y  \\\n",
       "44                   0.040000            21165.840000                5.953643   \n",
       "181                  0.023256            17768.968992                5.963228   \n",
       "217                  0.000000            25078.548387                5.607778   \n",
       "237                  0.019868            19612.453642                5.682630   \n",
       "607                  0.022059            18004.727941                5.569303   \n",
       "...                       ...                     ...                     ...   \n",
       "77617                0.039648            16883.255507                5.067273   \n",
       "77729                0.020747            17294.406639                5.379502   \n",
       "77830                0.020000            18749.953333                5.629960   \n",
       "78060                0.026316            19098.743860                5.628999   \n",
       "78102                0.020202            22025.297980                5.513581   \n",
       "\n",
       "       F_coupon_to_item_quantile_y  \n",
       "44                        7.230000  \n",
       "181                       6.348837  \n",
       "217                       7.419355  \n",
       "237                       6.632450  \n",
       "607                       6.632353  \n",
       "...                            ...  \n",
       "77617                     6.414097  \n",
       "77729                     6.821577  \n",
       "77830                     6.590000  \n",
       "78060                     6.964912  \n",
       "78102                     6.909091  \n",
       "\n",
       "[729 rows x 188 columns]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train[df_train.redemption_status==1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 1080x1080 with 0 Axes>"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "<Figure size 1080x1080 with 0 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import seaborn as sns\n",
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "plt.figure(figsize=(15,15))\n",
    "\n",
    "# sns.heatmap(df_train.corr())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "from catboost import CatBoostClassifier,Pool, cv\n",
    "from lightgbm import LGBMClassifier\n",
    "from sklearn.model_selection import StratifiedKFold,train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.metrics import accuracy_score,confusion_matrix,roc_auc_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['campaign_id_x', 'coupon_id', 'customer_id_x', 'id',\n",
       "       'redemption_status', 'campaign_type', 'start_date', 'end_date',\n",
       "       'diff_d_x', 'diff_m_x',\n",
       "       ...\n",
       "       'F_tot_disc_max_y', 'F_tot_disc_min_y', 'F_tot_disc_sum_y',\n",
       "       'F_tot_disc_std_y', 'F_coupon_to_item_mean_y', 'F_coupon_to_item_max_y',\n",
       "       'F_coupon_to_item_min_y', 'F_coupon_to_item_sum_y',\n",
       "       'F_coupon_to_item_std_y', 'F_coupon_to_item_quantile_y'],\n",
       "      dtype='object', length=188)"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "# X,y=df_train.drop(['id','redemption_status'],axis=1),df_train['redemption_status']\n",
    "# Xtest=df_test.drop(['id','redemption_status'],axis=1)\n",
    "# col_to_drop=['id','redemption_status','start_date','end_date','F_quantity_min','F_other_discount_max','F_coupon_discount_max','F_discount_bin_min',\n",
    "#              'F_disc_percent_min','F_brand_type_nunique','F_marked_by_sale_min','customer_id','campaign_id','coupon_id']\n",
    "\n",
    "col_to_drop=['id','redemption_status','start_date','end_date']\n",
    "\n",
    "X,y=df_train.drop(col_to_drop,axis=1),df_train['redemption_status']\n",
    "Xtest=df_test.drop(col_to_drop,axis=1)\n",
    "\n",
    "# X,y=df_train.drop(['id','redemption_status','start_date','end_date','customer_id','coupon_id','campaign_id'],axis=1),df_train['redemption_status']\n",
    "# Xtest=df_test.drop(['id','redemption_status','start_date','end_date','customer_id','coupon_id','campaign_id'],axis=1)\n",
    "\n",
    "# X=pd.get_dummies(X,drop_first=True)\n",
    "\n",
    "# from sklearn.ensemble import IsolationForest\n",
    "# clf = IsolationForest(contamination = 'auto', random_state=1994,behaviour=\"new\",bootstrap=True)\n",
    "# clf.fit(X)\n",
    "# df_train['iso_out']=clf.predict(X)\n",
    "# print(df_train['iso_out'].value_counts())\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "# print(df_train[df_train['iso_out']==1].shape)\n",
    "# print(df_train[df_train['iso_out']==-1].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(78369, 184) (50226, 184)\n"
     ]
    }
   ],
   "source": [
    "# X['iso_out'].value_counts()\n",
    "# X=X[X.iso_out==1].copy()\n",
    "print(X.shape,Xtest.shape)\n",
    "X_train,X_val,y_train,y_val = train_test_split(X,y,test_size=0.3,random_state = 1994,stratify=y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['campaign_id_x', 'coupon_id', 'customer_id_x', 'campaign_type',\n",
       "       'diff_d_x', 'diff_m_x', 'diff_w_x', 'within_date_x', 'F_date_d_count_x',\n",
       "       'F_date_d_nunique_x',\n",
       "       ...\n",
       "       'F_tot_disc_max_y', 'F_tot_disc_min_y', 'F_tot_disc_sum_y',\n",
       "       'F_tot_disc_std_y', 'F_coupon_to_item_mean_y', 'F_coupon_to_item_max_y',\n",
       "       'F_coupon_to_item_min_y', 'F_coupon_to_item_sum_y',\n",
       "       'F_coupon_to_item_std_y', 'F_coupon_to_item_quantile_y'],\n",
       "      dtype='object', length=184)"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "col=['campaign_id', 'coupon_id', 'customer_id', 'campaign_type','within_date', 'within_date_discount']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # for j in col:\n",
    "# X_train,X_val,y_train,y_val = train_test_split(X,y,test_size=0.3,random_state = 1994,stratify=y)\n",
    "# print('Dropped->',j)\n",
    "# X_train.drop(col,inplace=True,axis=1)\n",
    "# X_val.drop(col,inplace=True,axis=1)\n",
    "# m=LGBMClassifier(n_estimators=1500,random_state=1994,learning_rate=0.03,reg_alpha=0.2,colsample_bytree=0.5,bagging_fraction=0.9)\n",
    "# # m=RidgeCV(cv=4)\n",
    "# m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_val, y_val.values)],eval_metric='auc', early_stopping_rounds=100,verbose=200)\n",
    "# p=m.predict_proba(X_val)[:,-1]\n",
    "\n",
    "# print(roc_auc_score(y_val,p))\n",
    "# print('---------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### LGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 100 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00633047\ttraining's auc: 0.999969\tvalid_1's binary_logloss: 0.0229654\tvalid_1's auc: 0.987345\n",
      "Early stopping, best iteration is:\n",
      "[230]\ttraining's binary_logloss: 0.00556215\ttraining's auc: 0.999987\tvalid_1's binary_logloss: 0.0229254\tvalid_1's auc: 0.987675\n",
      "0.9876750360913304\n"
     ]
    }
   ],
   "source": [
    "m=LGBMClassifier(n_estimators=1500,random_state=1994,learning_rate=0.03,reg_alpha=0.2,colsample_bytree=0.5,bagging_fraction=0.9)\n",
    "# m=RidgeCV(cv=4)\n",
    "m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_val, y_val.values)],eval_metric='auc', early_stopping_rounds=100,verbose=200)\n",
    "p=m.predict_proba(X_val)[:,-1]\n",
    "\n",
    "print(roc_auc_score(y_val,p))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 56, 207,  65,  31,  41,  23,  11, 347,  24,  28,  12,   0,   6,\n",
       "        38,  63,  36,   0,  53,  44,  37,  64,  37,  59,  64,  65,   0,\n",
       "        71,  68,  84,  49,   0,  80,  95,  73,   2,  41,  24,  31,  14,\n",
       "        14,  11, 110, 133,   0,   0, 111,  91,   0,  42,  71,  56,  41,\n",
       "        47,  58,  75,   0,  28,  45, 102,  72,  59,  33,  70,   5,   0,\n",
       "        38,  44,   0,  18,  45,  42,  41,   2,  28,  47,  41,  31,  38,\n",
       "         0,  16,  43,  40,  38,  55,   5,  27,  50,  43,  42,   2,  61,\n",
       "        48,  42,  23,  49,  22,  22,  11, 237,   8,  46,   3,  53,   3,\n",
       "        25,  47,  46,   0,  38,  45,  36,  32,  25,  23,  38,  50,   0,\n",
       "        64,  19,  46,  36,   0,  50,  51,  37,   0,  15,   6,   6,   5,\n",
       "        33,   2,  27,  48,  39,   0,  43,  47,   0,  26,  39,  38,  18,\n",
       "        39,  32,  49,   0,  25,  50,  42,  60,  35,  10,  64,   1,   0,\n",
       "        14,  17,   0,  11,  27,  11,  17,  27,  31,  31,  19,  11,   7,\n",
       "        75,  17,  25,  15,  28,  16,  54,  17,  15,  30,  28,  44,  19,\n",
       "        39,  27])"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m.feature_importances_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[23260,    32],\n",
       "       [  160,    59]])"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "confusion_matrix(y_val,p>0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(347, 'within_date_x'),\n",
       " (237, 'within_date_y'),\n",
       " (207, 'coupon_id'),\n",
       " (133, 'F_discount_bin_mean_x'),\n",
       " (111, 'F_discount_bin_sum_x'),\n",
       " (110, 'F_coupon_id_nunique_x'),\n",
       " (102, 'F_price_per_quan_mean_x'),\n",
       " (95, 'F_coupon_discount_sum_x'),\n",
       " (91, 'F_discount_bin_std_x'),\n",
       " (84, 'F_other_discount_std_x'),\n",
       " (80, 'F_coupon_discount_min_x'),\n",
       " (75, 'F_item_bought_min_y'),\n",
       " (75, 'F_disc_percent_max_x'),\n",
       " (73, 'F_coupon_discount_std_x'),\n",
       " (72, 'F_price_per_quan_max_x'),\n",
       " (71, 'F_other_discount_min_x'),\n",
       " (71, 'F_marked_price_max_x'),\n",
       " (70, 'F_price_per_quan_std_x'),\n",
       " (68, 'F_other_discount_sum_x'),\n",
       " (65, 'customer_id_x'),\n",
       " (65, 'F_other_discount_mean_x'),\n",
       " (64, 'F_selling_price_std_x'),\n",
       " (64, 'F_selling_price_max_x'),\n",
       " (64, 'F_price_per_quan_std_y'),\n",
       " (64, 'F_other_discount_min_y'),\n",
       " (63, 'F_quantity_mean_x'),\n",
       " (61, 'F_coupon_to_item_sum_x'),\n",
       " (60, 'F_price_per_quan_max_y'),\n",
       " (59, 'F_selling_price_sum_x'),\n",
       " (59, 'F_price_per_quan_min_x'),\n",
       " (58, 'F_disc_percent_mean_x'),\n",
       " (56, 'campaign_id_x'),\n",
       " (56, 'F_marked_price_min_x'),\n",
       " (55, 'F_tot_disc_max_x'),\n",
       " (54, 'F_tot_disc_min_y'),\n",
       " (53, 'F_quantity_sum_x'),\n",
       " (53, 'F_date_m_nunique_y'),\n",
       " (51, 'F_coupon_discount_sum_y'),\n",
       " (50, 'F_tot_disc_std_x'),\n",
       " (50, 'F_other_discount_mean_y'),\n",
       " (50, 'F_disc_percent_std_y'),\n",
       " (50, 'F_coupon_discount_min_y'),\n",
       " (49, 'customer_id_y'),\n",
       " (49, 'F_disc_percent_max_y'),\n",
       " (49, 'F_coupon_discount_mean_x'),\n",
       " (48, 'F_discount_bin_mean_y'),\n",
       " (48, 'F_coupon_to_item_std_x'),\n",
       " (47, 'F_quantity_mean_y'),\n",
       " (47, 'F_marked_price_std_x'),\n",
       " (47, 'F_discount_bin_std_y'),\n",
       " (47, 'F_coupon_aquired_std_x'),\n",
       " (46, 'F_quantity_max_y'),\n",
       " (46, 'F_other_discount_std_y'),\n",
       " (46, 'F_date_d_nunique_y'),\n",
       " (45, 'F_quantity_std_y'),\n",
       " (45, 'F_marked_by_sale_std_x'),\n",
       " (45, 'F_disc_percent_std_x'),\n",
       " (44, 'F_quantity_std_x'),\n",
       " (44, 'F_marked_by_sale_max_x'),\n",
       " (44, 'F_coupon_to_item_min_y'),\n",
       " (43, 'F_item_bought_std_x'),\n",
       " (43, 'F_discount_bin_sum_y'),\n",
       " (43, 'F_coupon_to_item_mean_x'),\n",
       " (42, 'F_price_per_quan_mean_y'),\n",
       " (42, 'F_marked_price_mean_x'),\n",
       " (42, 'F_coupon_to_item_quantile_x'),\n",
       " (42, 'F_coupon_to_item_max_x'),\n",
       " (42, 'F_coupon_aquired_mean_x'),\n",
       " (41, 'diff_d_x'),\n",
       " (41, 'F_marked_price_sum_x'),\n",
       " (41, 'F_item_id_nunique_x'),\n",
       " (41, 'F_coupon_aquired_quantile_x'),\n",
       " (41, 'F_coupon_aquired_max_x'),\n",
       " (40, 'F_item_bought_quantile_x'),\n",
       " (39, 'F_marked_price_std_y'),\n",
       " (39, 'F_marked_price_max_y'),\n",
       " (39, 'F_discount_bin_max_y'),\n",
       " (39, 'F_coupon_to_item_std_y'),\n",
       " (38, 'F_tot_disc_mean_x'),\n",
       " (38, 'F_selling_price_std_y'),\n",
       " (38, 'F_quantity_sum_y'),\n",
       " (38, 'F_marked_price_min_y'),\n",
       " (38, 'F_marked_by_sale_mean_x'),\n",
       " (38, 'F_item_bought_max_x'),\n",
       " (38, 'F_date_w_nunique_x'),\n",
       " (37, 'F_selling_price_min_x'),\n",
       " (37, 'F_selling_price_mean_x'),\n",
       " (37, 'F_coupon_discount_std_y'),\n",
       " (36, 'F_selling_price_mean_y'),\n",
       " (36, 'F_quantity_max_x'),\n",
       " (36, 'F_coupon_discount_mean_y'),\n",
       " (35, 'F_price_per_quan_min_y'),\n",
       " (33, 'F_price_per_quan_sum_x'),\n",
       " (33, 'F_category_nunique_y'),\n",
       " (32, 'F_selling_price_max_y'),\n",
       " (32, 'F_disc_percent_mean_y'),\n",
       " (31, 'campaign_type'),\n",
       " (31, 'F_item_bought_mean_x'),\n",
       " (31, 'F_coupon_aquired_sum_y'),\n",
       " (31, 'F_coupon_aquired_std_y'),\n",
       " (31, 'F_brand_nunique_x'),\n",
       " (30, 'F_coupon_to_item_mean_y'),\n",
       " (28, 'F_tot_disc_mean_y'),\n",
       " (28, 'F_disc_percent_sum_x'),\n",
       " (28, 'F_date_d_nunique_x'),\n",
       " (28, 'F_coupon_to_item_max_y'),\n",
       " (28, 'F_coupon_aquired_sum_x'),\n",
       " (27, 'F_tot_disc_sum_x'),\n",
       " (27, 'F_marked_by_sale_std_y'),\n",
       " (27, 'F_coupon_to_item_quantile_y'),\n",
       " (27, 'F_coupon_id_nunique_y'),\n",
       " (27, 'F_coupon_aquired_min_y'),\n",
       " (26, 'F_marked_price_mean_y'),\n",
       " (25, 'F_selling_price_min_y'),\n",
       " (25, 'F_item_bought_std_y'),\n",
       " (25, 'F_disc_percent_sum_y'),\n",
       " (25, 'F_date_w_nunique_y'),\n",
       " (24, 'F_date_d_count_x'),\n",
       " (24, 'F_brand_count_x'),\n",
       " (23, 'diff_m_x'),\n",
       " (23, 'campaign_id_y'),\n",
       " (23, 'F_selling_price_sum_y'),\n",
       " (22, 'diff_m_y'),\n",
       " (22, 'diff_d_y'),\n",
       " (19, 'F_other_discount_sum_y'),\n",
       " (19, 'F_coupon_to_item_sum_y'),\n",
       " (19, 'F_coupon_aquired_quantile_y'),\n",
       " (18, 'F_marked_price_sum_y'),\n",
       " (18, 'F_marked_by_sale_sum_x'),\n",
       " (17, 'F_tot_disc_sum_y'),\n",
       " (17, 'F_marked_by_sale_max_y'),\n",
       " (17, 'F_item_bought_sum_y'),\n",
       " (17, 'F_coupon_aquired_max_y'),\n",
       " (16, 'F_tot_disc_max_y'),\n",
       " (16, 'F_item_bought_sum_x'),\n",
       " (15, 'F_tot_disc_std_y'),\n",
       " (15, 'F_item_id_nunique_y'),\n",
       " (15, 'F_item_bought_quantile_y'),\n",
       " (14, 'F_marked_by_sale_mean_y'),\n",
       " (14, 'F_category_nunique_x'),\n",
       " (14, 'F_category_count_x'),\n",
       " (12, 'F_date_m_count_x'),\n",
       " (11, 'diff_w_y'),\n",
       " (11, 'diff_w_x'),\n",
       " (11, 'F_marked_by_sale_sum_y'),\n",
       " (11, 'F_item_bought_mean_y'),\n",
       " (11, 'F_coupon_id_count_x'),\n",
       " (11, 'F_coupon_aquired_mean_y'),\n",
       " (10, 'F_price_per_quan_sum_y'),\n",
       " (8, 'F_date_d_count_y'),\n",
       " (7, 'F_item_bought_max_y'),\n",
       " (6, 'F_date_w_count_x'),\n",
       " (6, 'F_brand_nunique_y'),\n",
       " (6, 'F_brand_count_y'),\n",
       " (5, 'F_tot_disc_min_x'),\n",
       " (5, 'F_category_count_y'),\n",
       " (5, 'F_brand_type_count_x'),\n",
       " (3, 'F_date_w_count_y'),\n",
       " (3, 'F_date_m_count_y'),\n",
       " (2, 'F_item_id_count_x'),\n",
       " (2, 'F_coupon_to_item_min_x'),\n",
       " (2, 'F_coupon_id_count_y'),\n",
       " (2, 'F_coupon_aquired_min_x'),\n",
       " (1, 'F_brand_type_count_y'),\n",
       " (0, 'F_quantity_min_y'),\n",
       " (0, 'F_quantity_min_x'),\n",
       " (0, 'F_other_discount_max_y'),\n",
       " (0, 'F_other_discount_max_x'),\n",
       " (0, 'F_marked_by_sale_min_y'),\n",
       " (0, 'F_marked_by_sale_min_x'),\n",
       " (0, 'F_item_id_count_y'),\n",
       " (0, 'F_item_bought_min_x'),\n",
       " (0, 'F_discount_bin_quantile_y'),\n",
       " (0, 'F_discount_bin_quantile_x'),\n",
       " (0, 'F_discount_bin_min_y'),\n",
       " (0, 'F_discount_bin_min_x'),\n",
       " (0, 'F_discount_bin_max_x'),\n",
       " (0, 'F_disc_percent_min_y'),\n",
       " (0, 'F_disc_percent_min_x'),\n",
       " (0, 'F_date_m_nunique_x'),\n",
       " (0, 'F_coupon_discount_max_y'),\n",
       " (0, 'F_coupon_discount_max_x'),\n",
       " (0, 'F_brand_type_nunique_y'),\n",
       " (0, 'F_brand_type_nunique_x')]"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sorted(zip(m.feature_importances_,X_train),reverse=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " ### StratifiedKFold LGBM + Feature Selection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00747445\ttraining's auc: 0.99988\tvalid_1's binary_logloss: 0.0229257\tvalid_1's auc: 0.988482\n",
      "Early stopping, best iteration is:\n",
      "[180]\ttraining's binary_logloss: 0.00808889\ttraining's auc: 0.999824\tvalid_1's binary_logloss: 0.0228905\tvalid_1's auc: 0.988424\n",
      "err:  0.988423916495522\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00741135\ttraining's auc: 0.999891\tvalid_1's binary_logloss: 0.0245927\tvalid_1's auc: 0.983965\n",
      "[400]\ttraining's binary_logloss: 0.00384672\ttraining's auc: 0.999997\tvalid_1's binary_logloss: 0.0258373\tvalid_1's auc: 0.984422\n",
      "Early stopping, best iteration is:\n",
      "[222]\ttraining's binary_logloss: 0.00680912\ttraining's auc: 0.999927\tvalid_1's binary_logloss: 0.0244868\tvalid_1's auc: 0.984422\n",
      "err:  0.984422307382863\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00743288\ttraining's auc: 0.999892\tvalid_1's binary_logloss: 0.0243528\tvalid_1's auc: 0.984966\n",
      "Early stopping, best iteration is:\n",
      "[171]\ttraining's binary_logloss: 0.00832444\ttraining's auc: 0.999814\tvalid_1's binary_logloss: 0.0243096\tvalid_1's auc: 0.984892\n",
      "err:  0.9848916319084217\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00749405\ttraining's auc: 0.999889\tvalid_1's binary_logloss: 0.0223318\tvalid_1's auc: 0.988839\n",
      "Early stopping, best iteration is:\n",
      "[184]\ttraining's binary_logloss: 0.00797929\ttraining's auc: 0.999848\tvalid_1's binary_logloss: 0.022242\tvalid_1's auc: 0.988884\n",
      "err:  0.9888844191315026\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00731128\ttraining's auc: 0.999909\tvalid_1's binary_logloss: 0.0242036\tvalid_1's auc: 0.986294\n",
      "[400]\ttraining's binary_logloss: 0.0037746\ttraining's auc: 0.999998\tvalid_1's binary_logloss: 0.0247899\tvalid_1's auc: 0.987102\n",
      "Early stopping, best iteration is:\n",
      "[223]\ttraining's binary_logloss: 0.00669128\ttraining's auc: 0.999944\tvalid_1's binary_logloss: 0.0240853\tvalid_1's auc: 0.986555\n",
      "err:  0.986555440282865\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00730416\ttraining's auc: 0.999893\tvalid_1's binary_logloss: 0.0256072\tvalid_1's auc: 0.98502\n",
      "Early stopping, best iteration is:\n",
      "[179]\ttraining's binary_logloss: 0.00793566\ttraining's auc: 0.999842\tvalid_1's binary_logloss: 0.0255711\tvalid_1's auc: 0.98493\n",
      "err:  0.9849304482225656\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00745102\ttraining's auc: 0.999896\tvalid_1's binary_logloss: 0.0237285\tvalid_1's auc: 0.986723\n",
      "[400]\ttraining's binary_logloss: 0.00377056\ttraining's auc: 0.999998\tvalid_1's binary_logloss: 0.0239833\tvalid_1's auc: 0.988159\n",
      "Early stopping, best iteration is:\n",
      "[263]\ttraining's binary_logloss: 0.00591256\ttraining's auc: 0.999973\tvalid_1's binary_logloss: 0.0235356\tvalid_1's auc: 0.987526\n",
      "err:  0.9875258481364639\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00757733\ttraining's auc: 0.999882\tvalid_1's binary_logloss: 0.0210988\tvalid_1's auc: 0.990802\n",
      "[400]\ttraining's binary_logloss: 0.00391952\ttraining's auc: 0.999997\tvalid_1's binary_logloss: 0.0214466\tvalid_1's auc: 0.991002\n",
      "Early stopping, best iteration is:\n",
      "[240]\ttraining's binary_logloss: 0.00653326\ttraining's auc: 0.999943\tvalid_1's binary_logloss: 0.0209777\tvalid_1's auc: 0.991023\n",
      "err:  0.9910228451652516\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00756639\ttraining's auc: 0.999891\tvalid_1's binary_logloss: 0.0210582\tvalid_1's auc: 0.99028\n",
      "[400]\ttraining's binary_logloss: 0.00388475\ttraining's auc: 0.999998\tvalid_1's binary_logloss: 0.02164\tvalid_1's auc: 0.990391\n",
      "Early stopping, best iteration is:\n",
      "[223]\ttraining's binary_logloss: 0.00692759\ttraining's auc: 0.99993\tvalid_1's binary_logloss: 0.0210511\tvalid_1's auc: 0.990275\n",
      "err:  0.9902747489290227\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00751655\ttraining's auc: 0.999892\tvalid_1's binary_logloss: 0.0225152\tvalid_1's auc: 0.986862\n",
      "[400]\ttraining's binary_logloss: 0.00386858\ttraining's auc: 0.999998\tvalid_1's binary_logloss: 0.0226389\tvalid_1's auc: 0.988136\n",
      "Early stopping, best iteration is:\n",
      "[315]\ttraining's binary_logloss: 0.00503574\ttraining's auc: 0.999991\tvalid_1's binary_logloss: 0.022308\tvalid_1's auc: 0.987977\n",
      "err:  0.9879769162516457\n"
     ]
    }
   ],
   "source": [
    "err=[]\n",
    "y_pred_tot=[]\n",
    "\n",
    "feature_importance_df = pd.DataFrame()\n",
    "\n",
    "from sklearn.model_selection import KFold,StratifiedKFold\n",
    "fold=StratifiedKFold(n_splits=10,shuffle=True,random_state=1994)\n",
    "i=1\n",
    "for train_index, test_index in fold.split(X,y):\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    m=LGBMClassifier(n_estimators=5000,random_state=1994,learning_rate=0.03,reg_alpha=0.2,colsample_bytree=0.5)\n",
    "    m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_test, y_test)],eval_metric='auc', early_stopping_rounds=200,verbose=200)\n",
    "    \n",
    "    preds=m.predict_proba(X_test,num_iteration=m.best_iteration_)[:,-1]\n",
    "    \n",
    "    fold_importance_df = pd.DataFrame()\n",
    "    fold_importance_df[\"feature\"] = X_train.columns\n",
    "    fold_importance_df[\"importance\"] = m.feature_importances_\n",
    "    fold_importance_df[\"fold\"] = i + 1\n",
    "    feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n",
    "    \n",
    "    \n",
    "    print(\"err: \",roc_auc_score(y_test,preds))\n",
    "    err.append(roc_auc_score(y_test,preds))\n",
    "    p = m.predict_proba(Xtest)[:,-1]\n",
    "    i=i+1\n",
    "    y_pred_tot.append(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9874908521906123"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(err,0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Feature Selection & Importance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>feature</th>\n",
       "      <th>importance</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>within_date_x</td>\n",
       "      <td>315.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>within_date_y</td>\n",
       "      <td>225.7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>coupon_id</td>\n",
       "      <td>205.4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>F_discount_bin_mean_x</td>\n",
       "      <td>137.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>F_price_per_quan_mean_x</td>\n",
       "      <td>104.2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>145</td>\n",
       "      <td>F_brand_count_y</td>\n",
       "      <td>13.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>146</td>\n",
       "      <td>F_date_m_count_x</td>\n",
       "      <td>12.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>147</td>\n",
       "      <td>F_category_count_x</td>\n",
       "      <td>11.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>148</td>\n",
       "      <td>F_brand_nunique_y</td>\n",
       "      <td>11.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>149</td>\n",
       "      <td>diff_w_y</td>\n",
       "      <td>11.2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>150 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     feature  importance\n",
       "0              within_date_x       315.3\n",
       "1              within_date_y       225.7\n",
       "2                  coupon_id       205.4\n",
       "3      F_discount_bin_mean_x       137.5\n",
       "4    F_price_per_quan_mean_x       104.2\n",
       "..                       ...         ...\n",
       "145          F_brand_count_y        13.0\n",
       "146         F_date_m_count_x        12.3\n",
       "147       F_category_count_x        11.9\n",
       "148        F_brand_nunique_y        11.3\n",
       "149                 diff_w_y        11.2\n",
       "\n",
       "[150 rows x 2 columns]"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "all_features = feature_importance_df[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(by=\"importance\", ascending=False)\n",
    "all_features.reset_index(inplace=True)\n",
    "important_features = list(all_features[0:150]['feature'])\n",
    "all_features[0:150]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing Most Correlated Feats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "32\n",
      "['F_coupon_aquired_std_x', 'F_marked_by_sale_max_x', 'F_coupon_aquired_quantile_x', 'F_marked_by_sale_std_x', 'F_item_bought_max_x', 'F_item_bought_mean_x', 'F_marked_by_sale_mean_x', 'F_marked_price_sum_x', 'F_marked_by_sale_std_y', 'F_date_d_count_x', 'F_marked_by_sale_max_y', 'F_item_bought_sum_x', 'F_disc_percent_sum_y', 'F_coupon_aquired_sum_y', 'F_price_per_quan_sum_y', 'F_marked_by_sale_sum_x', 'F_item_bought_quantile_y', 'F_coupon_to_item_sum_y', 'F_item_id_nunique_y', 'F_item_bought_max_y', 'F_item_bought_std_y', 'diff_m_x', 'F_marked_price_sum_y', 'F_item_bought_mean_y', 'F_marked_by_sale_mean_y', 'diff_m_y', 'F_date_d_count_y', 'F_brand_count_y', 'F_date_m_count_x', 'F_category_count_x', 'F_brand_nunique_y', 'diff_w_y']\n"
     ]
    }
   ],
   "source": [
    "df1 = X[important_features]\n",
    "corr_matrix = df1.corr().abs()\n",
    "\n",
    "# Select upper triangle of correlation matrix\n",
    "upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))\n",
    "\n",
    "# Find index of feature columns with correlation greater than 0.95\n",
    "high_cor = [column for column in upper.columns if any(upper[column] > 0.98)]\n",
    "print(len(high_cor))\n",
    "print(high_cor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "118\n",
      "['within_date_x', 'within_date_y', 'coupon_id', 'F_discount_bin_mean_x', 'F_price_per_quan_mean_x', 'F_discount_bin_sum_x', 'F_coupon_discount_sum_x', 'F_other_discount_std_x', 'F_price_per_quan_min_x', 'F_coupon_id_nunique_x', 'F_discount_bin_std_x', 'customer_id_x', 'F_coupon_discount_std_x', 'F_other_discount_min_x', 'F_coupon_discount_mean_x', 'F_coupon_discount_min_x', 'F_marked_price_std_x', 'F_price_per_quan_max_x', 'F_other_discount_mean_x', 'F_marked_price_min_x', 'F_disc_percent_max_x', 'F_tot_disc_std_x', 'F_selling_price_max_x', 'F_disc_percent_mean_x', 'F_quantity_mean_x', 'F_marked_price_max_x', 'F_price_per_quan_std_y', 'F_coupon_to_item_std_x', 'F_price_per_quan_max_y', 'F_item_bought_std_x', 'F_item_bought_min_y', 'F_date_w_nunique_x', 'campaign_id_x', 'F_quantity_std_x', 'F_price_per_quan_std_x', 'F_coupon_discount_min_y', 'F_quantity_std_y', 'F_selling_price_mean_x', 'F_disc_percent_max_y', 'F_quantity_max_x', 'F_tot_disc_max_x', 'F_selling_price_std_x', 'F_disc_percent_std_y', 'F_other_discount_min_y', 'F_date_m_nunique_y', 'F_price_per_quan_mean_y', 'F_other_discount_sum_x', 'customer_id_y', 'F_coupon_aquired_mean_x', 'F_discount_bin_mean_y', 'F_coupon_aquired_max_x', 'F_discount_bin_max_y', 'F_tot_disc_min_y', 'F_disc_percent_std_x', 'F_quantity_sum_x', 'diff_d_x', 'F_marked_price_mean_x', 'F_item_bought_quantile_x', 'F_coupon_to_item_mean_x', 'F_quantity_max_y', 'F_tot_disc_mean_x', 'F_coupon_to_item_max_x', 'campaign_type', 'F_quantity_mean_y', 'F_other_discount_std_y', 'F_marked_price_min_y', 'F_brand_nunique_x', 'F_coupon_to_item_min_y', 'F_other_discount_mean_y', 'F_marked_price_max_y', 'F_selling_price_sum_x', 'F_disc_percent_sum_x', 'F_date_d_nunique_y', 'F_disc_percent_mean_y', 'F_coupon_to_item_sum_x', 'F_item_id_nunique_x', 'F_discount_bin_sum_y', 'F_selling_price_min_x', 'F_tot_disc_sum_x', 'F_coupon_discount_std_y', 'F_coupon_discount_mean_y', 'F_coupon_to_item_std_y', 'F_category_nunique_y', 'F_selling_price_std_y', 'F_coupon_aquired_min_y', 'F_coupon_aquired_sum_x', 'F_quantity_sum_y', 'F_coupon_discount_sum_y', 'F_marked_price_mean_y', 'F_price_per_quan_min_y', 'F_price_per_quan_sum_x', 'F_coupon_to_item_max_y', 'F_coupon_to_item_mean_y', 'F_coupon_id_nunique_y', 'diff_d_y', 'F_discount_bin_std_y', 'F_selling_price_max_y', 'F_selling_price_min_y', 'F_coupon_aquired_quantile_y', 'F_selling_price_mean_y', 'F_coupon_to_item_quantile_x', 'F_brand_count_x', 'F_coupon_to_item_quantile_y', 'F_other_discount_sum_y', 'F_marked_price_std_y', 'F_selling_price_sum_y', 'F_date_w_nunique_y', 'F_tot_disc_mean_y', 'F_tot_disc_sum_y', 'F_category_nunique_x', 'F_tot_disc_max_y', 'F_tot_disc_std_y', 'F_item_bought_sum_y', 'F_coupon_aquired_mean_y', 'F_coupon_aquired_max_y', 'F_date_d_nunique_x', 'F_coupon_aquired_std_y', 'campaign_id_y']\n"
     ]
    }
   ],
   "source": [
    "features = [i for i in important_features if i not in high_cor]\n",
    "print(len(features))\n",
    "print(features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "X=X[features]\n",
    "Xtest=Xtest[features]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StratifiedKFold LGBM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00760738\ttraining's auc: 0.999871\tvalid_1's binary_logloss: 0.0221997\tvalid_1's auc: 0.989158\n",
      "[400]\ttraining's binary_logloss: 0.00387871\ttraining's auc: 0.999998\tvalid_1's binary_logloss: 0.0232418\tvalid_1's auc: 0.988809\n",
      "Early stopping, best iteration is:\n",
      "[216]\ttraining's binary_logloss: 0.00713625\ttraining's auc: 0.999906\tvalid_1's binary_logloss: 0.0221546\tvalid_1's auc: 0.989197\n",
      "err:  0.9891967140225699\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00745603\ttraining's auc: 0.999898\tvalid_1's binary_logloss: 0.0234688\tvalid_1's auc: 0.985541\n",
      "[400]\ttraining's binary_logloss: 0.00383453\ttraining's auc: 0.999997\tvalid_1's binary_logloss: 0.0246998\tvalid_1's auc: 0.98579\n",
      "Early stopping, best iteration is:\n",
      "[220]\ttraining's binary_logloss: 0.00694068\ttraining's auc: 0.999926\tvalid_1's binary_logloss: 0.0234348\tvalid_1's auc: 0.985751\n",
      "err:  0.9857508839533357\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00748728\ttraining's auc: 0.999896\tvalid_1's binary_logloss: 0.0239866\tvalid_1's auc: 0.985753\n",
      "[400]\ttraining's binary_logloss: 0.0038295\ttraining's auc: 0.999998\tvalid_1's binary_logloss: 0.0249127\tvalid_1's auc: 0.986236\n",
      "Early stopping, best iteration is:\n",
      "[216]\ttraining's binary_logloss: 0.00701723\ttraining's auc: 0.999929\tvalid_1's binary_logloss: 0.0239174\tvalid_1's auc: 0.985938\n",
      "err:  0.985937908012393\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.0076477\ttraining's auc: 0.999881\tvalid_1's binary_logloss: 0.023222\tvalid_1's auc: 0.988563\n",
      "[400]\ttraining's binary_logloss: 0.00394566\ttraining's auc: 0.999997\tvalid_1's binary_logloss: 0.0238521\tvalid_1's auc: 0.989228\n",
      "Early stopping, best iteration is:\n",
      "[220]\ttraining's binary_logloss: 0.00708177\ttraining's auc: 0.999922\tvalid_1's binary_logloss: 0.0230687\tvalid_1's auc: 0.988851\n",
      "err:  0.9888508959511055\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00770249\ttraining's auc: 0.99987\tvalid_1's binary_logloss: 0.0247274\tvalid_1's auc: 0.986347\n",
      "[400]\ttraining's binary_logloss: 0.00395297\ttraining's auc: 0.999997\tvalid_1's binary_logloss: 0.0251111\tvalid_1's auc: 0.98721\n",
      "Early stopping, best iteration is:\n",
      "[309]\ttraining's binary_logloss: 0.00522173\ttraining's auc: 0.999983\tvalid_1's binary_logloss: 0.0245799\tvalid_1's auc: 0.986942\n",
      "err:  0.986941839046389\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00749255\ttraining's auc: 0.999891\tvalid_1's binary_logloss: 0.0259766\tvalid_1's auc: 0.984816\n",
      "Early stopping, best iteration is:\n",
      "[146]\ttraining's binary_logloss: 0.00944181\ttraining's auc: 0.999685\tvalid_1's binary_logloss: 0.0257994\tvalid_1's auc: 0.984925\n",
      "err:  0.9849251550888187\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00764126\ttraining's auc: 0.999896\tvalid_1's binary_logloss: 0.0227266\tvalid_1's auc: 0.986894\n",
      "[400]\ttraining's binary_logloss: 0.00392944\ttraining's auc: 0.999998\tvalid_1's binary_logloss: 0.0230837\tvalid_1's auc: 0.988517\n",
      "Early stopping, best iteration is:\n",
      "[265]\ttraining's binary_logloss: 0.00599972\ttraining's auc: 0.999975\tvalid_1's binary_logloss: 0.0225054\tvalid_1's auc: 0.988064\n",
      "err:  0.9880639834007326\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00773802\ttraining's auc: 0.999882\tvalid_1's binary_logloss: 0.0214837\tvalid_1's auc: 0.990615\n",
      "[400]\ttraining's binary_logloss: 0.00401301\ttraining's auc: 0.999997\tvalid_1's binary_logloss: 0.022181\tvalid_1's auc: 0.990194\n",
      "Early stopping, best iteration is:\n",
      "[244]\ttraining's binary_logloss: 0.00656984\ttraining's auc: 0.999947\tvalid_1's binary_logloss: 0.0212068\tvalid_1's auc: 0.990913\n",
      "err:  0.9909134537344824\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.00763537\ttraining's auc: 0.999897\tvalid_1's binary_logloss: 0.0215233\tvalid_1's auc: 0.98971\n",
      "[400]\ttraining's binary_logloss: 0.00393094\ttraining's auc: 0.999997\tvalid_1's binary_logloss: 0.0219487\tvalid_1's auc: 0.990105\n",
      "Early stopping, best iteration is:\n",
      "[266]\ttraining's binary_logloss: 0.00599911\ttraining's auc: 0.99997\tvalid_1's binary_logloss: 0.0213878\tvalid_1's auc: 0.990026\n",
      "err:  0.9900259716429182\n",
      "Training until validation scores don't improve for 200 rounds.\n",
      "[200]\ttraining's binary_logloss: 0.0077246\ttraining's auc: 0.999871\tvalid_1's binary_logloss: 0.0233345\tvalid_1's auc: 0.985836\n",
      "[400]\ttraining's binary_logloss: 0.00396991\ttraining's auc: 0.999997\tvalid_1's binary_logloss: 0.0231112\tvalid_1's auc: 0.987635\n",
      "Early stopping, best iteration is:\n",
      "[262]\ttraining's binary_logloss: 0.00615103\ttraining's auc: 0.999961\tvalid_1's binary_logloss: 0.0229569\tvalid_1's auc: 0.986753\n",
      "err:  0.9867533201671532\n"
     ]
    }
   ],
   "source": [
    "err=[]\n",
    "y_pred_tot=[]\n",
    "\n",
    "# feature_importance_df = pd.DataFrame()\n",
    "\n",
    "from sklearn.model_selection import KFold,StratifiedKFold\n",
    "fold=StratifiedKFold(n_splits=10,shuffle=True,random_state=1994)\n",
    "i=1\n",
    "for train_index, test_index in fold.split(X,y):\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    m=LGBMClassifier(n_estimators=5000,random_state=1994,learning_rate=0.03,reg_alpha=0.2,colsample_bytree=0.5)\n",
    "    m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_test, y_test)],eval_metric='auc', early_stopping_rounds=200,verbose=200)\n",
    "    \n",
    "    preds=m.predict_proba(X_test,num_iteration=m.best_iteration_)[:,-1]\n",
    "    \n",
    "#     fold_importance_df = pd.DataFrame()\n",
    "#     fold_importance_df[\"feature\"] = X_train.columns\n",
    "#     fold_importance_df[\"importance\"] = m.feature_importances_\n",
    "#     fold_importance_df[\"fold\"] = i + 1\n",
    "#     feature_importance_df = pd.concat([feature_importance_df, fold_importance_df], axis=0)\n",
    "    \n",
    "    \n",
    "    print(\"err: \",roc_auc_score(y_test,preds))\n",
    "    err.append(roc_auc_score(y_test,preds))\n",
    "    p = m.predict_proba(Xtest)[:,-1]\n",
    "    i=i+1\n",
    "    y_pred_tot.append(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9877360125019898"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(err,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.227818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0.050355</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.001558</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>0.000030</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.000120</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  redemption_status\n",
       "0   3           0.227818\n",
       "1   4           0.050355\n",
       "2   5           0.001558\n",
       "3   8           0.000030\n",
       "4  10           0.000120"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s['redemption_status']=np.mean(y_pred_tot,0)\n",
    "s.head()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "91"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(s.redemption_status>0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50226, 2)"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.to_csv('AV_amex_lgb_folds_v28.csv',index=False)\n",
    "s.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Catboost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(78369, 118) (50226, 118)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([62])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(X.shape,Xtest.shape)\n",
    "\n",
    "X_train,X_val,y_train,y_val = train_test_split(X,y,test_size=0.25,random_state = 1994,stratify=y)\n",
    "categorical_features_indices = np.where(X_train.dtypes =='category')[0]\n",
    "categorical_features_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.6327550\tbest: 0.6327550 (0)\ttotal: 111ms\tremaining: 4m 36s\n",
      "200:\ttest: 0.9858867\tbest: 0.9858867 (200)\ttotal: 10.9s\tremaining: 2m 4s\n",
      "400:\ttest: 0.9872119\tbest: 0.9872119 (400)\ttotal: 20.6s\tremaining: 1m 47s\n",
      "600:\ttest: 0.9878884\tbest: 0.9878980 (596)\ttotal: 30.2s\tremaining: 1m 35s\n",
      "800:\ttest: 0.9882603\tbest: 0.9882603 (800)\ttotal: 39.9s\tremaining: 1m 24s\n",
      "1000:\ttest: 0.9883727\tbest: 0.9884171 (923)\ttotal: 49.7s\tremaining: 1m 14s\n",
      "1200:\ttest: 0.9884913\tbest: 0.9884913 (1200)\ttotal: 59.5s\tremaining: 1m 4s\n",
      "1400:\ttest: 0.9885530\tbest: 0.9885935 (1292)\ttotal: 1m 9s\tremaining: 54.4s\n",
      "1600:\ttest: 0.9886945\tbest: 0.9887141 (1569)\ttotal: 1m 19s\tremaining: 44.5s\n",
      "1800:\ttest: 0.9887081\tbest: 0.9887342 (1713)\ttotal: 1m 28s\tremaining: 34.5s\n",
      "2000:\ttest: 0.9886637\tbest: 0.9887342 (1713)\ttotal: 1m 38s\tremaining: 24.6s\n",
      "Stopped by overfitting detector  (300 iterations wait)\n",
      "\n",
      "bestTest = 0.9887341549\n",
      "bestIteration = 1713\n",
      "\n",
      "Shrink model to first 1714 iterations.\n",
      "0.9887341549285807\n"
     ]
    }
   ],
   "source": [
    "m=CatBoostClassifier(n_estimators=2500,random_state=1994,learning_rate=0.03,eval_metric='AUC')\n",
    "# m=RidgeCV(cv=4)\n",
    "m.fit(X_train,y_train,eval_set=[(X_val, y_val.values)], early_stopping_rounds=300,verbose=200,cat_features=categorical_features_indices)\n",
    "p=m.predict_proba(X_val)[:,-1]\n",
    "print(roc_auc_score(y_val,p))\n",
    "\n",
    "\n",
    "# 0:\ttest: 0.7072835\tbest: 0.7072835 (0)\ttotal: 94.6ms\tremaining: 3m 56s\n",
    "# 200:\ttest: 0.9846892\tbest: 0.9846892 (200)\ttotal: 6.49s\tremaining: 1m 14s\n",
    "# 400:\ttest: 0.9857940\tbest: 0.9857940 (400)\ttotal: 13.1s\tremaining: 1m 8s\n",
    "# 600:\ttest: 0.9860940\tbest: 0.9860980 (590)\ttotal: 19.6s\tremaining: 1m 1s\n",
    "# 800:\ttest: 0.9861993\tbest: 0.9862259 (737)\ttotal: 25.9s\tremaining: 54.9s\n",
    "# 1000:\ttest: 0.9862786\tbest: 0.9863095 (882)\ttotal: 32.2s\tremaining: 48.2s\n",
    "# 1200:\ttest: 0.9863154\tbest: 0.9863839 (1056)\ttotal: 38.5s\tremaining: 41.6s\n",
    "# 1400:\ttest: 0.9863995\tbest: 0.9864448 (1265)\ttotal: 44.8s\tremaining: 35.1s\n",
    "# Stopped by overfitting detector  (300 iterations wait)\n",
    "\n",
    "# bestTest = 0.9864447541\n",
    "# bestIteration = 1265\n",
    "\n",
    "# Shrink model to first 1266 iterations.\n",
    "# 0.9864447540507506"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StratifiedKFold CatBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0:\ttest: 0.6119047\ttest1: 0.5973074\tbest: 0.5973074 (0)\ttotal: 61.4ms\tremaining: 5m 6s\n",
      "200:\ttest: 0.9900179\ttest1: 0.9858570\tbest: 0.9858570 (200)\ttotal: 12.2s\tremaining: 4m 51s\n",
      "400:\ttest: 0.9941425\ttest1: 0.9872331\tbest: 0.9872488 (392)\ttotal: 24.1s\tremaining: 4m 35s\n",
      "600:\ttest: 0.9963987\ttest1: 0.9879112\tbest: 0.9879349 (599)\ttotal: 36.3s\tremaining: 4m 25s\n",
      "800:\ttest: 0.9977695\ttest1: 0.9881005\tbest: 0.9881951 (711)\ttotal: 47.8s\tremaining: 4m 10s\n",
      "1000:\ttest: 0.9985384\ttest1: 0.9885224\tbest: 0.9885460 (993)\ttotal: 59.3s\tremaining: 3m 57s\n",
      "1200:\ttest: 0.9991091\ttest1: 0.9887195\tbest: 0.9887353 (1183)\ttotal: 1m 10s\tremaining: 3m 44s\n",
      "1400:\ttest: 0.9994799\ttest1: 0.9887274\tbest: 0.9888063 (1380)\ttotal: 1m 22s\tremaining: 3m 32s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9888062644\n",
      "bestIteration = 1380\n",
      "\n",
      "Shrink model to first 1381 iterations.\n",
      "err_cb:  0.9888062643913825\n",
      "0:\ttest: 0.6114484\ttest1: 0.5944863\tbest: 0.5944863 (0)\ttotal: 80.7ms\tremaining: 6m 43s\n",
      "200:\ttest: 0.9902400\ttest1: 0.9858136\tbest: 0.9858215 (199)\ttotal: 12.6s\tremaining: 5m\n",
      "400:\ttest: 0.9941476\ttest1: 0.9870202\tbest: 0.9870754 (380)\ttotal: 24.8s\tremaining: 4m 44s\n",
      "600:\ttest: 0.9964631\ttest1: 0.9876747\tbest: 0.9877338 (577)\ttotal: 37s\tremaining: 4m 30s\n",
      "800:\ttest: 0.9977868\ttest1: 0.9878008\tbest: 0.9879664 (741)\ttotal: 49.2s\tremaining: 4m 17s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9879664385\n",
      "bestIteration = 741\n",
      "\n",
      "Shrink model to first 742 iterations.\n",
      "err_cb:  0.9879664385073967\n",
      "0:\ttest: 0.6114360\ttest1: 0.5878190\tbest: 0.5878190 (0)\ttotal: 60.8ms\tremaining: 5m 4s\n",
      "200:\ttest: 0.9901635\ttest1: 0.9824819\tbest: 0.9824819 (200)\ttotal: 12.5s\tremaining: 4m 59s\n",
      "400:\ttest: 0.9942532\ttest1: 0.9835347\tbest: 0.9836057 (398)\ttotal: 24.8s\tremaining: 4m 44s\n",
      "600:\ttest: 0.9964770\ttest1: 0.9841261\tbest: 0.9841537 (598)\ttotal: 36.9s\tremaining: 4m 29s\n",
      "800:\ttest: 0.9978391\ttest1: 0.9844455\tbest: 0.9844652 (799)\ttotal: 49s\tremaining: 4m 16s\n",
      "1000:\ttest: 0.9986330\ttest1: 0.9844179\tbest: 0.9844731 (868)\ttotal: 1m 1s\tremaining: 4m 6s\n",
      "1200:\ttest: 0.9992083\ttest1: 0.9845914\tbest: 0.9846032 (1127)\ttotal: 1m 14s\tremaining: 3m 54s\n",
      "1400:\ttest: 0.9995138\ttest1: 0.9845559\tbest: 0.9847175 (1343)\ttotal: 1m 26s\tremaining: 3m 41s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9847175346\n",
      "bestIteration = 1343\n",
      "\n",
      "Shrink model to first 1344 iterations.\n",
      "err_cb:  0.984717534618175\n",
      "0:\ttest: 0.6131721\ttest1: 0.5849584\tbest: 0.5849584 (0)\ttotal: 70.8ms\tremaining: 5m 53s\n",
      "200:\ttest: 0.9904716\ttest1: 0.9809994\tbest: 0.9810467 (194)\ttotal: 12.4s\tremaining: 4m 56s\n",
      "400:\ttest: 0.9943757\ttest1: 0.9818550\tbest: 0.9819339 (387)\ttotal: 24.8s\tremaining: 4m 44s\n",
      "600:\ttest: 0.9965805\ttest1: 0.9820955\tbest: 0.9822493 (492)\ttotal: 37s\tremaining: 4m 30s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9822493139\n",
      "bestIteration = 492\n",
      "\n",
      "Shrink model to first 493 iterations.\n",
      "err_cb:  0.9822493139450524\n",
      "0:\ttest: 0.6150387\ttest1: 0.5526113\tbest: 0.5526113 (0)\ttotal: 59.9ms\tremaining: 4m 59s\n",
      "200:\ttest: 0.9900983\ttest1: 0.9835702\tbest: 0.9835820 (197)\ttotal: 12.5s\tremaining: 4m 58s\n",
      "400:\ttest: 0.9940711\ttest1: 0.9855692\tbest: 0.9856047 (394)\ttotal: 24.7s\tremaining: 4m 42s\n",
      "600:\ttest: 0.9963286\ttest1: 0.9856914\tbest: 0.9858176 (550)\ttotal: 36.9s\tremaining: 4m 30s\n",
      "800:\ttest: 0.9979016\ttest1: 0.9858176\tbest: 0.9859083 (660)\ttotal: 49.1s\tremaining: 4m 17s\n",
      "1000:\ttest: 0.9987082\ttest1: 0.9859438\tbest: 0.9860305 (924)\ttotal: 1m 1s\tremaining: 4m 4s\n",
      "1200:\ttest: 0.9992316\ttest1: 0.9858176\tbest: 0.9861015 (1126)\ttotal: 1m 13s\tremaining: 3m 52s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.986101473\n",
      "bestIteration = 1126\n",
      "\n",
      "Shrink model to first 1127 iterations.\n",
      "err_cb:  0.9861014730467148\n",
      "0:\ttest: 0.6112565\ttest1: 0.5933587\tbest: 0.5933587 (0)\ttotal: 60.1ms\tremaining: 5m\n",
      "200:\ttest: 0.9898830\ttest1: 0.9871424\tbest: 0.9871424 (200)\ttotal: 12.7s\tremaining: 5m 3s\n",
      "400:\ttest: 0.9940344\ttest1: 0.9889285\tbest: 0.9889324 (398)\ttotal: 25.1s\tremaining: 4m 47s\n",
      "600:\ttest: 0.9964296\ttest1: 0.9894608\tbest: 0.9895160 (593)\ttotal: 37.1s\tremaining: 4m 31s\n",
      "800:\ttest: 0.9978072\ttest1: 0.9899379\tbest: 0.9900759 (745)\ttotal: 49.3s\tremaining: 4m 18s\n",
      "1000:\ttest: 0.9986201\ttest1: 0.9904031\tbest: 0.9904071 (996)\ttotal: 1m 1s\tremaining: 4m 5s\n",
      "1200:\ttest: 0.9991602\ttest1: 0.9904268\tbest: 0.9905253 (1165)\ttotal: 1m 13s\tremaining: 3m 53s\n",
      "1400:\ttest: 0.9994913\ttest1: 0.9906870\tbest: 0.9907422 (1338)\ttotal: 1m 26s\tremaining: 3m 41s\n",
      "1600:\ttest: 0.9997023\ttest1: 0.9905569\tbest: 0.9907777 (1479)\ttotal: 1m 38s\tremaining: 3m 29s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9907776867\n",
      "bestIteration = 1479\n",
      "\n",
      "Shrink model to first 1480 iterations.\n",
      "err_cb:  0.9907776866542599\n",
      "0:\ttest: 0.6127886\ttest1: 0.5710757\tbest: 0.5710757 (0)\ttotal: 61.1ms\tremaining: 5m 5s\n",
      "200:\ttest: 0.9902041\ttest1: 0.9815081\tbest: 0.9815081 (200)\ttotal: 12.7s\tremaining: 5m 3s\n",
      "400:\ttest: 0.9940966\ttest1: 0.9829551\tbest: 0.9830063 (396)\ttotal: 25s\tremaining: 4m 46s\n",
      "600:\ttest: 0.9962402\ttest1: 0.9839960\tbest: 0.9840591 (598)\ttotal: 37.9s\tremaining: 4m 37s\n",
      "800:\ttest: 0.9975798\ttest1: 0.9849896\tbest: 0.9850132 (798)\ttotal: 50s\tremaining: 4m 22s\n",
      "1000:\ttest: 0.9985384\ttest1: 0.9857072\tbest: 0.9858097 (982)\ttotal: 1m 2s\tremaining: 4m 8s\n",
      "1200:\ttest: 0.9990732\ttest1: 0.9860778\tbest: 0.9862434 (1107)\ttotal: 1m 14s\tremaining: 3m 54s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9862434154\n",
      "bestIteration = 1107\n",
      "\n",
      "Shrink model to first 1108 iterations.\n",
      "err_cb:  0.986243415449642\n",
      "0:\ttest: 0.6122243\ttest1: 0.5823246\tbest: 0.5823246 (0)\ttotal: 60.3ms\tremaining: 5m 1s\n",
      "200:\ttest: 0.9898265\ttest1: 0.9876313\tbest: 0.9876313 (200)\ttotal: 12.5s\tremaining: 4m 59s\n",
      "400:\ttest: 0.9940180\ttest1: 0.9889561\tbest: 0.9889561 (400)\ttotal: 24.7s\tremaining: 4m 42s\n",
      "600:\ttest: 0.9963722\ttest1: 0.9893109\tbest: 0.9895317 (563)\ttotal: 36.8s\tremaining: 4m 29s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9895317478\n",
      "bestIteration = 563\n",
      "\n",
      "Shrink model to first 564 iterations.\n",
      "err_cb:  0.9895317477841213\n",
      "0:\ttest: 0.6095623\ttest1: 0.6112592\tbest: 0.6112592 (0)\ttotal: 60.4ms\tremaining: 5m 1s\n",
      "200:\ttest: 0.9903689\ttest1: 0.9817328\tbest: 0.9817407 (193)\ttotal: 12.6s\tremaining: 5m\n",
      "400:\ttest: 0.9942400\ttest1: 0.9827540\tbest: 0.9830182 (314)\ttotal: 24.8s\tremaining: 4m 44s\n",
      "600:\ttest: 0.9965155\ttest1: 0.9827264\tbest: 0.9830891 (464)\ttotal: 36.9s\tremaining: 4m 30s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9830891398\n",
      "bestIteration = 464\n",
      "\n",
      "Shrink model to first 465 iterations.\n",
      "err_cb:  0.9830891398290383\n",
      "0:\ttest: 0.7866621\ttest1: 0.7751924\tbest: 0.7751924 (0)\ttotal: 72.6ms\tremaining: 6m 2s\n",
      "200:\ttest: 0.9901299\ttest1: 0.9880981\tbest: 0.9881464 (196)\ttotal: 12.5s\tremaining: 4m 59s\n",
      "400:\ttest: 0.9941884\ttest1: 0.9887582\tbest: 0.9887582 (395)\ttotal: 24.7s\tremaining: 4m 42s\n",
      "600:\ttest: 0.9964802\ttest1: 0.9889554\tbest: 0.9892010 (546)\ttotal: 36.8s\tremaining: 4m 29s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9892009596\n",
      "bestIteration = 546\n",
      "\n",
      "Shrink model to first 547 iterations.\n",
      "err_cb:  0.9892009595569294\n",
      "0:\ttest: 0.7869482\ttest1: 0.7854320\tbest: 0.7854320 (0)\ttotal: 72.4ms\tremaining: 6m 1s\n",
      "200:\ttest: 0.9905100\ttest1: 0.9865203\tbest: 0.9865284 (199)\ttotal: 12.4s\tremaining: 4m 56s\n",
      "400:\ttest: 0.9944663\ttest1: 0.9875105\tbest: 0.9875105 (400)\ttotal: 24.6s\tremaining: 4m 42s\n",
      "600:\ttest: 0.9965177\ttest1: 0.9880538\tbest: 0.9880820 (599)\ttotal: 36.6s\tremaining: 4m 27s\n",
      "800:\ttest: 0.9979102\ttest1: 0.9883557\tbest: 0.9884121 (798)\ttotal: 48.7s\tremaining: 4m 15s\n",
      "1000:\ttest: 0.9987120\ttest1: 0.9885811\tbest: 0.9886375 (916)\ttotal: 1m\tremaining: 4m 2s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9886374614\n",
      "bestIteration = 916\n",
      "\n",
      "Shrink model to first 917 iterations.\n",
      "err_cb:  0.9886374613601236\n",
      "0:\ttest: 0.7903844\ttest1: 0.7276009\tbest: 0.7276009 (0)\ttotal: 72.5ms\tremaining: 6m 2s\n",
      "200:\ttest: 0.9902605\ttest1: 0.9899416\tbest: 0.9900543 (187)\ttotal: 12.5s\tremaining: 4m 59s\n",
      "400:\ttest: 0.9941594\ttest1: 0.9902314\tbest: 0.9905534 (265)\ttotal: 24.6s\tremaining: 4m 41s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9905533552\n",
      "bestIteration = 265\n",
      "\n",
      "Shrink model to first 266 iterations.\n",
      "err_cb:  0.9905533552292632\n",
      "0:\ttest: 0.7868431\ttest1: 0.7687021\tbest: 0.7687021 (0)\ttotal: 74.5ms\tremaining: 6m 12s\n",
      "200:\ttest: 0.9901178\ttest1: 0.9866048\tbest: 0.9866089 (198)\ttotal: 13s\tremaining: 5m 9s\n",
      "400:\ttest: 0.9941407\ttest1: 0.9883436\tbest: 0.9883919 (379)\ttotal: 25s\tremaining: 4m 46s\n",
      "600:\ttest: 0.9963732\ttest1: 0.9889474\tbest: 0.9889554 (599)\ttotal: 37.1s\tremaining: 4m 31s\n",
      "800:\ttest: 0.9977589\ttest1: 0.9894062\tbest: 0.9894183 (782)\ttotal: 49s\tremaining: 4m 16s\n",
      "1000:\ttest: 0.9985785\ttest1: 0.9895069\tbest: 0.9896397 (940)\ttotal: 1m\tremaining: 4m 3s\n",
      "1200:\ttest: 0.9991652\ttest1: 0.9894908\tbest: 0.9896679 (1091)\ttotal: 1m 13s\tremaining: 3m 51s\n",
      "1400:\ttest: 0.9995029\ttest1: 0.9898450\tbest: 0.9898892 (1378)\ttotal: 1m 25s\tremaining: 3m 39s\n",
      "1600:\ttest: 0.9997230\ttest1: 0.9897765\tbest: 0.9899053 (1462)\ttotal: 1m 37s\tremaining: 3m 26s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9899053323\n",
      "bestIteration = 1462\n",
      "\n",
      "Shrink model to first 1463 iterations.\n",
      "err_cb:  0.9899053323029365\n",
      "0:\ttest: 0.7827333\ttest1: 0.8105217\tbest: 0.8105217 (0)\ttotal: 73.8ms\tremaining: 6m 8s\n",
      "200:\ttest: 0.9902735\ttest1: 0.9843750\tbest: 0.9843750 (200)\ttotal: 12.4s\tremaining: 4m 55s\n",
      "400:\ttest: 0.9942054\ttest1: 0.9862345\tbest: 0.9862466 (399)\ttotal: 24.6s\tremaining: 4m 41s\n",
      "600:\ttest: 0.9963466\ttest1: 0.9867900\tbest: 0.9868785 (588)\ttotal: 36.9s\tremaining: 4m 30s\n",
      "800:\ttest: 0.9976859\ttest1: 0.9873092\tbest: 0.9873173 (792)\ttotal: 48.9s\tremaining: 4m 16s\n",
      "1000:\ttest: 0.9985313\ttest1: 0.9881021\tbest: 0.9881625 (996)\ttotal: 1m 1s\tremaining: 4m 3s\n",
      "1200:\ttest: 0.9991063\ttest1: 0.9881867\tbest: 0.9883155 (1176)\ttotal: 1m 13s\tremaining: 3m 51s\n",
      "1400:\ttest: 0.9994644\ttest1: 0.9883114\tbest: 0.9883316 (1230)\ttotal: 1m 25s\tremaining: 3m 39s\n",
      "1600:\ttest: 0.9997094\ttest1: 0.9885328\tbest: 0.9885328 (1600)\ttotal: 1m 37s\tremaining: 3m 26s\n",
      "1800:\ttest: 0.9998486\ttest1: 0.9885851\tbest: 0.9886737 (1680)\ttotal: 1m 49s\tremaining: 3m 14s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9886736862\n",
      "bestIteration = 1680\n",
      "\n",
      "Shrink model to first 1681 iterations.\n",
      "err_cb:  0.988673686244204\n",
      "0:\ttest: 0.7843707\ttest1: 0.7980422\tbest: 0.7980422 (0)\ttotal: 71.9ms\tremaining: 5m 59s\n",
      "200:\ttest: 0.9904086\ttest1: 0.9825396\tbest: 0.9825436 (199)\ttotal: 12.4s\tremaining: 4m 56s\n",
      "400:\ttest: 0.9942174\ttest1: 0.9846004\tbest: 0.9846286 (391)\ttotal: 24.6s\tremaining: 4m 42s\n",
      "600:\ttest: 0.9965116\ttest1: 0.9856831\tbest: 0.9857636 (594)\ttotal: 36.7s\tremaining: 4m 28s\n",
      "800:\ttest: 0.9978408\ttest1: 0.9863513\tbest: 0.9863955 (780)\ttotal: 48.3s\tremaining: 4m 13s\n",
      "1000:\ttest: 0.9986157\ttest1: 0.9869751\tbest: 0.9869953 (994)\ttotal: 59.7s\tremaining: 3m 58s\n",
      "1200:\ttest: 0.9991666\ttest1: 0.9872488\tbest: 0.9872810 (1179)\ttotal: 1m 11s\tremaining: 3m 45s\n",
      "1400:\ttest: 0.9994972\ttest1: 0.9875749\tbest: 0.9876030 (1383)\ttotal: 1m 23s\tremaining: 3m 34s\n",
      "1600:\ttest: 0.9997211\ttest1: 0.9879170\tbest: 0.9881021 (1590)\ttotal: 1m 36s\tremaining: 3m 24s\n",
      "Stopped by overfitting detector  (200 iterations wait)\n",
      "\n",
      "bestTest = 0.9881021381\n",
      "bestIteration = 1590\n",
      "\n",
      "Shrink model to first 1591 iterations.\n",
      "err_cb:  0.9881021380731582\n"
     ]
    }
   ],
   "source": [
    "errCB=[]\n",
    "y_pred_tot_cb=[]\n",
    "from sklearn.model_selection import KFold,StratifiedKFold\n",
    "fold=StratifiedKFold(n_splits=15,shuffle=True,random_state=1994)\n",
    "i=1\n",
    "for train_index, test_index in fold.split(X,y):\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    m=CatBoostClassifier(n_estimators=5000,random_state=1994,eval_metric='AUC',learning_rate=0.03)\n",
    "    m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_test, y_test)], early_stopping_rounds=200,verbose=200,cat_features=categorical_features_indices)\n",
    "    preds=m.predict_proba(X_test)[:,-1]\n",
    "    print(\"err_cb: \",roc_auc_score(y_test,preds))\n",
    "    errCB.append(roc_auc_score(y_test,preds))\n",
    "    p = m.predict_proba(Xtest)[:,-1]\n",
    "    i=i+1\n",
    "    y_pred_tot_cb.append(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9876370631328267"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(errCB,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.252884</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0.037019</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000079</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>0.000003</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.000004</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  redemption_status\n",
       "0   3           0.252884\n",
       "1   4           0.037019\n",
       "2   5           0.000079\n",
       "3   8           0.000003\n",
       "4  10           0.000004"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s['redemption_status']=np.mean(y_pred_tot_cb,0)\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "130"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(s.redemption_status>0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50226, 2)"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.to_csv('AV_amex_cb_folds_v28.csv',index=False)\n",
    "s.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.234085</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0.047021</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.001188</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>0.000023</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.000091</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  redemption_status\n",
       "0   3           0.234085\n",
       "1   4           0.047021\n",
       "2   5           0.001188\n",
       "3   8           0.000023\n",
       "4  10           0.000091"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s['redemption_status']=np.mean(y_pred_tot_cb,0)*0.25+np.mean(y_pred_tot,0)*0.75\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "94"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sum(s.redemption_status>0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "s.to_csv('AV_amex_stack2_folds_v28.csv',index=False)\n",
    "# s.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(78369, 118) (50226, 118)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([], dtype=int64)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(X.shape,Xtest.shape)\n",
    "\n",
    "X=pd.get_dummies(X,drop_first=True)\n",
    "Xtest=pd.get_dummies(Xtest,drop_first=True)\n",
    "\n",
    "X_train,X_val,y_train,y_val = train_test_split(X,y,test_size=0.25,random_state = 1994,stratify=y)\n",
    "categorical_features_indices = np.where(X_train.dtypes =='category')[0]\n",
    "categorical_features_indices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### StratifiedKFold XGB"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\tvalidation_0-auc:0.970545\tvalidation_1-auc:0.964131\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.986395\tvalidation_1-auc:0.985703\n",
      "[400]\tvalidation_0-auc:0.991917\tvalidation_1-auc:0.987985\n",
      "[600]\tvalidation_0-auc:0.994723\tvalidation_1-auc:0.988128\n",
      "[800]\tvalidation_0-auc:0.996379\tvalidation_1-auc:0.988373\n",
      "[1000]\tvalidation_0-auc:0.997494\tvalidation_1-auc:0.988447\n",
      "[1200]\tvalidation_0-auc:0.998273\tvalidation_1-auc:0.988554\n",
      "[1400]\tvalidation_0-auc:0.998826\tvalidation_1-auc:0.988685\n",
      "[1600]\tvalidation_0-auc:0.999204\tvalidation_1-auc:0.988794\n",
      "[1800]\tvalidation_0-auc:0.999456\tvalidation_1-auc:0.988842\n",
      "Stopping. Best iteration:\n",
      "[1652]\tvalidation_0-auc:0.999276\tvalidation_1-auc:0.988932\n",
      "\n",
      "err_xgb:  0.9889320573352247\n",
      "[0]\tvalidation_0-auc:0.965959\tvalidation_1-auc:0.962824\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.986413\tvalidation_1-auc:0.982582\n",
      "[400]\tvalidation_0-auc:0.992214\tvalidation_1-auc:0.985179\n",
      "[600]\tvalidation_0-auc:0.99491\tvalidation_1-auc:0.985379\n",
      "[800]\tvalidation_0-auc:0.996621\tvalidation_1-auc:0.985564\n",
      "[1000]\tvalidation_0-auc:0.99769\tvalidation_1-auc:0.98592\n",
      "[1200]\tvalidation_0-auc:0.998423\tvalidation_1-auc:0.985936\n",
      "[1400]\tvalidation_0-auc:0.998937\tvalidation_1-auc:0.986047\n",
      "Stopping. Best iteration:\n",
      "[1364]\tvalidation_0-auc:0.998863\tvalidation_1-auc:0.986137\n",
      "\n",
      "err_xgb:  0.9861372827168597\n",
      "[0]\tvalidation_0-auc:0.970555\tvalidation_1-auc:0.967999\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.986528\tvalidation_1-auc:0.981649\n",
      "[400]\tvalidation_0-auc:0.991881\tvalidation_1-auc:0.983764\n",
      "[600]\tvalidation_0-auc:0.994496\tvalidation_1-auc:0.984553\n",
      "[800]\tvalidation_0-auc:0.996109\tvalidation_1-auc:0.985107\n",
      "[1000]\tvalidation_0-auc:0.99731\tvalidation_1-auc:0.985456\n",
      "[1200]\tvalidation_0-auc:0.998126\tvalidation_1-auc:0.985823\n",
      "[1400]\tvalidation_0-auc:0.998674\tvalidation_1-auc:0.986093\n",
      "[1600]\tvalidation_0-auc:0.999095\tvalidation_1-auc:0.986058\n",
      "[1800]\tvalidation_0-auc:0.999375\tvalidation_1-auc:0.986314\n",
      "[2000]\tvalidation_0-auc:0.99958\tvalidation_1-auc:0.986531\n",
      "Stopping. Best iteration:\n",
      "[1960]\tvalidation_0-auc:0.999536\tvalidation_1-auc:0.986566\n",
      "\n",
      "err_xgb:  0.9865660265503589\n",
      "[0]\tvalidation_0-auc:0.970466\tvalidation_1-auc:0.965301\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.986434\tvalidation_1-auc:0.983185\n",
      "[400]\tvalidation_0-auc:0.992026\tvalidation_1-auc:0.986868\n",
      "[600]\tvalidation_0-auc:0.994626\tvalidation_1-auc:0.987965\n",
      "[800]\tvalidation_0-auc:0.9963\tvalidation_1-auc:0.988671\n",
      "[1000]\tvalidation_0-auc:0.997404\tvalidation_1-auc:0.989467\n",
      "[1200]\tvalidation_0-auc:0.998216\tvalidation_1-auc:0.989994\n",
      "[1400]\tvalidation_0-auc:0.998743\tvalidation_1-auc:0.990264\n",
      "[1600]\tvalidation_0-auc:0.999118\tvalidation_1-auc:0.990425\n",
      "[1800]\tvalidation_0-auc:0.999403\tvalidation_1-auc:0.990453\n",
      "[2000]\tvalidation_0-auc:0.999581\tvalidation_1-auc:0.99067\n",
      "[2200]\tvalidation_0-auc:0.999702\tvalidation_1-auc:0.990696\n",
      "[2400]\tvalidation_0-auc:0.999789\tvalidation_1-auc:0.990801\n",
      "[2600]\tvalidation_0-auc:0.999851\tvalidation_1-auc:0.990786\n",
      "Stopping. Best iteration:\n",
      "[2575]\tvalidation_0-auc:0.999845\tvalidation_1-auc:0.990808\n",
      "\n",
      "err_xgb:  0.9908075910595443\n",
      "[0]\tvalidation_0-auc:0.968788\tvalidation_1-auc:0.958936\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.98655\tvalidation_1-auc:0.983691\n",
      "[400]\tvalidation_0-auc:0.992088\tvalidation_1-auc:0.985906\n",
      "[600]\tvalidation_0-auc:0.994867\tvalidation_1-auc:0.98613\n",
      "Stopping. Best iteration:\n",
      "[587]\tvalidation_0-auc:0.994746\tvalidation_1-auc:0.986176\n",
      "\n",
      "err_xgb:  0.9861760990310038\n",
      "[0]\tvalidation_0-auc:0.970737\tvalidation_1-auc:0.968319\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.98652\tvalidation_1-auc:0.98271\n",
      "[400]\tvalidation_0-auc:0.991923\tvalidation_1-auc:0.985576\n",
      "[600]\tvalidation_0-auc:0.994755\tvalidation_1-auc:0.986333\n",
      "[800]\tvalidation_0-auc:0.996499\tvalidation_1-auc:0.986278\n",
      "Stopping. Best iteration:\n",
      "[601]\tvalidation_0-auc:0.994771\tvalidation_1-auc:0.98639\n",
      "\n",
      "err_xgb:  0.9863895887587953\n",
      "[0]\tvalidation_0-auc:0.969916\tvalidation_1-auc:0.975778\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.986259\tvalidation_1-auc:0.98477\n",
      "[400]\tvalidation_0-auc:0.991985\tvalidation_1-auc:0.98612\n",
      "[600]\tvalidation_0-auc:0.994625\tvalidation_1-auc:0.987184\n",
      "[800]\tvalidation_0-auc:0.996332\tvalidation_1-auc:0.987656\n",
      "[1000]\tvalidation_0-auc:0.997489\tvalidation_1-auc:0.988119\n",
      "[1200]\tvalidation_0-auc:0.99829\tvalidation_1-auc:0.988535\n",
      "[1400]\tvalidation_0-auc:0.998815\tvalidation_1-auc:0.98859\n",
      "[1600]\tvalidation_0-auc:0.999165\tvalidation_1-auc:0.988583\n",
      "Stopping. Best iteration:\n",
      "[1537]\tvalidation_0-auc:0.999067\tvalidation_1-auc:0.98869\n",
      "\n",
      "err_xgb:  0.988690337560783\n",
      "[0]\tvalidation_0-auc:0.969981\tvalidation_1-auc:0.975388\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.985744\tvalidation_1-auc:0.988193\n",
      "[400]\tvalidation_0-auc:0.991458\tvalidation_1-auc:0.991245\n",
      "[600]\tvalidation_0-auc:0.994347\tvalidation_1-auc:0.991642\n",
      "[800]\tvalidation_0-auc:0.996157\tvalidation_1-auc:0.991875\n",
      "[1000]\tvalidation_0-auc:0.99739\tvalidation_1-auc:0.992134\n",
      "[1200]\tvalidation_0-auc:0.998245\tvalidation_1-auc:0.992436\n",
      "[1400]\tvalidation_0-auc:0.998807\tvalidation_1-auc:0.992344\n",
      "Stopping. Best iteration:\n",
      "[1217]\tvalidation_0-auc:0.998306\tvalidation_1-auc:0.992489\n",
      "\n",
      "err_xgb:  0.9924890432131439\n",
      "[0]\tvalidation_0-auc:0.970583\tvalidation_1-auc:0.970265\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.986366\tvalidation_1-auc:0.984431\n",
      "[400]\tvalidation_0-auc:0.991906\tvalidation_1-auc:0.987939\n",
      "[600]\tvalidation_0-auc:0.994764\tvalidation_1-auc:0.9888\n",
      "[800]\tvalidation_0-auc:0.996347\tvalidation_1-auc:0.989565\n",
      "[1000]\tvalidation_0-auc:0.997447\tvalidation_1-auc:0.990132\n",
      "[1200]\tvalidation_0-auc:0.998235\tvalidation_1-auc:0.990594\n",
      "[1400]\tvalidation_0-auc:0.998721\tvalidation_1-auc:0.990541\n",
      "[1600]\tvalidation_0-auc:0.999099\tvalidation_1-auc:0.990568\n",
      "[1800]\tvalidation_0-auc:0.999356\tvalidation_1-auc:0.990644\n",
      "Stopping. Best iteration:\n",
      "[1790]\tvalidation_0-auc:0.999342\tvalidation_1-auc:0.990688\n",
      "\n",
      "err_xgb:  0.9906876133612811\n",
      "[0]\tvalidation_0-auc:0.970924\tvalidation_1-auc:0.966983\n",
      "Multiple eval metrics have been passed: 'validation_1-auc' will be used for early stopping.\n",
      "\n",
      "Will train until validation_1-auc hasn't improved in 200 rounds.\n",
      "[200]\tvalidation_0-auc:0.986653\tvalidation_1-auc:0.981828\n",
      "[400]\tvalidation_0-auc:0.992075\tvalidation_1-auc:0.984977\n",
      "[600]\tvalidation_0-auc:0.994624\tvalidation_1-auc:0.985932\n",
      "[800]\tvalidation_0-auc:0.996222\tvalidation_1-auc:0.986431\n",
      "[1000]\tvalidation_0-auc:0.99748\tvalidation_1-auc:0.987061\n",
      "[1200]\tvalidation_0-auc:0.998282\tvalidation_1-auc:0.987553\n",
      "[1400]\tvalidation_0-auc:0.998759\tvalidation_1-auc:0.988066\n",
      "[1600]\tvalidation_0-auc:0.999118\tvalidation_1-auc:0.988152\n",
      "[1800]\tvalidation_0-auc:0.999375\tvalidation_1-auc:0.988338\n",
      "[2000]\tvalidation_0-auc:0.999564\tvalidation_1-auc:0.988512\n",
      "[2200]\tvalidation_0-auc:0.999704\tvalidation_1-auc:0.988587\n",
      "Stopping. Best iteration:\n",
      "[2067]\tvalidation_0-auc:0.99961\tvalidation_1-auc:0.988608\n",
      "\n",
      "err_xgb:  0.9886083920087011\n"
     ]
    }
   ],
   "source": [
    "from xgboost import XGBClassifier\n",
    "\n",
    "errxgb=[]\n",
    "y_pred_tot_xgb=[]\n",
    "from sklearn.model_selection import KFold,StratifiedKFold\n",
    "fold=StratifiedKFold(n_splits=10,shuffle=True,random_state=1994)\n",
    "i=1\n",
    "for train_index, test_index in fold.split(X,y):\n",
    "    X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "    y_train, y_test = y[train_index], y[test_index]\n",
    "    m=XGBClassifier(n_estimators=5000,random_state=1994,eval_metric='auc',learning_rate=0.03)\n",
    "    m.fit(X_train,y_train,eval_set=[(X_train,y_train),(X_test, y_test)], early_stopping_rounds=200,verbose=200)\n",
    "    preds=m.predict_proba(X_test)[:,-1]\n",
    "    print(\"err_xgb: \",roc_auc_score(y_test,preds))\n",
    "    errxgb.append(roc_auc_score(y_test,preds))\n",
    "    p = m.predict_proba(Xtest)[:,-1]\n",
    "    i=i+1\n",
    "    y_pred_tot_xgb.append(p)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9885484031595695"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.mean(errxgb,0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.227403</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0.025819</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000053</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>0.000006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.000009</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  redemption_status\n",
       "0   3           0.227403\n",
       "1   4           0.025819\n",
       "2   5           0.000053\n",
       "3   8           0.000006\n",
       "4  10           0.000009"
      ]
     },
     "execution_count": 84,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s['redemption_status']=np.mean(y_pred_tot_xgb,0)\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(50226, 2)"
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s.to_csv('AV_amex_xgb_folds_v28.csv',index=False)\n",
    "s.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stacking & Ensemble"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>redemption_status</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.236035</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>0.037731</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>0.000563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>0.000013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.000044</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   id  redemption_status\n",
       "0   3           0.236035\n",
       "1   4           0.037731\n",
       "2   5           0.000563\n",
       "3   8           0.000013\n",
       "4  10           0.000044"
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s['redemption_status']=(np.mean(y_pred_tot_cb,0)+np.mean(y_pred_tot,0)+np.mean(y_pred_tot_xgb,0))/3\n",
    "s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "s.to_csv('AV_amex_stack3_folds_v28.csv',index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [],
   "source": [
    "# s['redemption_status']=np.mean(y_pred_tot_xgb,0)*0.5+np.mean(y_pred_tot,0)*0.5\n",
    "# s.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [],
   "source": [
    "# s.to_csv('AV_amex_stack4_folds_v17.csv',index=False)\n",
    "# s.shape"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}