Hyper Parameter Optimization Techniques.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### All Techniques Of Hyper Parameter Optimization\n",
    "\n",
    "1. GridSearchCV\n",
    "2. RandomizedSearchCV\n",
    "3. Bayesian Optimization -Automate Hyperparameter Tuning (Hyperopt)\n",
    "4. Sequential Model Based Optimization(Tuning a scikit-learn estimator with skopt)\n",
    "4. Optuna- Automate Hyperparameter Tuning\n",
    "5. Genetic Algorithms (TPOT Classifier)\n",
    "\n",
    "###### References\n",
    "- https://github.com/fmfn/BayesianOptimization\n",
    "- https://github.com/hyperopt/hyperopt\n",
    "- https://www.jeremyjordan.me/hyperparameter-tuning/\n",
    "- https://optuna.org/\n",
    "- https://towardsdatascience.com/hyperparameters-optimization-526348bb8e2d(By Pier Paolo Ippolito )\n",
    "- https://scikit-optimize.github.io/stable/auto_examples/hyperparameter-optimization.html\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import warnings\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "df=pd.read_csv('diabetes.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148.0</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85.0</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183.0</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89.0</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6    148.0             72             35        0  33.6   \n",
       "1            1     85.0             66             29        0  26.6   \n",
       "2            8    183.0             64              0        0  23.3   \n",
       "3            1     89.0             66             23       94  28.1   \n",
       "4            0    137.0             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "df['Glucose']=np.where(df['Glucose']==0,df['Glucose'].median(),df['Glucose'])\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Independent And Dependent features\n",
    "X=df.drop('Outcome',axis=1)\n",
    "y=df['Outcome']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148.0</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85.0</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183.0</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89.0</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137.0</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>763</th>\n",
       "      <td>10</td>\n",
       "      <td>101.0</td>\n",
       "      <td>76</td>\n",
       "      <td>48</td>\n",
       "      <td>180</td>\n",
       "      <td>32.9</td>\n",
       "      <td>0.171</td>\n",
       "      <td>63</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>764</th>\n",
       "      <td>2</td>\n",
       "      <td>122.0</td>\n",
       "      <td>70</td>\n",
       "      <td>27</td>\n",
       "      <td>0</td>\n",
       "      <td>36.8</td>\n",
       "      <td>0.340</td>\n",
       "      <td>27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>765</th>\n",
       "      <td>5</td>\n",
       "      <td>121.0</td>\n",
       "      <td>72</td>\n",
       "      <td>23</td>\n",
       "      <td>112</td>\n",
       "      <td>26.2</td>\n",
       "      <td>0.245</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>766</th>\n",
       "      <td>1</td>\n",
       "      <td>126.0</td>\n",
       "      <td>60</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>30.1</td>\n",
       "      <td>0.349</td>\n",
       "      <td>47</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>767</th>\n",
       "      <td>1</td>\n",
       "      <td>93.0</td>\n",
       "      <td>70</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "      <td>30.4</td>\n",
       "      <td>0.315</td>\n",
       "      <td>23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>768 rows × 8 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0              6    148.0             72             35        0  33.6   \n",
       "1              1     85.0             66             29        0  26.6   \n",
       "2              8    183.0             64              0        0  23.3   \n",
       "3              1     89.0             66             23       94  28.1   \n",
       "4              0    137.0             40             35      168  43.1   \n",
       "..           ...      ...            ...            ...      ...   ...   \n",
       "763           10    101.0             76             48      180  32.9   \n",
       "764            2    122.0             70             27        0  36.8   \n",
       "765            5    121.0             72             23      112  26.2   \n",
       "766            1    126.0             60              0        0  30.1   \n",
       "767            1     93.0             70             31        0  30.4   \n",
       "\n",
       "     DiabetesPedigreeFunction  Age  \n",
       "0                       0.627   50  \n",
       "1                       0.351   31  \n",
       "2                       0.672   32  \n",
       "3                       0.167   21  \n",
       "4                       2.288   33  \n",
       "..                        ...  ...  \n",
       "763                     0.171   63  \n",
       "764                     0.340   27  \n",
       "765                     0.245   30  \n",
       "766                     0.349   47  \n",
       "767                     0.315   23  \n",
       "\n",
       "[768 rows x 8 columns]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X,columns=df.columns[:-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "#### Train Test Split\n",
    "from sklearn.model_selection import train_test_split\n",
    "X_train,X_test,y_train,y_test=train_test_split(X,y,test_size=0.20,random_state=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rf_classifier=RandomForestClassifier(n_estimators=10).fit(X_train,y_train)\n",
    "prediction=rf_classifier.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    500\n",
       "1    268\n",
       "Name: Outcome, dtype: int64"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[94 13]\n",
      " [22 25]]\n",
      "0.7727272727272727\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.81      0.88      0.84       107\n",
      "           1       0.66      0.53      0.59        47\n",
      "\n",
      "    accuracy                           0.77       154\n",
      "   macro avg       0.73      0.71      0.72       154\n",
      "weighted avg       0.76      0.77      0.77       154\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix,classification_report,accuracy_score\n",
    "print(confusion_matrix(y_test,prediction))\n",
    "print(accuracy_score(y_test,prediction))\n",
    "print(classification_report(y_test,prediction))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main parameters used by a Random Forest Classifier are:\n",
    "\n",
    "- criterion = the function used to evaluate the quality of a split.\n",
    "- max_depth = maximum number of levels allowed in each tree.\n",
    "- max_features = maximum number of features considered when splitting a node.\n",
    "- min_samples_leaf = minimum number of samples which can be stored in a tree leaf.\n",
    "- min_samples_split = minimum number of samples necessary in a node to cause node splitting.\n",
    "- n_estimators = number of trees in the ensamble."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[98  9]\n",
      " [18 29]]\n",
      "0.8246753246753247\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.84      0.92      0.88       107\n",
      "           1       0.76      0.62      0.68        47\n",
      "\n",
      "    accuracy                           0.82       154\n",
      "   macro avg       0.80      0.77      0.78       154\n",
      "weighted avg       0.82      0.82      0.82       154\n",
      "\n"
     ]
    }
   ],
   "source": [
    "### Manual Hyperparameter Tuning\n",
    "model=RandomForestClassifier(n_estimators=300,criterion='entropy',\n",
    "                             max_features='sqrt',min_samples_leaf=10,random_state=100).fit(X_train,y_train)\n",
    "predictions=model.predict(X_test)\n",
    "print(confusion_matrix(y_test,predictions))\n",
    "print(accuracy_score(y_test,predictions))\n",
    "print(classification_report(y_test,predictions))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Randomized Search Cv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000], 'max_features': ['auto', 'sqrt', 'log2'], 'max_depth': [10, 120, 230, 340, 450, 560, 670, 780, 890, 1000], 'min_samples_split': [2, 5, 10, 14], 'min_samples_leaf': [1, 2, 4, 6, 8], 'criterion': ['entropy', 'gini']}\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "# Number of trees in random forest\n",
    "n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]\n",
    "# Number of features to consider at every split\n",
    "max_features = ['auto', 'sqrt','log2']\n",
    "# Maximum number of levels in tree\n",
    "max_depth = [int(x) for x in np.linspace(10, 1000,10)]\n",
    "# Minimum number of samples required to split a node\n",
    "min_samples_split = [2, 5, 10,14]\n",
    "# Minimum number of samples required at each leaf node\n",
    "min_samples_leaf = [1, 2, 4,6,8]\n",
    "# Create the random grid\n",
    "random_grid = {'n_estimators': n_estimators,\n",
    "               'max_features': max_features,\n",
    "               'max_depth': max_depth,\n",
    "               'min_samples_split': min_samples_split,\n",
    "               'min_samples_leaf': min_samples_leaf,\n",
    "              'criterion':['entropy','gini']}\n",
    "print(random_grid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 100 candidates, totalling 300 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  25 tasks      | elapsed:   27.9s\n",
      "[Parallel(n_jobs=-1)]: Done 146 tasks      | elapsed:  1.6min\n",
      "[Parallel(n_jobs=-1)]: Done 300 out of 300 | elapsed:  3.2min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=3, estimator=RandomForestClassifier(), n_iter=100,\n",
       "                   n_jobs=-1,\n",
       "                   param_distributions={'criterion': ['entropy', 'gini'],\n",
       "                                        'max_depth': [10, 120, 230, 340, 450,\n",
       "                                                      560, 670, 780, 890,\n",
       "                                                      1000],\n",
       "                                        'max_features': ['auto', 'sqrt',\n",
       "                                                         'log2'],\n",
       "                                        'min_samples_leaf': [1, 2, 4, 6, 8],\n",
       "                                        'min_samples_split': [2, 5, 10, 14],\n",
       "                                        'n_estimators': [200, 400, 600, 800,\n",
       "                                                         1000, 1200, 1400, 1600,\n",
       "                                                         1800, 2000]},\n",
       "                   random_state=100, verbose=2)"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf=RandomForestClassifier()\n",
    "rf_randomcv=RandomizedSearchCV(estimator=rf,param_distributions=random_grid,n_iter=100,cv=3,verbose=2,\n",
    "                               random_state=100,n_jobs=-1)\n",
    "### fit the randomized model\n",
    "rf_randomcv.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_estimators': 200,\n",
       " 'min_samples_split': 10,\n",
       " 'min_samples_leaf': 1,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_depth': 560,\n",
       " 'criterion': 'gini'}"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_randomcv.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=3, estimator=RandomForestClassifier(), n_iter=100,\n",
       "                   n_jobs=-1,\n",
       "                   param_distributions={'criterion': ['entropy', 'gini'],\n",
       "                                        'max_depth': [10, 120, 230, 340, 450,\n",
       "                                                      560, 670, 780, 890,\n",
       "                                                      1000],\n",
       "                                        'max_features': ['auto', 'sqrt',\n",
       "                                                         'log2'],\n",
       "                                        'min_samples_leaf': [1, 2, 4, 6, 8],\n",
       "                                        'min_samples_split': [2, 5, 10, 14],\n",
       "                                        'n_estimators': [200, 400, 600, 800,\n",
       "                                                         1000, 1200, 1400, 1600,\n",
       "                                                         1800, 2000]},\n",
       "                   random_state=100, verbose=2)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_randomcv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_random_grid=rf_randomcv.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[95 12]\n",
      " [15 32]]\n",
      "Accuracy Score 0.8246753246753247\n",
      "Classification report:               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.86      0.89      0.88       107\n",
      "           1       0.73      0.68      0.70        47\n",
      "\n",
      "    accuracy                           0.82       154\n",
      "   macro avg       0.80      0.78      0.79       154\n",
      "weighted avg       0.82      0.82      0.82       154\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "y_pred=best_random_grid.predict(X_test)\n",
    "print(confusion_matrix(y_test,y_pred))\n",
    "print(\"Accuracy Score {}\".format(accuracy_score(y_test,y_pred)))\n",
    "print(\"Classification report: {}\".format(classification_report(y_test,y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GridSearch CV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_estimators': 200,\n",
       " 'min_samples_split': 10,\n",
       " 'min_samples_leaf': 1,\n",
       " 'max_features': 'sqrt',\n",
       " 'max_depth': 560,\n",
       " 'criterion': 'gini'}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_randomcv.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'criterion': ['gini'], 'max_depth': [560], 'max_features': ['sqrt'], 'min_samples_leaf': [1, 3, 5], 'min_samples_split': [8, 9, 10, 11, 12], 'n_estimators': [0, 100, 200, 300, 400]}\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "param_grid = {\n",
    "    'criterion': [rf_randomcv.best_params_['criterion']],\n",
    "    'max_depth': [rf_randomcv.best_params_['max_depth']],\n",
    "    'max_features': [rf_randomcv.best_params_['max_features']],\n",
    "    'min_samples_leaf': [rf_randomcv.best_params_['min_samples_leaf'], \n",
    "                         rf_randomcv.best_params_['min_samples_leaf']+2, \n",
    "                         rf_randomcv.best_params_['min_samples_leaf'] + 4],\n",
    "    'min_samples_split': [rf_randomcv.best_params_['min_samples_split'] - 2,\n",
    "                          rf_randomcv.best_params_['min_samples_split'] - 1,\n",
    "                          rf_randomcv.best_params_['min_samples_split'], \n",
    "                          rf_randomcv.best_params_['min_samples_split'] +1,\n",
    "                          rf_randomcv.best_params_['min_samples_split'] + 2],\n",
    "    'n_estimators': [rf_randomcv.best_params_['n_estimators'] - 200, rf_randomcv.best_params_['n_estimators'] - 100, \n",
    "                     rf_randomcv.best_params_['n_estimators'], \n",
    "                     rf_randomcv.best_params_['n_estimators'] + 100, rf_randomcv.best_params_['n_estimators'] + 200]\n",
    "}\n",
    "\n",
    "print(param_grid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 10 folds for each of 75 candidates, totalling 750 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 8 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done  34 tasks      | elapsed:    4.7s\n",
      "[Parallel(n_jobs=-1)]: Done 170 tasks      | elapsed:   19.1s\n",
      "[Parallel(n_jobs=-1)]: Done 373 tasks      | elapsed:   40.6s\n",
      "[Parallel(n_jobs=-1)]: Done 656 tasks      | elapsed:  1.2min\n",
      "[Parallel(n_jobs=-1)]: Done 750 out of 750 | elapsed:  1.4min finished\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=10, estimator=RandomForestClassifier(), n_jobs=-1,\n",
       "             param_grid={'criterion': ['gini'], 'max_depth': [560],\n",
       "                         'max_features': ['sqrt'],\n",
       "                         'min_samples_leaf': [1, 3, 5],\n",
       "                         'min_samples_split': [8, 9, 10, 11, 12],\n",
       "                         'n_estimators': [0, 100, 200, 300, 400]},\n",
       "             verbose=2)"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#### Fit the grid_search to the data\n",
    "rf=RandomForestClassifier()\n",
    "grid_search=GridSearchCV(estimator=rf,param_grid=param_grid,cv=10,n_jobs=-1,verbose=2)\n",
    "grid_search.fit(X_train,y_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(max_depth=560, max_features='sqrt', min_samples_leaf=3,\n",
       "                       min_samples_split=10, n_estimators=300)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_grid=grid_search.best_estimator_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(max_depth=560, max_features='sqrt', min_samples_leaf=3,\n",
       "                       min_samples_split=10, n_estimators=300)"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best_grid"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[96 11]\n",
      " [16 31]]\n",
      "Accuracy Score 0.8246753246753247\n",
      "Classification report:               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.86      0.90      0.88       107\n",
      "           1       0.74      0.66      0.70        47\n",
      "\n",
      "    accuracy                           0.82       154\n",
      "   macro avg       0.80      0.78      0.79       154\n",
      "weighted avg       0.82      0.82      0.82       154\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred=best_grid.predict(X_test)\n",
    "print(confusion_matrix(y_test,y_pred))\n",
    "print(\"Accuracy Score {}\".format(accuracy_score(y_test,y_pred)))\n",
    "print(\"Classification report: {}\".format(classification_report(y_test,y_pred)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Automated Hyperparameter Tuning\n",
    "Automated Hyperparameter Tuning can be done by using techniques such as \n",
    "- Bayesian Optimization\n",
    "- Gradient Descent\n",
    "- Evolutionary Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bayesian Optimization\n",
    "Bayesian optimization uses probability to find the minimum of a function. The final aim is to find the input value to a function which can gives us the lowest possible output value.It usually performs better than random,grid and manual search providing better performance in the testing phase and reduced optimization time.\n",
    "In Hyperopt, Bayesian Optimization can be implemented giving 3 three main parameters to the function fmin.\n",
    "\n",
    "- Objective Function = defines the loss function to minimize.\n",
    "- Domain Space = defines the range of input values to test (in Bayesian Optimization this space creates a probability distribution for each of the used Hyperparameters).\n",
    "- Optimization Algorithm = defines the search algorithm to use to select the best input values to use in each new iteration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "from hyperopt import hp,fmin,tpe,STATUS_OK,Trials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "space = {'criterion': hp.choice('criterion', ['entropy', 'gini']),\n",
    "        'max_depth': hp.quniform('max_depth', 10, 1200, 10),\n",
    "        'max_features': hp.choice('max_features', ['auto', 'sqrt','log2', None]),\n",
    "        'min_samples_leaf': hp.uniform('min_samples_leaf', 0, 0.5),\n",
    "        'min_samples_split' : hp.uniform ('min_samples_split', 0, 1),\n",
    "        'n_estimators' : hp.choice('n_estimators', [10, 50, 300, 750, 1200,1300,1500])\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'criterion': <hyperopt.pyll.base.Apply at 0x20092b30c10>,\n",
       " 'max_depth': <hyperopt.pyll.base.Apply at 0x20092b30a90>,\n",
       " 'max_features': <hyperopt.pyll.base.Apply at 0x20092b30820>,\n",
       " 'min_samples_leaf': <hyperopt.pyll.base.Apply at 0x20092b30670>,\n",
       " 'min_samples_split': <hyperopt.pyll.base.Apply at 0x20092b30520>,\n",
       " 'n_estimators': <hyperopt.pyll.base.Apply at 0x20092b307f0>}"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def objective(space):\n",
    "    model = RandomForestClassifier(criterion = space['criterion'], max_depth = space['max_depth'],\n",
    "                                 max_features = space['max_features'],\n",
    "                                 min_samples_leaf = space['min_samples_leaf'],\n",
    "                                 min_samples_split = space['min_samples_split'],\n",
    "                                 n_estimators = space['n_estimators'], \n",
    "                                 )\n",
    "    \n",
    "    accuracy = cross_val_score(model, X_train, y_train, cv = 5).mean()\n",
    "\n",
    "    # We aim to maximize accuracy, therefore we return it as a negative value\n",
    "    return {'loss': -accuracy, 'status': STATUS_OK }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████| 80/80 [12:25<00:00,  9.32s/trial, best loss: -0.765480474476876]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'criterion': 0,\n",
       " 'max_depth': 420.0,\n",
       " 'max_features': 3,\n",
       " 'min_samples_leaf': 0.07909179100172778,\n",
       " 'min_samples_split': 0.0477920214214586,\n",
       " 'n_estimators': 2}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "trials = Trials()\n",
    "best = fmin(fn= objective,\n",
    "            space= space,\n",
    "            algo= tpe.suggest,\n",
    "            max_evals = 80,\n",
    "            trials= trials)\n",
    "best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "crit = {0: 'entropy', 1: 'gini'}\n",
    "feat = {0: 'auto', 1: 'sqrt', 2: 'log2', 3: None}\n",
    "est = {0: 10, 1: 50, 2: 300, 3: 750, 4: 1200,5:1300,6:1500}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "entropy\n",
      "None\n",
      "300\n"
     ]
    }
   ],
   "source": [
    "print(crit[best['criterion']])\n",
    "print(feat[best['max_features']])\n",
    "print(est[best['n_estimators']])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07909179100172778"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "best['min_samples_leaf']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[96 11]\n",
      " [23 24]]\n",
      "0.7792207792207793\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.81      0.90      0.85       107\n",
      "           1       0.69      0.51      0.59        47\n",
      "\n",
      "    accuracy                           0.78       154\n",
      "   macro avg       0.75      0.70      0.72       154\n",
      "weighted avg       0.77      0.78      0.77       154\n",
      "\n"
     ]
    }
   ],
   "source": [
    "trainedforest = RandomForestClassifier(criterion = crit[best['criterion']], max_depth = best['max_depth'], \n",
    "                                       max_features = feat[best['max_features']], \n",
    "                                       min_samples_leaf = best['min_samples_leaf'], \n",
    "                                       min_samples_split = best['min_samples_split'], \n",
    "                                       n_estimators = est[best['n_estimators']]).fit(X_train,y_train)\n",
    "predictionforest = trainedforest.predict(X_test)\n",
    "print(confusion_matrix(y_test,predictionforest))\n",
    "print(accuracy_score(y_test,predictionforest))\n",
    "print(classification_report(y_test,predictionforest))\n",
    "acc5 = accuracy_score(y_test,predictionforest)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Genetic Algorithms\n",
    "Genetic Algorithms tries to apply natural selection mechanisms to Machine Learning contexts.\n",
    "\n",
    "Let's immagine we create a population of N Machine Learning models with some predifined Hyperparameters. We can then calculate the accuracy of each model and decide to keep just half of the models (the ones that performs best). We can now generate some offsprings having similar Hyperparameters to the ones of the best models so that go get again a population of N models. At this point we can again caltulate the accuracy of each model and repeate the cycle for a defined number of generations. In this way, just the best models will survive at the end of the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000], 'max_features': ['auto', 'sqrt', 'log2'], 'max_depth': [10, 120, 230, 340, 450, 560, 670, 780, 890, 1000], 'min_samples_split': [2, 5, 10, 14], 'min_samples_leaf': [1, 2, 4, 6, 8], 'criterion': ['entropy', 'gini']}\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "# Number of trees in random forest\n",
    "n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]\n",
    "# Number of features to consider at every split\n",
    "max_features = ['auto', 'sqrt','log2']\n",
    "# Maximum number of levels in tree\n",
    "max_depth = [int(x) for x in np.linspace(10, 1000,10)]\n",
    "# Minimum number of samples required to split a node\n",
    "min_samples_split = [2, 5, 10,14]\n",
    "# Minimum number of samples required at each leaf node\n",
    "min_samples_leaf = [1, 2, 4,6,8]\n",
    "# Create the random grid\n",
    "param = {'n_estimators': n_estimators,\n",
    "               'max_features': max_features,\n",
    "               'max_depth': max_depth,\n",
    "               'min_samples_split': min_samples_split,\n",
    "               'min_samples_leaf': min_samples_leaf,\n",
    "              'criterion':['entropy','gini']}\n",
    "print(param)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, description='Optimization Progress', max=84.0, style=ProgressStyle(des…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Generation 1 - Current best internal CV score: 0.7589975384093031\n",
      "\n",
      "Generation 2 - Current best internal CV score: 0.7589975384093031\n",
      "\n",
      "Generation 3 - Current best internal CV score: 0.760557253204312\n",
      "\n",
      "Generation 4 - Current best internal CV score: 0.760557253204312\n",
      "\n",
      "Generation 5 - Current best internal CV score: 0.760557253204312\n",
      "\n",
      "Best pipeline: RandomForestClassifier(RandomForestClassifier(CombineDFs(input_matrix, input_matrix), criterion=gini, max_depth=340, max_features=log2, min_samples_leaf=8, min_samples_split=5, n_estimators=400), criterion=gini, max_depth=890, max_features=sqrt, min_samples_leaf=2, min_samples_split=14, n_estimators=600)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "TPOTClassifier(config_dict={'sklearn.ensemble.RandomForestClassifier': {'criterion': ['entropy',\n",
       "                                                                                      'gini'],\n",
       "                                                                        'max_depth': [10,\n",
       "                                                                                      120,\n",
       "                                                                                      230,\n",
       "                                                                                      340,\n",
       "                                                                                      450,\n",
       "                                                                                      560,\n",
       "                                                                                      670,\n",
       "                                                                                      780,\n",
       "                                                                                      890,\n",
       "                                                                                      1000],\n",
       "                                                                        'max_features': ['auto',\n",
       "                                                                                         'sqrt',\n",
       "                                                                                         'log2'],\n",
       "                                                                        'min_samples_leaf': [1,\n",
       "                                                                                             2,\n",
       "                                                                                             4,\n",
       "                                                                                             6,\n",
       "                                                                                             8],\n",
       "                                                                        'min_samples_split': [2,\n",
       "                                                                                              5,\n",
       "                                                                                              10,\n",
       "                                                                                              14],\n",
       "                                                                        'n_estimators': [200,\n",
       "                                                                                         400,\n",
       "                                                                                         600,\n",
       "                                                                                         800,\n",
       "                                                                                         1000,\n",
       "                                                                                         1200,\n",
       "                                                                                         1400,\n",
       "                                                                                         1600,\n",
       "                                                                                         1800,\n",
       "                                                                                         2000]}},\n",
       "               cv=4, early_stop=12, generations=5, offspring_size=12,\n",
       "               population_size=24, scoring='accuracy', verbosity=2)"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "from tpot import TPOTClassifier\n",
    "\n",
    "\n",
    "tpot_classifier = TPOTClassifier(generations= 5, population_size= 24, offspring_size= 12,\n",
    "                                 verbosity= 2, early_stop= 12,\n",
    "                                 config_dict={'sklearn.ensemble.RandomForestClassifier': param}, \n",
    "                                 cv = 4, scoring = 'accuracy')\n",
    "tpot_classifier.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.8311688311688312\n"
     ]
    }
   ],
   "source": [
    "\n",
    "accuracy = tpot_classifier.score(X_test, y_test)\n",
    "print(accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optimize hyperparameters of the model using Optuna\n",
    "\n",
    "The hyperparameters of the above algorithm are `n_estimators` and `max_depth` for which we can try different values to see if the model accuracy can be improved. The `objective` function is modified to accept a trial object. This trial has several methods for sampling hyperparameters. We create a study to run the hyperparameter optimization and finally read the best hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "import optuna\n",
    "import sklearn.svm\n",
    "def objective(trial):\n",
    "\n",
    "    classifier = trial.suggest_categorical('classifier', ['RandomForest', 'SVC'])\n",
    "    \n",
    "    if classifier == 'RandomForest':\n",
    "        n_estimators = trial.suggest_int('n_estimators', 200, 2000,10)\n",
    "        max_depth = int(trial.suggest_float('max_depth', 10, 100, log=True))\n",
    "\n",
    "        clf = sklearn.ensemble.RandomForestClassifier(\n",
    "            n_estimators=n_estimators, max_depth=max_depth)\n",
    "    else:\n",
    "        c = trial.suggest_float('svc_c', 1e-10, 1e10, log=True)\n",
    "        \n",
    "        clf = sklearn.svm.SVC(C=c, gamma='auto')\n",
    "\n",
    "    return sklearn.model_selection.cross_val_score(\n",
    "        clf,X_train,y_train, n_jobs=-1, cv=3).mean()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2020-12-15 15:38:41,648]\u001b[0m A new study created in memory with name: no-name-8c38ec7c-12c6-43f6-bef8-c3e1cebc3a65\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:38:47,068]\u001b[0m Trial 0 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 0.7601918583457238}. Best is trial 0 with value: 0.640068547744301.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:38:50,934]\u001b[0m Trial 1 finished with value: 0.7426749561613263 and parameters: {'classifier': 'RandomForest', 'n_estimators': 340, 'max_depth': 40.629200632832166}. Best is trial 1 with value: 0.7426749561613263.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:38:54,708]\u001b[0m Trial 2 finished with value: 0.7556830862426271 and parameters: {'classifier': 'RandomForest', 'n_estimators': 400, 'max_depth': 80.81002123438643}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:38:54,803]\u001b[0m Trial 3 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 6.569777587644741e-10}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:38:54,875]\u001b[0m Trial 4 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 367211529.14856416}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:38:56,015]\u001b[0m Trial 5 finished with value: 0.7459269886816515 and parameters: {'classifier': 'RandomForest', 'n_estimators': 300, 'max_depth': 10.690716908750247}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:38:56,103]\u001b[0m Trial 6 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 1494028.0622470523}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:38:56,183]\u001b[0m Trial 7 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 729024298.0003759}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:01,291]\u001b[0m Trial 8 finished with value: 0.7491790212019768 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1420, 'max_depth': 10.643573688227363}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:08,154]\u001b[0m Trial 9 finished with value: 0.7524390243902439 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1910, 'max_depth': 49.535112510639095}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:10,901]\u001b[0m Trial 10 finished with value: 0.7426590148254424 and parameters: {'classifier': 'RandomForest', 'n_estimators': 780, 'max_depth': 96.13169493718797}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:17,849]\u001b[0m Trial 11 finished with value: 0.7491790212019768 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1980, 'max_depth': 70.67261760616597}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:24,686]\u001b[0m Trial 12 finished with value: 0.7475530049418141 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1960, 'max_depth': 48.125653692136964}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:29,055]\u001b[0m Trial 13 finished with value: 0.7540570699824646 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1270, 'max_depth': 21.420433677619148}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:32,973]\u001b[0m Trial 14 finished with value: 0.7475609756097561 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1140, 'max_depth': 21.425232959935652}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:36,158]\u001b[0m Trial 15 finished with value: 0.752415112386418 and parameters: {'classifier': 'RandomForest', 'n_estimators': 870, 'max_depth': 20.386962421616637}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:41,049]\u001b[0m Trial 16 finished with value: 0.75242308305436 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1450, 'max_depth': 21.726352736895272}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:43,305]\u001b[0m Trial 17 finished with value: 0.7377889367128966 and parameters: {'classifier': 'RandomForest', 'n_estimators': 640, 'max_depth': 15.197961062313556}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:47,898]\u001b[0m Trial 18 finished with value: 0.7475609756097561 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1380, 'max_depth': 30.934346088514907}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:51,577]\u001b[0m Trial 19 finished with value: 0.7556751155746851 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1100, 'max_depth': 93.65774558693442}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:53,377]\u001b[0m Trial 20 finished with value: 0.7507970667941973 and parameters: {'classifier': 'RandomForest', 'n_estimators': 530, 'max_depth': 97.0331618536863}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:39:57,221]\u001b[0m Trial 21 finished with value: 0.7475530049418141 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1110, 'max_depth': 71.73281458662053}. Best is trial 2 with value: 0.7556830862426271.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:01,237]\u001b[0m Trial 22 finished with value: 0.7573170731707317 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1250, 'max_depth': 70.27658246744014}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:06,954]\u001b[0m Trial 23 finished with value: 0.7524390243902439 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1680, 'max_depth': 73.03028579999658}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:10,115]\u001b[0m Trial 24 finished with value: 0.744293001753547 and parameters: {'classifier': 'RandomForest', 'n_estimators': 930, 'max_depth': 97.94058561662146}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:15,796]\u001b[0m Trial 25 finished with value: 0.7410329985652798 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1640, 'max_depth': 58.37261743027528}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:19,190]\u001b[0m Trial 26 finished with value: 0.7426749561613263 and parameters: {'classifier': 'RandomForest', 'n_estimators': 980, 'max_depth': 82.33188962016798}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:23,343]\u001b[0m Trial 27 finished with value: 0.7491869918699187 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1210, 'max_depth': 58.25759933896509}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:28,770]\u001b[0m Trial 28 finished with value: 0.744293001753547 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1590, 'max_depth': 84.08326127610971}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:28,841]\u001b[0m Trial 29 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 1.6405987386989474e-10}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:31,224]\u001b[0m Trial 30 finished with value: 0.7556830862426271 and parameters: {'classifier': 'RandomForest', 'n_estimators': 650, 'max_depth': 35.32973739383278}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:33,044]\u001b[0m Trial 31 finished with value: 0.7410409692332217 and parameters: {'classifier': 'RandomForest', 'n_estimators': 520, 'max_depth': 29.344194769291867}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:33,788]\u001b[0m Trial 32 finished with value: 0.7426510441575004 and parameters: {'classifier': 'RandomForest', 'n_estimators': 230, 'max_depth': 58.91146699508621}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:35,968]\u001b[0m Trial 33 finished with value: 0.744293001753547 and parameters: {'classifier': 'RandomForest', 'n_estimators': 700, 'max_depth': 38.582576788458645}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:37,450]\u001b[0m Trial 34 finished with value: 0.7573170731707317 and parameters: {'classifier': 'RandomForest', 'n_estimators': 450, 'max_depth': 83.49782634419522}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2020-12-15 15:40:38,872]\u001b[0m Trial 35 finished with value: 0.7508130081300813 and parameters: {'classifier': 'RandomForest', 'n_estimators': 430, 'max_depth': 66.51820289481061}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:38,946]\u001b[0m Trial 36 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 0.0013161542437025479}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:39,640]\u001b[0m Trial 37 finished with value: 0.7508130081300813 and parameters: {'classifier': 'RandomForest', 'n_estimators': 200, 'max_depth': 38.64604823517601}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:39,699]\u001b[0m Trial 38 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 7.287305886373373e-06}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:41,038]\u001b[0m Trial 39 finished with value: 0.7459190180137095 and parameters: {'classifier': 'RandomForest', 'n_estimators': 400, 'max_depth': 48.77913171457546}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:42,977]\u001b[0m Trial 40 finished with value: 0.75242308305436 and parameters: {'classifier': 'RandomForest', 'n_estimators': 560, 'max_depth': 83.04204032890499}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:46,400]\u001b[0m Trial 41 finished with value: 0.7442770604176631 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1010, 'max_depth': 98.54745971627327}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:47,665]\u001b[0m Trial 42 finished with value: 0.7426430734895585 and parameters: {'classifier': 'RandomForest', 'n_estimators': 360, 'max_depth': 86.1848142901668}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:50,244]\u001b[0m Trial 43 finished with value: 0.7507970667941973 and parameters: {'classifier': 'RandomForest', 'n_estimators': 770, 'max_depth': 62.89931121310966}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:54,769]\u001b[0m Trial 44 finished with value: 0.7524390243902439 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1290, 'max_depth': 77.77493643799801}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:55,690]\u001b[0m Trial 45 finished with value: 0.7556910569105691 and parameters: {'classifier': 'RandomForest', 'n_estimators': 250, 'max_depth': 26.106054818818276}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:56,725]\u001b[0m Trial 46 finished with value: 0.7491630798660928 and parameters: {'classifier': 'RandomForest', 'n_estimators': 290, 'max_depth': 16.35657683992267}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:56,791]\u001b[0m Trial 47 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 7606.536920307913}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:57,498]\u001b[0m Trial 48 finished with value: 0.7556671449067432 and parameters: {'classifier': 'RandomForest', 'n_estimators': 200, 'max_depth': 24.964231389798414}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:40:59,145]\u001b[0m Trial 49 finished with value: 0.75242308305436 and parameters: {'classifier': 'RandomForest', 'n_estimators': 490, 'max_depth': 34.06057734628813}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:01,321]\u001b[0m Trial 50 finished with value: 0.7475530049418141 and parameters: {'classifier': 'RandomForest', 'n_estimators': 610, 'max_depth': 25.653584407309616}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:02,436]\u001b[0m Trial 51 finished with value: 0.7524310537223019 and parameters: {'classifier': 'RandomForest', 'n_estimators': 320, 'max_depth': 43.564470152119995}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:06,040]\u001b[0m Trial 52 finished with value: 0.7508050374621393 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1040, 'max_depth': 89.99635512588058}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:10,251]\u001b[0m Trial 53 finished with value: 0.7491790212019768 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1170, 'max_depth': 17.807855108278165}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:13,455]\u001b[0m Trial 54 finished with value: 0.7475450342738722 and parameters: {'classifier': 'RandomForest', 'n_estimators': 870, 'max_depth': 26.140289467530334}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:18,856]\u001b[0m Trial 55 finished with value: 0.7540490993145226 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1320, 'max_depth': 11.26017100851049}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:24,717]\u001b[0m Trial 56 finished with value: 0.7540650406504065 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1460, 'max_depth': 54.03954664256802}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:26,388]\u001b[0m Trial 57 finished with value: 0.7459269886816515 and parameters: {'classifier': 'RandomForest', 'n_estimators': 470, 'max_depth': 33.73272218056058}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:27,526]\u001b[0m Trial 58 finished with value: 0.7507970667941973 and parameters: {'classifier': 'RandomForest', 'n_estimators': 280, 'max_depth': 71.8743165886551}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:30,331]\u001b[0m Trial 59 finished with value: 0.7459269886816515 and parameters: {'classifier': 'RandomForest', 'n_estimators': 720, 'max_depth': 43.44557956222402}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:31,813]\u001b[0m Trial 60 finished with value: 0.7540650406504065 and parameters: {'classifier': 'RandomForest', 'n_estimators': 380, 'max_depth': 92.81976327195814}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:32,663]\u001b[0m Trial 61 finished with value: 0.7459190180137095 and parameters: {'classifier': 'RandomForest', 'n_estimators': 200, 'max_depth': 24.015686462162897}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:36,493]\u001b[0m Trial 62 finished with value: 0.7459110473457676 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1080, 'max_depth': 28.01977971995723}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:37,356]\u001b[0m Trial 63 finished with value: 0.7540411286465806 and parameters: {'classifier': 'RandomForest', 'n_estimators': 240, 'max_depth': 19.174335607884405}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:39,619]\u001b[0m Trial 64 finished with value: 0.7475530049418141 and parameters: {'classifier': 'RandomForest', 'n_estimators': 610, 'max_depth': 22.511052535654297}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:43,925]\u001b[0m Trial 65 finished with value: 0.7573170731707317 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1190, 'max_depth': 13.188526536315694}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:48,104]\u001b[0m Trial 66 finished with value: 0.7524390243902439 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1200, 'max_depth': 77.3418682799883}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:53,418]\u001b[0m Trial 67 finished with value: 0.7491710505340348 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1250, 'max_depth': 12.499704656059338}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:41:59,915]\u001b[0m Trial 68 finished with value: 0.7556830862426271 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1340, 'max_depth': 12.846823339250241}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:05,829]\u001b[0m Trial 69 finished with value: 0.752415112386418 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1380, 'max_depth': 13.383030676146}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\u001b[32m[I 2020-12-15 15:42:05,917]\u001b[0m Trial 70 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 2283.1551439306886}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:11,215]\u001b[0m Trial 71 finished with value: 0.7524390243902439 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1510, 'max_depth': 10.26691959439627}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:15,403]\u001b[0m Trial 72 finished with value: 0.7507970667941973 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1130, 'max_depth': 13.237596067027456}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:20,409]\u001b[0m Trial 73 finished with value: 0.744285031085605 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1350, 'max_depth': 15.357294581658595}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:23,787]\u001b[0m Trial 74 finished with value: 0.7508050374621393 and parameters: {'classifier': 'RandomForest', 'n_estimators': 930, 'max_depth': 11.347234402202993}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:28,539]\u001b[0m Trial 75 finished with value: 0.7573091025027897 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1270, 'max_depth': 99.81232944607433}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:33,501]\u001b[0m Trial 76 finished with value: 0.7410409692332217 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1260, 'max_depth': 66.04431837515774}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:39,149]\u001b[0m Trial 77 finished with value: 0.7540730113183485 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1550, 'max_depth': 77.78202093049175}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:45,506]\u001b[0m Trial 78 finished with value: 0.7459190180137095 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1420, 'max_depth': 54.47330486537615}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:50,094]\u001b[0m Trial 79 finished with value: 0.755699027578511 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1230, 'max_depth': 14.464044165516349}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:54,491]\u001b[0m Trial 80 finished with value: 0.7556671449067432 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1170, 'max_depth': 14.331504193084921}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:42:58,987]\u001b[0m Trial 81 finished with value: 0.7491630798660928 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1220, 'max_depth': 11.948758596813715}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:05,654]\u001b[0m Trial 82 finished with value: 0.7475450342738722 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1320, 'max_depth': 14.065164039769755}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:10,834]\u001b[0m Trial 83 finished with value: 0.7475530049418141 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1080, 'max_depth': 16.951999571745137}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:18,898]\u001b[0m Trial 84 finished with value: 0.7507970667941973 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1780, 'max_depth': 88.4923388332337}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:20,573]\u001b[0m Trial 85 finished with value: 0.7459110473457676 and parameters: {'classifier': 'RandomForest', 'n_estimators': 430, 'max_depth': 10.055473764343684}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:25,470]\u001b[0m Trial 86 finished with value: 0.7475530049418141 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1380, 'max_depth': 19.134624989262054}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:30,115]\u001b[0m Trial 87 finished with value: 0.7524390243902439 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1290, 'max_depth': 15.564899145888445}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:30,223]\u001b[0m Trial 88 finished with value: 0.640068547744301 and parameters: {'classifier': 'SVC', 'svc_c': 7.106632792646168e-07}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:31,451]\u001b[0m Trial 89 finished with value: 0.7508209787980232 and parameters: {'classifier': 'RandomForest', 'n_estimators': 340, 'max_depth': 97.78924638366236}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:36,585]\u001b[0m Trial 90 finished with value: 0.7491710505340348 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1480, 'max_depth': 12.35330654124254}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:40,369]\u001b[0m Trial 91 finished with value: 0.7540730113183485 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1140, 'max_depth': 80.99473822522634}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:44,049]\u001b[0m Trial 92 finished with value: 0.7540570699824646 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1030, 'max_depth': 92.86258896277278}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:48,305]\u001b[0m Trial 93 finished with value: 0.744293001753547 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1220, 'max_depth': 68.47590482043174}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:52,347]\u001b[0m Trial 94 finished with value: 0.7426669854933844 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1170, 'max_depth': 99.92763829265347}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:56,132]\u001b[0m Trial 95 finished with value: 0.7524469950581859 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1080, 'max_depth': 78.34459426342944}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:43:59,480]\u001b[0m Trial 96 finished with value: 0.7475450342738722 and parameters: {'classifier': 'RandomForest', 'n_estimators': 970, 'max_depth': 85.52317279846142}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:44:03,869]\u001b[0m Trial 97 finished with value: 0.7459269886816515 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1310, 'max_depth': 61.97518901196247}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:44:06,593]\u001b[0m Trial 98 finished with value: 0.7508050374621393 and parameters: {'classifier': 'RandomForest', 'n_estimators': 840, 'max_depth': 11.100301397689092}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n",
      "\u001b[32m[I 2020-12-15 15:44:10,866]\u001b[0m Trial 99 finished with value: 0.7540570699824646 and parameters: {'classifier': 'RandomForest', 'n_estimators': 1250, 'max_depth': 93.8486261846434}. Best is trial 22 with value: 0.7573170731707317.\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.7573170731707317\n",
      "Best hyperparameters: {'classifier': 'RandomForest', 'n_estimators': 1250, 'max_depth': 70.27658246744014}\n"
     ]
    }
   ],
   "source": [
    "study = optuna.create_study(direction='maximize')\n",
    "study.optimize(objective, n_trials=100)\n",
    "\n",
    "trial = study.best_trial\n",
    "\n",
    "print('Accuracy: {}'.format(trial.value))\n",
    "print(\"Best hyperparameters: {}\".format(trial.params))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "FrozenTrial(number=22, value=0.7573170731707317, datetime_start=datetime.datetime(2020, 12, 15, 15, 39, 57, 226329), datetime_complete=datetime.datetime(2020, 12, 15, 15, 40, 1, 237671), params={'classifier': 'RandomForest', 'n_estimators': 1250, 'max_depth': 70.27658246744014}, distributions={'classifier': CategoricalDistribution(choices=('RandomForest', 'SVC')), 'n_estimators': IntUniformDistribution(high=2000, low=200, step=10), 'max_depth': LogUniformDistribution(high=100, low=10)}, user_attrs={}, system_attrs={}, intermediate_values={}, trial_id=22, state=TrialState.COMPLETE)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'classifier': 'RandomForest',\n",
       " 'n_estimators': 1250,\n",
       " 'max_depth': 70.27658246744014}"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "study.best_params"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "RandomForestClassifier(max_depth=30, n_estimators=330)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf=RandomForestClassifier(n_estimators=330,max_depth=30)\n",
    "rf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[94 13]\n",
      " [16 31]]\n",
      "0.8116883116883117\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.85      0.88      0.87       107\n",
      "           1       0.70      0.66      0.68        47\n",
      "\n",
      "    accuracy                           0.81       154\n",
      "   macro avg       0.78      0.77      0.77       154\n",
      "weighted avg       0.81      0.81      0.81       154\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_pred=rf.predict(X_test)\n",
    "print(confusion_matrix(y_test,y_pred))\n",
    "print(accuracy_score(y_test,y_pred))\n",
    "print(classification_report(y_test,y_pred))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}