7-Hyperparameters Tuning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4e1af8c7",
   "metadata": {},
   "source": [
    "# Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06547e47",
   "metadata": {},
   "source": [
    "## Overview"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ccb75e25",
   "metadata": {},
   "source": [
    "in machine learning, we have model parameters and hyperparameters. **Model Parameters** are the parameters that are estimated by the model (if the model is parametric) from the given data. the **Model hyperparameters** are the parameters that the model used to estimate the model parameters. so each model have its hyperparameters, and different hyperparameter will give us different answers, so we have to find the best hyperparameters to get the best accuracy.so **Hyperparameter Tuning** (or hyperparameter optimization) is the process of determining the right combination of hyperparameters in order to maximize the model performance. it works by running multiple trials in a single training process. it is an important step in any ML project since it leads to optimal results for a model. but how to find these hyperparameters?<br>\n",
    "in general, we need another dataset besides the training and test set, which is called the **Cross Validation** or **Development** set. so after choosing our hyperparameters and training our model using training data, we calculate our model performance using the development set (dev set) and keep changing the hyperparameters until we reach our optimal performance on the dev set. then choose those hyperparameters and train our model another time on the training set and evaluate its final accuracy using the test set. by using the dev set we prevent data leakage from the test set to our training process, so we get a more realistic performance number. so before we talk about methods of hyperparameters tuning, let's talk about a method that helps us perform cross validation, without the need for a separate dataset as the dev set, and just using our usual training and test dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2ecc8e0",
   "metadata": {},
   "source": [
    "## K-Fold Cross Validation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0d0c29d",
   "metadata": {},
   "source": [
    "in **K-Fold Cross Validation** we are using training set both as training and dev set. we split the training set into K equal parts and train our model k time on k-1 of those parts with fixed hyperparameters and calculate our performance on the remaining part. at the end we will have k performance numbers, so we use average of those numbers as our final model accuracy on dev set correspond to those hyperparameters. then we change our hyperparameters and repeat this process and so on.<br>\n",
    "\n",
    "**NOTE:** this method preferable when we don't have big dataset, and by splitting our data into train/dev/test set we will have small data for training our model. so in other words, we somehow train our data on more data.<br>\n",
    "\n",
    "**NOTE:** for big datasets, we won't use this method and use train/dev/test set split with portion of 0.98/0.01/0.01 (or something like this).<br>\n",
    "\n",
    "let's see how we can perform K-Fold cross validation by using scikit learn."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e408c6c",
   "metadata": {},
   "source": [
    "### Implementing K-Fold cross validation in scikit learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "0e8a3596",
   "metadata": {},
   "outputs": [],
   "source": [
    "# all the libraries we need through out the entire notebook\n",
    "import numpy as np\n",
    "from sklearn.experimental import enable_halving_search_cv\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.ensemble import RandomForestClassifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "062072be",
   "metadata": {},
   "outputs": [],
   "source": [
    "# our breast cancer dataset- so we just split our data into train and test set\n",
    "X,y = load_breast_cancer(return_X_y=True)\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42,stratify=y)\n",
    "scale = StandardScaler()\n",
    "scale.fit(X_train)\n",
    "X_train = scale.transform(X_train)\n",
    "X_test = scale.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d1b4531b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([0.93478261, 1.        , 1.        , 0.97826087, 0.93478261,\n",
       "        1.        , 1.        , 1.        , 1.        , 0.97777778]),\n",
       " 0.9825603864734299)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import cross_val_score\n",
    "lr = LogisticRegression(penalty='l2',C=1) # using logistic regression with these hyperparameters to model our data\n",
    "lr_cv = cross_val_score(lr,X_train,y_train,cv=10) # cv : number of split (our k)\n",
    "accuracy = np.mean(lr_cv)\n",
    "lr_cv,accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12553946",
   "metadata": {},
   "source": [
    "as you can see we have 10 (equal to our k) calculated accuracy, so our final accuracy using these hyperparameters will be average of these numbers.<br>\n",
    "now that we know how to calculate the accuracy of our model using each of these hyperparameters, lets talk about methods that can help us find the optimum hyperparameters."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61b6704c",
   "metadata": {},
   "source": [
    "## Hyperparameter Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81f2ce10",
   "metadata": {},
   "source": [
    "in general, there are two types of hyperparameter tuning:<br>\n",
    "   1. **Manual** hyperparameter tuning\n",
    "   2. **Automated** hyperparameter tuning<br>\n",
    "\n",
    "in manual HP tuning, as you can gauss, we pick hyperparameters by hand and test them one by one. in this method we have more control over the process but tuning usually consist of many trials so keeping track of them is hard and time-consuming, so this isn't a very practical approach when there are a lot of hyperparameters to consider (and that's usually the case).<br>\n",
    "in automated HP tuning, we are using algorithms to do these processes and find the optimum HPs. in these algorithms generally we have two steps:<br>\n",
    "   * Step 1: we have to specify a set of hyperparameters and their limits.in other words, we somehow specify a grid of search for our algorithm.\n",
    "   * Step 2: then the algorithm does the rest. it runs these trials (train our model with these hyperparameters) and find the best hyperparameters that give us optimum results.<br>\n",
    "\n",
    "how do the algorithms do this? well, there are a lot of classes of algorithms with many variations, and the advanced ones are completely related to the field of optimization because, in the end, it is an optimization problem. here I will talk about 4 of these methods and I think knowing those are enough! "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "97b0ffef",
   "metadata": {},
   "source": [
    "### GridSearch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15c92f8f",
   "metadata": {},
   "source": [
    "in **GridSearch** we define grid of possible values for hyperparameters, and the algorithm will perform the model on each and every possible combination and record the model performance. finally it returns the best model with the best hyperparameters. let's implement it with scikit learn."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27511472",
   "metadata": {},
   "source": [
    "#### Implementing GridSearch with scikit learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8b2495f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "# our models that we want to perform on our dataset, we will use them through entire notebook.\n",
    "# as you can see i fixed some of hyperparameters but in general you can search over them too\n",
    "\n",
    "lr = LogisticRegression(max_iter=1000,penalty='l2',random_state=42) #Logistic Regression\n",
    "rf = RandomForestClassifier(random_state=42,n_jobs=-1)# Random forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "441d233e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# our grid of hyperparameters\n",
    "lr_params = {'C': np.logspace(4,-4,30),\n",
    "             'solver':['lbfgs','liblinear']\n",
    "             \n",
    "}\n",
    "\n",
    "rf_params = {'n_estimators':[200,400,600,800,1000],\n",
    "             'criterion':['gini','entropy'],\n",
    "             'bootstrap':[True,False],\n",
    "             'max_depth':[3,7,9,11,13,None]\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c67aa58e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 391 ms\n",
      "Wall time: 2.85 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=LogisticRegression(max_iter=1000, random_state=42),\n",
       "             n_jobs=-1,\n",
       "             param_grid={&#x27;C&#x27;: array([1.00000000e+04, 5.29831691e+03, 2.80721620e+03, 1.48735211e+03,\n",
       "       7.88046282e+02, 4.17531894e+02, 2.21221629e+02, 1.17210230e+02,\n",
       "       6.21016942e+01, 3.29034456e+01, 1.74332882e+01, 9.23670857e+00,\n",
       "       4.89390092e+00, 2.59294380e+00, 1.37382380e+00, 7.27895384e-01,\n",
       "       3.85662042e-01, 2.04335972e-01, 1.08263673e-01, 5.73615251e-02,\n",
       "       3.03919538e-02, 1.61026203e-02, 8.53167852e-03, 4.52035366e-03,\n",
       "       2.39502662e-03, 1.26896100e-03, 6.72335754e-04, 3.56224789e-04,\n",
       "       1.88739182e-04, 1.00000000e-04]),\n",
       "                         &#x27;solver&#x27;: [&#x27;lbfgs&#x27;, &#x27;liblinear&#x27;]})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=LogisticRegression(max_iter=1000, random_state=42),\n",
       "             n_jobs=-1,\n",
       "             param_grid={&#x27;C&#x27;: array([1.00000000e+04, 5.29831691e+03, 2.80721620e+03, 1.48735211e+03,\n",
       "       7.88046282e+02, 4.17531894e+02, 2.21221629e+02, 1.17210230e+02,\n",
       "       6.21016942e+01, 3.29034456e+01, 1.74332882e+01, 9.23670857e+00,\n",
       "       4.89390092e+00, 2.59294380e+00, 1.37382380e+00, 7.27895384e-01,\n",
       "       3.85662042e-01, 2.04335972e-01, 1.08263673e-01, 5.73615251e-02,\n",
       "       3.03919538e-02, 1.61026203e-02, 8.53167852e-03, 4.52035366e-03,\n",
       "       2.39502662e-03, 1.26896100e-03, 6.72335754e-04, 3.56224789e-04,\n",
       "       1.88739182e-04, 1.00000000e-04]),\n",
       "                         &#x27;solver&#x27;: [&#x27;lbfgs&#x27;, &#x27;liblinear&#x27;]})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000, random_state=42)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000, random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=10,\n",
       "             estimator=LogisticRegression(max_iter=1000, random_state=42),\n",
       "             n_jobs=-1,\n",
       "             param_grid={'C': array([1.00000000e+04, 5.29831691e+03, 2.80721620e+03, 1.48735211e+03,\n",
       "       7.88046282e+02, 4.17531894e+02, 2.21221629e+02, 1.17210230e+02,\n",
       "       6.21016942e+01, 3.29034456e+01, 1.74332882e+01, 9.23670857e+00,\n",
       "       4.89390092e+00, 2.59294380e+00, 1.37382380e+00, 7.27895384e-01,\n",
       "       3.85662042e-01, 2.04335972e-01, 1.08263673e-01, 5.73615251e-02,\n",
       "       3.03919538e-02, 1.61026203e-02, 8.53167852e-03, 4.52035366e-03,\n",
       "       2.39502662e-03, 1.26896100e-03, 6.72335754e-04, 3.56224789e-04,\n",
       "       1.88739182e-04, 1.00000000e-04]),\n",
       "                         'solver': ['lbfgs', 'liblinear']})"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "#logistic regression\n",
    "lr_g_clf = GridSearchCV(lr,param_grid=lr_params,cv=10,n_jobs=-1)\n",
    "lr_g_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a62737a0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'C': 0.7278953843983146, 'solver': 'liblinear'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# our best parameters\n",
    "lr_g_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "bea31e77",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9825603864734299"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# score of our best parameters\n",
    "lr_g_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "34410ebc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9824561403508771"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# our model with best hyperparameters. now we test our model with chosen hyperparameter on our test dataset.\n",
    "lr_g_clf.best_estimator_.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d826b193",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "158faf4e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 3.58 s\n",
      "Wall time: 4min 47s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=RandomForestClassifier(n_jobs=-1, random_state=42),\n",
       "             n_jobs=-1,\n",
       "             param_grid={&#x27;bootstrap&#x27;: [True, False],\n",
       "                         &#x27;criterion&#x27;: [&#x27;gini&#x27;, &#x27;entropy&#x27;],\n",
       "                         &#x27;max_depth&#x27;: [3, 7, 9, 11, 13, None],\n",
       "                         &#x27;n_estimators&#x27;: [200, 400, 600, 800, 1000]})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">GridSearchCV</label><div class=\"sk-toggleable__content\"><pre>GridSearchCV(cv=10,\n",
       "             estimator=RandomForestClassifier(n_jobs=-1, random_state=42),\n",
       "             n_jobs=-1,\n",
       "             param_grid={&#x27;bootstrap&#x27;: [True, False],\n",
       "                         &#x27;criterion&#x27;: [&#x27;gini&#x27;, &#x27;entropy&#x27;],\n",
       "                         &#x27;max_depth&#x27;: [3, 7, 9, 11, 13, None],\n",
       "                         &#x27;n_estimators&#x27;: [200, 400, 600, 800, 1000]})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_jobs=-1, random_state=42)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_jobs=-1, random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "GridSearchCV(cv=10,\n",
       "             estimator=RandomForestClassifier(n_jobs=-1, random_state=42),\n",
       "             n_jobs=-1,\n",
       "             param_grid={'bootstrap': [True, False],\n",
       "                         'criterion': ['gini', 'entropy'],\n",
       "                         'max_depth': [3, 7, 9, 11, 13, None],\n",
       "                         'n_estimators': [200, 400, 600, 800, 1000]})"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "#random forest\n",
    "rf_g_clf = GridSearchCV(rf,param_grid=rf_params,cv=10,n_jobs=-1)\n",
    "rf_g_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "79d5050b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'bootstrap': False,\n",
       " 'criterion': 'entropy',\n",
       " 'max_depth': 7,\n",
       " 'n_estimators': 600}"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_g_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "357926f2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9648309178743961"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_g_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "225dfd0d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9649122807017544"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_g_clf.best_estimator_.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8339cf8",
   "metadata": {},
   "source": [
    "**Some Notes:**<br>\n",
    "   * as you can see grid search is time consuming process and with finer grid we will have very slow process.\n",
    "   * another problem of grid search is that our domain is discrete, and maybe the optimum number is the number between what we define.\n",
    "   * `GridSearchCV` will perform k-fold cross validation for us so we don't have to worry about it. so we have specify the k number and `cv=` is for that.\n",
    "\n",
    "can we do better? yes we can, let's see how."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd650f45",
   "metadata": {},
   "source": [
    "### RandomSearch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "83553377",
   "metadata": {},
   "source": [
    "in **RandomSearch**, again we define our grid, but not in discrete form, instead we define distribution for our hyperparameters. then the algorithm will pick randomly from our domain. how many? well, we have to specify it. let's see how we can implement it with scikit learn."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "802c68aa",
   "metadata": {},
   "source": [
    "#### Implementing RandomSearch with scikit learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "eb724308",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import RandomizedSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "8212f7de",
   "metadata": {},
   "outputs": [],
   "source": [
    "lr_distribution = {\n",
    "    'C':np.logspace(-4,4,num=1000),\n",
    "    'solver':['lbfgs','liblinear']\n",
    "}\n",
    "\n",
    "rf_distribution = {\n",
    "    'n_estimators':np.arange(200,1600,200),\n",
    "             'criterion':['gini','entropy'],\n",
    "             'bootstrap':[True,False],\n",
    "             'max_depth':np.arange(3,15,2)\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "2632735c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 203 ms\n",
      "Wall time: 1.07 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomizedSearchCV(cv=10,\n",
       "                   estimator=LogisticRegression(max_iter=1000, random_state=42),\n",
       "                   n_iter=50, n_jobs=-1,\n",
       "                   param_distributions={&#x27;C&#x27;: array([1.00000000e-04, 1.01861017e-04, 1.03756668e-04, 1.05687597e-04,\n",
       "       1.07654461e-04, 1.09657929e-04, 1.11698682e-04, 1.13777413e-04,\n",
       "       1.15894830e-04, 1.18051653e-04, 1.20248614e-04, 1.22486461e-04,\n",
       "       1.24765955e-04, 1.27087871e-04, 1.2...\n",
       "       7.04446228e+03, 7.17556092e+03, 7.30909933e+03, 7.44512291e+03,\n",
       "       7.58367791e+03, 7.72481145e+03, 7.86857151e+03, 8.01500696e+03,\n",
       "       8.16416760e+03, 8.31610415e+03, 8.47086827e+03, 8.62851257e+03,\n",
       "       8.78909065e+03, 8.95265713e+03, 9.11926760e+03, 9.28897872e+03,\n",
       "       9.46184819e+03, 9.63793480e+03, 9.81729841e+03, 1.00000000e+04]),\n",
       "                                        &#x27;solver&#x27;: [&#x27;lbfgs&#x27;, &#x27;liblinear&#x27;]},\n",
       "                   random_state=42)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomizedSearchCV</label><div class=\"sk-toggleable__content\"><pre>RandomizedSearchCV(cv=10,\n",
       "                   estimator=LogisticRegression(max_iter=1000, random_state=42),\n",
       "                   n_iter=50, n_jobs=-1,\n",
       "                   param_distributions={&#x27;C&#x27;: array([1.00000000e-04, 1.01861017e-04, 1.03756668e-04, 1.05687597e-04,\n",
       "       1.07654461e-04, 1.09657929e-04, 1.11698682e-04, 1.13777413e-04,\n",
       "       1.15894830e-04, 1.18051653e-04, 1.20248614e-04, 1.22486461e-04,\n",
       "       1.24765955e-04, 1.27087871e-04, 1.2...\n",
       "       7.04446228e+03, 7.17556092e+03, 7.30909933e+03, 7.44512291e+03,\n",
       "       7.58367791e+03, 7.72481145e+03, 7.86857151e+03, 8.01500696e+03,\n",
       "       8.16416760e+03, 8.31610415e+03, 8.47086827e+03, 8.62851257e+03,\n",
       "       8.78909065e+03, 8.95265713e+03, 9.11926760e+03, 9.28897872e+03,\n",
       "       9.46184819e+03, 9.63793480e+03, 9.81729841e+03, 1.00000000e+04]),\n",
       "                                        &#x27;solver&#x27;: [&#x27;lbfgs&#x27;, &#x27;liblinear&#x27;]},\n",
       "                   random_state=42)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000, random_state=42)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000, random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "RandomizedSearchCV(cv=10,\n",
       "                   estimator=LogisticRegression(max_iter=1000, random_state=42),\n",
       "                   n_iter=50, n_jobs=-1,\n",
       "                   param_distributions={'C': array([1.00000000e-04, 1.01861017e-04, 1.03756668e-04, 1.05687597e-04,\n",
       "       1.07654461e-04, 1.09657929e-04, 1.11698682e-04, 1.13777413e-04,\n",
       "       1.15894830e-04, 1.18051653e-04, 1.20248614e-04, 1.22486461e-04,\n",
       "       1.24765955e-04, 1.27087871e-04, 1.2...\n",
       "       7.04446228e+03, 7.17556092e+03, 7.30909933e+03, 7.44512291e+03,\n",
       "       7.58367791e+03, 7.72481145e+03, 7.86857151e+03, 8.01500696e+03,\n",
       "       8.16416760e+03, 8.31610415e+03, 8.47086827e+03, 8.62851257e+03,\n",
       "       8.78909065e+03, 8.95265713e+03, 9.11926760e+03, 9.28897872e+03,\n",
       "       9.46184819e+03, 9.63793480e+03, 9.81729841e+03, 1.00000000e+04]),\n",
       "                                        'solver': ['lbfgs', 'liblinear']},\n",
       "                   random_state=42)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "# random search for logistic regression\n",
    "lr_r_clf = RandomizedSearchCV(lr,param_distributions=lr_distribution\n",
    "                              ,cv=10,n_iter=50,n_jobs=-1,random_state=42)\n",
    "lr_r_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "3bd07888",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'solver': 'lbfgs', 'C': 0.14831025143361043}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_r_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "b46ed418",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9824637681159419"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_r_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "cb0e880c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9824561403508771"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_r_clf.best_estimator_.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "178a4c0e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "9d3ee593",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 3.23 s\n",
      "Wall time: 23 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-4 {color: black;background-color: white;}#sk-container-id-4 pre{padding: 0;}#sk-container-id-4 div.sk-toggleable {background-color: white;}#sk-container-id-4 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-4 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-4 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-4 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-4 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-4 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-4 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-4 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-4 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-4 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-4 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-4 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-4 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-4 div.sk-item {position: relative;z-index: 1;}#sk-container-id-4 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-4 div.sk-item::before, #sk-container-id-4 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-4 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-4 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-4 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-4 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-4 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-4 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-4 div.sk-label-container {text-align: center;}#sk-container-id-4 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-4 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-4\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomizedSearchCV(cv=10,\n",
       "                   estimator=RandomForestClassifier(n_jobs=-1, random_state=42),\n",
       "                   n_iter=5, n_jobs=-1,\n",
       "                   param_distributions={&#x27;bootstrap&#x27;: [True, False],\n",
       "                                        &#x27;criterion&#x27;: [&#x27;gini&#x27;, &#x27;entropy&#x27;],\n",
       "                                        &#x27;max_depth&#x27;: array([ 3,  5,  7,  9, 11, 13]),\n",
       "                                        &#x27;n_estimators&#x27;: array([ 200,  300,  400,  500,  600,  700,  800,  900, 1000, 1100, 1200,\n",
       "       1300, 1400, 1500])})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomizedSearchCV</label><div class=\"sk-toggleable__content\"><pre>RandomizedSearchCV(cv=10,\n",
       "                   estimator=RandomForestClassifier(n_jobs=-1, random_state=42),\n",
       "                   n_iter=5, n_jobs=-1,\n",
       "                   param_distributions={&#x27;bootstrap&#x27;: [True, False],\n",
       "                                        &#x27;criterion&#x27;: [&#x27;gini&#x27;, &#x27;entropy&#x27;],\n",
       "                                        &#x27;max_depth&#x27;: array([ 3,  5,  7,  9, 11, 13]),\n",
       "                                        &#x27;n_estimators&#x27;: array([ 200,  300,  400,  500,  600,  700,  800,  900, 1000, 1100, 1200,\n",
       "       1300, 1400, 1500])})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_jobs=-1, random_state=42)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_jobs=-1, random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "RandomizedSearchCV(cv=10,\n",
       "                   estimator=RandomForestClassifier(n_jobs=-1, random_state=42),\n",
       "                   n_iter=5, n_jobs=-1,\n",
       "                   param_distributions={'bootstrap': [True, False],\n",
       "                                        'criterion': ['gini', 'entropy'],\n",
       "                                        'max_depth': array([ 3,  5,  7,  9, 11, 13]),\n",
       "                                        'n_estimators': array([ 200,  300,  400,  500,  600,  700,  800,  900, 1000, 1100, 1200,\n",
       "       1300, 1400, 1500])})"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "#random search for random forest\n",
    "rf_r_clf = RandomizedSearchCV(rf,param_distributions={\n",
    "    'n_estimators':np.arange(200,1600,100),\n",
    "             'criterion':['gini','entropy'],\n",
    "             'bootstrap':[True,False],\n",
    "             'max_depth':np.arange(3,15,2)},cv=10,n_iter=5,n_jobs=-1)\n",
    "rf_r_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "005e8220",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_estimators': 1300,\n",
       " 'max_depth': 13,\n",
       " 'criterion': 'entropy',\n",
       " 'bootstrap': False}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_r_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "b462b66f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.962657004830918"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_r_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "0d5f9a70",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9649122807017544"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_r_clf.best_estimator_.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd14480c",
   "metadata": {},
   "source": [
    "**Some Notes:**<br>\n",
    "   * in random search it is more likely to get the optimum hyperparameters.\n",
    "   * here we define the distribution, so it will choose from all the possible numbers that we define. (I'm not defining distribution here, check `scipy` library for that)\n",
    "   * `n_iter=` is the number of times that our algorithm will sample hyperparameters and try them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "470d735c",
   "metadata": {},
   "source": [
    "### Bayesian Optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "970926ad",
   "metadata": {},
   "source": [
    "in **Bayesian Optimization** instead of using random combinations of values for hyperparameters, we look at the result we get so far and use them to predict the region in our hyperparameter space that might give better results. we need to predict how well a new combination will do and also model the uncertainty of that prediction.<br>\n",
    "the entire concept of Bayesian optimization is to reduce the number of times we need to train our model on our dataset by choosing only the most promising set of hyperparameters to evaluate based on previous results. in contrast, in the grid and random search, we are completely uninformed by post evaluations results and often spend a significant amount of time evaluating bad hyperparameters.<br>\n",
    "in these models, we have 3 main parts:<br>\n",
    "   1. our grid of hyperparameters.\n",
    "   2. **objective function**, which is our model.\n",
    "   3. **surrogate function**, which is a probabilistic model(usually) of our objective function and is so much easier to optimize compared to our objective function.<br>\n",
    "\n",
    "so the model will build a surrogate function and randomly choose hyperparameters from our grid, then choose the best combinations of hyperparameters that work best on our surrogate function. after that apply these hyperparameters to our objective function (our model) and calculate the results. then by using the result it updates the surrogate function and repeats its process until we reach our maximum number of iterations.<br>\n",
    "\n",
    "**NOTE:** there are different methods used for construct surrogate function like:\n",
    "   * Gaussian Process (GP)\n",
    "   * Tree Parzen Estimator (TPE)\n",
    "   * Random Forrest\n",
    "   * Neural Network<br>\n",
    "\n",
    "several good libraries use Bayesian optimization like Scikit-Optimize or Hyperopt, but I will use scikit-Optimize because, besides its own format, it has scikit learn format too. before using it be sure to install the package first and read the documentation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e23c705",
   "metadata": {},
   "source": [
    "#### Implementing Bayesian Optimization using Scikit-Optimize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "12f7cb66",
   "metadata": {},
   "outputs": [],
   "source": [
    "from skopt import BayesSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "4ff245e4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 6.2 s\n",
      "Wall time: 6.81 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-5 {color: black;background-color: white;}#sk-container-id-5 pre{padding: 0;}#sk-container-id-5 div.sk-toggleable {background-color: white;}#sk-container-id-5 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-5 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-5 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-5 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-5 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-5 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-5 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-5 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-5 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-5 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-5 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-5 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-5 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-5 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-5 div.sk-item {position: relative;z-index: 1;}#sk-container-id-5 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-5 div.sk-item::before, #sk-container-id-5 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-5 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-5 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-5 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-5 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-5 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-5 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-5 div.sk-label-container {text-align: center;}#sk-container-id-5 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-5 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-5\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BayesSearchCV(cv=10,\n",
       "              estimator=LogisticRegression(max_iter=1000, random_state=42),\n",
       "              n_iter=20, n_jobs=-1, random_state=42,\n",
       "              search_spaces={&#x27;C&#x27;: (0.0001, 10000.0, &#x27;log-uniform&#x27;),\n",
       "                             &#x27;solver&#x27;: [&#x27;lbfgs&#x27;, &#x27;liblinear&#x27;]})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BayesSearchCV</label><div class=\"sk-toggleable__content\"><pre>BayesSearchCV(cv=10,\n",
       "              estimator=LogisticRegression(max_iter=1000, random_state=42),\n",
       "              n_iter=20, n_jobs=-1, random_state=42,\n",
       "              search_spaces={&#x27;C&#x27;: (0.0001, 10000.0, &#x27;log-uniform&#x27;),\n",
       "                             &#x27;solver&#x27;: [&#x27;lbfgs&#x27;, &#x27;liblinear&#x27;]})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000, random_state=42)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000, random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "BayesSearchCV(cv=10,\n",
       "              estimator=LogisticRegression(max_iter=1000, random_state=42),\n",
       "              n_iter=20, n_jobs=-1, random_state=42,\n",
       "              search_spaces={'C': (0.0001, 10000.0, 'log-uniform'),\n",
       "                             'solver': ['lbfgs', 'liblinear']})"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "lr_bo_clf = BayesSearchCV(lr,search_spaces={\n",
    "    'C':(1e-4,1e4,'log-uniform'),\n",
    "    'solver':['lbfgs','liblinear']\n",
    "},n_iter=20,cv=10,n_jobs=-1,random_state=42)\n",
    "lr_bo_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "3c164b40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('C', 0.9278459556357788), ('solver', 'lbfgs')])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_bo_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "e8dda702",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9825603864734299"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_bo_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "412beae5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9824561403508771"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_bo_clf.best_estimator_.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ade58010",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "55eabc09",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 1.94 s\n",
      "Wall time: 27.7 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-6 {color: black;background-color: white;}#sk-container-id-6 pre{padding: 0;}#sk-container-id-6 div.sk-toggleable {background-color: white;}#sk-container-id-6 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-6 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-6 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-6 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-6 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-6 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-6 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-6 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-6 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-6 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-6 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-6 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-6 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-6 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-6 div.sk-item {position: relative;z-index: 1;}#sk-container-id-6 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-6 div.sk-item::before, #sk-container-id-6 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-6 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-6 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-6 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-6 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-6 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-6 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-6 div.sk-label-container {text-align: center;}#sk-container-id-6 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-6 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-6\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>BayesSearchCV(cv=10,\n",
       "              estimator=RandomForestClassifier(n_jobs=-1, random_state=42),\n",
       "              n_iter=10, n_jobs=-1,\n",
       "              search_spaces={&#x27;bootstrap&#x27;: [True, False],\n",
       "                             &#x27;criterion&#x27;: [&#x27;gini&#x27;, &#x27;entropy&#x27;],\n",
       "                             &#x27;max_depth&#x27;: array([ 3,  5,  7,  9, 11, 13]),\n",
       "                             &#x27;n_estimators&#x27;: array([ 200,  400,  600,  800, 1000, 1200, 1400])})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">BayesSearchCV</label><div class=\"sk-toggleable__content\"><pre>BayesSearchCV(cv=10,\n",
       "              estimator=RandomForestClassifier(n_jobs=-1, random_state=42),\n",
       "              n_iter=10, n_jobs=-1,\n",
       "              search_spaces={&#x27;bootstrap&#x27;: [True, False],\n",
       "                             &#x27;criterion&#x27;: [&#x27;gini&#x27;, &#x27;entropy&#x27;],\n",
       "                             &#x27;max_depth&#x27;: array([ 3,  5,  7,  9, 11, 13]),\n",
       "                             &#x27;n_estimators&#x27;: array([ 200,  400,  600,  800, 1000, 1200, 1400])})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_jobs=-1, random_state=42)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_jobs=-1, random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "BayesSearchCV(cv=10,\n",
       "              estimator=RandomForestClassifier(n_jobs=-1, random_state=42),\n",
       "              n_iter=10, n_jobs=-1,\n",
       "              search_spaces={'bootstrap': [True, False],\n",
       "                             'criterion': ['gini', 'entropy'],\n",
       "                             'max_depth': array([ 3,  5,  7,  9, 11, 13]),\n",
       "                             'n_estimators': array([ 200,  400,  600,  800, 1000, 1200, 1400])})"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "rf_bo_clf = BayesSearchCV(rf,search_spaces={\n",
    "    'n_estimators':np.arange(200,1600,200),\n",
    "    'criterion':['gini','entropy'],\n",
    "    'bootstrap':[True,False],\n",
    "    'max_depth':np.arange(3,15,2)\n",
    "},cv=10,n_iter=10,n_jobs=-1)\n",
    "rf_bo_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "62ea1c02",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('bootstrap', False),\n",
       "             ('criterion', 'gini'),\n",
       "             ('max_depth', 13),\n",
       "             ('n_estimators', 600)])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_bo_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "efe48279",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9604830917874396"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_bo_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "3b8beb61",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.956140350877193"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_bo_clf.best_estimator_.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cbb15b4",
   "metadata": {},
   "source": [
    "### Successive Halving and Hyperband"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2e7374c",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "if a hyperparameter configuration is bad, it will be bad at the early stage of our computation or will be bad at the simpler form of our problem. that is the whole idea of **Successive Halving**. in successive halving (SH) we have 3 parameters:\n",
    "   * Number of configuration (${N}$): which is our number of hyperparameter combinations.\n",
    "   * our budget (${B}$): it can be the number of samples, number of iterations, or number of estimators in case of boosting methods.\n",
    "   * our factor (${k}$): usually it is 2 or 3.<br>\n",
    "\n",
    "and the SH algorithm will be something like this:\n",
    "   1. Randomly sample a set of hyperparameter configurations. how many? well, it depends on our budget and factor.\n",
    "   2. Evaluate the performances of all correctly remaining configurations.\n",
    "   3. keep $\\frac{1}{K}$ of best scoring configurations and increase the number of our budget with the factor of ${k}$.\n",
    "   4. repeat steps 2 and 3 until one configuration remains.\n",
    "\n",
    "so instead of wasting our time on configurations that won't perform well, we keep the most promising ones and test them on more resources in each iteration.<br>\n",
    "as you can see there is a trade-off between the number of configurations and our budget. so the question is how do we specify these numbers? that is where we use **Hyperband**. hyperband will perform SH with different budgets to find the best configurations.<br>\n",
    "**NOTE:** there is a variation of hyperband that use Bayesian Optimization to find our budget (instead of blindly testing all budgets), and it is called **BOHB** (short for Bayesian Optimization Hyperband). now let's implement SH using scikit learn."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f41f63e2",
   "metadata": {},
   "source": [
    "#### Implementing SH in Scikit learn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "146f660c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.experimental import enable_halving_search_cv # this method is still experimental so we have to import it\n",
    "from sklearn.model_selection import HalvingRandomSearchCV"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "3ec18acd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 172 ms\n",
      "Wall time: 222 ms\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-7 {color: black;background-color: white;}#sk-container-id-7 pre{padding: 0;}#sk-container-id-7 div.sk-toggleable {background-color: white;}#sk-container-id-7 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-7 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-7 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-7 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-7 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-7 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-7 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-7 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-7 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-7 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-7 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-7 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-7 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-7 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-7 div.sk-item {position: relative;z-index: 1;}#sk-container-id-7 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-7 div.sk-item::before, #sk-container-id-7 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-7 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-7 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-7 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-7 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-7 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-7 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-7 div.sk-label-container {text-align: center;}#sk-container-id-7 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-7 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-7\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>HalvingRandomSearchCV(cv=10,\n",
       "                      estimator=LogisticRegression(max_iter=1000,\n",
       "                                                   random_state=42),\n",
       "                      n_jobs=-1,\n",
       "                      param_distributions={&#x27;C&#x27;: array([1.00000000e-04, 1.01861017e-04, 1.03756668e-04, 1.05687597e-04,\n",
       "       1.07654461e-04, 1.09657929e-04, 1.11698682e-04, 1.13777413e-04,\n",
       "       1.15894830e-04, 1.18051653e-04, 1.20248614e-04, 1.22486461e-04,\n",
       "       1.24765955e-04, 1.27087871e-04, 1.29452998...\n",
       "       6.54358602e+03, 6.66536327e+03, 6.78940681e+03, 6.91575883e+03,\n",
       "       7.04446228e+03, 7.17556092e+03, 7.30909933e+03, 7.44512291e+03,\n",
       "       7.58367791e+03, 7.72481145e+03, 7.86857151e+03, 8.01500696e+03,\n",
       "       8.16416760e+03, 8.31610415e+03, 8.47086827e+03, 8.62851257e+03,\n",
       "       8.78909065e+03, 8.95265713e+03, 9.11926760e+03, 9.28897872e+03,\n",
       "       9.46184819e+03, 9.63793480e+03, 9.81729841e+03, 1.00000000e+04]),\n",
       "                                           &#x27;solver&#x27;: [&#x27;lbfgs&#x27;, &#x27;liblinear&#x27;]})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">HalvingRandomSearchCV</label><div class=\"sk-toggleable__content\"><pre>HalvingRandomSearchCV(cv=10,\n",
       "                      estimator=LogisticRegression(max_iter=1000,\n",
       "                                                   random_state=42),\n",
       "                      n_jobs=-1,\n",
       "                      param_distributions={&#x27;C&#x27;: array([1.00000000e-04, 1.01861017e-04, 1.03756668e-04, 1.05687597e-04,\n",
       "       1.07654461e-04, 1.09657929e-04, 1.11698682e-04, 1.13777413e-04,\n",
       "       1.15894830e-04, 1.18051653e-04, 1.20248614e-04, 1.22486461e-04,\n",
       "       1.24765955e-04, 1.27087871e-04, 1.29452998...\n",
       "       6.54358602e+03, 6.66536327e+03, 6.78940681e+03, 6.91575883e+03,\n",
       "       7.04446228e+03, 7.17556092e+03, 7.30909933e+03, 7.44512291e+03,\n",
       "       7.58367791e+03, 7.72481145e+03, 7.86857151e+03, 8.01500696e+03,\n",
       "       8.16416760e+03, 8.31610415e+03, 8.47086827e+03, 8.62851257e+03,\n",
       "       8.78909065e+03, 8.95265713e+03, 9.11926760e+03, 9.28897872e+03,\n",
       "       9.46184819e+03, 9.63793480e+03, 9.81729841e+03, 1.00000000e+04]),\n",
       "                                           &#x27;solver&#x27;: [&#x27;lbfgs&#x27;, &#x27;liblinear&#x27;]})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000, random_state=42)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-21\" type=\"checkbox\" ><label for=\"sk-estimator-id-21\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000, random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "HalvingRandomSearchCV(cv=10,\n",
       "                      estimator=LogisticRegression(max_iter=1000,\n",
       "                                                   random_state=42),\n",
       "                      n_jobs=-1,\n",
       "                      param_distributions={'C': array([1.00000000e-04, 1.01861017e-04, 1.03756668e-04, 1.05687597e-04,\n",
       "       1.07654461e-04, 1.09657929e-04, 1.11698682e-04, 1.13777413e-04,\n",
       "       1.15894830e-04, 1.18051653e-04, 1.20248614e-04, 1.22486461e-04,\n",
       "       1.24765955e-04, 1.27087871e-04, 1.29452998...\n",
       "       6.54358602e+03, 6.66536327e+03, 6.78940681e+03, 6.91575883e+03,\n",
       "       7.04446228e+03, 7.17556092e+03, 7.30909933e+03, 7.44512291e+03,\n",
       "       7.58367791e+03, 7.72481145e+03, 7.86857151e+03, 8.01500696e+03,\n",
       "       8.16416760e+03, 8.31610415e+03, 8.47086827e+03, 8.62851257e+03,\n",
       "       8.78909065e+03, 8.95265713e+03, 9.11926760e+03, 9.28897872e+03,\n",
       "       9.46184819e+03, 9.63793480e+03, 9.81729841e+03, 1.00000000e+04]),\n",
       "                                           'solver': ['lbfgs', 'liblinear']})"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "lr_sh_clf = HalvingRandomSearchCV(lr,param_distributions={\n",
    "    'C':np.logspace(-4,4,num=1000),\n",
    "    'solver':['lbfgs','liblinear']\n",
    "},factor=3,cv=10,n_jobs=-1)\n",
    "lr_sh_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "ed29c627",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9804761904761904"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sh_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "914490b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'solver': 'lbfgs', 'C': 12.856096069432965}"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sh_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "dd4f709d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9649122807017544"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lr_sh_clf.best_estimator_.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7f5087ea",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "6492519e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 3.33 s\n",
      "Wall time: 43.7 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-8 {color: black;background-color: white;}#sk-container-id-8 pre{padding: 0;}#sk-container-id-8 div.sk-toggleable {background-color: white;}#sk-container-id-8 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-8 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-8 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-8 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-8 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-8 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-8 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-8 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-8 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-8 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-8 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-8 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-8 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-8 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-8 div.sk-item {position: relative;z-index: 1;}#sk-container-id-8 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-8 div.sk-item::before, #sk-container-id-8 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-8 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-8 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-8 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-8 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-8 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-8 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-8 div.sk-label-container {text-align: center;}#sk-container-id-8 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-8 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-8\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>HalvingRandomSearchCV(cv=10,\n",
       "                      estimator=RandomForestClassifier(n_jobs=-1,\n",
       "                                                       random_state=42),\n",
       "                      n_jobs=-1,\n",
       "                      param_distributions={&#x27;bootstrap&#x27;: [True, False],\n",
       "                                           &#x27;criterion&#x27;: [&#x27;gini&#x27;, &#x27;entropy&#x27;],\n",
       "                                           &#x27;max_depth&#x27;: array([ 3,  5,  7,  9, 11, 13, 15]),\n",
       "                                           &#x27;n_estimators&#x27;: array([ 200,  400,  600,  800, 1000, 1200, 1400])})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-22\" type=\"checkbox\" ><label for=\"sk-estimator-id-22\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">HalvingRandomSearchCV</label><div class=\"sk-toggleable__content\"><pre>HalvingRandomSearchCV(cv=10,\n",
       "                      estimator=RandomForestClassifier(n_jobs=-1,\n",
       "                                                       random_state=42),\n",
       "                      n_jobs=-1,\n",
       "                      param_distributions={&#x27;bootstrap&#x27;: [True, False],\n",
       "                                           &#x27;criterion&#x27;: [&#x27;gini&#x27;, &#x27;entropy&#x27;],\n",
       "                                           &#x27;max_depth&#x27;: array([ 3,  5,  7,  9, 11, 13, 15]),\n",
       "                                           &#x27;n_estimators&#x27;: array([ 200,  400,  600,  800, 1000, 1200, 1400])})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-23\" type=\"checkbox\" ><label for=\"sk-estimator-id-23\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_jobs=-1, random_state=42)</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-24\" type=\"checkbox\" ><label for=\"sk-estimator-id-24\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(n_jobs=-1, random_state=42)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "HalvingRandomSearchCV(cv=10,\n",
       "                      estimator=RandomForestClassifier(n_jobs=-1,\n",
       "                                                       random_state=42),\n",
       "                      n_jobs=-1,\n",
       "                      param_distributions={'bootstrap': [True, False],\n",
       "                                           'criterion': ['gini', 'entropy'],\n",
       "                                           'max_depth': array([ 3,  5,  7,  9, 11, 13, 15]),\n",
       "                                           'n_estimators': array([ 200,  400,  600,  800, 1000, 1200, 1400])})"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "rf_sh_clf = HalvingRandomSearchCV(rf,param_distributions={\n",
    "    'n_estimators':np.arange(200,1600,200),\n",
    "             'criterion':['gini','entropy'],\n",
    "             'bootstrap':[True,False],\n",
    "             'max_depth':np.arange(3,17,2)\n",
    "},factor=3,cv=10,n_jobs=-1)\n",
    "rf_sh_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "a2ce05c1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'n_estimators': 1400,\n",
       " 'max_depth': 5,\n",
       " 'criterion': 'entropy',\n",
       " 'bootstrap': False}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_sh_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "88b9728a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.954920634920635"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_sh_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "097e551f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9649122807017544"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rf_sh_clf.best_estimator_.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2b9885c",
   "metadata": {},
   "source": [
    "**NOTE:** as you can see this method is experimental, so check documentation before use it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d435646",
   "metadata": {},
   "source": [
    "## Final notes on hyperparameters Tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29aa4826",
   "metadata": {},
   "source": [
    "### Using pipeline in HP tuning"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f34e076",
   "metadata": {},
   "source": [
    "we can use pipeline in our tuning process let's see how."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "e9ef5af9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-11 {color: black;background-color: white;}#sk-container-id-11 pre{padding: 0;}#sk-container-id-11 div.sk-toggleable {background-color: white;}#sk-container-id-11 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-11 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-11 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-11 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-11 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-11 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-11 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-11 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-11 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-11 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-11 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-11 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-11 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-11 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-11 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-11 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-11 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-11 div.sk-item {position: relative;z-index: 1;}#sk-container-id-11 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-11 div.sk-item::before, #sk-container-id-11 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-11 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-11 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-11 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-11 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-11 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-11 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-11 div.sk-label-container {text-align: center;}#sk-container-id-11 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-11 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-11\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>RandomizedSearchCV(cv=10,\n",
       "                   estimator=Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                              StandardScaler()),\n",
       "                                             (&#x27;pca&#x27;, PCA()),\n",
       "                                             (&#x27;kneighborsclassifier&#x27;,\n",
       "                                              KNeighborsClassifier())]),\n",
       "                   n_iter=50,\n",
       "                   param_distributions={&#x27;kneighborsclassifier__metric&#x27;: [&#x27;euclidean&#x27;,\n",
       "                                                                         &#x27;minkowski&#x27;,\n",
       "                                                                         &#x27;manhattan&#x27;],\n",
       "                                        &#x27;kneighborsclassifier__n_neighbors&#x27;: [3,\n",
       "                                                                              5,\n",
       "                                                                              7,\n",
       "                                                                              9,\n",
       "                                                                              11,\n",
       "                                                                              13,\n",
       "                                                                              15],\n",
       "                                        &#x27;pca__n_components&#x27;: [0.99, 0.95, 0.9,\n",
       "                                                              None]})</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-35\" type=\"checkbox\" ><label for=\"sk-estimator-id-35\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomizedSearchCV</label><div class=\"sk-toggleable__content\"><pre>RandomizedSearchCV(cv=10,\n",
       "                   estimator=Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                              StandardScaler()),\n",
       "                                             (&#x27;pca&#x27;, PCA()),\n",
       "                                             (&#x27;kneighborsclassifier&#x27;,\n",
       "                                              KNeighborsClassifier())]),\n",
       "                   n_iter=50,\n",
       "                   param_distributions={&#x27;kneighborsclassifier__metric&#x27;: [&#x27;euclidean&#x27;,\n",
       "                                                                         &#x27;minkowski&#x27;,\n",
       "                                                                         &#x27;manhattan&#x27;],\n",
       "                                        &#x27;kneighborsclassifier__n_neighbors&#x27;: [3,\n",
       "                                                                              5,\n",
       "                                                                              7,\n",
       "                                                                              9,\n",
       "                                                                              11,\n",
       "                                                                              13,\n",
       "                                                                              15],\n",
       "                                        &#x27;pca__n_components&#x27;: [0.99, 0.95, 0.9,\n",
       "                                                              None]})</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-36\" type=\"checkbox\" ><label for=\"sk-estimator-id-36\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">estimator: Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;standardscaler&#x27;, StandardScaler()), (&#x27;pca&#x27;, PCA()),\n",
       "                (&#x27;kneighborsclassifier&#x27;, KNeighborsClassifier())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-37\" type=\"checkbox\" ><label for=\"sk-estimator-id-37\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-38\" type=\"checkbox\" ><label for=\"sk-estimator-id-38\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">PCA</label><div class=\"sk-toggleable__content\"><pre>PCA()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-39\" type=\"checkbox\" ><label for=\"sk-estimator-id-39\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KNeighborsClassifier</label><div class=\"sk-toggleable__content\"><pre>KNeighborsClassifier()</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "RandomizedSearchCV(cv=10,\n",
       "                   estimator=Pipeline(steps=[('standardscaler',\n",
       "                                              StandardScaler()),\n",
       "                                             ('pca', PCA()),\n",
       "                                             ('kneighborsclassifier',\n",
       "                                              KNeighborsClassifier())]),\n",
       "                   n_iter=50,\n",
       "                   param_distributions={'kneighborsclassifier__metric': ['euclidean',\n",
       "                                                                         'minkowski',\n",
       "                                                                         'manhattan'],\n",
       "                                        'kneighborsclassifier__n_neighbors': [3,\n",
       "                                                                              5,\n",
       "                                                                              7,\n",
       "                                                                              9,\n",
       "                                                                              11,\n",
       "                                                                              13,\n",
       "                                                                              15],\n",
       "                                        'pca__n_components': [0.99, 0.95, 0.9,\n",
       "                                                              None]})"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.decomposition import PCA \n",
    "from sklearn.pipeline import make_pipeline\n",
    "from sklearn.neighbors import KNeighborsClassifier \n",
    "\n",
    "knn_pipe = make_pipeline(StandardScaler(),PCA(),KNeighborsClassifier())\n",
    "\n",
    "knn_params = {'kneighborsclassifier__n_neighbors':[3,5,7,9,11,13,15],\n",
    "              'kneighborsclassifier__metric':['euclidean','minkowski','manhattan'],\n",
    "              'pca__n_components':[0.99,0.95,0.90,None]\n",
    "    \n",
    "}\n",
    "\n",
    "knn_clf = RandomizedSearchCV(knn_pipe,param_distributions=knn_params,cv=10,n_iter=50)\n",
    "knn_clf.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "2d170e4e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.973719806763285"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_clf.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "d0d1303c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'pca__n_components': None,\n",
       " 'kneighborsclassifier__n_neighbors': 7,\n",
       " 'kneighborsclassifier__metric': 'minkowski'}"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_clf.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "5b6df1eb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9736842105263158"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_clf.best_estimator_.score(X_test,y_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7212e79f",
   "metadata": {},
   "source": [
    "### HP tuning Packages"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "147b2751",
   "metadata": {},
   "source": [
    "various packages perform HP tuning for us and each has its pros and cons, these are some of the most popular ones:<br>\n",
    "\n",
    "   * Scikit-Optimize\n",
    "   * Hyperopt\n",
    "   * Ray-tune\n",
    "   * SMAC\n",
    "   * Scikit-Learn\n",
    "\n",
    "and usually, most frameworks (like PyTorch, Tensorflow, xgboost, etc) have their HP Tuning packages, so check them too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "da768b9d",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}