integrations/optuna/showcase/Neptune-Optuna.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G4ny3mHCFrqG",
    "tags": [
     "header"
    ]
   },
   "source": [
    "# Neptune + Optuna"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8iu66Yz2SIWz",
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Before you start"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xndcBi_EUFVi",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Install dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jl90ahSMvhmV",
    "tags": [
     "installation"
    ]
   },
   "outputs": [],
   "source": [
    "! pip install --quiet optuna==2.3.0 lightgbm==3.1.0 plotly==4.13.0 neptune-client==0.4.132 neptune-contrib[monitoring]==0.25.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "1As3OW3LGqKg",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Import libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FufE4GTzvRZG",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "import lightgbm as lgb\n",
    "import optuna\n",
    "from sklearn.datasets import load_breast_cancer\n",
    "from sklearn.metrics import roc_auc_score\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "K-FNh8EKV40p",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Create a sample `objective` function for Optuna"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RHuuJb5VV94w",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "def objective(trial):\n",
    "   \n",
    "   data, target = load_breast_cancer(return_X_y=True)\n",
    "   train_x, test_x, train_y, test_y = train_test_split(data, target, test_size=0.25)\n",
    "   dtrain = lgb.Dataset(train_x, label=train_y)\n",
    "\n",
    "   param = {\n",
    "      'verbose': -1,\n",
    "      'objective': 'binary',\n",
    "      'metric': 'binary_logloss',\n",
    "      'num_leaves': trial.suggest_int('num_leaves', 2, 256),\n",
    "      'feature_fraction': trial.suggest_uniform('feature_fraction', 0.2, 1.0),\n",
    "      'bagging_fraction': trial.suggest_uniform('bagging_fraction', 0.2, 1.0),\n",
    "      'min_child_samples': trial.suggest_int('min_child_samples', 3, 100),\n",
    "   }\n",
    "\n",
    "   gbm = lgb.train(param, dtrain)\n",
    "   preds = gbm.predict(test_x)\n",
    "   accuracy = roc_auc_score(test_y, preds)\n",
    "   \n",
    "   return accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "If9d2A8zUynE",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Initialize Neptune"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "FQgpdwXVvcY1",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "import neptune\n",
    "\n",
    "neptune.init(api_token='ANONYMOUS', project_qualified_name='shared/optuna-integration')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "auuzaMR3Sei2",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Neptune gives you an option of logging data under a public folder as an anonymous user. This is great when you are just trying out the application and don't have a Neptune account yet.  \n",
    "\n",
    "---\n",
    "\n",
    "If you already have a [Neptune account](https://neptune.ai/register), you can create your own experiment and start logging to it using your personal API token. Pass your `username` to the `project_qualified_name` argument of the `neptune.init()` method: `project_qualified_name='YOUR_USERNAME/YOUR_PROJECT_NAME`. If you don't have a project yet, keep `/sandbox` at the end. The `sandbox` project is automatically created for you."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "X9i-HAeJDLO-",
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Quickstart"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CHp9zfqBToiX",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Step 1: Create an Experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Be3xPNTHTsIz",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "This creates an experiment in Neptune.\n",
    "\n",
    "Once you have a live experiment you can log things to it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "t5VBefIDwBjR",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "neptune.create_experiment('optuna-sweep')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_QdNmNLIXYXx",
    "tags": [
     "comment"
    ]
   },
   "source": [
    "Click on the link above to open this experiment in Neptune.\n",
    "\n",
    "For now it is empty but keep the tab with experiment open to see what happens next. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "XomuTzlsVILc",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Step 2: Create the Neptune Callback"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "82879lQuVfsJ",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "import neptunecontrib.monitoring.optuna as opt_utils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ybpXmhwywmJ2",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "neptune_callback = opt_utils.NeptuneCallback()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xbkNjVVLVl8t",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Step 3: Run Optuna with the Neptune Callback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "You can view the logging live in the Neptune tab once Optuna you run the below cell"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "40McUYc5wrva",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "study = optuna.create_study(direction='maximize')\n",
    "study.optimize(objective, n_trials=100, callbacks=[neptune_callback])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "P04k_nFiXl63",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Step 4: Stop logging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "When you track experiments with Neptune in Jupyter notebooks you need to explicitly stop the experiment by running ```neptune.stop()```.\n",
    "\n",
    "If you are running Neptune in regular ```.py``` scripts it will stop automatically when your code stops running."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "T5teGbiLwuEi",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "neptune.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YHQrdIgOIo_E",
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Advanced Options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q9zxE-EzLAbe",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Log charts and study object during sweep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "To log interactive charts from `optuna.visualization` and the `optuna.study` object itself, you can set `log_study=True` and `log_charts=True` while creating the Neptune Callback. \n",
    "\n",
    "This will log the charts after every iteration, so they can be explored while the sweep is running.\n",
    "\n",
    "<font color=yellow>Warning:</font> Depending on the size of the `optuna.study` object and the charts, this might add overhead to the sweep. To avoid this, you can log the study object and charts after the sweep."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EcRlqlGNLQ-3",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "# Create experiment\n",
    "neptune.create_experiment('optuna-sweep-advanced')\n",
    "\n",
    "# Create callback to log advanced options during the sweep\n",
    "neptune_callback = opt_utils.NeptuneCallback(log_study=True, log_charts=True)\n",
    "\n",
    "# Run Optuna with Neptune Callback\n",
    "study = optuna.create_study(direction='maximize')\n",
    "study.optimize(objective, n_trials=20, callbacks=[neptune_callback])\n",
    "\n",
    "# Stop logging \n",
    "neptune.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iAJZLVQKL8Dt",
    "tags": [
     "header"
    ]
   },
   "source": [
    "### Log charts and study object after the sweep"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": [
     "comment"
    ]
   },
   "source": [
    "If you don't want charts from `optuna.visualizations` and the `optuna.study` object to be updated after each iteration, you can log all of them after the sweep is complete to prevent overhead during the sweep. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Hw_lQq9AL5PR",
    "tags": [
     "code"
    ]
   },
   "outputs": [],
   "source": [
    "# Create experiment\n",
    "neptune.create_experiment('optuna-sweep-advanced')\n",
    "\n",
    "# Create Neptune callback\n",
    "neptune_callback = opt_utils.NeptuneCallback()\n",
    "\n",
    "# Run Optuna with Neptune Callback\n",
    "study = optuna.create_study(direction='maximize')\n",
    "study.optimize(objective, n_trials=100, callbacks=[neptune_callback])\n",
    "\n",
    "# Log Optuna charts and study object after the sweep is complete\n",
    "opt_utils.log_study_info(study)\n",
    "\n",
    "# Stop logging \n",
    "neptune.stop()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Tcq00KslIdDd",
    "tags": [
     "header"
    ]
   },
   "source": [
    "## Explore results in the Neptune UI"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "Neptune-Optuna.ipynb",
   "private_outputs": true,
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}