mlflow_tracking_with_h2o_minio.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# MLflow Tracking: H2O example (remote server w/ Minio)\n",
    "- The MLflow Tracking component is an API and UI for logging parameters, code versions, metrics, and output files when running your machine learning code and for later visualizing the results.\n",
    "- This example uses H2O binary classification model on an public dataset\n",
    "- MLflow Tracking can be done locally or using a remote server.\n",
    "    - this notebook will use a `remote server` for tracking\n",
    "    - MLflow requires a cloud bucket storage for artifacts\n",
    "        - a Minio server provides a bucket for artifacts (plots,images,log files ..any unstructured data)\n",
    "- set environmental variables for Minio bucket storage before running notebook\n",
    "    - export MLFLOW_URL=mlflow_url\n",
    "    - export MLFLOW_S3_ENDPOINT_URL=minio_url\n",
    "    - export AWS_ACCESS_KEY_ID=minio_access_key\n",
    "    - export AWS_SECRET_ACCESS_KEY=minio_secret_key\n",
    "- this notebook was tested on Windows Subsystem for Linux\n",
    "- references:\n",
    "    - h2o: https://www.h2o.ai/\n",
    "    - mlflow: https://mlflow.org/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import Dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import h2o\n",
    "import mlflow\n",
    "import mlflow.h2o\n",
    "import mlflow.server\n",
    "from h2o.estimators.random_forest import H2ORandomForestEstimator\n",
    "from h2o.automl import H2OAutoML\n",
    "import numpy as np\n",
    "import os.path\n",
    "import time\n",
    "import matplotlib.pyplot as plt\n",
    "import itertools\n",
    "import json\n",
    "import os\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize H2O"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h2o.init()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import dataset\n",
    "- predict if someone has diabetes\n",
    "- reponse column: Outcome \n",
    "- feature columns: Pregnancies,Glucose,BloodPressure,Insulin,BMI,DiabetesPedigreeFunction,Age"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hf = h2o.import_file(path=\"https://raw.githubusercontent.com/plotly/datasets/master/diabetes.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hf.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add more descriptive reponse labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hf['OutcomeClass'] = 'Sick'\n",
    "mask = hf['Outcome'] == 0\n",
    "hf[mask,'OutcomeClass'] = 'NotSick'\n",
    "hf['OutcomeClass'] = hf['OutcomeClass'].asfactor() #make sure it the type is categorical\n",
    "hf['Outcome'] = hf['Outcome'].asfactor()\n",
    "\n",
    "hf.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Check class balance"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hf.group_by('OutcomeClass').count().get_frame()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Select features and response columns\n",
    "- display the list of features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y = 'OutcomeClass'\n",
    "X = hf.columns\n",
    "X.remove(y)\n",
    "X.remove('Outcome')\n",
    "\n",
    "X"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Split data into test/train subsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hf_train, hf_test = hf.split_frame(ratios = [0.8],seed=42)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now we'll do some modeling and track with MLflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set mlflow url\n",
    "- MLFLOW_URL env variable should be set before running notebook\n",
    "- choose experiement name\n",
    "- and activate it using .set_experiment()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlflow_url = os.environ['MLFLOW_URL'] \n",
    "mlflow.tracking.set_tracking_uri(mlflow_url)\n",
    "\n",
    "mlflow_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## List existing experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlflow.tracking.MlflowClient().list_experiments()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a new experiment\n",
    "- this will generate and experiement id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ex_id = mlflow.create_experiment(name='h2o_diabetes2')\n",
    "\n",
    "ex_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## We'll now run RandomForest Models with a grid/list of a hyper-parameter\n",
    "- hyper-parameter: number of trees\n",
    "- we'll log the number of trees, logloss,auc\n",
    "- we'll also save each model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Helper functions\n",
    "- save scoring,variable_importance,roc, confusion_matrix plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def save_plot_scoring_history(model,image_name):\n",
    "    df = model.scoring_history()\n",
    "    plt.plot(df['number_of_trees'],df['training_logloss'])\n",
    "    plt.plot(df['number_of_trees'],df['validation_logloss'])\n",
    "    plt.xlabel('number of trees',fontsize=14)\n",
    "    plt.ylabel('logloss',fontsize=14)\n",
    "    plt.title('Scoring History',fontsize=18)\n",
    "    plt.legend(['training','validation'])\n",
    "    plt.grid()\n",
    "    plt.savefig(image_name)\n",
    "    plt.close()\n",
    "\n",
    "def save_plot_varimp(model,image_name):\n",
    "    plt.rcdefaults()\n",
    "    fig, ax = plt.subplots()\n",
    "    variables = model._model_json['output']['variable_importances']['variable']\n",
    "    y_pos = np.arange(len(variables))\n",
    "    scaled_importance = model._model_json['output']['variable_importances']['scaled_importance']\n",
    "    ax.barh(y_pos, scaled_importance, align='center', color='blue', ecolor='black')\n",
    "    ax.set_yticks(y_pos)\n",
    "    ax.set_yticklabels(variables)\n",
    "    ax.invert_yaxis()\n",
    "    ax.set_xlabel('Scaled Importance')\n",
    "    ax.set_title('Variable Importance')\n",
    "    fig.savefig(image_name)\n",
    "    plt.close()\n",
    "\n",
    "    \n",
    "def save_plot_roc(model,image_name):\n",
    "    perf = model.model_performance(valid=True) # roc for validation frame\n",
    "    plt.xlabel('False Positive Rate (FPR)')\n",
    "    plt.ylabel('True Positive Rate (TPR)')\n",
    "    plt.title('ROC Curve')\n",
    "    plt.text(0.5, 0.5, r'AUC={0:.4f}'.format(perf._metric_json[\"AUC\"]))\n",
    "    plt.plot(perf.fprs, perf.tprs, 'b--')\n",
    "    plt.legend(['validation'])\n",
    "    plt.axis([0, 1, 0, 1])\n",
    "    plt.grid()\n",
    "    plt.savefig(image_name)\n",
    "    plt.close()\n",
    "\n",
    "    \n",
    "def plot_confusion_matrix(cm, classes,\n",
    "                          normalize=False,\n",
    "                          title='Confusion matrix',\n",
    "                          cmap=plt.cm.Blues,\n",
    "                          image_name='confusion_matrix.png'):\n",
    "    \n",
    "    #reference: https://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html\n",
    "    \n",
    "    if normalize:\n",
    "        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
    "        #print(\"Normalized confusion matrix\")\n",
    "    #else:\n",
    "        #print('Confusion matrix, without normalization')\n",
    "\n",
    "    #print(cm)\n",
    "\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    fmt = '.2f' if normalize else 'd'\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], fmt),\n",
    "                 horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "    plt.tight_layout()\n",
    "    plt.savefig(image_name)\n",
    "    plt.close()\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### create a folder for local artifact storage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not os.path.exists(\"artifact_folder\"):\n",
    "    os.makedirs(\"artifact_folder\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### grid number of trees random forest function\n",
    "- this is a modification of the example here: https://github.com/mlflow/mlflow/blob/master/examples/h2o/random_forest.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def trainRandomForest(ntrees):\n",
    "    with mlflow.start_run(experiment_id=ex_id):\n",
    "        rf = H2ORandomForestEstimator(ntrees=ntrees)\n",
    "        rf.train(x=X,\n",
    "                 y=y,\n",
    "                 training_frame=hf_train,\n",
    "                 validation_frame=hf_test)\n",
    "        \n",
    "        mlflow.log_param(\"ntrees\", ntrees)\n",
    "        mlflow.log_metric(\"auc\", rf.auc())\n",
    "        mlflow.log_metric(\"logloss\", rf.logloss())\n",
    "        \n",
    "        mlflow.h2o.log_model(rf, \"model\")\n",
    "        \n",
    "        cnf_matrix = np.zeros((2, 2))\n",
    "        cnf_matrix = cnf_matrix.astype('int')\n",
    "        perf = rf.model_performance(valid=True) # roc for validation frame\n",
    "        conf_list = perf.confusion_matrix().to_list()\n",
    "        cnf_matrix[0,0] = conf_list[0][0]\n",
    "        cnf_matrix[1,0] = conf_list[1][0]\n",
    "        cnf_matrix[0,1] = conf_list[0][1]\n",
    "        cnf_matrix[1,1] = conf_list[1][1]\n",
    "        np.set_printoptions(precision=2)\n",
    "        plot_confusion_matrix(cnf_matrix, classes = ['NotSick','Sick'],\n",
    "                              title='Confusion matrix, without normalization',\n",
    "                              image_name = 'artifact_folder/confusion_matrix.png')\n",
    "        \n",
    "        catch_kill = cnf_matrix[1,1]/cnf_matrix[1,0]\n",
    "        mlflow.log_metric(\"catch/kill\",catch_kill)\n",
    "        \n",
    "        while not os.path.exists('artifact_folder/confusion_matrix.png'):\n",
    "            time.sleep(1)\n",
    "        mlflow.log_artifact(\"artifact_folder/confusion_matrix.png\")\n",
    "        \n",
    "        save_plot_scoring_history(rf,'artifact_folder/score_history.png')\n",
    "        while not os.path.exists('artifact_folder/score_history.png'):\n",
    "            time.sleep(1)\n",
    "        mlflow.log_artifact(\"artifact_folder/score_history.png\")\n",
    "        \n",
    "        save_plot_varimp(rf,'artifact_folder/varimp.png')\n",
    "        while not os.path.exists('artifact_folder/varimp.png'):\n",
    "            time.sleep(1)\n",
    "        mlflow.log_artifact(\"artifact_folder/varimp.png\")\n",
    "        \n",
    "        save_plot_roc(rf,'artifact_folder/roc.png')\n",
    "        while not os.path.exists('artifact_folder/roc.png'):\n",
    "            time.sleep(1)\n",
    "        mlflow.log_artifact(\"artifact_folder/roc.png\")\n",
    "        \n",
    "        # in this case we'll delete the local plots on each iteration\n",
    "        os.remove(\"artifact_folder/score_history.png\")\n",
    "        os.remove(\"artifact_folder/varimp.png\")\n",
    "        os.remove(\"artifact_folder/roc.png\")\n",
    "        os.remove(\"artifact_folder/confusion_matrix.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## train the model on grid search of varying number of trees"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for ntrees in [10, 20, 50, 100, 200]:\n",
    "    trainRandomForest(ntrees)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Open the MLflow UI\n",
    "- open a browser to the uri provided below  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mlflow.tracking.get_tracking_uri()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## If you need to make any additions to a run of an experiment:\n",
    "- mlflow.start_run(experiment_id=ex_id,run_uuid = '')\n",
    "    - you can get the run_uuid from the mlflow UI\n",
    "- make additions\n",
    "- mlflow.end_run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shutdown h2o cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "h2o.cluster().shutdown()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}