Démonstration FairLearn avec Azure ML.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Démonstration FairLearn avec intégration Azure ML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://fairlearn.github.io/_static/images/header-image.png\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unfairness Mitigation with Fairlearn and Azure Machine Learning\n",
    "**This notebook shows how to upload results from Fairlearn's GridSearch mitigation algorithm into a dashboard in Azure Machine Learning Studio**\n",
    "\n",
    "## Table of Contents\n",
    "\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Loading the Data](#LoadingData)\n",
    "1. [Training an Unmitigated Model](#UnmitigatedModel)\n",
    "1. [Mitigation with GridSearch](#Mitigation)\n",
    "1. [Uploading a Fairness Dashboard to Azure](#AzureUpload)\n",
    "    1. Registering models\n",
    "    1. Computing Fairness Metrics\n",
    "    1. Uploading to Azure\n",
    "1. [Conclusion](#Conclusion)\n",
    "\n",
    "<a id=\"Introduction\"></a>\n",
    "## Introduction\n",
    "This notebook shows how to use [Fairlearn (an open source fairness assessment and unfairness mitigation package)](http://fairlearn.github.io) and Azure Machine Learning Studio for a binary classification problem. This example uses the well-known adult census dataset. For the purposes of this notebook, we shall treat this as a loan decision problem. We will pretend that the label indicates whether or not each individual repaid a loan in the past. We will use the data to train a predictor to predict whether previously unseen individuals will repay a loan or not. The assumption is that the model predictions are used to decide whether an individual should be offered a loan. Its purpose is purely illustrative of a workflow including a fairness dashboard - in particular, we do **not** include a full discussion of the detailed issues which arise when considering fairness in machine learning. For such discussions, please [refer to the Fairlearn website](http://fairlearn.github.io/).\n",
    "\n",
    "We will apply the [grid search algorithm](https://fairlearn.github.io/api_reference/fairlearn.reductions.html#fairlearn.reductions.GridSearch) from the Fairlearn package using a specific notion of fairness called Demographic Parity. This produces a set of models, and we will view these in a dashboard both locally and in the Azure Machine Learning Studio.\n",
    "\n",
    "### Setup\n",
    "\n",
    "To use this notebook, an Azure Machine Learning workspace is required.\n",
    "Please see the [configuration notebook](../../configuration.ipynb) for information about creating one, if required.\n",
    "This notebook also requires the following packages:\n",
    "* `azureml-contrib-fairness`\n",
    "* `fairlearn==0.4.6`\n",
    "* `joblib`\n",
    "* `shap`\n",
    "\n",
    "\n",
    "<a id=\"LoadingData\"></a>\n",
    "## Loading the Data\n",
    "We use the well-known `adult` census dataset, which we load using `shap` (for convenience). We start with a fairly unremarkable set of imports:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'3.6.9 |Anaconda, Inc.| (default, Jul 30 2019, 19:07:31) \\n[GCC 7.3.0]'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import sys\n",
    "sys.version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2020-07-08 08:24:33.317453\n"
     ]
    }
   ],
   "source": [
    "import datetime\n",
    "now = datetime.datetime.now()\n",
    "print(now)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version Azure ML : 1.9.0\n"
     ]
    }
   ],
   "source": [
    "#Version\n",
    "import azureml.core\n",
    "print(\"Version Azure ML :\", azureml.core.VERSION)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install fairlearn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "#!pip install azureml-contrib-fairness"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fairlearn.reductions import GridSearch, DemographicParity, ErrorRate\n",
    "from fairlearn.widget import FairlearnDashboard\n",
    "from sklearn import svm\n",
    "from sklearn.preprocessing import LabelEncoder, StandardScaler\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "import pandas as pd\n",
    "import shap"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now load and inspect the data from the `shap` package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{4: 27816, 2: 3124, 1: 1039, 0: 311, 3: 271}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_raw, Y = shap.datasets.adult()\n",
    "X_raw[\"Race\"].value_counts().to_dict()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to treat the sex of each individual as a protected attribute (where 0 indicates female and 1 indicates male), and in this particular case we are going separate this attribute out and drop it from the main data (this is not always the best option - see the [Fairlearn website](http://fairlearn.github.io/) for further discussion). We also separate out the Race column, but we will not perform any mitigation based on it. Finally, we perform some standard data preprocessing steps to convert the data into a format suitable for the ML algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = X_raw[['Sex','Race']]\n",
    "X = X_raw.drop(labels=['Sex', 'Race'],axis = 1)\n",
    "X = pd.get_dummies(X)\n",
    "\n",
    "le = LabelEncoder()\n",
    "Y = le.fit_transform(Y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our data prepared, we can make the conventional split in to 'test' and 'train' subsets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "A value is trying to be set on a copy of a slice from a DataFrame\n",
      "\n",
      "See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy\n"
     ]
    }
   ],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "X_train, X_test, Y_train, Y_test, A_train, A_test = train_test_split(X_raw, \n",
    "                                                    Y, \n",
    "                                                    A,\n",
    "                                                    test_size = 0.2,\n",
    "                                                    random_state=0,\n",
    "                                                    stratify=Y)\n",
    "\n",
    "# Work around indexing issue\n",
    "X_train = X_train.reset_index(drop=True)\n",
    "A_train = A_train.reset_index(drop=True)\n",
    "X_test = X_test.reset_index(drop=True)\n",
    "A_test = A_test.reset_index(drop=True)\n",
    "\n",
    "# Improve labels\n",
    "A_test.Sex.loc[(A_test['Sex'] == 0)] = 'female'\n",
    "A_test.Sex.loc[(A_test['Sex'] == 1)] = 'male'\n",
    "\n",
    "\n",
    "A_test.Race.loc[(A_test['Race'] == 0)] = 'Amer-Indian-Eskimo'\n",
    "A_test.Race.loc[(A_test['Race'] == 1)] = 'Asian-Pac-Islander'\n",
    "A_test.Race.loc[(A_test['Race'] == 2)] = 'Black'\n",
    "A_test.Race.loc[(A_test['Race'] == 3)] = 'Other'\n",
    "A_test.Race.loc[(A_test['Race'] == 4)] = 'White'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"UnmitigatedModel\"></a>\n",
    "## Training an Unmitigated Model\n",
    "\n",
    "So we have a point of comparison, we first train a model (specifically, logistic regression from scikit-learn) on the raw data, without applying any mitigation algorithm:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(solver='liblinear')"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "unmitigated_predictor = LogisticRegression(solver='liblinear', fit_intercept=True)\n",
    "\n",
    "unmitigated_predictor.fit(X_train, Y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can view this model in the fairness dashboard, and see the disparities which appear:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ce6576678e2c45c3a0131f875442e97c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FairlearnWidget(value={'true_y': [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<fairlearn.widget._fairlearn_dashboard.FairlearnDashboard at 0x7ff1b57eba20>"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FairlearnDashboard(sensitive_features=A_test, sensitive_feature_names=['Sex', 'Race'],\n",
    "                   y_true=Y_test,\n",
    "                   y_pred={\"unmitigated\": unmitigated_predictor.predict(X_test)})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the disparity in accuracy when we select 'Sex' as the sensitive feature, we see that males have an error rate about three times greater than the females. More interesting is the disparity in opportunitiy - males are offered loans at three times the rate of females.\n",
    "\n",
    "Despite the fact that we removed the feature from the training data, our predictor still discriminates based on sex. This demonstrates that simply ignoring a protected attribute when fitting a predictor rarely eliminates unfairness. There will generally be enough other features correlated with the removed attribute to lead to disparate impact."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"Mitigation\"></a>\n",
    "## Mitigation with GridSearch\n",
    "\n",
    "The `GridSearch` class in `Fairlearn` implements a simplified version of the exponentiated gradient reduction of [Agarwal et al. 2018](https://arxiv.org/abs/1803.02453). The user supplies a standard ML estimator, which is treated as a blackbox - for this simple example, we shall use the logistic regression estimator from scikit-learn. `GridSearch` works by generating a sequence of relabellings and reweightings, and trains a predictor for each.\n",
    "\n",
    "For this example, we specify demographic parity (on the protected attribute of sex) as the fairness metric. Demographic parity requires that individuals are offered the opportunity (a loan in this example) independent of membership in the protected class (i.e., females and males should be offered loans at the same rate). *We are using this metric for the sake of simplicity* in this example; the appropriate fairness metric can only be selected after *careful examination of the broader context* in which the model is to be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "sweep = GridSearch(LogisticRegression(solver='liblinear', fit_intercept=True),\n",
    "                   constraints=DemographicParity(),\n",
    "                   grid_size=71)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With our estimator created, we can fit it to the data. After `fit()` completes, we extract the full set of predictors from the `GridSearch` object.\n",
    "\n",
    "The following cell trains a many copies of the underlying estimator, and may take a minute or two to run:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "sweep.fit(X_train, Y_train,\n",
    "          sensitive_features=A_train.Sex)\n",
    "\n",
    "predictors = sweep._predictors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We could load these predictors into the Fairness dashboard now. However, the plot would be somewhat confusing due to their number. In this case, we are going to remove the predictors which are dominated in the error-disparity space by others from the sweep (note that the disparity will only be calculated for the protected attribute; other potentially protected attributes will *not* be mitigated). In general, one might not want to do this, since there may be other considerations beyond the strict optimisation of error and disparity (of the given protected attribute)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "errors, disparities = [], []\n",
    "for m in predictors:\n",
    "    classifier = lambda X: m.predict(X)\n",
    "    \n",
    "    error = ErrorRate()\n",
    "    error.load_data(X_train, pd.Series(Y_train), sensitive_features=A_train.Sex)\n",
    "    disparity = DemographicParity()\n",
    "    disparity.load_data(X_train, pd.Series(Y_train), sensitive_features=A_train.Sex)\n",
    "    \n",
    "    errors.append(error.gamma(classifier)[0])\n",
    "    disparities.append(disparity.gamma(classifier).max())\n",
    "    \n",
    "all_results = pd.DataFrame( {\"predictor\": predictors, \"error\": errors, \"disparity\": disparities})\n",
    "\n",
    "dominant_models_dict = dict()\n",
    "base_name_format = \"census_gs_model_{0}\"\n",
    "row_id = 0\n",
    "for row in all_results.itertuples():\n",
    "    model_name = base_name_format.format(row_id)\n",
    "    errors_for_lower_or_eq_disparity = all_results[\"error\"][all_results[\"disparity\"]<=row.disparity]\n",
    "    if row.error <= errors_for_lower_or_eq_disparity.min():\n",
    "        dominant_models_dict[model_name] = row.predictor\n",
    "    row_id = row_id + 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can construct predictions for the dominant models (we include the unmitigated predictor as well, for comparison):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_dominant = {\"census_unmitigated\": unmitigated_predictor.predict(X_test)}\n",
    "models_dominant = {\"census_unmitigated\": unmitigated_predictor}\n",
    "for name, predictor in dominant_models_dict.items():\n",
    "    value = predictor.predict(X_test)\n",
    "    predictions_dominant[name] = value\n",
    "    models_dominant[name] = predictor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These predictions may then be viewed in the fairness dashboard. We include the race column from the dataset, as an alternative basis for assessing the models. However, since we have not based our mitigation on it, the variation in the models with respect to race can be large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "078ba95979df423baa45a9203e7b5a90",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "FairlearnWidget(value={'true_y': [0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 0…"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "<fairlearn.widget._fairlearn_dashboard.FairlearnDashboard at 0x7ff1b414ef98>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "FairlearnDashboard(sensitive_features=A_test, \n",
    "                   sensitive_feature_names=['Sex', 'Race'],\n",
    "                   y_true=Y_test.tolist(),\n",
    "                   y_pred=predictions_dominant)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When using sex as the sensitive feature, we see a Pareto front forming - the set of predictors which represent optimal tradeoffs between accuracy and disparity in predictions. In the ideal case, we would have a predictor at (1,0) - perfectly accurate and without any unfairness under demographic parity (with respect to the protected attribute \"sex\"). The Pareto front represents the closest we can come to this ideal based on our data and choice of estimator. Note the range of the axes - the disparity axis covers more values than the accuracy, so we can reduce disparity substantially for a small loss in accuracy. Finally, we also see that the unmitigated model is towards the top right of the plot, with high accuracy, but worst disparity.\n",
    "\n",
    "By clicking on individual models on the plot, we can inspect their metrics for disparity and accuracy in greater detail. In a real example, we would then pick the model which represented the best trade-off between accuracy and disparity given the relevant business constraints."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"AzureUpload\"></a>\n",
    "## Uploading a Fairness Dashboard to Azure\n",
    "\n",
    "Uploading a fairness dashboard to Azure is a two stage process. The `FairlearnDashboard` invoked in the previous section relies on the underlying Python kernel to compute metrics on demand. This is obviously not available when the fairness dashboard is rendered in AzureML Studio. By default, the dashboard in Azure Machine Learning Studio also requires the models to be registered. The required stages are therefore:\n",
    "1. Register the dominant models\n",
    "1. Precompute all the required metrics\n",
    "1. Upload to Azure\n",
    "\n",
    "Before that, we need to connect to Azure Machine Learning Studio:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace, Experiment, Model\n",
    "\n",
    "ws = Workspace.from_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"RegisterModels\"></a>\n",
    "### Registering Models\n",
    "\n",
    "The fairness dashboard is designed to integrate with registered models, so we need to do this for the models we want in the Studio portal. The assumption is that the names of the models specified in the dashboard dictionary correspond to the `id`s (i.e. `<name>:<version>` pairs) of registered models in the workspace. We register each of the models in the `models_dominant` dictionary into the workspace. For this, we have to save each model to a file, and then register that file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Registering  census_unmitigated\n",
      "Registering model census_unmitigated\n",
      "Registered  census_unmitigated:3\n",
      "Registering  census_gs_model_30\n",
      "Registering model census_gs_model_30\n",
      "Registered  census_gs_model_30:3\n",
      "Registering  census_gs_model_31\n",
      "Registering model census_gs_model_31\n",
      "Registered  census_gs_model_31:3\n",
      "Registering  census_gs_model_32\n",
      "Registering model census_gs_model_32\n",
      "Registered  census_gs_model_32:3\n",
      "Registering  census_gs_model_33\n",
      "Registering model census_gs_model_33\n",
      "Registered  census_gs_model_33:3\n",
      "Registering  census_gs_model_34\n",
      "Registering model census_gs_model_34\n",
      "Registered  census_gs_model_34:3\n",
      "Registering  census_gs_model_35\n",
      "Registering model census_gs_model_35\n",
      "Registered  census_gs_model_35:3\n"
     ]
    }
   ],
   "source": [
    "import joblib\n",
    "import os\n",
    "\n",
    "os.makedirs('models', exist_ok=True)\n",
    "def register_model(name, model):\n",
    "    print(\"Registering \", name)\n",
    "    model_path = \"models/{0}.pkl\".format(name)\n",
    "    joblib.dump(value=model, filename=model_path)\n",
    "    registered_model = Model.register(model_path=model_path,\n",
    "                                      model_name=name,\n",
    "                                      workspace=ws)\n",
    "    print(\"Registered \", registered_model.id)\n",
    "    return registered_model.id\n",
    "\n",
    "model_name_id_mapping = dict()\n",
    "for name, model in models_dominant.items():\n",
    "    m_id = register_model(name, model)\n",
    "    model_name_id_mapping[name] = m_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, produce new predictions dictionaries, with the updated names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "predictions_dominant_ids = dict()\n",
    "for name, y_pred in predictions_dominant.items():\n",
    "    predictions_dominant_ids[model_name_id_mapping[name]] = y_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"PrecomputeMetrics\"></a>\n",
    "### Precomputing Metrics\n",
    "\n",
    "We create a _dashboard dictionary_ using Fairlearn's `metrics` package. The `_create_group_metric_set` method has arguments similar to the Dashboard constructor, except that the sensitive features are passed as a dictionary (to ensure that names are available), and we must specify the type of prediction. Note that we use the `predictions_dominant_ids` dictionary we just created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "sf = { 'sex': A_test.Sex, 'race': A_test.Race }\n",
    "\n",
    "from fairlearn.metrics._group_metric_set import _create_group_metric_set\n",
    "\n",
    "\n",
    "dash_dict = _create_group_metric_set(y_true=Y_test,\n",
    "                                     predictions=predictions_dominant_ids,\n",
    "                                     sensitive_features=sf,\n",
    "                                     prediction_type='binary_classification')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"DashboardUpload\"></a>\n",
    "### Uploading the Dashboard\n",
    "\n",
    "Now, we import our `contrib` package which contains the routine to perform the upload:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.contrib.fairness import upload_dashboard_dictionary, download_dashboard_by_upload_id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can create an Experiment, then a Run, and upload our dashboard to it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Experiment(Name: FairLearnDemo,\n",
      "Workspace: workshopAML2020)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_dashboard_validation.py:Starting validation of dashboard dictionary\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_dashboard_validation.py:Validation of dashboard dictionary successful\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_azureml_validation.py:Validating model ids exist\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_azureml_validation.py:Checking census_gs_model_30:3\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_azureml_validation.py:Checking census_gs_model_31:3\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_azureml_validation.py:Checking census_gs_model_32:3\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_azureml_validation.py:Checking census_gs_model_33:3\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_azureml_validation.py:Checking census_gs_model_34:3\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_azureml_validation.py:Checking census_gs_model_35:3\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_azureml_validation.py:Checking census_unmitigated:3\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_azureml_validation.py:Validation of model ids complete\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Uploading y_true\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_true/7ca3e030-acaa-45c7-b47a-40ee5635643f.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded y_true to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_true/7ca3e030-acaa-45c7-b47a-40ee5635643f.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Found 7 predictions\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/2b0eae4b-ecd8-4718-a154-6644b6d27ac5.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded prediction to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/2b0eae4b-ecd8-4718-a154-6644b6d27ac5.json\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/3095d1c4-a0b1-4ccb-b54b-13e2bee554c0.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded prediction to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/3095d1c4-a0b1-4ccb-b54b-13e2bee554c0.json\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/549ccf81-6322-42fa-9b41-4d6cc13f0273.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded prediction to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/549ccf81-6322-42fa-9b41-4d6cc13f0273.json\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/a53f1e5e-c482-4a31-8c93-a612ca5f86d1.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded prediction to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/a53f1e5e-c482-4a31-8c93-a612ca5f86d1.json\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/5ad7ee3a-4980-4d2d-a4c4-bd6b3887f83c.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded prediction to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/5ad7ee3a-4980-4d2d-a4c4-bd6b3887f83c.json\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/a8225241-679e-4e99-9a4b-eaeb2891c8a9.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded prediction to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/a8225241-679e-4e99-9a4b-eaeb2891c8a9.json\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/7f0b530a-65f4-4bca-bc67-e436e8c5bf3a.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded prediction to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/7f0b530a-65f4-4bca-bc67-e436e8c5bf3a.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded 7 predictions\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Found {0} sensitive features\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/sensitive_features_column/d928aca5-f578-442b-9ad2-8352753613b8.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded sensitive feature column to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/sensitive_features_column/d928aca5-f578-442b-9ad2-8352753613b8.json\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/sensitive_features_column/aa307cb8-744b-4430-8eb8-8f0c2b829b48.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded sensitive feature column to prefix azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/sensitive_features_column/aa307cb8-744b-4430-8eb8-8f0c2b829b48.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded 2 sensitive features\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Uploading metrics\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/c4f11f69-167c-4186-9ee2-866dd70054cd.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 0 and sensitive_feature 0\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/885fff26-9cc4-450b-a68e-2ad4207daf26.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 1 and sensitive_feature 0\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/e5bea314-3e45-42e8-adce-966693b949f2.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 2 and sensitive_feature 0\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/b3c5eb01-83fa-456a-83d7-43fcab797d74.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 3 and sensitive_feature 0\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/747ff714-bb79-451a-9681-95f2db493071.json\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 4 and sensitive_feature 0\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/4f23a44b-53d4-4ed6-aa6a-58b5b6ffb0ad.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 5 and sensitive_feature 0\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/5aaa297d-6350-4780-899a-a5d908dd6145.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 6 and sensitive_feature 0\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/e4ecc3ab-daf1-4b27-aad8-c2ef2a21e00a.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 0 and sensitive_feature 1\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/29a7f3e0-8340-4257-b0e7-74d1abd3e31b.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 1 and sensitive_feature 1\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/14d7dd3d-9480-42da-9924-657f12a17453.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 2 and sensitive_feature 1\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/07e3aa01-2959-4c60-b5f1-e46bf26c5b04.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 3 and sensitive_feature 1\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/bdb5dd7d-49f5-466f-ac6e-9ce427d5940f.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 4 and sensitive_feature 1\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/a7225ed4-ef3a-45f7-8e6e-bdaac13f12ba.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 5 and sensitive_feature 1\n",
      "INFO:azureml.FairnessArtifactClient:Uploading to azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/ef859964-b3c3-44fe-978e-a3bff089f57e.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_specific_uploaders.py:Uploaded metrics data for prediction 6 and sensitive_feature 1\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Creating CUF Assets\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id a2b793e0e14b49b1b38826131f7b71e7\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id c29c5b99f975471a87629446db6b27e5\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id ba8260dbdd844c33953a00f1a5ab856a\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id f951c7c44d4d47cca502266a5c0f885e\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id 8e0c2861e7594aac955d2019a55ed154\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id e8ca543f674f4c868d6a1324cb63aa3b\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id 6f508d9ed5cc4fcaa69c794251d0ea0f\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id 47638ae907f64d33872065d81f7778b3\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id 872abbd056b847c1ba69b793a65355b2\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id 8e253cf3c81e46b58877332d57671f86\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id 4310da0bc633419a922b54d6625d836e\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id 0b5b69593dbc40eda7e50e568d3aaa8c\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id cce17c5c67624cedb652ff998c87cad5\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Asset uploaded with id 8bfce97fe2374119905341bac25cff93\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Fetching asset list\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Populating y_true\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_true/7ca3e030-acaa-45c7-b47a-40ee5635643f.json\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Uploaded to id: 50b2a2cf-5f06-43e5-80cc-22330b8d0ede\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Populating y_pred\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/2b0eae4b-ecd8-4718-a154-6644b6d27ac5.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/3095d1c4-a0b1-4ccb-b54b-13e2bee554c0.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/549ccf81-6322-42fa-9b41-4d6cc13f0273.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/a53f1e5e-c482-4a31-8c93-a612ca5f86d1.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/5ad7ee3a-4980-4d2d-a4c4-bd6b3887f83c.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/a8225241-679e-4e99-9a4b-eaeb2891c8a9.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/y_pred/7f0b530a-65f4-4bca-bc67-e436e8c5bf3a.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Populating sensitive features\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/sensitive_features_column/d928aca5-f578-442b-9ad2-8352753613b8.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/sensitive_features_column/aa307cb8-744b-4430-8eb8-8f0c2b829b48.json\n",
      "INFO:/anaconda/envs/azureml_py36/lib/python3.6/site-packages/azureml/contrib/fairness/_fairness_client.py:Populating metrics\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/c4f11f69-167c-4186-9ee2-866dd70054cd.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/885fff26-9cc4-450b-a68e-2ad4207daf26.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/e5bea314-3e45-42e8-adce-966693b949f2.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/b3c5eb01-83fa-456a-83d7-43fcab797d74.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/747ff714-bb79-451a-9681-95f2db493071.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/4f23a44b-53d4-4ed6-aa6a-58b5b6ffb0ad.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/5aaa297d-6350-4780-899a-a5d908dd6145.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/e4ecc3ab-daf1-4b27-aad8-c2ef2a21e00a.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/29a7f3e0-8340-4257-b0e7-74d1abd3e31b.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/14d7dd3d-9480-42da-9924-657f12a17453.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/07e3aa01-2959-4c60-b5f1-e46bf26c5b04.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/bdb5dd7d-49f5-466f-ac6e-9ce427d5940f.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/a7225ed4-ef3a-45f7-8e6e-bdaac13f12ba.json\n",
      "INFO:azureml.FairnessArtifactClient:Downloading from azureml.fairness/dashboard.metrics/50b2a2cf-5f06-43e5-80cc-22330b8d0ede/metrics_set/ef859964-b3c3-44fe-978e-a3bff089f57e.json\n"
     ]
    }
   ],
   "source": [
    "exp = Experiment(ws, \"FairLearnDemo\")\n",
    "print(exp)\n",
    "\n",
    "run = exp.start_logging()\n",
    "try:\n",
    "    dashboard_title = \"Dominant Models from GridSearch\"\n",
    "    upload_id = upload_dashboard_dictionary(run,\n",
    "                                            dash_dict,\n",
    "                                            dashboard_name=dashboard_title)\n",
    "    print(\"\\nUploaded to id: {0}\\n\".format(upload_id))\n",
    "\n",
    "    downloaded_dict = download_dashboard_by_upload_id(run, upload_id)\n",
    "finally:\n",
    "    run.complete()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dashboard can be viewed in the Run Details page.\n",
    "\n",
    "Finally, we can verify that the dashboard dictionary which we downloaded matches our upload:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(dash_dict == downloaded_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The FairLearn dashboard is available in the Fairness tab from the experiments view of Azure ML Studio:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://github.com/retkowsky/images/blob/master/Fairlearn.jpg?raw=true\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"Conclusion\"></a>\n",
    "## Conclusion\n",
    "\n",
    "In this notebook we have demonstrated how to use the `GridSearch` algorithm from Fairlearn to generate a collection of models, and then present them in the fairness dashboard in Azure Machine Learning Studio. Please remember that this notebook has not attempted to discuss the many considerations which should be part of any approach to unfairness mitigation. The [Fairlearn website](http://fairlearn.github.io/) provides that discussion"
   ]
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "riedgar"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3.6 - AzureML",
   "language": "python",
   "name": "python3-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}