Supervised_Classification_ML_Pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A Rigorous Machine Learning Pipeline (Supervised Binary Classification):\n",
    "* Author: Ryan Urbanowicz, PhD \n",
    "*  Affiliation: University of Pennsylvania - Department of Biostatistics, Epidemiology, and Informatics & Institute for Biomedical Informatics (IBI) \n",
    "* Date: 8/25/20\n",
    "***\n",
    "## Introduction\n",
    "This notebook presents an example of a 'rigorous' machine learning (ML) analysis pipeline that could be reasonablly applied to various supervised learning classification tasks. This pipeline was developed with biomedical data mining/modeling in mind. While this pipeline offers a comprehensive and rigorous proposed analysis, we do not suggest that this is necessarily the 'best' or 'only' way to conduct an appropriate binary classification ML analysis. \n",
    "***\n",
    "## Environment Requirements\n",
    "In order to run this pipeline as a Jupyter Notebook you must have the proper environment set up on your computer. Python 3 as well as a number of Python packages are required.  Most of these requirements are satisfied by installing the most recent version of anaconda (https://docs.anaconda.com/anaconda/install/). We used Anaconda3 with python version 3.7.7 during this pipeline development. In addition to the packages included in anaconda, the following packages will need to be installed separately (or possibly updated, if you have an older version installed):\n",
    "\n",
    "* scikit-rebate (To install: pip install skrebate)\n",
    "* xgboost (To install: pip install xgboost)\n",
    "* lightgbm (To install: pip install lightgbm)\n",
    "* optuna (To install: pip install optuna)\n",
    "\n",
    "Additionally, while currently commented out in the file (modeling_methods.py) if you want the optuna hypterparameter sweep figures to appear within the jupyter notebook (via the command 'fig.show()' ) you will need to run the following installation commands.  This should only be required if you edit the python file to uncomment this line for any or all of the ML modeling algorithms. \n",
    "\n",
    "* pip install -U plotly>=4.0.0\n",
    "* conda install -c plotly plotly-orca\n",
    "\n",
    "Lastly, in order to include the stand-alone algorithm 'ExSTraCS' we needed to call this from the command line within this Jupyter Notebook.  As a result, the part of this notebook running ExSTraCS will only run properly if the path to the working directory used to run this notebook includes no spaces.  In other words if your path includes a folder called 'My Folder' vs. 'My_Folder' you will likely get a run error for ExSTraCS (at least on a Windows machine). Thus, make sure to check that wherever you are running this notebook from, that the entire path to the working directory does note include any spaces. \n",
    "***\n",
    "## Dataset Requirements\n",
    "This notebook loads a single dataset to be run through the entire pipeline. Here we summarize the requirements for this dataset:\n",
    "* Ensure your data is in a single file: (If you have a pre-partitioned training/testing dataset, you should combine them into a single dataset before running this notebook)\n",
    "* Any dataset specific cleaning, feature transformation, or feature engineering that may be needed in order to maximize ML performance should be conducted by the user separately or added to the beginning of this notebook. \n",
    "* The dataset should be in tab-delimited .txt format to run this notebook (as is).  Commented-out code to load a comma separated file (.csv) and excel file (.xlsx) is included in the notebook as an alternative. \n",
    "* Missing data values should be empty or indicated with an 'NA'.\n",
    "* Dataset includes a header with column names. This should include a column for the binary class label and (optionally) a column for the instance ID, as well as columns for other 'features', e.g. independend variables. \n",
    "* The class labels should be 0 for the major class (i.e. the most frequent class), and 1 for the minor class.  This is important for generation of the precision/recall curve (PRC) plots. \n",
    "* This dataset is saved in the working directory containing the jupyter notebook file, and all other files in this repository.\n",
    "* All variables in the dataset have been numerically encoded (otherwise additional data preprocessing may be needed)\n",
    "***\n",
    "## Notebook Organization\n",
    "#### Part 1: Exploratory analysis, data cleaning, and creating n-fold CV partitioned datasets \n",
    "- Instances missing a class value are excluded\n",
    "- The user can indicate other columns that should be excluded from the analysis\n",
    "- The user can turn on/off the option to apply standard scaling to the data prior to CV partitioning or imputation\n",
    "    - We use no scaling by default. This is because most methods should work properly without it, and in applying the model downstream, it is difficult to properly scale new data so that models may be re-applied later.\n",
    "    - ANN modeling is sensitive to feature scaling, thus without it, performance not be as good. However this is only one of many challenges in getting ANN to perform well. \n",
    "- The user can turn on/off the option to impute missing values following CV partitioning\n",
    "- The user can turn on/off the option for the code to automatically attempt to discriminate nominal from ordinal features\n",
    "- The user can choose the number of CV partitions as well as the strategy for CV partitioning (i.e.  random (R), stratified (S), and matched (M) \n",
    "- CV training and testing datasets are saved as .txt files so that the same partitions may be analyzed external to this code\n",
    "    \n",
    "#### Part 2: Feature selection\n",
    "- The user can turn on/off the option to filter out the lowest scoring features in the data (i.e. to conduct not just feature importance evaluation but feature selection)\n",
    "- Feature importance evaluation and feature selection are conducted within each respective CV training partition\n",
    "- The pipeline reports feature importance estimates via two feature selection algorithms:\n",
    "    - Mutual Information: Proficient at detecting univariate associations\n",
    "    - MultiSURF: Proficient at detecting univariate associations, 2-way epistatic interactions, and heterogeneous associations\n",
    "    \n",
    "- When selected by the user, feature selection conservatively keeps any feature identified as 'potentially relevant' (i.e. score > 0) by either algorithm\n",
    "- Since MultiSURF scales quadratically with the number of training instances, there is an option to utilize a random subset of instances when running this algorithm to save computational time\n",
    "    \n",
    "#### Part 3: Machine learning modeling\n",
    "- Seven ML modeling algorithms have been implemented in this pipeline:\n",
    "    - Logistic Regression (scikit learn)\n",
    "    - Decision Tree (scikit learn)\n",
    "    - Random Forest (scikit learn)\n",
    "    - Naïve Bayes (scikit learn)\n",
    "    - XGBoost (separate python package)\n",
    "    - LightGBM (separate python package)\n",
    "    - SVM (scikit learn)\n",
    "    - ANN (scikit learn)\n",
    "    - ExSTraCS (v2.0.2.1) - a Learning Classifier System (LCS) algorithm manually configured to run in this notebook\n",
    "- User can select any subset of these methods to run\n",
    "- ML modeling is conducted within each respective CV training partition on the respective feature subset selected within the given CV partition\n",
    "- ML modeling begins with a hyperparameter sweep conducted with a grid search of hard coded run parameter options (user can edit as needed)\n",
    "- Balanced accuracy is applied as the evaluation metric for the hyperparameter sweep\n",
    "\n",
    "#### Part 4: ML feature importance vizualization\n",
    "\n",
    "***\n",
    "## Schematic of ML Analysis Pipeline\n",
    "\n",
    "<img src=\"ML pipeline schematic2.png\" />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Import Necessary Python Packages "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import os\n",
    "import random\n",
    "import copy\n",
    "import sys\n",
    "import pickle\n",
    "import time\n",
    "\n",
    "from data_processing_methods import cv_partitioner\n",
    "from data_processing_methods import identifyCategoricalFeatures\n",
    "from data_processing_methods import imputeCVData\n",
    "from data_processing_methods import dataScaling\n",
    "\n",
    "from feature_selection_methods import reportAllFS\n",
    "from feature_selection_methods import reportTopFS\n",
    "from feature_selection_methods import sort_save_fi_scores\n",
    "from feature_selection_methods import sort_save_fs_fi_scores\n",
    "from feature_selection_methods import run_mi\n",
    "from feature_selection_methods import run_multisurf\n",
    "from feature_selection_methods import selectFeatures\n",
    "from feature_selection_methods import genFilteredDatasets\n",
    "\n",
    "from modeling_methods import classEval\n",
    "from modeling_methods import roc_plot_single\n",
    "from modeling_methods import save_performance\n",
    "from modeling_methods import save_FI\n",
    "from modeling_methods import eval_Algorithm_FI\n",
    "from modeling_methods import eval_LCS\n",
    "from modeling_methods import eval_LCS_QRF\n",
    "\n",
    "#LCS package\n",
    "#sys.path.append('C:/home/work/research_urbslab/Core_ML_Pipeline/ml_pipeline_lcs/exstracs_2.0.2.1_noclassmutate_lynch')\n",
    "#import exstracs_main\n",
    "\n",
    "#Statistics packages\n",
    "from scipy import stats\n",
    "\n",
    "#Visualization Packages:\n",
    "#This code ensures that the output of plotting commands is displayed inline directly below the code cell that produced it.\n",
    "%matplotlib inline \n",
    "import matplotlib.pyplot as plt\n",
    "pd.set_option('display.max_columns', None) # display all the columns\n",
    "import scipy.stats as scs\n",
    "from scipy.stats import randint\n",
    "from matplotlib import rc\n",
    "import seaborn as sns\n",
    "\n",
    "import warnings\n",
    "warnings.filterwarnings('ignore')\n",
    "\n",
    "# Jupyter Notebook Hack: This code ensures that the results of multiple commands within a given cell are all displayed, rather than just the last. \n",
    "from IPython.core.interactiveshell import InteractiveShell\n",
    "InteractiveShell.ast_node_interactivity = \"all\"\n",
    "\n",
    "#Import Progress bar:\n",
    "from tqdm import tnrange, tqdm_notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Set Dataset Pipeline Variables (Mandatory)\n",
    "These variables values will have to be respecified to run this pipeline on a given dataset or computer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Unique dataset name (include dataset extension) (must be updated for a given dataset analysis)\n",
    "target_data = 'hcc-data_example.txt' \n",
    "\n",
    "#Dataset specific constants (Must be updated for a given dataset)\n",
    "outcomeLabel = 'Class Attribute' #i.e. class outcome column label\n",
    "categoricalOutcome = True  #Is outcome nominal (i.e. discrete/classification) This script will not function correctly if False!\n",
    "instLabel = None #If data includes instance labels, given respective column name here, otherwise put None\n",
    "ignore_columns = [] #list of column names to exclude from the analysis (only insert column names if needed, otherwise leave empty)\n",
    "categorical_variables = [] #Leave empty for 'auto-detect' otherwise list feature names to be treated as categorical."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Set Other Pipeline Variables (Optional) \n",
    "These variables can be left as is, or modified to update key aspects of how the ML pipeline will be run and what methods will be included"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Descriptive Analysis options ------------------------------------------------------------------------------------\n",
    "doCorrelation = True #Generate visualization of feature correlations (Not recommended for feature sets > 100)\n",
    "doUA = True #Perform a univariate analysis of features in the dataset \n",
    "\n",
    "#Cross Validation run parameters (change as needed) --------------------------------------------------------------\n",
    "cv_partitions = 10 #number of CV partitions (3-10 recommended)\n",
    "partition_method = 'S' #Options: random (R), stratified (S), matched (M)\n",
    "matchName = None #Name of column used for matched CV partitioning. Put None otherwise.\n",
    "\n",
    "#Data cleaning options--------------------------------------------------------------------------------------------\n",
    "scaleData = True  #Prescale original dataset values (important for ML and feature importance interpretation of some algorithms)\n",
    "imputeData = True  #Impute missing values (separate methods used for categorical vs ordinal variables)\n",
    "autoDetectCategorical = True #keep in mind that this can incorectly identify some ordinal values as being categorical (e.g. tumor state/grade)\n",
    "categoricalCutoff = 10 #If the number of unique values is greater than this value, the feature is assumed to be ordinal/continuous-valued\n",
    "\n",
    "#Feature Selection----------------------------------------------------------------------------------------------\n",
    "algorithms = ['mutual_information','multisurf']\n",
    "instanceSubset = 2000 #Sample subset size to use with MultiSURF (since they scale quandratically with n)\n",
    "maxFeaturesToKeep = 50 #Maximum number of features to keep during feature selection - pute None if no max desired)\n",
    "filterPoorFeatures = True\n",
    "topResults = 20 #Top features to illustrate in feature selection results figures\n",
    "\n",
    "#ML Modeling -----------------------------------------------------------------------------------------------------\n",
    "#To use all available algorithms, use the following (feel free to adjust the colors):\n",
    "#algorithmsToRun = ['logistic_regression','decision_tree', 'random_forest', 'naive_bayes', 'XGB', 'LGB','SVM','ANN','LCS','LCS_QRF'] \n",
    "#methodsKey = ('Logistic Regression', 'Decision Tree', 'Random Forest', 'Naïve Bayes','XGB','LGB', 'SVM', 'ANN','LCS','LCS_QRF')\n",
    "#algColors = ['black','yellow','orange','grey','purple', 'aqua', 'red', 'pink', 'green', 'blue']\n",
    "\n",
    "algorithmsToRun = ['logistic_regression','decision_tree', 'random_forest', 'naive_bayes', 'XGB', 'LGB','SVM','ANN','LCS','LCS_QRF'] \n",
    "methodsKey = ('Logistic Regression', 'Decision Tree', 'Random Forest', 'Naïve Bayes','XGB','LGB', 'SVM', 'ANN','LCS','LCS_QRF')\n",
    "algColors = ['black','yellow','orange','grey','purple', 'aqua', 'red', 'pink', 'green', 'blue']\n",
    "\n",
    "#Note that the statistical comparisons and compound feature importance plot sections of this notebook will fail if only one modeling algorithm is selected here.\n",
    "#To run a single ML modeling algorithm, use the following (example):\n",
    "#algorithmsToRun = ['decision_tree'] \n",
    "#methodsKey = ('Decision Tree')\n",
    "#algColors = ['black']\n",
    "\n",
    "#Notebook run parameters (i.e. set random seed for notebook replication consistency)\n",
    "randomSeed = 42\n",
    "random.seed(randomSeed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ML Modeling Hyperparamters (Optional)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Optima Hyperparameter sweep run parameters\n",
    "n_trials = 100 #number of bayesian hyperparameter optimization trials (using optuna)\n",
    "scoring_metric = 'balanced_accuracy' #metric used to pick best hyperparamters\n",
    "hype_cv = 3 #Internal cv partitions for hyperparamter sweep evaluations\n",
    "timeout = 300 #Seconds until hyperparameter sweep stopped - 3600 sec = 1 hour, 300 sec = 5 min\n",
    "\n",
    "#Logistic Regression\n",
    "param_grid_LR = {'penalty' : ['l2','l1'],\n",
    "                'C' : [1e-5, 1e5], \n",
    "                'dual' : [True, False],\n",
    "                'solver' : ['newton-cg','lbfgs', 'liblinear', 'sag','saga'],\n",
    "                'class_weight' : [None, 'balanced'],\n",
    "                'max_iter' : [10, 1000],\n",
    "                'n_jobs' : [-1]}\n",
    "\n",
    "#Decision Tree\n",
    "param_grid_DT = {'criterion' : ['gini', 'entropy'],\n",
    "                'splitter' : ['best', 'random'],\n",
    "                'max_depth' : [1, 30],\n",
    "                'min_samples_split' : [2, 50], \n",
    "                'min_samples_leaf' : [1, 50],\n",
    "                'max_features' : [None, 'auto','log2'],\n",
    "                'class_weight' : [None, 'balanced']}\n",
    "\n",
    "#Random Forest\n",
    "param_grid_RF = {'n_estimators': [10,1000],\n",
    "                'criterion' : ['gini', 'entropy'],\n",
    "                'max_depth' : [1, 30],\n",
    "                'min_samples_split' : [2, 50], \n",
    "                'min_samples_leaf' : [1, 50],\n",
    "                'max_features' : [None, 'auto','log2'],\n",
    "                'bootstrap' : [True],\n",
    "                'oob_score' : [False, True],\n",
    "                'n_jobs' : [-1],\n",
    "                'class_weight' : [None, 'balanced']}\n",
    "\n",
    "#XG Boost - note: class weight balance is included as option internally\n",
    "param_grid_XGB = {'booster': ['gbtree'],\n",
    "                'objective' : ['binary:logistic'],\n",
    "                'verbosity' : [0],\n",
    "                'reg_lambda' : [1e-8, 1.0],\n",
    "                'alpha' : [1e-8, 1.0],\n",
    "                'eta' : [1e-8, 1.0],\n",
    "                'gamma' : [1e-8, 1.0],\n",
    "                'max_depth' : [1, 30],\n",
    "                'grow_policy' : ['depthwise', 'lossguide'],\n",
    "                'n_estimators': [10,1000],\n",
    "                'min_samples_split' : [2, 50], \n",
    "                'min_samples_leaf' : [1, 50],\n",
    "                'subsample' : [0.5, 1.0],\n",
    "                'min_child_weight': [0.1, 10],\n",
    "                'colsample_bytree': [0.1, 1.0]}\n",
    "\n",
    "#LG Boost - note: class weight balance is included as option internally\n",
    "param_grid_LGB = {'objective' : ['binary'],\n",
    "                'metric': ['binary_logloss'],\n",
    "                'verbosity' : [-1],\n",
    "                'boosting_type': ['gbdt'],\n",
    "                'num_leaves': [2,256],\n",
    "                'max_depth' : [1, 30],\n",
    "                'lambda_l1': [1e-8, 10.0],\n",
    "                'lambda_l2': [1e-8, 10.0],\n",
    "                'feature_fraction': [0.4, 1.0],\n",
    "                'bagging_fraction': [0.4, 1.0],\n",
    "                'bagging_freq': [1,7],\n",
    "                'min_child_samples': [5,100],\n",
    "                'n_estimators': [10,1000]}\n",
    "\n",
    "#SVM\n",
    "param_grid_SVM = {'kernel' : ['linear','poly','rbf'],\n",
    "                'C': [0.1, 1000],\n",
    "                'gamma': ['scale'],\n",
    "                'degree': [1,6],\n",
    "                'probability' : [True],\n",
    "                'class_weight' : [None, 'balanced']}\n",
    "        \n",
    "#ANN\n",
    "param_grid_ANN = {'n_layers' : [1,3],\n",
    "                'layer_size' : [1,100],\n",
    "                'activation': ['identity','logistic', 'tanh', 'relu'],\n",
    "                'learning_rate': ['constant', 'invscaling', 'adaptive'],\n",
    "                'momentum': [.1,.9],\n",
    "                'solver': ['sgd', 'adam'],\n",
    "                'batch_size': ['auto'],\n",
    "                'alpha': [0.0001, 0.05],\n",
    "                'max_iter' : [200]}\n",
    "\n",
    "param_grid = {} #stores all user selected algorithm hyperparameters\n",
    "\n",
    "for algorithm in algorithmsToRun: #['logistic_regression','decision_tree', 'random_forest', 'naive_bayes', 'xg_boost','SVM','ANN'] \n",
    "    if algorithm == 'logistic_regression':\n",
    "        param_grid['logistic_regression'] = param_grid_LR\n",
    "    elif algorithm == 'decision_tree':\n",
    "        param_grid['decision_tree'] = param_grid_DT\n",
    "    elif algorithm == 'random_forest':\n",
    "        param_grid['random_forest'] = param_grid_RF\n",
    "    elif algorithm == 'naive_bayes':\n",
    "        pass #no hyperparameters\n",
    "    elif algorithm == 'XGB':\n",
    "        param_grid['XGB'] = param_grid_XGB\n",
    "    elif algorithm == 'LGB':\n",
    "        param_grid['LGB'] = param_grid_LGB\n",
    "    elif algorithm == 'SVM':\n",
    "        param_grid['SVM'] = param_grid_SVM\n",
    "    elif algorithm == 'ANN':\n",
    "        param_grid['ANN'] = param_grid_ANN\n",
    "    elif algorithm =='LCS':\n",
    "        pass #parameters are set directly\n",
    "    elif algorithm =='LCS_QRF':\n",
    "        pass #parameters are set directly\n",
    "    else:\n",
    "        print(\"Error: Algorithm not found!\")\n",
    "\n",
    "#LCS Hyperparameters (not set up for parameter sweep - uses specified and default hyperparameters)\n",
    "lcs_path = 'exstracs_2.0.2.1_noclassmutate_lynch/exstracs_main.py' #LCS file to call\n",
    "iterations = 20000\n",
    "popsize = 1000\n",
    "lcs_alg = 'ExSTraCS_2.0.2.1'\n",
    "\n",
    "#Visualization ---------------------------------------------------------------------------------------------------\n",
    "focus_metric = 'Balanced Accuracy'\n",
    "\n",
    "#Statistical Significance-----------------------------------------------------------------------------------------\n",
    "sig_cutoff = 0.05"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Notebook/Folder Initialization\n",
    "For simplicity it is recommended that this not be altered."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Working Directory: C:/Users/Moore Lab/Desktop/ExSTraCS_ML_Pipeline_Binary_Notebook-master/\n"
     ]
    }
   ],
   "source": [
    "#Standard Names (no need to change)\n",
    "output_folder = 'results'\n",
    "cv_data_folder = 'cv_datasets'\n",
    "ua_folder = 'univariate'\n",
    "model_folder = 'models'\n",
    "lcs_folder = 'lcs'\n",
    "\n",
    "#Working directory path automatically detected\n",
    "wd_path = os.getcwd()\n",
    "wd_path = wd_path.replace('\\\\','/')\n",
    "wd_path = wd_path+'/'\n",
    "print(\"Working Directory: \"+wd_path)\n",
    "\n",
    "#Save data name without extension for later file labeling\n",
    "data_name = target_data.split('.')[0] \n",
    "\n",
    "#Create output folder if it doesn't already exist\n",
    "if not os.path.exists(wd_path+(output_folder)):\n",
    "    os.mkdir(wd_path+(output_folder))\n",
    "\n",
    "#Create CV dataset folder if it doesn't already exist\n",
    "if not os.path.exists(wd_path+(cv_data_folder)):\n",
    "    os.mkdir(wd_path+(cv_data_folder))\n",
    "\n",
    "#Create univariate analysis folder if it doesn't already exist\n",
    "if not os.path.exists(wd_path+(ua_folder)):\n",
    "    os.mkdir(wd_path+(ua_folder))\n",
    "    \n",
    "#Create model pickle folder if it doesn't already exist\n",
    "if not os.path.exists(wd_path+(model_folder)):\n",
    "    os.mkdir(wd_path+(model_folder))\n",
    "    \n",
    "#Create LCS output folder if it doesn't already exist\n",
    "if not os.path.exists(wd_path+(lcs_folder)):\n",
    "    os.mkdir(wd_path+(lcs_folder))\n",
    "\n",
    "data_changed = False #Keeps track of whether cleaning changed original dataset (leave this set to False)\n",
    "\n",
    "run_time_dict = {}\n",
    "notebook_start_time = time.time()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Load Dataset\n",
    "This code assumes that the data is being loaded as a tab delimited '.txt' file with columns as features/outcome, and rows as instances. It is also assumed that the target file includes a header and that missing values are indicated with 'NA' or an empty cell. This command can be updated depending on users dataset format. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(165, 50)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "td = pd.read_csv(target_data, na_values='NA', sep = \"\\t\") #Data load command for tab-delimited .txt file\n",
    "#td = pd.read_csv(target_data, na_values='NA', sep = \",\") #Data load command for comma-separated .csv file\n",
    "#td = pd.read_excel(target_data) #Data load command for excel file\n",
    "\n",
    "td.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Debugging data simplification (This code should be commented out during normal run)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "#td = td.sample(frac=0.5, replace=False, random_state=randomSeed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Initial Exploratory Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Dataset contains 165 rows.\n",
      "Dataset contains 50 columns.\n"
     ]
    }
   ],
   "source": [
    "# Count the number of rows and columns\n",
    "num_rows = td.shape[0]\n",
    "print('Dataset contains '+str(num_rows)+' rows.')\n",
    "num_cols = td.shape[1]\n",
    "print('Dataset contains '+str(num_cols)+' columns.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 165 entries, 0 to 164\n",
      "Data columns (total 50 columns):\n",
      " #   Column                            Non-Null Count  Dtype  \n",
      "---  ------                            --------------  -----  \n",
      " 0   Gender                            165 non-null    int64  \n",
      " 1   Symptoms                          147 non-null    float64\n",
      " 2   Alcohol                           165 non-null    int64  \n",
      " 3   Hepatitis B Surface Antigen       148 non-null    float64\n",
      " 4   Hepatitis B e Antigen             126 non-null    float64\n",
      " 5   Hepatitis B Core Antibody         141 non-null    float64\n",
      " 6   Hepatitis C Virus Antibody        156 non-null    float64\n",
      " 7   Cirrhosis                         165 non-null    int64  \n",
      " 8   Endemic Countries                 126 non-null    float64\n",
      " 9   Smoking                           124 non-null    float64\n",
      " 10  Diabetes                          162 non-null    float64\n",
      " 11  Obesity                           155 non-null    float64\n",
      " 12  Hemochromatosis                   142 non-null    float64\n",
      " 13  Arterial Hypertension             162 non-null    float64\n",
      " 14  Chronic Renal Insufficiency       163 non-null    float64\n",
      " 15  Human Immunodeficiency Virus      151 non-null    float64\n",
      " 16  Nonalcoholic Steatohepatitis      143 non-null    float64\n",
      " 17  Esophageal Varices                113 non-null    float64\n",
      " 18  Splenomegaly                      150 non-null    float64\n",
      " 19  Portal Hypertension               154 non-null    float64\n",
      " 20  Portal Vein Thrombosis            162 non-null    float64\n",
      " 21  Liver Metastasis                  161 non-null    float64\n",
      " 22  Radiological Hallmark             163 non-null    float64\n",
      " 23  Age at diagnosis                  165 non-null    int64  \n",
      " 24  Grams of Alcohol per day          117 non-null    float64\n",
      " 25  Packs of cigarets per year        112 non-null    float64\n",
      " 26  Performance Status*               165 non-null    int64  \n",
      " 27  Encephalopathy degree*            164 non-null    float64\n",
      " 28  Ascites degree*                   163 non-null    float64\n",
      " 29  International Normalised Ratio*   161 non-null    float64\n",
      " 30  Alpha-Fetoprotein (ng/mL)         157 non-null    float64\n",
      " 31  Haemoglobin (g/dL)                162 non-null    float64\n",
      " 32  Mean Corpuscular Volume           162 non-null    float64\n",
      " 33  Leukocytes(G/L)                   162 non-null    float64\n",
      " 34  Platelets                         162 non-null    float64\n",
      " 35  Albumin (mg/dL)                   159 non-null    float64\n",
      " 36  Total Bilirubin(mg/dL)            160 non-null    float64\n",
      " 37  Alanine transaminase (U/L)        161 non-null    float64\n",
      " 38  Aspartate transaminase (U/L)      162 non-null    float64\n",
      " 39  Gamma glutamyl transferase (U/L)  162 non-null    float64\n",
      " 40  Alkaline phosphatase (U/L)        162 non-null    float64\n",
      " 41  Total Proteins (g/dL)             154 non-null    float64\n",
      " 42  Creatinine (mg/dL)                158 non-null    float64\n",
      " 43  Number of Nodules                 163 non-null    float64\n",
      " 44  Major dimension of nodule (cm)    145 non-null    float64\n",
      " 45  Direct Bilirubin (mg/dL)          121 non-null    float64\n",
      " 46  Iron                              86 non-null     float64\n",
      " 47  Oxygen Saturation (%)             85 non-null     float64\n",
      " 48  Ferritin (ng/mL)                  85 non-null     float64\n",
      " 49  Class Attribute                   165 non-null    int64  \n",
      "dtypes: float64(44), int64(6)\n",
      "memory usage: 64.6 KB\n"
     ]
    }
   ],
   "source": [
    "td.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Gender</th>\n",
       "      <th>Symptoms</th>\n",
       "      <th>Alcohol</th>\n",
       "      <th>Hepatitis B Surface Antigen</th>\n",
       "      <th>Hepatitis B e Antigen</th>\n",
       "      <th>Hepatitis B Core Antibody</th>\n",
       "      <th>Hepatitis C Virus Antibody</th>\n",
       "      <th>Cirrhosis</th>\n",
       "      <th>Endemic Countries</th>\n",
       "      <th>Smoking</th>\n",
       "      <th>Diabetes</th>\n",
       "      <th>Obesity</th>\n",
       "      <th>Hemochromatosis</th>\n",
       "      <th>Arterial Hypertension</th>\n",
       "      <th>Chronic Renal Insufficiency</th>\n",
       "      <th>Human Immunodeficiency Virus</th>\n",
       "      <th>Nonalcoholic Steatohepatitis</th>\n",
       "      <th>Esophageal Varices</th>\n",
       "      <th>Splenomegaly</th>\n",
       "      <th>Portal Hypertension</th>\n",
       "      <th>Portal Vein Thrombosis</th>\n",
       "      <th>Liver Metastasis</th>\n",
       "      <th>Radiological Hallmark</th>\n",
       "      <th>Age at diagnosis</th>\n",
       "      <th>Grams of Alcohol per day</th>\n",
       "      <th>Packs of cigarets per year</th>\n",
       "      <th>Performance Status*</th>\n",
       "      <th>Encephalopathy degree*</th>\n",
       "      <th>Ascites degree*</th>\n",
       "      <th>International Normalised Ratio*</th>\n",
       "      <th>Alpha-Fetoprotein (ng/mL)</th>\n",
       "      <th>Haemoglobin (g/dL)</th>\n",
       "      <th>Mean Corpuscular Volume</th>\n",
       "      <th>Leukocytes(G/L)</th>\n",
       "      <th>Platelets</th>\n",
       "      <th>Albumin (mg/dL)</th>\n",
       "      <th>Total Bilirubin(mg/dL)</th>\n",
       "      <th>Alanine transaminase (U/L)</th>\n",
       "      <th>Aspartate transaminase (U/L)</th>\n",
       "      <th>Gamma glutamyl transferase (U/L)</th>\n",
       "      <th>Alkaline phosphatase (U/L)</th>\n",
       "      <th>Total Proteins (g/dL)</th>\n",
       "      <th>Creatinine (mg/dL)</th>\n",
       "      <th>Number of Nodules</th>\n",
       "      <th>Major dimension of nodule (cm)</th>\n",
       "      <th>Direct Bilirubin (mg/dL)</th>\n",
       "      <th>Iron</th>\n",
       "      <th>Oxygen Saturation (%)</th>\n",
       "      <th>Ferritin (ng/mL)</th>\n",
       "      <th>Class Attribute</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>165.000000</td>\n",
       "      <td>147.000000</td>\n",
       "      <td>165.000000</td>\n",
       "      <td>148.000000</td>\n",
       "      <td>126.000000</td>\n",
       "      <td>141.000000</td>\n",
       "      <td>156.000000</td>\n",
       "      <td>165.000000</td>\n",
       "      <td>126.000000</td>\n",
       "      <td>124.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>155.000000</td>\n",
       "      <td>142.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>163.000000</td>\n",
       "      <td>151.000000</td>\n",
       "      <td>143.000000</td>\n",
       "      <td>113.000000</td>\n",
       "      <td>150.00000</td>\n",
       "      <td>154.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>161.000000</td>\n",
       "      <td>163.000000</td>\n",
       "      <td>165.000000</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>112.000000</td>\n",
       "      <td>165.000000</td>\n",
       "      <td>164.000000</td>\n",
       "      <td>163.000000</td>\n",
       "      <td>161.000000</td>\n",
       "      <td>1.570000e+02</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>159.000000</td>\n",
       "      <td>160.000000</td>\n",
       "      <td>161.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>154.000000</td>\n",
       "      <td>158.000000</td>\n",
       "      <td>163.000000</td>\n",
       "      <td>145.000000</td>\n",
       "      <td>121.000000</td>\n",
       "      <td>86.000000</td>\n",
       "      <td>85.000000</td>\n",
       "      <td>85.000000</td>\n",
       "      <td>165.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.806061</td>\n",
       "      <td>0.639456</td>\n",
       "      <td>0.739394</td>\n",
       "      <td>0.108108</td>\n",
       "      <td>0.007937</td>\n",
       "      <td>0.269504</td>\n",
       "      <td>0.217949</td>\n",
       "      <td>0.903030</td>\n",
       "      <td>0.079365</td>\n",
       "      <td>0.508065</td>\n",
       "      <td>0.345679</td>\n",
       "      <td>0.129032</td>\n",
       "      <td>0.049296</td>\n",
       "      <td>0.364198</td>\n",
       "      <td>0.122699</td>\n",
       "      <td>0.019868</td>\n",
       "      <td>0.055944</td>\n",
       "      <td>0.610619</td>\n",
       "      <td>0.56000</td>\n",
       "      <td>0.714286</td>\n",
       "      <td>0.222222</td>\n",
       "      <td>0.223602</td>\n",
       "      <td>0.680982</td>\n",
       "      <td>64.690909</td>\n",
       "      <td>71.008547</td>\n",
       "      <td>20.464286</td>\n",
       "      <td>1.018182</td>\n",
       "      <td>1.158537</td>\n",
       "      <td>1.441718</td>\n",
       "      <td>1.421851</td>\n",
       "      <td>1.929995e+04</td>\n",
       "      <td>12.879012</td>\n",
       "      <td>95.119753</td>\n",
       "      <td>1473.961549</td>\n",
       "      <td>113206.442654</td>\n",
       "      <td>3.445535</td>\n",
       "      <td>3.087938</td>\n",
       "      <td>67.093168</td>\n",
       "      <td>96.382716</td>\n",
       "      <td>268.026543</td>\n",
       "      <td>212.211605</td>\n",
       "      <td>8.961039</td>\n",
       "      <td>1.127089</td>\n",
       "      <td>2.736196</td>\n",
       "      <td>6.851172</td>\n",
       "      <td>1.930000</td>\n",
       "      <td>85.598837</td>\n",
       "      <td>37.028941</td>\n",
       "      <td>438.997647</td>\n",
       "      <td>0.381818</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.396586</td>\n",
       "      <td>0.481800</td>\n",
       "      <td>0.440302</td>\n",
       "      <td>0.311571</td>\n",
       "      <td>0.089087</td>\n",
       "      <td>0.445284</td>\n",
       "      <td>0.414182</td>\n",
       "      <td>0.296817</td>\n",
       "      <td>0.271387</td>\n",
       "      <td>0.501963</td>\n",
       "      <td>0.477064</td>\n",
       "      <td>0.336322</td>\n",
       "      <td>0.217251</td>\n",
       "      <td>0.482697</td>\n",
       "      <td>0.329103</td>\n",
       "      <td>0.140009</td>\n",
       "      <td>0.230621</td>\n",
       "      <td>0.489782</td>\n",
       "      <td>0.49805</td>\n",
       "      <td>0.453228</td>\n",
       "      <td>0.417029</td>\n",
       "      <td>0.417959</td>\n",
       "      <td>0.467533</td>\n",
       "      <td>13.319534</td>\n",
       "      <td>76.277680</td>\n",
       "      <td>51.565130</td>\n",
       "      <td>1.181528</td>\n",
       "      <td>0.428137</td>\n",
       "      <td>0.685819</td>\n",
       "      <td>0.477816</td>\n",
       "      <td>1.490983e+05</td>\n",
       "      <td>2.145237</td>\n",
       "      <td>8.405846</td>\n",
       "      <td>2909.106006</td>\n",
       "      <td>107118.632481</td>\n",
       "      <td>0.685132</td>\n",
       "      <td>5.499379</td>\n",
       "      <td>57.539747</td>\n",
       "      <td>87.484241</td>\n",
       "      <td>258.750319</td>\n",
       "      <td>167.943548</td>\n",
       "      <td>11.728739</td>\n",
       "      <td>0.955856</td>\n",
       "      <td>1.797901</td>\n",
       "      <td>5.095042</td>\n",
       "      <td>4.210452</td>\n",
       "      <td>55.698995</td>\n",
       "      <td>28.993552</td>\n",
       "      <td>457.114454</td>\n",
       "      <td>0.487311</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>20.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.840000</td>\n",
       "      <td>1.200000e+00</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>69.500000</td>\n",
       "      <td>2.200000</td>\n",
       "      <td>1.710000</td>\n",
       "      <td>1.900000</td>\n",
       "      <td>0.300000</td>\n",
       "      <td>11.000000</td>\n",
       "      <td>17.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>1.280000</td>\n",
       "      <td>3.900000</td>\n",
       "      <td>0.200000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.500000</td>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>57.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.170000</td>\n",
       "      <td>5.200000e+00</td>\n",
       "      <td>11.425000</td>\n",
       "      <td>89.775000</td>\n",
       "      <td>5.100000</td>\n",
       "      <td>255.750000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.800000</td>\n",
       "      <td>31.000000</td>\n",
       "      <td>46.250000</td>\n",
       "      <td>91.250000</td>\n",
       "      <td>108.250000</td>\n",
       "      <td>6.300000</td>\n",
       "      <td>0.700000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>0.370000</td>\n",
       "      <td>40.500000</td>\n",
       "      <td>16.000000</td>\n",
       "      <td>84.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>66.000000</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.300000</td>\n",
       "      <td>3.300000e+01</td>\n",
       "      <td>13.050000</td>\n",
       "      <td>94.950000</td>\n",
       "      <td>7.200000</td>\n",
       "      <td>93000.000000</td>\n",
       "      <td>3.400000</td>\n",
       "      <td>1.400000</td>\n",
       "      <td>50.000000</td>\n",
       "      <td>71.000000</td>\n",
       "      <td>179.500000</td>\n",
       "      <td>162.000000</td>\n",
       "      <td>7.050000</td>\n",
       "      <td>0.850000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.700000</td>\n",
       "      <td>83.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>295.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>74.000000</td>\n",
       "      <td>100.000000</td>\n",
       "      <td>30.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.530000</td>\n",
       "      <td>6.150000e+02</td>\n",
       "      <td>14.600000</td>\n",
       "      <td>100.675000</td>\n",
       "      <td>19.525000</td>\n",
       "      <td>171500.000000</td>\n",
       "      <td>4.050000</td>\n",
       "      <td>2.925000</td>\n",
       "      <td>78.000000</td>\n",
       "      <td>110.250000</td>\n",
       "      <td>345.250000</td>\n",
       "      <td>261.500000</td>\n",
       "      <td>7.575000</td>\n",
       "      <td>1.100000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>9.000000</td>\n",
       "      <td>1.400000</td>\n",
       "      <td>118.000000</td>\n",
       "      <td>56.000000</td>\n",
       "      <td>706.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>93.000000</td>\n",
       "      <td>500.000000</td>\n",
       "      <td>510.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>4.820000</td>\n",
       "      <td>1.810346e+06</td>\n",
       "      <td>18.700000</td>\n",
       "      <td>119.600000</td>\n",
       "      <td>13000.000000</td>\n",
       "      <td>459000.000000</td>\n",
       "      <td>4.900000</td>\n",
       "      <td>40.500000</td>\n",
       "      <td>420.000000</td>\n",
       "      <td>553.000000</td>\n",
       "      <td>1575.000000</td>\n",
       "      <td>980.000000</td>\n",
       "      <td>102.000000</td>\n",
       "      <td>7.600000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>22.000000</td>\n",
       "      <td>29.300000</td>\n",
       "      <td>224.000000</td>\n",
       "      <td>126.000000</td>\n",
       "      <td>2230.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           Gender   Symptoms      Alcohol  Hepatitis B Surface Antigen  \\\n",
       "count  165.000000  147.000000  165.000000                   148.000000   \n",
       "mean     0.806061    0.639456    0.739394                     0.108108   \n",
       "std      0.396586    0.481800    0.440302                     0.311571   \n",
       "min      0.000000    0.000000    0.000000                     0.000000   \n",
       "25%      1.000000    0.000000    0.000000                     0.000000   \n",
       "50%      1.000000    1.000000    1.000000                     0.000000   \n",
       "75%      1.000000    1.000000    1.000000                     0.000000   \n",
       "max      1.000000    1.000000    1.000000                     1.000000   \n",
       "\n",
       "       Hepatitis B e Antigen  Hepatitis B Core Antibody  \\\n",
       "count             126.000000                 141.000000   \n",
       "mean                0.007937                   0.269504   \n",
       "std                 0.089087                   0.445284   \n",
       "min                 0.000000                   0.000000   \n",
       "25%                 0.000000                   0.000000   \n",
       "50%                 0.000000                   0.000000   \n",
       "75%                 0.000000                   1.000000   \n",
       "max                 1.000000                   1.000000   \n",
       "\n",
       "       Hepatitis C Virus Antibody   Cirrhosis  Endemic Countries     Smoking  \\\n",
       "count                  156.000000  165.000000         126.000000  124.000000   \n",
       "mean                     0.217949    0.903030           0.079365    0.508065   \n",
       "std                      0.414182    0.296817           0.271387    0.501963   \n",
       "min                      0.000000    0.000000           0.000000    0.000000   \n",
       "25%                      0.000000    1.000000           0.000000    0.000000   \n",
       "50%                      0.000000    1.000000           0.000000    1.000000   \n",
       "75%                      0.000000    1.000000           0.000000    1.000000   \n",
       "max                      1.000000    1.000000           1.000000    1.000000   \n",
       "\n",
       "         Diabetes     Obesity  Hemochromatosis  Arterial Hypertension  \\\n",
       "count  162.000000  155.000000       142.000000             162.000000   \n",
       "mean     0.345679    0.129032         0.049296               0.364198   \n",
       "std      0.477064    0.336322         0.217251               0.482697   \n",
       "min      0.000000    0.000000         0.000000               0.000000   \n",
       "25%      0.000000    0.000000         0.000000               0.000000   \n",
       "50%      0.000000    0.000000         0.000000               0.000000   \n",
       "75%      1.000000    0.000000         0.000000               1.000000   \n",
       "max      1.000000    1.000000         1.000000               1.000000   \n",
       "\n",
       "       Chronic Renal Insufficiency  Human Immunodeficiency Virus  \\\n",
       "count                   163.000000                    151.000000   \n",
       "mean                      0.122699                      0.019868   \n",
       "std                       0.329103                      0.140009   \n",
       "min                       0.000000                      0.000000   \n",
       "25%                       0.000000                      0.000000   \n",
       "50%                       0.000000                      0.000000   \n",
       "75%                       0.000000                      0.000000   \n",
       "max                       1.000000                      1.000000   \n",
       "\n",
       "       Nonalcoholic Steatohepatitis  Esophageal Varices  Splenomegaly  \\\n",
       "count                    143.000000          113.000000     150.00000   \n",
       "mean                       0.055944            0.610619       0.56000   \n",
       "std                        0.230621            0.489782       0.49805   \n",
       "min                        0.000000            0.000000       0.00000   \n",
       "25%                        0.000000            0.000000       0.00000   \n",
       "50%                        0.000000            1.000000       1.00000   \n",
       "75%                        0.000000            1.000000       1.00000   \n",
       "max                        1.000000            1.000000       1.00000   \n",
       "\n",
       "       Portal Hypertension  Portal Vein Thrombosis  Liver Metastasis  \\\n",
       "count           154.000000              162.000000        161.000000   \n",
       "mean              0.714286                0.222222          0.223602   \n",
       "std               0.453228                0.417029          0.417959   \n",
       "min               0.000000                0.000000          0.000000   \n",
       "25%               0.000000                0.000000          0.000000   \n",
       "50%               1.000000                0.000000          0.000000   \n",
       "75%               1.000000                0.000000          0.000000   \n",
       "max               1.000000                1.000000          1.000000   \n",
       "\n",
       "       Radiological Hallmark  Age at diagnosis  Grams of Alcohol per day  \\\n",
       "count             163.000000        165.000000                117.000000   \n",
       "mean                0.680982         64.690909                 71.008547   \n",
       "std                 0.467533         13.319534                 76.277680   \n",
       "min                 0.000000         20.000000                  0.000000   \n",
       "25%                 0.000000         57.000000                  0.000000   \n",
       "50%                 1.000000         66.000000                 75.000000   \n",
       "75%                 1.000000         74.000000                100.000000   \n",
       "max                 1.000000         93.000000                500.000000   \n",
       "\n",
       "       Packs of cigarets per year  Performance Status*  \\\n",
       "count                  112.000000           165.000000   \n",
       "mean                    20.464286             1.018182   \n",
       "std                     51.565130             1.181528   \n",
       "min                      0.000000             0.000000   \n",
       "25%                      0.000000             0.000000   \n",
       "50%                      0.000000             1.000000   \n",
       "75%                     30.500000             2.000000   \n",
       "max                    510.000000             4.000000   \n",
       "\n",
       "       Encephalopathy degree*  Ascites degree*  \\\n",
       "count              164.000000       163.000000   \n",
       "mean                 1.158537         1.441718   \n",
       "std                  0.428137         0.685819   \n",
       "min                  1.000000         1.000000   \n",
       "25%                  1.000000         1.000000   \n",
       "50%                  1.000000         1.000000   \n",
       "75%                  1.000000         2.000000   \n",
       "max                  3.000000         3.000000   \n",
       "\n",
       "       International Normalised Ratio*  Alpha-Fetoprotein (ng/mL)  \\\n",
       "count                       161.000000               1.570000e+02   \n",
       "mean                          1.421851               1.929995e+04   \n",
       "std                           0.477816               1.490983e+05   \n",
       "min                           0.840000               1.200000e+00   \n",
       "25%                           1.170000               5.200000e+00   \n",
       "50%                           1.300000               3.300000e+01   \n",
       "75%                           1.530000               6.150000e+02   \n",
       "max                           4.820000               1.810346e+06   \n",
       "\n",
       "       Haemoglobin (g/dL)  Mean Corpuscular Volume  Leukocytes(G/L)  \\\n",
       "count          162.000000               162.000000       162.000000   \n",
       "mean            12.879012                95.119753      1473.961549   \n",
       "std              2.145237                 8.405846      2909.106006   \n",
       "min              5.000000                69.500000         2.200000   \n",
       "25%             11.425000                89.775000         5.100000   \n",
       "50%             13.050000                94.950000         7.200000   \n",
       "75%             14.600000               100.675000        19.525000   \n",
       "max             18.700000               119.600000     13000.000000   \n",
       "\n",
       "           Platelets  Albumin (mg/dL)  Total Bilirubin(mg/dL)  \\\n",
       "count     162.000000       159.000000              160.000000   \n",
       "mean   113206.442654         3.445535                3.087938   \n",
       "std    107118.632481         0.685132                5.499379   \n",
       "min         1.710000         1.900000                0.300000   \n",
       "25%       255.750000         3.000000                0.800000   \n",
       "50%     93000.000000         3.400000                1.400000   \n",
       "75%    171500.000000         4.050000                2.925000   \n",
       "max    459000.000000         4.900000               40.500000   \n",
       "\n",
       "       Alanine transaminase (U/L)  Aspartate transaminase (U/L)  \\\n",
       "count                  161.000000                    162.000000   \n",
       "mean                    67.093168                     96.382716   \n",
       "std                     57.539747                     87.484241   \n",
       "min                     11.000000                     17.000000   \n",
       "25%                     31.000000                     46.250000   \n",
       "50%                     50.000000                     71.000000   \n",
       "75%                     78.000000                    110.250000   \n",
       "max                    420.000000                    553.000000   \n",
       "\n",
       "       Gamma glutamyl transferase (U/L)  Alkaline phosphatase (U/L)  \\\n",
       "count                        162.000000                  162.000000   \n",
       "mean                         268.026543                  212.211605   \n",
       "std                          258.750319                  167.943548   \n",
       "min                           23.000000                    1.280000   \n",
       "25%                           91.250000                  108.250000   \n",
       "50%                          179.500000                  162.000000   \n",
       "75%                          345.250000                  261.500000   \n",
       "max                         1575.000000                  980.000000   \n",
       "\n",
       "       Total Proteins (g/dL)  Creatinine (mg/dL)  Number of Nodules  \\\n",
       "count             154.000000          158.000000         163.000000   \n",
       "mean                8.961039            1.127089           2.736196   \n",
       "std                11.728739            0.955856           1.797901   \n",
       "min                 3.900000            0.200000           0.000000   \n",
       "25%                 6.300000            0.700000           1.000000   \n",
       "50%                 7.050000            0.850000           2.000000   \n",
       "75%                 7.575000            1.100000           5.000000   \n",
       "max               102.000000            7.600000           5.000000   \n",
       "\n",
       "       Major dimension of nodule (cm)  Direct Bilirubin (mg/dL)        Iron  \\\n",
       "count                      145.000000                121.000000   86.000000   \n",
       "mean                         6.851172                  1.930000   85.598837   \n",
       "std                          5.095042                  4.210452   55.698995   \n",
       "min                          1.500000                  0.100000    0.000000   \n",
       "25%                          3.000000                  0.370000   40.500000   \n",
       "50%                          5.000000                  0.700000   83.000000   \n",
       "75%                          9.000000                  1.400000  118.000000   \n",
       "max                         22.000000                 29.300000  224.000000   \n",
       "\n",
       "       Oxygen Saturation (%)  Ferritin (ng/mL)  Class Attribute  \n",
       "count              85.000000         85.000000       165.000000  \n",
       "mean               37.028941        438.997647         0.381818  \n",
       "std                28.993552        457.114454         0.487311  \n",
       "min                 0.000000          0.000000         0.000000  \n",
       "25%                16.000000         84.000000         0.000000  \n",
       "50%                27.000000        295.000000         0.000000  \n",
       "75%                56.000000        706.000000         1.000000  \n",
       "max               126.000000       2230.000000         1.000000  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "td.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender                                int64\n",
       "Symptoms                            float64\n",
       "Alcohol                               int64\n",
       "Hepatitis B Surface Antigen         float64\n",
       "Hepatitis B e Antigen               float64\n",
       "Hepatitis B Core Antibody           float64\n",
       "Hepatitis C Virus Antibody          float64\n",
       "Cirrhosis                             int64\n",
       "Endemic Countries                   float64\n",
       "Smoking                             float64\n",
       "Diabetes                            float64\n",
       "Obesity                             float64\n",
       "Hemochromatosis                     float64\n",
       "Arterial Hypertension               float64\n",
       "Chronic Renal Insufficiency         float64\n",
       "Human Immunodeficiency Virus        float64\n",
       "Nonalcoholic Steatohepatitis        float64\n",
       "Esophageal Varices                  float64\n",
       "Splenomegaly                        float64\n",
       "Portal Hypertension                 float64\n",
       "Portal Vein Thrombosis              float64\n",
       "Liver Metastasis                    float64\n",
       "Radiological Hallmark               float64\n",
       "Age at diagnosis                      int64\n",
       "Grams of Alcohol per day            float64\n",
       "Packs of cigarets per year          float64\n",
       "Performance Status*                   int64\n",
       "Encephalopathy degree*              float64\n",
       "Ascites degree*                     float64\n",
       "International Normalised Ratio*     float64\n",
       "Alpha-Fetoprotein (ng/mL)           float64\n",
       "Haemoglobin (g/dL)                  float64\n",
       "Mean Corpuscular Volume             float64\n",
       "Leukocytes(G/L)                     float64\n",
       "Platelets                           float64\n",
       "Albumin (mg/dL)                     float64\n",
       "Total Bilirubin(mg/dL)              float64\n",
       "Alanine transaminase (U/L)          float64\n",
       "Aspartate transaminase (U/L)        float64\n",
       "Gamma glutamyl transferase (U/L)    float64\n",
       "Alkaline phosphatase (U/L)          float64\n",
       "Total Proteins (g/dL)               float64\n",
       "Creatinine (mg/dL)                  float64\n",
       "Number of Nodules                   float64\n",
       "Major dimension of nodule (cm)      float64\n",
       "Direct Bilirubin (mg/dL)            float64\n",
       "Iron                                float64\n",
       "Oxygen Saturation (%)               float64\n",
       "Ferritin (ng/mL)                    float64\n",
       "Class Attribute                       int64\n",
       "dtype: object"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "td.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Gender                                2\n",
       "Symptoms                              2\n",
       "Alcohol                               2\n",
       "Hepatitis B Surface Antigen           2\n",
       "Hepatitis B e Antigen                 2\n",
       "Hepatitis B Core Antibody             2\n",
       "Hepatitis C Virus Antibody            2\n",
       "Cirrhosis                             2\n",
       "Endemic Countries                     2\n",
       "Smoking                               2\n",
       "Diabetes                              2\n",
       "Obesity                               2\n",
       "Hemochromatosis                       2\n",
       "Arterial Hypertension                 2\n",
       "Chronic Renal Insufficiency           2\n",
       "Human Immunodeficiency Virus          2\n",
       "Nonalcoholic Steatohepatitis          2\n",
       "Esophageal Varices                    2\n",
       "Splenomegaly                          2\n",
       "Portal Hypertension                   2\n",
       "Portal Vein Thrombosis                2\n",
       "Liver Metastasis                      2\n",
       "Radiological Hallmark                 2\n",
       "Age at diagnosis                     51\n",
       "Grams of Alcohol per day             19\n",
       "Packs of cigarets per year           30\n",
       "Performance Status*                   5\n",
       "Encephalopathy degree*                3\n",
       "Ascites degree*                       3\n",
       "International Normalised Ratio*      87\n",
       "Alpha-Fetoprotein (ng/mL)           132\n",
       "Haemoglobin (g/dL)                   71\n",
       "Mean Corpuscular Volume             128\n",
       "Leukocytes(G/L)                     105\n",
       "Platelets                           131\n",
       "Albumin (mg/dL)                      41\n",
       "Total Bilirubin(mg/dL)               62\n",
       "Alanine transaminase (U/L)           93\n",
       "Aspartate transaminase (U/L)        107\n",
       "Gamma glutamyl transferase (U/L)    139\n",
       "Alkaline phosphatase (U/L)          124\n",
       "Total Proteins (g/dL)                46\n",
       "Creatinine (mg/dL)                   84\n",
       "Number of Nodules                     6\n",
       "Major dimension of nodule (cm)       68\n",
       "Direct Bilirubin (mg/dL)             41\n",
       "Iron                                 68\n",
       "Oxygen Saturation (%)                57\n",
       "Ferritin (ng/mL)                     84\n",
       "Class Attribute                       2\n",
       "dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "td.nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assess Missingness in Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Symptoms  \t\t% Missing(N/A) =  10.909090909090908\n",
      "Hepatitis B Surface Antigen \t\t% Missing(N/A) =  10.303030303030303\n",
      "Hepatitis B e Antigen \t\t% Missing(N/A) =  23.636363636363637\n",
      "Hepatitis B Core Antibody \t\t% Missing(N/A) =  14.545454545454545\n",
      "Hepatitis C Virus Antibody \t\t% Missing(N/A) =  5.454545454545454\n",
      "Endemic Countries \t\t% Missing(N/A) =  23.636363636363637\n",
      "Smoking \t\t% Missing(N/A) =  24.848484848484848\n",
      "Diabetes \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Obesity \t\t% Missing(N/A) =  6.0606060606060606\n",
      "Hemochromatosis \t\t% Missing(N/A) =  13.939393939393941\n",
      "Arterial Hypertension \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Chronic Renal Insufficiency \t\t% Missing(N/A) =  1.2121212121212122\n",
      "Human Immunodeficiency Virus \t\t% Missing(N/A) =  8.484848484848486\n",
      "Nonalcoholic Steatohepatitis \t\t% Missing(N/A) =  13.333333333333334\n",
      "Esophageal Varices \t\t% Missing(N/A) =  31.515151515151512\n",
      "Splenomegaly \t\t% Missing(N/A) =  9.090909090909092\n",
      "Portal Hypertension \t\t% Missing(N/A) =  6.666666666666667\n",
      "Portal Vein Thrombosis \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Liver Metastasis \t\t% Missing(N/A) =  2.4242424242424243\n",
      "Radiological Hallmark \t\t% Missing(N/A) =  1.2121212121212122\n",
      "Grams of Alcohol per day \t\t% Missing(N/A) =  29.09090909090909\n",
      "Packs of cigarets per year \t\t% Missing(N/A) =  32.121212121212125\n",
      "Encephalopathy degree* \t\t% Missing(N/A) =  0.6060606060606061\n",
      "Ascites degree* \t\t% Missing(N/A) =  1.2121212121212122\n",
      "International Normalised Ratio* \t\t% Missing(N/A) =  2.4242424242424243\n",
      "Alpha-Fetoprotein (ng/mL) \t\t% Missing(N/A) =  4.848484848484849\n",
      "Haemoglobin (g/dL) \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Mean Corpuscular Volume \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Leukocytes(G/L) \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Platelets \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Albumin (mg/dL) \t\t% Missing(N/A) =  3.6363636363636362\n",
      "Total Bilirubin(mg/dL) \t\t% Missing(N/A) =  3.0303030303030303\n",
      "Alanine transaminase (U/L) \t\t% Missing(N/A) =  2.4242424242424243\n",
      "Aspartate transaminase (U/L) \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Gamma glutamyl transferase (U/L) \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Alkaline phosphatase (U/L) \t\t% Missing(N/A) =  1.8181818181818181\n",
      "Total Proteins (g/dL) \t\t% Missing(N/A) =  6.666666666666667\n",
      "Creatinine (mg/dL) \t\t% Missing(N/A) =  4.242424242424243\n",
      "Number of Nodules \t\t% Missing(N/A) =  1.2121212121212122\n",
      "Major dimension of nodule (cm) \t\t% Missing(N/A) =  12.121212121212121\n",
      "Direct Bilirubin (mg/dL) \t\t% Missing(N/A) =  26.666666666666668\n",
      "Iron \t\t% Missing(N/A) =  47.878787878787875\n",
      "Oxygen Saturation (%) \t\t% Missing(N/A) =  48.484848484848484\n",
      "Ferritin (ng/mL) \t\t% Missing(N/A) =  48.484848484848484\n"
     ]
    }
   ],
   "source": [
    "for i,c in enumerate(td.columns):\n",
    "    na_percent = np.sum(td[c].isnull())/len(td)*100\n",
    "    if na_percent > 0:\n",
    "        print(c,'\\t\\t% Missing(N/A) = ',np.sum(td[c].isnull())/len(td)*100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 0, 'Missing Value Counts')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Frequency')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Histogram of Missing Value Counts In Feature Set')"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAs0AAAEWCAYAAACdXqrwAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAjkUlEQVR4nO3de7wcdX3/8ddb4g2CoKJpBTRiKVZBEaLVWm0i1YIoaqsVixdslVq1Xqq/Nvqwam1ptfVSvLQVWysIEhEVUdSC2EC1KiWIBgSLVZSbgBcuARTQz++PmWOWkz1ndsLZsxvyej4e55Gd2Zn5fveze07e+93vzqSqkCRJkjS3O0y6A5IkSdK0MzRLkiRJHQzNkiRJUgdDsyRJktTB0CxJkiR1MDRLkiRJHQzN0hYmyXlJVk66H5OU5GlJLk6yIcnDbsNxPpPkebdh/39J8pebu/84JFmZ5JJJ90OSbm8MzdIUSXJRkt+ete7QJF+YWa6qB1fV2o7jLE9SSZaMqauT9lbgpVW1tKq+OvvO9rFfMfj4kyxJcmWSX5ycvqoOqKqjNrcTVfWiqvrrzd1/mCR3SXJ1kscNue8dSU5YyPb6SONlSc5Ncn2SS5J8JMleY253s1/PC7TvhoGfr/U9zkL15za0+eAkpyT5cfvaWpfkiSPuu8nfJGlrZWiW1NsUhPH7Aed1bHM1cMDA8hOBH4+rQwulqn4CfBh47uD6JNsAzwI2O+QvgCOAlwMvA+4B/CpwInDgBPu0GHZs36AtraqHTrIj7RuXvv93fxI4FVgG3Jvm+bt2ofsm3d4ZmqUtzODIT5JHJDkrybXtyOrb283OaP+9uh0de1SSOyR5XZLvtiOuRyfZYeC4z23v+2GSv5zVzhuTnJDkmCTXAoe2bX+pHbm6PMm7k9xp4HiV5MVJLkxyXZK/TvKAdp9rkxw/uP2sxzi0r0nunGQDsA3wtST/N0+pPsitg+dzgaNntbM2yQva27+S5PQk1yT5QZIPt+vTjvBe2d739SR7tvd9IMnftLdXtiOvr2q3vTzJ8wfaumeST7aP/X+S/E0GPkGY5Sjg95JsO7Dud2j+Zn8myfOTnN/W9dtJ/niuIrTPw68MLP+iz+3yk5Kc0z6P/53kIXMcZ3fgJcCzqurzVfXTqrqhqo6tqje32+zQPldXtc/d62YCXvsaOmbgeLcacW2fi79O8sX2cZ2SZKd282Gv56HPV5f28b8nycltO19J8oBR9p11nAcmOTXJj5J8M8nvD9x3YJKvts/1xUneOLDrsMcySm0OT/JF4AZgt/nan9XPnYD7A++rqpvany9W1RcGthn6GkjyQeC+wCfbvv553zpJtyeGZmnLdgRwRFXdDXgAcHy7/rHtvzMjZF8CDm1/VgG7AUuBdwMkeRDwT8AhwC8DOwA7z2rrKcAJwI7AscDPgFcCOwGPAvYDXjxrn/2BfYFHAn8OHNm2sSuwJ83I6TBD+9oGtaXtNg+tqvnCzonAY5PsmGRH4DHAJ+bZ/q+BU4C7A7sA72rXP4Gmnr9K89ifCfxwjmP8Ehtr90fAe5Lcvb3vPcD17TbPa3+Gqqr/Bi4Hfndg9XOAD1XVLcCVwJOAuwHPB96RZJ95HttQ7T7vB/4YuCfwXuCkJHcesvl+wCVVdeY8h3wXzePfDfgtmjcqz59n+9n+oN3+3sCdgFe364e9nud6vkbxLOCv2n2/BRzeY1+SbEczcvuhtq/PAv4pyYPbTa6neew70ozC/0mSp87zWEbxHOAwYHvgqo72B/2Q5jEek+SpSZbNeixzvgaq6jnA94Ant339+xH7Kt0uGZql6XNiO+JzdZKracLsXG4GfiXJTlW1oaq+PM+2hwBvr6pvV9UG4DXAwe1o1tOBT1bVF6rqJuD1QM3a/0tVdWJV/byqbqyqdVX15aq6paouovnP9rdm7fOWqrq2qs4DzgVOadu/BvgMMNeX+Obr66h+QvOx9DOBg4GT2nVzuZlm2sd9quonAyNxN9MElQcCqarzq+ryeY7xpqq6uao+DWwA9kgzteL3gDe0o7PfoHuaxdG0I+VJ7kbzpuUogKo6uar+rxqn04THx3Qcb5gXAu+tqq9U1c/a+d0/pXmTM9s9aYL8UO1jfCbwmqq6rn1NvI0m7I3q36vqf6vqRpo3gHvPs+1cz9coPlZVZ7ZvQI7taAfgBwO/k6+mecNyUVX9e/v6Pxv4KM3vEVW1tqrWt78rXweOY9Pfjb4+UFXntX3ef772B1VV0bz5vIjm+bg8yRntJwfQ7zUgbdUMzdL0eWpV7Tjzw6ajt4P+iGYE9IL2I/8nzbPtfYDvDix/F1hCM8/xPsDFM3dU1Q1sOpp68eBCkl9N8qkk308zZeNvaUadB10xcPvGIctLGW6+vvYxEzw3mZoxxJ8DAc5Mc4aSPwSoqs/TjMi/B7giyZFtiB3mh22omXEDzWO8V9v/wRreqp5z9H1Vkp1pwtC3Zr70mOSAJF9uP5q/mma+9uzaj+J+wKtmvUnblab+s/2Q5lOIuexEMzo8+3mb/YnFfL4/cHumdnMZ+nyNoR2AnQZ+J99KU7dfn1W3Q2g+RSDJryf5z3aayjXAi9i852fQ4Otl3vZnq6pLquql7Scz96MZCT964FijvgakrZqhWdqCVdWFVfUsmo9o3wKc0H50PHuUGOAymv8gZ9wXuIUmyF5O8xE3AEnuSjOyeKvmZi3/M3ABsHs7PeS1NCFmIczX1z7+iyboLQPmHYmsqu9X1Qur6j40H1X/U9q5wFX1zqraF3gwzZuU/9ezH1e1/d9lYN2uHf35Xtv/Q2hGa48GaKdOfJTmDCLL2jdWn2bu2t8ADM6NHgxWFwOHD75Jq6ptq+q4Icc5DdglyYo52vkBG0d/Z9wXuLS9ff08/eiyyet5vudrEVwMnD6rbkur6k/a+z9E88nGrlW1A/AvbHx+hv1ujlKbwf262p9TVV1M8wZwz4FjzfcaGNZfaatkaJa2YEmeneReVfVzmrNFQDPX+Crg5zRzS2ccB7wyyf2TLKUZGf5wOzJ6AvDkJL+R5st5f0V3AN6e5hv4G5I8EOj8D7uH+fo6svaj6ScDB7W355TkGUlmQu2PacLCz5I8vB05vCNNuPkJTY379ONnwMeANybZtq3Xczt2g2Y6xkuBR9NMI4BmNPfOtEE8yQE0867ncg7wB0m2SbI/t54m8D7gRe3jS5Lt2i+xbT/kMVxIM1XouDRferxTmtPjHZxkdfsYjwcOT7J9kvsBfwbMfMHtHJo55vdN8wXU14zw+Gds8nqe6/nqcczb4lPAryZ5TpI7tj8PT/Jr7f3bAz+qqp8keQTNXO0Zw343z6Ffbbra/4Ukd0/yV2m+OHmHNF8M/ENgZipX12vgill9lbZahmZpy7Y/cF6aM0ocARzczu+8gebLTV9sP3J9JM2XfT5I8+3979CEvz8FaOcc/ymwhmbU+TqaL5v9dJ62X00TBq6j+Y93pLMXjGjOvvbVzgPtOj0dwMOBr7S1PAl4eVV9h+bLdu+jCWbfpZmm8NbN6MpLab4k932ax3Yc89cXmjczdwdOm5lHXVXX0Zwy7Pi2T3/Q9ncuL6d543A1zaj1iTN3VNVZNHNa390e61s0X8Ccy8vYOFXlauD/gKfRzB2H5jm6Hvg2zcj+h2ieS6rqVJrXyNeBdTTBbyRzvJ7ner7Grn0OnkAzV/4ymuf0LTRvZqCZUvWmJNfRfD/g+IF9N3ksfWszQvuDbgKWA5+jeZN7Ls3r7tD2WF2vgb8DXjcwn1vaaqVj8EXSVqgd3b2aZurFogSRrU2StwC/VFWbfUVCSdLicaRZEgBJntxOHdiOZiR1Pc037rUA0pxX9yHtR+CPoPkS58cn3S9J0mgMzZJmPIXmo97LgN1ppnr4UdTC2Z5mXvP1NB/Xv435zxstSZoiTs+QJEmSOjjSLEmSJHXoc3Wtidlpp51q+fLli97u9ddfz3bbbbfo7W6prFd/1qwf69WP9erHevVjvfqxXv1Msl7r1q37QVXda/b6LSI0L1++nLPOOmvR2127di0rV65c9Ha3VNarP2vWj/Xqx3r1Y736sV79WK9+JlmvJN8dtt7pGZIkSVIHQ7MkSZLUwdAsSZIkdTA0S5IkSR0MzZIkSVIHQ7MkSZLUYWyhOcn7k1yZ5NyBdfdIcmqSC9t/7z6u9iVJkqSFMs6R5g8A+89atxo4rap2B05rlyVJkqSpNrbQXFVnAD+atfopwFHt7aOAp46rfUmSJGmhpKrGd/BkOfCpqtqzXb66qnYcuP/HVTV0ikaSw4DDAJYtW7bvmjVrxtbPuWzYsIGlS5eO5djrL73mF7f32nmHsbSx2MZZr9sra9aP9erHevVjvfqxXv1Yr34mWa9Vq1atq6oVs9dP7WW0q+pI4EiAFStW1CQupTjOSzgeuvrkX9y+6JDxtLHYvERof9asH+vVj/Xqx3r1Y736sV79TGO9FvvsGVck+WWA9t8rF7l9SZIkqbfFDs0nAc9rbz8P+MQity9JkiT1Ns5Tzh0HfAnYI8klSf4IeDPw+CQXAo9vlyVJkqSpNrY5zVX1rDnu2m9cbUqSJEnj4BUBJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOkwkNCd5ZZLzkpyb5Lgkd5lEPyRJkqRRLHpoTrIz8DJgRVXtCWwDHLzY/ZAkSZJGNanpGUuAuyZZAmwLXDahfkiSJEmdUlWL32jycuBw4EbglKo6ZMg2hwGHASxbtmzfNWvWLG4ngQ0bNrB06dKxHHv9pdf84vZeO+8wljYW2zjrdXtlzfqxXv1Yr36sVz/Wqx/r1c8k67Vq1ap1VbVi9vpFD81J7g58FHgmcDXwEeCEqjpmrn1WrFhRZ5111uJ0cMDatWtZuXLlWI69fPXJv7h90ZsPHEsbi22c9bq9smb9WK9+rFc/1qsf69WP9epnkvVKMjQ0T2J6xm8D36mqq6rqZuBjwG9MoB+SJEnSSCYRmr8HPDLJtkkC7AecP4F+SJIkSSNZ9NBcVV8BTgDOBta3fThysfshSZIkjWrJJBqtqjcAb5hE25IkSVJfXhFQkiRJ6mBoliRJkjoYmiVJkqQOhmZJkiSpg6FZkiRJ6mBoliRJkjoYmiVJkqQOhmZJkiSpg6FZkiRJ6mBoliRJkjoYmiVJkqQOhmZJkiSpg6FZkiRJ6mBoliRJkjoYmiVJkqQOhmZJkiSpg6FZkiRJ6mBoliRJkjoYmiVJkqQOhmZJkiSpg6FZkiRJ6mBoliRJkjoYmiVJkqQOhmZJkiSpg6FZkiRJ6mBoliRJkjoYmiVJkqQOhmZJkiSpw0ihOcme4+6IJEmSNK1GHWn+lyRnJnlxkh3H2SFJkiRp2owUmqvqN4FDgF2Bs5J8KMnjx9ozSZIkaUqMPKe5qi4EXgf8BfBbwDuTXJDkd8fVOUmSJGkajDqn+SFJ3gGcDzwOeHJV/Vp7+x19G02yY5IT2tB9fpJH9T2GJEmStFiWjLjdu4H3Aa+tqhtnVlbVZUletxntHgF8tqqenuROwLabcQxJkiRpUYwamp8I3FhVPwNIcgfgLlV1Q1V9sE+DSe4GPBY4FKCqbgJu6nMMSZIkaTGNOqf5c8BdB5a3bddtjt2Aq4B/T/LVJP+aZLvNPJYkSZI0dqmq7o2Sc6pq7651IzWYrAC+DDy6qr6S5Ajg2qr6y1nbHQYcBrBs2bJ916xZ07ep2+zKH13DFTduXN5r5x0W7NjrL71mLMedpA0bNrB06dJJd2OLYs36sV79WK9+rFc/1qsf69XPJOu1atWqdVW1Yvb6UadnXJ9kn6o6GyDJvsCNHfvM5RLgkqr6Srt8ArB69kZVdSRwJMCKFStq5cqVm9nc5nvXsZ/gbes3luiiQxauD4euPnksx52ktWvXMonnaUtmzfqxXv1Yr36sVz/Wqx/r1c801mvU0PwK4CNJLmuXfxl45uY0WFXfT3Jxkj2q6pvAfsA3NudYkiRJ0mIYKTRX1f8keSCwBxDggqq6+Ta0+6fAse2ZM74NPP82HEuSJEkaq1FHmgEeDixv93lYEqrq6M1ptKrOATaZKyJJkiRNo5FCc5IPAg8AzgF+1q4uYLNCsyRJkrQlGXWkeQXwoBrlVBuSJEnS7cyo52k+F/ilcXZEkiRJmlajjjTvBHwjyZnAT2dWVtVBY+mVJEmSNEVGDc1vHGcnJEmSpGk26innTk9yP2D3qvpckm2BbcbbNUmSJGk6jDSnOckLaa7c99521c7AiWPqkyRJkjRVRv0i4EuARwPXAlTVhcC9x9UpSZIkaZqMGpp/WlU3zSwkWUJznmZJkiTpdm/U0Hx6ktcCd03yeOAjwCfH1y1JkiRpeowamlcDVwHrgT8GPg28blydkiRJkqbJqGfP+DnwvvZHkiRJ2qqMFJqTfIchc5irarcF75EkSZI0ZUa9uMmKgdt3AZ4B3GPhuyNJkiRNn5HmNFfVDwd+Lq2qfwQeN96uSZIkSdNh1OkZ+wws3oFm5Hn7sfRIkiRJmjKjTs9428DtW4CLgN9f8N5IkiRJU2jUs2esGndHJEmSpGk16vSMP5vv/qp6+8J0R5IkSZo+fc6e8XDgpHb5ycAZwMXj6JQkSZI0TUYNzTsB+1TVdQBJ3gh8pKpeMK6OSZIkSdNi1Mto3xe4aWD5JmD5gvdGkiRJmkKjjjR/EDgzycdprgz4NODosfVKkiRJmiKjnj3j8CSfAR7Trnp+VX11fN2SJEmSpseo0zMAtgWuraojgEuS3H9MfZIkSZKmykihOckbgL8AXtOuuiNwzLg6JUmSJE2TUUeanwYcBFwPUFWX4WW0JUmStJUYNTTfVFVF8yVAkmw3vi5JkiRJ02XU0Hx8kvcCOyZ5IfA54H3j65YkSZI0PTrPnpEkwIeBBwLXAnsAr6+qU8fcN0mSJGkqdIbmqqokJ1bVvoBBWZIkSVudUadnfDnJw8faE0mSJGlKjXpFwFXAi5JcRHMGjdAMQj9kXB2TJEmSpsW8oTnJfavqe8ABC91wkm2As4BLq+pJC318SZIkaaF0jTSfCOxTVd9N8tGq+r0FbPvlwPnA3RbwmJIkSdKC65rTnIHbuy1Uo0l2AQ4E/nWhjilJkiSNS5prlsxxZ3J2Ve0z+/ZtbjQ5Afg7mqsKvnrY9IwkhwGHASxbtmzfNWvWLETTvVz5o2u44saNy3vtvMPI+66/9JpbLc/ed/D+PsedZhs2bGDp0qWT7sYWxZr1Y736sV79WK9+rFc/1qufSdZr1apV66pqxez1XdMzHprkWpoR57u2t2HjFwF7T61I8iTgyqpal2TlXNtV1ZHAkQArVqyolSvn3HRs3nXsJ3jb+o0luuiQ0ftw6OqTb7U8e9/B+/scd5qtXbuWSTxPWzJr1o/16sd69WO9+rFe/VivfqaxXvOG5qraZgxtPho4KMkTgbsAd0tyTFU9ewxtSZIkSbfZqOdpXjBV9Zqq2qWqlgMHA583MEuSJGmaLXpoliRJkrY0o17cZCyqai2wdpJ9kCRJkro40ixJkiR1MDRLkiRJHQzNkiRJUgdDsyRJktTB0CxJkiR1MDRLkiRJHQzNkiRJUgdDsyRJktTB0CxJkiR1MDRLkiRJHQzNkiRJUgdDsyRJktTB0CxJkiR1MDRLkiRJHQzNkiRJUoclk+6ANrV89cm3Wr7ozQdOqCeSJEkCR5olSZKkToZmSZIkqYOhWZIkSepgaJYkSZI6GJolSZKkDoZmSZIkqYOhWZIkSepgaJYkSZI6GJolSZKkDoZmSZIkqYOhWZIkSepgaJYkSZI6GJolSZKkDoZmSZIkqYOhWZIkSepgaJYkSZI6LHpoTrJrkv9Mcn6S85K8fLH7IEmSJPWxZAJt3gK8qqrOTrI9sC7JqVX1jQn0RZIkSeq06CPNVXV5VZ3d3r4OOB/YebH7IUmSJI0qVTW5xpPlwBnAnlV17az7DgMOA1i2bNm+a9asWfT+Xfmja7jixo3Le+28w8j7rr/0mlstz9538P757uvb7iRt2LCBpUuXbrJ+vse6tZurZhrOem00yu+V9erHevVjvfqxXvObnX3uv8M2E6vXqlWr1lXVitnrJxaakywFTgcOr6qPzbftihUr6qyzzlqcjg1417Gf4G3rN85guejNB4687/LVJ99qefa+g/fPd1/fdidp7dq1rFy5cpP18z3Wrd1cNdNw1mujUX6vrFc/1qsf69WP9Zrf7Ozzgf23m1i9kgwNzRM5e0aSOwIfBY7tCsySJEnSpE3i7BkB/g04v6revtjtS5IkSX1NYqT50cBzgMclOaf9eeIE+iFJkiSNZNFPOVdVXwCy2O1KkiRJm8srAkqSJEkdDM2SJElSB0OzJEmS1MHQLEmSJHUwNEuSJEkdDM2SJElSB0OzJEmS1MHQLEmSJHUwNEuSJEkdDM2SJElSB0OzJEmS1MHQLEmSJHUwNEuSJEkdDM2SJElSB0OzJEmS1GHJpDuwJVu++uRf3L7ozQcuepvD2p2vT/PtuxDHfdVet3Do6pMnUos+j3Va2lm++uTNqtm4XgPaOozrNTAtr60t7e+ytDWZlr8Tm8uRZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKnDREJzkv2TfDPJt5KsnkQfJEmSpFEtemhOsg3wHuAA4EHAs5I8aLH7IUmSJI1qEiPNjwC+VVXfrqqbgDXAUybQD0mSJGkkqarFbTB5OrB/Vb2gXX4O8OtV9dJZ2x0GHNYu7gF8c1E72tgJ+MEE2t1SWa/+rFk/1qsf69WP9erHevVjvfqZZL3uV1X3mr1yyQQ6kiHrNknuVXUkcOT4uzO3JGdV1YpJ9mFLYr36s2b9WK9+rFc/1qsf69WP9epnGus1iekZlwC7DizvAlw2gX5IkiRJI5lEaP4fYPck909yJ+Bg4KQJ9EOSJEkayaJPz6iqW5K8FPgPYBvg/VV13mL3Y0QTnR6yBbJe/VmzfqxXP9arH+vVj/Xqx3r1M3X1WvQvAkqSJElbGq8IKEmSJHUwNEuSJEkdDM1z8FLf80vy/iRXJjl3YN09kpya5ML237tPso/TJMmuSf4zyflJzkvy8na9NRsiyV2SnJnka229/qpdb73mkWSbJF9N8ql22XrNIclFSdYnOSfJWe066zWHJDsmOSHJBe3fsUdZr+GS7NG+rmZ+rk3yCus1tySvbP/Wn5vkuPb/gKmrl6F5CC/1PZIPAPvPWrcaOK2qdgdOa5fVuAV4VVX9GvBI4CXta8qaDfdT4HFV9VBgb2D/JI/EenV5OXD+wLL1mt+qqtp74Fyw1mtuRwCfraoHAg+leZ1ZryGq6pvt62pvYF/gBuDjWK+hkuwMvAxYUVV70pwk4mCmsF6G5uG81HeHqjoD+NGs1U8BjmpvHwU8dTH7NM2q6vKqOru9fR3Nfzg7Y82GqsaGdvGO7U9hveaUZBfgQOBfB1Zbr36s1xBJ7gY8Fvg3gKq6qaquxnqNYj/g/6rqu1iv+SwB7ppkCbAtzfU7pq5ehubhdgYuHli+pF2n+S2rqsuhCYnAvSfcn6mUZDnwMOArWLM5tVMNzgGuBE6tKus1v38E/hz4+cA66zW3Ak5Jsi7JYe066zXcbsBVwL+303/+Ncl2WK9RHAwc1962XkNU1aXAW4HvAZcD11TVKUxhvQzNw410qW+pryRLgY8Cr6iqayfdn2lWVT9rP97cBXhEkj0n3KWpleRJwJVVtW7SfdmCPLqq9qGZhveSJI+ddIem2BJgH+Cfq+phwPVMwUfl0669gNtBwEcm3Zdp1s5Vfgpwf+A+wHZJnj3ZXg1naB7OS31vniuS/DJA+++VE+7PVElyR5rAfGxVfaxdbc06tB8Dr6WZQ2+9hns0cFCSi2imkz0uyTFYrzlV1WXtv1fSzDd9BNZrLpcAl7Sf9gCcQBOirdf8DgDOrqor2mXrNdxvA9+pqquq6mbgY8BvMIX1MjQP56W+N89JwPPa288DPjHBvkyVJKGZD3h+Vb194C5rNkSSeyXZsb19V5o/qhdgvYaqqtdU1S5VtZzm79Xnq+rZWK+hkmyXZPuZ28ATgHOxXkNV1feBi5Ps0a7aD/gG1qvLs9g4NQOs11y+Bzwyybbt/5X70XzvZ+rq5RUB55DkiTRzBGcu9X34ZHs0XZIcB6wEdgKuAN4AnAgcD9yX5pfgGVU1+8uCW6Ukvwn8F7CejXNOX0szr9mazZLkITRf/NiG5s398VX1piT3xHrNK8lK4NVV9STrNVyS3WhGl6GZevChqjrces0tyd40XzK9E/Bt4Pm0v5tYr00k2Zbmu1G7VdU17TpfX3NoTyv6TJozTX0VeAGwlCmrl6FZkiRJ6uD0DEmSJKmDoVmSJEnqYGiWJEmSOhiaJUmSpA6GZkmSJKmDoVmSWkkqyQcHlpckuSrJp9rlg5L0vhJakv9egL5tl+SHSXaYtf7EJL8/z34bbmvbA8d6bpJzk5yX5BtJXr1Qxx5o47ULfUxJWgiGZkna6Hpgz/aCKgCPBy6dubOqTqqqN/c9aFX9xm3tWFVdD5wCPHVmXRugfxP41G09fpckBwCvAJ5QVQ+muSLcNWNoytAsaSoZmiXp1j4DHNjevtUVvZIcmuTd7e1ntKOuX0tyRrvuwUnOTHJOkq8n2b1dv6H9d2WStUlOSHJBkmPbK2CR5Intui8keefM6PYsx9Fc8W/G04DPAndIclqSs5OsT/KU2Tu2bX9qYPndSQ5tb++b5PQk65L8x8yla2d5Dc1FU2YuP/2Tqnpfu//eSb7cPuaPJ7l7u35tkhXt7Z3ay3zP1PFjST6b5MIkf9+ufzNw17Z+x7aj6ye3NT43yTPneM4kaewMzZJ0a2uAg5PcBXgIzVUbh3k98DtV9VDgoHbdi4AjqmpvYAVwyZD9HkYzYvsgYDfg0W1b7wUOqKrfBO41R5ufBfZtrywGTYA+DvgJ8LSq2gdYBbxtJox3SXJH4F3A06tqX+D9wLAroO4JrJvjMEcDf1FVD6G56uUbRmh6b5orgO0FPDPJrlW1GrixqvauqkOA/YHLquqhVbUnzeOXpIkwNEvSgKr6OrCcZpT50/Ns+kXgA0leSHO5b4AvAa9N8hfA/arqxiH7nVlVl1TVz4Fz2rYeCHy7qr7TbnPckP2oqpuAk4CnJ9mJJnieAgT42yRfBz4H7AwsG+XxAnvQBOJTk5wDvA7YZcR9Z6aI7FhVp7erjgIeO8Kup1XVNVX1E+AbwP2GbLMe+O0kb0nymJnLEUvSJBiaJWlTJwFvZY7wClBVL6IJmLsC5yS5Z1V9iGbU+UbgP5I8bsiuPx24/TNgCU3oHdXMFI2nA5+oqpuBQ2hGp/dtR7mvAO4ya79buPXf/Jn7A5zXju7uXVV7VdUThrR7HrBvj37ObnN2f4bV4Vaq6n/bNtcDf5fk9T3bl6QFY2iWpE29H3hTVa2fa4MkD6iqr1TV64EfALsm2Y1mxPidNMH7ISO2dwGwW5Ll7fJ8c3f/E9gdeAkbQ/0OwJVVdXOSVQwftf0u8KAkd25Hh/dr138TuFeSR7WP645JHjxk/78D/j7JL7Xb3TnJy9rR3x8neUy73XOAmVHni9gYtJ8+z2MadHM7ZYQk9wFuqKpjaN7E7DPiMSRpwW3yzl6StnZVdQlwRMdm/9B+0S/AacDXgNXAs5PcDHwfeNOI7d2Y5MXAZ5P8ADhznm1/nuSjwDOAM9rVxwKfTHIWzZSPC4bsd3GS44GvAxcCX23X35Tk6cA72zC9BPhHmpHlwf0/nWQZ8Ll2vnTRvLkAeB7wL0m2Bb4NPL9d/1bg+CTPAT4/Si2AI4GvJzmbZq70PyT5OXAz8CcjHkOSFlyqatJ9kKStXpKlVbWhDaTvAS6sqndMul+SpIbTMyRpOryw/SLeeTTTLd472e5IkgY50ixJkiR1cKRZkiRJ6mBoliRJkjoYmiVJkqQOhmZJkiSpg6FZkiRJ6vD/AR/6nL9cBt1pAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 864x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "missing_count = td.isnull().sum()\n",
    "#Plot a histogram of the missingness observed over all features in the dataset\n",
    "ax = missing_count.hist(bins=num_rows,figsize=(12,4))\n",
    "ax.set_xlabel(\"Missing Value Counts\")\n",
    "ax.set_ylabel(\"Frequency\")\n",
    "ax.set_title(\"Histogram of Missing Value Counts In Feature Set\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Data Cleaning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove instances (rows) with missing outcome values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of training instances in original dataset =  165\n",
      "Number of training instances after removing instances with missing endpoint =  165\n",
      "Number of instances removed =  0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(165, 50)"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Number of training instances in original dataset = \",num_rows)\n",
    "td = td.dropna(axis=0,how='any',subset=[outcomeLabel])\n",
    "print(\"Number of training instances after removing instances with missing endpoint = \",len(td))\n",
    "diff = num_rows-len(td)\n",
    "print(\"Number of instances removed = \", diff)\n",
    "num_rows = len(td) #reassign value (if changed)\n",
    "if diff > 0:\n",
    "    data_changed = True\n",
    "td.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(165, 50)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 165 entries, 0 to 164\n",
      "Data columns (total 50 columns):\n",
      " #   Column                            Non-Null Count  Dtype  \n",
      "---  ------                            --------------  -----  \n",
      " 0   Gender                            165 non-null    int64  \n",
      " 1   Symptoms                          147 non-null    float64\n",
      " 2   Alcohol                           165 non-null    int64  \n",
      " 3   Hepatitis B Surface Antigen       148 non-null    float64\n",
      " 4   Hepatitis B e Antigen             126 non-null    float64\n",
      " 5   Hepatitis B Core Antibody         141 non-null    float64\n",
      " 6   Hepatitis C Virus Antibody        156 non-null    float64\n",
      " 7   Cirrhosis                         165 non-null    int64  \n",
      " 8   Endemic Countries                 126 non-null    float64\n",
      " 9   Smoking                           124 non-null    float64\n",
      " 10  Diabetes                          162 non-null    float64\n",
      " 11  Obesity                           155 non-null    float64\n",
      " 12  Hemochromatosis                   142 non-null    float64\n",
      " 13  Arterial Hypertension             162 non-null    float64\n",
      " 14  Chronic Renal Insufficiency       163 non-null    float64\n",
      " 15  Human Immunodeficiency Virus      151 non-null    float64\n",
      " 16  Nonalcoholic Steatohepatitis      143 non-null    float64\n",
      " 17  Esophageal Varices                113 non-null    float64\n",
      " 18  Splenomegaly                      150 non-null    float64\n",
      " 19  Portal Hypertension               154 non-null    float64\n",
      " 20  Portal Vein Thrombosis            162 non-null    float64\n",
      " 21  Liver Metastasis                  161 non-null    float64\n",
      " 22  Radiological Hallmark             163 non-null    float64\n",
      " 23  Age at diagnosis                  165 non-null    int64  \n",
      " 24  Grams of Alcohol per day          117 non-null    float64\n",
      " 25  Packs of cigarets per year        112 non-null    float64\n",
      " 26  Performance Status*               165 non-null    int64  \n",
      " 27  Encephalopathy degree*            164 non-null    float64\n",
      " 28  Ascites degree*                   163 non-null    float64\n",
      " 29  International Normalised Ratio*   161 non-null    float64\n",
      " 30  Alpha-Fetoprotein (ng/mL)         157 non-null    float64\n",
      " 31  Haemoglobin (g/dL)                162 non-null    float64\n",
      " 32  Mean Corpuscular Volume           162 non-null    float64\n",
      " 33  Leukocytes(G/L)                   162 non-null    float64\n",
      " 34  Platelets                         162 non-null    float64\n",
      " 35  Albumin (mg/dL)                   159 non-null    float64\n",
      " 36  Total Bilirubin(mg/dL)            160 non-null    float64\n",
      " 37  Alanine transaminase (U/L)        161 non-null    float64\n",
      " 38  Aspartate transaminase (U/L)      162 non-null    float64\n",
      " 39  Gamma glutamyl transferase (U/L)  162 non-null    float64\n",
      " 40  Alkaline phosphatase (U/L)        162 non-null    float64\n",
      " 41  Total Proteins (g/dL)             154 non-null    float64\n",
      " 42  Creatinine (mg/dL)                158 non-null    float64\n",
      " 43  Number of Nodules                 163 non-null    float64\n",
      " 44  Major dimension of nodule (cm)    145 non-null    float64\n",
      " 45  Direct Bilirubin (mg/dL)          121 non-null    float64\n",
      " 46  Iron                              86 non-null     float64\n",
      " 47  Oxygen Saturation (%)             85 non-null     float64\n",
      " 48  Ferritin (ng/mL)                  85 non-null     float64\n",
      " 49  Class Attribute                   165 non-null    int64  \n",
      "dtypes: float64(44), int64(6)\n",
      "memory usage: 64.6 KB\n"
     ]
    }
   ],
   "source": [
    "#It's critical to reset the index after dropping rows to work with data frame later with pd.concat of separate columns\n",
    "td = td.reset_index(drop=True)\n",
    "td.shape\n",
    "#Ensure Class variable is cast as an int\n",
    "td[outcomeLabel] = td[outcomeLabel].astype(dtype='int64')\n",
    "td.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "No missing values found in: Class Attribute\n"
     ]
    }
   ],
   "source": [
    "#Confirm no missing values in outcome variable\n",
    "na_percent = np.sum(td[outcomeLabel].isnull())/len(td[outcomeLabel])*100\n",
    "if na_percent > 0:\n",
    "    print(c,'\\t\\t% Missing(N/A) = ',np.sum(td[outcomeLabel].isnull())/len(td)*100)\n",
    "else:\n",
    "    print('No missing values found in: '+ str(outcomeLabel))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Remove columns to be ignored in analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of columns in original dataset =  50\n",
      "Number of columns after removing ignored columns =  50\n",
      "Number of columns removed =  0\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(165, 50)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(\"Number of columns in original dataset = \", num_cols)\n",
    "td = td.drop(ignore_columns, axis=1)\n",
    "print(\"Number of columns after removing ignored columns = \", len(td.columns))\n",
    "diff = num_cols-len(td.columns)\n",
    "print(\"Number of columns removed = \", diff)\n",
    "num_cols = len(td.columns)\n",
    "if diff > 0:\n",
    "    data_changed = True\n",
    "td.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Assess class imbalance (assuming discrete outcome)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Counts of each class\n",
      "0    102\n",
      "1     63\n",
      "Name: Class Attribute, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:>"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0, 0.5, 'Count')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Class Counts (Checking for Imbalance)')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(165, 50)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEFCAYAAAAMk/uQAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAWb0lEQVR4nO3de7hcdX3v8fdHIqKCQCDcL1GJd496Gq13PcU+rccqaIviNVpa9DlFtFBttB61rfbgqfrIsagHLzUWCoJVQaQKRrHeDhAUrRgVtUAiCBsKCF65fM8fv5XFsN1JdkL2rJ3M+/U8+5mZ37p9Z82a9Vnrt2Zmp6qQJAngbkMXIEmaPwwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUNiKJHlzkpOGrmNrkOQrSR41y3EXJ6kkC7ZwDR9O8pb1DHtfkv+5JZc3Mu9nJ1mT5ObZroO5cle22SQvTfLlLV3TJiz/Hkm+m2SPoWoYgqEwzyR5QZJV3Rv6qiT/muSJA9WSJEcn+XaSnyVZm+T0JA+f4+XepZ10kmcCN1XVN0baHtDVfm2SG5N8K8kxSbbbYoVvgqp6RVX97RzN/u3AUVW14+g62FxJzkvyJ1ugrq1KVf0K+BDwl0PXMk6GwjyS5BjgXcDfAXsCBwDvAQ4ZqKTjgVcBRwMLgQcAnwSeMVA9s/UK4J/WPUhyf+B8YA3w8KraGTgMWArsNEiFc+tA4JLNmXCokJzH/hlYluQeQxcyNlXl3zz4A3YGbgYO28A4bwZOGnl8OvAT4Ebg34CHjgz778B3gJuAHwN/0bXvDpwF3AD8J/Al4G4zLGsJcBvwmI3U/BFgCrgceMO6ec1Q62KggAXd4/OAvwW+0tV4DrB7N+yKbtybu7/HAQcBX+ye67XAR9dT0/bAL4D9RtpOAj69geexrrZl3bKvBf5qZPjdgOXAD4HrgNOAhSPDnwh8tVuna4CXdu0fBt7S3d8J+ALwf4BMG/ZUYC1wLHANcBXwspH57wZ8CvgpcCHwFuDLMzyPe3Trq4CfAT/s2h/cre8baGHxrJFpPgy8Fzi7m+ZpM8z3POBPptX62pFaD6Vtb9+nbVOvn7bNfgz4aPc6fx14xMjwdev1Jtr2+uyRYS8dfZ60g5Q13Xq4CHjStOWcRtseb+qe59KR4fsDH6dtq9cB/zAy7I+B1cD1wGeBA6c9/0uBpwy9jxjXn2cK88fjgB2AT2zCNP9K23nvQXuznTwy7IPAy6tqJ+BhwOe79mNpb+pFtLOR19N2ItMdDKytqgs2sPx304LhfsBTgJcAL9uE+l/Qjb8HbWf+F137k7vbXap1gXyNFiDnALsC+3XLnskS4PaqWjvS9jTajmljngg8kPbc35jkwV370bQd31OAfWg7jxMAkhxAex3eTVunjwQuHp1pkt2AlcBXquro6vY00+xFW5f7AkcAJyTZtRt2Am2HvRctuJbNVHxV/aqqduwePqKq7p/k7rRAOYe2nl8JnJzkgSOTvgB4Ky24ZtOHvxdtW90XeCPwfuBFwG8BT6Ktu/uNjH8I7QBmIe3I+5NdXdAC4Undc/9r4KQke69nuRfS1u+6+ZyeZIeR4c8CTgV2Ac4E/gH6s5+zaAcui7u6T+2GHUp7DzyH9vp9CThl2nJXA4/Y8CrZdhgK88duwLVVdetsJ6iqD1XVTdX6Pt8MPCLJzt3gW4CHJLlPVV1fVV8fad+bdjR0S1V9aT07qd1oR4Ez6t5ozwNe19VwGfAO4MWzrR/4x6r6flX9gnaU98gNjHsLrVtkn6r6ZVWtb+e1C+1IcdQGn8uIv66qX1TVN4FvcseO4OW0M4e1I+v6j7prHi8EPldVp3Tr87qqunhknvvQznBOr6o3bOT5/U03j7NpR/wP7NbzHwJvqqqfV9V3gBWzeC7rPBbYETiuqn5dVZ+n7SCfPzLOGVX1laq6vap+OYt53gK8tapuoe1cdweO77aDS2hH6f9lZPyLqupj3fjvpAXKYwGq6vSqurJb9kdpR+WPmWmhVXVSt35vrap30M6MRsPty1V1dlXdRus+XPf6PYb2Orymqn42bft5OfC/qmp19977O+CRSQ4cme9NtO1qIhgK88d1wO6zvbiaZLskxyX5YZKfApd1g3bvbv+Qdkp/eZIvJnlc1/73wA+Ac5L8KMnyDdSzviO2dcvZnnb0tc7ltKOw2frJyP2f03Ze6/NaWrfLBUkuSfLH6xnven7zOsHGnsvG6jkQ+ESSG5LcQDtyvI12prU/7Wh3fZ4B3BN430aWfd20A4J1y18ELKB1m6wzen9j9gHWVNXtI23TX6dNmd+6Wm/r7v+iu716ZPgvuPNr2c+/q2NtVxdJXpLk4pF1+zDu2IbvJMmxSVZ3HxS4gXZ2MTru9Ndvh+79tD9w+XoOuA4Ejh9Z/n/StrPR9bMTrettIhgK88fXgF/Suilm4wW00/Kn0d4ci7v2AFTVhVV1CK3L4JO0I3G6o7ljq+p+wDOBY5IcPMP8VwL7JVm6nuVfyx1H7+scQLt+Aa27414jw/aa5fOCGbqzquonVfWnVbUP7ejuPUkOmmHaS2kfnBp9U3+OFpKbaw3w9KraZeRvh6r6cTfs/huY9v3AZ4Czk9x7M5Y9BdxK6zJbZ/9NmP5KYP8ko+/10dcJZu4+3JL6ers69gOu7I7G3w8cBexWVbsA36bbhkcleRLtU0DPBXbtxr1xpnFnsAY4YD0HXGto3ayjr+09q+qrI+M8mHbmOBEMhXmiqm6k9c+ekOTQJPdKcvckT0/yv2eYZCfgV7Sj4HvRTnsBSLJ9khcm2bk7Zf8p7ciWJH+Q5KAkGWm/bfrMq+pS2iefTkny1G6eOyQ5PMny7kjxNOCtSXbq3uDH0C7qQutXf3KSA7ourddtwuqYAm6nXatY95wOS7Jux3g9bUc2U9230ELgKSPNbwIen+Tvk+zVze+gJCcl2WUW9byP9jwP7KZdlGTdJ8JOBp6W5LlJFiTZLckjp01/FPA94Kwk95zF8kafz220C6Rv7raJB9Gu3czW+bSAfm23PT2VdjBw6qbUcRf9VpLndDvlV9O22/8H3Jv2Ok4BJHkZ7UxhJjvRwnEKWJDkjcB9Zrn8C2jdh8cluXe3HT+hG/Y+4HVJHtrVsHOSw9ZN2B1cLOzqnQiGwjxSVe+k7VjfQNv419B2KJ+cYfSP0LoBfkz71Mb0jfbFwGVd19IraBcCoV2I/Rytz/prwHuq6rz1lHQ07WLdCbTT5x8Cz6ZduIR20fJnwI9oFyj/mfa5bqrqXNonTr5F+6TIWRt7/utU1c9pFz6/0p3WPxZ4NHB+kptpFxFfVVX/sZ5Z/F9Grm1U1Q9pF/IXA5ckuRH4F2AVv3n9YSbHd8s8J8lNtHX92928r6B10x1L63q4mGkXJbtrNkfSXs8zpl0cnY2jaGeDP6H1lZ9C27FuVFX9mnYB9um0s7v3AC+pqu9uYg13xRm060/X016X53TXTr5Duw71NVr308Npn0abyWdpF/S/T9vuf8ksu726YH0m7RNsV9C6r57XDfsE8Dbg1O698m3aulrnBcCK7lrSRMjM1xilrVvaN2FfWVvgy1vzTZK3AXtV1YyfQtKW0X034ZvAk6vqmqHrGRdDQZrnui6j7YF/p50xnU373sAnh6xL26Yt+lsvkubETrQuo31oXxh7B61LRtriPFOQJPW80CxJ6hkKkqTeVn1NYffdd6/FixcPXYYkbVUuuuiia6tq0UzDtupQWLx4MatWrRq6DEnaqiS5fH3D7D6SJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSz1CQJPUMBUlSb6v+8trWYvHyTw9dwjblsuOeMXQJ0jZrzs4UknwoyTVJvj3StjDJuUku7W53HRn2uiQ/SPK9JL83V3VJktZvLruPPgz8/rS25cDKqlpC+8fwywGSPAQ4HHhoN817kmw3h7VJkmYwZ6FQVf9G+5+1ow4BVnT3VwCHjrSfWlW/6v7v7g+Ax8xVbZKkmY37QvOeVXUVQHe7R9e+L3f+J9xru7bfkOTIJKuSrJqamprTYiVp0syXTx9lhrYZ/yVcVZ1YVUuraumiRTP+8qskaTONOxSuTrI3QHd7Tde+Fth/ZLz9gCvHXJskTbxxh8KZwLLu/jLu+OfjZwKHJ7lHkvsCS4ALxlybJE28OfueQpJTgKcCuydZC7wJOA44LckRwBXAYQBVdUmS04DvALcCf1ZVt81VbZKkmc1ZKFTV89cz6OD1jP9W4K1zVY8kaePmy4VmSdI8YChIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpZyhIknqGgiSpN0goJPnzJJck+XaSU5LskGRhknOTXNrd7jpEbZI0ycYeCkn2BY4GllbVw4DtgMOB5cDKqloCrOweS5LGaKjuowXAPZMsAO4FXAkcAqzohq8ADh2mNEmaXGMPhar6MfB24ArgKuDGqjoH2LOqrurGuQrYY9y1SdKkG6L7aFfaWcF9gX2Aeyd50SZMf2SSVUlWTU1NzVWZkjSRhug+ehrwH1U1VVW3AB8HHg9cnWRvgO72mpkmrqoTq2ppVS1dtGjR2IqWpEkwRChcATw2yb2SBDgYWA2cCSzrxlkGnDFAbZI00RaMe4FVdX6SjwFfB24FvgGcCOwInJbkCFpwHDbu2iRp0o09FACq6k3Am6Y1/4p21iBJGojfaJYk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVLPUJAk9QwFSVJvwdAFSBrW4uWfHrqEbcZlxz1j6BLuMs8UJEm9QUIhyS5JPpbku0lWJ3lckoVJzk1yaXe76xC1SdIkG+pM4XjgM1X1IOARwGpgObCyqpYAK7vHkqQxGnsoJLkP8GTggwBV9euqugE4BFjRjbYCOHTctUnSpBviTOF+wBTwj0m+keQDSe4N7FlVVwF0t3sMUJskTbRZhUKSJ8ymbZYWAP8VeG9VPQr4GZvQVZTkyCSrkqyamprazBIkSTOZ7ZnCu2fZNhtrgbVVdX73+GO0kLg6yd4A3e01M01cVSdW1dKqWrpo0aLNLEGSNJMNfk8hyeOAxwOLkhwzMug+wHabs8Cq+kmSNUkeWFXfAw4GvtP9LQOO627P2Jz5S5I238a+vLY9sGM33k4j7T8F/uguLPeVwMlJtgd+BLyMdtZyWpIjgCuAw+7C/CVJm2GDoVBVXwS+mOTDVXX5llpoVV0MLJ1h0MFbahmSpE0325+5uEeSE4HFo9NU1e/MRVGSpGHMNhROB94HfAC4be7KkSQNabahcGtVvXdOK5EkDW62H0n9VJL/kWTv7jeKFiZZOKeVSZLGbrZnCsu629eMtBXt28mSpG3ErEKhqu4714VIkoY3q1BI8pKZ2qvqI1u2HEnSkGbbffTokfs70L5P8HXAUJCkbchsu49eOfo4yc7AP81JRZKkwWzuT2f/HFiyJQuRJA1vttcUPkX7tBG0H8J7MHDaXBUlSRrGbK8pvH3k/q3A5VW1dg7qkSQNaFbdR90P432X9kupuwK/nsuiJEnDmO1/XnsucAHt56yfC5yf5K78dLYkaR6abffRXwGPrqprAJIsAj5H+69pkqRtxGw/fXS3dYHQuW4TppUkbSVme6bwmSSfBU7pHj8POHtuSpIkDWVj/6P5IGDPqnpNkucATwQCfA04eQz1SZLGaGNdQO8CbgKoqo9X1TFV9ee0s4R3zW1pkqRx21goLK6qb01vrKpVtH/NKUnahmwsFHbYwLB7bslCJEnD21goXJjkT6c3JjkCuGhuSpIkDWVjnz56NfCJJC/kjhBYCmwPPHsO65IkDWCDoVBVVwOPT/LfgId1zZ+uqs/PeWWSpLGb7f9T+ALwhTmuRZI0ML+VLEnqGQqSpJ6hIEnqGQqSpJ6hIEnqGQqSpN5goZBkuyTfSHJW93hhknOTXNrd7jpUbZI0qYY8U3gVsHrk8XJgZVUtAVZ2jyVJYzRIKCTZD3gG8IGR5kOAFd39FcChYy5LkibeUGcK7wJeC9w+0rZnVV0F0N3uMUBdkjTRxh4KSf4AuKaqNutXVpMcmWRVklVTU1NbuDpJmmxDnCk8AXhWksuAU4HfSXIScHWSvQG622tmmriqTqyqpVW1dNGiReOqWZImwthDoapeV1X7VdVi4HDg81X1IuBMYFk32jLgjHHXJkmTbj59T+E44HeTXAr8bvdYkjRGs/rp7LlSVecB53X3rwMOHrIeSZp08+lMQZI0MENBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQzFCRJPUNBktQbeygk2T/JF5KsTnJJkld17QuTnJvk0u5213HXJkmTbogzhVuBY6vqwcBjgT9L8hBgObCyqpYAK7vHkqQxGnsoVNVVVfX17v5NwGpgX+AQYEU32grg0HHXJkmTbtBrCkkWA48Czgf2rKqroAUHsMeApUnSRBosFJLsCPwL8Oqq+ukmTHdkklVJVk1NTc1dgZI0gQYJhSR3pwXCyVX18a756iR7d8P3Bq6ZadqqOrGqllbV0kWLFo2nYEmaEEN8+ijAB4HVVfXOkUFnAsu6+8uAM8ZdmyRNugUDLPMJwIuBf09ycdf2euA44LQkRwBXAIcNUJskTbSxh0JVfRnIegYfPM5aJEl35jeaJUk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1DMUJEk9Q0GS1Jt3oZDk95N8L8kPkiwfuh5JmiTzKhSSbAecADwdeAjw/CQPGbYqSZoc8yoUgMcAP6iqH1XVr4FTgUMGrkmSJsaCoQuYZl9gzcjjtcBvj46Q5EjgyO7hzUm+N6baJsHuwLVDF7ExedvQFWgAbptb1oHrGzDfQiEztNWdHlSdCJw4nnImS5JVVbV06Dqk6dw2x2e+dR+tBfYfebwfcOVAtUjSxJlvoXAhsCTJfZNsDxwOnDlwTZI0MeZV91FV3ZrkKOCzwHbAh6rqkoHLmiR2y2m+ctsck1TVxseSJE2E+dZ9JEkakKEgSeoZCpKk3ry60KzxSvIg2jfG96V9H+RK4MyqWj1oYZIG45nChEryl7SfEQlwAe3jwAFO8YcINZ8lednQNWzL/PTRhEryfeChVXXLtPbtgUuqaskwlUkbluSKqjpg6Dq2VXYfTa7bgX2Ay6e1790NkwaT5FvrGwTsOc5aJo2hMLleDaxMcil3/AjhAcBBwFFDFSV19gR+D7h+WnuAr46/nMlhKEyoqvpMkgfQfq58X9qbbS1wYVXdNmhxEpwF7FhVF08fkOS8sVczQbymIEnq+ekjSVLPUJAk9QwFSVLPUJAk9QwFSVLv/wN66tTojd2GegAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"Counts of each class\")\n",
    "print(td[outcomeLabel].value_counts())\n",
    "td[outcomeLabel].value_counts().plot(kind='bar')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Class Counts (Checking for Imbalance)')\n",
    "td.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify categorical variables in dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(165, 50)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "if autoDetectCategorical:\n",
    "    #Separate independent variables from the rest\n",
    "    if instLabel == None or instLabel == 'None':\n",
    "        x_data = td.drop([outcomeLabel], axis=1)\n",
    "    else:\n",
    "        x_data = td.drop([outcomeLabel,instLabel], axis=1)\n",
    "\n",
    "    categorical_variables = identifyCategoricalFeatures(x_data,categoricalCutoff)\n",
    "    #print(categorical_variables)\n",
    "    td.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Example Code for other potential data cleaning/processing\n",
    "- Not applied in this example script, but available here as an example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete Rows/Instances with any missing values\n",
    "- An extreme alternative to imputation when missing values must be eliminated from the dataset prior to machine learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "#num_rows = td.shape[0]\n",
    "#print(\"Number of training instances in original dataset = \",num_rows)\n",
    "#td = td.dropna()\n",
    "#print(\"Number of training instances after removing instances with any missing values = \",td.shape[0])\n",
    "#diff = num_rows-td.shape[0]\n",
    "#if diff > 0:\n",
    "#    data_changed = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete Columns with any missing values\n",
    "- An extreme alternative to imputation when missing values must be eliminated from the dataset prior to machine learning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "#num_cols = td.shape[1]\n",
    "#print(\"Number of training instances in original dataset = \",num_cols)\n",
    "#td = df.dropna(axis='columns')\n",
    "#print(\"Number of training instances after removing instances with any missing values = \",td.shape[1])\n",
    "#diff = num_cols-td.shape[1]\n",
    "#if diff > 0:\n",
    "#    data_changed = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One hot encoding of categorical variables\n",
    "- Ensures that categorical variables are not treated in an ordinal manner by ML algorithms\n",
    "- The code below is an example of how to convert a single column to one hot encoding, and is not meant to be run as is. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Example below assumes a data frame 'df' with a column called 'B'\n",
    "#one_hot = pd.get_dummies(df['B'])\n",
    "#df = df.drop('B',axis = 1)\n",
    "#df = df.join(one_hot)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other possible cleaning steps:\n",
    "- encode text values as numerics\n",
    "- oversample or undersample to reduce class imbalance\n",
    "- drop additional rows based on assigned cutoffs, etc\n",
    "- cast variable types as nominal or ordinal\n",
    "- construct missingness features with MissingIndicator: https://scikit-learn.org/stable/modules/impute.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Saved for later, determines if imputation attempted.\n",
    "isMissingData = False\n",
    "if td.isnull().values.any():\n",
    "    isMissingData = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save cleaned dataset if it has been modified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if data_changed:\n",
    "    print('Saving cleaned data.')\n",
    "    td.shape\n",
    "    data_name = data_name +'_clean'\n",
    "    td.to_csv(data_name+'.txt', index=None, sep='\\t')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Further Exploratory/Descriptive Analysis\n",
    "- Visualize feature correlations in data\n",
    "- Appropriate univariate analyses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature Correlations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if doCorrelation:\n",
    "    #Drop outcome column for correlation analysis\n",
    "    td_cor = td.drop([outcomeLabel], axis=1)\n",
    "    \n",
    "    #Compute correlation between the outcome and each feature , excluding NA/null values.\n",
    "    corrmat = td_cor.corr(method='pearson') \n",
    "    f, ax = plt.subplots(figsize = (40,20)) # Note: need large size\n",
    "    sns.heatmap(corrmat, vmax =1.0,square = True) # Set max value and square-shaped\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Univariate Analysis and Plots\n",
    "Below we have encoded a method to automatically select an appropriate plot and univariate association test between a single feature and the target outcome in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Code to analyze the univariate relationship between the outcome and each feature\n",
    "def test_selector(featureName, outcomeLabel, td, categorical_variables):\n",
    "    p_val = 0\n",
    "    #Feature and Outcome are discrete/categorical/binary\n",
    "    if featureName in categorical_variables:  \n",
    "        #Calculate Contingency Table - Counts\n",
    "        table = pd.crosstab(td[featureName], td[outcomeLabel])\n",
    "        \n",
    "        #Univariate association test (Chi Square Test of Independence - Non-parametric)\n",
    "        c, p, dof, expected = scs.chi2_contingency(table)\n",
    "        p_val = p\n",
    "        \n",
    "    #Feature is continuous and Outcome is discrete/categorical/binary\n",
    "    else: \n",
    "        #Univariate association test (Mann-Whitney Test - Non-parametric)\n",
    "        c, p = scs.mannwhitneyu(x=td[featureName].loc[td[outcomeLabel] == 0],y=td[featureName].loc[td[outcomeLabel] == 1])\n",
    "        p_val = p\n",
    "        \n",
    "    return p_val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Function select appropriate visualizations to examine univariate relationships between each feature and outcome\n",
    "\n",
    "def graph_selector(featureName, outcomeLabel, td, categorical_variables):\n",
    "    #Feature and Outcome are discrete/categorical/binary\n",
    "    if featureName in categorical_variables:  \n",
    "        #Generate contingency table count bar plot. ------------------------------------------------------------------------\n",
    "        #Calculate Contingency Table - Counts\n",
    "        table = pd.crosstab(td[featureName], td[outcomeLabel])\n",
    "        geom_bar_data = pd.DataFrame(table)\n",
    "        mygraph = geom_bar_data.plot(kind='bar')\n",
    "        plt.ylabel('Count')\n",
    "        new_feature_name = featureName.replace(\" \",\"\") #Deal with the dataset specific characters causing problems in this dataset.\n",
    "        new_feature_name = new_feature_name.replace(\"*\",\"\") #Deal with the dataset specific characters causing problems in this dataset.\n",
    "        new_feature_name = new_feature_name.replace(\"/\",\"\") #Deal with the dataset specific characters causing problems in this dataset.\n",
    "        plt.savefig((wd_path+ua_folder+'/'+'UA_Barplot_' + str(new_feature_name)+'_'+outcomeLabel+'_'+data_name), bbox_inches = \"tight\",format='png')\n",
    "        plt.show()\n",
    "        \n",
    "    #Feature is continuous and Outcome is discrete/categorical/binary    \n",
    "    else: \n",
    "        #Generate boxplot-----------------------------------------------------------------------------------------------------\n",
    "        mygraph = td.boxplot(column=featureName,by=outcomeLabel)\n",
    "        plt.ylabel(featureName)\n",
    "        plt.title('')\n",
    "        new_feature_name = featureName.replace(\" \",\"\") #Deal with the dataset specific characters causing problems in this dataset.\n",
    "        new_feature_name = new_feature_name.replace(\"*\",\"\") #Deal with the dataset specific characters causing problems in this dataset.\n",
    "        new_feature_name = new_feature_name.replace(\"/\",\"\") #Deal with the dataset specific characters causing problems in this dataset.\n",
    "        plt.savefig((wd_path+ua_folder+'/'+'UA_Boxplot_' + str(new_feature_name)+'_'+outcomeLabel+'_'+data_name), bbox_inches = \"tight\",format='png')\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if doUA:\n",
    "    p_val_dict = {}\n",
    "    # run appropriate univariate association test for each independent variable\n",
    "    for each in td:\n",
    "        if each != outcomeLabel and each != instLabel: \n",
    "            p_val_dict[each] = test_selector(each, outcomeLabel, td, categorical_variables)\n",
    "    \n",
    "    num_features = td.shape[1] - 1\n",
    "    \n",
    "    sorted_p_list = sorted(p_val_dict.items(), key = lambda item:item[1])\n",
    "    \n",
    "    #Save all p-values to file\n",
    "    pval_df = pd.DataFrame.from_dict(p_val_dict, orient='index')\n",
    "    filepath = wd_path+ua_folder+'/'+'UA_Significance_'+data_name+'.csv'\n",
    "    pval_df.to_csv(filepath, header=['p-value'], index=True)  \n",
    "    \n",
    "    min_num = min(topResults,num_features)\n",
    "    sorted_p_list = sorted_p_list[: min_num]\n",
    "    print('Plotting top significant '+ str(min_num) + ' features.')\n",
    "    \n",
    "    # summarize significant values of selected number of features\n",
    "    print('###################################################')\n",
    "    print('Significant Univariate Associations:')\n",
    "    for each in sorted_p_list[:min_num]:\n",
    "        print(each[0]+\": (p-val = \"+str(each[1]) +\")\")\n",
    "\n",
    "    #generate appropriate descriptive plot for selected number of features\n",
    "    for i in sorted_p_list:\n",
    "        for j in td:\n",
    "            if j == i[0]:# if the feature name is within the selected feature list\n",
    "                graph_selector(j, outcomeLabel, td, categorical_variables) #plot appropriate figure for each feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Create n-fold Cross Validation (CV) Datasets \n",
    "- i.e. Training and Testing Datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create a header list of independant feature names\n",
    "header = td.columns.values.tolist()\n",
    "header.remove(outcomeLabel)\n",
    "if instLabel != None and instLabel != 'None':\n",
    "    header.remove(instLabel)\n",
    "\n",
    "#Perform CV partitioning (get back list of training and testing partition dataframes)\n",
    "train_dfs, test_dfs = cv_partitioner(td, cv_partitions, partition_method, outcomeLabel, categoricalOutcome, matchName, randomSeed)\n",
    "\n",
    "if partition_method == 'M':\n",
    "    header.remove(matchName)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Transform data - Standard Scaling\n",
    "- Features are independently scaled to allow them to be more comparable in downstream logistic regression modeling, etc\n",
    "- Scaling is learned on the training data and saved with pickel so that the same scaling may be applied to any future testing data\n",
    "- Completed within each training and testing partition independently"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if scaleData:\n",
    "    print('Scaling Data')\n",
    "    data_name = data_name +'_scale'\n",
    "    name_path = wd_path+model_folder+'/'+'Scalar_' + data_name+'_'\n",
    "    train_dfs, test_dfs = dataScaling(train_dfs, test_dfs, outcomeLabel, instLabel, name_path, header)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Impute Missing Data Values\n",
    "- Missing data values in categorical features are imputed with simple 'mode' imputation\n",
    "- Remaining missing data values are imputed using Iterative Imputer (i.e. MICE) \n",
    "- Completed within each training and testing partition independently"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if imputeData and isMissingData:\n",
    "    print('Imputing Data')\n",
    "    data_name = data_name + '_imp'\n",
    "    train_dfs, test_dfs = imputeCVData(outcomeLabel, instLabel, categorical_variables, header, train_dfs, test_dfs, randomSeed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Confirm Missing Data Imputation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Training Datasets\")\n",
    "for each in train_dfs:\n",
    "    each.isnull().sum().sum()\n",
    "\n",
    "print(\"Testing Datasets\")\n",
    "for each in test_dfs:\n",
    "    each.isnull().sum().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save Training and Testing Datasets as .txt Files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_name = data_name+'_CV_'+str(partition_method)\n",
    "\n",
    "counter = 0\n",
    "for each in train_dfs:\n",
    "    each.to_csv(cv_data_folder+'/'+data_name+'_'+str(counter)+'_Train.txt', index=None, sep='\\t')\n",
    "    counter += 1\n",
    "    \n",
    "counter = 0\n",
    "for each in test_dfs:\n",
    "    each.to_csv(cv_data_folder+'/'+data_name+'_'+str(counter)+'_Test.txt', index=None, sep='\\t')\n",
    "    counter += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Load, Check, and Prepare Saved CV Datasets for ML"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load CV Partition Datasets\n",
    "Re-loads training and testing datasets from saved .txt files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create lists to hold training and testing set dataframes.\n",
    "trainList = []\n",
    "testList = []\n",
    "\n",
    "for i in range(cv_partitions):\n",
    "    #Load training partition\n",
    "    trainSet = pd.read_csv(cv_data_folder+'/'+data_name+'_'+str(i)+'_Train.txt', na_values='NA', sep = \"\\t\")\n",
    "    trainList.append(trainSet)\n",
    "    \n",
    "    #Load testing partition\n",
    "    testSet = pd.read_csv(cv_data_folder+'/'+data_name+'_'+str(i)+'_Test.txt', na_values='NA', sep = \"\\t\")\n",
    "    testList.append(testSet)\n",
    "    \n",
    "print('Number of Training Partitions: '+ str(len(trainList)))\n",
    "print('Number of Testing Partitions: '+ str(len(testList)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Quality Control Check (Checking one of the partitions)\n",
    "Merge the first training/testing datasets to confirm original dataset is reconstituted"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Confirm case/control counts when merging one training and testing dataset\n",
    "classesTrain = trainList[0].apply(lambda x: True if x[outcomeLabel] == 1 else False , axis=1)\n",
    "casesTrain = len(classesTrain[classesTrain == True].index)\n",
    "controlsTrain = len(classesTrain[classesTrain == False].index)\n",
    "\n",
    "classesTest = testList[0].apply(lambda x: True if x[outcomeLabel] == 1 else False , axis=1)\n",
    "casesTest = len(classesTest[classesTest == True].index)\n",
    "controlsTest = len(classesTest[classesTest == False].index)\n",
    "\n",
    "print(\"Number of Cases in Original Dataset: \"+str(casesTrain+casesTest))\n",
    "print(\"Number of Controls in Original Dataset: \"+str(controlsTrain+controlsTest))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Confirm training and testing set dimensions\n",
    "print(\"Training Set Dimensions: \" + str(trainList[0].shape))\n",
    "print(\"Testing Set Dimensions: \" + str(testList[0].shape))\n",
    "numColumns = trainList[0].shape[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Partition Data Preprocessing\n",
    "If present, remove instance id's from datasets prior to analysis. Also confirm that training and testing sets have same set of columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop Instance ID from all training and test sets\n",
    "if instLabel != None and instLabel != 'None':\n",
    "    for i in range(cv_partitions):\n",
    "        trainList[i] = trainList[i].drop([instLabel] , axis = 1)\n",
    "        testList[i] = testList[i].drop([instLabel] , axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "trainColumnNames = list(trainList[0])\n",
    "testColumnNames = list(testList[0])\n",
    "\n",
    "if not trainColumnNames == testColumnNames:\n",
    "    print('Error: Training columns do not match testing columns!')\n",
    "\n",
    "# As we can see, instLabel is no longer a column name for any of the sets\n",
    "#print(trainColumnNames)\n",
    "#print(testColumnNames)\n",
    "\n",
    "#Create master list of feature names (excluding outcome column)\n",
    "ordered_feature_names = copy.deepcopy(trainColumnNames) #Stores original ordered feature list \n",
    "ordered_feature_names.remove(outcomeLabel)\n",
    "#ordered_feature_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Format data for Scikit learn (separate features from outcome)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Separate data into training and test features and outcomes\n",
    "xTrainList = []\n",
    "yTrainList = []\n",
    "xTestList = []\n",
    "yTestList = []\n",
    "\n",
    "for i in range(cv_partitions):\n",
    "    xTrainList.append(trainList[i].iloc[:, 1:].values)\n",
    "    yTrainList.append(trainList[i].iloc[:, 0].values)\n",
    " \n",
    "    xTestList.append(testList[i].iloc[:, 1:].values)\n",
    "    yTestList.append(testList[i].iloc[:, 0].values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Feature Importance Evaluation\n",
    "- Completed independently within each training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Store indexes of 'interesting' features identified by each algorithm in algorithm labeled dictionary\n",
    "selectedFeatureLists = {}\n",
    "metaScoreDict = {}\n",
    "metaFeatureRanks = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mutual information"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Run Mutual Information for each CV------------------------------------------------------------------------------\n",
    "algorithm = 'mutual_information'\n",
    "if algorithm in algorithms:\n",
    "    start_time = time.time()\n",
    "    scoreSet = [] #Used to save excel filel with data-ordered scores for each CV partition\n",
    "    featureNameRanks = [] #list of feature name ranks for each CV partition\n",
    "    scoreDictSet = [] #list of feature/score dictionaries (one for each cv partition)\n",
    "    cvKeepList = [] #Used for initial feature selection attempt to keep only features with 'informative' support\n",
    "    \n",
    "    for i in tqdm_notebook(range(cv_partitions), desc='1st loop'):\n",
    "        #Run algorithm and return scores and sorted score dictionary\n",
    "        scores, scoreDict, score_sorted_features = run_mi(xTrainList[i],yTrainList[i], i, data_name, output_folder,randomSeed,ordered_feature_names,algorithm)\n",
    "        scoreSet.append(scores) #store data-ordered scores for this cv partition\n",
    "        scoreDictSet.append(scoreDict)\n",
    "        featureNameRanks.append(score_sorted_features)\n",
    "        \n",
    "        #Add list of feature names that meet the 'informative' cutoff\n",
    "        keepList = []\n",
    "        for each in scoreDict:\n",
    "            if scoreDict[each] > 0:\n",
    "                keepList.append(each)\n",
    "                #keepList.append(ordered_feature_names.index(each))\n",
    "        cvKeepList.append(keepList)\n",
    "        \n",
    "        #Create dictionary of average scores\n",
    "        if i == 0:\n",
    "            scoreSum = copy.deepcopy(scoreDict)\n",
    "        else:\n",
    "            for each in scoreDict:\n",
    "                scoreSum[each] += scoreDict[each]\n",
    "    \n",
    "    #Save/report analysis results\n",
    "    #cv_output_name = data_name+'_clean_imp_CV_'+partition_method\n",
    "    reportAllFS(scoreSet, algorithm, ordered_feature_names,output_folder,data_name) #Save CV scores file for algorithm\n",
    "    reportTopFS(scoreSum, algorithm, cv_partitions,topResults,wd_path,output_folder,data_name)\n",
    "    \n",
    "    #Store info for downstream feature selection\n",
    "    selectedFeatureLists[algorithm] = cvKeepList\n",
    "    metaScoreDict[algorithm] = scoreDictSet\n",
    "    metaFeatureRanks[algorithm] = featureNameRanks\n",
    "\n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('Mutual Information Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MultiSURF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#Run MultiSURF for each CV------------------------------------------------------------------------------\n",
    "algorithm = 'multisurf'\n",
    "if algorithm in algorithms:\n",
    "    start_time = time.time()\n",
    "    \n",
    "    scoreSet = [] #Used to save excel filel with data-ordered scores for each CV partition\n",
    "    scoreDictSet = [] #list of feature/score dictionaries (one for each cv partition)\n",
    "    cvKeepList = [] #Used for initial feature selection attempt to keep only features with 'informative' support\n",
    "    \n",
    "    for i in tqdm_notebook(range(cv_partitions), desc='1st loop'):\n",
    "        #If training set is large MultiSURF will run very slowly - quadradic time complexity with # instances (so use subset)\n",
    "        tempData = pd.concat([pd.DataFrame(yTrainList[i]),pd.DataFrame(xTrainList[i])], axis=1, sort=False)\n",
    "        if len(tempData) < instanceSubset:\n",
    "            dataSample = tempData.sample(n=len(tempData), axis = 0)\n",
    "        else:\n",
    "            dataSample = tempData.sample(n=instanceSubset, axis = 0)\n",
    "\n",
    "        xTrain = dataSample.iloc[:, 1:].values\n",
    "        yTrain = dataSample.iloc[:, 0].values\n",
    "        #Run algorithm and return scores and sorted score dictionary\n",
    "        scores, scoreDict, score_sorted_features = run_multisurf(xTrain,yTrain, i, data_name, output_folder,randomSeed,ordered_feature_names,algorithm)\n",
    "        scoreSet.append(scores) #store data-ordered scores for this cv partition\n",
    "        scoreDictSet.append(scoreDict)\n",
    "        featureNameRanks.append(score_sorted_features)\n",
    "        \n",
    "        #Add list of feature names that meet the 'informative' cutoff\n",
    "        keepList = []\n",
    "        for each in scoreDict:\n",
    "            if scoreDict[each] > 0:\n",
    "                keepList.append(each)\n",
    "                #keepList.append(ordered_feature_names.index(each))\n",
    "        cvKeepList.append(keepList)\n",
    "        \n",
    "        #Create dictionary of average scores\n",
    "        if i == 0:\n",
    "            scoreSum = copy.deepcopy(scoreDict)\n",
    "        else:\n",
    "            for each in scoreDict:\n",
    "                scoreSum[each] += scoreDict[each]\n",
    "    \n",
    "    #Save/report analysis results\n",
    "    reportAllFS(scoreSet, algorithm, ordered_feature_names,output_folder,data_name) #Save CV scores file for algorithm\n",
    "    reportTopFS(scoreSum, algorithm, cv_partitions,topResults,wd_path,output_folder,data_name)\n",
    "    \n",
    "    #Store info for downstream feature selection\n",
    "    selectedFeatureLists[algorithm] = cvKeepList\n",
    "    metaScoreDict[algorithm] = scoreDictSet\n",
    "    metaFeatureRanks[algorithm] = featureNameRanks\n",
    "\n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('MultiSURF Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Feature Selection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify union of 'interesting' features from different algorithms\n",
    "- Operates no matter how many feature selection methods are being applied\n",
    "- But assumes that all methods have larger scores as being more informative"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cv_Selected_List = selectFeatures(algorithms, cv_partitions, selectedFeatureLists, maxFeaturesToKeep,metaFeatureRanks)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Report counts of 'interesting' features identified by each algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Original feature count: '+str(len(ordered_feature_names)))\n",
    "\n",
    "for each in algorithms:\n",
    "    print(each)\n",
    "    for cv in selectedFeatureLists[each]:\n",
    "        print(len(cv))\n",
    "    \n",
    "print(\"Selected Counts\")\n",
    "for each in cv_Selected_List:\n",
    "    print(len(each))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Filtered Datasets (Save as new .txt files)\n",
    "- These datasets included only the selected features for each respective partition\n",
    "- Note that the feature order from the original datasets is no longer preserved in these datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if filterPoorFeatures:\n",
    "    genFilteredDatasets(cv_Selected_List, outcomeLabel, instLabel,cv_partitions,cv_data_folder,data_name)\n",
    "    data_name = data_name +'_FS'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate Filtered MultiSURF Files (Save as new .txt files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'multisurf' in algorithms: #This method is only meant for working with MultiSURF scores\n",
    "    if filterPoorFeatures:\n",
    "        for i in range(cv_partitions):\n",
    "            tempDict = scoreDictSet[i]\n",
    "\n",
    "            #create new dictionary with only selected features\n",
    "            scoreDict = {}\n",
    "            for j in cv_Selected_List[i]:\n",
    "                scoreDict[j] = tempDict[j]\n",
    "            \n",
    "            filename = output_folder+'/'+'multisurf'+'_'+data_name+'_'+str(i)+'_Train.txt'\n",
    "            sort_save_fs_fi_scores(scoreDict, 'multisurf', filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Load, Check, and Prepare Filtered CV Datasets for ML\n",
    "- If user opted not to filter datasets then the code below will simply reload the original CV datasets again."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load CV Partition Datasets (Filtered Datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create lists to hold training and testing set dataframes.\n",
    "trainList = []\n",
    "testList = []\n",
    "\n",
    "for i in range(cv_partitions):\n",
    "    #Load training partition\n",
    "    trainset_name = cv_data_folder+'/'+data_name+'_'+str(i)+'_Train.txt'\n",
    "    trainSet = pd.read_csv(trainset_name, na_values='NA', sep = \"\\t\")\n",
    "    trainList.append(trainSet)\n",
    "\n",
    "    #Load testing partition\n",
    "    testset_name = cv_data_folder+'/'+data_name+'_'+str(i)+'_Test.txt'\n",
    "    testSet = pd.read_csv(testset_name, na_values='NA', sep = \"\\t\")\n",
    "    testList.append(testSet)\n",
    "    \n",
    "print('Number of Training Partitions: '+ str(len(trainList)))\n",
    "print('Number of Testing Partitions: '+ str(len(testList)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CV partition data cleaning (Filtered Datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop Instance ID from all training and test sets\n",
    "if instLabel != None and instLabel != 'None':\n",
    "    for i in range(cv_partitions):\n",
    "        trainList[i] = trainList[i].drop([instLabel] , axis = 1)\n",
    "        testList[i] = testList[i].drop([instLabel] , axis = 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "global_ordered_features = [] #Holds the new order of features in the FS CV datasets\n",
    "for i in range(cv_partitions):\n",
    "    cv_ordered_features = list(trainList[i])\n",
    "    cv_ordered_features.remove(outcomeLabel)\n",
    "    #print(cv_ordered_features)\n",
    "    global_ordered_features.append(cv_ordered_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Format data for Scikit learn (filtered datasets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Separate data into training and test features and outcomes\n",
    "xTrainList = []\n",
    "yTrainList = []\n",
    "xTestList = []\n",
    "yTestList = []\n",
    "\n",
    "for i in range(cv_partitions):\n",
    "    xTrainList.append(trainList[i].iloc[:, 1:].values)\n",
    "    yTrainList.append(trainList[i].iloc[:, 0].values)\n",
    " \n",
    "    xTestList.append(testList[i].iloc[:, 1:].values)\n",
    "    yTestList.append(testList[i].iloc[:, 0].values)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Machine Learning Modeling\n",
    "ML algorithms are run one after the other below. Within each we conduct a hyperparameter sweep, train a best model and evaluate it indendently within each CV training/testing partition. \n",
    "- All standard classification metrics are computed for the resulting models as well as an ROC curve and precision/recall curve"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Logistic Regression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'logistic_regression'\n",
    "if algorithm in algorithmsToRun:\n",
    "    start_time = time.time()\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec, FI_ave = eval_Algorithm_FI(algorithm,ordered_feature_names,\n",
    "        xTrainList,yTrainList,xTestList,yTestList,cv_partitions,global_ordered_features,wd_path,output_folder,\n",
    "        data_name,randomSeed,param_grid,model_folder,algColor,hype_cv,n_trials,scoring_metric,timeout)\n",
    "    \n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))\n",
    "\n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)\n",
    "    \n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('Logistic Regression Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'decision_tree'\n",
    "if algorithm in algorithmsToRun:\n",
    "    start_time = time.time()\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec, FI_ave = eval_Algorithm_FI(algorithm,ordered_feature_names,\n",
    "        xTrainList,yTrainList,xTestList,yTestList,cv_partitions,global_ordered_features,wd_path,output_folder,\n",
    "        data_name,randomSeed,param_grid,model_folder,algColor,hype_cv,n_trials,scoring_metric,timeout)\n",
    "    \n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))\n",
    "    \n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)\n",
    "    \n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('Decision Tree Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Random Forest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'random_forest'\n",
    "if algorithm in algorithmsToRun:\n",
    "    start_time = time.time()\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec, FI_ave = eval_Algorithm_FI(algorithm,ordered_feature_names,\n",
    "        xTrainList,yTrainList,xTestList,yTestList,cv_partitions,global_ordered_features,wd_path,output_folder,\n",
    "        data_name,randomSeed,param_grid,model_folder,algColor,hype_cv,n_trials,scoring_metric,timeout)\n",
    "    \n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))\n",
    "\n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)\n",
    "    \n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('Random Forest Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run Naive Bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'naive_bayes'\n",
    "if algorithm in algorithmsToRun:\n",
    "    start_time = time.time()\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec, FI_ave = eval_Algorithm_FI(algorithm,ordered_feature_names,\n",
    "        xTrainList,yTrainList,xTestList,yTestList,cv_partitions,global_ordered_features,wd_path,output_folder,\n",
    "        data_name,randomSeed,param_grid,model_folder,algColor,hype_cv,n_trials,scoring_metric,timeout)\n",
    "    \n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))\n",
    "        \n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)\n",
    "    \n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('Naive Bayes Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run XGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'XGB'\n",
    "if algorithm in algorithmsToRun:\n",
    "    start_time = time.time()\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec, FI_ave = eval_Algorithm_FI(algorithm,ordered_feature_names,\n",
    "        xTrainList,yTrainList,xTestList,yTestList,cv_partitions,global_ordered_features,wd_path,output_folder,\n",
    "        data_name,randomSeed,param_grid,model_folder,algColor,hype_cv,n_trials,scoring_metric,timeout)\n",
    "    \n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))\n",
    "\n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)\n",
    "    \n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('XGBoost Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run LGBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'LGB'\n",
    "if algorithm in algorithmsToRun:\n",
    "    start_time = time.time()\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec, FI_ave = eval_Algorithm_FI(algorithm,ordered_feature_names,\n",
    "        xTrainList,yTrainList,xTestList,yTestList,cv_partitions,global_ordered_features,wd_path,output_folder,\n",
    "        data_name,randomSeed,param_grid,model_folder,algColor,hype_cv,n_trials,scoring_metric,timeout)\n",
    "    \n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))\n",
    "\n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)\n",
    "        \n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('LGBoost Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run SVM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'SVM'\n",
    "if algorithm in algorithmsToRun:\n",
    "    start_time = time.time()\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec, FI_ave = eval_Algorithm_FI(algorithm,ordered_feature_names,\n",
    "        xTrainList,yTrainList,xTestList,yTestList,cv_partitions,global_ordered_features,wd_path,output_folder,\n",
    "        data_name,randomSeed,param_grid,model_folder,algColor,hype_cv,n_trials,scoring_metric,timeout)\n",
    "    \n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))\n",
    "\n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)\n",
    "    \n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('SVM Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run ANN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'ANN'\n",
    "if algorithm in algorithmsToRun:\n",
    "    start_time = time.time()\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec, FI_ave = eval_Algorithm_FI(algorithm,ordered_feature_names,\n",
    "        xTrainList,yTrainList,xTestList,yTestList,cv_partitions,global_ordered_features,wd_path,output_folder,\n",
    "        data_name,randomSeed,param_grid,model_folder,algColor,hype_cv,n_trials,scoring_metric,timeout)\n",
    "    \n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))\n",
    "\n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)\n",
    "\n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('ANN Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Run LCS"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def makeConfigFile(train_dataset,test_dataset,outpath,external_ek_file,iterations,popsize,configfile,outcomelabel,instancelabel,discthresh):\n",
    "    \"\"\" Construct Configuration File for CV Analysis \"\"\"\n",
    "    configFile = open(configfile,'w')\n",
    "    \n",
    "    #Write to Config File#############################################################################################################\n",
    "    configFile.write('offlineData=1# \\n')\n",
    "    configFile.write('trainFile='+str(train_dataset)+'# \\n')\n",
    "    configFile.write('testFile='+str(test_dataset)+'# \\n')\n",
    "    configFile.write('internalCrossValidation=0# \\n')\n",
    "    \n",
    "    configFile.write('outFileName='+str(outpath)+'# \\n')\n",
    "    configFile.write('randomSeed=1# \\n')\n",
    "    configFile.write('labelInstanceID='+str(instancelabel)+'# \\n')\n",
    "    configFile.write('labelPhenotype='+str(outcomelabel)+'# \\n')\n",
    "    configFile.write('discreteAttributeLimit='+str(discthresh)+'# \\n')\n",
    "    configFile.write('labelMissingData=NA# \\n')\n",
    "    configFile.write('outputSummary=1# \\n')\n",
    "    configFile.write('outputPopulation=1# \\n')\n",
    "    configFile.write('outputAttCoOccur=1# \\n')\n",
    "    configFile.write('outputTestPredictions=1# \\n')  \n",
    "    configFile.write('maxAttsCoOccur=50# \\n')\n",
    "        \n",
    "    configFile.write('trackingFrequency=0# \\n')\n",
    "    configFile.write('learningIterations='+str(iterations)+'# \\n')\n",
    "    \n",
    "    configFile.write('N='+str(popsize)+'# \\n') \n",
    "    configFile.write('nu=1# \\n')\n",
    "    configFile.write('chi=0.8# \\n')\n",
    "    configFile.write('upsilon=0.04# \\n')\n",
    "    configFile.write('theta_GA=25# \\n')\n",
    "    configFile.write('theta_del=20# \\n')   \n",
    "    configFile.write('theta_sub=20# \\n')    \n",
    "    configFile.write('acc_sub=0.99# \\n')    \n",
    "    configFile.write('beta=0.2# \\n')         \n",
    "    configFile.write('delta=0.1# \\n')     \n",
    "    configFile.write('init_fit=0.01# \\n')\n",
    "    configFile.write('fitnessReduction=0.1# \\n')\n",
    "    configFile.write('theta_sel=0.5# \\n')\n",
    "    configFile.write('RSL_Override=0# \\n')\n",
    "\n",
    "    configFile.write('doSubsumption=1# \\n')\n",
    "    configFile.write('selectionMethod=tournament# \\n')\n",
    "    \n",
    "    configFile.write('doAttributeTracking=1# \\n')\n",
    "    configFile.write('doAttributeFeedback=1# \\n')\n",
    "    \n",
    "    configFile.write('useExpertKnowledge=1# \\n')\n",
    "    configFile.write('external_EK_Generation='+str(external_ek_file)+'# \\n')\n",
    "    configFile.write('outEKFileName=None# \\n')\n",
    "    \n",
    "    configFile.write('filterAlgorithm=None# \\n')\n",
    "    configFile.write('turfPercent=0.2# \\n')\n",
    "    configFile.write('reliefNeighbors=10# \\n')\n",
    "    configFile.write('reliefSampleFraction=1# \\n')\n",
    "    configFile.write('onlyEKScores=0# \\n')\n",
    "    \n",
    "    configFile.write('doRuleCompaction=1# \\n')\n",
    "    configFile.write('onlyRC=0# \\n')\n",
    "    configFile.write('ruleCompactionMethod=QRF# \\n')\n",
    "    \n",
    "    configFile.write('doPopulationReboot=0# \\n')\n",
    "    configFile.write('popRebootIteration=0# \\n')\n",
    "    \n",
    "    configFile.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate LCS Configuration Files (needed to pass all run/hyperparamters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'LCS'\n",
    "if algorithm in algorithmsToRun:\n",
    "    start_time = time.time()\n",
    "    # Generate LCS configuration files (needed to pass all run/hyper parameters to LCS)\n",
    "    for i in tqdm_notebook(range(cv_partitions), desc='1st loop'):\n",
    "        trainFile = wd_path+cv_data_folder+'/'+data_name+'_'+str(i)+'_Train.txt'\n",
    "        testFile = wd_path+cv_data_folder+'/'+data_name+'_'+str(i)+'_Test.txt'\n",
    "        ekFile = wd_path+output_folder+'/'+'multisurf'+'_'+data_name+'_'+str(i)+'_Train.txt'\n",
    "        configfile = wd_path+lcs_folder+'/'+lcs_alg+'_'+data_name+'_'+str(i)+'_Train_ConfigFile.txt'\n",
    "        outFold = wd_path+lcs_folder+'/'\n",
    "\n",
    "        makeConfigFile(trainFile,testFile,outFold,ekFile,iterations,popsize,configfile,outcomeLabel,instLabel,categoricalCutoff)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'LCS'\n",
    "if algorithm in algorithmsToRun:\n",
    "    #Use Python Magic to run Exstracs from command line\n",
    "    for i in tqdm_notebook(range(cv_partitions), desc='1st loop'):\n",
    "        #lcs_path = 'exstracs_2.0.2.1_noclassmutate_lynch'+'/'+'exstracs_main.py'\n",
    "        configFile = wd_path+lcs_folder+'/'+lcs_alg+'_'+data_name+'_'+str(i)+'_Train_ConfigFile.txt'\n",
    "        %run {lcs_path} {configFile}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and Evaluate LCS Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "algorithm = 'LCS'\n",
    "if algorithm in algorithmsToRun:\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec = eval_LCS(algorithm,cv_partitions,wd_path,output_folder,lcs_folder,data_name,iterations,algColor)\n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))\n",
    "    \n",
    "    elapsed_time = time.time() - start_time\n",
    "    print('LCS Run Time: '+ str(elapsed_time))\n",
    "    run_time_dict[algorithm] = elapsed_time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and Evaluate LCS with QRF Predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'LCS_QRF'\n",
    "if algorithm in algorithmsToRun:\n",
    "    algColor = algColors[algorithmsToRun.index(algorithm)]\n",
    "    mean_fpr, mean_tpr, mean_auc, mean_prec, mean_pr_auc, mean_ave_prec = eval_LCS_QRF(algorithm,cv_partitions,wd_path,output_folder,lcs_folder,data_name,iterations,algColor)\n",
    "    result_dict = {'algorithm':algorithm,'fpr':mean_fpr, 'tpr':mean_tpr, 'auc':mean_auc, 'prec':mean_prec, 'pr_auc':mean_pr_auc, 'ave_prec':mean_ave_prec}\n",
    "    pickle.dump(result_dict, open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and Evaluate LCS Feature Importance Estimates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'LCS'\n",
    "if algorithm in algorithmsToRun:\n",
    "    FI_all = []\n",
    "    FI_ave = [0]*len(ordered_feature_names) #Holds only the selected feature FI results for each partition\n",
    "    \n",
    "    #Gather all LCS feature importance results (specificity sums)\n",
    "    for i in tqdm_notebook(range(cv_partitions), desc='1st loop'):\n",
    "        scoreDict = {}\n",
    "        fiFile = wd_path+lcs_folder+'/'+'ExSTraCS'+'_'+data_name+'_'+str(i)+'_Train_'+str(iterations)+'_PopStats.txt'\n",
    "        fileObject = open(fiFile, 'r') \n",
    "        \n",
    "        counter = 0\n",
    "        tempList1 = []\n",
    "        tempList2 = []\n",
    "        for line in fileObject:\n",
    "            if counter == 9:\n",
    "                tempList1 = line.strip().split('\\t')\n",
    "            if counter == 10:\n",
    "                tempList2 = line.strip().split('\\t')\n",
    "            counter += 1\n",
    "        fileObject.close()\n",
    "        \n",
    "        for j in range(len(tempList1)):\n",
    "            scoreDict[tempList1[j]] = float(tempList2[j])\n",
    " \n",
    "        #Format feature importance scores as list (takes into account that all features are not in each CV partition)\n",
    "        tempList = []\n",
    "        j = 0\n",
    "        for each in ordered_feature_names:\n",
    "            if each in scoreDict:\n",
    "                FI_ave[j] += float(scoreDict[each])\n",
    "                tempList.append(float(scoreDict[each]))\n",
    "            else:\n",
    "                tempList.append(0)\n",
    "            j += 1\n",
    "            \n",
    "        FI_all.append(tempList)\n",
    "    \n",
    "    \n",
    "    dr = pd.DataFrame(FI_all)\n",
    "    filepath = output_folder+'/'+algorithm+'_FI_'+data_name+'.csv'\n",
    "    dr.to_csv(filepath, header=ordered_feature_names, index=False)  \n",
    "    \n",
    "    #Calculate Averages\n",
    "    for i in range(0,len(FI_ave)):\n",
    "        FI_ave[i] = FI_ave[i]/float(cv_partitions)\n",
    "        \n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load and Evaluate LCS with QRF Feature Importance Estimates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "algorithm = 'LCS_QRF'\n",
    "if algorithm in algorithmsToRun:\n",
    "    FI_all = []\n",
    "    FI_ave = [0]*len(ordered_feature_names) #Holds only the selected feature FI results for each partition\n",
    "    \n",
    "    #Gather all LCS feature importance results (specificity sums)\n",
    "    for i in tqdm_notebook(range(cv_partitions), desc='1st loop'):\n",
    "        scoreDict = {}\n",
    "        fiFile = wd_path+lcs_folder+'/'+'ExSTraCS'+'_'+data_name+'_'+str(i)+'_Train_RC_QRF_'+str(iterations)+'_PopStats.txt'\n",
    "        fileObject = open(fiFile, 'r') \n",
    "        \n",
    "        counter = 0\n",
    "        tempList1 = []\n",
    "        tempList2 = []\n",
    "        for line in fileObject:\n",
    "            if counter == 9:\n",
    "                tempList1 = line.strip().split('\\t')\n",
    "            if counter == 10:\n",
    "                tempList2 = line.strip().split('\\t')\n",
    "            counter += 1\n",
    "        fileObject.close()\n",
    "        \n",
    "        for j in range(len(tempList1)):\n",
    "            scoreDict[tempList1[j]] = float(tempList2[j])\n",
    " \n",
    "        #Format feature importance scores as list (takes into account that all features are not in each CV partition)\n",
    "        tempList = []\n",
    "        j = 0\n",
    "        for each in ordered_feature_names:\n",
    "            if each in scoreDict:\n",
    "                FI_ave[j] += float(scoreDict[each])\n",
    "                tempList.append(float(scoreDict[each]))\n",
    "            else:\n",
    "                tempList.append(0)\n",
    "            j += 1\n",
    "            \n",
    "        FI_all.append(tempList)\n",
    "    \n",
    "    \n",
    "    dr = pd.DataFrame(FI_all)\n",
    "    filepath = output_folder+'/'+algorithm+'_FI_'+data_name+'.csv'\n",
    "    dr.to_csv(filepath, header=ordered_feature_names, index=False)  \n",
    "    \n",
    "    #Calculate Averages\n",
    "    for i in range(0,len(FI_ave)):\n",
    "        FI_ave[i] = FI_ave[i]/float(cv_partitions)\n",
    "        \n",
    "    #Sort averages (decreasing order and print top 'n' and plot top 'n'\n",
    "    names_scores = {'Names':ordered_feature_names, 'Scores':FI_ave} \n",
    "    ns = pd.DataFrame(names_scores)\n",
    "    ns = ns.sort_values(by='Scores',ascending = False)\n",
    "\n",
    "    #Select top 'n' to report and plot\n",
    "    print(\"\\n---Feature Importances---\\n\")\n",
    "    ns.head(topResults)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Review result table (for global ROC and PRC figure building)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Load pickle files of results\n",
    "result_table = []\n",
    "\n",
    "for algorithm in algorithmsToRun:\n",
    "    tempDict = pickle.load(open(wd_path+model_folder+'/Results_'+algorithm+'.sav', 'rb'))\n",
    "    result_table.append(tempDict)\n",
    "    \n",
    "result_table = pd.DataFrame.from_dict(result_table)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result_table\n",
    "result_table.set_index('algorithm', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# ROC Plot Summarizing Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Multi-algorithm ROC plot\n",
    "fig = plt.figure(figsize=(8,8))\n",
    "count = 0\n",
    "for i in result_table.index:\n",
    "    plt.plot(result_table.loc[i]['fpr'], \n",
    "             result_table.loc[i]['tpr'], color = algColors[count],\n",
    "             label=\"{}, AUC={:.3f}\".format(i, result_table.loc[i]['auc']))\n",
    "    count += 1\n",
    "    \n",
    "plt.plot([0,1], [0,1], color='orange', linestyle='--')\n",
    "\n",
    "plt.xticks(np.arange(0.0, 1.1, step=0.1))\n",
    "plt.xlabel(\"False Positive Rate\", fontsize=15)\n",
    "\n",
    "plt.yticks(np.arange(0.0, 1.1, step=0.1))\n",
    "plt.ylabel(\"True Positive Rate\", fontsize=15)\n",
    "\n",
    "plt.title('Comparing Algorithms: Testing Data with CV', fontweight='bold', fontsize=15)\n",
    "plt.legend(prop={'size':13}, loc='best')\n",
    "plt.savefig((wd_path+output_folder+'/'+'Compare_ROC_' + data_name), bbox_inches = \"tight\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# PRC Plot Summarizing Analysis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Multi-algorithm PRC plot ['algorithm','fpr','tpr','auc','prec','pr_auc','ave_prec'])\n",
    "#result_table.set_index('algorithm', inplace=True)\n",
    "\n",
    "fig = plt.figure(figsize=(8,8))\n",
    "\n",
    "count = 0\n",
    "for i in result_table.index:\n",
    "    plt.plot(result_table.loc[i]['fpr'], \n",
    "             result_table.loc[i]['prec'], color = algColors[count],\n",
    "             label=\"{}, AUC={:.3f}, APS={:.3f}\".format(i, result_table.loc[i]['pr_auc'],result_table.loc[i]['ave_prec']))\n",
    "    count += 1\n",
    "noskill = len(yTestList[0][yTestList[0]==1]) / len(yTestList[0]) #Fraction of cases\n",
    "plt.plot([0,1], [noskill,noskill], color='orange', linestyle='--')\n",
    "\n",
    "plt.xticks(np.arange(0.0, 1.1, step=0.1))\n",
    "plt.xlabel(\"Recall\", fontsize=15)\n",
    "\n",
    "plt.yticks(np.arange(0.0, 1.1, step=0.1))\n",
    "plt.ylabel(\"Precision\", fontsize=15)\n",
    "\n",
    "plt.title('Comparing Algorithms: Testing Data with CV', fontweight='bold', fontsize=15)\n",
    "plt.legend(prop={'size':13}, loc='best')\n",
    "plt.savefig((wd_path+output_folder+'/'+'Compare_PRC_' + data_name), bbox_inches = \"tight\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Summarize Algorithm Metrics "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td = pd.read_csv(wd_path+output_folder+'/'+algorithmsToRun[0]+'_Metrics_'+data_name+'.csv')\n",
    "metrics = list(td.columns.values.tolist()) \n",
    "\n",
    "metric_summary = pd.DataFrame(index=metrics,columns=algorithmsToRun)\n",
    "master_list = []\n",
    "\n",
    "for algorithm in algorithmsToRun:\n",
    "    #Open file\n",
    "    td = pd.read_csv(wd_path+output_folder+'/'+algorithm+'_Metrics_'+data_name+'.csv')\n",
    "    master_list.append(td)\n",
    "    for metric in metrics:\n",
    "        ave = td[metric].mean()\n",
    "        sd = td[metric].std()\n",
    "        metric_summary.at[metric, algorithm] = str(round(ave,4))+' ('+str(round(sd,4))+')'\n",
    "    \n",
    "metric_summary.to_csv(wd_path+output_folder+'/'+'Summary_Algorithm_Metrics_'+data_name+'.csv')   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Performance Metric Boxplots Comparing Algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#For each metric\n",
    "for metric in metrics:\n",
    "    i = 0\n",
    "    tempList = []\n",
    "    for algorithm in algorithmsToRun:\n",
    "        temp_td = master_list[i]\n",
    "        tempList.append(temp_td[metric].tolist())\n",
    "        i += 1\n",
    "        \n",
    "    td = pd.DataFrame(tempList)\n",
    "    td = td.transpose()\n",
    "    td.columns = algorithmsToRun\n",
    "        \n",
    "    boxplot = td.boxplot(column=algorithmsToRun,rot=45)\n",
    "    plt.title('Comparing Algorithm '+ str(metric))\n",
    "    plt.ylabel(str(metric))\n",
    "    plt.xlabel('ML Algorithm')\n",
    "    plt.savefig((wd_path+output_folder+'/'+'Compare_'+str(metric)+'_Boxplot_'+data_name), bbox_inches = \"tight\")\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Statistical Comparisons\n",
    "- Kruskal Wallis AOV comparing ML algorithms\n",
    "- Mann-Whitney U test - Pairwise algorithm differences  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kruskal Wallis Test - Algorithm comparisons for each metric\n",
    "- Are any algorithms performing statistically better or worse? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mann-Whiney Test - Pairwise Post-Hoc Analysis \n",
    "- For any metric where kruskal wallis test was significant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(algorithmsToRun) > 1:\n",
    "    label = ['statistic','pvalue','sig']\n",
    "    kruskal_summary = pd.DataFrame(index=metrics,columns=label)\n",
    "    for metric in metrics:\n",
    "        tempArray = []\n",
    "        for i in range(len(algorithmsToRun)):\n",
    "            tempArray.append(master_list[i][metric])\n",
    "        result = stats.kruskal(*tempArray)\n",
    "        #result = stats.kruskal(master_list[0][metric],master_list[1][metric],master_list[2][metric],master_list[3][metric],master_list[4][metric],master_list[5][metric],master_list[6][metric]) \n",
    "        kruskal_summary.at[metric, 'statistic'] = str(round(result[0],6))\n",
    "        kruskal_summary.at[metric, 'pvalue'] = str(round(result[1],6))\n",
    "        if result[1] < sig_cutoff:\n",
    "            kruskal_summary.at[metric, 'sig'] = str('*')\n",
    "        else:\n",
    "            kruskal_summary.at[metric, 'sig'] = str('')\n",
    "\n",
    "    kruskal_summary\n",
    "\n",
    "    kruskal_summary.to_csv(wd_path+output_folder+'/'+'Summary_Algorithm_KruskalWallis_'+data_name+'.csv')   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if len(algorithmsToRun) > 1:\n",
    "    algCount = len(algorithmsToRun)\n",
    "    for metric in metrics:\n",
    "        if kruskal_summary['sig'][metric] == '*':\n",
    "            mann_stats = []\n",
    "            #Balanced Accuracy\n",
    "            for i in range(0,algCount-1):\n",
    "                for j in range(i+1,algCount):\n",
    "                    set1 = master_list[i][metric]\n",
    "                    set2 = master_list[j][metric]\n",
    "\n",
    "                    #print('Mann-WhitneyU: '+str(algorithmsToRun[i])+' vs. '+str(algorithmsToRun[j]))\n",
    "                    report = stats.mannwhitneyu(set1,set2)\n",
    "                    #report\n",
    "                    tempstats = [algorithmsToRun[i], algorithmsToRun[j], report[0], report[1], '']\n",
    "                    if report[1] < sig_cutoff:\n",
    "                        tempstats[4] = '*'\n",
    "                    mann_stats.append(tempstats)\n",
    "\n",
    "            mann_stats_df = pd.DataFrame(mann_stats)\n",
    "            mann_stats_df.columns = ['Algorithm 1', 'Algorithm 2', 'statistic', 'p-value', 'sig']\n",
    "            filepath = wd_path+output_folder+'/'+'MannWhitney_'+str(metric)+'_'+data_name+'.csv'\n",
    "            mann_stats_df.to_csv(filepath, index = False) \n",
    "            mann_stats_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "# Visualize Feature Importance Results Across Algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Specify dataset specific paths/names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "figx = 24\n",
    "figy = 12\n",
    "    \n",
    "#Feature names to use in the final publication figures\n",
    "df = pd.read_csv(wd_path+output_folder+'/'+algorithmsToRun[0]+'_FI_'+data_name+'.csv')\n",
    "\n",
    "printfeatureNames = list(df.columns)\n",
    "print(printfeatureNames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load feature importance and balanced accuracy results\n",
    "- Do this generically so that future algorithms can be added here\n",
    "- Identify a feature list for each algorithm (only features with average scores over zero)\n",
    "- Identify union of these feature lists - master feature list to be included in bar chart\n",
    "- Have a max_features to visualize - use ranking to reduce above list as needed to top n features only.\n",
    "- Add ability to load LCS or other external algorithm to these results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#algorithm feature importance dataframe list (used to generate FI boxplots for each algorithm)\n",
    "fi_df_list = []\n",
    "#algorithm feature importance averages list (used to generate composite FI barplots)\n",
    "fi_ave_list = []\n",
    "#algorithm focus metric averages list (used in weighted FI viz)\n",
    "ave_metric_list = []\n",
    "#list of pre-feature selection features as they appear in FI reports for each algorithm\n",
    "all_feature_list = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for each in algorithmsToRun:\n",
    "    #Get relevant feature importance info\n",
    "    temp_df = pd.read_csv(wd_path+output_folder+'/'+each+'_FI_'+data_name+'.csv')\n",
    "    if each == algorithmsToRun[0]: #Should be same for all algorithm files (i.e. all original features in standard CV dataset order)\n",
    "        all_feature_list = temp_df.columns.tolist()\n",
    "    fi_df_list.append(temp_df) \n",
    "    fi_ave_list.append(temp_df.mean().tolist())\n",
    "    \n",
    "    #Get relevant metric info\n",
    "    tmp_df = pd.read_csv(wd_path+output_folder+'/'+each+'_Metrics_'+data_name+'.csv')\n",
    "    ave_metric_list.append(tmp_df[focus_metric].mean())\n",
    "ave_metric_list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Normalize average scores (Range 0-1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Normalize average scores so all values range between 0 and 1 (important for combining FI's)\n",
    "fi_ave_norm_list = []\n",
    "for each in fi_ave_list: #each algorithm   \n",
    "    normList = []\n",
    "    for i in range(len(each)):\n",
    "        if each[i] <= 0:\n",
    "            normList.append(0)\n",
    "        else:\n",
    "            normList.append((each[i])/(max(each)))\n",
    "    fi_ave_norm_list.append(normList)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify features with non-zero averages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For each algorithm identify all non-zero features\n",
    "alg_non_zero_FI_list = []\n",
    "for each in fi_ave_list: #each algorithm\n",
    "    \n",
    "    temp_non_zero_list = []\n",
    "    for i in range(len(each)): #each feature\n",
    "        if each[i] > 0.0:\n",
    "            temp_non_zero_list.append(all_feature_list[i])\n",
    "    \n",
    "    alg_non_zero_FI_list.append(temp_non_zero_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify union of features with non-zero averages over all algorithms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Identify a union list over all algorithms of features with non-zero FI (These are candidates for vizualization)\n",
    "non_zero_union_features = alg_non_zero_FI_list[0] #grab first algorithm's list\n",
    "\n",
    "for j in range(1,len(algorithmsToRun)):\n",
    "    #print(j)\n",
    "    non_zero_union_features = list(set(non_zero_union_features) | set(alg_non_zero_FI_list[j]))\n",
    "\n",
    "#print(non_zero_union_features)\n",
    "\n",
    "non_zero_union_indexes = []\n",
    "for i in non_zero_union_features:\n",
    "    non_zero_union_indexes.append(all_feature_list.index(i))\n",
    "non_zero_union_indexes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Identify list of top features over all algorithms to visualize"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#If remaining non-zero feature list is still larger than we want to visualize, pick only top features for visualization\n",
    "featuresToViz = None\n",
    "if len(non_zero_union_features) > topResults:\n",
    "    #Identify a top set of feature values\n",
    "    scoreSumDict = {}\n",
    "    i = 0\n",
    "    for each in non_zero_union_features: #for each non-zero feature\n",
    "        for j in range(len(algorithmsToRun)): #for each algorithm\n",
    "            #grab target score from each algorithm\n",
    "            score = fi_ave_norm_list[j][non_zero_union_indexes[i]]\n",
    "            #multiply score by algorithm performance weight\n",
    "            score = score*ave_metric_list[j]\n",
    "            if not each in scoreSumDict:\n",
    "                scoreSumDict[each] = score\n",
    "            else:\n",
    "                scoreSumDict[each] += score\n",
    "        i += 1\n",
    "        \n",
    "    for each in scoreSumDict:\n",
    "        scoreSumDict[each] = scoreSumDict[each]/ len(algorithmsToRun)\n",
    "    #print(scoreSumDict)\n",
    "\n",
    "    #Rank and select top features to visualize\n",
    "\n",
    "    #Sort features by decreasing score\n",
    "    scoreSumDict_features = sorted(scoreSumDict, key=lambda x: scoreSumDict[x], reverse=True)\n",
    "    #scoreSumDict_features\n",
    "\n",
    "    featuresToViz = scoreSumDict_features[0:topResults]\n",
    "    #featuresToViz\n",
    "else:\n",
    "    featuresToViz = non_zero_union_features #Ranked feature name order"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Generate individual feature importance boxplots for each algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "counter = 0\n",
    "for df in fi_df_list:\n",
    "    fig = plt.figure(figsize=(15,4))\n",
    "    boxplot = df.boxplot(rot=90)\n",
    "    plt.title(algorithmsToRun[counter])\n",
    "    plt.ylabel('Feature Importance Score')\n",
    "    plt.xlabel('Features')\n",
    "    plt.xticks(np.arange(1,len(printfeatureNames)+1), printfeatureNames,rotation='vertical')\n",
    "    plt.savefig((wd_path+output_folder+'/'+algorithmsToRun[counter]+'_boxplot_' + data_name), bbox_inches = \"tight\")\n",
    "    plt.show()\n",
    "\n",
    "    counter += 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Normalized dataframes with feature viz subsets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "featuresToViz\n",
    "\n",
    "feature_indexToViz = []\n",
    "for i in featuresToViz:\n",
    "    feature_indexToViz.append(all_feature_list.index(i))\n",
    "feature_indexToViz\n",
    "\n",
    "#Preserve features in original dataset order for consistency\n",
    "top_fi_ave_norm_list = []\n",
    "for i in range(len(algorithmsToRun)):\n",
    "    tempList = []\n",
    "    for j in range(len(fi_ave_norm_list[i])):\n",
    "        if j in feature_indexToViz:\n",
    "            tempList.append(fi_ave_norm_list[i][j])\n",
    "    top_fi_ave_norm_list.append(tempList)\n",
    "\n",
    "#Create feature name list in propper order\n",
    "all_feature_listToViz = []\n",
    "for j in (all_feature_list):\n",
    "    if j in featuresToViz:\n",
    "        all_feature_listToViz.append(j)\n",
    "all_feature_listToViz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compound_FI_plot(fi_list,algorithmsToRun,algColors,all_feature_listToViz,figName):\n",
    "    \n",
    "    # y-axis in bold\n",
    "    rc('font', weight='bold', size=16)\n",
    "    \n",
    "    # The position of the bars on the x-axis\n",
    "    r = all_feature_listToViz\n",
    "    barWidth = 0.75\n",
    "    plt.figure(figsize=(figx,figy))\n",
    "            \n",
    "    lines = None\n",
    "    if len(algorithmsToRun) == 1:\n",
    "        print(\"Plotting with 1 algorithm.\")\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        lines = (p1[0])\n",
    "        \n",
    "    elif len(algorithmsToRun) == 2:\n",
    "        print(\"Plotting with 2 algorithms.\")\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        p2 = plt.bar(r, fi_list[1], bottom=fi_list[0], color=algColors[1], edgecolor='white', width=barWidth)\n",
    "        lines = (p1[0],p2[0])\n",
    "        \n",
    "    elif len(algorithmsToRun) == 3:\n",
    "        print(\"Plotting with 3 algorithms.\")\n",
    "        bottom3 = [sum(i) for i in zip(fi_list[0], fi_list[1])]\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        p2 = plt.bar(r, fi_list[1], bottom=fi_list[0], color=algColors[1], edgecolor='white', width=barWidth)\n",
    "        p3 = plt.bar(r, fi_list[2], bottom=bottom3, color=algColors[2], edgecolor='white', width=barWidth)\n",
    "        lines = (p1[0],p2[0],p3[0])\n",
    "        \n",
    "    elif len(algorithmsToRun) == 4:\n",
    "        print(\"Plotting with 4 algorithms.\")\n",
    "        bottom3 = [sum(i) for i in zip(fi_list[0], fi_list[1])]\n",
    "        bottom4 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2])]\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        p2 = plt.bar(r, fi_list[1], bottom=fi_list[0], color=algColors[1], edgecolor='white', width=barWidth)\n",
    "        p3 = plt.bar(r, fi_list[2], bottom=bottom3, color=algColors[2], edgecolor='white', width=barWidth)\n",
    "        p4 = plt.bar(r, fi_list[3], bottom=bottom4, color=algColors[3], edgecolor='white', width=barWidth)\n",
    "        lines = (p1[0],p2[0],p3[0],p4[0])\n",
    "        \n",
    "    elif len(algorithmsToRun) == 5:\n",
    "        print(\"Plotting with 5 algorithms.\")\n",
    "        bottom3 = [sum(i) for i in zip(fi_list[0], fi_list[1])]\n",
    "        bottom4 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2])]\n",
    "        bottom5 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3])]\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        p2 = plt.bar(r, fi_list[1], bottom=fi_list[0], color=algColors[1], edgecolor='white', width=barWidth)\n",
    "        p3 = plt.bar(r, fi_list[2], bottom=bottom3, color=algColors[2], edgecolor='white', width=barWidth)\n",
    "        p4 = plt.bar(r, fi_list[3], bottom=bottom4, color=algColors[3], edgecolor='white', width=barWidth)\n",
    "        p5 = plt.bar(r, fi_list[4], bottom=bottom5, color=algColors[4], edgecolor='white', width=barWidth)\n",
    "        lines = (p1[0],p2[0],p3[0],p4[0],p5[0])\n",
    "        \n",
    "    elif len(algorithmsToRun) == 6:\n",
    "        print(\"Plotting with 6 algorithms.\")\n",
    "        bottom3 = [sum(i) for i in zip(fi_list[0], fi_list[1])]\n",
    "        bottom4 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2])]\n",
    "        bottom5 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3])]\n",
    "        bottom6 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4])]\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        p2 = plt.bar(r, fi_list[1], bottom=fi_list[0], color=algColors[1], edgecolor='white', width=barWidth)\n",
    "        p3 = plt.bar(r, fi_list[2], bottom=bottom3, color=algColors[2], edgecolor='white', width=barWidth)\n",
    "        p4 = plt.bar(r, fi_list[3], bottom=bottom4, color=algColors[3], edgecolor='white', width=barWidth)\n",
    "        p5 = plt.bar(r, fi_list[4], bottom=bottom5, color=algColors[4], edgecolor='white', width=barWidth)\n",
    "        p6 = plt.bar(r, fi_list[5], bottom=bottom6, color=algColors[5], edgecolor='white', width=barWidth)\n",
    "        lines = (p1[0],p2[0],p3[0],p4[0],p5[0],p6[0])\n",
    "        \n",
    "    elif len(algorithmsToRun) == 7:\n",
    "        print(\"Plotting with 7 algorithms.\")\n",
    "        bottom3 = [sum(i) for i in zip(fi_list[0], fi_list[1])]\n",
    "        bottom4 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2])]\n",
    "        bottom5 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3])]\n",
    "        bottom6 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4])]\n",
    "        bottom7 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5])]\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        p2 = plt.bar(r, fi_list[1], bottom=fi_list[0], color=algColors[1], edgecolor='white', width=barWidth)\n",
    "        p3 = plt.bar(r, fi_list[2], bottom=bottom3, color=algColors[2], edgecolor='white', width=barWidth)\n",
    "        p4 = plt.bar(r, fi_list[3], bottom=bottom4, color=algColors[3], edgecolor='white', width=barWidth)\n",
    "        p5 = plt.bar(r, fi_list[4], bottom=bottom5, color=algColors[4], edgecolor='white', width=barWidth)\n",
    "        p6 = plt.bar(r, fi_list[5], bottom=bottom6, color=algColors[5], edgecolor='white', width=barWidth)\n",
    "        p7 = plt.bar(r, fi_list[6], bottom=bottom7, color=algColors[6], edgecolor='white', width=barWidth)\n",
    "        lines = (p1[0],p2[0],p3[0],p4[0],p5[0],p6[0],p7[0])\n",
    "\n",
    "    elif len(algorithmsToRun) == 8:\n",
    "        print(\"Plotting with 8 algorithms.\")\n",
    "        bottom3 = [sum(i) for i in zip(fi_list[0], fi_list[1])]\n",
    "        bottom4 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2])]\n",
    "        bottom5 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3])]\n",
    "        bottom6 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4])]\n",
    "        bottom7 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5])]\n",
    "        bottom8 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5], fi_list[6])]\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        p2 = plt.bar(r, fi_list[1], bottom=fi_list[0], color=algColors[1], edgecolor='white', width=barWidth)\n",
    "        p3 = plt.bar(r, fi_list[2], bottom=bottom3, color=algColors[2], edgecolor='white', width=barWidth)\n",
    "        p4 = plt.bar(r, fi_list[3], bottom=bottom4, color=algColors[3], edgecolor='white', width=barWidth)\n",
    "        p5 = plt.bar(r, fi_list[4], bottom=bottom5, color=algColors[4], edgecolor='white', width=barWidth)\n",
    "        p6 = plt.bar(r, fi_list[5], bottom=bottom6, color=algColors[5], edgecolor='white', width=barWidth)\n",
    "        p7 = plt.bar(r, fi_list[6], bottom=bottom7, color=algColors[6], edgecolor='white', width=barWidth)\n",
    "        p8 = plt.bar(r, fi_list[7], bottom=bottom8, color=algColors[7], edgecolor='white', width=barWidth)\n",
    "        lines = (p1[0],p2[0],p3[0],p4[0],p5[0],p6[0],p7[0],p8[0])\n",
    "        \n",
    "    elif len(algorithmsToRun) == 9:\n",
    "        print(\"Plotting with 9 algorithms.\")\n",
    "        bottom3 = [sum(i) for i in zip(fi_list[0], fi_list[1])]\n",
    "        bottom4 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2])]\n",
    "        bottom5 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3])]\n",
    "        bottom6 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4])]\n",
    "        bottom7 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5])]\n",
    "        bottom8 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5], fi_list[6])]\n",
    "        bottom9 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5], fi_list[6], fi_list[7])]\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        p2 = plt.bar(r, fi_list[1], bottom=fi_list[0], color=algColors[1], edgecolor='white', width=barWidth)\n",
    "        p3 = plt.bar(r, fi_list[2], bottom=bottom3, color=algColors[2], edgecolor='white', width=barWidth)\n",
    "        p4 = plt.bar(r, fi_list[3], bottom=bottom4, color=algColors[3], edgecolor='white', width=barWidth)\n",
    "        p5 = plt.bar(r, fi_list[4], bottom=bottom5, color=algColors[4], edgecolor='white', width=barWidth)\n",
    "        p6 = plt.bar(r, fi_list[5], bottom=bottom6, color=algColors[5], edgecolor='white', width=barWidth)\n",
    "        p7 = plt.bar(r, fi_list[6], bottom=bottom7, color=algColors[6], edgecolor='white', width=barWidth)\n",
    "        p8 = plt.bar(r, fi_list[7], bottom=bottom8, color=algColors[7], edgecolor='white', width=barWidth)\n",
    "        p9 = plt.bar(r, fi_list[8], bottom=bottom9, color=algColors[8], edgecolor='white', width=barWidth)\n",
    "        lines = (p1[0],p2[0],p3[0],p4[0],p5[0],p6[0],p7[0],p8[0],p9[0]) \n",
    "\n",
    "    elif len(algorithmsToRun) == 10:\n",
    "        print(\"Plotting with 10 algorithms.\")\n",
    "        bottom3 = [sum(i) for i in zip(fi_list[0], fi_list[1])]\n",
    "        bottom4 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2])]\n",
    "        bottom5 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3])]\n",
    "        bottom6 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4])]\n",
    "        bottom7 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5])]\n",
    "        bottom8 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5], fi_list[6])]\n",
    "        bottom9 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5], fi_list[6], fi_list[7])]\n",
    "        bottom10 = [sum(i) for i in zip(fi_list[0], fi_list[1], fi_list[2], fi_list[3], fi_list[4], fi_list[5], fi_list[6], fi_list[7], fi_list[8])]\n",
    "\n",
    "        p1 = plt.bar(r, fi_list[0], color=algColors[0], edgecolor='white', width=barWidth)\n",
    "        p2 = plt.bar(r, fi_list[1], bottom=fi_list[0], color=algColors[1], edgecolor='white', width=barWidth)\n",
    "        p3 = plt.bar(r, fi_list[2], bottom=bottom3, color=algColors[2], edgecolor='white', width=barWidth)\n",
    "        p4 = plt.bar(r, fi_list[3], bottom=bottom4, color=algColors[3], edgecolor='white', width=barWidth)\n",
    "        p5 = plt.bar(r, fi_list[4], bottom=bottom5, color=algColors[4], edgecolor='white', width=barWidth)\n",
    "        p6 = plt.bar(r, fi_list[5], bottom=bottom6, color=algColors[5], edgecolor='white', width=barWidth)\n",
    "        p7 = plt.bar(r, fi_list[6], bottom=bottom7, color=algColors[6], edgecolor='white', width=barWidth)\n",
    "        p8 = plt.bar(r, fi_list[7], bottom=bottom8, color=algColors[7], edgecolor='white', width=barWidth)\n",
    "        p9 = plt.bar(r, fi_list[8], bottom=bottom9, color=algColors[8], edgecolor='white', width=barWidth)\n",
    "        p10 = plt.bar(r, fi_list[9], bottom=bottom10, color=algColors[9], edgecolor='white', width=barWidth)\n",
    "\n",
    "        lines = (p1[0],p2[0],p3[0],p4[0],p5[0],p6[0],p7[0],p8[0],p9[0],p10[0]) \n",
    "        \n",
    "    # Custom X axis\n",
    "    plt.xticks(np.arange(len(all_feature_listToViz)), all_feature_listToViz,rotation='vertical')\n",
    "    plt.xlabel(\"Feature\", fontsize=20)\n",
    "    plt.ylabel(\"Normalized Feature Importance\", fontsize=20)\n",
    "    plt.legend(lines, methodsKey, loc=0,fontsize=16)\n",
    "    plt.savefig(wd_path+output_folder+'/Compare_FI_'+figName+'_'+data_name, bbox_inches = 'tight')\n",
    "    # Show graphic\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compound Feature Importance Plot (Normalized only)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compound_FI_plot(top_fi_ave_norm_list,algorithmsToRun,algColors,all_feature_listToViz,'Norm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Fraction Adjustment of Normalized Feature importances\n",
    "- Each scores from each algorithm divided by sum of all scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fracLists = []\n",
    "\n",
    "for each in top_fi_ave_norm_list:\n",
    "    fracList = []\n",
    "    for i in range(len(each)):\n",
    "        fracList.append((each[i]/(sum(each))))\n",
    "    fracLists.append(fracList)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compound Feature Importance Plot (Normalized + Fraction Adjustment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compound_FI_plot(fracLists,algorithmsToRun,algColors,all_feature_listToViz,'Norm_Frac')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Performance Weighting of Normalized Feature importances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Prepare weights\n",
    "weights = []\n",
    "\n",
    "# replace all balanced accuraces <=.5 with 0\n",
    "for i in range(len(ave_metric_list)):\n",
    "    if ave_metric_list[i] <= .5:\n",
    "        ave_metric_list[i] = 0\n",
    "\n",
    "# normalize balanced accuracies\n",
    "for i in range(len(ave_metric_list)):\n",
    "    if ave_metric_list[i] == 0:\n",
    "        weights.append(0)\n",
    "    else:\n",
    "        weights.append((ave_metric_list[i]-0.5)/0.5)\n",
    "        \n",
    "        \n",
    "# Weight normalized feature importances\n",
    "weightedLists = []\n",
    "\n",
    "for i in range(len(top_fi_ave_norm_list)):\n",
    "    weightList = np.multiply(weights[i],top_fi_ave_norm_list[i]).tolist()\n",
    "    weightedLists.append(weightList)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compound Feature Importance Plot (Normalized + Perf. Weight Adjustment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compound_FI_plot(weightedLists,algorithmsToRun,algColors,all_feature_listToViz,'Norm_Weight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Performance Weighting of Fract. Adj. and Normalized Feature importances"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Weight normalized feature importances\n",
    "weightedFracLists = []\n",
    "\n",
    "for i in range(len(fracLists)):\n",
    "    weightList = np.multiply(weights[i],fracLists[i]).tolist()\n",
    "    weightedFracLists.append(weightList)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compound Feature Importance Plot (Normalized + Frac. + Perf. Weight Adjustment)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "compound_FI_plot(weightedFracLists,algorithmsToRun,algColors,all_feature_listToViz,'Norm_Frac_Weight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Runtime Report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "notebook_elapsed_time = time.time() - notebook_start_time\n",
    "print('Pipeline Run Time: '+ str(notebook_elapsed_time))\n",
    "run_time_dict['pipeline'] = notebook_elapsed_time\n",
    "print(run_time_dict)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}