ML Pipeline Preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ML Pipeline Preparation\n",
    "Follow the instructions below to help you create your ML pipeline.\n",
    "### 1. Import libraries and load data from database.\n",
    "- Import Python libraries\n",
    "- Load dataset from database with [`read_sql_table`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql_table.html)\n",
    "- Define feature and target variables X and Y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\jasper.kuller\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to\n",
      "[nltk_data]     C:\\Users\\jasper.kuller\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     C:\\Users\\jasper.kuller\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n"
     ]
    }
   ],
   "source": [
    "# import libraries\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sqlalchemy import create_engine\n",
    "\n",
    "import nltk\n",
    "nltk.download(['punkt', 'wordnet', 'averaged_perceptron_tagger'])\n",
    "\n",
    "import re\n",
    "from nltk.tokenize import word_tokenize\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "\n",
    "from sklearn.multioutput import MultiOutputClassifier\n",
    "from sklearn.pipeline import Pipeline, FeatureUnion\n",
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer\n",
    "from sklearn.ensemble import RandomForestClassifier, AdaBoostClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load data from database\n",
    "engine = create_engine('sqlite:///DisasterResponse.db')\n",
    "df = pd.read_sql_table('Messages', engine)\n",
    "X = df.iloc[:,1]\n",
    "y = df.iloc[:,4:]\n",
    "category_names = y.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Write a tokenization function to process your text data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(text):\n",
    "    tokens = word_tokenize(text)\n",
    "    lemmatizer = WordNetLemmatizer()\n",
    "\n",
    "    clean_tokens = []\n",
    "    for tok in tokens:\n",
    "        clean_tok = lemmatizer.lemmatize(tok).lower().strip()\n",
    "        clean_tokens.append(clean_tok)\n",
    "\n",
    "    return clean_tokens"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Build a machine learning pipeline\n",
    "This machine pipeline should take in the `message` column as input and output classification results on the other 36 categories in the dataset. You may find the [MultiOutputClassifier](http://scikit-learn.org/stable/modules/generated/sklearn.multioutput.MultiOutputClassifier.html) helpful for predicting multiple target variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline([\n",
    "    ('vect', CountVectorizer(tokenizer=tokenize)),\n",
    "    ('tfidf', TfidfTransformer()),\n",
    "    ('clf', MultiOutputClassifier(RandomForestClassifier()))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Train pipeline\n",
    "- Split data into train and test sets\n",
    "- Train pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n",
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Pipeline(memory=None,\n",
       "         steps=[('vect',\n",
       "                 CountVectorizer(analyzer='word', binary=False,\n",
       "                                 decode_error='strict',\n",
       "                                 dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                 input='content', lowercase=True, max_df=1.0,\n",
       "                                 max_features=None, min_df=1,\n",
       "                                 ngram_range=(1, 1), preprocessor=None,\n",
       "                                 stop_words=None, strip_accents=None,\n",
       "                                 token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                 tokenizer=<function tokenize at...\n",
       "                 MultiOutputClassifier(estimator=RandomForestClassifier(bootstrap=True,\n",
       "                                                                        class_weight=None,\n",
       "                                                                        criterion='gini',\n",
       "                                                                        max_depth=None,\n",
       "                                                                        max_features='auto',\n",
       "                                                                        max_leaf_nodes=None,\n",
       "                                                                        min_impurity_decrease=0.0,\n",
       "                                                                        min_impurity_split=None,\n",
       "                                                                        min_samples_leaf=1,\n",
       "                                                                        min_samples_split=2,\n",
       "                                                                        min_weight_fraction_leaf=0.0,\n",
       "                                                                        n_estimators='warn',\n",
       "                                                                        n_jobs=None,\n",
       "                                                                        oob_score=False,\n",
       "                                                                        random_state=None,\n",
       "                                                                        verbose=0,\n",
       "                                                                        warm_start=False),\n",
       "                                       n_jobs=None))],\n",
       "         verbose=False)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(X, y)\n",
    "\n",
    "pipeline.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Test your model\n",
    "Report the f1 score, precision and recall for each output category of the dataset. You can do this by iterating through the columns and calling sklearn's `classification_report` on each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = pipeline.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# y_pred.shape\n",
    "# type(y_pred)\n",
    "# y_pred.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# y_test.shape\n",
    "# type(y_test)\n",
    "# y_pred.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Category: related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.60      0.33      0.43      1523\n",
      "           1       0.82      0.93      0.87      5006\n",
      "\n",
      "    accuracy                           0.79      6529\n",
      "   macro avg       0.71      0.63      0.65      6529\n",
      "weighted avg       0.77      0.79      0.77      6529\n",
      "\n",
      "Category: request \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.98      0.93      5412\n",
      "           1       0.82      0.37      0.51      1117\n",
      "\n",
      "    accuracy                           0.88      6529\n",
      "   macro avg       0.85      0.68      0.72      6529\n",
      "weighted avg       0.87      0.88      0.86      6529\n",
      "\n",
      "Category: offer \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6494\n",
      "           1       0.00      0.00      0.00        35\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.50      0.50      0.50      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: aid_related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.71      0.88      0.79      3814\n",
      "           1       0.75      0.50      0.60      2715\n",
      "\n",
      "    accuracy                           0.72      6529\n",
      "   macro avg       0.73      0.69      0.69      6529\n",
      "weighted avg       0.73      0.72      0.71      6529\n",
      "\n",
      "Category: medical_help \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.96      6025\n",
      "           1       0.65      0.07      0.13       504\n",
      "\n",
      "    accuracy                           0.93      6529\n",
      "   macro avg       0.79      0.54      0.55      6529\n",
      "weighted avg       0.91      0.93      0.90      6529\n",
      "\n",
      "Category: medical_products \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6233\n",
      "           1       0.83      0.07      0.13       296\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.90      0.53      0.55      6529\n",
      "weighted avg       0.95      0.96      0.94      6529\n",
      "\n",
      "Category: search_and_rescue \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6363\n",
      "           1       0.67      0.04      0.07       166\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.82      0.52      0.53      6529\n",
      "weighted avg       0.97      0.98      0.96      6529\n",
      "\n",
      "Category: security \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6424\n",
      "           1       0.00      0.00      0.00       105\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.49      0.50      0.50      6529\n",
      "weighted avg       0.97      0.98      0.98      6529\n",
      "\n",
      "Category: military \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6332\n",
      "           1       0.53      0.04      0.08       197\n",
      "\n",
      "    accuracy                           0.97      6529\n",
      "   macro avg       0.75      0.52      0.53      6529\n",
      "weighted avg       0.96      0.97      0.96      6529\n",
      "\n",
      "Category:"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " child_alone \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6529\n",
      "\n",
      "    accuracy                           1.00      6529\n",
      "   macro avg       1.00      1.00      1.00      6529\n",
      "weighted avg       1.00      1.00      1.00      6529\n",
      "\n",
      "Category: water \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6110\n",
      "           1       0.85      0.19      0.31       419\n",
      "\n",
      "    accuracy                           0.95      6529\n",
      "   macro avg       0.90      0.59      0.64      6529\n",
      "weighted avg       0.94      0.95      0.93      6529\n",
      "\n",
      "Category: food \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.99      0.96      5797\n",
      "           1       0.83      0.40      0.54       732\n",
      "\n",
      "    accuracy                           0.92      6529\n",
      "   macro avg       0.88      0.69      0.75      6529\n",
      "weighted avg       0.92      0.92      0.91      6529\n",
      "\n",
      "Category: shelter \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.96      5977\n",
      "           1       0.80      0.20      0.32       552\n",
      "\n",
      "    accuracy                           0.93      6529\n",
      "   macro avg       0.86      0.60      0.64      6529\n",
      "weighted avg       0.92      0.93      0.91      6529\n",
      "\n",
      "Category: clothing \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6449\n",
      "           1       0.64      0.11      0.19        80\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.82      0.56      0.59      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: money \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6377\n",
      "           1       0.62      0.05      0.10       152\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.80      0.53      0.54      6529\n",
      "weighted avg       0.97      0.98      0.97      6529\n",
      "\n",
      "Category: missing_people \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6460\n",
      "           1       0.00      0.00      0.00        69\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.49      0.50      0.50      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: refugees \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6321\n",
      "           1       0.33      0.02      0.04       208\n",
      "\n",
      "    accuracy                           0.97      6529\n",
      "   macro avg       0.65      0.51      0.51      6529\n",
      "weighted avg       0.95      0.97      0.95      6529\n",
      "\n",
      "Category: death \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6229\n",
      "           1       0.79      0.18      0.29       300\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.88      0.59      0.63      6529\n",
      "weighted avg       0.95      0.96      0.95      6529\n",
      "\n",
      "Category: other_aid \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      1.00      0.93      5643\n",
      "           1       0.61      0.04      0.07       886\n",
      "\n",
      "    accuracy                           0.87      6529\n",
      "   macro avg       0.74      0.52      0.50      6529\n",
      "weighted avg       0.83      0.87      0.81      6529\n",
      "\n",
      "Category: infrastructure_related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97      6117\n",
      "           1       0.29      0.00      0.01       412\n",
      "\n",
      "    accuracy                           0.94      6529\n",
      "   macro avg       0.61      0.50      0.49      6529\n",
      "weighted avg       0.90      0.94      0.91      6529\n",
      "\n",
      "Category: transport \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6231\n",
      "           1       0.95      0.06      0.12       298\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.95      0.53      0.55      6529\n",
      "weighted avg       0.96      0.96      0.94      6529\n",
      "\n",
      "Category: buildings \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6226\n",
      "           1       0.76      0.11      0.19       303\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.86      0.55      0.58      6529\n",
      "weighted avg       0.95      0.96      0.94      6529\n",
      "\n",
      "Category: electricity \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6399\n",
      "           1       1.00      0.04      0.07       130\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.99      0.52      0.53      6529\n",
      "weighted avg       0.98      0.98      0.97      6529\n",
      "\n",
      "Category: tools \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6479\n",
      "           1       0.00      0.00      0.00        50\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.50      0.50      0.50      6529\n",
      "weighted avg       0.98      0.99      0.99      6529\n",
      "\n",
      "Category: hospitals \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6463\n",
      "           1       0.00      0.00      0.00        66\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.49      0.50      0.50      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: shops \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6491\n",
      "           1       0.00      0.00      0.00        38\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.50      0.50      0.50      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: aid_centers \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6459\n",
      "           1       0.00      0.00      0.00        70\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.49      0.50      0.50      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: other_infrastructure \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6257\n",
      "           1       0.00      0.00      0.00       272\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.48      0.50      0.49      6529\n",
      "weighted avg       0.92      0.96      0.94      6529\n",
      "\n",
      "Category: weather_related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.84      0.96      0.90      4684\n",
      "           1       0.84      0.54      0.66      1845\n",
      "\n",
      "    accuracy                           0.84      6529\n",
      "   macro avg       0.84      0.75      0.78      6529\n",
      "weighted avg       0.84      0.84      0.83      6529\n",
      "\n",
      "Category: floods \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97      6028\n",
      "           1       0.83      0.29      0.42       501\n",
      "\n",
      "    accuracy                           0.94      6529\n",
      "   macro avg       0.89      0.64      0.70      6529\n",
      "weighted avg       0.93      0.94      0.93      6529\n",
      "\n",
      "Category: storm \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      0.99      0.96      5900\n",
      "           1       0.76      0.33      0.46       629\n",
      "\n",
      "    accuracy                           0.93      6529\n",
      "   macro avg       0.85      0.66      0.71      6529\n",
      "weighted avg       0.92      0.93      0.91      6529\n",
      "\n",
      "Category: fire \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6458\n",
      "           1       0.00      0.00      0.00        71\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.49      0.50      0.50      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: earthquake \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      5886\n",
      "           1       0.91      0.69      0.79       643\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.94      0.84      0.88      6529\n",
      "weighted avg       0.96      0.96      0.96      6529\n",
      "\n",
      "Category: cold \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6383\n",
      "           1       0.67      0.05      0.10       146\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.82      0.53      0.55      6529\n",
      "weighted avg       0.97      0.98      0.97      6529\n",
      "\n",
      "Category: other_weather \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6204\n",
      "           1       0.39      0.02      0.04       325\n",
      "\n",
      "    accuracy                           0.95      6529\n",
      "   macro avg       0.67      0.51      0.51      6529\n",
      "weighted avg       0.92      0.95      0.93      6529\n",
      "\n",
      "Category: direct_report \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.85      0.98      0.91      5252\n",
      "           1       0.81      0.30      0.43      1277\n",
      "\n",
      "    accuracy                           0.85      6529\n",
      "   macro avg       0.83      0.64      0.67      6529\n",
      "weighted avg       0.84      0.85      0.82      6529\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(category_names)):\n",
    "    print(\"Category:\", category_names[i],\"\\n\", classification_report(y_test.iloc[:, i].values, y_pred[:, i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Improve your model\n",
    "Use grid search to find better parameters. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'memory': None,\n",
       " 'steps': [('vect',\n",
       "   CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "                   dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "                   lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "                   ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "                   strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                   tokenizer=<function tokenize at 0x0000021E88B26678>,\n",
       "                   vocabulary=None)),\n",
       "  ('tfidf',\n",
       "   TfidfTransformer(norm='l2', smooth_idf=True, sublinear_tf=False, use_idf=True)),\n",
       "  ('clf',\n",
       "   MultiOutputClassifier(estimator=RandomForestClassifier(bootstrap=True,\n",
       "                                                          class_weight=None,\n",
       "                                                          criterion='gini',\n",
       "                                                          max_depth=None,\n",
       "                                                          max_features='auto',\n",
       "                                                          max_leaf_nodes=None,\n",
       "                                                          min_impurity_decrease=0.0,\n",
       "                                                          min_impurity_split=None,\n",
       "                                                          min_samples_leaf=1,\n",
       "                                                          min_samples_split=2,\n",
       "                                                          min_weight_fraction_leaf=0.0,\n",
       "                                                          n_estimators='warn',\n",
       "                                                          n_jobs=None,\n",
       "                                                          oob_score=False,\n",
       "                                                          random_state=None,\n",
       "                                                          verbose=0,\n",
       "                                                          warm_start=False),\n",
       "                         n_jobs=None))],\n",
       " 'verbose': False,\n",
       " 'vect': CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "                 dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "                 lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "                 ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "                 strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                 tokenizer=<function tokenize at 0x0000021E88B26678>,\n",
       "                 vocabulary=None),\n",
       " 'tfidf': TfidfTransformer(norm='l2', smooth_idf=True, sublinear_tf=False, use_idf=True),\n",
       " 'clf': MultiOutputClassifier(estimator=RandomForestClassifier(bootstrap=True,\n",
       "                                                        class_weight=None,\n",
       "                                                        criterion='gini',\n",
       "                                                        max_depth=None,\n",
       "                                                        max_features='auto',\n",
       "                                                        max_leaf_nodes=None,\n",
       "                                                        min_impurity_decrease=0.0,\n",
       "                                                        min_impurity_split=None,\n",
       "                                                        min_samples_leaf=1,\n",
       "                                                        min_samples_split=2,\n",
       "                                                        min_weight_fraction_leaf=0.0,\n",
       "                                                        n_estimators='warn',\n",
       "                                                        n_jobs=None,\n",
       "                                                        oob_score=False,\n",
       "                                                        random_state=None,\n",
       "                                                        verbose=0,\n",
       "                                                        warm_start=False),\n",
       "                       n_jobs=None),\n",
       " 'vect__analyzer': 'word',\n",
       " 'vect__binary': False,\n",
       " 'vect__decode_error': 'strict',\n",
       " 'vect__dtype': numpy.int64,\n",
       " 'vect__encoding': 'utf-8',\n",
       " 'vect__input': 'content',\n",
       " 'vect__lowercase': True,\n",
       " 'vect__max_df': 1.0,\n",
       " 'vect__max_features': None,\n",
       " 'vect__min_df': 1,\n",
       " 'vect__ngram_range': (1, 1),\n",
       " 'vect__preprocessor': None,\n",
       " 'vect__stop_words': None,\n",
       " 'vect__strip_accents': None,\n",
       " 'vect__token_pattern': '(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       " 'vect__tokenizer': <function __main__.tokenize(text)>,\n",
       " 'vect__vocabulary': None,\n",
       " 'tfidf__norm': 'l2',\n",
       " 'tfidf__smooth_idf': True,\n",
       " 'tfidf__sublinear_tf': False,\n",
       " 'tfidf__use_idf': True,\n",
       " 'clf__estimator__bootstrap': True,\n",
       " 'clf__estimator__class_weight': None,\n",
       " 'clf__estimator__criterion': 'gini',\n",
       " 'clf__estimator__max_depth': None,\n",
       " 'clf__estimator__max_features': 'auto',\n",
       " 'clf__estimator__max_leaf_nodes': None,\n",
       " 'clf__estimator__min_impurity_decrease': 0.0,\n",
       " 'clf__estimator__min_impurity_split': None,\n",
       " 'clf__estimator__min_samples_leaf': 1,\n",
       " 'clf__estimator__min_samples_split': 2,\n",
       " 'clf__estimator__min_weight_fraction_leaf': 0.0,\n",
       " 'clf__estimator__n_estimators': 'warn',\n",
       " 'clf__estimator__n_jobs': None,\n",
       " 'clf__estimator__oob_score': False,\n",
       " 'clf__estimator__random_state': None,\n",
       " 'clf__estimator__verbose': 0,\n",
       " 'clf__estimator__warm_start': False,\n",
       " 'clf__estimator': RandomForestClassifier(bootstrap=True, class_weight=None, criterion='gini',\n",
       "                        max_depth=None, max_features='auto', max_leaf_nodes=None,\n",
       "                        min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "                        min_samples_leaf=1, min_samples_split=2,\n",
       "                        min_weight_fraction_leaf=0.0, n_estimators='warn',\n",
       "                        n_jobs=None, oob_score=False, random_state=None,\n",
       "                        verbose=0, warm_start=False),\n",
       " 'clf__n_jobs': None}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters = {\n",
    "    'clf__estimator__n_estimators': [10, 20, 30, 40, 50],\n",
    "    'clf__estimator__min_samples_leaf': [1, 2, 3, 4, 5]\n",
    "}\n",
    "\n",
    "cv = GridSearchCV(pipeline, param_grid=parameters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 7. Test your model\n",
    "Show the accuracy, precision, and recall of the tuned model.  \n",
    "\n",
    "Since this project focuses on code quality, process, and  pipelines, there is no minimum performance metric needed to pass. However, make sure to fine tune your models for accuracy, precision and recall to make your project stand out - especially for your portfolio!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_split.py:1978: FutureWarning: The default value of cv will change from 3 to 5 in version 0.22. Specify it explicitly to silence this warning.\n",
      "  warnings.warn(CV_WARNING, FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv='warn', error_score='raise-deprecating',\n",
       "             estimator=Pipeline(memory=None,\n",
       "                                steps=[('vect',\n",
       "                                        CountVectorizer(analyzer='word',\n",
       "                                                        binary=False,\n",
       "                                                        decode_error='strict',\n",
       "                                                        dtype=<class 'numpy.int64'>,\n",
       "                                                        encoding='utf-8',\n",
       "                                                        input='content',\n",
       "                                                        lowercase=True,\n",
       "                                                        max_df=1.0,\n",
       "                                                        max_features=None,\n",
       "                                                        min_df=1,\n",
       "                                                        ngram_range=(1, 1),\n",
       "                                                        preprocessor=None,\n",
       "                                                        stop_words=None,\n",
       "                                                        strip_accents=Non...\n",
       "                                                                                               min_weight_fraction_leaf=0.0,\n",
       "                                                                                               n_estimators='warn',\n",
       "                                                                                               n_jobs=None,\n",
       "                                                                                               oob_score=False,\n",
       "                                                                                               random_state=None,\n",
       "                                                                                               verbose=0,\n",
       "                                                                                               warm_start=False),\n",
       "                                                              n_jobs=None))],\n",
       "                                verbose=False),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'clf__estimator__min_samples_leaf': [1, 2, 3, 4, 5],\n",
       "                         'clf__estimator__n_estimators': [10, 20, 30, 40, 50]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'clf__estimator__min_samples_leaf': 1, 'clf__estimator__n_estimators': 50}"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_cv = cv.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Category: related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.75      0.27      0.40      1523\n",
      "           1       0.81      0.97      0.89      5006\n",
      "\n",
      "    accuracy                           0.81      6529\n",
      "   macro avg       0.78      0.62      0.64      6529\n",
      "weighted avg       0.80      0.81      0.77      6529\n",
      "\n",
      "Category: request \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.99      0.94      5412\n",
      "           1       0.87      0.42      0.57      1117\n",
      "\n",
      "    accuracy                           0.89      6529\n",
      "   macro avg       0.88      0.70      0.75      6529\n",
      "weighted avg       0.89      0.89      0.87      6529\n",
      "\n",
      "Category: offer \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6494\n",
      "           1       0.00      0.00      0.00        35\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.50      0.50      0.50      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: aid_related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.75      0.89      0.81      3814\n",
      "           1       0.78      0.58      0.67      2715\n",
      "\n",
      "    accuracy                           0.76      6529\n",
      "   macro avg       0.77      0.73      0.74      6529\n",
      "weighted avg       0.76      0.76      0.75      6529\n",
      "\n",
      "Category: medical_help \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.96      6025\n",
      "           1       0.69      0.05      0.10       504\n",
      "\n",
      "    accuracy                           0.93      6529\n",
      "   macro avg       0.81      0.53      0.53      6529\n",
      "weighted avg       0.91      0.93      0.89      6529\n",
      "\n",
      "Category: medical_products \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6233\n",
      "           1       0.79      0.06      0.12       296\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.87      0.53      0.55      6529\n",
      "weighted avg       0.95      0.96      0.94      6529\n",
      "\n",
      "Category: search_and_rescue \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6363\n",
      "           1       0.88      0.04      0.08       166\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.93      0.52      0.53      6529\n",
      "weighted avg       0.97      0.98      0.96      6529\n",
      "\n",
      "Category: security \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6424\n",
      "           1       0.00      0.00      0.00       105\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.49      0.50      0.50      6529\n",
      "weighted avg       0.97      0.98      0.98      6529\n",
      "\n",
      "Category: military \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6332\n",
      "           1       0.50      0.02      0.04       197\n",
      "\n",
      "    accuracy                           0.97      6529\n",
      "   macro avg       0.74      0.51      0.51      6529\n",
      "weighted avg       0.96      0.97      0.96      6529\n",
      "\n",
      "Category: child_alone \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6529\n",
      "\n",
      "    accuracy                           1.00      6529\n",
      "   macro avg       1.00      1.00      1.00      6529\n",
      "weighted avg       1.00      1.00      1.00      6529\n",
      "\n",
      "Category: water \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97      6110\n",
      "           1       0.90      0.15      0.26       419\n",
      "\n",
      "    accuracy                           0.94      6529\n",
      "   macro avg       0.92      0.57      0.61      6529\n",
      "weighted avg       0.94      0.94      0.93      6529\n",
      "\n",
      "Category:"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\metrics\\classification.py:1437: UndefinedMetricWarning: Precision and F-score are ill-defined and being set to 0.0 in labels with no predicted samples.\n",
      "  'precision', 'predicted', average, warn_for)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " food \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.92      0.99      0.96      5797\n",
      "           1       0.88      0.35      0.50       732\n",
      "\n",
      "    accuracy                           0.92      6529\n",
      "   macro avg       0.90      0.67      0.73      6529\n",
      "weighted avg       0.92      0.92      0.91      6529\n",
      "\n",
      "Category: shelter \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.93      1.00      0.96      5977\n",
      "           1       0.81      0.20      0.32       552\n",
      "\n",
      "    accuracy                           0.93      6529\n",
      "   macro avg       0.87      0.60      0.64      6529\n",
      "weighted avg       0.92      0.93      0.91      6529\n",
      "\n",
      "Category: clothing \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6449\n",
      "           1       0.43      0.04      0.07        80\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.71      0.52      0.53      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: money \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6377\n",
      "           1       0.75      0.04      0.07       152\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.86      0.52      0.53      6529\n",
      "weighted avg       0.97      0.98      0.97      6529\n",
      "\n",
      "Category: missing_people \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6460\n",
      "           1       1.00      0.01      0.03        69\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.99      0.51      0.51      6529\n",
      "weighted avg       0.99      0.99      0.98      6529\n",
      "\n",
      "Category: refugees \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      1.00      0.98      6321\n",
      "           1       0.86      0.03      0.06       208\n",
      "\n",
      "    accuracy                           0.97      6529\n",
      "   macro avg       0.91      0.51      0.52      6529\n",
      "weighted avg       0.97      0.97      0.95      6529\n",
      "\n",
      "Category: death \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6229\n",
      "           1       0.96      0.08      0.15       300\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.96      0.54      0.56      6529\n",
      "weighted avg       0.96      0.96      0.94      6529\n",
      "\n",
      "Category: other_aid \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.87      1.00      0.93      5643\n",
      "           1       0.73      0.02      0.04       886\n",
      "\n",
      "    accuracy                           0.87      6529\n",
      "   macro avg       0.80      0.51      0.48      6529\n",
      "weighted avg       0.85      0.87      0.81      6529\n",
      "\n",
      "Category: infrastructure_related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      1.00      0.97      6117\n",
      "           1       1.00      0.00      0.01       412\n",
      "\n",
      "    accuracy                           0.94      6529\n",
      "   macro avg       0.97      0.50      0.49      6529\n",
      "weighted avg       0.94      0.94      0.91      6529\n",
      "\n",
      "Category: transport \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6231\n",
      "           1       0.80      0.03      0.05       298\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.88      0.51      0.51      6529\n",
      "weighted avg       0.95      0.96      0.93      6529\n",
      "\n",
      "Category: buildings \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6226\n",
      "           1       0.85      0.09      0.17       303\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.90      0.55      0.57      6529\n",
      "weighted avg       0.95      0.96      0.94      6529\n",
      "\n",
      "Category: electricity \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6399\n",
      "           1       1.00      0.05      0.10       130\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.99      0.53      0.55      6529\n",
      "weighted avg       0.98      0.98      0.97      6529\n",
      "\n",
      "Category: tools \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6479\n",
      "           1       0.00      0.00      0.00        50\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.50      0.50      0.50      6529\n",
      "weighted avg       0.98      0.99      0.99      6529\n",
      "\n",
      "Category: hospitals \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6463\n",
      "           1       0.00      0.00      0.00        66\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.49      0.50      0.50      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: shops \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6491\n",
      "           1       0.00      0.00      0.00        38\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.50      0.50      0.50      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: aid_centers \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6459\n",
      "           1       0.00      0.00      0.00        70\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.49      0.50      0.50      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: other_infrastructure \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6257\n",
      "           1       0.00      0.00      0.00       272\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.48      0.50      0.49      6529\n",
      "weighted avg       0.92      0.96      0.94      6529\n",
      "\n",
      "Category: weather_related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.86      0.96      0.91      4684\n",
      "           1       0.86      0.60      0.71      1845\n",
      "\n",
      "    accuracy                           0.86      6529\n",
      "   macro avg       0.86      0.78      0.81      6529\n",
      "weighted avg       0.86      0.86      0.85      6529\n",
      "\n",
      "Category: floods \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6028\n",
      "           1       0.87      0.37      0.52       501\n",
      "\n",
      "    accuracy                           0.95      6529\n",
      "   macro avg       0.91      0.68      0.74      6529\n",
      "weighted avg       0.94      0.95      0.94      6529\n",
      "\n",
      "Category: storm \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.96      5900\n",
      "           1       0.80      0.37      0.51       629\n",
      "\n",
      "    accuracy                           0.93      6529\n",
      "   macro avg       0.87      0.68      0.74      6529\n",
      "weighted avg       0.92      0.93      0.92      6529\n",
      "\n",
      "Category: fire \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6458\n",
      "           1       0.00      0.00      0.00        71\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.49      0.50      0.50      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: earthquake \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      5886\n",
      "           1       0.92      0.69      0.79       643\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.94      0.84      0.88      6529\n",
      "weighted avg       0.96      0.96      0.96      6529\n",
      "\n",
      "Category: cold \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6383\n",
      "           1       1.00      0.08      0.14       146\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.99      0.54      0.56      6529\n",
      "weighted avg       0.98      0.98      0.97      6529\n",
      "\n",
      "Category: other_weather \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      1.00      0.97      6204\n",
      "           1       0.45      0.03      0.05       325\n",
      "\n",
      "    accuracy                           0.95      6529\n",
      "   macro avg       0.70      0.51      0.51      6529\n",
      "weighted avg       0.93      0.95      0.93      6529\n",
      "\n",
      "Category: direct_report \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.86      0.99      0.92      5252\n",
      "           1       0.88      0.35      0.50      1277\n",
      "\n",
      "    accuracy                           0.86      6529\n",
      "   macro avg       0.87      0.67      0.71      6529\n",
      "weighted avg       0.86      0.86      0.84      6529\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(category_names)):\n",
    "    print(\"Category:\", category_names[i],\"\\n\", classification_report(y_test.iloc[:, i].values, y_pred_cv[:, i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 8. Try improving your model further. Here are a few ideas:\n",
    "* try other machine learning algorithms\n",
    "* add other features besides the TF-IDF"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "class StartingVerbExtractor(BaseEstimator, TransformerMixin):\n",
    "\n",
    "    def starting_verb(self, text):\n",
    "        sentence_list = nltk.sent_tokenize(text)\n",
    "        for sentence in sentence_list:\n",
    "            pos_tags = nltk.pos_tag(tokenize(sentence))\n",
    "            first_word, first_tag = pos_tags[0]\n",
    "            if first_tag in ['VB', 'VBP'] or first_word == 'RT':\n",
    "                return True\n",
    "        return False\n",
    "\n",
    "    def fit(self, x, y=None):\n",
    "        return self\n",
    "\n",
    "    def transform(self, X):\n",
    "        X_tagged = pd.Series(X).apply(self.starting_verb)\n",
    "        return pd.DataFrame(X_tagged)\n",
    "\n",
    "pipeline2 = Pipeline([\n",
    "        ('features', FeatureUnion([\n",
    "\n",
    "            ('text_pipeline', Pipeline([\n",
    "                ('vect', CountVectorizer(tokenizer=tokenize)),\n",
    "                ('tfidf', TfidfTransformer())\n",
    "            ])),\n",
    "\n",
    "            ('starting_verb', StartingVerbExtractor())\n",
    "        ])),\n",
    "\n",
    "        ('clf', MultiOutputClassifier(AdaBoostClassifier()))\n",
    "    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'memory': None, 'steps': [('features', FeatureUnion(n_jobs=None,\n",
       "                transformer_list=[('text_pipeline',\n",
       "                                   Pipeline(memory=None,\n",
       "                                            steps=[('vect',\n",
       "                                                    CountVectorizer(analyzer='word',\n",
       "                                                                    binary=False,\n",
       "                                                                    decode_error='strict',\n",
       "                                                                    dtype=<class 'numpy.int64'>,\n",
       "                                                                    encoding='utf-8',\n",
       "                                                                    input='content',\n",
       "                                                                    lowercase=True,\n",
       "                                                                    max_df=1.0,\n",
       "                                                                    max_features=None,\n",
       "                                                                    min_df=1,\n",
       "                                                                    ngram_range=(1,\n",
       "                                                                                 1),\n",
       "                                                                    preprocessor=None,\n",
       "                                                                    stop_words=None,\n",
       "                                                                    strip_accents=None,\n",
       "                                                                    token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                                                    tokenizer=<function tokenize at 0x0000021E88B26678>,\n",
       "                                                                    vocabulary=None)),\n",
       "                                                   ('tfidf',\n",
       "                                                    TfidfTransformer(norm='l2',\n",
       "                                                                     smooth_idf=True,\n",
       "                                                                     sublinear_tf=False,\n",
       "                                                                     use_idf=True))],\n",
       "                                            verbose=False)),\n",
       "                                  ('starting_verb', StartingVerbExtractor())],\n",
       "                transformer_weights=None, verbose=False)),\n",
       "  ('clf',\n",
       "   MultiOutputClassifier(estimator=AdaBoostClassifier(algorithm='SAMME.R',\n",
       "                                                      base_estimator=None,\n",
       "                                                      learning_rate=1.0,\n",
       "                                                      n_estimators=50,\n",
       "                                                      random_state=None),\n",
       "                         n_jobs=None))], 'verbose': False, 'features': FeatureUnion(n_jobs=None,\n",
       "              transformer_list=[('text_pipeline',\n",
       "                                 Pipeline(memory=None,\n",
       "                                          steps=[('vect',\n",
       "                                                  CountVectorizer(analyzer='word',\n",
       "                                                                  binary=False,\n",
       "                                                                  decode_error='strict',\n",
       "                                                                  dtype=<class 'numpy.int64'>,\n",
       "                                                                  encoding='utf-8',\n",
       "                                                                  input='content',\n",
       "                                                                  lowercase=True,\n",
       "                                                                  max_df=1.0,\n",
       "                                                                  max_features=None,\n",
       "                                                                  min_df=1,\n",
       "                                                                  ngram_range=(1,\n",
       "                                                                               1),\n",
       "                                                                  preprocessor=None,\n",
       "                                                                  stop_words=None,\n",
       "                                                                  strip_accents=None,\n",
       "                                                                  token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                                                  tokenizer=<function tokenize at 0x0000021E88B26678>,\n",
       "                                                                  vocabulary=None)),\n",
       "                                                 ('tfidf',\n",
       "                                                  TfidfTransformer(norm='l2',\n",
       "                                                                   smooth_idf=True,\n",
       "                                                                   sublinear_tf=False,\n",
       "                                                                   use_idf=True))],\n",
       "                                          verbose=False)),\n",
       "                                ('starting_verb', StartingVerbExtractor())],\n",
       "              transformer_weights=None, verbose=False), 'clf': MultiOutputClassifier(estimator=AdaBoostClassifier(algorithm='SAMME.R',\n",
       "                                                    base_estimator=None,\n",
       "                                                    learning_rate=1.0,\n",
       "                                                    n_estimators=50,\n",
       "                                                    random_state=None),\n",
       "                       n_jobs=None), 'features__n_jobs': None, 'features__transformer_list': [('text_pipeline',\n",
       "   Pipeline(memory=None,\n",
       "            steps=[('vect',\n",
       "                    CountVectorizer(analyzer='word', binary=False,\n",
       "                                    decode_error='strict',\n",
       "                                    dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                    input='content', lowercase=True, max_df=1.0,\n",
       "                                    max_features=None, min_df=1,\n",
       "                                    ngram_range=(1, 1), preprocessor=None,\n",
       "                                    stop_words=None, strip_accents=None,\n",
       "                                    token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                    tokenizer=<function tokenize at 0x0000021E88B26678>,\n",
       "                                    vocabulary=None)),\n",
       "                   ('tfidf',\n",
       "                    TfidfTransformer(norm='l2', smooth_idf=True,\n",
       "                                     sublinear_tf=False, use_idf=True))],\n",
       "            verbose=False)),\n",
       "  ('starting_verb',\n",
       "   StartingVerbExtractor())], 'features__transformer_weights': None, 'features__verbose': False, 'features__text_pipeline': Pipeline(memory=None,\n",
       "          steps=[('vect',\n",
       "                  CountVectorizer(analyzer='word', binary=False,\n",
       "                                  decode_error='strict',\n",
       "                                  dtype=<class 'numpy.int64'>, encoding='utf-8',\n",
       "                                  input='content', lowercase=True, max_df=1.0,\n",
       "                                  max_features=None, min_df=1,\n",
       "                                  ngram_range=(1, 1), preprocessor=None,\n",
       "                                  stop_words=None, strip_accents=None,\n",
       "                                  token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                                  tokenizer=<function tokenize at 0x0000021E88B26678>,\n",
       "                                  vocabulary=None)),\n",
       "                 ('tfidf',\n",
       "                  TfidfTransformer(norm='l2', smooth_idf=True,\n",
       "                                   sublinear_tf=False, use_idf=True))],\n",
       "          verbose=False), 'features__starting_verb': StartingVerbExtractor(), 'features__text_pipeline__memory': None, 'features__text_pipeline__steps': [('vect',\n",
       "   CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "                   dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "                   lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "                   ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "                   strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                   tokenizer=<function tokenize at 0x0000021E88B26678>,\n",
       "                   vocabulary=None)),\n",
       "  ('tfidf',\n",
       "   TfidfTransformer(norm='l2', smooth_idf=True, sublinear_tf=False, use_idf=True))], 'features__text_pipeline__verbose': False, 'features__text_pipeline__vect': CountVectorizer(analyzer='word', binary=False, decode_error='strict',\n",
       "                 dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',\n",
       "                 lowercase=True, max_df=1.0, max_features=None, min_df=1,\n",
       "                 ngram_range=(1, 1), preprocessor=None, stop_words=None,\n",
       "                 strip_accents=None, token_pattern='(?u)\\\\b\\\\w\\\\w+\\\\b',\n",
       "                 tokenizer=<function tokenize at 0x0000021E88B26678>,\n",
       "                 vocabulary=None), 'features__text_pipeline__tfidf': TfidfTransformer(norm='l2', smooth_idf=True, sublinear_tf=False, use_idf=True), 'features__text_pipeline__vect__analyzer': 'word', 'features__text_pipeline__vect__binary': False, 'features__text_pipeline__vect__decode_error': 'strict', 'features__text_pipeline__vect__dtype': numpy.int64, 'features__text_pipeline__vect__encoding': 'utf-8', 'features__text_pipeline__vect__input': 'content', 'features__text_pipeline__vect__lowercase': True, 'features__text_pipeline__vect__max_df': 1.0, 'features__text_pipeline__vect__max_features': None, 'features__text_pipeline__vect__min_df': 1, 'features__text_pipeline__vect__ngram_range': (1,\n",
       "  1), 'features__text_pipeline__vect__preprocessor': None, 'features__text_pipeline__vect__stop_words': None, 'features__text_pipeline__vect__strip_accents': None, 'features__text_pipeline__vect__token_pattern': '(?u)\\\\b\\\\w\\\\w+\\\\b', 'features__text_pipeline__vect__tokenizer': <function __main__.tokenize(text)>, 'features__text_pipeline__vect__vocabulary': None, 'features__text_pipeline__tfidf__norm': 'l2', 'features__text_pipeline__tfidf__smooth_idf': True, 'features__text_pipeline__tfidf__sublinear_tf': False, 'features__text_pipeline__tfidf__use_idf': True, 'clf__estimator__algorithm': 'SAMME.R', 'clf__estimator__base_estimator': None, 'clf__estimator__learning_rate': 1.0, 'clf__estimator__n_estimators': 50, 'clf__estimator__random_state': None, 'clf__estimator': AdaBoostClassifier(algorithm='SAMME.R', base_estimator=None, learning_rate=1.0,\n",
       "                    n_estimators=50, random_state=None), 'clf__n_jobs': None}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipeline2.get_params()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "parameters2 = {\n",
    "    'clf__estimator__n_estimators': [10, 20, 30, 40, 50]\n",
    "}\n",
    "\n",
    "cv2 = GridSearchCV(pipeline2, param_grid=parameters2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\jasper.kuller\\AppData\\Local\\Continuum\\anaconda3\\lib\\site-packages\\sklearn\\model_selection\\_split.py:1978: FutureWarning: The default value of cv will change from 3 to 5 in version 0.22. Specify it explicitly to silence this warning.\n",
      "  warnings.warn(CV_WARNING, FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv='warn', error_score='raise-deprecating',\n",
       "             estimator=Pipeline(memory=None,\n",
       "                                steps=[('features',\n",
       "                                        FeatureUnion(n_jobs=None,\n",
       "                                                     transformer_list=[('text_pipeline',\n",
       "                                                                        Pipeline(memory=None,\n",
       "                                                                                 steps=[('vect',\n",
       "                                                                                         CountVectorizer(analyzer='word',\n",
       "                                                                                                         binary=False,\n",
       "                                                                                                         decode_error='strict',\n",
       "                                                                                                         dtype=<class 'numpy.int64'>,\n",
       "                                                                                                         encoding='utf-8',\n",
       "                                                                                                         input='content',\n",
       "                                                                                                         lowercase=True,\n",
       "                                                                                                         max_df=...\n",
       "                                                     verbose=False)),\n",
       "                                       ('clf',\n",
       "                                        MultiOutputClassifier(estimator=AdaBoostClassifier(algorithm='SAMME.R',\n",
       "                                                                                           base_estimator=None,\n",
       "                                                                                           learning_rate=1.0,\n",
       "                                                                                           n_estimators=50,\n",
       "                                                                                           random_state=None),\n",
       "                                                              n_jobs=None))],\n",
       "                                verbose=False),\n",
       "             iid='warn', n_jobs=None,\n",
       "             param_grid={'clf__estimator__n_estimators': [10, 20, 30, 40, 50]},\n",
       "             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,\n",
       "             scoring=None, verbose=0)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv2.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'clf__estimator__n_estimators': 40}"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# I'll take this configuration for the final model (python script)\n",
    "cv2.best_params_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_cv2 = cv2.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Category: related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.65      0.35      0.46      1523\n",
      "           1       0.83      0.94      0.88      5006\n",
      "\n",
      "    accuracy                           0.81      6529\n",
      "   macro avg       0.74      0.65      0.67      6529\n",
      "weighted avg       0.79      0.81      0.78      6529\n",
      "\n",
      "Category: request \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.91      0.96      0.93      5412\n",
      "           1       0.75      0.52      0.62      1117\n",
      "\n",
      "    accuracy                           0.89      6529\n",
      "   macro avg       0.83      0.74      0.78      6529\n",
      "weighted avg       0.88      0.89      0.88      6529\n",
      "\n",
      "Category: offer \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6494\n",
      "           1       0.00      0.00      0.00        35\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.50      0.50      0.50      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: aid_related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.74      0.87      0.80      3814\n",
      "           1       0.76      0.58      0.66      2715\n",
      "\n",
      "    accuracy                           0.75      6529\n",
      "   macro avg       0.75      0.73      0.73      6529\n",
      "weighted avg       0.75      0.75      0.74      6529\n",
      "\n",
      "Category: medical_help \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.96      6025\n",
      "           1       0.58      0.24      0.34       504\n",
      "\n",
      "    accuracy                           0.93      6529\n",
      "   macro avg       0.76      0.61      0.65      6529\n",
      "weighted avg       0.91      0.93      0.91      6529\n",
      "\n",
      "Category: medical_products \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6233\n",
      "           1       0.68      0.33      0.45       296\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.82      0.66      0.71      6529\n",
      "weighted avg       0.96      0.96      0.96      6529\n",
      "\n",
      "Category: search_and_rescue \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6363\n",
      "           1       0.58      0.23      0.33       166\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.78      0.61      0.66      6529\n",
      "weighted avg       0.97      0.98      0.97      6529\n",
      "\n",
      "Category: security \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6424\n",
      "           1       0.23      0.05      0.08       105\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.61      0.52      0.53      6529\n",
      "weighted avg       0.97      0.98      0.98      6529\n",
      "\n",
      "Category: military \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.99      6332\n",
      "           1       0.57      0.34      0.43       197\n",
      "\n",
      "    accuracy                           0.97      6529\n",
      "   macro avg       0.78      0.67      0.71      6529\n",
      "weighted avg       0.97      0.97      0.97      6529\n",
      "\n",
      "Category: child_alone \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       1.00      1.00      1.00      6529\n",
      "\n",
      "    accuracy                           1.00      6529\n",
      "   macro avg       1.00      1.00      1.00      6529\n",
      "weighted avg       1.00      1.00      1.00      6529\n",
      "\n",
      "Category: water \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.98      0.98      6110\n",
      "           1       0.73      0.66      0.69       419\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.85      0.82      0.84      6529\n",
      "weighted avg       0.96      0.96      0.96      6529\n",
      "\n",
      "Category: food \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.98      0.97      5797\n",
      "           1       0.81      0.70      0.75       732\n",
      "\n",
      "    accuracy                           0.95      6529\n",
      "   macro avg       0.88      0.84      0.86      6529\n",
      "weighted avg       0.95      0.95      0.95      6529\n",
      "\n",
      "Category: shelter \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.98      0.97      5977\n",
      "           1       0.76      0.54      0.63       552\n",
      "\n",
      "    accuracy                           0.95      6529\n",
      "   macro avg       0.86      0.76      0.80      6529\n",
      "weighted avg       0.94      0.95      0.94      6529\n",
      "\n",
      "Category: clothing \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6449\n",
      "           1       0.61      0.39      0.47        80\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.80      0.69      0.73      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: money \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      1.00      0.99      6377\n",
      "           1       0.58      0.25      0.35       152\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.78      0.62      0.67      6529\n",
      "weighted avg       0.97      0.98      0.97      6529\n",
      "\n",
      "Category: missing_people \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6460\n",
      "           1       0.57      0.25      0.34        69\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.78      0.62      0.67      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: refugees \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6321\n",
      "           1       0.51      0.22      0.31       208\n",
      "\n",
      "    accuracy                           0.97      6529\n",
      "   macro avg       0.74      0.61      0.65      6529\n",
      "weighted avg       0.96      0.97      0.96      6529\n",
      "\n",
      "Category: death \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.98      0.99      0.98      6229\n",
      "           1       0.81      0.47      0.60       300\n",
      "\n",
      "    accuracy                           0.97      6529\n",
      "   macro avg       0.89      0.73      0.79      6529\n",
      "weighted avg       0.97      0.97      0.97      6529\n",
      "\n",
      "Category: other_aid \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.98      0.92      5643\n",
      "           1       0.45      0.11      0.18       886\n",
      "\n",
      "    accuracy                           0.86      6529\n",
      "   macro avg       0.66      0.54      0.55      6529\n",
      "weighted avg       0.82      0.86      0.82      6529\n",
      "\n",
      "Category: infrastructure_related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.94      0.99      0.97      6117\n",
      "           1       0.44      0.09      0.16       412\n",
      "\n",
      "    accuracy                           0.94      6529\n",
      "   macro avg       0.69      0.54      0.56      6529\n",
      "weighted avg       0.91      0.94      0.92      6529\n",
      "\n",
      "Category: transport \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      1.00      0.98      6231\n",
      "           1       0.72      0.24      0.37       298\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.84      0.62      0.67      6529\n",
      "weighted avg       0.95      0.96      0.95      6529\n",
      "\n",
      "Category: buildings \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6226\n",
      "           1       0.57      0.38      0.46       303\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.77      0.68      0.72      6529\n",
      "weighted avg       0.95      0.96      0.95      6529\n",
      "\n",
      "Category: electricity \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6399\n",
      "           1       0.54      0.28      0.37       130\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.76      0.64      0.68      6529\n",
      "weighted avg       0.98      0.98      0.98      6529\n",
      "\n",
      "Category: tools \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6479\n",
      "           1       0.50      0.04      0.07        50\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.75      0.52      0.54      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: hospitals \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6463\n",
      "           1       0.22      0.08      0.11        66\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.60      0.54      0.55      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: shops \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6491\n",
      "           1       0.00      0.00      0.00        38\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.50      0.50      0.50      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: aid_centers \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6459\n",
      "           1       0.24      0.09      0.13        70\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.62      0.54      0.56      6529\n",
      "weighted avg       0.98      0.99      0.98      6529\n",
      "\n",
      "Category: other_infrastructure \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.98      6257\n",
      "           1       0.31      0.08      0.13       272\n",
      "\n",
      "    accuracy                           0.95      6529\n",
      "   macro avg       0.63      0.54      0.55      6529\n",
      "weighted avg       0.93      0.95      0.94      6529\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Category: weather_related \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.96      0.91      4684\n",
      "           1       0.86      0.65      0.74      1845\n",
      "\n",
      "    accuracy                           0.87      6529\n",
      "   macro avg       0.87      0.81      0.83      6529\n",
      "weighted avg       0.87      0.87      0.87      6529\n",
      "\n",
      "Category: floods \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      6028\n",
      "           1       0.82      0.58      0.68       501\n",
      "\n",
      "    accuracy                           0.96      6529\n",
      "   macro avg       0.89      0.78      0.83      6529\n",
      "weighted avg       0.95      0.96      0.95      6529\n",
      "\n",
      "Category: storm \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.95      0.98      0.97      5900\n",
      "           1       0.77      0.50      0.60       629\n",
      "\n",
      "    accuracy                           0.94      6529\n",
      "   macro avg       0.86      0.74      0.78      6529\n",
      "weighted avg       0.93      0.94      0.93      6529\n",
      "\n",
      "Category: fire \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      1.00      6458\n",
      "           1       0.67      0.31      0.42        71\n",
      "\n",
      "    accuracy                           0.99      6529\n",
      "   macro avg       0.83      0.65      0.71      6529\n",
      "weighted avg       0.99      0.99      0.99      6529\n",
      "\n",
      "Category: earthquake \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.97      0.99      0.98      5886\n",
      "           1       0.90      0.75      0.82       643\n",
      "\n",
      "    accuracy                           0.97      6529\n",
      "   macro avg       0.94      0.87      0.90      6529\n",
      "weighted avg       0.97      0.97      0.97      6529\n",
      "\n",
      "Category: cold \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.99      1.00      0.99      6383\n",
      "           1       0.69      0.34      0.46       146\n",
      "\n",
      "    accuracy                           0.98      6529\n",
      "   macro avg       0.84      0.67      0.72      6529\n",
      "weighted avg       0.98      0.98      0.98      6529\n",
      "\n",
      "Category: other_weather \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.96      0.99      0.97      6204\n",
      "           1       0.43      0.17      0.24       325\n",
      "\n",
      "    accuracy                           0.95      6529\n",
      "   macro avg       0.69      0.58      0.61      6529\n",
      "weighted avg       0.93      0.95      0.94      6529\n",
      "\n",
      "Category: direct_report \n",
      "               precision    recall  f1-score   support\n",
      "\n",
      "           0       0.88      0.96      0.92      5252\n",
      "           1       0.73      0.45      0.55      1277\n",
      "\n",
      "    accuracy                           0.86      6529\n",
      "   macro avg       0.80      0.70      0.74      6529\n",
      "weighted avg       0.85      0.86      0.85      6529\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for i in range(len(category_names)):\n",
    "    print(\"Category:\", category_names[i],\"\\n\", classification_report(y_test.iloc[:, i].values, y_pred_cv2[:, i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 9. Export your model as a pickle file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('model.pkl', 'wb') as file:\n",
    "    pickle.dump(cv2, file)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 10. Use this notebook to complete `train.py`\n",
    "Use the template file attached in the Resources folder to write a script that runs the steps above to create a database and export a model based on a new dataset specified by the user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}