examples/classification-r8-twidf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Multi-class Classification Task on the R8 dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goal of this Notebook\n",
    "\n",
    "The goal of this notebook is to demonstrate the usage of the TW-IDF model implemented in the [gowpy library](https://github.com/GuillaumeDD/gowpy) for a multi-class classification task. More precisely, TW-IDF is compared to a standard TF-IDF on the R8 dataset. \n",
    "\n",
    "Long story short: the TW-IDF model shows improved performance on this dataset compared to a standard TF-IDF model w.r.t.  accuracy, F1 (macro, micro) and MCC metrics.\n",
    "\n",
    "The R8 dataset is the preprocessed Reuters dataset with the top 8 classes. It contains 5,495 training documents and 2,189 testing documents, with 8 different labels. Preprocessing involves: tokenization, stop-words removal and stemming to the initial texts. The version of the dataset comes from this [github repository](https://github.com/Nath-B/Graph-Of-Words)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python Environment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preparation of the python environment:\n",
    "```bash\n",
    "pip install gowpy pandas\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loading the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('champion product approv stock split champion product inc board director approv two for stock split common share for sharehold record april compani board vote recommend sharehold annual meet april increas author capit stock mln mln share reuter',\n",
       " 'earn')"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.read_csv('datasets/r8/r8-train-stemmed.txt',\n",
    "                        header = None, \n",
    "                        sep='\\t', \n",
    "                        names = ['label', 'document'])\n",
    "X_train = df_train['document']\n",
    "y_train = df_train['label']\n",
    "\n",
    "X_train[0], y_train[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "earn        2840\n",
       "acq         1596\n",
       "crude        253\n",
       "trade        251\n",
       "money-fx     206\n",
       "interest     190\n",
       "ship         108\n",
       "grain         41\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "earn        1083\n",
       "acq          696\n",
       "crude        121\n",
       "money-fx      87\n",
       "interest      81\n",
       "trade         75\n",
       "ship          36\n",
       "grain         10\n",
       "Name: label, dtype: int64"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test = pd.read_csv('datasets/r8/r8-test-stemmed.txt',\n",
    "                        header = None, \n",
    "                        sep='\\t', \n",
    "                        names = ['label', 'document'])\n",
    "X_test = df_test['document']\n",
    "y_test = df_test['label']\n",
    "\n",
    "y_test.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classification task"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.pipeline import FeatureUnion\n",
    "\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer, TfidfTransformer\n",
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from pprint import pprint\n",
    "from time import time\n",
    "\n",
    "from sklearn.metrics import classification_report, matthews_corrcoef, accuracy_score, f1_score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hyper-optimisation metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import matthews_corrcoef, make_scorer\n",
    "scorer_mcc = make_scorer(matthews_corrcoef)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TF-IDF model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameter search and cross-validation score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline([\n",
    "    ('vect', TfidfVectorizer()),\n",
    "    ('svm', LinearSVC()),\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'vect__min_df': [5, 10, 20],\n",
    "    'vect__max_df': [0.85, 0.9, 0.95],\n",
    "    'vect__ngram_range': ((1, 1), (1, 2)),  # unigrams or bigrams\n",
    "    #\n",
    "    'svm__C' : [1, 10, 100, 1000],\n",
    "    'svm__class_weight' : [None, 'balanced']\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performing grid search...\n",
      "pipeline: ['vect', 'svm']\n",
      "parameters:\n",
      "{'svm__C': [1, 10, 100, 1000],\n",
      " 'svm__class_weight': [None, 'balanced'],\n",
      " 'vect__max_df': [0.85, 0.9, 0.95],\n",
      " 'vect__min_df': [5, 10, 20],\n",
      " 'vect__ngram_range': ((1, 1), (1, 2))}\n",
      "Fitting 10 folds for each of 144 candidates, totalling 1440 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done   5 tasks      | elapsed:    2.0s\n",
      "[Parallel(n_jobs=-1)]: Done  10 tasks      | elapsed:    2.6s\n",
      "[Parallel(n_jobs=-1)]: Done  17 tasks      | elapsed:    4.9s\n",
      "[Parallel(n_jobs=-1)]: Done  24 tasks      | elapsed:    6.1s\n",
      "[Parallel(n_jobs=-1)]: Done  33 tasks      | elapsed:    8.0s\n",
      "[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:   10.2s\n",
      "[Parallel(n_jobs=-1)]: Done  53 tasks      | elapsed:   12.4s\n",
      "[Parallel(n_jobs=-1)]: Done  64 tasks      | elapsed:   14.8s\n",
      "[Parallel(n_jobs=-1)]: Done  77 tasks      | elapsed:   18.1s\n",
      "[Parallel(n_jobs=-1)]: Done  90 tasks      | elapsed:   20.5s\n",
      "[Parallel(n_jobs=-1)]: Done 105 tasks      | elapsed:   24.7s\n",
      "[Parallel(n_jobs=-1)]: Done 120 tasks      | elapsed:   29.0s\n",
      "[Parallel(n_jobs=-1)]: Done 137 tasks      | elapsed:   33.8s\n",
      "[Parallel(n_jobs=-1)]: Done 154 tasks      | elapsed:   37.7s\n",
      "[Parallel(n_jobs=-1)]: Done 173 tasks      | elapsed:   43.1s\n",
      "[Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:   48.5s\n",
      "[Parallel(n_jobs=-1)]: Done 213 tasks      | elapsed:   54.4s\n",
      "[Parallel(n_jobs=-1)]: Done 234 tasks      | elapsed:  1.0min\n",
      "[Parallel(n_jobs=-1)]: Done 257 tasks      | elapsed:  1.1min\n",
      "[Parallel(n_jobs=-1)]: Done 280 tasks      | elapsed:  1.2min\n",
      "[Parallel(n_jobs=-1)]: Done 305 tasks      | elapsed:  1.3min\n",
      "[Parallel(n_jobs=-1)]: Done 330 tasks      | elapsed:  1.5min\n",
      "[Parallel(n_jobs=-1)]: Done 357 tasks      | elapsed:  1.6min\n",
      "[Parallel(n_jobs=-1)]: Done 384 tasks      | elapsed:  1.7min\n",
      "[Parallel(n_jobs=-1)]: Done 413 tasks      | elapsed:  1.9min\n",
      "[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed:  2.1min\n",
      "[Parallel(n_jobs=-1)]: Done 473 tasks      | elapsed:  2.2min\n",
      "[Parallel(n_jobs=-1)]: Done 504 tasks      | elapsed:  2.4min\n",
      "[Parallel(n_jobs=-1)]: Done 537 tasks      | elapsed:  2.6min\n",
      "[Parallel(n_jobs=-1)]: Done 570 tasks      | elapsed:  2.8min\n",
      "[Parallel(n_jobs=-1)]: Done 605 tasks      | elapsed:  3.0min\n",
      "[Parallel(n_jobs=-1)]: Done 640 tasks      | elapsed:  3.2min\n",
      "[Parallel(n_jobs=-1)]: Done 677 tasks      | elapsed:  3.5min\n",
      "[Parallel(n_jobs=-1)]: Done 714 tasks      | elapsed:  3.7min\n",
      "[Parallel(n_jobs=-1)]: Done 753 tasks      | elapsed:  4.1min\n",
      "[Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed:  4.4min\n",
      "[Parallel(n_jobs=-1)]: Done 833 tasks      | elapsed:  4.7min\n",
      "[Parallel(n_jobs=-1)]: Done 874 tasks      | elapsed:  5.1min\n",
      "[Parallel(n_jobs=-1)]: Done 917 tasks      | elapsed:  5.5min\n",
      "[Parallel(n_jobs=-1)]: Done 960 tasks      | elapsed:  5.9min\n",
      "[Parallel(n_jobs=-1)]: Done 1005 tasks      | elapsed:  6.3min\n",
      "[Parallel(n_jobs=-1)]: Done 1050 tasks      | elapsed:  6.7min\n",
      "[Parallel(n_jobs=-1)]: Done 1097 tasks      | elapsed:  7.1min\n",
      "[Parallel(n_jobs=-1)]: Done 1144 tasks      | elapsed:  7.4min\n",
      "[Parallel(n_jobs=-1)]: Done 1193 tasks      | elapsed:  7.9min\n",
      "[Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed:  8.3min\n",
      "[Parallel(n_jobs=-1)]: Done 1293 tasks      | elapsed:  8.7min\n",
      "[Parallel(n_jobs=-1)]: Done 1344 tasks      | elapsed:  9.2min\n",
      "[Parallel(n_jobs=-1)]: Done 1397 tasks      | elapsed:  9.7min\n",
      "[Parallel(n_jobs=-1)]: Done 1440 out of 1440 | elapsed: 10.0min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done in 600.469s\n",
      "\n",
      "Best score: 0.962\n",
      "Best parameters set:\n",
      "\tsvm__C: 1\n",
      "\tsvm__class_weight: 'balanced'\n",
      "\tvect__max_df: 0.85\n",
      "\tvect__min_df: 20\n",
      "\tvect__ngram_range: (1, 2)\n"
     ]
    }
   ],
   "source": [
    "# find the best parameters for both the feature extraction and the\n",
    "# classifier\n",
    "grid_search = GridSearchCV(pipeline, parameters, \n",
    "                           cv=10,\n",
    "                           scoring=scorer_mcc,\n",
    "                           n_jobs=-1, \n",
    "                           verbose=10)\n",
    "\n",
    "print(\"Performing grid search...\")\n",
    "print(\"pipeline:\", [name for name, _ in pipeline.steps])\n",
    "print(\"parameters:\")\n",
    "pprint(parameters)\n",
    "t0 = time()\n",
    "grid_search.fit(X_train, y_train)\n",
    "print(\"done in %0.3fs\" % (time() - t0))\n",
    "print()\n",
    "\n",
    "print(\"Best score: %0.3f\" % grid_search.best_score_)\n",
    "print(\"Best parameters set:\")\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "Best score: 0.962\n",
    "Best parameters set:\n",
    "\tsvm__C: 1\n",
    "\tsvm__class_weight: 'balanced'\n",
    "\tvect__max_df: 0.85\n",
    "\tvect__min_df: 20\n",
    "\tvect__ngram_range: (1, 2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fitting the final TF-IDF model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('vect',\n",
       "                 TfidfVectorizer(max_df=0.85, min_df=20, ngram_range=(1, 2))),\n",
       "                ('svm', LinearSVC(C=1, class_weight='balanced'))])"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\n",
    "# /!\\ manually set the best parameters\n",
    "#\n",
    "pipeline_tfidf = Pipeline([\n",
    "    ('vect', TfidfVectorizer(\n",
    "        min_df=20,\n",
    "        max_df=0.85,\n",
    "        ngram_range=(1, 2),\n",
    "    )),\n",
    "    ('svm', LinearSVC(\n",
    "        C=1,\n",
    "        class_weight='balanced',\n",
    "    )),\n",
    "])\n",
    "\n",
    "pipeline_tfidf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluation on the test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = pipeline_tfidf.predict(X_test)\n",
    "y_true = y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         acq       0.97      0.98      0.97       696\n",
      "       crude       0.95      0.93      0.94       121\n",
      "        earn       0.99      0.99      0.99      1083\n",
      "       grain       1.00      0.90      0.95        10\n",
      "    interest       0.89      0.86      0.87        81\n",
      "    money-fx       0.88      0.79      0.84        87\n",
      "        ship       0.84      0.89      0.86        36\n",
      "       trade       0.89      0.99      0.94        75\n",
      "\n",
      "    accuracy                           0.97      2189\n",
      "   macro avg       0.93      0.92      0.92      2189\n",
      "weighted avg       0.97      0.97      0.97      2189\n",
      "\n",
      "mcc=0.9519401384452915 ; accuracy=0.9689355870260393 ; f1-micro=0.9689355870260393\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_true, y_pred))\n",
    "\n",
    "mcc = matthews_corrcoef(y_true, y_pred)\n",
    "accuracy = accuracy_score(y_true, y_pred)\n",
    "f1 = f1_score(y_true, y_pred, average='micro')\n",
    "print(f\"mcc={mcc} ; accuracy={accuracy} ; f1-micro={f1}\")"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "              precision    recall  f1-score   support\n",
    "\n",
    "         acq       0.97      0.98      0.97       696\n",
    "       crude       0.95      0.93      0.94       121\n",
    "        earn       0.99      0.99      0.99      1083\n",
    "       grain       1.00      0.90      0.95        10\n",
    "    interest       0.89      0.86      0.87        81\n",
    "    money-fx       0.88      0.79      0.84        87\n",
    "        ship       0.84      0.89      0.86        36\n",
    "       trade       0.89      0.99      0.94        75\n",
    "\n",
    "    accuracy                           0.97      2189\n",
    "   macro avg       0.93      0.92      0.92      2189\n",
    "weighted avg       0.97      0.97      0.97      2189\n",
    "\n",
    "mcc=0.9519401384452915 ; accuracy=0.9689355870260393 ; f1-micro=0.9689355870260393"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TW-IDF model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "from gowpy.feature_extraction.gow import TwidfVectorizer\n",
    "from gowpy.feature_extraction.gow.tw_vectorizer import TERM_WEIGHT_DEGREE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Hyperparameter search  and cross-validation score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipeline = Pipeline([\n",
    "    ('gow', TwidfVectorizer()),\n",
    "    ('svm', LinearSVC()),\n",
    "])\n",
    "\n",
    "parameters = {\n",
    "    'gow__window_size' : [2, 4, 8, 16],\n",
    "    'gow__b' : [0.0, 0.003],\n",
    "    'gow__directed' : [False, True],\n",
    "    'gow__term_weighting' : [TERM_WEIGHT_DEGREE],\n",
    "#\n",
    "    'gow__min_df' : [5, 10, 20],\n",
    "    'gow__max_df' : [0.85, 0.9, 0.95],\n",
    "#\n",
    "    'svm__C' : [1, 10, 100, 1000],\n",
    "    'svm__class_weight' : [None, 'balanced'],\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Performing grid search...\n",
      "pipeline: ['gow', 'svm']\n",
      "parameters:\n",
      "{'gow__b': [0.0, 0.003],\n",
      " 'gow__directed': [False, True],\n",
      " 'gow__max_df': [0.85, 0.9, 0.95],\n",
      " 'gow__min_df': [5, 10, 20],\n",
      " 'gow__term_weighting': ['degree'],\n",
      " 'gow__window_size': [2, 4, 8, 16],\n",
      " 'svm__C': [1, 10, 100, 1000],\n",
      " 'svm__class_weight': [None, 'balanced']}\n",
      "Fitting 10 folds for each of 1152 candidates, totalling 11520 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 4 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done   5 tasks      | elapsed:    6.0s\n",
      "[Parallel(n_jobs=-1)]: Done  10 tasks      | elapsed:    8.6s\n",
      "[Parallel(n_jobs=-1)]: Done  17 tasks      | elapsed:   13.8s\n",
      "[Parallel(n_jobs=-1)]: Done  24 tasks      | elapsed:   17.1s\n",
      "[Parallel(n_jobs=-1)]: Done  33 tasks      | elapsed:   26.1s\n",
      "[Parallel(n_jobs=-1)]: Done  42 tasks      | elapsed:   34.0s\n",
      "[Parallel(n_jobs=-1)]: Done  53 tasks      | elapsed:   46.0s\n",
      "[Parallel(n_jobs=-1)]: Done  64 tasks      | elapsed:   55.1s\n",
      "[Parallel(n_jobs=-1)]: Done  77 tasks      | elapsed:  1.2min\n",
      "[Parallel(n_jobs=-1)]: Done  90 tasks      | elapsed:  1.4min\n",
      "[Parallel(n_jobs=-1)]: Done 105 tasks      | elapsed:  1.8min\n",
      "[Parallel(n_jobs=-1)]: Done 120 tasks      | elapsed:  2.1min\n",
      "[Parallel(n_jobs=-1)]: Done 137 tasks      | elapsed:  2.5min\n",
      "[Parallel(n_jobs=-1)]: Done 154 tasks      | elapsed:  2.9min\n",
      "[Parallel(n_jobs=-1)]: Done 173 tasks      | elapsed:  3.6min\n",
      "[Parallel(n_jobs=-1)]: Done 192 tasks      | elapsed:  4.2min\n",
      "[Parallel(n_jobs=-1)]: Done 213 tasks      | elapsed:  5.0min\n",
      "[Parallel(n_jobs=-1)]: Done 234 tasks      | elapsed:  5.8min\n",
      "[Parallel(n_jobs=-1)]: Done 257 tasks      | elapsed:  7.1min\n",
      "[Parallel(n_jobs=-1)]: Done 280 tasks      | elapsed:  8.4min\n",
      "[Parallel(n_jobs=-1)]: Done 305 tasks      | elapsed:  9.9min\n",
      "[Parallel(n_jobs=-1)]: Done 330 tasks      | elapsed: 10.9min\n",
      "[Parallel(n_jobs=-1)]: Done 357 tasks      | elapsed: 11.3min\n",
      "[Parallel(n_jobs=-1)]: Done 384 tasks      | elapsed: 11.7min\n",
      "[Parallel(n_jobs=-1)]: Done 413 tasks      | elapsed: 12.2min\n",
      "[Parallel(n_jobs=-1)]: Done 442 tasks      | elapsed: 12.9min\n",
      "[Parallel(n_jobs=-1)]: Done 473 tasks      | elapsed: 13.6min\n",
      "[Parallel(n_jobs=-1)]: Done 504 tasks      | elapsed: 14.6min\n",
      "[Parallel(n_jobs=-1)]: Done 537 tasks      | elapsed: 15.9min\n",
      "[Parallel(n_jobs=-1)]: Done 570 tasks      | elapsed: 17.3min\n",
      "[Parallel(n_jobs=-1)]: Done 605 tasks      | elapsed: 19.4min\n",
      "[Parallel(n_jobs=-1)]: Done 640 tasks      | elapsed: 21.4min\n",
      "[Parallel(n_jobs=-1)]: Done 677 tasks      | elapsed: 21.9min\n",
      "[Parallel(n_jobs=-1)]: Done 714 tasks      | elapsed: 22.5min\n",
      "[Parallel(n_jobs=-1)]: Done 753 tasks      | elapsed: 23.3min\n",
      "[Parallel(n_jobs=-1)]: Done 792 tasks      | elapsed: 24.2min\n",
      "[Parallel(n_jobs=-1)]: Done 833 tasks      | elapsed: 25.6min\n",
      "[Parallel(n_jobs=-1)]: Done 874 tasks      | elapsed: 27.1min\n",
      "[Parallel(n_jobs=-1)]: Done 917 tasks      | elapsed: 29.5min\n",
      "[Parallel(n_jobs=-1)]: Done 960 tasks      | elapsed: 31.9min\n",
      "[Parallel(n_jobs=-1)]: Done 1005 tasks      | elapsed: 32.6min\n",
      "[Parallel(n_jobs=-1)]: Done 1050 tasks      | elapsed: 33.4min\n",
      "[Parallel(n_jobs=-1)]: Done 1097 tasks      | elapsed: 34.5min\n",
      "[Parallel(n_jobs=-1)]: Done 1144 tasks      | elapsed: 35.9min\n",
      "[Parallel(n_jobs=-1)]: Done 1193 tasks      | elapsed: 37.8min\n",
      "[Parallel(n_jobs=-1)]: Done 1242 tasks      | elapsed: 40.5min\n",
      "[Parallel(n_jobs=-1)]: Done 1293 tasks      | elapsed: 42.9min\n",
      "[Parallel(n_jobs=-1)]: Done 1344 tasks      | elapsed: 43.7min\n",
      "[Parallel(n_jobs=-1)]: Done 1397 tasks      | elapsed: 44.7min\n",
      "[Parallel(n_jobs=-1)]: Done 1450 tasks      | elapsed: 46.1min\n",
      "[Parallel(n_jobs=-1)]: Done 1505 tasks      | elapsed: 48.2min\n",
      "[Parallel(n_jobs=-1)]: Done 1560 tasks      | elapsed: 51.1min\n",
      "[Parallel(n_jobs=-1)]: Done 1617 tasks      | elapsed: 53.7min\n",
      "[Parallel(n_jobs=-1)]: Done 1674 tasks      | elapsed: 54.6min\n",
      "[Parallel(n_jobs=-1)]: Done 1733 tasks      | elapsed: 55.9min\n",
      "[Parallel(n_jobs=-1)]: Done 1792 tasks      | elapsed: 57.6min\n",
      "[Parallel(n_jobs=-1)]: Done 1853 tasks      | elapsed: 60.2min\n",
      "[Parallel(n_jobs=-1)]: Done 1914 tasks      | elapsed: 63.8min\n",
      "[Parallel(n_jobs=-1)]: Done 1977 tasks      | elapsed: 64.9min\n",
      "[Parallel(n_jobs=-1)]: Done 2040 tasks      | elapsed: 66.2min\n",
      "[Parallel(n_jobs=-1)]: Done 2105 tasks      | elapsed: 68.2min\n",
      "[Parallel(n_jobs=-1)]: Done 2170 tasks      | elapsed: 70.9min\n",
      "[Parallel(n_jobs=-1)]: Done 2237 tasks      | elapsed: 74.9min\n",
      "[Parallel(n_jobs=-1)]: Done 2304 tasks      | elapsed: 76.0min\n",
      "[Parallel(n_jobs=-1)]: Done 2373 tasks      | elapsed: 77.5min\n",
      "[Parallel(n_jobs=-1)]: Done 2442 tasks      | elapsed: 79.6min\n",
      "[Parallel(n_jobs=-1)]: Done 2513 tasks      | elapsed: 83.0min\n",
      "[Parallel(n_jobs=-1)]: Done 2584 tasks      | elapsed: 86.1min\n",
      "[Parallel(n_jobs=-1)]: Done 2657 tasks      | elapsed: 87.3min\n",
      "[Parallel(n_jobs=-1)]: Done 2730 tasks      | elapsed: 89.2min\n",
      "[Parallel(n_jobs=-1)]: Done 2805 tasks      | elapsed: 92.1min\n",
      "[Parallel(n_jobs=-1)]: Done 2880 tasks      | elapsed: 96.4min\n",
      "[Parallel(n_jobs=-1)]: Done 2957 tasks      | elapsed: 98.4min\n",
      "[Parallel(n_jobs=-1)]: Done 3034 tasks      | elapsed: 101.7min\n",
      "[Parallel(n_jobs=-1)]: Done 3113 tasks      | elapsed: 107.2min\n",
      "[Parallel(n_jobs=-1)]: Done 3192 tasks      | elapsed: 116.9min\n",
      "[Parallel(n_jobs=-1)]: Done 3273 tasks      | elapsed: 119.7min\n",
      "[Parallel(n_jobs=-1)]: Done 3354 tasks      | elapsed: 123.0min\n",
      "[Parallel(n_jobs=-1)]: Done 3437 tasks      | elapsed: 128.8min\n",
      "[Parallel(n_jobs=-1)]: Done 3520 tasks      | elapsed: 138.2min\n",
      "[Parallel(n_jobs=-1)]: Done 3605 tasks      | elapsed: 140.6min\n",
      "[Parallel(n_jobs=-1)]: Done 3690 tasks      | elapsed: 144.5min\n",
      "[Parallel(n_jobs=-1)]: Done 3777 tasks      | elapsed: 151.6min\n",
      "[Parallel(n_jobs=-1)]: Done 3864 tasks      | elapsed: 159.2min\n",
      "[Parallel(n_jobs=-1)]: Done 3953 tasks      | elapsed: 162.1min\n",
      "[Parallel(n_jobs=-1)]: Done 4042 tasks      | elapsed: 167.2min\n",
      "[Parallel(n_jobs=-1)]: Done 4133 tasks      | elapsed: 176.2min\n",
      "[Parallel(n_jobs=-1)]: Done 4224 tasks      | elapsed: 180.9min\n",
      "[Parallel(n_jobs=-1)]: Done 4317 tasks      | elapsed: 184.7min\n",
      "[Parallel(n_jobs=-1)]: Done 4410 tasks      | elapsed: 192.2min\n",
      "[Parallel(n_jobs=-1)]: Done 4505 tasks      | elapsed: 201.7min\n",
      "[Parallel(n_jobs=-1)]: Done 4600 tasks      | elapsed: 205.1min\n",
      "[Parallel(n_jobs=-1)]: Done 4697 tasks      | elapsed: 211.4min\n",
      "[Parallel(n_jobs=-1)]: Done 4794 tasks      | elapsed: 222.8min\n",
      "[Parallel(n_jobs=-1)]: Done 4893 tasks      | elapsed: 226.4min\n",
      "[Parallel(n_jobs=-1)]: Done 4992 tasks      | elapsed: 232.0min\n",
      "[Parallel(n_jobs=-1)]: Done 5093 tasks      | elapsed: 242.0min\n",
      "[Parallel(n_jobs=-1)]: Done 5194 tasks      | elapsed: 246.9min\n",
      "[Parallel(n_jobs=-1)]: Done 5297 tasks      | elapsed: 251.8min\n",
      "[Parallel(n_jobs=-1)]: Done 5400 tasks      | elapsed: 260.9min\n",
      "[Parallel(n_jobs=-1)]: Done 5505 tasks      | elapsed: 267.2min\n",
      "[Parallel(n_jobs=-1)]: Done 5610 tasks      | elapsed: 271.7min\n",
      "[Parallel(n_jobs=-1)]: Done 5717 tasks      | elapsed: 281.0min\n",
      "[Parallel(n_jobs=-1)]: Done 5824 tasks      | elapsed: 286.8min\n",
      "[Parallel(n_jobs=-1)]: Done 5933 tasks      | elapsed: 289.5min\n",
      "[Parallel(n_jobs=-1)]: Done 6042 tasks      | elapsed: 294.6min\n",
      "[Parallel(n_jobs=-1)]: Done 6153 tasks      | elapsed: 298.0min\n",
      "[Parallel(n_jobs=-1)]: Done 6264 tasks      | elapsed: 300.9min\n",
      "[Parallel(n_jobs=-1)]: Done 6377 tasks      | elapsed: 306.5min\n",
      "[Parallel(n_jobs=-1)]: Done 6490 tasks      | elapsed: 309.1min\n",
      "[Parallel(n_jobs=-1)]: Done 6605 tasks      | elapsed: 312.4min\n",
      "[Parallel(n_jobs=-1)]: Done 6720 tasks      | elapsed: 318.4min\n",
      "[Parallel(n_jobs=-1)]: Done 6837 tasks      | elapsed: 320.5min\n",
      "[Parallel(n_jobs=-1)]: Done 6954 tasks      | elapsed: 324.4min\n",
      "[Parallel(n_jobs=-1)]: Done 7073 tasks      | elapsed: 329.8min\n",
      "[Parallel(n_jobs=-1)]: Done 7192 tasks      | elapsed: 332.3min\n",
      "[Parallel(n_jobs=-1)]: Done 7313 tasks      | elapsed: 337.4min\n",
      "[Parallel(n_jobs=-1)]: Done 7434 tasks      | elapsed: 341.3min\n",
      "[Parallel(n_jobs=-1)]: Done 7557 tasks      | elapsed: 344.6min\n",
      "[Parallel(n_jobs=-1)]: Done 7680 tasks      | elapsed: 350.9min\n",
      "[Parallel(n_jobs=-1)]: Done 7805 tasks      | elapsed: 353.2min\n",
      "[Parallel(n_jobs=-1)]: Done 7930 tasks      | elapsed: 357.8min\n",
      "[Parallel(n_jobs=-1)]: Done 8057 tasks      | elapsed: 362.8min\n",
      "[Parallel(n_jobs=-1)]: Done 8184 tasks      | elapsed: 365.9min\n",
      "[Parallel(n_jobs=-1)]: Done 8313 tasks      | elapsed: 372.4min\n",
      "[Parallel(n_jobs=-1)]: Done 8442 tasks      | elapsed: 374.9min\n",
      "[Parallel(n_jobs=-1)]: Done 8573 tasks      | elapsed: 379.6min\n",
      "[Parallel(n_jobs=-1)]: Done 8704 tasks      | elapsed: 385.2min\n",
      "[Parallel(n_jobs=-1)]: Done 8837 tasks      | elapsed: 391.7min\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Done 8970 tasks      | elapsed: 404.3min\n",
      "[Parallel(n_jobs=-1)]: Done 9105 tasks      | elapsed: 409.0min\n",
      "[Parallel(n_jobs=-1)]: Done 9240 tasks      | elapsed: 420.0min\n",
      "[Parallel(n_jobs=-1)]: Done 9377 tasks      | elapsed: 427.4min\n",
      "[Parallel(n_jobs=-1)]: Done 9514 tasks      | elapsed: 435.4min\n",
      "[Parallel(n_jobs=-1)]: Done 9653 tasks      | elapsed: 446.3min\n",
      "[Parallel(n_jobs=-1)]: Done 9792 tasks      | elapsed: 452.8min\n",
      "[Parallel(n_jobs=-1)]: Done 9933 tasks      | elapsed: 465.8min\n",
      "[Parallel(n_jobs=-1)]: Done 10074 tasks      | elapsed: 470.8min\n",
      "[Parallel(n_jobs=-1)]: Done 10217 tasks      | elapsed: 483.5min\n",
      "[Parallel(n_jobs=-1)]: Done 10360 tasks      | elapsed: 489.8min\n",
      "[Parallel(n_jobs=-1)]: Done 10505 tasks      | elapsed: 500.1min\n",
      "[Parallel(n_jobs=-1)]: Done 10650 tasks      | elapsed: 508.9min\n",
      "[Parallel(n_jobs=-1)]: Done 10797 tasks      | elapsed: 517.5min\n",
      "[Parallel(n_jobs=-1)]: Done 10944 tasks      | elapsed: 528.6min\n",
      "[Parallel(n_jobs=-1)]: Done 11093 tasks      | elapsed: 536.4min\n",
      "[Parallel(n_jobs=-1)]: Done 11242 tasks      | elapsed: 548.5min\n",
      "[Parallel(n_jobs=-1)]: Done 11393 tasks      | elapsed: 555.3min\n",
      "[Parallel(n_jobs=-1)]: Done 11520 out of 11520 | elapsed: 567.8min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done in 34067.734s\n",
      "\n",
      "Best score: 0.960\n",
      "Best parameters set:\n",
      "\tgow__b: 0.0\n",
      "\tgow__directed: False\n",
      "\tgow__max_df: 0.85\n",
      "\tgow__min_df: 5\n",
      "\tgow__term_weighting: 'degree'\n",
      "\tgow__window_size: 2\n",
      "\tsvm__C: 1\n",
      "\tsvm__class_weight: 'balanced'\n"
     ]
    }
   ],
   "source": [
    "grid_search = GridSearchCV(pipeline, \n",
    "                           parameters, \n",
    "                           cv=10,\n",
    "                           scoring=scorer_mcc,\n",
    "                           n_jobs=-1, \n",
    "                           verbose=10)\n",
    "\n",
    "print(\"Performing grid search...\")\n",
    "print(\"pipeline:\", [name for name, _ in pipeline.steps])\n",
    "print(\"parameters:\")\n",
    "pprint(parameters)\n",
    "t0 = time()\n",
    "grid_search.fit(X_train, y_train)\n",
    "print(\"done in %0.3fs\" % (time() - t0))\n",
    "print()\n",
    "\n",
    "print(\"Best score: %0.3f\" % grid_search.best_score_)\n",
    "print(\"Best parameters set:\")\n",
    "best_parameters = grid_search.best_estimator_.get_params()\n",
    "for param_name in sorted(parameters.keys()):\n",
    "    print(\"\\t%s: %r\" % (param_name, best_parameters[param_name]))"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "done in 34067.734s\n",
    "\n",
    "Best score: 0.960\n",
    "Best parameters set:\n",
    "\tgow__b: 0.0\n",
    "\tgow__directed: False\n",
    "\tgow__max_df: 0.85\n",
    "\tgow__min_df: 5\n",
    "\tgow__term_weighting: 'degree'\n",
    "\tgow__window_size: 2\n",
    "\tsvm__C: 1\n",
    "\tsvm__class_weight: 'balanced'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Fitting the final TW-IDF model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('gow',\n",
       "                 TwidfVectorizer(directed=False, max_df=0.85, min_df=5,\n",
       "                                 tokenizer=<function default_tokenizer at 0x7fd04029a950>,\n",
       "                                 window_size=2)),\n",
       "                ('svm', LinearSVC(C=1, class_weight='balanced'))])"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#\n",
    "# /!\\ manually set the best parameters\n",
    "#\n",
    "pipeline_gow = Pipeline([\n",
    "    ('gow', TwidfVectorizer(\n",
    "        b=0.0,\n",
    "        directed=False,\n",
    "        min_df=5,\n",
    "        max_df=0.85,\n",
    "        window_size=2,\n",
    "        term_weighting=TERM_WEIGHT_DEGREE\n",
    "    )),\n",
    "    ('svm', LinearSVC(\n",
    "        C=1,\n",
    "        class_weight='balanced',\n",
    "    )),\n",
    "])\n",
    "\n",
    "pipeline_gow.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Evaluation on the test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = pipeline_gow.predict(X_test)\n",
    "y_true = y_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         acq       0.98      0.98      0.98       696\n",
      "       crude       0.95      0.95      0.95       121\n",
      "        earn       0.99      0.99      0.99      1083\n",
      "       grain       1.00      1.00      1.00        10\n",
      "    interest       0.91      0.84      0.87        81\n",
      "    money-fx       0.88      0.85      0.87        87\n",
      "        ship       0.91      0.86      0.89        36\n",
      "       trade       0.90      0.99      0.94        75\n",
      "\n",
      "    accuracy                           0.97      2189\n",
      "   macro avg       0.94      0.93      0.94      2189\n",
      "weighted avg       0.97      0.97      0.97      2189\n",
      "\n",
      "mcc=0.9597248748927998 ; accuracy=0.9739607126541799 ; f1-micro=0.9739607126541799\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_true, y_pred))\n",
    "\n",
    "mcc = matthews_corrcoef(y_true, y_pred)\n",
    "accuracy = accuracy_score(y_true, y_pred)\n",
    "f1 = f1_score(y_true, y_pred, average='micro')\n",
    "print(f\"mcc={mcc} ; accuracy={accuracy} ; f1-micro={f1}\")"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "              precision    recall  f1-score   support\n",
    "\n",
    "         acq       0.98      0.98      0.98       696\n",
    "       crude       0.95      0.95      0.95       121\n",
    "        earn       0.99      0.99      0.99      1083\n",
    "       grain       1.00      1.00      1.00        10\n",
    "    interest       0.91      0.84      0.87        81\n",
    "    money-fx       0.88      0.85      0.87        87\n",
    "        ship       0.91      0.86      0.89        36\n",
    "       trade       0.90      0.99      0.94        75\n",
    "\n",
    "    accuracy                           0.97      2189\n",
    "   macro avg       0.94      0.93      0.94      2189\n",
    "weighted avg       0.97      0.97      0.97      2189\n",
    "\n",
    "mcc=0.9597248748927998 ; accuracy=0.9739607126541799 ; f1-micro=0.9739607126541799"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:gowpy-classification]",
   "language": "python",
   "name": "conda-env-gowpy-classification-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "370.6px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}