GPT-3.5_InContextLearning_Validation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.0 Introduction\n",
    "\n",
    "In this notebook we will explore few-shot in-context learning where we will use randomly selected samples from the training dataset as examples in the prompt to classify a text from the validation dataset.\n",
    "\n",
    "The training and validation csv files are the same as used in all other notebooks in this repository.\n",
    "\n",
    "The GPT-3.5 Turbo model with 16K context window will be used as it is more than large enough to contain the generated prompt with all added training examples.\n",
    "\n",
    "!Note: The code in this notebook is based on the new OpenAI API version (1.X or higher)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import Modules\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import time\n",
    "from sklearn.metrics import classification_report\n",
    "from tqdm.notebook import tqdm\n",
    "\n",
    "# OpenAI\n",
    "from openai import OpenAI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0 Load Datasets\n",
    "\n",
    "We will load the training and validation CSV files that were generated earlier with the notebook 'Prepare_Train_and_Validation_Datasets.ipynb'.\n",
    "\n",
    "The validation dataset will be used in the same way as in all other notebook. The training dataset however will be used to continuously get random samples to use as examples in the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(3069, 11)\n",
      "(1559, 11)\n"
     ]
    }
   ],
   "source": [
    "# Load Datasets\n",
    "train_df = pd.read_csv('./data/train_df.csv')\n",
    "val_df = pd.read_csv('./data/val_df.csv')\n",
    "\n",
    "# Summary\n",
    "print(train_df.shape)\n",
    "print(val_df.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's review a small subset of the training data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>mainSection</th>\n",
       "      <th>published_at</th>\n",
       "      <th>publisher</th>\n",
       "      <th>partisan</th>\n",
       "      <th>url</th>\n",
       "      <th>text_wordcount</th>\n",
       "      <th>max_words_text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>10706318</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorlog</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorl...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2018-10-07</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/ogen-als-schoteltjes-bij-de-...</td>\n",
       "      <td>539</td>\n",
       "      <td>Ogen als schoteltjes bij de Tachtigjarige Oorl...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>12633805</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>/amsterdam</td>\n",
       "      <td>2019-05-10</td>\n",
       "      <td>parool</td>\n",
       "      <td>True</td>\n",
       "      <td>www.parool.nl/amsterdam/geen-beeld-maar-een-mo...</td>\n",
       "      <td>662</td>\n",
       "      <td>Geen beeld, maar een monument voor Mandela in ...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>7140125</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>/</td>\n",
       "      <td>2017-04-18</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>494</td>\n",
       "      <td>Hoe ga je een onveilige arbeidscultuur zoals i...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4490774</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokikker</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokik...</td>\n",
       "      <td>/</td>\n",
       "      <td>2017-03-14</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>NaN</td>\n",
       "      <td>291</td>\n",
       "      <td>Wetenschappers ontdekken lichtgevende discokik...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10592180</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2018-09-11</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/meer-fouten-kabinet-bij-steu...</td>\n",
       "      <td>471</td>\n",
       "      <td>Meer fouten kabinet bij steun aan strijdgroepe...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "0  10706318   Ogen als schoteltjes bij de Tachtigjarige Oorlog   \n",
       "1  12633805  Geen beeld, maar een monument voor Mandela in ...   \n",
       "2   7140125  Hoe ga je een onveilige arbeidscultuur zoals i...   \n",
       "3   4490774  Wetenschappers ontdekken lichtgevende discokikker   \n",
       "4  10592180  Meer fouten kabinet bij steun aan strijdgroepe...   \n",
       "\n",
       "                                                text mainSection published_at  \\\n",
       "0  Ogen als schoteltjes bij de Tachtigjarige Oorl...       /home   2018-10-07   \n",
       "1  Geen beeld, maar een monument voor Mandela in ...  /amsterdam   2019-05-10   \n",
       "2  Hoe ga je een onveilige arbeidscultuur zoals i...           /   2017-04-18   \n",
       "3  Wetenschappers ontdekken lichtgevende discokik...           /   2017-03-14   \n",
       "4  Meer fouten kabinet bij steun aan strijdgroepe...       /home   2018-09-11   \n",
       "\n",
       "  publisher  partisan                                                url  \\\n",
       "0     trouw      True  www.trouw.nl/home/ogen-als-schoteltjes-bij-de-...   \n",
       "1    parool      True  www.parool.nl/amsterdam/geen-beeld-maar-een-mo...   \n",
       "2     trouw      True                                                NaN   \n",
       "3     trouw      True                                                NaN   \n",
       "4     trouw      True  www.trouw.nl/home/meer-fouten-kabinet-bij-steu...   \n",
       "\n",
       "   text_wordcount                                     max_words_text  labels  \n",
       "0             539  Ogen als schoteltjes bij de Tachtigjarige Oorl...       1  \n",
       "1             662  Geen beeld, maar een monument voor Mandela in ...       1  \n",
       "2             494  Hoe ga je een onveilige arbeidscultuur zoals i...       1  \n",
       "3             291  Wetenschappers ontdekken lichtgevende discokik...       1  \n",
       "4             471  Meer fouten kabinet bij steun aan strijdgroepe...       1  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and also the validation data..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>title</th>\n",
       "      <th>text</th>\n",
       "      <th>mainSection</th>\n",
       "      <th>published_at</th>\n",
       "      <th>publisher</th>\n",
       "      <th>partisan</th>\n",
       "      <th>url</th>\n",
       "      <th>text_wordcount</th>\n",
       "      <th>max_words_text</th>\n",
       "      <th>labels</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9266995</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-12-18</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/verdachte-dodelijke-steek...</td>\n",
       "      <td>188</td>\n",
       "      <td>Verdachte dodelijke steekpartijen Maastricht l...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>4130077</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2017-02-11</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/honderden-arrestaties-bij...</td>\n",
       "      <td>122</td>\n",
       "      <td>Honderden arrestaties bij acties tegen mensen ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11147268</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>/home</td>\n",
       "      <td>2019-01-20</td>\n",
       "      <td>trouw</td>\n",
       "      <td>True</td>\n",
       "      <td>www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...</td>\n",
       "      <td>262</td>\n",
       "      <td>Waarom de 'oudejaarsbonus' voor de jongeren va...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>10749100</td>\n",
       "      <td>Klaar voor de verdediging</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-16</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/binnenland/klaar-voor-de-verdediging...</td>\n",
       "      <td>411</td>\n",
       "      <td>Klaar voor de verdedigingOver ruim een week be...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>10700707</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>/nieuws</td>\n",
       "      <td>2018-10-05</td>\n",
       "      <td>ad</td>\n",
       "      <td>False</td>\n",
       "      <td>www.ad.nl/buitenland/windvlaag-grijpt-springma...</td>\n",
       "      <td>286</td>\n",
       "      <td>Windvlaag grijpt springmatras en doodt 2-jarig...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         id                                              title  \\\n",
       "0   9266995  Verdachte dodelijke steekpartijen Maastricht l...   \n",
       "1   4130077  Honderden arrestaties bij acties tegen mensen ...   \n",
       "2  11147268  Waarom de 'oudejaarsbonus' voor de jongeren va...   \n",
       "3  10749100                          Klaar voor de verdediging   \n",
       "4  10700707  Windvlaag grijpt springmatras en doodt 2-jarig...   \n",
       "\n",
       "                                                text mainSection published_at  \\\n",
       "0  Verdachte dodelijke steekpartijen Maastricht l...     /nieuws   2017-12-18   \n",
       "1  Honderden arrestaties bij acties tegen mensen ...     /nieuws   2017-02-11   \n",
       "2  Waarom de 'oudejaarsbonus' voor de jongeren va...       /home   2019-01-20   \n",
       "3  Klaar voor de verdedigingOver ruim een week be...     /nieuws   2018-10-16   \n",
       "4  Windvlaag grijpt springmatras en doodt 2-jarig...     /nieuws   2018-10-05   \n",
       "\n",
       "  publisher  partisan                                                url  \\\n",
       "0        ad     False  www.ad.nl/binnenland/verdachte-dodelijke-steek...   \n",
       "1        ad     False  www.ad.nl/buitenland/honderden-arrestaties-bij...   \n",
       "2     trouw      True  www.trouw.nl/home/waarom-de-oudejaarsbonus-voo...   \n",
       "3        ad     False  www.ad.nl/binnenland/klaar-voor-de-verdediging...   \n",
       "4        ad     False  www.ad.nl/buitenland/windvlaag-grijpt-springma...   \n",
       "\n",
       "   text_wordcount                                     max_words_text  labels  \n",
       "0             188  Verdachte dodelijke steekpartijen Maastricht l...       0  \n",
       "1             122  Honderden arrestaties bij acties tegen mensen ...       0  \n",
       "2             262  Waarom de 'oudejaarsbonus' voor de jongeren va...       1  \n",
       "3             411  Klaar voor de verdedigingOver ruim een week be...       0  \n",
       "4             286  Windvlaag grijpt springmatras en doodt 2-jarig...       0  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Summary\n",
    "val_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the InContext Learning we will further subsample the validation set to 100 / 100 samples of Politiek and Neutraal.\n",
    "\n",
    "I did a lot of small experiments with various different prompts and in-context added samples. The achieved accuracy varied between 40% to around 60% on the validation set for all experiments. \n",
    "The dataset is clearly not the best one suitable for few-shot classification.\n",
    "\n",
    "Since this is one of my many side projects I won't create a nice credit card bill for validating on all validation samples ;-)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "labels\n",
       "1    100\n",
       "0    100\n",
       "Name: count, dtype: int64"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Subsample Validation Dataset\n",
    "label0_val_df = val_df[val_df.labels == 0].sample(n = 100)\n",
    "label1_val_df = val_df[val_df.labels == 1].sample(n = 100)\n",
    "\n",
    "# Concatenate\n",
    "small_val_df = pd.concat([label0_val_df, label1_val_df]).sample(frac = 1.0)\n",
    "\n",
    "# Summary\n",
    "small_val_df.labels.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0 In-context Learning with OpenAI GPT-3.5 Model\n",
    "\n",
    "In this section we will determine the classification accuracy on the validation dataset when using few shot in-context learning.\n",
    "\n",
    "In each request to determine the classification label on a sample text from the validation dataset we will provide 3 rxamples for each of the 2 labels.\n",
    "\n",
    "The examples for the 2 labels will be sampled at random from the training dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Settings and Support Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Constants\n",
    "MAX_WORDS = 192\n",
    "N_SAMPLES = 3\n",
    "\n",
    "# OpenAI API Key\n",
    "client = OpenAI(api_key = os.environ[\"OPENAI_API_KEY\"])\n",
    "\n",
    "# Set Model\n",
    "openai_model = \"gpt-3.5-turbo-1106\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following function will randomly sample from the training dataset the examples that will be put in the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_incontext_samples(train_df):\n",
    "    politic_samples = train_df[train_df.labels == 1].sample(n = N_SAMPLES)['max_words_text'].tolist()\n",
    "    neutral_samples = train_df[train_df.labels == 0].sample(n = N_SAMPLES)['max_words_text'].tolist()\n",
    "\n",
    "    return politic_samples, neutral_samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 In-context learning prompt\n",
    "\n",
    "Next is the function to create the prompt text. It contains the necessary prompt engineering to specify the examples that are provided for the 2 labels 'Politiek' and 'Neutraal'.\n",
    "\n",
    "In the final part the text to be classified is added and again we ask the model to only respond with one of the two allowed labels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_prompt(item_text, politic_samples, neutral_samples):\n",
    "    # Base Prompt\n",
    "    prompt_text = [\n",
    "    {\"role\": \"system\", \n",
    "     \"content\": \n",
    "\"\"\"\n",
    "Je bent redacteur bij een krant.\n",
    "Je beoordeeld een krantenartikel of het politiek of neutraal is. \n",
    "Je mag de beoordeling alleen aangeven met 1 van de 2 volgende labels:\n",
    "Politiek\n",
    "Neutraal\n",
    "\n",
    "Hieronder staan een aantal voorbeelden van Politieke en Neutrale teksten aangegeven met '# Voorbeeld: ' en vervolgens de tekst.\n",
    "Eronder staat met 'Label: ' aangegeven of het Politiek of Neutraal is.\n",
    "\n",
    "\"\"\"\n",
    "     }, \n",
    "    {\"role\": \"user\", \"content\": \"\"}]\n",
    "\n",
    "    # Set Text and Label\n",
    "    prompt_text[1]['content'] = f\"\"\"\n",
    "\\n\\n# Voorbeeld: {politic_samples[0]}\n",
    "\\nLabel: Politiek\n",
    "\\n\\n# Voorbeeld: {neutral_samples[0]}\n",
    "\\nLabel: Neutraal \n",
    "\\n\\n# Voorbeeld: {politic_samples[1]}\n",
    "\\nLabel: Politiek\n",
    "\\n\\n# Voorbeeld: {neutral_samples[1]}\n",
    "\\nLabel: Neutraal\n",
    "\\n\\n# Voorbeeld: {politic_samples[2]}\n",
    "\\nLabel: Politiek\n",
    "\\n\\n# Voorbeeld: {neutral_samples[2]}\n",
    "\\nLabel: Neutraal\n",
    "\n",
    "Beoordeel de nu volgende tekst. \n",
    "Antwoord met alleen de labels Politiek of Neutraal.\n",
    "\\n{item_text}\n",
    "Label: \n",
    "\"\"\"\n",
    "    \n",
    "    return prompt_text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3 Validation\n",
    "\n",
    "In this part we loop through the validation samples and have the default GPT-3.5 model determine the classification label based on the provided examples in the prompt."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f98ba4ae7ac247f598aba843e2e79870",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "  0%|          | 0/200 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Placeholders\n",
    "gt_labels, pred_labels = [], []\n",
    "\n",
    "# Inference on Validation Dataframe\n",
    "for index, row in tqdm(small_val_df.iterrows(), total = small_val_df.shape[0]):\n",
    "    text = row['max_words_text']\n",
    "    label = row['labels']\n",
    "    gt_labels.append(label)\n",
    "\n",
    "    # Get In Context Samples\n",
    "    politics_samples, neutral_samples = get_incontext_samples(train_df)\n",
    "    \n",
    "    # Create Prompt\n",
    "    prompt = create_prompt(text, politics_samples, neutral_samples)\n",
    "\n",
    "    # Call API for inference on FineTuned Model\n",
    "    # Simple error handling .... earlier this summer I experienced occasional errors with the model not responding/not being available.. A simple wait period was enough to have it working.\n",
    "    # Recently no bad experiences with that...however I keep the error handling for it in-place\n",
    "    try:\n",
    "        completion = client.chat.completions.create(model = openai_model,\n",
    "                                                    messages = prompt, \n",
    "                                                    temperature = 0.0)\n",
    "    except Exception as err:\n",
    "        print(err)\n",
    "        time.sleep(120.0)\n",
    "        completion = client.chat.completions.create(model = openai_model,\n",
    "                                                    messages = prompt,\n",
    "                                                    temperature = 0.0)\n",
    "\n",
    "    # Response Determine label based on OpenAI ChatCompletion Response\n",
    "    predicted_label = -1\n",
    "    prediction_text = completion.choices[0].message.content\n",
    "    if prediction_text == 'Politiek':\n",
    "        predicted_label = 1\n",
    "    if prediction_text == 'Neutraal':\n",
    "        predicted_label = 0\n",
    "\n",
    "    # Set the predicted label wrong in case Politiek or Neutraal was not set.\n",
    "    if predicted_label == -1:\n",
    "        predicted_label = 1 - label\n",
    "\n",
    "    # Store Result\n",
    "    pred_labels.append(predicted_label)\n",
    "    #print(f'Index: {index}   GT: {label}   Pred: {predicted_label}')\n",
    "\n",
    "    # Delay...To stay within the limits for tokens/requests per minute\n",
    "    time.sleep(0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4 Classification Results\n",
    "\n",
    "Below the classification results for the in-context learning approach.\n",
    "\n",
    "With an achieved accuracy of 56% on the validation dataset it is clear that in-context learning is performing very bad for this particular dataset.\n",
    "\n",
    "The validation accuracy is significantly lower than the 85% - 90% validation accuracy we could achieve when fine-tuning an OpenAI GPT-3.5 (or any other..) model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "    Neutraal      0.583     0.420     0.488       100\n",
      "    Politiek      0.547     0.700     0.614       100\n",
      "\n",
      "    accuracy                          0.560       200\n",
      "   macro avg      0.565     0.560     0.551       200\n",
      "weighted avg      0.565     0.560     0.551       200\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Classification Results\n",
    "print(classification_report(gt_labels, pred_labels, \n",
    "                            target_names = ['Neutraal', 'Politiek'], \n",
    "                            digits = 3))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}