Assignment_2.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3GM9DBN-Qz3k"
      },
      "source": [
        "# Assignment 2\n",
        "\n",
        "**Due to**: 23/12/2021 (dd/mm/yyyy)\n",
        "\n",
        "**Credits**: Andrea Galassi, Federico Ruggeri, Paolo Torroni\n",
        "\n",
        "**Summary**: Fact checking, Neural Languange Inference (**NLI**)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tO_-4CZeRCO7"
      },
      "source": [
        "# Intro\n",
        "\n",
        "This assignment is centred on a particular and emerging NLP task, formally known as **fact checking** (or fake checking). As AI techniques become more and more powerful, reaching amazing results, such as image and text generation, it is more than ever necessary to build tools able to distinguish what is real from what is fake.\n",
        "\n",
        "Here we focus on a small portion of the whole fact checking problem, which aims to determine whether a given statement (fact) conveys a trustworthy information or not. \n",
        "\n",
        "More precisely, given a set of evidences and a fact to verify, we would like our model to correctly predict whether the fact is true or fake.\n",
        "\n",
        "In particular, we will see:\n",
        "\n",
        "*   Dataset preparation (analysis and pre-processing)\n",
        "*   Problem formulation: multi-input binary classification\n",
        "*   Defining an evaluation method\n",
        "*   Simple sentence embedding\n",
        "*   Neural building blocks\n",
        "*   Neural architecture extension"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hGDwg78PS_uy"
      },
      "source": [
        "# The FEVER dataset\n",
        "\n",
        "First of all, we need to choose a dataset. In this assignment we will rely on the [FEVER dataset](https://fever.ai).\n",
        "\n",
        "The dataset is about facts taken from Wikipedia documents that have to be verified. In particular, facts could face manual modifications in order to define fake information or to give different formulations of the same concept.\n",
        "\n",
        "The dataset consists of 185,445 claims manually verified against the introductory sections of Wikipedia pages and classified as ```Supported```, ```Refuted``` or ```NotEnoughInfo```. For the first two classes, systems and annotators need to also return the combination of sentences forming the necessary evidence supporting or refuting the claim."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4Oa5FpVpT7p4"
      },
      "source": [
        "## 2.1 Dataset structure\n",
        "\n",
        "Relevant data is divided into two file types. Information concerning the fact to verify, its verdict and associated supporting/opposing statements are stored in **.jsonl** format. In particular, each JSON element is a python dictionary with the following relevant fields:\n",
        "\n",
        "*    **ID**: ID associated to the fact to verify.\n",
        "\n",
        "*    **Verifiable**: whether the fact has been verified or not: ```VERIFIABLE``` or ```NOT VERIFIABLE```.\n",
        "    \n",
        "*    **Label**: the final verdict on the fact to verify: ```SUPPORTS```, ```REFUTES``` or ```NOT ENOUGH INFO```.\n",
        "    \n",
        "*    **Claim**: the fact to verify.\n",
        "    \n",
        "*    **Evidence**: a nested list of document IDs along with the sentence ID that is associated to the fact to verify. In particular, each list element is a tuple of four elements: the first two are internal annotator IDs that can be safely ignored; the third term is the document ID (called URL) and the last one is the sentence number (ID) in the pointed document to consider.\n",
        "\n",
        "**Some Examples**\n",
        "\n",
        "---\n",
        "\n",
        "**Verifiable**\n",
        "\n",
        "```\n",
        "{\"id\": 202314, \"verifiable\": \"VERIFIABLE\", \"label\": \"REFUTES\", \"claim\": \"The New Jersey Turnpike has zero shoulders.\", \"evidence\": [[[238335, 240393, \"New_Jersey_Turnpike\", 15]]]}\n",
        "```\n",
        "\n",
        "---\n",
        "\n",
        "**Not Verifiable**\n",
        "\n",
        "```\n",
        "{\"id\": 113501, \"verifiable\": \"NOT VERIFIABLE\", \"label\": \"NOT ENOUGH INFO\", \"claim\": \"Grease had bad reviews.\", \"evidence\": [[[133128, null, null, null]]]}\n",
        "```\n",
        "\n",
        "---"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nex_8UM4VWuY"
      },
      "source": [
        "## 2.2 Some simplifications and pre-processing\n",
        "\n",
        "We are only interested in verifiable facts. Thus, we can filter out all non-verifiable claims.\n",
        "\n",
        "Additionally, the current dataset format does not contain all necessary information for our classification purposes. In particular, we need to download Wikipedia documents and replace reported evidence IDs with the corresponding text.\n",
        "\n",
        "Don't worry about that! We are providing you the already pre-processed dataset so that you can concentrate on the classification pipeline (pre-processing, model definition, evaluation and training).\n",
        "\n",
        "You can download the zip file containing all set splits (train, validation and test) of the FEVER dataset by clicking on this [link](https://drive.google.com/file/d/1wArZhF9_SHW17WKNGeLmX-QTYw9Zscl1/view?usp=sharing). Alternatively, run the below code cell to automatically download it on this notebook.\n",
        "\n",
        "**Note**: each dataset split is in .csv format. Feel free to inspect the whole dataset!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "BspxZcRjW0NG",
        "outputId": "23dd2947-797f-4d0a-e307-7666d1e2e7ae"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Downloading FEVER data splits...\n",
            "Download completed!\n",
            "Extracting dataset...\n",
            "Extraction completed!\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "import requests\n",
        "import zipfile\n",
        "\n",
        "def save_response_content(response, destination):\n",
        "    CHUNK_SIZE = 32768\n",
        "\n",
        "    with open(destination, \"wb\") as f:\n",
        "        for chunk in response.iter_content(CHUNK_SIZE):\n",
        "            if chunk: # filter out keep-alive new chunks\n",
        "                f.write(chunk)\n",
        "\n",
        "def download_data(data_path):\n",
        "    toy_data_path = os.path.join(data_path, 'fever_data.zip')\n",
        "    toy_data_url_id = \"1wArZhF9_SHW17WKNGeLmX-QTYw9Zscl1\"\n",
        "    toy_url = \"https://docs.google.com/uc?export=download\"\n",
        "\n",
        "    if not os.path.exists(data_path):\n",
        "        os.makedirs(data_path)\n",
        "\n",
        "    if not os.path.exists(toy_data_path):\n",
        "        print(\"Downloading FEVER data splits...\")\n",
        "        with requests.Session() as current_session:\n",
        "            response = current_session.get(toy_url,\n",
        "                                   params={'id': toy_data_url_id},\n",
        "                                   stream=True)\n",
        "        save_response_content(response, toy_data_path)\n",
        "        print(\"Download completed!\")\n",
        "\n",
        "        print(\"Extracting dataset...\")\n",
        "        with zipfile.ZipFile(toy_data_path) as loaded_zip:\n",
        "            loaded_zip.extractall(data_path)\n",
        "        print(\"Extraction completed!\")\n",
        "\n",
        "download_data('dataset')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RbH_8errW5MH"
      },
      "source": [
        "# Classification dataset\n",
        "\n",
        "At this point, you should have a reay-to-go dataset! Note that the dataset format changed as well! In particular, we split the evidence set associated to each claim, in order to build `(claim, evidence)` pairs. The classification label is propagated as well.\n",
        "\n",
        "We'll motivate this decision in the next section!\n",
        "\n",
        "Just for clarity, here's an example of the pre-processed dataset:\n",
        "\n",
        "---\n",
        "\n",
        "**Claim**: \"Wentworth Miller is yet to make his screenwriting debut.\"\n",
        "\n",
        "**Evidence**: \"2\tHe made his screenwriting debut with the 2013 thriller film Stoker .\tStoker\tStoker (film)\"\n",
        "\n",
        "**Label**: Refutes\n",
        "\n",
        "---\n",
        "\n",
        "[**Note**]: The dataset requires some text cleaning as you may have noticed!\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gH8hIK21Xrl0"
      },
      "source": [
        "# Problem formulation\n",
        "\n",
        "As mentioned at the beginning of the assignment, we are going to formulate the fact checking problem as a binary classification task.\n",
        "\n",
        "In particular, each dataset sample is comprised of:\n",
        "\n",
        "*     A claim to verify\n",
        "*     A set of semantically related statements (evidence set)\n",
        "*     Fact checking label: either evidences support or refute the claim.\n",
        "\n",
        "Handling the evidence set from the point of view of neural models may imply some additional complexity: if the evidence set is comprised of several sentences we might incur in memory problems.\n",
        "\n",
        "To this end, we further simplify the problem by building (claim, evidence) pairs. The fact checking label is propagated as well.\n",
        "\n",
        "Example:\n",
        "\n",
        "     Claim: c1 \n",
        "     Evidence set: [e1, e2, e3]\n",
        "     Label: S (support)\n",
        "\n",
        "--->\n",
        "\n",
        "    (c1, e1, S),\n",
        "    (c1, e2, S),\n",
        "    (c1, e3, S)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "E46flIz_zQy-"
      },
      "source": [
        "## 4.1 Schema\n",
        "\n",
        "The overall binary classification problem is summed up by the following (simplified) schema\n",
        "\n",
        "![](https://drive.google.com/uc?export=view&id=1Wm_YBnFwgJtxcWEBpPbTBEVkpKaL08Jp)\n",
        "\n",
        "Don't worry too much about the **Encoding** block for now. We'll give you some simple guidelines about its definition. For the moment, stick to the binary classification task definition where, in this case, we have 2 inputs: the claim to verify and one of its associated evidences."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LsiTV-NVdgsF"
      },
      "source": [
        "# Architecture Guidelines\n",
        "\n",
        "There are many neural architectures that follow the above schema. To avoid phenomena like the writer's block, in this section we are going to give you some implementation guidelines.\n",
        "\n",
        "In particular, we would like you to test some implementations so that you explore basic approaches (neural baselines) and use them as building blocks for possible extensions."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZJBQm47fe7iE"
      },
      "source": [
        "## 5.1 Handling multiple inputs\n",
        "\n",
        "The first thing to notice is that we are in a multi-input scenario. In particular, each sample is comprised of a fact and its asssociated evidence statement.\n",
        "\n",
        "Each of these input is encoded as a sequence of tokens. In particular, we will have the following input matrices:\n",
        "\n",
        "*    Claim: `[batch_size, max_tokens]`\n",
        "*    Evidence: `[batch_size, max_tokens]`\n",
        "\n",
        "Moreover, after the embedding layer, we'll have:\n",
        "\n",
        "*    Claim: `[batch_size, max_tokens, embedding_dim]`\n",
        "*    Evidence: `[batch_size, max_tokens, embedding_dim]`\n",
        "\n",
        "But, we would like to have a 2D input to our classifier, since we have to give an answer at pair level. Therefore, for each sample, we would expect the following input shape to our classification block:\n",
        "\n",
        "*   Classification input shape: `[batch_size, dim]`\n",
        "\n",
        "**How to do that?**\n",
        "\n",
        "We inherently need to reduce the token sequence to a single representation. This operation is formally known as **sentence embedding**. Indeed, we are trying to compress the information of a whole sequence into a single embedding vector.\n",
        "\n",
        "Here are some simple solutions that we ask you to try out:\n",
        "\n",
        "1.   Encode token sequences via a RNN and take the last state as the sentence embedding.\n",
        "\n",
        "2.  Encode token sequences via a RNN and average all the output states.\n",
        "\n",
        "3.  Encode token sequences via a simple MLP layer. In particular, if your input is a `[batch_size, max_tokens, embedding_dim]` tensor, the matrix multiplication works on the **max_tokens** dimension, resulting in a `[batch_size, embedding_dim]` 2D matrix. Alternatively, you can reshape the 3D input tensor from `[batch_size, max_tokens, embedding_dim]` to `[batch_size, max_tokens * embedding_dim]` and then apply the MLP layer.\n",
        "\n",
        "4.   Compute the sentence embedding as the mean of its token embeddings (**bag of vectors**)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Gypl5z5ElJo1"
      },
      "source": [
        "## 5.2 Merging multi-inputs\n",
        "\n",
        "At this point, we have to think about **how** we should merge evidence and claim sentence embeddings.\n",
        "\n",
        "For simplicity, we stick to simple merging strategies:\n",
        "\n",
        "*     **Concatenation**: define the classification input as the concatenation of evidence and claim sentence embeddings\n",
        "\n",
        "*     **Sum**: define the classification input as the sum of evidence and claim sentence embeddings\n",
        "\n",
        "*     **Mean**: define the classification input as the mean of evidence and claim sentence embeddings\n",
        "\n",
        "For clarity, if the sentence embedding of a single input has shape `[batch_size, embedding_dim]`, then the classification input has shape:\n",
        "\n",
        "*     **Concatenation**: `[batch_size, 2 * embedding_dim]`\n",
        "\n",
        "*     **Sum**: `[batch_size, embedding_dim]`\n",
        "\n",
        "*     **Mean**: `[batch_size, embedding_dim]`"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IhVg9ZLymOUc"
      },
      "source": [
        "# A simple extension\n",
        "\n",
        "Lastly, we ask you to modify previously defined neural architectures by adding an additional feature to the classification input.\n",
        "\n",
        "We would like to see if some similarity information between the claim to verify and one of its associated evidence might be useful to the classification.\n",
        "\n",
        "Compute the cosine similarity metric between the two sentence embeddings and concatenate the result to the classification input.\n",
        "\n",
        "For clarity, since the cosine similarity of two vectors outputs a scalar value, the classification input shape is modified as follows:\n",
        "\n",
        "*     **Concatenation**: `[batch_size, 2 * embedding_dim + 1]`\n",
        "\n",
        "*     **Sum**: `[batch_size, embedding_dim + 1]`\n",
        "\n",
        "*     **Mean**: `[batch_size, embedding_dim + 1]`\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Nd74ULgpnJrc"
      },
      "source": [
        "# Performance evaluation\n",
        "\n",
        "Due to our simplifications, obtained results are not directly compatible with a traditional fact checking method that considers the evidence set as a whole.\n",
        "\n",
        "Thus, we need to consider two types of evaluations.\n",
        "\n",
        "---\n",
        "\n",
        "A. **Multi-input classification evaluation**\n",
        "\n",
        "This type of evaluation is the easiest and concerns computing evaluation metrics, such as accuracy, f1-score, recall and precision, of our pre-processed dataset.\n",
        "\n",
        "In other words, we assess the performance of chosen classifiers.\n",
        "\n",
        "---\n",
        "\n",
        "B. **Claim verification evaluation**\n",
        "\n",
        "However, if we want to give an answer concerning the claim itself, we need to consider the whole evidence set. \n",
        "\n",
        "Intuitively, for a given claim, we consider all its corresponding (claim, evidence) pairs and their corresponding classification outputs. \n",
        "\n",
        "At this point, all we need to do is to compute the final predicted claim label via majority voting.\n",
        "\n",
        "---\n",
        "\n",
        "Example:\n",
        "\n",
        "    Claim: c1\n",
        "    Evidence set: e1, e2, e3\n",
        "    True label: S\n",
        "\n",
        "    Pair outputs:\n",
        "    (c1, e1) -> S (supports)\n",
        "    (c1, e2) -> S (supports)\n",
        "    (c1, e3) -> R (refutes)\n",
        "\n",
        "    Majority voting:\n",
        "    S -> 2 votes\n",
        "    R -> 1 vote\n",
        "\n",
        "    Final label:\n",
        "    c1 -> S\n",
        "\n",
        "Lastly, we have to compute classification metrics just like before.\n",
        "\n",
        "Shortly speaking, implement both strategies for your classification metrics."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W4LJ2yPxsUOV"
      },
      "source": [
        "# Tips and Extras"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Uf80UVRNrXve"
      },
      "source": [
        "## 8.1 Extensions are welcome!\n",
        "\n",
        "Is this task too easy for you? Are you curious to try out things you have seen during lectures (e.g. attention)? Feel free to try everything you want!\n",
        "\n",
        "**Don't forget to try neural baselines first!**"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "COXeCXdYsBEf"
      },
      "source": [
        "## 8.2 Comments and documentation\n",
        "\n",
        "Remember to properly comment your code (it is not necessary to comment each single line) and don't forget to describe your work!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ejv6SDE8xc4_"
      },
      "source": [
        "## 8.3 Organization\n",
        "\n",
        "We suggest you to divide your work into sections. This allows you to build clean and modular code, as well as easy to read and to debug.\n",
        "\n",
        "A possible schema:\n",
        "\n",
        "*   Dataset pre-processing\n",
        "*   Dataset conversion\n",
        "*   Model definition\n",
        "*   Training\n",
        "*   Evaluation\n",
        "*   Comments/Summary"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "19QWjgGzIKOq"
      },
      "source": [
        "# Evaluation\n",
        "\n",
        "Which are the evaluation criteria on which we'll judge you and your work?\n",
        "\n",
        "1. Pre-processing: whether you have done some pre-processing or not.\n",
        "2. Sentence embedding: you should implement all required strategies (with an example and working code for each). That is, we, as evaluators, should be able to test all strategies without writing down new code.\n",
        "3. Multiple inputs merging strategies: you should implement all required strategies (with an example and working code for each).\n",
        "4. Similarity extension: you should implement the cosine similarity extension (with an example and working code).\n",
        "5. Voting strategy: you should implement the majority voting strategy and provide results.\n",
        "6. Report: when submitting your notebook, you should also attach a small summary report that describes what you have done (provide motivations as well for abitrary steps. For instance, \"We've applied L2 regularization since the model was overfitting\".\n",
        "\n",
        "Extras (possible extra points):\n",
        "\n",
        "1. Any well defined extension is welcome!\n",
        "2. Well organized and commented code is as important as any other criteria."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9DR70uh7pabo"
      },
      "source": [
        "# Contact\n",
        "\n",
        "For any doubt, question, issue or help, you can always contact us at the following email addresses:\n",
        "\n",
        "Teaching Assistants:\n",
        "\n",
        "* Andrea Galassi -> a.galassi@unibo.it\n",
        "* Federico Ruggeri -> federico.ruggeri6@unibo.it\n",
        "\n",
        "Professor:\n",
        "\n",
        "* Paolo Torroni -> p.torroni@unibo.it\n",
        "\n",
        "*Note*: We highly recommend you to check the [course useful material](https://virtuale.unibo.it/pluginfile.php/1036039/mod_resource/content/2/NLP_Course_Useful_Material.pdf) for additional information before contacting us!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Jc0gNWU2pgKQ"
      },
      "source": [
        "# FAQ\n",
        "\n",
        "---\n",
        "\n",
        "**Question**: Can I do something text pre-processing?\n",
        "\n",
        "**Answer:** You have to! If you check text data, the majority of sentences need some cleaning.\n",
        "\n",
        "---\n",
        "\n",
        "**Question**: The model architecture schema is not so clear, are we doing end-to-end training?\n",
        "\n",
        "**Answer**: Exactly! All models can be thought as:\n",
        "\n",
        "1. Input\n",
        "2. (word) Embedding\n",
        "3. Sentence embedding\n",
        "4. Multiple inputs merging\n",
        "5. Classification\n",
        "\n",
        "---\n",
        "\n",
        "**Question**: Can I extend models by adding more layers?\n",
        "\n",
        "**Answer**: Feel free to define model architectures as you wish, but remember satisfy our requirements. This assignment should not be thought as a competition to achieve the best performing model: fancy students that want to show off but miss required assignment objectives will be punished!!\n",
        "\n",
        "---\n",
        "\n",
        "**Question**: I'm struggling with the implementation. Can you help me?\n",
        "\n",
        "**Answer**: Yes sure! Contact us and describe your issue. If you are looking for a particular type of operation, you can easily check the documentation of the deep learning framework you are using (google is your friend).\n",
        "\n",
        "---\n",
        "\n",
        "**Question**: Can I try other encoding strategies or neural architectures?\n",
        "\n",
        "**Answer:** Absolutely! Remember to try out recommended neural baselines first and only then proceed with your extensions.\n",
        "\n",
        "---\n",
        "\n",
        "**Question**: Do we have to test all possible sentence embedding and input merging combinations?\n",
        "\n",
        "**Answer**: Absolutely no! Feel free to pick one sentence embedding strategy and try all possible input merging strategies with it! For instance, pick the best performing sentence embedding method and proceed with next steps (extras included). Please, note that you still have to implement all mentioned strategies!\n",
        "\n",
        "---\n",
        "\n",
        "**Question**: I'm hitting out of memory error when training my models, do you have any suggestions?\n",
        "\n",
        "**Answer**: Here are some common workarounds:\n",
        "\n",
        "1. Try decreasing the mini-batch size\n",
        "2. Try applying a different padding strategy (if you are applying padding): e.g. use quantiles instead of maximum sequence length\n",
        "3. Check the efficiency of your custom code implementation (if any)\n",
        "4. Try to define same length mini-batches to avoid padding (**It should not be necessary here!**)\n",
        "\n",
        "---\n",
        "\n",
        "**Question**: I'm hitting CUDNN_STATUS_BAD_PARAM error! What I'm doing wrong?\n",
        "\n",
        "**Answer**: This error is a little bit tricky since the stack trace is not meaningful at all! This error occurs when the RNN is fed with a sequence of all 0s and pad masking is enabled (e.g. from the embedding layer). Please, check your conversion step, since there might be an error that leads to the encoding of a sentence to all 0s.\n",
        "\n",
        "---"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "name": "Assignment_2.ipynb",
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python",
      "version": "3.9.7"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}