Extraction_question_answering_T5.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "613868c9-ff2d-4f99-a035-3f4c60529852",
   "metadata": {},
   "source": [
    "Let's set up our environment to run the Hugging Face version of T5 and feed it a small snippet of text to see what kind of summary it produces.  Note that we could not feed the entire Wikipedia article we used above into T5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "2c299ee2-d52f-4f1c-91ac-8d00062c3caa",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-10-10 19:52:14.153034: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from transformers import T5Tokenizer, TFT5Model, TFT5ForConditionalGeneration"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d719e60f-5415-4e03-a215-26fa62484cfc",
   "metadata": {},
   "source": [
    "Here is the text that I will demonstrate\n",
    "\n",
    "Now let's look at an extractive question answering example.  We'll need to feed the model a context paragraph and a question.  The T5 model was pre-trained on the SQUAD dataset so it knows how to identify and extract the answer span. Note that we already have the prompt in the respective texts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c1761872-d615-4677-aa54-65a5c101d19d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All PyTorch model weights were used when initializing TFT5ForConditionalGeneration.\n",
      "\n",
      "All the weights of TFT5ForConditionalGeneration were initialized from the PyTorch model.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFT5ForConditionalGeneration for predictions without further training.\n",
      "/Users/peeti_mac/opt/anaconda3/lib/python3.9/site-packages/transformers/models/t5/tokenization_t5.py:220: FutureWarning: This tokenizer was incorrectly instantiated with a model max length of 512 which will be corrected in Transformers v5.\n",
      "For now, this behavior is kept to avoid breaking backwards compatibility when padding/encoding with `truncation is True`.\n",
      "- Be aware that you SHOULD NOT rely on t5-base automatically truncating your input to 512 when padding/encoding.\n",
      "- If you want to encode/pad to sequences longer than 512 you can either instantiate this tokenizer with `model_max_length` or pass `max_length` when encoding/padding.\n",
      "- To avoid this warning, please instantiate this tokenizer with `model_max_length` set to your preferred value.\n",
      "  warnings.warn(\n",
      "You are using the default legacy behaviour of the <class 'transformers.models.t5.tokenization_t5.T5Tokenizer'>. If you see this, DO NOT PANIC! This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thouroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"tft5_for_conditional_generation\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " shared (Embedding)          multiple                  24674304  \n",
      "                                                                 \n",
      " encoder (TFT5MainLayer)     multiple                  109628544 \n",
      "                                                                 \n",
      " decoder (TFT5MainLayer)     multiple                  137949312 \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 222903552 (850.31 MB)\n",
      "Trainable params: 222903552 (850.31 MB)\n",
      "Non-trainable params: 0 (0.00 Byte)\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "t5_model = TFT5ForConditionalGeneration.from_pretrained('t5-base') #also t5-small and t5-large\n",
    "t5_tokenizer = T5Tokenizer.from_pretrained('t5-base')\n",
    "\n",
    "t5_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "bd3ac9a3-6117-4381-8894-f1166e79a64d",
   "metadata": {},
   "outputs": [],
   "source": [
    "t5_context_text = \"\"\"context: Hyperbaric (high-pressure) medicine uses special oxygen \n",
    "chambers to increase the partial pressure of O 2 around the patient and, when needed, \n",
    "the medical staff. Carbon monoxide poisoning, gas gangrene, and decompression sickness \n",
    "(the ’bends’) are sometimes treated using these devices. Increased O 2 concentration \n",
    "in the lungs helps to displace carbon monoxide from the heme group of hemoglobin. \n",
    "Oxygen gas is poisonous to the anaerobic bacteria that cause gas gangrene, so increasing \n",
    "its partial pressure helps kill them. Decompression sickness occurs in divers who \n",
    "decompress too quickly after a dive, resulting in bubbles of inert gas, mostly nitrogen \n",
    "and helium, forming in their blood. Increasing the pressure of O 2 as soon as possible \n",
    "is part of the treatment.\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "006ed76a-2b58-4b76-adf8-7b64a26c313e",
   "metadata": {},
   "outputs": [],
   "source": [
    "t5_question_text = \"\"\"question: What does increased oxygen concentrations in the patient’s\n",
    "lungs displace? \"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6cc85eb3-84c5-49f5-9e35-3083fac74930",
   "metadata": {},
   "outputs": [],
   "source": [
    "t5_qa_input_text = t5_question_text + t5_context_text"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3b67760-81c2-41d7-b27a-3e9c8ddcd9be",
   "metadata": {},
   "source": [
    "Here's the output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "911f74c3-873f-4df4-939a-d50c92562e9c",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/peeti_mac/opt/anaconda3/lib/python3.9/site-packages/transformers/generation/tf_utils.py:834: UserWarning: Using the model-agnostic default `max_length` (=20) to control the generation length.  recommend setting `max_new_tokens` to control the maximum length of the generation.\n",
      "  warnings.warn(\n",
      "2023-10-10 19:55:18.800513: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7fcaa033c6f0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:\n",
      "2023-10-10 19:55:18.800539: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version\n",
      "2023-10-10 19:55:18.801066: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
      "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
      "2023-10-10 19:55:18.809671: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:255] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n",
      "2023-10-10 19:55:18.846257: I ./tensorflow/compiler/jit/device_compiler.h:186] Compiled cluster using XLA!  This line is logged at most once for the lifetime of the process.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['carbon monoxide']\n"
     ]
    }
   ],
   "source": [
    "t5_inputs = t5_tokenizer([t5_qa_input_text], return_tensors='tf')\n",
    "\n",
    "t5_summary_ids = t5_model.generate(t5_inputs['input_ids'])\n",
    "\n",
    "print([t5_tokenizer.decode(g, skip_special_tokens=True,\n",
    "                           clean_up_tokenization_spaces=False) for g in t5_summary_ids])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd7764f7-d89a-4517-983a-e45a75bc87d9",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}