notebooks/day3_Part_2_title_generator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hxCRj64GFtkx"
   },
   "source": [
    "<img src=\"https://raw.githubusercontent.com/cebirnie92/KAUST-Iraya_SummerSchool2021/main/figures/logo.jpg\">\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DSyWJmERnDpL"
   },
   "source": [
    "# DAY THREE, Task(2): Title Generation\n",
    "\n",
    "In this tutorial we will learn how to generate a title for a given paper's abstract. Title generation is a special type of text summarization, which is a well-known challenge of NLP.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EWmQ2YtSoqp3"
   },
   "source": [
    "\n",
    "# Problem Defenition\n",
    "Given an abstract, the task is to generate a title for this paper.\n",
    "### Example:\n",
    "<a href=\"https://ibb.co/12CZLxn\"><img src=\"https://i.ibb.co/WKrn613/Example.png\" alt=\"Example\" border=\"0\"></a>\"\n",
    "\n",
    "In this tutorial, we will generate titles based on the BERT model, which is one of highly powerful pre-trained models for solving Natural Language Processing problems. We will use a python library named [**Headliner**](https://as-ideas.github.io/headliner/) that includes BERT model for text generation. \n",
    "\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7aoRkeQ-oqzY"
   },
   "source": [
    "# Headliner Library\n",
    "\n",
    "\n",
    "**Headliner** is a sequence2sequence modeling library with the following features: \n",
    "*   It provides an easy-to-use API for both training and deployment. You can very easily use it in a few lines of code. \n",
    "\n",
    "*   It supports different encoder-decoder models, such as: \n",
    "  *   basic encoder-decoder\n",
    "  *   encoder-decoder with attention\n",
    "  *   transformer\n",
    "  *   pre-trained BERT-model.\n",
    "\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/1826/1*MqFZFkCBd9N-yLLC75awZA.jpeg\">\n",
    "\n",
    "Figure: Encoder-decoder sequence-to-sequence model.\n",
    "\n",
    "Source: https://as-ideas.github.io/headliner/\n",
    "\n",
    "\n",
    "## **So, let’s have fun and try to generate titles!**\n",
    "\n",
    "\n",
    " \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VLSLA9hGr20B"
   },
   "source": [
    "## Notebook Setup\n",
    "To run this tutorial, first make sure to use GPU. \n",
    "\n",
    "If you are running on Colab please select Runtime--> change runtime  type--> GPU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "m_JO3jJ5sZ1q"
   },
   "source": [
    "**Required libraries and packages:**\n",
    "\n",
    "If you are using your local machine, please install all these requirenments (provided in the environment.yml on github) (if you are using Colab, please jump to the next part)\n",
    "\n",
    "*   pandas : https://pandas.pydata.org/\n",
    "*   scikit-learn: https://scikit-learn.org/stable/\n",
    "\n",
    "*   tenserflow-gpu: pip install tensorflow-gpu\n",
    "*   tensorflow_datasets: pip install tensorflow_datasets\n",
    "\n",
    "\n",
    "\n",
    "*   [headliner library](https://github.com/as-ideas/headliner ): pip install headliner\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JnIz4w0tCOF0"
   },
   "source": [
    "### Download all needed packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "37CZKYyf9p8N",
    "outputId": "e6eba208-e030-4fe7-bd2b-aec2dfaf6c37"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting tensorflow-gpu\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/1d/a2/5ccf0a418eb22e0a2ae9edc1e7f5456d0a4b8b49524572897564b4030a9b/tensorflow_gpu-2.5.0-cp37-cp37m-manylinux2010_x86_64.whl (454.3MB)\n",
      "\u001b[K     |████████████████████████████████| 454.3MB 36kB/s \n",
      "\u001b[?25hRequirement already satisfied: tensorflow-estimator<2.6.0,>=2.5.0rc0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (2.5.0)\n",
      "Requirement already satisfied: keras-preprocessing~=1.1.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (1.1.2)\n",
      "Requirement already satisfied: tensorboard~=2.5 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (2.5.0)\n",
      "Requirement already satisfied: gast==0.4.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (0.4.0)\n",
      "Requirement already satisfied: keras-nightly~=2.5.0.dev in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (2.5.0.dev2021032900)\n",
      "Requirement already satisfied: absl-py~=0.10 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (0.12.0)\n",
      "Requirement already satisfied: astunparse~=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (1.6.3)\n",
      "Requirement already satisfied: opt-einsum~=3.3.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (3.3.0)\n",
      "Requirement already satisfied: h5py~=3.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (3.1.0)\n",
      "Requirement already satisfied: termcolor~=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (1.1.0)\n",
      "Requirement already satisfied: wrapt~=1.12.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (1.12.1)\n",
      "Requirement already satisfied: numpy~=1.19.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (1.19.5)\n",
      "Requirement already satisfied: typing-extensions~=3.7.4 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (3.7.4.3)\n",
      "Requirement already satisfied: wheel~=0.35 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (0.36.2)\n",
      "Requirement already satisfied: six~=1.15.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (1.15.0)\n",
      "Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (3.12.4)\n",
      "Requirement already satisfied: flatbuffers~=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (1.12)\n",
      "Requirement already satisfied: grpcio~=1.34.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (1.34.1)\n",
      "Requirement already satisfied: google-pasta~=0.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow-gpu) (0.2.0)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow-gpu) (3.3.4)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow-gpu) (2.23.0)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow-gpu) (57.0.0)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow-gpu) (0.4.4)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow-gpu) (0.6.1)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow-gpu) (1.8.0)\n",
      "Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow-gpu) (1.31.0)\n",
      "Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard~=2.5->tensorflow-gpu) (1.0.1)\n",
      "Requirement already satisfied: cached-property; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from h5py~=3.1.0->tensorflow-gpu) (1.5.2)\n",
      "Requirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard~=2.5->tensorflow-gpu) (4.5.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow-gpu) (2021.5.30)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow-gpu) (1.24.3)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow-gpu) (2.10)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard~=2.5->tensorflow-gpu) (3.0.4)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow-gpu) (1.3.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4; python_version >= \"3.6\" in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow-gpu) (4.7.2)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow-gpu) (4.2.2)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow-gpu) (0.2.8)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->markdown>=2.6.8->tensorboard~=2.5->tensorflow-gpu) (3.4.1)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard~=2.5->tensorflow-gpu) (3.1.1)\n",
      "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.7/dist-packages (from rsa<5,>=3.1.4; python_version >= \"3.6\"->google-auth<2,>=1.6.3->tensorboard~=2.5->tensorflow-gpu) (0.4.8)\n",
      "Installing collected packages: tensorflow-gpu\n",
      "Successfully installed tensorflow-gpu-2.5.0\n",
      "Collecting headliner\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/74/e4/386e9f58b8464261d4e220abaebe66da2426d55b6ea4186ec2cb828195ef/headliner-1.0.2-py3-none-any.whl (65kB)\n",
      "\u001b[K     |████████████████████████████████| 71kB 8.2MB/s \n",
      "\u001b[?25hRequirement already satisfied: spacy>=2.2.2 in /usr/local/lib/python3.7/dist-packages (from headliner) (2.2.4)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.7/dist-packages (from headliner) (3.13)\n",
      "Requirement already satisfied: scikit-learn in /usr/local/lib/python3.7/dist-packages (from headliner) (0.22.2.post1)\n",
      "Requirement already satisfied: nltk in /usr/local/lib/python3.7/dist-packages (from headliner) (3.2.5)\n",
      "Collecting transformers>=2.2.2\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d5/43/cfe4ee779bbd6a678ac6a97c5a5cdeb03c35f9eaebbb9720b036680f9a2d/transformers-4.6.1-py3-none-any.whl (2.2MB)\n",
      "\u001b[K     |████████████████████████████████| 2.3MB 38.4MB/s \n",
      "\u001b[?25hRequirement already satisfied: preshed<3.1.0,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (3.0.5)\n",
      "Requirement already satisfied: catalogue<1.1.0,>=0.0.7 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (1.0.0)\n",
      "Requirement already satisfied: blis<0.5.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (0.4.1)\n",
      "Requirement already satisfied: thinc==7.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (7.4.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (57.0.0)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (2.23.0)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (2.0.5)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (1.19.5)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (4.41.1)\n",
      "Requirement already satisfied: srsly<1.1.0,>=1.0.2 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (1.0.5)\n",
      "Requirement already satisfied: plac<1.2.0,>=0.9.6 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (1.1.3)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.4.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (0.8.2)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /usr/local/lib/python3.7/dist-packages (from spacy>=2.2.2->headliner) (1.0.5)\n",
      "Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->headliner) (1.0.1)\n",
      "Requirement already satisfied: scipy>=0.17.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn->headliner) (1.4.1)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from nltk->headliner) (1.15.0)\n",
      "Collecting tokenizers<0.11,>=0.10.1\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/d4/e2/df3543e8ffdab68f5acc73f613de9c2b155ac47f162e725dcac87c521c11/tokenizers-0.10.3-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (3.3MB)\n",
      "\u001b[K     |████████████████████████████████| 3.3MB 47.5MB/s \n",
      "\u001b[?25hCollecting sacremoses\n",
      "\u001b[?25l  Downloading https://files.pythonhosted.org/packages/75/ee/67241dc87f266093c533a2d4d3d69438e57d7a90abb216fa076e7d475d4a/sacremoses-0.0.45-py3-none-any.whl (895kB)\n",
      "\u001b[K     |████████████████████████████████| 901kB 39.6MB/s \n",
      "\u001b[?25hRequirement already satisfied: importlib-metadata; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from transformers>=2.2.2->headliner) (4.5.0)\n",
      "Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from transformers>=2.2.2->headliner) (20.9)\n",
      "Collecting huggingface-hub==0.0.8\n",
      "  Downloading https://files.pythonhosted.org/packages/a1/88/7b1e45720ecf59c6c6737ff332f41c955963090a18e72acbcbeac6b25e86/huggingface_hub-0.0.8-py3-none-any.whl\n",
      "Requirement already satisfied: filelock in /usr/local/lib/python3.7/dist-packages (from transformers>=2.2.2->headliner) (3.0.12)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.7/dist-packages (from transformers>=2.2.2->headliner) (2019.12.20)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->headliner) (3.0.4)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->headliner) (1.24.3)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->headliner) (2.10)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3.0.0,>=2.13.0->spacy>=2.2.2->headliner) (2021.5.30)\n",
      "Requirement already satisfied: click in /usr/local/lib/python3.7/dist-packages (from sacremoses->transformers>=2.2.2->headliner) (7.1.2)\n",
      "Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers>=2.2.2->headliner) (3.4.1)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4; python_version < \"3.8\" in /usr/local/lib/python3.7/dist-packages (from importlib-metadata; python_version < \"3.8\"->transformers>=2.2.2->headliner) (3.7.4.3)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.7/dist-packages (from packaging->transformers>=2.2.2->headliner) (2.4.7)\n",
      "Installing collected packages: tokenizers, sacremoses, huggingface-hub, transformers, headliner\n",
      "Successfully installed headliner-1.0.2 huggingface-hub-0.0.8 sacremoses-0.0.45 tokenizers-0.10.3 transformers-4.6.1\n",
      "Requirement already satisfied: tensorflow_datasets in /usr/local/lib/python3.7/dist-packages (4.0.1)\n",
      "Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (1.19.5)\n",
      "Requirement already satisfied: future in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (0.16.0)\n",
      "Requirement already satisfied: attrs>=18.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (21.2.0)\n",
      "Requirement already satisfied: dill in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (0.3.3)\n",
      "Requirement already satisfied: protobuf>=3.6.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (3.12.4)\n",
      "Requirement already satisfied: six in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (1.15.0)\n",
      "Requirement already satisfied: requests>=2.19.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (2.23.0)\n",
      "Requirement already satisfied: tensorflow-metadata in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (1.0.0)\n",
      "Requirement already satisfied: dm-tree in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (0.1.6)\n",
      "Requirement already satisfied: absl-py in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (0.12.0)\n",
      "Requirement already satisfied: promise in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (2.3)\n",
      "Requirement already satisfied: tqdm in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (4.41.1)\n",
      "Requirement already satisfied: importlib-resources; python_version < \"3.9\" in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (5.1.4)\n",
      "Requirement already satisfied: termcolor in /usr/local/lib/python3.7/dist-packages (from tensorflow_datasets) (1.1.0)\n",
      "Requirement already satisfied: setuptools in /usr/local/lib/python3.7/dist-packages (from protobuf>=3.6.1->tensorflow_datasets) (57.0.0)\n",
      "Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tensorflow_datasets) (2.10)\n",
      "Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tensorflow_datasets) (3.0.4)\n",
      "Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tensorflow_datasets) (1.24.3)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests>=2.19.0->tensorflow_datasets) (2021.5.30)\n",
      "Requirement already satisfied: googleapis-common-protos<2,>=1.52.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow-metadata->tensorflow_datasets) (1.53.0)\n",
      "Requirement already satisfied: zipp>=3.1.0; python_version < \"3.10\" in /usr/local/lib/python3.7/dist-packages (from importlib-resources; python_version < \"3.9\"->tensorflow_datasets) (3.4.1)\n"
     ]
    }
   ],
   "source": [
    "## Uncomment if running on CoLab\n",
    "# # Install TensorFlow and library package via PyPI\n",
    "# !pip install tensorflow-gpu\n",
    "# !pip install headliner\n",
    "# !pip install tensorflow_datasets==1.2.0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IllJxlJF5IjA"
   },
   "source": [
    "## Import the data\n",
    "**Working on CoLab:**\n",
    "Please add the file 'paper_info.csv' to your Drive"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "b1076dfc-b9ad-4769-8c92-a6c4dae69d19",
    "_uuid": "8f2839f25d086af736a60e9eeb907d3b93b6e0e5",
    "execution": {
     "iopub.execute_input": "2021-06-08T13:11:18.291577Z",
     "iopub.status.busy": "2021-06-08T13:11:18.290943Z",
     "iopub.status.idle": "2021-06-08T13:11:19.105969Z",
     "shell.execute_reply": "2021-06-08T13:11:19.104627Z",
     "shell.execute_reply.started": "2021-06-08T13:11:18.291523Z"
    },
    "id": "OcgJGv2A5IjE"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "v9au7QhCp_gL"
   },
   "outputs": [],
   "source": [
    "## Uncomment if running on CoLab\n",
    "#from google.colab import drive\n",
    "#drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 409
    },
    "id": "sLhMRtl2aEZo",
    "outputId": "0aa49147-d162-435a-da33-bbc293056173"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>An algorithm to compute the full set of many-t...</td>\n",
       "      <td>The paper proposes an algorithm to compute the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Relative entropy maximization and directed dif...</td>\n",
       "      <td>Motivated by the concept of maximum entropy me...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>On the secant varieties to the tangential vari...</td>\n",
       "      <td>We study the dimensions of the higher secant v...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The $L(2,1)$-Labeling Problem on Graphs</td>\n",
       "      <td>1. Introduction. The channel assignment probl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Probabilities of causation: Bounds and identif...</td>\n",
       "      <td>This paper deals with the problem of estimati...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title  \\\n",
       "0  An algorithm to compute the full set of many-t...   \n",
       "1  Relative entropy maximization and directed dif...   \n",
       "2  On the secant varieties to the tangential vari...   \n",
       "3            The $L(2,1)$-Labeling Problem on Graphs   \n",
       "4  Probabilities of causation: Bounds and identif...   \n",
       "\n",
       "                                            abstract  \n",
       "0  The paper proposes an algorithm to compute the...  \n",
       "1  Motivated by the concept of maximum entropy me...  \n",
       "2  We study the dimensions of the higher secant v...  \n",
       "3   1. Introduction. The channel assignment probl...  \n",
       "4   This paper deals with the problem of estimati...  "
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Data directory\n",
    "data_dir = \"../data/\" \n",
    "#data_dir = '/content/drive/My Drive/'  # Uncomment if running on CoLab\n",
    "\n",
    "#read dataset file\n",
    "df = pd.read_csv(data_dir+'paper_info.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uAxWmUUo17Xk",
    "outputId": "f0376e07-cfe0-4e68-cb02-56d9eb96ad1f"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1517"
      ]
     },
     "execution_count": 3,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#check how many examples we have in our dataset\n",
    "len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TyXIkE3A5IjJ"
   },
   "source": [
    "## Clean the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "execution": {
     "iopub.execute_input": "2021-06-08T13:11:26.081495Z",
     "iopub.status.busy": "2021-06-08T13:11:26.081142Z",
     "iopub.status.idle": "2021-06-08T13:11:26.119374Z",
     "shell.execute_reply": "2021-06-08T13:11:26.118367Z",
     "shell.execute_reply.started": "2021-06-08T13:11:26.081463Z"
    },
    "id": "4Wu1B4iI5IjK",
    "outputId": "5b7ba26b-b80a-4dc0-a90a-d8765b4ae2e2"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>An algorithm to compute the full set of many-t...</td>\n",
       "      <td>The paper proposes an algorithm to compute the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Relative entropy maximization and directed dif...</td>\n",
       "      <td>Motivated by the concept of maximum entropy me...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>On the secant varieties to the tangential vari...</td>\n",
       "      <td>We study the dimensions of the higher secant v...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The $L(2,1)$-Labeling Problem on Graphs</td>\n",
       "      <td>1. Introduction. The channel assignment probl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Probabilities of causation: Bounds and identif...</td>\n",
       "      <td>This paper deals with the problem of estimati...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title  \\\n",
       "0  An algorithm to compute the full set of many-t...   \n",
       "1  Relative entropy maximization and directed dif...   \n",
       "2  On the secant varieties to the tangential vari...   \n",
       "3            The $L(2,1)$-Labeling Problem on Graphs   \n",
       "4  Probabilities of causation: Bounds and identif...   \n",
       "\n",
       "                                            abstract  \n",
       "0  The paper proposes an algorithm to compute the...  \n",
       "1  Motivated by the concept of maximum entropy me...  \n",
       "2  We study the dimensions of the higher secant v...  \n",
       "3   1. Introduction. The channel assignment probl...  \n",
       "4   This paper deals with the problem of estimati...  "
      ]
     },
     "execution_count": 4,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# drop nan values\n",
    "df = df.dropna()\n",
    "# resetting the index\n",
    "df = df.reset_index(drop=True)\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "execution": {
     "iopub.execute_input": "2021-06-08T13:11:30.669608Z",
     "iopub.status.busy": "2021-06-08T13:11:30.669124Z",
     "iopub.status.idle": "2021-06-08T13:11:55.729441Z",
     "shell.execute_reply": "2021-06-08T13:11:55.728728Z",
     "shell.execute_reply.started": "2021-06-08T13:11:30.669566Z"
    },
    "id": "tK0W34cY5IjL",
    "outputId": "6e166c3c-6f78-4482-9e7a-cbcc7c572c1a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>abstract</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>An algorithm to compute the full set of many-t...</td>\n",
       "      <td>The paper proposes an algorithm to compute the...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Relative entropy maximization and directed dif...</td>\n",
       "      <td>Motivated by the concept of maximum entropy me...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>On the secant varieties to the tangential vari...</td>\n",
       "      <td>We study the dimensions of the higher secant v...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The $L(2,1)$-Labeling Problem on Graphs</td>\n",
       "      <td>1. Introduction. The channel assignment probl...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Probabilities of causation: Bounds and identif...</td>\n",
       "      <td>This paper deals with the problem of estimati...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                               title  \\\n",
       "0  An algorithm to compute the full set of many-t...   \n",
       "1  Relative entropy maximization and directed dif...   \n",
       "2  On the secant varieties to the tangential vari...   \n",
       "3            The $L(2,1)$-Labeling Problem on Graphs   \n",
       "4  Probabilities of causation: Bounds and identif...   \n",
       "\n",
       "                                            abstract  \n",
       "0  The paper proposes an algorithm to compute the...  \n",
       "1  Motivated by the concept of maximum entropy me...  \n",
       "2  We study the dimensions of the higher secant v...  \n",
       "3   1. Introduction. The channel assignment probl...  \n",
       "4   This paper deals with the problem of estimati...  "
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# remove noise from the text, remove new lines, tabs.\n",
    "for i in range(0,len(df['abstract'])):\n",
    "    if type(df['abstract'][i]) == str:\n",
    "        df['abstract'][i] = df['abstract'][i].replace('\\n','').replace('\\r','').replace('/','')\n",
    "        df['title'][i] = df['title'][i].replace('\\n','').replace('\\r','').replace('/','')\n",
    "    else:\n",
    "        print(str(df['abstract'][i]))\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3tWk7cOnSAav",
    "outputId": "48041152-8fa5-42a7-85ea-c2ba153c8199"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1517"
      ]
     },
     "execution_count": 6,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "p101hU7O5IjO"
   },
   "source": [
    "## Split the data into training and testing sets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-06-08T13:12:17.148468Z",
     "iopub.status.busy": "2021-06-08T13:12:17.148112Z",
     "iopub.status.idle": "2021-06-08T13:12:19.472533Z",
     "shell.execute_reply": "2021-06-08T13:12:19.471393Z",
     "shell.execute_reply.started": "2021-06-08T13:12:17.148438Z"
    },
    "id": "XnDLEmav5IjP",
    "outputId": "8be53996-8775-4b96-d723-b3bbe619890f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "304\n",
      "1213\n"
     ]
    }
   ],
   "source": [
    "#Split data into train, test\n",
    "import sklearn.model_selection as model_selection\n",
    "train_df, test_df = model_selection.train_test_split(df, test_size = 0.2)\n",
    "\n",
    "print(len(test_df))\n",
    "print(len(train_df))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "v9e0hspbOYW6"
   },
   "outputs": [],
   "source": [
    "#arrange the training data into list\n",
    "train_data = []\n",
    "for i in range(0,len(train_df['abstract'])):\n",
    "    train_data.append((df['abstract'][i], df['title'][i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-ug3HR8r5IjU"
   },
   "source": [
    "## Preprocessing\n",
    "\n",
    "In this part we need to add BERT-specific start and end token to our training data using bert_preprocessor and spaCy packages\n",
    "\n",
    "Here, the tokens **[CLS]** and **[SEP]** will be added to the start and end of each sentence.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-06-08T12:30:01.268727Z",
     "iopub.status.busy": "2021-06-08T12:30:01.268186Z",
     "iopub.status.idle": "2021-06-08T12:30:01.391619Z",
     "shell.execute_reply": "2021-06-08T12:30:01.390082Z",
     "shell.execute_reply.started": "2021-06-08T12:30:01.268677Z"
    },
    "id": "pZSFjz_d5IjU"
   },
   "outputs": [],
   "source": [
    "from headliner.preprocessing.bert_preprocessor import BertPreprocessor\n",
    "from spacy.lang.en import English\n",
    "\n",
    "# use BERT-specific start and end token\n",
    "preprocessor = BertPreprocessor(nlp=English())\n",
    "train_prep = [preprocessor(t) for t in train_data]\n",
    "targets_prep = [t[1] for t in train_prep]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 90
    },
    "id": "veqaJzLn7m0c",
    "outputId": "b3f6aa90-4980-4e1f-f273-dc5889d7c286"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[CLS] The paper proposes an algorithm to compute the full set of many-to-many stable matchings when agents have substitutable preferences. [SEP] [CLS] The algorithm starts by calculating the two optimal stable matchings using the deferred-acceptance algorithm. [SEP] [CLS] Then, it computes each remaining stable matching as the firm-optimal stable matching corresponding to a new preference profile, which is obtained after modifying the preferences of a previously identified sequence of firms. [SEP] [CLS] © 2003 Elsevier B.V. All rights reserved. [SEP]'"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Check one sample of the abstracts after preprocessing\n",
    "#it Should add the tokens before and and after each sentence.\n",
    "train_prep[0][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "CVR7YfR68FXg",
    "outputId": "048a3842-efed-49b7-8461-ca4058098295"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'[CLS] On the secant varieties to the tangential varieties of a Veronesean [SEP]'"
      ]
     },
     "execution_count": 11,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Check one sample of the titles after preprocessing\n",
    "targets_prep[2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pw2rHlKE5IjV"
   },
   "source": [
    "## Training \n",
    "\n",
    "Here is the main part of the tutorial . We define the model and train it.\n",
    "\n",
    "You can play with different parameters to see how the model works. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "_cell_guid": "79c7e3d0-c299-4dcb-8224-4455121ee9b0",
    "_uuid": "d629ff2d2480ee46fbb7e2d37f6b5fab8052498a",
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000,
     "referenced_widgets": [
      "8d99a96aca3c4140aa83332a8fa38dd7",
      "756cb78309be4aee9a87c5ffef3ffc97",
      "c0c4bc0029c64ec6ab5996b2fffb054a",
      "e2c021a7fe0940928b633e4bffe10992",
      "cc47c8493efc4ab7805a05c6ffb69a1f",
      "596471dd7146450ca539376e9d72b747",
      "369a8fdc81ba44e4b178992e4ed34877",
      "1339875a08794aff990637693e5f4f90",
      "2c2f8562ac1f46e7a4ba82fd1ff5f06d",
      "fc572a59965144d39bfc21696bb757ef",
      "cf37738d358b4819aeb945ec5a69e51f",
      "8f07544e21a5484483862beb82ec0c86",
      "4b376439dca649bfa0ab11df63b8466a",
      "b3aeb42c860d4c7c83ae9c7c8a5328f5",
      "e720c2d40abf44f8960f66d44e157303",
      "5ac53ba4abbb4da99929367a4ed0af10",
      "98ceb4fbac9040e48d311068725a1765",
      "9dc605935f8c480b8bdadcbdc2a43f67",
      "d580b5669bd0404f974792321d61361b",
      "5c830657c2e44251a0b62011542780a9",
      "31cbfd36a52b4638b2677745d16812da",
      "41a4ec5f76de458383024422a5b1faf2",
      "b1eb8aaf03ed437bb75f254796938539",
      "a163976c22b8469d9112306a4e82580c",
      "9fd9edb1271949cb8064fe09bd801303",
      "775f74cb728944588f527d8ffd6ac7a9",
      "3898c7f1337e4ca4bbde3b9298d9f265",
      "935cf044b05a420f8801c0ac73e83d74",
      "699a541fa0bd4ad2a424523aa7662fc7",
      "a6074d382cb44903afaa9592e438ee14",
      "3f712265be4a49779631f1c4ebb52f63",
      "c10a7f750ed64e7cab0d164576225996",
      "98ec4e207c474fb89aa5f59c9588733e",
      "8c67f0909f0742e08957e52ca206c734",
      "80de91ca85ef4e48aa2e9f720c53d2ea",
      "a883f8f7718245a087e9298d20243078",
      "d405946ea39c4ddc859218dac8c75170",
      "0947c71b077c4d77b11a0695550ec7ba",
      "bad195eb9a22490caf2853ea14c0529b",
      "3d12a086bd9e4be7af26564078debb2c"
     ]
    },
    "id": "-eFE0dyn5IjX",
    "outputId": "f2426d9c-e58e-4853-ec80-26eddfce870d"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 607\n",
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 303\n",
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 151\n",
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 75\n",
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 37\n",
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 18\n",
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 9\n",
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 4\n",
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 2\n",
      "INFO:absl:SubwordTextEncoder build: trying min_token_count 1\n",
      "Some layers from the model checkpoint at bert-base-uncased were not used when initializing TFBertModel: ['nsp___cls', 'mlm___cls']\n",
      "- This IS expected if you are initializing TFBertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing TFBertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n",
      "All the layers of TFBertModel were initialized from the model checkpoint at bert-base-uncased.\n",
      "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertModel for predictions without further training.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training an already initialized model...\n",
      "WARNING:tensorflow:AutoGraph could not transform <function BertSummarizer.new_train_step.<locals>.train_step at 0x1490e5124950> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <function BertSummarizer.new_train_step.<locals>.train_step at 0x1490e5124950> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <function BertSummarizer.new_train_step.<locals>.train_step at 0x1490e5124950> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method Encoder.call of <headliner.model.bert_model.Encoder object at 0x1490e5684ed0>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method Encoder.call of <headliner.model.bert_model.Encoder object at 0x1490e5684ed0>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method Encoder.call of <headliner.model.bert_model.Encoder object at 0x1490e5684ed0>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertModel.call of <transformers.models.bert.modeling_tf_bert.TFBertModel object at 0x1490e5875110>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertModel.call of <transformers.models.bert.modeling_tf_bert.TFBertModel object at 0x1490e5875110>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method TFBertModel.call of <transformers.models.bert.modeling_tf_bert.TFBertModel object at 0x1490e5875110>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertMainLayer.call of <transformers.models.bert.modeling_tf_bert.TFBertMainLayer object at 0x1490e5875510>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertMainLayer.call of <transformers.models.bert.modeling_tf_bert.TFBertMainLayer object at 0x1490e5875510>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method TFBertMainLayer.call of <transformers.models.bert.modeling_tf_bert.TFBertMainLayer object at 0x1490e5875510>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertEmbeddings.call of <transformers.models.bert.modeling_tf_bert.TFBertEmbeddings object at 0x1490e57c2250>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertEmbeddings.call of <transformers.models.bert.modeling_tf_bert.TFBertEmbeddings object at 0x1490e57c2250>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method TFBertEmbeddings.call of <transformers.models.bert.modeling_tf_bert.TFBertEmbeddings object at 0x1490e57c2250>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertEncoder.call of <transformers.models.bert.modeling_tf_bert.TFBertEncoder object at 0x1490e56eee10>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertEncoder.call of <transformers.models.bert.modeling_tf_bert.TFBertEncoder object at 0x1490e56eee10>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method TFBertEncoder.call of <transformers.models.bert.modeling_tf_bert.TFBertEncoder object at 0x1490e56eee10>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertLayer.call of <transformers.models.bert.modeling_tf_bert.TFBertLayer object at 0x1490e56eee50>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertLayer.call of <transformers.models.bert.modeling_tf_bert.TFBertLayer object at 0x1490e56eee50>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method TFBertLayer.call of <transformers.models.bert.modeling_tf_bert.TFBertLayer object at 0x1490e56eee50>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertAttention.call of <transformers.models.bert.modeling_tf_bert.TFBertAttention object at 0x1490e56eeed0>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertAttention.call of <transformers.models.bert.modeling_tf_bert.TFBertAttention object at 0x1490e56eeed0>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method TFBertAttention.call of <transformers.models.bert.modeling_tf_bert.TFBertAttention object at 0x1490e56eeed0>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfAttention.call of <transformers.models.bert.modeling_tf_bert.TFBertSelfAttention object at 0x1490e56ee3d0>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertSelfAttention.call of <transformers.models.bert.modeling_tf_bert.TFBertSelfAttention object at 0x1490e56ee3d0>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method TFBertSelfAttention.call of <transformers.models.bert.modeling_tf_bert.TFBertSelfAttention object at 0x1490e56ee3d0>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertPooler.call of <transformers.models.bert.modeling_tf_bert.TFBertPooler object at 0x1490e56eea50>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method TFBertPooler.call of <transformers.models.bert.modeling_tf_bert.TFBertPooler object at 0x1490e56eea50>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method TFBertPooler.call of <transformers.models.bert.modeling_tf_bert.TFBertPooler object at 0x1490e56eea50>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method Decoder.call of <headliner.model.bert_model.Decoder object at 0x1490e5871e50>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method Decoder.call of <headliner.model.bert_model.Decoder object at 0x1490e5871e50>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method Decoder.call of <headliner.model.bert_model.Decoder object at 0x1490e5871e50>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:AutoGraph could not transform <bound method MultiHeadAttention.call of <headliner.model.transformer_util.MultiHeadAttention object at 0x1490e50a0750>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:AutoGraph could not transform <bound method MultiHeadAttention.call of <headliner.model.transformer_util.MultiHeadAttention object at 0x1490e50a0750>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING: AutoGraph could not transform <bound method MultiHeadAttention.call of <headliner.model.transformer_util.MultiHeadAttention object at 0x1490e50a0750>> and will run it as-is.\n",
      "Please report this to the TensorFlow team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output.\n",
      "Cause: module 'gast' has no attribute 'Index'\n",
      "To silence this warning, decorate the function with @tf.autograph.experimental.do_not_convert\n",
      "WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The parameters `output_attentions`, `output_hidden_states` and `use_cache` cannot be updated when calling a model.They have to be set to True/False in the config object (i.e.: `config=XConfig.from_pretrained('name', output_attentions=True)`).\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:The parameter `return_dict` cannot be set in graph mode and will always be set to `True`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:Gradients do not exist for variables ['tf_bert_model/bert/pooler/dense/kernel:0', 'tf_bert_model/bert/pooler/dense/bias:0'] when minimizing the loss.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 0, batch 10, logs: {'loss': 7.167372131347657}\n",
      "epoch 0, batch 20, logs: {'loss': 7.1193702697753904}\n",
      "epoch 0, batch 30, logs: {'loss': 7.256068420410156}\n",
      "epoch 0, batch 40, logs: {'loss': 7.254052734375}\n",
      "epoch 0, batch 50, logs: {'loss': 7.2374853515625}\n",
      "epoch 0, batch 60, logs: {'loss': 7.250812784830729}\n",
      "epoch 0, batch 70, logs: {'loss': 7.141945975167411}\n",
      "epoch 0, batch 80, logs: {'loss': 7.1068885803222654}\n",
      "epoch 0, batch 90, logs: {'loss': 7.070761447482639}\n",
      "epoch 0, batch 100, logs: {'loss': 7.04937744140625}\n",
      "epoch 0, batch 110, logs: {'loss': 7.0407936789772725}\n",
      "epoch 0, batch 120, logs: {'loss': 7.005577087402344}\n",
      "epoch 0, batch 130, logs: {'loss': 6.99504159780649}\n",
      "epoch 0, batch 140, logs: {'loss': 6.9932861328125}\n",
      "epoch 0, batch 150, logs: {'loss': 6.960576171875}\n",
      "epoch 0, batch 160, logs: {'loss': 6.9315338134765625}\n",
      "epoch 0, batch 170, logs: {'loss': 6.921579877068015}\n",
      "epoch 0, batch 180, logs: {'loss': 6.918975151909722}\n",
      "epoch 0, batch 190, logs: {'loss': 6.90308195415296}\n",
      "epoch 0, batch 200, logs: {'loss': 6.8684521484375}\n",
      "epoch 0, batch 210, logs: {'loss': 6.858997744605655}\n",
      "epoch 0, batch 220, logs: {'loss': 6.8512662020596595}\n",
      "epoch 0, batch 230, logs: {'loss': 6.823725692085598}\n",
      "epoch 0, batch 240, logs: {'loss': 6.813185119628907}\n",
      "epoch 0, batch 250, logs: {'loss': 6.78995068359375}\n",
      "epoch 0, batch 260, logs: {'loss': 6.764026817908654}\n",
      "epoch 0, batch 270, logs: {'loss': 6.728141728153935}\n",
      "epoch 0, batch 280, logs: {'loss': 6.69922354561942}\n",
      "epoch 0, batch 290, logs: {'loss': 6.671557196255388}\n",
      "epoch 0, batch 300, logs: {'loss': 6.648816731770833}\n",
      "epoch 0, batch 310, logs: {'loss': 6.636476184475806}\n",
      "epoch 0, batch 320, logs: {'loss': 6.6179359436035154}\n",
      "epoch 0, batch 330, logs: {'loss': 6.600156102035985}\n",
      "epoch 0, batch 340, logs: {'loss': 6.585112448299633}\n",
      "epoch 0, batch 350, logs: {'loss': 6.563353097098214}\n",
      "epoch 0, batch 360, logs: {'loss': 6.545132107204861}\n",
      "epoch 0, batch 370, logs: {'loss': 6.533315957559122}\n",
      "epoch 0, batch 380, logs: {'loss': 6.528677528782895}\n",
      "epoch 0, batch 390, logs: {'loss': 6.517132411858975}\n",
      "epoch 0, batch 400, logs: {'loss': 6.5025244140625}\n",
      "epoch 0, batch 410, logs: {'loss': 6.491062071265244}\n",
      "epoch 0, batch 420, logs: {'loss': 6.488541666666666}\n",
      "epoch 0, batch 430, logs: {'loss': 6.4764500817587205}\n",
      "epoch 0, batch 440, logs: {'loss': 6.476151344992898}\n",
      "epoch 0, batch 450, logs: {'loss': 6.468624674479167}\n",
      "epoch 0, batch 460, logs: {'loss': 6.461868418817935}\n",
      "epoch 0, batch 470, logs: {'loss': 6.449772481715425}\n",
      "epoch 0, batch 480, logs: {'loss': 6.4445749918619795}\n",
      "epoch 0, batch 490, logs: {'loss': 6.429249043367347}\n",
      "epoch 0, batch 500, logs: {'loss': 6.4196015625}\n",
      "epoch 1, batch 510, logs: {'loss': 5.820395660400391}\n",
      "epoch 1, batch 520, logs: {'loss': 5.945095825195312}\n",
      "epoch 1, batch 530, logs: {'loss': 5.8736216227213545}\n",
      "epoch 1, batch 540, logs: {'loss': 5.856154251098633}\n",
      "epoch 1, batch 550, logs: {'loss': 5.856688842773438}\n",
      "epoch 1, batch 560, logs: {'loss': 5.911790974934896}\n",
      "epoch 1, batch 570, logs: {'loss': 5.93895525251116}\n",
      "epoch 1, batch 580, logs: {'loss': 5.9599559783935545}\n",
      "epoch 1, batch 590, logs: {'loss': 5.96869371202257}\n",
      "epoch 1, batch 600, logs: {'loss': 5.96154052734375}\n",
      "finished iterating over dataset, total batches: 606\n",
      "epoch 1, batch 610, logs: {'loss': 5.966403475674716}\n",
      "epoch 1, batch 620, logs: {'loss': 5.965849304199219}\n",
      "epoch 1, batch 630, logs: {'loss': 5.927955979567308}\n",
      "epoch 1, batch 640, logs: {'loss': 5.912181309291294}\n",
      "epoch 1, batch 650, logs: {'loss': 5.914322509765625}\n",
      "epoch 1, batch 660, logs: {'loss': 5.8979133605957035}\n",
      "epoch 1, batch 670, logs: {'loss': 5.87616792566636}\n",
      "epoch 1, batch 680, logs: {'loss': 5.850662570529514}\n",
      "epoch 1, batch 690, logs: {'loss': 5.837029708059211}\n",
      "epoch 1, batch 700, logs: {'loss': 5.847933959960938}\n",
      "epoch 1, batch 710, logs: {'loss': 5.839919317336309}\n",
      "epoch 1, batch 720, logs: {'loss': 5.829951615767046}\n",
      "epoch 1, batch 730, logs: {'loss': 5.800606636379076}\n",
      "epoch 1, batch 740, logs: {'loss': 5.791690063476563}\n",
      "epoch 1, batch 750, logs: {'loss': 5.78642626953125}\n",
      "epoch 1, batch 760, logs: {'loss': 5.771666541466346}\n",
      "epoch 1, batch 770, logs: {'loss': 5.778276909722222}\n",
      "epoch 1, batch 780, logs: {'loss': 5.766195678710938}\n",
      "epoch 1, batch 790, logs: {'loss': 5.760021130792026}\n",
      "epoch 1, batch 800, logs: {'loss': 5.745762532552083}\n",
      "epoch 1, batch 810, logs: {'loss': 5.735399603074597}\n",
      "epoch 1, batch 820, logs: {'loss': 5.730392837524414}\n",
      "epoch 1, batch 830, logs: {'loss': 5.7280854196259465}\n",
      "epoch 1, batch 840, logs: {'loss': 5.732284366383272}\n",
      "epoch 1, batch 850, logs: {'loss': 5.724409877232143}\n",
      "epoch 1, batch 860, logs: {'loss': 5.7282958984375}\n",
      "epoch 1, batch 870, logs: {'loss': 5.727023067989865}\n",
      "epoch 1, batch 880, logs: {'loss': 5.727047568873355}\n",
      "epoch 1, batch 890, logs: {'loss': 5.722357647235577}\n",
      "epoch 1, batch 900, logs: {'loss': 5.711500854492187}\n",
      "epoch 1, batch 910, logs: {'loss': 5.708075695503049}\n",
      "epoch 1, batch 920, logs: {'loss': 5.708215913318452}\n",
      "epoch 1, batch 930, logs: {'loss': 5.709190475109011}\n",
      "epoch 1, batch 940, logs: {'loss': 5.718842662464489}\n",
      "epoch 1, batch 950, logs: {'loss': 5.720417209201389}\n",
      "epoch 1, batch 960, logs: {'loss': 5.717151409646739}\n",
      "epoch 1, batch 970, logs: {'loss': 5.708040018284574}\n",
      "epoch 1, batch 980, logs: {'loss': 5.710747273763021}\n",
      "epoch 1, batch 990, logs: {'loss': 5.709044164540816}\n",
      "epoch 1, batch 1000, logs: {'loss': 5.709525390625}\n",
      "epoch 2, batch 1010, logs: {'loss': 5.739593124389648}\n",
      "epoch 2, batch 1020, logs: {'loss': 5.5596565246582035}\n",
      "epoch 2, batch 1030, logs: {'loss': 5.448111470540365}\n",
      "epoch 2, batch 1040, logs: {'loss': 5.458777236938476}\n",
      "epoch 2, batch 1050, logs: {'loss': 5.49279052734375}\n",
      "epoch 2, batch 1060, logs: {'loss': 5.5217742919921875}\n",
      "epoch 2, batch 1070, logs: {'loss': 5.529515729631696}\n",
      "epoch 2, batch 1080, logs: {'loss': 5.455258560180664}\n",
      "epoch 2, batch 1090, logs: {'loss': 5.464704047309028}\n",
      "epoch 2, batch 1100, logs: {'loss': 5.469182739257812}\n",
      "epoch 2, batch 1110, logs: {'loss': 5.485716663707386}\n",
      "epoch 2, batch 1120, logs: {'loss': 5.509097798665365}\n",
      "epoch 2, batch 1130, logs: {'loss': 5.518426983173077}\n",
      "epoch 2, batch 1140, logs: {'loss': 5.5378579275948665}\n",
      "epoch 2, batch 1150, logs: {'loss': 5.546582845052083}\n",
      "epoch 2, batch 1160, logs: {'loss': 5.5563201904296875}\n",
      "epoch 2, batch 1170, logs: {'loss': 5.542395378561581}\n",
      "epoch 2, batch 1180, logs: {'loss': 5.532154676649306}\n",
      "epoch 2, batch 1190, logs: {'loss': 5.5205810546875}\n",
      "epoch 2, batch 1200, logs: {'loss': 5.516620483398437}\n",
      "epoch 2, batch 1210, logs: {'loss': 5.5311006091889885}\n",
      "finished iterating over dataset, total batches: 1212\n",
      "epoch 2, batch 1220, logs: {'loss': 5.548907470703125}\n",
      "epoch 2, batch 1230, logs: {'loss': 5.532236646569293}\n",
      "epoch 2, batch 1240, logs: {'loss': 5.520064290364584}\n",
      "epoch 2, batch 1250, logs: {'loss': 5.50482177734375}\n",
      "epoch 2, batch 1260, logs: {'loss': 5.501266714242789}\n",
      "epoch 2, batch 1270, logs: {'loss': 5.485373716001157}\n",
      "epoch 2, batch 1280, logs: {'loss': 5.487682233537947}\n",
      "epoch 2, batch 1290, logs: {'loss': 5.47570001010237}\n",
      "epoch 2, batch 1300, logs: {'loss': 5.482117106119792}\n",
      "epoch 2, batch 1310, logs: {'loss': 5.486887679561492}\n",
      "epoch 2, batch 1320, logs: {'loss': 5.503039932250976}\n",
      "epoch 2, batch 1330, logs: {'loss': 5.490622410629735}\n",
      "epoch 2, batch 1340, logs: {'loss': 5.489971923828125}\n",
      "epoch 2, batch 1350, logs: {'loss': 5.484560198102678}\n",
      "epoch 2, batch 1360, logs: {'loss': 5.471719699435764}\n",
      "epoch 2, batch 1370, logs: {'loss': 5.474228317673142}\n",
      "epoch 2, batch 1380, logs: {'loss': 5.470874665912829}\n",
      "epoch 2, batch 1390, logs: {'loss': 5.461340019030449}\n",
      "epoch 2, batch 1400, logs: {'loss': 5.453600463867187}\n",
      "epoch 2, batch 1410, logs: {'loss': 5.458713438452744}\n",
      "epoch 2, batch 1420, logs: {'loss': 5.459172130766369}\n",
      "epoch 2, batch 1430, logs: {'loss': 5.46071039244186}\n",
      "epoch 2, batch 1440, logs: {'loss': 5.453531161221591}\n",
      "epoch 2, batch 1450, logs: {'loss': 5.457232530381944}\n",
      "epoch 2, batch 1460, logs: {'loss': 5.452427606997283}\n",
      "epoch 2, batch 1470, logs: {'loss': 5.453689120678192}\n",
      "epoch 2, batch 1480, logs: {'loss': 5.450410970052084}\n",
      "epoch 2, batch 1490, logs: {'loss': 5.448751395089285}\n",
      "epoch 2, batch 1500, logs: {'loss': 5.45175}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 3, batch 1510, logs: {'loss': 5.545142364501953}\n",
      "epoch 3, batch 1520, logs: {'loss': 5.417619705200195}\n",
      "epoch 3, batch 1530, logs: {'loss': 5.433860270182292}\n",
      "epoch 3, batch 1540, logs: {'loss': 5.375527191162109}\n",
      "epoch 3, batch 1550, logs: {'loss': 5.350341796875}\n",
      "epoch 3, batch 1560, logs: {'loss': 5.3614501953125}\n",
      "epoch 3, batch 1570, logs: {'loss': 5.371289498465401}\n",
      "epoch 3, batch 1580, logs: {'loss': 5.392351150512695}\n",
      "epoch 3, batch 1590, logs: {'loss': 5.437780083550347}\n",
      "epoch 3, batch 1600, logs: {'loss': 5.455440063476562}\n",
      "epoch 3, batch 1610, logs: {'loss': 5.442047119140625}\n",
      "epoch 3, batch 1620, logs: {'loss': 5.442609151204427}\n",
      "epoch 3, batch 1630, logs: {'loss': 5.433273550180289}\n",
      "epoch 3, batch 1640, logs: {'loss': 5.441652570452009}\n",
      "epoch 3, batch 1650, logs: {'loss': 5.459882405598958}\n",
      "epoch 3, batch 1660, logs: {'loss': 5.435660934448242}\n",
      "epoch 3, batch 1670, logs: {'loss': 5.423834587545956}\n",
      "epoch 3, batch 1680, logs: {'loss': 5.426718139648438}\n",
      "epoch 3, batch 1690, logs: {'loss': 5.422353644120066}\n",
      "epoch 3, batch 1700, logs: {'loss': 5.412247924804688}\n",
      "epoch 3, batch 1710, logs: {'loss': 5.432611374627976}\n",
      "epoch 3, batch 1720, logs: {'loss': 5.418137428977273}\n",
      "epoch 3, batch 1730, logs: {'loss': 5.414496114979619}\n",
      "epoch 3, batch 1740, logs: {'loss': 5.413611857096354}\n",
      "epoch 3, batch 1750, logs: {'loss': 5.39150439453125}\n",
      "epoch 3, batch 1760, logs: {'loss': 5.385413067157452}\n",
      "epoch 3, batch 1770, logs: {'loss': 5.396253345630787}\n",
      "epoch 3, batch 1780, logs: {'loss': 5.410441371372768}\n",
      "epoch 3, batch 1790, logs: {'loss': 5.416368787042026}\n",
      "epoch 3, batch 1800, logs: {'loss': 5.405116780598958}\n",
      "epoch 3, batch 1810, logs: {'loss': 5.42603011592742}\n",
      "finished iterating over dataset, total batches: 1818\n",
      "epoch 3, batch 1820, logs: {'loss': 5.41859016418457}\n",
      "epoch 3, batch 1830, logs: {'loss': 5.414044004498106}\n",
      "epoch 3, batch 1840, logs: {'loss': 5.393081844554228}\n",
      "epoch 3, batch 1850, logs: {'loss': 5.387955147879464}\n",
      "epoch 3, batch 1860, logs: {'loss': 5.376393296983507}\n",
      "epoch 3, batch 1870, logs: {'loss': 5.364376913534628}\n",
      "epoch 3, batch 1880, logs: {'loss': 5.350356573807566}\n",
      "epoch 3, batch 1890, logs: {'loss': 5.340953650841346}\n",
      "epoch 3, batch 1900, logs: {'loss': 5.328906860351562}\n",
      "epoch 3, batch 1910, logs: {'loss': 5.317940763147866}\n",
      "epoch 3, batch 1920, logs: {'loss': 5.314039248511905}\n",
      "epoch 3, batch 1930, logs: {'loss': 5.30978152252907}\n",
      "epoch 3, batch 1940, logs: {'loss': 5.308648681640625}\n",
      "epoch 3, batch 1950, logs: {'loss': 5.310753038194444}\n",
      "epoch 3, batch 1960, logs: {'loss': 5.303946586277174}\n",
      "epoch 3, batch 1970, logs: {'loss': 5.298045316655585}\n",
      "epoch 3, batch 1980, logs: {'loss': 5.297257995605468}\n",
      "epoch 3, batch 1990, logs: {'loss': 5.289710718271683}\n",
      "epoch 3, batch 2000, logs: {'loss': 5.28811474609375}\n",
      "epoch 4, batch 2010, logs: {'loss': 4.828479385375976}\n",
      "epoch 4, batch 2020, logs: {'loss': 5.013887786865235}\n",
      "epoch 4, batch 2030, logs: {'loss': 4.926529947916666}\n",
      "epoch 4, batch 2040, logs: {'loss': 4.980269241333008}\n",
      "epoch 4, batch 2050, logs: {'loss': 5.0477056884765625}\n",
      "epoch 4, batch 2060, logs: {'loss': 5.0502685546875}\n",
      "epoch 4, batch 2070, logs: {'loss': 5.057455008370535}\n",
      "epoch 4, batch 2080, logs: {'loss': 5.040332412719726}\n",
      "epoch 4, batch 2090, logs: {'loss': 5.029476928710937}\n",
      "epoch 4, batch 2100, logs: {'loss': 5.042489929199219}\n",
      "epoch 4, batch 2110, logs: {'loss': 5.072971968217329}\n",
      "epoch 4, batch 2120, logs: {'loss': 5.056130981445312}\n",
      "epoch 4, batch 2130, logs: {'loss': 5.04781728891226}\n",
      "epoch 4, batch 2140, logs: {'loss': 5.033146449497768}\n",
      "epoch 4, batch 2150, logs: {'loss': 5.020926513671875}\n",
      "epoch 4, batch 2160, logs: {'loss': 5.019388198852539}\n",
      "epoch 4, batch 2170, logs: {'loss': 4.995489142922795}\n",
      "epoch 4, batch 2180, logs: {'loss': 5.010884263780382}\n",
      "epoch 4, batch 2190, logs: {'loss': 5.027041786595395}\n",
      "epoch 4, batch 2200, logs: {'loss': 5.0182763671875}\n",
      "epoch 4, batch 2210, logs: {'loss': 5.031708635602679}\n",
      "epoch 4, batch 2220, logs: {'loss': 5.037876753373579}\n",
      "epoch 4, batch 2230, logs: {'loss': 5.032146951426631}\n",
      "epoch 4, batch 2240, logs: {'loss': 5.019737752278646}\n",
      "epoch 4, batch 2250, logs: {'loss': 5.01762939453125}\n",
      "epoch 4, batch 2260, logs: {'loss': 5.0189767690805285}\n",
      "epoch 4, batch 2270, logs: {'loss': 5.02175880714699}\n",
      "epoch 4, batch 2280, logs: {'loss': 5.023746163504464}\n",
      "epoch 4, batch 2290, logs: {'loss': 5.018704960264008}\n",
      "epoch 4, batch 2300, logs: {'loss': 5.022516276041666}\n",
      "epoch 4, batch 2310, logs: {'loss': 5.018462544102823}\n",
      "epoch 4, batch 2320, logs: {'loss': 5.031297302246093}\n",
      "epoch 4, batch 2330, logs: {'loss': 5.034445652817235}\n",
      "epoch 4, batch 2340, logs: {'loss': 5.0501920812270225}\n",
      "epoch 4, batch 2350, logs: {'loss': 5.059976981026786}\n",
      "epoch 4, batch 2360, logs: {'loss': 5.0404039171006945}\n",
      "epoch 4, batch 2370, logs: {'loss': 5.031560454497466}\n",
      "epoch 4, batch 2380, logs: {'loss': 5.038329756887335}\n",
      "epoch 4, batch 2390, logs: {'loss': 5.0411868364383015}\n",
      "epoch 4, batch 2400, logs: {'loss': 5.03660888671875}\n",
      "epoch 4, batch 2410, logs: {'loss': 5.03712783441311}\n",
      "epoch 4, batch 2420, logs: {'loss': 5.037366885230655}\n",
      "finished iterating over dataset, total batches: 2424\n",
      "epoch 4, batch 2430, logs: {'loss': 5.032849972747093}\n",
      "epoch 4, batch 2440, logs: {'loss': 5.027726606889204}\n",
      "epoch 4, batch 2450, logs: {'loss': 5.020534939236111}\n",
      "epoch 4, batch 2460, logs: {'loss': 5.011078676970109}\n",
      "epoch 4, batch 2470, logs: {'loss': 5.004791909075798}\n",
      "epoch 4, batch 2480, logs: {'loss': 4.997576904296875}\n",
      "epoch 4, batch 2490, logs: {'loss': 4.991196488360969}\n",
      "epoch 4, batch 2500, logs: {'loss': 4.9899013671875}\n",
      "epoch 5, batch 2510, logs: {'loss': 4.7241065979003904}\n",
      "epoch 5, batch 2520, logs: {'loss': 4.554470062255859}\n",
      "epoch 5, batch 2530, logs: {'loss': 4.524687703450521}\n",
      "epoch 5, batch 2540, logs: {'loss': 4.507642364501953}\n",
      "epoch 5, batch 2550, logs: {'loss': 4.543350830078125}\n",
      "epoch 5, batch 2560, logs: {'loss': 4.56037343343099}\n",
      "epoch 5, batch 2570, logs: {'loss': 4.591670445033482}\n",
      "epoch 5, batch 2580, logs: {'loss': 4.591893005371094}\n",
      "epoch 5, batch 2590, logs: {'loss': 4.577385118272569}\n",
      "epoch 5, batch 2600, logs: {'loss': 4.573631286621094}\n",
      "epoch 5, batch 2610, logs: {'loss': 4.595783857865767}\n",
      "epoch 5, batch 2620, logs: {'loss': 4.60434315999349}\n",
      "epoch 5, batch 2630, logs: {'loss': 4.617247596153846}\n",
      "epoch 5, batch 2640, logs: {'loss': 4.616924176897322}\n",
      "epoch 5, batch 2650, logs: {'loss': 4.632891438802083}\n",
      "epoch 5, batch 2660, logs: {'loss': 4.621212768554687}\n",
      "epoch 5, batch 2670, logs: {'loss': 4.615102610868566}\n",
      "epoch 5, batch 2680, logs: {'loss': 4.645652262369792}\n",
      "epoch 5, batch 2690, logs: {'loss': 4.653626451994244}\n",
      "epoch 5, batch 2700, logs: {'loss': 4.658792419433594}\n",
      "epoch 5, batch 2710, logs: {'loss': 4.659545317150298}\n",
      "epoch 5, batch 2720, logs: {'loss': 4.6568370472301135}\n",
      "epoch 5, batch 2730, logs: {'loss': 4.6589562457540765}\n",
      "epoch 5, batch 2740, logs: {'loss': 4.654863993326823}\n",
      "epoch 5, batch 2750, logs: {'loss': 4.6575576171875}\n",
      "epoch 5, batch 2760, logs: {'loss': 4.667027400090144}\n",
      "epoch 5, batch 2770, logs: {'loss': 4.671079734519676}\n",
      "epoch 5, batch 2780, logs: {'loss': 4.67792489188058}\n",
      "epoch 5, batch 2790, logs: {'loss': 4.681310614224138}\n",
      "epoch 5, batch 2800, logs: {'loss': 4.67395263671875}\n",
      "epoch 5, batch 2810, logs: {'loss': 4.673959252142137}\n",
      "epoch 5, batch 2820, logs: {'loss': 4.669622802734375}\n",
      "epoch 5, batch 2830, logs: {'loss': 4.6736779415246215}\n",
      "epoch 5, batch 2840, logs: {'loss': 4.666538373161765}\n",
      "epoch 5, batch 2850, logs: {'loss': 4.66440185546875}\n",
      "epoch 5, batch 2860, logs: {'loss': 4.658865356445313}\n",
      "epoch 5, batch 2870, logs: {'loss': 4.661511890308277}\n",
      "epoch 5, batch 2880, logs: {'loss': 4.653396124588816}\n",
      "epoch 5, batch 2890, logs: {'loss': 4.649948980869391}\n",
      "epoch 5, batch 2900, logs: {'loss': 4.656160583496094}\n",
      "epoch 5, batch 2910, logs: {'loss': 4.656459901391006}\n",
      "epoch 5, batch 2920, logs: {'loss': 4.657967994326637}\n",
      "epoch 5, batch 2930, logs: {'loss': 4.658870821220931}\n",
      "epoch 5, batch 2940, logs: {'loss': 4.6600880015980115}\n",
      "epoch 5, batch 2950, logs: {'loss': 4.656319986979167}\n",
      "epoch 5, batch 2960, logs: {'loss': 4.649624235733696}\n",
      "epoch 5, batch 2970, logs: {'loss': 4.653817943816489}\n",
      "epoch 5, batch 2980, logs: {'loss': 4.656601969401041}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 5, batch 2990, logs: {'loss': 4.658330676020408}\n",
      "epoch 5, batch 3000, logs: {'loss': 4.66214404296875}\n",
      "epoch 6, batch 3010, logs: {'loss': 4.239145278930664}\n",
      "epoch 6, batch 3020, logs: {'loss': 4.536834335327148}\n",
      "epoch 6, batch 3030, logs: {'loss': 4.591202799479166}\n",
      "finished iterating over dataset, total batches: 3030\n",
      "epoch 6, batch 3040, logs: {'loss': 4.4915412902832035}\n",
      "epoch 6, batch 3050, logs: {'loss': 4.346260986328125}\n",
      "epoch 6, batch 3060, logs: {'loss': 4.322457377115885}\n",
      "epoch 6, batch 3070, logs: {'loss': 4.273846435546875}\n",
      "epoch 6, batch 3080, logs: {'loss': 4.261737823486328}\n",
      "epoch 6, batch 3090, logs: {'loss': 4.247124565972222}\n",
      "epoch 6, batch 3100, logs: {'loss': 4.255319213867187}\n",
      "epoch 6, batch 3110, logs: {'loss': 4.243368530273438}\n",
      "epoch 6, batch 3120, logs: {'loss': 4.2493441263834635}\n",
      "epoch 6, batch 3130, logs: {'loss': 4.227216045673077}\n",
      "epoch 6, batch 3140, logs: {'loss': 4.212058367047991}\n",
      "epoch 6, batch 3150, logs: {'loss': 4.183502604166667}\n",
      "epoch 6, batch 3160, logs: {'loss': 4.1920318603515625}\n",
      "epoch 6, batch 3170, logs: {'loss': 4.208134550206801}\n",
      "epoch 6, batch 3180, logs: {'loss': 4.18472663031684}\n",
      "epoch 6, batch 3190, logs: {'loss': 4.181661505448191}\n",
      "epoch 6, batch 3200, logs: {'loss': 4.1967724609375}\n",
      "epoch 6, batch 3210, logs: {'loss': 4.199772135416667}\n",
      "epoch 6, batch 3220, logs: {'loss': 4.193987759676847}\n",
      "epoch 6, batch 3230, logs: {'loss': 4.185785973590353}\n",
      "epoch 6, batch 3240, logs: {'loss': 4.178279113769531}\n",
      "epoch 6, batch 3250, logs: {'loss': 4.18630029296875}\n",
      "epoch 6, batch 3260, logs: {'loss': 4.183508770282452}\n",
      "epoch 6, batch 3270, logs: {'loss': 4.189479347511574}\n",
      "epoch 6, batch 3280, logs: {'loss': 4.188921247209821}\n",
      "epoch 6, batch 3290, logs: {'loss': 4.192512038658405}\n",
      "epoch 6, batch 3300, logs: {'loss': 4.191649983723958}\n",
      "epoch 6, batch 3310, logs: {'loss': 4.200390625}\n",
      "epoch 6, batch 3320, logs: {'loss': 4.203249740600586}\n",
      "epoch 6, batch 3330, logs: {'loss': 4.2031124230587125}\n",
      "epoch 6, batch 3340, logs: {'loss': 4.208502556295956}\n",
      "epoch 6, batch 3350, logs: {'loss': 4.205703473772322}\n",
      "epoch 6, batch 3360, logs: {'loss': 4.202362399631077}\n",
      "epoch 6, batch 3370, logs: {'loss': 4.198964381862331}\n",
      "epoch 6, batch 3380, logs: {'loss': 4.197171502364309}\n",
      "epoch 6, batch 3390, logs: {'loss': 4.1899792793469555}\n",
      "epoch 6, batch 3400, logs: {'loss': 4.195765686035156}\n",
      "epoch 6, batch 3410, logs: {'loss': 4.198982945884146}\n",
      "epoch 6, batch 3420, logs: {'loss': 4.2022580101376485}\n",
      "epoch 6, batch 3430, logs: {'loss': 4.202132823855378}\n",
      "epoch 6, batch 3440, logs: {'loss': 4.199863225763495}\n",
      "epoch 6, batch 3450, logs: {'loss': 4.200490993923611}\n",
      "epoch 6, batch 3460, logs: {'loss': 4.197726307744565}\n",
      "epoch 6, batch 3470, logs: {'loss': 4.186931983460771}\n",
      "epoch 6, batch 3480, logs: {'loss': 4.186116027832031}\n",
      "epoch 6, batch 3490, logs: {'loss': 4.186215521364796}\n",
      "epoch 6, batch 3500, logs: {'loss': 4.18057568359375}\n",
      "epoch 7, batch 3510, logs: {'loss': 3.945664978027344}\n",
      "epoch 7, batch 3520, logs: {'loss': 3.9121349334716795}\n",
      "epoch 7, batch 3530, logs: {'loss': 3.962365976969401}\n",
      "epoch 7, batch 3540, logs: {'loss': 3.946522903442383}\n",
      "epoch 7, batch 3550, logs: {'loss': 4.027789306640625}\n",
      "epoch 7, batch 3560, logs: {'loss': 4.039889017740886}\n",
      "epoch 7, batch 3570, logs: {'loss': 4.070957728794643}\n",
      "epoch 7, batch 3580, logs: {'loss': 4.085488128662109}\n",
      "epoch 7, batch 3590, logs: {'loss': 4.078685167100694}\n",
      "epoch 7, batch 3600, logs: {'loss': 4.111568298339844}\n",
      "epoch 7, batch 3610, logs: {'loss': 4.109257368607954}\n",
      "epoch 7, batch 3620, logs: {'loss': 4.084319814046224}\n",
      "epoch 7, batch 3630, logs: {'loss': 4.119368802584135}\n",
      "finished iterating over dataset, total batches: 3636\n",
      "epoch 7, batch 3640, logs: {'loss': 4.113755580357143}\n",
      "epoch 7, batch 3650, logs: {'loss': 4.072440185546875}\n",
      "epoch 7, batch 3660, logs: {'loss': 4.061788558959961}\n",
      "epoch 7, batch 3670, logs: {'loss': 4.015697524126838}\n",
      "epoch 7, batch 3680, logs: {'loss': 3.9976437038845485}\n",
      "epoch 7, batch 3690, logs: {'loss': 3.9889391447368423}\n",
      "epoch 7, batch 3700, logs: {'loss': 3.97056884765625}\n",
      "epoch 7, batch 3710, logs: {'loss': 3.9642342703683036}\n",
      "epoch 7, batch 3720, logs: {'loss': 3.946466064453125}\n",
      "epoch 7, batch 3730, logs: {'loss': 3.9397957179857337}\n",
      "epoch 7, batch 3740, logs: {'loss': 3.9217313130696616}\n",
      "epoch 7, batch 3750, logs: {'loss': 3.91012548828125}\n",
      "epoch 7, batch 3760, logs: {'loss': 3.8952970064603365}\n",
      "epoch 7, batch 3770, logs: {'loss': 3.8772659866898147}\n",
      "epoch 7, batch 3780, logs: {'loss': 3.869171578543527}\n",
      "epoch 7, batch 3790, logs: {'loss': 3.8595694706357757}\n",
      "epoch 7, batch 3800, logs: {'loss': 3.853826904296875}\n",
      "epoch 7, batch 3810, logs: {'loss': 3.84664306640625}\n",
      "epoch 7, batch 3820, logs: {'loss': 3.8367233276367188}\n",
      "epoch 7, batch 3830, logs: {'loss': 3.8344730261600377}\n",
      "epoch 7, batch 3840, logs: {'loss': 3.8261115579044116}\n",
      "epoch 7, batch 3850, logs: {'loss': 3.820459681919643}\n",
      "epoch 7, batch 3860, logs: {'loss': 3.816750081380208}\n",
      "epoch 7, batch 3870, logs: {'loss': 3.8095010293496623}\n",
      "epoch 7, batch 3880, logs: {'loss': 3.8028127569901318}\n",
      "epoch 7, batch 3890, logs: {'loss': 3.7967147435897437}\n",
      "epoch 7, batch 3900, logs: {'loss': 3.798703308105469}\n",
      "epoch 7, batch 3910, logs: {'loss': 3.7966275843178354}\n",
      "epoch 7, batch 3920, logs: {'loss': 3.796580868675595}\n",
      "epoch 7, batch 3930, logs: {'loss': 3.789325093114099}\n",
      "epoch 7, batch 3940, logs: {'loss': 3.790911865234375}\n",
      "epoch 7, batch 3950, logs: {'loss': 3.793628200954861}\n",
      "epoch 7, batch 3960, logs: {'loss': 3.787653384001359}\n",
      "epoch 7, batch 3970, logs: {'loss': 3.7870078228889628}\n",
      "epoch 7, batch 3980, logs: {'loss': 3.7834312438964846}\n",
      "epoch 7, batch 3990, logs: {'loss': 3.7856975944674747}\n",
      "epoch 7, batch 4000, logs: {'loss': 3.789970947265625}\n",
      "epoch 8, batch 4010, logs: {'loss': 3.7591156005859374}\n",
      "epoch 8, batch 4020, logs: {'loss': 3.7438480377197267}\n",
      "epoch 8, batch 4030, logs: {'loss': 3.68275883992513}\n",
      "epoch 8, batch 4040, logs: {'loss': 3.718663787841797}\n",
      "epoch 8, batch 4050, logs: {'loss': 3.69032958984375}\n",
      "epoch 8, batch 4060, logs: {'loss': 3.6849517822265625}\n",
      "epoch 8, batch 4070, logs: {'loss': 3.6846675327845984}\n",
      "epoch 8, batch 4080, logs: {'loss': 3.6852115631103515}\n",
      "epoch 8, batch 4090, logs: {'loss': 3.688126627604167}\n",
      "epoch 8, batch 4100, logs: {'loss': 3.6940228271484377}\n",
      "epoch 8, batch 4110, logs: {'loss': 3.6992162531072443}\n",
      "epoch 8, batch 4120, logs: {'loss': 3.672399393717448}\n",
      "epoch 8, batch 4130, logs: {'loss': 3.679418006310096}\n",
      "epoch 8, batch 4140, logs: {'loss': 3.6598096575055803}\n",
      "epoch 8, batch 4150, logs: {'loss': 3.6525480143229165}\n",
      "epoch 8, batch 4160, logs: {'loss': 3.6427589416503907}\n",
      "epoch 8, batch 4170, logs: {'loss': 3.6523828843060664}\n",
      "epoch 8, batch 4180, logs: {'loss': 3.6692179361979167}\n",
      "epoch 8, batch 4190, logs: {'loss': 3.6709646124588815}\n",
      "epoch 8, batch 4200, logs: {'loss': 3.65715576171875}\n",
      "epoch 8, batch 4210, logs: {'loss': 3.666531226748512}\n",
      "epoch 8, batch 4220, logs: {'loss': 3.6621975985440343}\n",
      "epoch 8, batch 4230, logs: {'loss': 3.664018448539402}\n",
      "epoch 8, batch 4240, logs: {'loss': 3.6697845458984375}\n",
      "finished iterating over dataset, total batches: 4242\n",
      "epoch 8, batch 4250, logs: {'loss': 3.65382666015625}\n",
      "epoch 8, batch 4260, logs: {'loss': 3.6285670353816104}\n",
      "epoch 8, batch 4270, logs: {'loss': 3.6088462547019677}\n",
      "epoch 8, batch 4280, logs: {'loss': 3.5886117117745537}\n",
      "epoch 8, batch 4290, logs: {'loss': 3.5742886247306034}\n",
      "epoch 8, batch 4300, logs: {'loss': 3.5534053548177083}\n",
      "epoch 8, batch 4310, logs: {'loss': 3.5410983177923385}\n",
      "epoch 8, batch 4320, logs: {'loss': 3.5335834503173826}\n",
      "epoch 8, batch 4330, logs: {'loss': 3.514056581439394}\n",
      "epoch 8, batch 4340, logs: {'loss': 3.4996416877297793}\n",
      "epoch 8, batch 4350, logs: {'loss': 3.484981166294643}\n",
      "epoch 8, batch 4360, logs: {'loss': 3.4793294270833335}\n",
      "epoch 8, batch 4370, logs: {'loss': 3.479773609058277}\n",
      "epoch 8, batch 4380, logs: {'loss': 3.4684207313939144}\n",
      "epoch 8, batch 4390, logs: {'loss': 3.4722837790464745}\n",
      "epoch 8, batch 4400, logs: {'loss': 3.4638934326171875}\n",
      "epoch 8, batch 4410, logs: {'loss': 3.4496754716082316}\n",
      "epoch 8, batch 4420, logs: {'loss': 3.44166986374628}\n",
      "epoch 8, batch 4430, logs: {'loss': 3.435649641170058}\n",
      "epoch 8, batch 4440, logs: {'loss': 3.4320778586647727}\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch 8, batch 4450, logs: {'loss': 3.4294694010416666}\n",
      "epoch 8, batch 4460, logs: {'loss': 3.4254654594089673}\n",
      "epoch 8, batch 4470, logs: {'loss': 3.4163789789727392}\n",
      "epoch 8, batch 4480, logs: {'loss': 3.4081581115722654}\n",
      "epoch 8, batch 4490, logs: {'loss': 3.3993806800063777}\n",
      "epoch 8, batch 4500, logs: {'loss': 3.389989501953125}\n",
      "epoch 9, batch 4510, logs: {'loss': 3.1116186141967774}\n",
      "epoch 9, batch 4520, logs: {'loss': 3.217171478271484}\n",
      "epoch 9, batch 4530, logs: {'loss': 3.205359141031901}\n",
      "epoch 9, batch 4540, logs: {'loss': 3.128487968444824}\n",
      "epoch 9, batch 4550, logs: {'loss': 3.1015313720703124}\n",
      "epoch 9, batch 4560, logs: {'loss': 3.0595672607421873}\n",
      "epoch 9, batch 4570, logs: {'loss': 3.06434326171875}\n",
      "epoch 9, batch 4580, logs: {'loss': 3.0796680450439453}\n",
      "epoch 9, batch 4590, logs: {'loss': 3.0890801323784722}\n",
      "epoch 9, batch 4600, logs: {'loss': 3.0751828002929686}\n",
      "epoch 9, batch 4610, logs: {'loss': 3.099008733575994}\n",
      "epoch 9, batch 4620, logs: {'loss': 3.0973670959472654}\n",
      "epoch 9, batch 4630, logs: {'loss': 3.1193885216346153}\n",
      "epoch 9, batch 4640, logs: {'loss': 3.1258313860212055}\n",
      "epoch 9, batch 4650, logs: {'loss': 3.153620402018229}\n",
      "epoch 9, batch 4660, logs: {'loss': 3.1771371841430662}\n",
      "epoch 9, batch 4670, logs: {'loss': 3.2019613826976103}\n",
      "epoch 9, batch 4680, logs: {'loss': 3.2136888292100694}\n",
      "epoch 9, batch 4690, logs: {'loss': 3.2223549290707236}\n",
      "epoch 9, batch 4700, logs: {'loss': 3.2152163696289064}\n",
      "epoch 9, batch 4710, logs: {'loss': 3.215897623697917}\n",
      "epoch 9, batch 4720, logs: {'loss': 3.2089000355113635}\n",
      "epoch 9, batch 4730, logs: {'loss': 3.2106076447860055}\n",
      "epoch 9, batch 4740, logs: {'loss': 3.210120137532552}\n",
      "epoch 9, batch 4750, logs: {'loss': 3.20980029296875}\n",
      "epoch 9, batch 4760, logs: {'loss': 3.207448167067308}\n",
      "epoch 9, batch 4770, logs: {'loss': 3.2136485912181714}\n",
      "epoch 9, batch 4780, logs: {'loss': 3.207135009765625}\n",
      "epoch 9, batch 4790, logs: {'loss': 3.202184637661638}\n",
      "epoch 9, batch 4800, logs: {'loss': 3.20490966796875}\n",
      "epoch 9, batch 4810, logs: {'loss': 3.197852743825605}\n",
      "epoch 9, batch 4820, logs: {'loss': 3.201498794555664}\n",
      "epoch 9, batch 4830, logs: {'loss': 3.1991477272727273}\n",
      "epoch 9, batch 4840, logs: {'loss': 3.199991383272059}\n",
      "finished iterating over dataset, total batches: 4848\n",
      "epoch 9, batch 4850, logs: {'loss': 3.1988821847098214}\n",
      "epoch 9, batch 4860, logs: {'loss': 3.1878604465060763}\n",
      "epoch 9, batch 4870, logs: {'loss': 3.1688334697001688}\n",
      "epoch 9, batch 4880, logs: {'loss': 3.147487921463816}\n",
      "epoch 9, batch 4890, logs: {'loss': 3.128184157151442}\n",
      "epoch 9, batch 4900, logs: {'loss': 3.1082244873046876}\n",
      "epoch 9, batch 4910, logs: {'loss': 3.0962625643102135}\n",
      "epoch 9, batch 4920, logs: {'loss': 3.0847202845982142}\n",
      "epoch 9, batch 4930, logs: {'loss': 3.0724510015443314}\n",
      "epoch 9, batch 4940, logs: {'loss': 3.0608725807883523}\n",
      "epoch 9, batch 4950, logs: {'loss': 3.0505558268229165}\n",
      "epoch 9, batch 4960, logs: {'loss': 3.042774764351223}\n",
      "epoch 9, batch 4970, logs: {'loss': 3.035817767204122}\n",
      "epoch 9, batch 4980, logs: {'loss': 3.02915522257487}\n",
      "epoch 9, batch 4990, logs: {'loss': 3.019081582828444}\n",
      "epoch 9, batch 5000, logs: {'loss': 3.008167724609375}\n",
      "finished iterating over dataset, total batches: 5000\n"
     ]
    }
   ],
   "source": [
    "# Define the model and train it\n",
    "# You need to be quite patient, since the model has a lot of params\n",
    "import tensorflow_datasets as tfds\n",
    "from transformers import BertTokenizer\n",
    "from headliner.model.bert_summarizer import BertSummarizer\n",
    "from headliner.preprocessing.bert_vectorizer import BertVectorizer\n",
    "from headliner.trainer import Trainer\n",
    "\n",
    "# Use a pre-trained BERT embedding and BERT tokenizer for the encoder \n",
    "tokenizer_input = BertTokenizer.from_pretrained('bert-base-uncased')\n",
    "tokenizer_target = tfds.features.text.SubwordTextEncoder.build_from_corpus(\n",
    "    targets_prep, target_vocab_size=2**13,  reserved_tokens=[preprocessor.start_token, preprocessor.end_token])\n",
    "\n",
    "\n",
    "vectorizer = BertVectorizer(tokenizer_input, tokenizer_target)\n",
    "\n",
    "#The parameters of the model\n",
    "summarizer = BertSummarizer(num_heads=2,\n",
    "                            feed_forward_dim=512,\n",
    "                            num_layers_encoder=0,\n",
    "                            num_layers_decoder=4,\n",
    "                            bert_embedding_encoder='bert-base-uncased',\n",
    "                            embedding_size_encoder=768,\n",
    "                            embedding_size_decoder=768,\n",
    "                            dropout_rate=0.1,\n",
    "                            max_prediction_len=80)\n",
    "summarizer.init_model(preprocessor, vectorizer)\n",
    "\n",
    "trainer = Trainer(batch_size=2)\n",
    "trainer.train(summarizer, train_data, num_epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RvU9hQhqS0zZ"
   },
   "source": [
    "### Test the model\n",
    "\n",
    "Let's test the model with some data from our testing set.\n",
    "\n",
    "We will check the ground truth titles with the generated ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zPqsFo9lhbdH"
   },
   "outputs": [],
   "source": [
    "#select some samples from the test set\n",
    "sample_test_set=test_df[:10]\n",
    "sample_test_set = sample_test_set.reset_index(drop=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "mwur8q6PUUxO",
    "outputId": "4b8458d5-8d11-412c-90a4-0017196ee959",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t: Positive definite matrices and the S-divergence\n",
      "p: The approximation of the smoothing problem [SEP]\n",
      "\n",
      "t: Anasazi software for the numerical solution of large-scale eigenvalue problems\n",
      "p: Algorithm 782: codes for the solution of large-scale eigenvalue problems [SEP]\n",
      "\n",
      "t: Approximate Qualitative Temporal Reasoning\n",
      "p: Comparing multiagent systems and applications [SEP]\n",
      "\n",
      "t: Towards Mathematical Knowledge Management for Electrical Engineering\n",
      "p: A new approach for the Complexity of Mathematical Knowledge [SEP]\n",
      "\n",
      "t: On the level sets of a distance function in a Minkowski space\n",
      "p: On a Convex Optimization [SEP]\n",
      "\n",
      "t: The complexity of Markov decision processes\n",
      "p: The complexity of a Markov Decision Processes [SEP]\n",
      "\n",
      "t: Network-Based Drug Ranking and Repositioning with Respect to DrugBank Therapeutic Categories\n",
      "p: A new approach for the Classification of Brain -based based based on Protein Interaction [SEP]\n",
      "\n",
      "t: Mining association rules among gene functions in clusters of similar gene expression maps\n",
      "p: Gene Expression Analysis of Gene Expression Data [SEP]\n",
      "\n",
      "t: Mining gene expression data focusing cancer therapeutics: a digest\n",
      "p: Using Classification and Gene Expression Data Using Classification [SEP]\n",
      "\n",
      "t: Selecting informative genes by Lasso and Dantzig selector for linear classifiers\n",
      "p: Using Classification and Gene Expression Data Using Gene Expression Data [SEP]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "#10 epochs - batch=2\n",
    "for i in range(0, len(sample_test_set['abstract'])):\n",
    "    #predict the title given the abstract\n",
    "    prediction = summarizer.predict(sample_test_set['abstract'][i])\n",
    "\n",
    "    #print the ground-truth title  \n",
    "    print('t:',sample_test_set['title'][i])\n",
    "    #print the predicted title\n",
    "    print('p:',prediction)\n",
    "    print('')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_FjpmLqdUqYD"
   },
   "source": [
    "## ToDo Task"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wqDza6c-Uw2L"
   },
   "source": [
    "\n",
    "\n",
    "\n",
    "\n",
    "1.   Test the model using geoscience data used in the previous tutorials.\n",
    "  1.   Read the abstracts into a list. \n",
    "  2.   Apply the testing part of this tutorial on this list.\n",
    "2.   Change some parameters of the model, like the epochs number and the batch size, and see what is the impact of these parameters.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8l6NCodAUvbg"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "zSeKealuUtBv"
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NeS4yMMl1FUG"
   },
   "source": [
    "## References:\n",
    "*   https://as-ideas.github.io/headliner/\n",
    "*   https://medium.com/axel-springer-tech/headliner-easy-training-and-deployment-of-seq2seq-models-2a26508b4dae\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y0zd8LDt1TWL"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "Day3_Task2_ title_generator.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0947c71b077c4d77b11a0695550ec7ba": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "1339875a08794aff990637693e5f4f90": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "2c2f8562ac1f46e7a4ba82fd1ff5f06d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_cf37738d358b4819aeb945ec5a69e51f",
       "IPY_MODEL_8f07544e21a5484483862beb82ec0c86"
      ],
      "layout": "IPY_MODEL_fc572a59965144d39bfc21696bb757ef"
     }
    },
    "31cbfd36a52b4638b2677745d16812da": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "369a8fdc81ba44e4b178992e4ed34877": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3898c7f1337e4ca4bbde3b9298d9f265": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a6074d382cb44903afaa9592e438ee14",
      "max": 570,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_699a541fa0bd4ad2a424523aa7662fc7",
      "value": 570
     }
    },
    "3d12a086bd9e4be7af26564078debb2c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3f712265be4a49779631f1c4ebb52f63": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "41a4ec5f76de458383024422a5b1faf2": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "4b376439dca649bfa0ab11df63b8466a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "596471dd7146450ca539376e9d72b747": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5ac53ba4abbb4da99929367a4ed0af10": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "5c830657c2e44251a0b62011542780a9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_a163976c22b8469d9112306a4e82580c",
      "placeholder": "​",
      "style": "IPY_MODEL_b1eb8aaf03ed437bb75f254796938539",
      "value": " 466k/466k [00:00&lt;00:00, 1.85MB/s]"
     }
    },
    "699a541fa0bd4ad2a424523aa7662fc7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "756cb78309be4aee9a87c5ffef3ffc97": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "775f74cb728944588f527d8ffd6ac7a9": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "80de91ca85ef4e48aa2e9f720c53d2ea": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_0947c71b077c4d77b11a0695550ec7ba",
      "max": 536063208,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_d405946ea39c4ddc859218dac8c75170",
      "value": 536063208
     }
    },
    "8c67f0909f0742e08957e52ca206c734": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8d99a96aca3c4140aa83332a8fa38dd7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_c0c4bc0029c64ec6ab5996b2fffb054a",
       "IPY_MODEL_e2c021a7fe0940928b633e4bffe10992"
      ],
      "layout": "IPY_MODEL_756cb78309be4aee9a87c5ffef3ffc97"
     }
    },
    "8f07544e21a5484483862beb82ec0c86": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_5ac53ba4abbb4da99929367a4ed0af10",
      "placeholder": "​",
      "style": "IPY_MODEL_e720c2d40abf44f8960f66d44e157303",
      "value": " 28.0/28.0 [00:00&lt;00:00, 35.0B/s]"
     }
    },
    "935cf044b05a420f8801c0ac73e83d74": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c10a7f750ed64e7cab0d164576225996",
      "placeholder": "​",
      "style": "IPY_MODEL_3f712265be4a49779631f1c4ebb52f63",
      "value": " 570/570 [00:00&lt;00:00, 654B/s]"
     }
    },
    "98ceb4fbac9040e48d311068725a1765": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_d580b5669bd0404f974792321d61361b",
       "IPY_MODEL_5c830657c2e44251a0b62011542780a9"
      ],
      "layout": "IPY_MODEL_9dc605935f8c480b8bdadcbdc2a43f67"
     }
    },
    "98ec4e207c474fb89aa5f59c9588733e": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_80de91ca85ef4e48aa2e9f720c53d2ea",
       "IPY_MODEL_a883f8f7718245a087e9298d20243078"
      ],
      "layout": "IPY_MODEL_8c67f0909f0742e08957e52ca206c734"
     }
    },
    "9dc605935f8c480b8bdadcbdc2a43f67": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9fd9edb1271949cb8064fe09bd801303": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_3898c7f1337e4ca4bbde3b9298d9f265",
       "IPY_MODEL_935cf044b05a420f8801c0ac73e83d74"
      ],
      "layout": "IPY_MODEL_775f74cb728944588f527d8ffd6ac7a9"
     }
    },
    "a163976c22b8469d9112306a4e82580c": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a6074d382cb44903afaa9592e438ee14": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a883f8f7718245a087e9298d20243078": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_3d12a086bd9e4be7af26564078debb2c",
      "placeholder": "​",
      "style": "IPY_MODEL_bad195eb9a22490caf2853ea14c0529b",
      "value": " 536M/536M [00:16&lt;00:00, 32.5MB/s]"
     }
    },
    "b1eb8aaf03ed437bb75f254796938539": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "b3aeb42c860d4c7c83ae9c7c8a5328f5": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bad195eb9a22490caf2853ea14c0529b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c0c4bc0029c64ec6ab5996b2fffb054a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_596471dd7146450ca539376e9d72b747",
      "max": 231508,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_cc47c8493efc4ab7805a05c6ffb69a1f",
      "value": 231508
     }
    },
    "c10a7f750ed64e7cab0d164576225996": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "cc47c8493efc4ab7805a05c6ffb69a1f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "cf37738d358b4819aeb945ec5a69e51f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_b3aeb42c860d4c7c83ae9c7c8a5328f5",
      "max": 28,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_4b376439dca649bfa0ab11df63b8466a",
      "value": 28
     }
    },
    "d405946ea39c4ddc859218dac8c75170": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "d580b5669bd0404f974792321d61361b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Downloading: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_41a4ec5f76de458383024422a5b1faf2",
      "max": 466062,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_31cbfd36a52b4638b2677745d16812da",
      "value": 466062
     }
    },
    "e2c021a7fe0940928b633e4bffe10992": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_1339875a08794aff990637693e5f4f90",
      "placeholder": "​",
      "style": "IPY_MODEL_369a8fdc81ba44e4b178992e4ed34877",
      "value": " 232k/232k [00:02&lt;00:00, 87.5kB/s]"
     }
    },
    "e720c2d40abf44f8960f66d44e157303": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "fc572a59965144d39bfc21696bb757ef": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}