test-with-quantized-openllm-ro-model.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f8659a38",
   "metadata": {
    "papermill": {
     "duration": 0.00533,
     "end_time": "2024-05-19T20:51:08.593002",
     "exception": false,
     "start_time": "2024-05-19T20:51:08.587672",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Introduction\n",
    "\n",
    "OpenLLM-Ro is a new initiative to develop a Romanian LLM, based on existing large language models. The first version is based on Llama2, fine-tuned with a corpus of Romanian language sources data.\n",
    "\n",
    "For more details about the project, check this link: https://ilds.ro/llm-for-romanian/\n",
    "        \n",
    "## The current model\n",
    "\n",
    "The model we are using here was obtained by quantization of the [OpenLLM-Ro/RoLlama2-7b-Instruct](https://huggingface.co/OpenLLM-Ro/RoLlama2-7b-Instruct) model using the notebook [Quantized Romanian LLM (OpenLLM-Ro) Tests](https://www.kaggle.com/code/gpreda/quantized-romanian-llm-openllm-ro-tests).\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4238d9af",
   "metadata": {
    "papermill": {
     "duration": 0.004306,
     "end_time": "2024-05-19T20:51:08.602040",
     "exception": false,
     "start_time": "2024-05-19T20:51:08.597734",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Load the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "024afb52",
   "metadata": {
    "_kg_hide-output": true,
    "execution": {
     "iopub.execute_input": "2024-05-19T20:51:08.612460Z",
     "iopub.status.busy": "2024-05-19T20:51:08.612144Z",
     "iopub.status.idle": "2024-05-19T20:51:39.701790Z",
     "shell.execute_reply": "2024-05-19T20:51:39.700888Z"
    },
    "papermill": {
     "duration": 31.09764,
     "end_time": "2024-05-19T20:51:39.704231",
     "exception": false,
     "start_time": "2024-05-19T20:51:08.606591",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: transformers in /opt/conda/lib/python3.10/site-packages (4.39.3)\r\n",
      "Collecting transformers\r\n",
      "  Downloading transformers-4.41.0-py3-none-any.whl.metadata (43 kB)\r\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.8/43.8 kB\u001b[0m \u001b[31m1.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hRequirement already satisfied: accelerate in /opt/conda/lib/python3.10/site-packages (0.29.3)\r\n",
      "Collecting accelerate\r\n",
      "  Downloading accelerate-0.30.1-py3-none-any.whl.metadata (18 kB)\r\n",
      "Collecting bitsandbytes\r\n",
      "  Downloading bitsandbytes-0.43.1-py3-none-manylinux_2_24_x86_64.whl.metadata (2.2 kB)\r\n",
      "Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from transformers) (3.13.1)\r\n",
      "Collecting huggingface-hub<1.0,>=0.23.0 (from transformers)\r\n",
      "  Downloading huggingface_hub-0.23.0-py3-none-any.whl.metadata (12 kB)\r\n",
      "Requirement already satisfied: numpy>=1.17 in /opt/conda/lib/python3.10/site-packages (from transformers) (1.26.4)\r\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.10/site-packages (from transformers) (21.3)\r\n",
      "Requirement already satisfied: pyyaml>=5.1 in /opt/conda/lib/python3.10/site-packages (from transformers) (6.0.1)\r\n",
      "Requirement already satisfied: regex!=2019.12.17 in /opt/conda/lib/python3.10/site-packages (from transformers) (2023.12.25)\r\n",
      "Requirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from transformers) (2.31.0)\r\n",
      "Collecting tokenizers<0.20,>=0.19 (from transformers)\r\n",
      "  Downloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\r\n",
      "Requirement already satisfied: safetensors>=0.4.1 in /opt/conda/lib/python3.10/site-packages (from transformers) (0.4.3)\r\n",
      "Requirement already satisfied: tqdm>=4.27 in /opt/conda/lib/python3.10/site-packages (from transformers) (4.66.1)\r\n",
      "Requirement already satisfied: psutil in /opt/conda/lib/python3.10/site-packages (from accelerate) (5.9.3)\r\n",
      "Requirement already satisfied: torch>=1.10.0 in /opt/conda/lib/python3.10/site-packages (from accelerate) (2.1.2)\r\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.23.0->transformers) (2024.2.0)\r\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /opt/conda/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.23.0->transformers) (4.9.0)\r\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.10/site-packages (from packaging>=20.0->transformers) (3.1.1)\r\n",
      "Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (1.12)\r\n",
      "Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (3.2.1)\r\n",
      "Requirement already satisfied: jinja2 in /opt/conda/lib/python3.10/site-packages (from torch>=1.10.0->accelerate) (3.1.2)\r\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.10/site-packages (from requests->transformers) (3.3.2)\r\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.10/site-packages (from requests->transformers) (3.6)\r\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /opt/conda/lib/python3.10/site-packages (from requests->transformers) (1.26.18)\r\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.10/site-packages (from requests->transformers) (2024.2.2)\r\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2->torch>=1.10.0->accelerate) (2.1.3)\r\n",
      "Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch>=1.10.0->accelerate) (1.3.0)\r\n",
      "Downloading transformers-4.41.0-py3-none-any.whl (9.1 MB)\r\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m9.1/9.1 MB\u001b[0m \u001b[31m69.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hDownloading accelerate-0.30.1-py3-none-any.whl (302 kB)\r\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m302.6/302.6 kB\u001b[0m \u001b[31m20.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hDownloading bitsandbytes-0.43.1-py3-none-manylinux_2_24_x86_64.whl (119.8 MB)\r\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m119.8/119.8 MB\u001b[0m \u001b[31m12.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hDownloading huggingface_hub-0.23.0-py3-none-any.whl (401 kB)\r\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m401.2/401.2 kB\u001b[0m \u001b[31m18.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hDownloading tokenizers-0.19.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)\r\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m77.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\r\n",
      "\u001b[?25hInstalling collected packages: huggingface-hub, tokenizers, bitsandbytes, accelerate, transformers\r\n",
      "  Attempting uninstall: huggingface-hub\r\n",
      "    Found existing installation: huggingface-hub 0.22.2\r\n",
      "    Uninstalling huggingface-hub-0.22.2:\r\n",
      "      Successfully uninstalled huggingface-hub-0.22.2\r\n",
      "  Attempting uninstall: tokenizers\r\n",
      "    Found existing installation: tokenizers 0.15.2\r\n",
      "    Uninstalling tokenizers-0.15.2:\r\n",
      "      Successfully uninstalled tokenizers-0.15.2\r\n",
      "  Attempting uninstall: accelerate\r\n",
      "    Found existing installation: accelerate 0.29.3\r\n",
      "    Uninstalling accelerate-0.29.3:\r\n",
      "      Successfully uninstalled accelerate-0.29.3\r\n",
      "  Attempting uninstall: transformers\r\n",
      "    Found existing installation: transformers 4.39.3\r\n",
      "    Uninstalling transformers-4.39.3:\r\n",
      "      Successfully uninstalled transformers-4.39.3\r\n",
      "Successfully installed accelerate-0.30.1 bitsandbytes-0.43.1 huggingface-hub-0.23.0 tokenizers-0.19.1 transformers-4.41.0\r\n"
     ]
    }
   ],
   "source": [
    "!pip install -U transformers accelerate bitsandbytes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "626c509f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:51:39.727937Z",
     "iopub.status.busy": "2024-05-19T20:51:39.727662Z",
     "iopub.status.idle": "2024-05-19T20:51:57.885509Z",
     "shell.execute_reply": "2024-05-19T20:51:57.884346Z"
    },
    "papermill": {
     "duration": 18.171946,
     "end_time": "2024-05-19T20:51:57.887809",
     "exception": false,
     "start_time": "2024-05-19T20:51:39.715863",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2024-05-19 20:51:45.603685: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n",
      "2024-05-19 20:51:45.603787: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n",
      "2024-05-19 20:51:45.722080: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n"
     ]
    }
   ],
   "source": [
    "from time import time\n",
    "from IPython.display import display, Markdown\n",
    "from transformers import pipeline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7c50dd85",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:51:57.910011Z",
     "iopub.status.busy": "2024-05-19T20:51:57.909202Z",
     "iopub.status.idle": "2024-05-19T20:52:50.793200Z",
     "shell.execute_reply": "2024-05-19T20:52:50.792221Z"
    },
    "papermill": {
     "duration": 52.897148,
     "end_time": "2024-05-19T20:52:50.795343",
     "exception": false,
     "start_time": "2024-05-19T20:51:57.898195",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Unused kwargs: ['_load_in_4bit', '_load_in_8bit', 'quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>.\n",
      "`low_cpu_mem_usage` was None, now set to True since model is quantized.\n",
      "You set `add_prefix_space`. The tokenizer needs to be converted from the slow tokenizers\n",
      "You are using the default legacy behaviour of the <class 'transformers.models.llama.tokenization_llama.LlamaTokenizer'>. This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n",
      "Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.\n"
     ]
    }
   ],
   "source": [
    "pipe = pipeline(\"text-generation\", \n",
    "                model=\"/kaggle/input/openllm-ro-q4b/transformers/7b-instruct-q4b/1/RoLlama2-7b-Instruct-v1-q4b\",\n",
    "               kwargs=['_load_in_4bit'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f70d2496",
   "metadata": {
    "papermill": {
     "duration": 0.010471,
     "end_time": "2024-05-19T20:52:50.816705",
     "exception": false,
     "start_time": "2024-05-19T20:52:50.806234",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Prepare the query function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "9b7d9a30",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:52:50.840242Z",
     "iopub.status.busy": "2024-05-19T20:52:50.839476Z",
     "iopub.status.idle": "2024-05-19T20:52:50.847881Z",
     "shell.execute_reply": "2024-05-19T20:52:50.847120Z"
    },
    "papermill": {
     "duration": 0.022674,
     "end_time": "2024-05-19T20:52:50.849792",
     "exception": false,
     "start_time": "2024-05-19T20:52:50.827118",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def query_model(\n",
    "        system_message,\n",
    "        user_message,\n",
    "        temperature=0.7,\n",
    "        max_length=1024\n",
    "        ):\n",
    "    start_time = time()\n",
    "    user_message = \"Question: \" + user_message + \" Answer:\"\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": system_message},\n",
    "        {\"role\": \"user\", \"content\": user_message},\n",
    "        ]\n",
    "    prompt = pipe.tokenizer.apply_chat_template(\n",
    "        messages, \n",
    "        tokenize=False, \n",
    "        add_generation_prompt=True\n",
    "        )\n",
    "    terminators = [\n",
    "        pipe.tokenizer.eos_token_id,\n",
    "        pipe.tokenizer.convert_tokens_to_ids(\"<|eot_id|>\")\n",
    "    ]\n",
    "    sequences = pipe(\n",
    "        prompt,\n",
    "        do_sample=True,\n",
    "        top_p=0.9,\n",
    "        temperature=temperature,\n",
    "        #num_return_sequences=1,\n",
    "        eos_token_id=terminators,\n",
    "        max_new_tokens=max_length,\n",
    "        return_full_text=False,\n",
    "        pad_token_id=pipe.model.config.eos_token_id\n",
    "    )\n",
    "    #answer = f\"{sequences[0]['generated_text'][len(prompt):]}\\n\"\n",
    "    answer = sequences[0]['generated_text']\n",
    "    end_time = time()\n",
    "    ttime = f\"Total time: {round(end_time-start_time, 2)} sec.\"\n",
    "\n",
    "    return user_message + \" \" + answer  + \" \" +  ttime\n",
    "\n",
    "\n",
    "system_message = \"\"\"\n",
    "You are an AI assistant designed to answer simple questions.\n",
    "Please restrict your answer to the exact question asked.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6589b59b",
   "metadata": {
    "papermill": {
     "duration": 0.010431,
     "end_time": "2024-05-19T20:52:50.870840",
     "exception": false,
     "start_time": "2024-05-19T20:52:50.860409",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Test with few questions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "3ce276ef",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:52:50.893265Z",
     "iopub.status.busy": "2024-05-19T20:52:50.892997Z",
     "iopub.status.idle": "2024-05-19T20:52:50.897693Z",
     "shell.execute_reply": "2024-05-19T20:52:50.896846Z"
    },
    "papermill": {
     "duration": 0.017833,
     "end_time": "2024-05-19T20:52:50.899548",
     "exception": false,
     "start_time": "2024-05-19T20:52:50.881715",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "def colorize_text(text):\n",
    "    for word, color in zip([\"Reasoning\", \"Question\", \"Answer\", \"Total time\"], [\"blue\", \"red\", \"green\", \"magenta\"]):\n",
    "        text = text.replace(f\"{word}:\", f\"\\n\\n**<font color='{color}'>{word}:</font>**\")\n",
    "    return text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6363790f",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:52:50.921616Z",
     "iopub.status.busy": "2024-05-19T20:52:50.921350Z",
     "iopub.status.idle": "2024-05-19T20:52:55.064815Z",
     "shell.execute_reply": "2024-05-19T20:52:55.063870Z"
    },
    "papermill": {
     "duration": 4.156928,
     "end_time": "2024-05-19T20:52:55.066925",
     "exception": false,
     "start_time": "2024-05-19T20:52:50.909997",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues.\n"
     ]
    },
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='red'>Question:</font>** Care este suprafata Romaniei? \n",
       "\n",
       "**<font color='green'>Answer:</font>**  Suprafața României este de aproximativ 238.391 de kilometri pătrați (92.04 \n",
       "\n",
       "**<font color='magenta'>Total time:</font>** 4.14 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_model(\n",
    "    system_message,\n",
    "    user_message=\"Care este suprafata Romaniei?\",\n",
    "    temperature=0.1,\n",
    "    max_length=32)\n",
    "display(Markdown(colorize_text(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b0b019cc",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:52:55.090599Z",
     "iopub.status.busy": "2024-05-19T20:52:55.090193Z",
     "iopub.status.idle": "2024-05-19T20:52:59.699391Z",
     "shell.execute_reply": "2024-05-19T20:52:59.698506Z"
    },
    "papermill": {
     "duration": 4.623363,
     "end_time": "2024-05-19T20:52:59.701462",
     "exception": false,
     "start_time": "2024-05-19T20:52:55.078099",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='red'>Question:</font>** In ce an a castigat faima mondiala Nadia Comaneci? \n",
       "\n",
       "**<font color='green'>Answer:</font>**  Nadia Comăneci a câștigat faima mondială în 1976, când a câștigat prima medalie de aur la Jocurile Olimpice de la Montreal.  \n",
       "\n",
       "**<font color='magenta'>Total time:</font>** 4.6 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_model(\n",
    "    system_message,\n",
    "    user_message=\"In ce an a castigat faima mondiala Nadia Comaneci?\",\n",
    "    temperature=0.1,\n",
    "    max_length=64)\n",
    "display(Markdown(colorize_text(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "484119a8",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:52:59.724958Z",
     "iopub.status.busy": "2024-05-19T20:52:59.724667Z",
     "iopub.status.idle": "2024-05-19T20:53:05.383591Z",
     "shell.execute_reply": "2024-05-19T20:53:05.382554Z"
    },
    "papermill": {
     "duration": 5.672935,
     "end_time": "2024-05-19T20:53:05.385733",
     "exception": false,
     "start_time": "2024-05-19T20:52:59.712798",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='red'>Question:</font>** Numeste cateva parcuri din Bucuresti. \n",
       "\n",
       "**<font color='green'>Answer:</font>**  Parcurile din București sunt:\n",
       "1. Herăstrău\n",
       "2. Cișmigiu\n",
       "3. Carol\n",
       "4. Kiseleff\n",
       "5. Cișmigiu\n",
       "6. Tineretului\n",
       "7. Floreasca\n",
       "8. Alexandru Io \n",
       "\n",
       "**<font color='magenta'>Total time:</font>** 5.65 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_model(\n",
    "    system_message,\n",
    "    user_message=\"Numeste cateva parcuri din Bucuresti.\",\n",
    "    temperature=0.1,\n",
    "    max_length=64)\n",
    "display(Markdown(colorize_text(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e9f3b803",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:53:05.410085Z",
     "iopub.status.busy": "2024-05-19T20:53:05.409778Z",
     "iopub.status.idle": "2024-05-19T20:53:08.926729Z",
     "shell.execute_reply": "2024-05-19T20:53:08.925784Z"
    },
    "papermill": {
     "duration": 3.531298,
     "end_time": "2024-05-19T20:53:08.928882",
     "exception": false,
     "start_time": "2024-05-19T20:53:05.397584",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='red'>Question:</font>** Cine e George Hagi? \n",
       "\n",
       "**<font color='green'>Answer:</font>**  George Hagi este un fotbalist român retras, care a jucat pentru echipa națională de fotbal a României.  \n",
       "\n",
       "**<font color='magenta'>Total time:</font>** 3.51 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_model(\n",
    "    system_message,\n",
    "    user_message=\"Cine e George Hagi?\",\n",
    "    temperature=0.1,\n",
    "    max_length=64)\n",
    "display(Markdown(colorize_text(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4abfd10b",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:53:08.954552Z",
     "iopub.status.busy": "2024-05-19T20:53:08.953631Z",
     "iopub.status.idle": "2024-05-19T20:53:12.056928Z",
     "shell.execute_reply": "2024-05-19T20:53:12.056019Z"
    },
    "papermill": {
     "duration": 3.118158,
     "end_time": "2024-05-19T20:53:12.058886",
     "exception": false,
     "start_time": "2024-05-19T20:53:08.940728",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='red'>Question:</font>** Cine e primarul Capitalei Romaniei, Bucuresti, incepand cu 2020? \n",
       "\n",
       "**<font color='green'>Answer:</font>**  Primarul Capitalei României, București, începând cu 2020, este Nicușor Dan.  \n",
       "\n",
       "**<font color='magenta'>Total time:</font>** 3.1 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_model(\n",
    "    system_message,\n",
    "    user_message=\"Cine e primarul Capitalei Romaniei, Bucuresti, incepand cu 2020?\",\n",
    "    temperature=0.1,\n",
    "    max_length=64)\n",
    "display(Markdown(colorize_text(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "cbad3c00",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:53:12.084325Z",
     "iopub.status.busy": "2024-05-19T20:53:12.083990Z",
     "iopub.status.idle": "2024-05-19T20:53:14.012164Z",
     "shell.execute_reply": "2024-05-19T20:53:14.011285Z"
    },
    "papermill": {
     "duration": 1.943175,
     "end_time": "2024-05-19T20:53:14.014291",
     "exception": false,
     "start_time": "2024-05-19T20:53:12.071116",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='red'>Question:</font>** Numiti 3 pesti din Delta Dunarii. \n",
       "\n",
       "**<font color='green'>Answer:</font>**  1. Sturion\n",
       "2. Crap\n",
       "3. Somon  \n",
       "\n",
       "**<font color='magenta'>Total time:</font>** 1.92 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_model(\n",
    "    system_message,\n",
    "    user_message=\"Numiti 3 pesti din Delta Dunarii.\",\n",
    "    temperature=0.1,\n",
    "    max_length=64)\n",
    "display(Markdown(colorize_text(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "bd42ca63",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:53:14.039878Z",
     "iopub.status.busy": "2024-05-19T20:53:14.039304Z",
     "iopub.status.idle": "2024-05-19T20:53:17.101671Z",
     "shell.execute_reply": "2024-05-19T20:53:17.100742Z"
    },
    "papermill": {
     "duration": 3.077243,
     "end_time": "2024-05-19T20:53:17.103927",
     "exception": false,
     "start_time": "2024-05-19T20:53:14.026684",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='red'>Question:</font>** Numiti 3 atractii din Bucuresti. \n",
       "\n",
       "**<font color='green'>Answer:</font>**  1. Palatul Parlamentului\n",
       "2. Muzeul National de Arta\n",
       "3. Muzeul National de Istorie a Romaniei  \n",
       "\n",
       "**<font color='magenta'>Total time:</font>** 3.06 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_model(\n",
    "    system_message,\n",
    "    user_message=\"Numiti 3 atractii din Bucuresti.\",\n",
    "    temperature=0.1,\n",
    "    max_length=64)\n",
    "display(Markdown(colorize_text(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "44803800",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:53:17.130088Z",
     "iopub.status.busy": "2024-05-19T20:53:17.129770Z",
     "iopub.status.idle": "2024-05-19T20:53:19.571940Z",
     "shell.execute_reply": "2024-05-19T20:53:19.571082Z"
    },
    "papermill": {
     "duration": 2.457347,
     "end_time": "2024-05-19T20:53:19.573942",
     "exception": false,
     "start_time": "2024-05-19T20:53:17.116595",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='red'>Question:</font>** Care e suprafata judetului Dambovita? \n",
       "\n",
       "**<font color='green'>Answer:</font>**  Suprafața județului Dâmbovița este de 6.083 km².  \n",
       "\n",
       "**<font color='magenta'>Total time:</font>** 2.44 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_model(\n",
    "    system_message,\n",
    "    user_message=\"Care e suprafata judetului Dambovita?\",\n",
    "    temperature=0.1,\n",
    "    max_length=64)\n",
    "display(Markdown(colorize_text(response)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "97843227",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2024-05-19T20:53:19.599746Z",
     "iopub.status.busy": "2024-05-19T20:53:19.599236Z",
     "iopub.status.idle": "2024-05-19T20:53:21.604331Z",
     "shell.execute_reply": "2024-05-19T20:53:21.603451Z"
    },
    "papermill": {
     "duration": 2.020159,
     "end_time": "2024-05-19T20:53:21.606488",
     "exception": false,
     "start_time": "2024-05-19T20:53:19.586329",
     "status": "completed"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/markdown": [
       "\n",
       "\n",
       "**<font color='red'>Question:</font>** Cine este actualul presedinte al Romaniei? \n",
       "\n",
       "**<font color='green'>Answer:</font>**  Actualul președinte al României este Klaus Iohannis.  \n",
       "\n",
       "**<font color='magenta'>Total time:</font>** 2.0 sec."
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "response = query_model(\n",
    "    system_message,\n",
    "    user_message=\"Cine este actualul presedinte al Romaniei?\",\n",
    "    temperature=0.1,\n",
    "    max_length=64)\n",
    "display(Markdown(colorize_text(response)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2463017d",
   "metadata": {
    "papermill": {
     "duration": 0.012371,
     "end_time": "2024-05-19T20:53:21.631717",
     "exception": false,
     "start_time": "2024-05-19T20:53:21.619346",
     "status": "completed"
    },
    "tags": []
   },
   "source": [
    "# Conclusions\n",
    "\n",
    "Using the quantized model derived from OpenLLM-Ro 7B Instruct (v1) model published by **ILDS**, we can run fast queries in Romanian language."
   ]
  }
 ],
 "metadata": {
  "kaggle": {
   "accelerator": "gpu",
   "dataSources": [
    {
     "isSourceIdPinned": true,
     "modelInstanceId": 43578,
     "sourceId": 51817,
     "sourceType": "modelInstanceVersion"
    }
   ],
   "dockerImageVersionId": 30698,
   "isGpuEnabled": true,
   "isInternetEnabled": true,
   "language": "python",
   "sourceType": "notebook"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  },
  "papermill": {
   "default_parameters": {},
   "duration": 138.185154,
   "end_time": "2024-05-19T20:53:24.076140",
   "environment_variables": {},
   "exception": null,
   "input_path": "__notebook__.ipynb",
   "output_path": "__notebook__.ipynb",
   "parameters": {},
   "start_time": "2024-05-19T20:51:05.890986",
   "version": "2.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}