02-multimodal-llm/01_Amazon_Nova.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "01b7e5f3",
   "metadata": {},
   "source": [
    "<h1 style=\"background: linear-gradient(to right, #ff6b6b, #4ecdc4); \n",
    "           color: white; \n",
    "           padding: 20px; \n",
    "           border-radius: 10px; \n",
    "           text-align: center; \n",
    "           font-family: Arial, sans-serif; \n",
    "           text-shadow: 2px 2px 4px rgba(0,0,0,0.5);\">\n",
    "  Amazon Nova\n",
    "</h1>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3314f09d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# # %%capture\n",
    "# !pip install matplotlib transformers datasets accelerate sentence-transformers\n",
    "# !pip install ipywidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09c3cdab",
   "metadata": {},
   "source": [
    "Amazon Nova is a new generation of multimodal understanding and creative content generation models that offer state-of-the-art quality, unparalleled customization, and the best price-performance. Amazon Nova models incorporate the same secure-by-design approach as all AWS services, with built-in controls for the safe and responsible use of AI.\n",
    "\n",
    "Amazon Nova has two categories of models: \n",
    " - **Understanding models** —These models are capable of reasoning over several input modalities, including text, video, and image, and output text. \n",
    "- **Creative Content Generation models** —These models generate images or videos based on a text or image prompt.\n",
    "  \n",
    "![imgs/model_intro.png](imgs/nova_intro.png)\n",
    "\n",
    "**Multimodal Understanding Models**\n",
    "- **Amazon Nova Micro**: Lightening fast, cost-effective text-only model\n",
    "- **Amazon Nova Lite**: Fastest, most affordable multimodal FM in the industry for its intelligence tier\n",
    "- **Amazon Nova Pro**:  The fastest, most cost-effective, state-of-the-art multimodal model in the industry\n",
    "\n",
    "**Creative Content Generation Models**\n",
    "- **Amazon Nova Canvas**: State-of-the-art image generation model\n",
    "- **Amazon Nova Reel**: State-of-the-art video generation model\n",
    "\n",
    "The following notebooks will be focused primarily on Amazon Nova Understanding Models. \n",
    "\n",
    "**Amazon Nova Multimodal understanding** foundation models (FMs) are a family of models that are capable of reasoning over several input modalities, including text, video, documents and/or images, and output text.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3aea9407",
   "metadata": {},
   "source": [
    "<h2 style=\"background: linear-gradient(to right, #ff6b6b, #4ecdc4); \n",
    "           color: white; \n",
    "           padding: 20px; \n",
    "           border-radius: 10px; \n",
    "           text-align: center; \n",
    "           font-family: Arial, sans-serif; \n",
    "           text-shadow: 2px 2px 4px rgba(0,0,0,0.5);\">\n",
    "  Setup\n",
    "</h2>\n",
    "\n",
    "\n",
    "**Gain Access to the Model**: If you have not yet requested for model access in Bedrock, you do so [request access following these instructions](https://docs.aws.amazon.com/bedrock/latest/userguide/model-access-modify.html).\n",
    "\n",
    "![imgs/model_access.png](imgs/model_access.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c633ddd-299c-4fb6-98ba-b01bb8b48f50",
   "metadata": {},
   "source": [
    "<h2 style=\"background: linear-gradient(to right, #ff6b6b, #4ecdc4); \n",
    "           color: white; \n",
    "           padding: 20px; \n",
    "           border-radius: 10px; \n",
    "           text-align: center; \n",
    "           font-family: Arial, sans-serif; \n",
    "           text-shadow: 2px 2px 4px rgba(0,0,0,0.5);\">\n",
    "  When to Use What?\n",
    "</h2>\n",
    "\n",
    "\n",
    "### When to Use `Amazon Nova Micro` \n",
    "\n",
    "Amazon Nova Micro (Text Input Only) is the fastest and most affordable option, optimized for large-scale, latency-sensitive deployments like conversational interfaces, chats, and high-volume tasks, such as classification, routing, entity extraction, and document summarization.\n",
    "\n",
    "### When to Use `Amazon Nova Lite` \n",
    "\n",
    "Amazon Nova Lite balances intelligence, latency, and cost-effectiveness. It’s optimized for complex scenarios where low latency (minimal delay) is crucial, such as interactive agents that need to orchestrate multiple tool calls simultaneously. Amazon Nova Lite supports image, video, and text inputs and outputs text. \n",
    "\n",
    "### When to Use `Amazon Nova Pro` \n",
    "Amazon Nova Pro is designed for highly complex use cases requiring advanced reasoning, creativity, and code generation. Amazon Nova pro supports image, video, and text inputs and outputs text. \n",
    "\n",
    "### Prerequisites\n",
    "\n",
    "Run the cells in this section to install the packages needed by the notebooks in this workshop. ⚠️ You will see pip dependency errors, you can safely ignore these errors."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c0c946f",
   "metadata": {},
   "source": [
    "<h2 style=\"background: linear-gradient(to right, #ff6b6b, #4ecdc4); \n",
    "           color: white; \n",
    "           padding: 20px; \n",
    "           border-radius: 10px; \n",
    "           text-align: center; \n",
    "           font-family: Arial, sans-serif; \n",
    "           text-shadow: 2px 2px 4px rgba(0,0,0,0.5);\">\n",
    "  Text Understanding\n",
    "</h2>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "32dd0c6f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import json\n",
    "import base64\n",
    "\n",
    "PRO_MODEL_ID = \"amazon.nova-pro-v1:0\"\n",
    "LITE_MODEL_ID = \"amazon.nova-lite-v1:0\"\n",
    "MICRO_MODEL_ID = \"amazon.nova-micro-v1:0\"\n",
    "\n",
    "# Create a Bedrock Runtime client in the AWS Region of your choice.\n",
    "client = boto3.client(\"bedrock-runtime\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3e62778-e5f5-4664-a81c-ac27b54d3c2c",
   "metadata": {},
   "source": [
    "### `InvokeModel` body and output\n",
    "\n",
    "The invoke_model() method of the Amazon Bedrock runtime client (InvokeModel API) will be the primary method we use for most of our Text Generation and Processing tasks\n",
    "\n",
    "Although the method is shared, the format of input and output varies depending on the foundation model used - as described below:\n",
    "\n",
    "\n",
    "```python\n",
    "{\n",
    "  \"system\": [\n",
    "    {\n",
    "      \"text\": string\n",
    "    }\n",
    "  ],\n",
    "  \"messages\": [\n",
    "    {\n",
    "      \"role\": \"user\",# first turn should always be the user turn\n",
    "      \"content\": [\n",
    "        {\n",
    "          \"text\": string\n",
    "        },\n",
    "        {\n",
    "          \"image\": {\n",
    "            \"format\": \"jpeg\"| \"png\" | \"gif\" | \"webp\",\n",
    "            \"source\": {\n",
    "              \"bytes\": \"base64EncodedImageDataHere...\"#  base64-encoded binary\n",
    "            }\n",
    "          }\n",
    "        },\n",
    "        {\n",
    "          \"video\": {\n",
    "            \"format\": \"mkv\" | \"mov\" | \"mp4\" | \"webm\" | \"three_gp\" | \"flv\" | \"mpeg\" | \"mpg\" | \"wmv\",\n",
    "            \"source\": {\n",
    "            # source can be s3 location of base64 bytes based on size of input file. \n",
    "               \"s3Location\": {\n",
    "                \"uri\": string, #  example: s3://my-bucket/object-key\n",
    "                \"bucketOwner\": string #  (Optional) example: 123456789012)\n",
    "               }\n",
    "              \"bytes\": \"base64EncodedImageDataHere...\" #  base64-encoded binary\n",
    "            }\n",
    "          }\n",
    "        },\n",
    "      ]\n",
    "    },\n",
    "    {\n",
    "      \"role\": \"assistant\",\n",
    "      \"content\": [\n",
    "        {\n",
    "          \"text\": string # prefilling assistant turn\n",
    "        }\n",
    "      ]\n",
    "    }\n",
    "  ],\n",
    " \"inferenceConfig\":{ # all Optional\n",
    "    \"max_new_tokens\": int, #  greater than 0, equal or less than 5k (default: dynamic*)\n",
    "    \"temperature\": float, # greater then 0 and less than 1.0 (default: 0.7)\n",
    "    \"top_p\": float, #  greater than 0, equal or less than 1.0 (default: 0.9)\n",
    "    \"top_k\": int #  0 or greater (default: 50)\n",
    "    \"stopSequences\": [string]\n",
    "  },\n",
    "  \"toolConfig\": { #  all Optional\n",
    "        \"tools\": [\n",
    "                {\n",
    "                    \"toolSpec\": {\n",
    "                        \"name\": string # menaingful tool name (Max char: 64)\n",
    "                        \"description\": string # meaningful description of the tool\n",
    "                        \"inputSchema\": {\n",
    "                            \"json\": { # The JSON schema for the tool. For more information, see JSON Schema Reference\n",
    "                                \"type\": \"object\",\n",
    "                                \"properties\": {\n",
    "                                    <args>: { # arguments \n",
    "                                        \"type\": string, # argument data type\n",
    "                                        \"description\": string # meaningful description\n",
    "                                    }\n",
    "                                },\n",
    "                                \"required\": [\n",
    "                                    string # args\n",
    "                                ]\n",
    "                            }\n",
    "                        }\n",
    "                    }\n",
    "                }\n",
    "            ],\n",
    "   \"toolChoice\": \"any\" //Amazon Nova models ONLY support tool choice of \"any\"\n",
    "        }\n",
    "    }\n",
    "}\n",
    "```\n",
    "\n",
    "The following are required parameters.\n",
    "\n",
    "* `system` – (Optional) The system prompt for the request.\n",
    "    A system prompt is a way of providing context and instructions to Amazon Nova, such as specifying a particular goal or role.\n",
    "* `messages` – (Required) The input messages.\n",
    "    * `role` – The role of the conversation turn. Valid values are user and assistant. \n",
    "    * `content` – (required) The content of the conversation turn.\n",
    "        * `type` – (required) The type of the content. Valid values are image, text. , video\n",
    "            * if chosen text (text content)\n",
    "                * `text` - The content of the conversation turn. \n",
    "            * If chosen Image (image content)\n",
    "                * `source` – (required) The base64 encoded image bytes for the image.\n",
    "                * `format` – (required) The type of the image. You can specify the following image formats. \n",
    "                    * `jpeg`\n",
    "                    * `png`\n",
    "                    * `webp`\n",
    "                    * `gif`\n",
    "            * If chosen video: (video content)\n",
    "                * `source` – (required) The base64 encoded image bytes for the video or S3 URI and bucket owner as shown in the above schema\n",
    "                * `format` – (required) The type of the video. You can specify the following video formats. \n",
    "                    * `mkv`\n",
    "                    *  `mov`  \n",
    "                    *  `mp4`\n",
    "                    *  `webm`\n",
    "                    *  `three_gp`\n",
    "                    *  `flv`  \n",
    "                    *  `mpeg`  \n",
    "                    *  `mpg`\n",
    "                    *  `wmv`\n",
    "* `inferenceConfig`: These are inference config values that can be passed in inference.\n",
    "    * `max_new_tokens` – (Optional) The maximum number of tokens to generate before stopping.\n",
    "        Note that Amazon Nova models might stop generating tokens before reaching the value of max_tokens. Maximum New Tokens value allowed is 5K.\n",
    "    * `temperature` – (Optional) The amount of randomness injected into the response.\n",
    "    * `top_p` – (Optional) Use nucleus sampling. Amazon Nova computes the cumulative distribution over all the options for each subsequent token in decreasing probability order and cuts it off once it reaches a particular probability specified by top_p. You should alter either temperature or top_p, but not both.\n",
    "    * `top_k` – (Optional) Only sample from the top K options for each subsequent token. Use top_k to remove long tail low probability responses.\n",
    "    * `stopSequences` – (Optional) Array of strings containing step sequences. If the model generates any of those strings, generation will stop and response is returned up until that point. \n",
    "    * `toolConfig` – (Optional) JSON object following ToolConfig schema,  containing the tool specification and tool choice. This schema is the same followed by the Converse API\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48a9f067",
   "metadata": {},
   "source": [
    "<h2 style=\"background: linear-gradient(to right, #ff6b6b, #4ecdc4); \n",
    "           color: white; \n",
    "           padding: 20px; \n",
    "           border-radius: 10px; \n",
    "           text-align: center; \n",
    "           font-family: Arial, sans-serif; \n",
    "           text-shadow: 2px 2px 4px rgba(0,0,0,0.5);\">\n",
    "  Synchronous API Call\n",
    "</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "353f4769-4397-4dae-a4de-8a295569f6fb",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define your system prompt(s).\n",
    "system_list = [\n",
    "    { \"text\": \"You should respond to all messages in french\" }\n",
    "]\n",
    "\n",
    "# Define one or more messages using the \"user\" and \"assistant\" roles.\n",
    "message_list = [\n",
    "    {\"role\": \"user\", \"content\": [{\"text\": \"FreeCodeCamp is a great platform to learn coding and technology in the field of computer science\"}]},\n",
    "]\n",
    "\n",
    "# Configure the inference parameters.\n",
    "inf_params = {\"max_new_tokens\": 300, \"top_p\": 0.9, \"top_k\": 20}\n",
    "\n",
    "native_request = {\n",
    "    \"messages\": message_list,\n",
    "    \"system\": system_list,\n",
    "    \"inferenceConfig\": inf_params,\n",
    "}\n",
    "\n",
    "# Invoke the model and extract the response body.\n",
    "response = client.invoke_model(modelId=LITE_MODEL_ID, body=json.dumps(native_request))\n",
    "request_id = response[\"ResponseMetadata\"][\"RequestId\"]\n",
    "model_response = json.loads(response[\"body\"].read())\n",
    "\n",
    "# # Pretty print the response JSON.\n",
    "# print(\"\\n[Full Response]\")\n",
    "# print(json.dumps(model_response, indent=2))\n",
    "\n",
    "# Print the text content for easy readability.\n",
    "content_text = model_response[\"output\"][\"message\"][\"content\"][0][\"text\"]\n",
    "print(content_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b9926bc",
   "metadata": {},
   "source": [
    "<h2 style=\"background: linear-gradient(to right, #ff6b6b, #4ecdc4); \n",
    "           color: white; \n",
    "           padding: 20px; \n",
    "           border-radius: 10px; \n",
    "           text-align: center; \n",
    "           font-family: Arial, sans-serif; \n",
    "           text-shadow: 2px 2px 4px rgba(0,0,0,0.5);\">\n",
    "  Streaming API Call\n",
    "</h2>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4728b5e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import datetime\n",
    "\n",
    "# Define your system prompt(s).\n",
    "system_list = [\n",
    "    { \"text\": \"Act as a creative writing assistant. When the user provides you with a topic, write a short story about that topic.\" }\n",
    "]\n",
    "\n",
    "# Define one or more messages using the \"user\" and \"assistant\" roles.\n",
    "message_list = [{\"role\": \"user\", \"content\": [{\"text\": \"A camping trip\"}]}]\n",
    "\n",
    "# Configure the inference parameters.\n",
    "inf_params = {\"max_new_tokens\": 1000, \"top_p\": 0.9, \"top_k\": 20}\n",
    "\n",
    "request_body = {\n",
    "    \"messages\": message_list,\n",
    "    \"system\": system_list,\n",
    "    \"inferenceConfig\": inf_params,\n",
    "}\n",
    "\n",
    "start_time = datetime.now()\n",
    "\n",
    "# Invoke the model with the response stream\n",
    "response = client.invoke_model_with_response_stream(\n",
    "    modelId=LITE_MODEL_ID, body=json.dumps(request_body)\n",
    ")\n",
    "\n",
    "chunk_count = 0\n",
    "time_to_first_token = None\n",
    "\n",
    "# Process the response stream\n",
    "stream = response.get(\"body\")\n",
    "if stream:\n",
    "    for event in stream:\n",
    "        chunk = event.get(\"chunk\")\n",
    "        if chunk:\n",
    "            # Print the response chunk\n",
    "            chunk_json = json.loads(chunk.get(\"bytes\").decode())\n",
    "            # Pretty print JSON\n",
    "            # print(json.dumps(chunk_json, indent=2, ensure_ascii=False))\n",
    "            content_block_delta = chunk_json.get(\"contentBlockDelta\")\n",
    "            if content_block_delta:\n",
    "                if time_to_first_token is None:\n",
    "                    time_to_first_token = datetime.now() - start_time\n",
    "                    print(f\"Time to first token: {time_to_first_token}\")\n",
    "\n",
    "                chunk_count += 1\n",
    "                current_time = datetime.now().strftime(\"%Y-%m-%d %H:%M:%S:%f\")\n",
    "                # print(f\"{current_time} - \", end=\"\")\n",
    "                print(content_block_delta.get(\"delta\").get(\"text\"), end=\"\")\n",
    "    print(f\"\\nTotal chunks: {chunk_count}\")\n",
    "else:\n",
    "    print(\"No response stream received.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c25e1a9-912e-4bb5-82ef-9b3f673bd215",
   "metadata": {},
   "source": [
    "<h2 style=\"background: linear-gradient(to right, #ff6b6b, #4ecdc4); \n",
    "           color: white; \n",
    "           padding: 20px; \n",
    "           border-radius: 10px; \n",
    "           text-align: center; \n",
    "           font-family: Arial, sans-serif; \n",
    "           text-shadow: 2px 2px 4px rgba(0,0,0,0.5);\">\n",
    "  Image Understanding \n",
    "</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4dc68099",
   "metadata": {},
   "source": [
    "Amazon Nova models allow you to include multiple images in the payload with a limitation of total payload size to not go beyond 25MB. \n",
    "Amazon Nova models can analyze the passed images and answer questions, classify an image, as well as summarize images based on provided instructions.\n",
    "\n",
    "## Image size information\n",
    "\n",
    "To provide the best possible results, Amazon Nova automatically rescales input images up or down depending on their aspect ratio and original resolution. For each image, Amazon Nova first identifies the closest aspect ratio from 1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 1:7, 1:8, 1:9 2:3, 2:4 and their transposes. Then the image is rescaled so that at least one side of the image is greater than 896px or the length of shorter side of the original image, while maintaining the closest aspect ratio. There's a maximum resolution of 8,000x8,000 pixels.\n",
    "\n",
    "## Image to tokens conversion\n",
    "\n",
    "As previously discussed, images are resized to maximize information extraction, while still maintaining the aspect ratio. What follows are some examples of sample image dimensions and approximate token calculations.\n",
    "\n",
    "| image_resolution (HxW or WxH) | 900 x 450 | 900 x 900 | 1400 x 900 | 1.8K x 900 | 1.3Kx1.3K |\n",
    "|------------------------------|-----------|-----------|------------|------------|-----------|\n",
    "| Estimated token count        | ~800      | ~1300     | ~1800      | ~2400      | ~2600     |\n",
    "\n",
    "So for example, consider an example image that is 800x400 in size, and you want to estimate the token count for this image. Based on the dimensions, to maintain an aspect ratio of 1:2, the closest resolution is 900x450. Therefore, the approximate token count for this image is about 800 tokens.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd0239ff-e022-462f-b44d-b6d840ceade5",
   "metadata": {},
   "source": [
    "Lets see how Nova model does on Image Understanding Usecase. \n",
    "\n",
    "Here we will pass this image and ask model to try to create 3 art titles for this image. \n",
    "\n",
    "![Snow](imgs/snow.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e01e09ea-112d-4cfd-a778-b06f6477be85",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "\n",
    "# Open the image you'd like to use and encode it as a Base64 string.\n",
    "with open(\"imgs/snow.png\", \"rb\") as image_file:\n",
    "    binary_data = image_file.read()\n",
    "    base_64_encoded_data = base64.b64encode(binary_data)\n",
    "    base64_string = base_64_encoded_data.decode(\"utf-8\")\n",
    "\n",
    "# Define your system prompt(s).\n",
    "system_list = [\n",
    "    { \"text\": \"You are an expert artist. When the user provides you with an image, provide 3 potential art titles\" }\n",
    "]\n",
    "\n",
    "# Define a \"user\" message including both the image and a text prompt.\n",
    "message_list = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"image\": {\n",
    "                    \"format\": \"png\",\n",
    "                    \"source\": {\"bytes\": base64_string},\n",
    "                }\n",
    "            },\n",
    "            {\"text\": \"Provide art titles for this image.\"},\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "# Configure the inference parameters.\n",
    "inf_params = {\"max_new_tokens\": 300, \"top_p\": 0.1, \"top_k\": 20}\n",
    "\n",
    "native_request = {\n",
    "    \"messages\": message_list,\n",
    "    \"system\": system_list,\n",
    "    \"inferenceConfig\": inf_params,\n",
    "}\n",
    "\n",
    "# Invoke the model and extract the response body.\n",
    "response = client.invoke_model(modelId=LITE_MODEL_ID, body=json.dumps(native_request))\n",
    "model_response = json.loads(response[\"body\"].read())\n",
    "\n",
    "# Pretty print the response JSON.\n",
    "print(\"[Full Response]\")\n",
    "print(json.dumps(model_response, indent=2))\n",
    "\n",
    "# Print the text content for easy readability.\n",
    "content_text = model_response[\"output\"][\"message\"][\"content\"][0][\"text\"]\n",
    "print(\"\\n[Response Content Text]\")\n",
    "print(content_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f12c7267",
   "metadata": {},
   "source": [
    "There can be multiple image contents. In this example we ask the model to find what two images have in common:\n",
    "\n",
    "![](imgs/cat.jpeg)\n",
    "\n",
    "![](imgs/dog.jpeg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90b13815",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Open the image you'd like to use and encode it as a Base64 string.\n",
    "with open(\"imgs/dog.jpeg\", \"rb\") as image_file:\n",
    "    binary_data = image_file.read()\n",
    "    base_64_encoded_data = base64.b64encode(binary_data)\n",
    "    dog_base64_string = base_64_encoded_data.decode(\"utf-8\")\n",
    "\n",
    "with open(\"imgs/cat.jpeg\", \"rb\") as image_file:\n",
    "    binary_data = image_file.read()\n",
    "    base_64_encoded_data = base64.b64encode(binary_data)\n",
    "    cat_base64_string = base_64_encoded_data.decode(\"utf-8\")\n",
    "\n",
    "# Define a \"user\" message including both the image and a text prompt.\n",
    "message_list = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"image\": {\n",
    "                    \"format\": \"jpeg\",\n",
    "                    \"source\": {\"bytes\": dog_base64_string},\n",
    "                }\n",
    "            },\n",
    "            {\n",
    "                \"image\": {\n",
    "                    \"format\": \"jpeg\",\n",
    "                    \"source\": {\"bytes\": cat_base64_string},\n",
    "                }\n",
    "            },\n",
    "            {\"text\": \"What do these two images have in common?\"},\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "# Configure the inference parameters.\n",
    "inf_params = {\"max_new_tokens\": 300, \"top_p\": 0.1, \"top_k\": 20}\n",
    "\n",
    "native_request = {\n",
    "    \"messages\": message_list,\n",
    "    \"inferenceConfig\": inf_params,\n",
    "}\n",
    "\n",
    "# Invoke the model and extract the response body.\n",
    "response = client.invoke_model(modelId=LITE_MODEL_ID, body=json.dumps(native_request))\n",
    "model_response = json.loads(response[\"body\"].read())\n",
    "\n",
    "# # Pretty print the response JSON.\n",
    "# print(\"[Full Response]\")\n",
    "# print(json.dumps(model_response, indent=2))\n",
    "\n",
    "# Print the text content for easy readability.\n",
    "content_text = model_response[\"output\"][\"message\"][\"content\"][0][\"text\"]\n",
    "print(content_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f2976fe-29dc-477b-b27a-758f8f450a05",
   "metadata": {},
   "source": [
    "<h2 style=\"background: linear-gradient(to right, #ff6b6b, #4ecdc4); \n",
    "           color: white; \n",
    "           padding: 20px; \n",
    "           border-radius: 10px; \n",
    "           text-align: center; \n",
    "           font-family: Arial, sans-serif; \n",
    "           text-shadow: 2px 2px 4px rgba(0,0,0,0.5);\">\n",
    "  Video Understanding\n",
    "</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2c0de24-c4cf-497c-b6a7-d38b7c6b963e",
   "metadata": {},
   "source": [
    "The Amazon Nova models allow you to include a single video in the payload, which can be provided either in base64 format or through an Amazon S3 URI. When using the base64 method, the overall payload size must remain within 25MB. However, you can specify an Amazon S3 URI for video understanding. This approach enables you to leverage the model for longer videos (up to 1GB in size) without being constrained by the overall payload size limitation. Amazon Nova models can analyze the passed video and answer questions, classify a video, and summarize information in the video based on provided instructions.\n",
    "\n",
    "| Media File Type | File Formats supported | Input Method |\n",
    "|----------------|------------------------|--------------|\n",
    "| Video | MP4, MOV, MKV, WebM, FLV, MPEG, MPG, WMV, 3GP | Base64 <br> _(Recommended for payload size less than 25MB)_ <br><br> Amazon S3 URI <br> _(Recommended for payload greater than 25MB up to 1GB)_ |\n",
    "\n",
    "There are no differences in the video input token count, regardless of whether the video is passed as base64 (as long as it fits within the size constraints) or via an Amazon S3 location.\n",
    "\n",
    "Note that for 3gp file format, the \"format\" field passed in the API request should be of the format \"three_gp\".\n",
    "\n",
    "When using Amazon S3, ensure that you are set the \"Content-Type\" metadata to the correct MIME type for the video.\n",
    "\n",
    "## Video size information\n",
    "\n",
    "Amazon Nova video understanding capabilities support Multi-Aspect Ratio. All videos are resized with distortion (up or down, based on the input) to 672*672 square dimensions before feeding it to the model. The model utilizes a dynamic sampling strategy based on the length of the video. For videos less than or equal to 16 minutes in duration, a 1 frame per second (FPS) sampling rate is employed. However, for videos exceeding 16 minutes in length, the sampling rate decreases in order to maintain a consistent 960 frames sampled, with the frame sampling rate varying accordingly. This approach is designed to provide more accurate scene-level video understanding for shorter videos compared to longer video content. We recommend that you keep the video length less than 1 hour for low motion, and less than 16 minutes for anything with higher motion.\n",
    "\n",
    "There should be no difference when analyzing a 4k version of a video and a Full HD version. Similarly, because the sampling rate is at most 1 FPS, a 60 FPS video should perform as well as a 30 FPS video. Because of the 1GB limit in video size, using higher than required resolution and FPS is not beneficial and will limit the video length that fits in that size limit. You might want to pre-process videos longer than 1GB.\n",
    "\n",
    "## Video tokens\n",
    "\n",
    "The length of the video is main factor impacting the number of tokens generated. To calculate the approximate cost, you should multiply the estimated number of video tokens by the per-token price of the specific model being utilized.\n",
    "\n",
    "This table has some approximations of frame sampling and token utilization per video length:\n",
    "\n",
    "| video_duration | 10 sec | 30 sec | 16 min | 20 min | 30 min | 45 min | 1 hr | 1.5 hr |\n",
    "|----------------|---------|---------|---------|---------|---------|---------|-------|---------|\n",
    "| frames_to_sample | 10 | 30 | 960 | 960 | 960 | 960 | 960 | 960 |\n",
    "| sample_rate_fps | 1 | 1 | 1 | 0.755 | 0.5 | 0.35556 | 0.14 | 0.07 |\n",
    "| Estimated token count | 2,880 | 8,640 | 276,480 | 276,480 | 276,480 | 276,480 | 276,480 | 276,480 |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7f9a327-79b2-472b-911e-735546aa51bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import Video\n",
    "\n",
    "Video(\"imgs/the-sea.mp4\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9253bf6e-b2dc-41ef-ae5f-b06e35568eba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open the image you'd like to use and encode it as a Base64 string.\n",
    "with open(\"imgs/the-sea.mp4\", \"rb\") as video_file:\n",
    "    binary_data = video_file.read()\n",
    "    base_64_encoded_data = base64.b64encode(binary_data)\n",
    "    base64_string = base_64_encoded_data.decode(\"utf-8\")\n",
    "\n",
    "# Define your system prompt(s).\n",
    "system_list = [\n",
    "    { \"text\": \"You are an expert media analyst. When the user provides you with a video, provide 3 potential video titles\" }\n",
    "]\n",
    "\n",
    "# Define a \"user\" message including both the image and a text prompt.\n",
    "message_list = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"video\": {\n",
    "                    \"format\": \"mp4\",\n",
    "                    \"source\": {\"bytes\": base64_string},\n",
    "                }\n",
    "            },\n",
    "            {\"text\": \"Provide video titles for this clip.\"},\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "# Configure the inference parameters.\n",
    "inf_params = {\"max_new_tokens\": 300, \"top_p\": 0.1, \"top_k\": 20}\n",
    "\n",
    "native_request = {\n",
    "    \"messages\": message_list,\n",
    "    \"system\": system_list,\n",
    "    \"inferenceConfig\": inf_params,\n",
    "}\n",
    "\n",
    "# Invoke the model and extract the response body.\n",
    "response = client.invoke_model(modelId=LITE_MODEL_ID, body=json.dumps(native_request))\n",
    "model_response = json.loads(response[\"body\"].read())\n",
    "\n",
    "# # Pretty print the response JSON.\n",
    "# print(\"[Full Response]\")\n",
    "# print(json.dumps(model_response, indent=2))\n",
    "\n",
    "# Print the text content for easy readability.\n",
    "content_text = model_response[\"output\"][\"message\"][\"content\"][0][\"text\"]\n",
    "print(\"\\n[Response Content Text]\")\n",
    "print(content_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48a29615-4915-4475-8511-fff08eeb97ad",
   "metadata": {},
   "source": [
    "### Video Understanding using S3 Path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a2b1bc4-3e98-46ed-ab0c-a68e90f7c541",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# Define your system prompt(s).\n",
    "system_list = [\n",
    "    { \"text\": \"You are an expert media analyst. When the user provides you with a video, provide 3 potential video titles\" }\n",
    "]\n",
    "\n",
    "# Define a \"user\" message including both the image and a text prompt.\n",
    "message_list = [\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": [\n",
    "            {\n",
    "                \"video\": {\n",
    "                    \"format\": \"mp4\",\n",
    "                    \"source\": {\n",
    "                        \"s3Location\": {\n",
    "                            # Replace the S3 URI\n",
    "                            \"uri\": \"s3://my-bedrock-dataset-1/the-sea.mp4\"\n",
    "                        }\n",
    "                    },\n",
    "                }\n",
    "            },\n",
    "            {\"text\": \"Provide video titles for this clip.\"},\n",
    "        ],\n",
    "    }\n",
    "]\n",
    "\n",
    "# Configure the inference parameters.\n",
    "inf_params = {\"max_new_tokens\": 300, \"top_p\": 0.1, \"top_k\": 20, \"temperature\": 0.3}\n",
    "\n",
    "native_request = {\n",
    "    \"schemaVersion\": \"messages-v1\",\n",
    "    \"messages\": message_list,\n",
    "    \"system\": system_list,\n",
    "    \"inferenceConfig\": inf_params,\n",
    "}\n",
    "\n",
    "# Invoke the model and extract the response body.\n",
    "response = client.invoke_model(modelId=PRO_MODEL_ID, body=json.dumps(native_request))\n",
    "model_response = json.loads(response[\"body\"].read())\n",
    "\n",
    "# # Pretty print the response JSON.\n",
    "# print(\"[Full Response]\")\n",
    "# print(json.dumps(model_response, indent=2))\n",
    "\n",
    "# Print the text content for easy readability.\n",
    "content_text = model_response[\"output\"][\"message\"][\"content\"][0][\"text\"]\n",
    "print(content_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8631db0f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}