Version bump v0.24.0 (#237)

* Multi-agent shared vector database (#202) * add example * create env variables * add md file * Update pull-request-links.yml * testing action * testing action * testing action * testing action * action works * PR updates * update to shared memory * vectors * Remove reference to `create_index()` for `VectorStoreDriver`s (#205) * remove reference to create_index() * Update image tool documentation for task memory support (#206) * Update ImageLoader interface (#208) * Add example mappings for Amazon OpenSearch index creation (#209) * Add example mappings for Amazon OpenSearch index creation * Removed carryover comment * Refactor header structure, fix a couple headings (#211) * Fix marqo query example (#214) * Fix directory structure to match new layout (#215) * Updated ToS (#213) * Updated Terms of Service Added instructions for generating from .docx with Pandoc * Updated Terms of Service Added instructions for generating from .docx with Pandoc * Switch to rtf format * Fix pandoc errors * More pandoc errors Maybe Pandox was a bad idea * Hack to exclude code snippets from pytest (#216) * Run integration tests on PRs (#217) * Run integration tests on PRs * Fix env vars * Update deps * Update readthedocs build def (#223) * Fix integration tests (#220) * Run integration tests on PRs * Fix env vars * Fix some examples * test code blocks in specific path/file with CODE_PATH env var * update engine tests to define prompt driver * Fix test failures in vector store drivers * Fix task memory example execution * Fix tasks example execution * Update prompt driver * Update env var thing * Add pgvector service container * Wire up pgvector vars * save workflow config * Replicate .csv assets locally * Test workflow updates * Update dependencies * Update workflow * Update spec * Widen test scope to all integration tests * misc and example fixes * Remove dev dependency * wire env var --------- Co-authored-by: Andrew French <andrew@afren.ch> * Add Image Query docs (#219) * Update docs for new config classes (#218) * Update docs for new config classes * Fix links * Revert poetry lock to main * Remove docs gruop * Ignore assets * Update env * Update config * Add back assets * Revert poetry * Revert group * Regerate lock file * Regenerate lockfile * Fixes * Fix tests * Update docs for loader extras (#210) * Update docs for loader extras * Add extras to loaders, add missing loader docs * Remove cloud links (#225) * Add Image Query Task docs (#226) * Remove references to default tokenizers (#212) * Update test deps (#233) * Update test deps * Fix tokenizer examples * Update simple tokenizer * Fix missing args * Update lock * Fix import * Fix missing models * Ignore marqo * Fix model names * Add docs for voyageai (#230) * Add docs for voyageai * Fix example, add key * Fix key * Regenerate lock * Add docs for gemini (#231) * Add docs for gemini * Fix reference links * Claude 3 image query docs (#234) * adding anthropic and bedrock image query docs * updating max_token name * fixing typo * updating to remove openai default model for image query driver * updated wording * updating path for image file * docs: max_tokens documentation for vision * updating engines page for max_tokens specification --------- Co-authored-by: Michael Bianco <iloveitaly@gmail.com> * Add web scraper driver docs (#227) * Update docs for Anthropic (#229) * Update docs for Anthropic * Link to example of overriding * Update link * Fix url * Update deps * Version bump v0.24.0 * Update poetry.lock --------- Co-authored-by: Matt Vallillo <matt@griptape.ai> Co-authored-by: Andrew French <andrew@afren.ch> Co-authored-by: Collin Dutter <collin@griptape.ai> Co-authored-by: Zach Giordano <32624672+zachgiordano@users.noreply.github.com> Co-authored-by: Emily Danielson <2302515+emjay07@users.noreply.github.com> Co-authored-by: Michael Bianco <iloveitaly@gmail.com> Co-authored-by: dylanholmes <4370153+dylanholmes@users.noreply.github.com>
griptape-ai · Mar 27, 2024 · e1b929a · e1b929a
1 parent 2c640eb
commit e1b929a
Show file tree

Hide file tree

Showing 19 changed files with 1,981 additions and 1,413 deletions.
diff --git a/.github/actions/init-environment/action.yml b/.github/actions/init-environment/action.yml
@@ -34,3 +34,7 @@ runs:
           source .venv/bin/activate
           echo PATH=$PATH >> $GITHUB_ENV
         shell: bash
+
+      - name: Install playwright
+        run: playwright install --with-deps
+        shell: bash
diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml
@@ -103,6 +103,7 @@ jobs:
       POSTGRES_DB: ${{ secrets.INTEG_POSTGRES_DB }}
       POSTGRES_HOST: ${{ secrets.INTEG_POSTGRES_HOST }}
       POSTGRES_PORT: ${{ secrets.INTEG_POSTGRES_PORT }}
+      VOYAGE_API_KEY: ${{ secrets.INTEG_VOYAGE_API_KEY }}
 
     steps:
       - name: Checkout actions

diff --git a/docs/examples/load-query-and-chat-marqo.md b/docs/examples/load-query-and-chat-marqo.md
@@ -1,4 +1,4 @@
-```python
+```python title="PYTEST_IGNORE"
 import os
 from griptape import utils
 from griptape.drivers import MarqoVectorStoreDriver

diff --git a/docs/griptape-framework/data/chunkers.md b/docs/griptape-framework/data/chunkers.md
@@ -15,7 +15,7 @@ from griptape.chunkers import TextChunker
 from griptape.tokenizers import OpenAiTokenizer
 TextChunker(
      # set an optional custom tokenizer
-     tokenizer=OpenAiTokenizer(model=OpenAiTokenizer.DEFAULT_OPENAI_GPT_4_MODEL),
+     tokenizer=OpenAiTokenizer(model="gpt-4"),
      # optionally modify default number of tokens
      max_tokens=100
 ).chunk("long text")

diff --git a/docs/griptape-framework/data/loaders.md b/docs/griptape-framework/data/loaders.md
@@ -6,6 +6,9 @@ multiple documents with [load_collection()](../../reference/griptape/loaders/bas
 
 ## Pdf Loader
 
+!!! info
+    This driver requires the `loaders-pdf` [extra](../index.md#extras).
+
 Inherits from the [TextLoader](../../reference/griptape/loaders/text_loader.md) and can be used to load PDFs from a path or from an IO stream:
 
 ```python
@@ -53,8 +56,34 @@ CsvLoader().load("tests/assets/cities.csv")
 
 CsvLoader().load_collection(["tests/assets/cities.csv", "tests/assets/addresses.csv"])
 ```
+
+
+## Dataframe Loader
+
+!!! info
+    This driver requires the `loaders-dataframe` [extra](../index.md#extras).
+
+Can be used to load [pandas](https://pandas.pydata.org/) [DataFrame](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.html)s into [CsvRowArtifact](../../reference/griptape/artifacts/csv_row_artifact.md)s:
+
+```python
+import urllib
+import pandas as pd
+from griptape.loaders import DataframeLoader
+
+urllib.request.urlretrieve("https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv", "cities.csv")
+
+artifacts = loader.load()
+
+DataframeLoader().load(pd.read_csv(path))
+
+urllib.request.urlretrieve("https://people.sc.fsu.edu/~jburkardt/data/csv/addresses.csv", "addresses.csv")
+
+CsvLoader().load_collection(
+    [pd.read_csv('cities.csv'), pd.read_csv('addresses.csv')]
+)
 ```
 
+
 ## Text Loader
 
 Used to load arbitrary text and text files:
@@ -83,6 +112,9 @@ You can set a custom [tokenizer](../../reference/griptape/loaders/text_loader.md
 
 ## Web Loader
 
+!!! info
+    This driver requires the `loaders-web` [extra](../index.md#extras).
+
 Inherits from the [TextLoader](../../reference/griptape/loaders/text_loader.md) and can be used to load web pages:
 
 ```python
@@ -99,6 +131,9 @@ WebLoader().load_collection(
 
 ## Image Loader
 
+!!! info
+    This driver requires the `loaders-image` [extra](../index.md#extras).
+
 The Image Loader is used to load an image as an [ImageArtifact](./artifacts.md#imageartifact). The Loader operates on image bytes that can be sourced from files on disk, downloaded images, or images in memory.
 
 ```python
@@ -117,3 +152,21 @@ from griptape.loaders import ImageLoader
 with open("tests/assets/mountain.png", "rb") as f:
     image_artifact_jpeg = ImageLoader(format="bmp").load(f.read())
 ```
+
+
+## Email Loader
+
+!!! info
+    This driver requires the `loaders-email` [extra](../index.md#extras).
+
+Can be used to load email from an imap server:
+
+```python
+from griptape.loaders import EmailLoader
+
+loader = EmailLoader(imap_url="an.email.server.hostname", username="username", password="password")
+
+loader.load(EmailLoader.EmailQuery(label="INBOX"))
+
+loader.load_collection([EmailLoader.EmailQuery(label="INBOX"), EmailLoader.EmailQuery(label="SENT")])
+```
diff --git a/docs/griptape-framework/drivers/embedding-drivers.md b/docs/griptape-framework/drivers/embedding-drivers.md
@@ -51,6 +51,24 @@ print(embeddings[:3])
 [-0.234375, -0.024902344, -0.14941406]
 ```
 
+### Google Embeddings
+!!! info
+    This driver requires the `drivers-embedding-google` [extra](../index.md#extras).
+
+The [GoogleEmbeddingDriver](../../reference/griptape/drivers/embedding/google_embedding_driver.md) uses the [Google Embeddings API](https://ai.google.dev/tutorials/python_quickstart#use_embeddings).
+
+```python
+from griptape.drivers import GoogleEmbeddingDriver
+
+embeddings = GoogleEmbeddingDriver().embed_string("Hello world!")
+
+# display the first 3 embeddings
+print(embeddings[:3])
+```
+```
+[0.0588633, 0.0033929371, -0.072810836]
+```
+
 ### Hugging Face Hub Embeddings
 
 !!! info
@@ -70,6 +88,7 @@ driver = HuggingFaceHubEmbeddingDriver(
     api_token=os.environ["HUGGINGFACE_HUB_ACCESS_TOKEN"],
     model="sentence-transformers/all-MiniLM-L6-v2",
     tokenizer=HuggingFaceTokenizer(
+        max_output_tokens=512,
         tokenizer=AutoTokenizer.from_pretrained(
             "sentence-transformers/all-MiniLM-L6-v2"
         )
@@ -125,22 +144,51 @@ embeddings = driver.embed_string("Hello world!")
 print(embeddings[:3])
 ```
 
+### VoyageAI Embeddings
+The [VoyageAiEmbeddingDriver](../../reference/griptape/drivers/embedding/voyageai_embedding_driver.md) uses the [VoyageAI Embeddings API](https://www.voyageai.com/).
+
+!!! info
+    This driver requires the `drivers-embedding-voyageai` [extra](../index.md#extras).
+
+```python
+import os
+from griptape.drivers import VoyageAiEmbeddingDriver
+
+driver = VoyageAiEmbeddingDriver(
+    api_key=os.environ["VOYAGE_API_KEY"]
+)
+
+embeddings = driver.embed_string("Hello world!")
+
+# display the first 3 embeddings
+print(embeddings[:3])
+```
+
 ### Override Default Structure Embedding Driver
 Here is how you can override the Embedding Driver that is used by default in Structures. 
 
 ```python
 from griptape.structures import Agent
 from griptape.tools import WebScraper, TaskMemoryClient
-from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver
-from griptape.config import StructureConfig, OpenAiStructureConfig
+from griptape.drivers import (
+    OpenAiChatPromptDriver,
+    VoyageAiEmbeddingDriver,
+)
+from griptape.config import (
+    StructureGlobalDriversConfig,
+    StructureConfig,
+)
 
 agent = Agent(
     tools=[WebScraper(), TaskMemoryClient(off_prompt=False)],
-    config=OpenAiStructureConfig(
+    config=StructureConfig(
         global_drivers=StructureGlobalDriversConfig(
-            embedding_driver=OpenAiEmbeddingDriver()
+            prompt_driver=OpenAiChatPromptDriver(model="gpt-4"),
+            embedding_driver=VoyageAiEmbeddingDriver(),
         )
-    )
+    ),
 )
 
 agent.run("based on https://www.griptape.ai/, tell me what Griptape is")
+```
+
diff --git a/docs/griptape-framework/drivers/image-query-drivers.md b/docs/griptape-framework/drivers/image-query-drivers.md
@@ -2,6 +2,63 @@
 
 Image Query Drivers are used by [Image Query Engines](../engines/image-query-engines.md) to execute natural language queries on the contents of images. You can specify the provider and model used to query the image by providing the Engine with a particular Image Query Driver.
 
+!!! info
+    All Image Query Drivers default to a `max_tokens` of 256. It is recommended that you set this value to correspond to the desired response length. 
+
+## AnthropicImageQueryDriver
+
+!!! info
+    To tune `max_tokens`, see [Anthropic's documentation on image tokens](https://docs.anthropic.com/claude/docs/vision#image-costs) for more information on how to relate token count to response length.
+
+The [AnthropicImageQueryDriver](../../reference/griptape/drivers/image_query/anthropic_image_query_driver.md) is used to query images using Anthropic's Claude 3 multi-modal model. Here is an example of how to use it:
+
+```python
+from griptape.drivers import AnthropicImageQueryDriver
+from griptape.engines import ImageQueryEngine
+from griptape.loaders import ImageLoader
+
+driver = AnthropicImageQueryDriver(
+    model="claude-3-sonnet-20240229",
+    max_tokens=1024,
+)
+
+engine = ImageQueryEngine(
+    image_query_driver=driver,
+)
+
+with open("tests/assets/mountain.png", "rb") as f:
+    image_artifact = ImageLoader().load(f.read())
+
+engine.run("Describe the weather in the image", [image_artifact])
+```
+
+You can also specify multiple images with a single text prompt. This applies the same text prompt to all images specified, up to a max of 20. However, you will still receive one text response from the model currently.
+
+```python
+from griptape.drivers import AnthropicImageQueryDriver
+from griptape.engines import ImageQueryEngine
+from griptape.loaders import ImageLoader
+
+driver = AnthropicImageQueryDriver(
+    model="claude-3-sonnet-20240229",
+    max_tokens=1024,
+)
+
+engine = ImageQueryEngine(
+    image_query_driver=driver,
+)
+
+with open("tests/assets/mountain.png", "rb") as f:
+    image_artifact1 = ImageLoader().load(f.read())
+
+with open("tests/assets/cow.png", "rb") as f:
+    image_artifact2 = ImageLoader().load(f.read())
+
+result = engine.run("Describe the weather in the image", [image_artifact1, image_artifact2])
+
+print(result)
+```
+
 ## OpenAiVisionImageQueryDriver
 
 !!! info
@@ -16,7 +73,7 @@ from griptape.loaders import ImageLoader
 
 driver = OpenAiVisionImageQueryDriver(
     model="gpt-4-vision-preview",
-    max_tokens=200,
+    max_tokens=256,
 )
 
 engine = ImageQueryEngine(
@@ -28,3 +85,40 @@ with open("tests/assets/mountain.png", "rb") as f:
 
 engine.run("Describe the weather in the image", [image_artifact])
 ```
+
+## AmazonBedrockImageQueryDriver
+
+The [Amazon Bedrock Image Query Driver](../../reference/griptape/drivers/image_query/amazon_bedrock_image_query_driver.md) provides multi-model access to image query models hosted by Amazon Bedrock. This Driver manages API calls to the Bedrock API, while the specific Model Drivers below format the API requests and parse the responses.
+
+### Claude
+
+The [BedrockClaudeImageQueryModelDriver](../../reference/griptape/drivers/image_query_model/bedrock_claude_image_query_model_driver.md) provides support for Claude models hosted by Bedrock.
+
+```python
+from griptape.drivers import AmazonBedrockImageQueryDriver, BedrockClaudeImageQueryModelDriver
+from griptape.engines import ImageQueryEngine
+from griptape.loaders import ImageLoader
+import boto3
+
+session = boto3.Session(
+    region_name="us-west-2"
+)
+
+driver = AmazonBedrockImageQueryDriver(
+    image_query_model_driver=BedrockClaudeImageQueryModelDriver(),
+    model="anthropic.claude-3-sonnet-20240229-v1:0",
+    session=session
+)
+
+engine = ImageQueryEngine(
+    image_query_driver=driver
+)
+
+with open("tests/assets/mountain.png", "rb") as f:
+    image_artifact = ImageLoader().load(f.read())
+
+
+result = engine.run("Describe the weather in the image", [image_artifact])
+
+print(result)
+```
diff --git a/docs/griptape-framework/drivers/prompt-drivers.md b/docs/griptape-framework/drivers/prompt-drivers.md
@@ -197,7 +197,7 @@ agent.run('What is the sentiment of this review? Review: "I really enjoyed this
 !!! info
     This driver requires the `drivers-prompt-anthropic` [extra](../index.md#extras).
 
-The [AnthropicPromptDriver](../../reference/griptape/drivers/prompt/anthropic_prompt_driver.md) connects to the Anthropic [Completions](https://docs.anthropic.com/claude/reference/complete_post) API.
+The [AnthropicPromptDriver](../../reference/griptape/drivers/prompt/anthropic_prompt_driver.md) connects to the Anthropic [Messages](https://docs.anthropic.com/claude/reference/messages_post) API.
 
 ```python
 import os
@@ -209,7 +209,7 @@ agent = Agent(
     config=StructureConfig(
         global_drivers=StructureGlobalDriversConfig(
             prompt_driver=AnthropicPromptDriver(
-                model="claude-2",
+                model="claude-3-opus-20240229",
                 api_key=os.environ['ANTHROPIC_API_KEY'],
             )
         )
@@ -219,6 +219,33 @@ agent = Agent(
 agent.run('Where is the best place to see cherry blossums in Japan?')
 ```
 
+### Google
+
+!!! info
+    This driver requires the `drivers-prompt-google` [extra](../index.md#extras).
+
+The [GooglePromptDriver](../../reference/griptape/drivers/prompt/google_prompt_driver.md) connects to the [Google Generative AI](https://ai.google.dev/tutorials/python_quickstart#generate_text_from_text_inputs) API.
+
+```python
+import os
+from griptape.structures import Agent
+from griptape.drivers import GooglePromptDriver
+from griptape.config import StructureConfig, StructureGlobalDriversConfig
+
+agent = Agent(
+    config=StructureConfig(
+        global_drivers=StructureGlobalDriversConfig(
+            prompt_driver=GooglePromptDriver(
+                model="gemini-pro",
+                api_key=os.environ['GOOGLE_API_KEY'],
+            )
+        )
+    )
+)
+
+agent.run('Briefly explain how a computer works to a young child.')
+```
+
 ### Hugging Face Hub
 
 !!! info
@@ -494,7 +521,7 @@ agent = Agent(
     config=StructureConfig(
         global_drivers=StructureGlobalDriversConfig(
             prompt_driver=AmazonBedrockPromptDriver(
-                model="anthropic.claude-v2",
+                model="anthropic.claude-3-sonnet-20240229-v1:0",
                 prompt_model_driver=BedrockClaudePromptModelDriver(
                     top_p=1,
                 ),