Release version v0.23.0 (#224)

* Multi-agent shared vector database (#202) * add example * create env variables * add md file * Update pull-request-links.yml * testing action * testing action * testing action * testing action * action works * PR updates * update to shared memory * vectors * Remove reference to `create_index()` for `VectorStoreDriver`s (#205) * remove reference to create_index() * Update image tool documentation for task memory support (#206) * Update ImageLoader interface (#208) * Add example mappings for Amazon OpenSearch index creation (#209) * Add example mappings for Amazon OpenSearch index creation * Removed carryover comment * Refactor header structure, fix a couple headings (#211) * Fix marqo query example (#214) * Fix directory structure to match new layout (#215) * Updated ToS (#213) * Updated Terms of Service Added instructions for generating from .docx with Pandoc * Updated Terms of Service Added instructions for generating from .docx with Pandoc * Switch to rtf format * Fix pandoc errors * More pandoc errors Maybe Pandox was a bad idea * Hack to exclude code snippets from pytest (#216) * Run integration tests on PRs (#217) * Run integration tests on PRs * Fix env vars * Update deps * Update readthedocs build def (#223) * Fix integration tests (#220) * Run integration tests on PRs * Fix env vars * Fix some examples * test code blocks in specific path/file with CODE_PATH env var * update engine tests to define prompt driver * Fix test failures in vector store drivers * Fix task memory example execution * Fix tasks example execution * Update prompt driver * Update env var thing * Add pgvector service container * Wire up pgvector vars * save workflow config * Replicate .csv assets locally * Test workflow updates * Update dependencies * Update workflow * Update spec * Widen test scope to all integration tests * misc and example fixes * Remove dev dependency * wire env var --------- Co-authored-by: Andrew French <andrew@afren.ch> * Add Image Query docs (#219) * Update docs for new config classes (#218) * Update docs for new config classes * Fix links * Revert poetry lock to main * Remove docs gruop * Ignore assets * Update env * Update config * Add back assets * Revert poetry * Revert group * Regerate lock file * Regenerate lockfile * Fixes * Fix tests * Version bump v0.23.0 --------- Co-authored-by: Matt Vallillo <matt@griptape.ai> Co-authored-by: Collin Dutter <collin@griptape.ai> Co-authored-by: cjkindel <cjkindel@users.noreply.github.com>
griptape-ai · Feb 29, 2024 · 2c640eb · 2c640eb
1 parent f94582a
commit 2c640eb
Show file tree

Hide file tree

Showing 48 changed files with 3,279 additions and 1,468 deletions.
diff --git a/.github/workflows/integration-tests.yml b/.github/workflows/integration-tests.yml
@@ -1,10 +1,26 @@
 name: Integration Tests
 
-on: workflow_dispatch
+on:
+  pull_request:
+    branches: [ "main", "dev" ]
 
 jobs:
   test:
     runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: pgvector/pgvector:pg16
+        ports:
+          - 5432:5432
+        env:
+          POSTGRES_USER: ${{ secrets.INTEG_POSTGRES_USER }}
+          POSTGRES_PASSWORD: ${{ secrets.INTEG_POSTGRES_PASSWORD }}
+          POSTGRES_DB: ${{ secrets.INTEG_POSTGRES_DB }}
+        options: >-
+          --health-cmd pg_isready
+          --health-interval 10s
+          --health-timeout 5s
+          --health-retries 5
     strategy:
       fail-fast: false
       matrix:
@@ -38,6 +54,15 @@ jobs:
       MONGODB_DATABASE_NAME: ${{ secrets.INTEG_MONGODB_DATABASE_NAME }}
       MONGODB_COLLECTION_NAME: ${{ secrets.INTEG_MONGODB_COLLECTION_NAME }}
       MONGODB_PASSWORD: ${{ secrets.INTEG_MONGODB_PASSWORD }}
+      MONGODB_INDEX_NAME: ${{ secrets.INTEG_MONGODB_INDEX_NAME }}
+      MONGODB_VECTOR_PATH: ${{ secrets.INTEG_MONGODB_VECTOR_PATH }}
+      AZURE_MONGODB_HOST: ${{ secrets.INTEG_AZURE_MONGODB_HOST }}
+      AZURE_MONGODB_USERNAME: ${{ secrets.INTEG_AZURE_MONGODB_USERNAME }}
+      AZURE_MONGODB_PASSWORD: ${{ secrets.INTEG_AZURE_MONGODB_PASSWORD }}
+      AZURE_MONGODB_DATABASE_NAME: ${{ secrets.INTEG_AZURE_MONGODB_DATABASE_NAME }}
+      AZURE_MONGODB_COLLECTION_NAME: ${{ secrets.INTEG_AZURE_MONGODB_COLLECTION_NAME }}
+      AZURE_MONGODB_INDEX_NAME: ${{ secrets.INTEG_AZURE_MONGODB_INDEX_NAME }}
+      AZURE_MONGODB_VECTOR_PATH: ${{ secrets.INTEG_AZURE_MONGODB_VECTOR_PATH }}
       AMAZON_OPENSEARCH_HOST: ${{ secrets.INTEG_AMAZON_OPENSEARCH_HOST }}
       AMAZON_OPENSEARCH_INDEX_NAME: ${{ secrets.INTEG_AMAZON_OPENSEARCH_INDEX_NAME }}
       GOOGLE_API_KEY: ${{ secrets.INTEG_GOOGLE_API_KEY }}
@@ -66,12 +91,18 @@ jobs:
       AZURE_OPENAI_DALL_E_3_DEPLOYMENT_ID: ${{ secrets.INTEG_AZURE_OPENAI_DALL_E_3_DEPLOYMENT_ID }}
       AZURE_OPENAI_DEV_2_API_BASE: ${{ secrets.INTEG_AZURE_OPENAI_DEV_2_API_BASE }}
       AZURE_OPENAI_DEV_2_API_KEY: ${{ secrets.INTEG_AZURE_OPENAI_DEV_2_API_KEY }}
+      AZURE_OPENAI_API_BASE: ${{ secrets.INTEG_AZURE_OPENAI_API_BASE }}
       COHERE_API_KEY: ${{ secrets.INTEG_COHERE_API_KEY }}
       GOOGLE_OWNER_EMAIL: ${{ secrets.INTEG_GOOGLE_OWNER_EMAIL }}
       LEONARDO_API_KEY: ${{ secrets.INTEG_LEONARDO_API_KEY }}
       LEONARDO_MODEL_ID: ${{ secrets.INTEG_LEONARDO_MODEL_ID }}
       SAGEMAKER_TENSORFLOW_HUB_MODEL: ${{ secrets.INTEG_SAGEMAKER_TENSORFLOW_HUB_MODEL }}
       SAGEMAKER_HUGGINGFACE_MODEL: ${{ secrets.INTEG_SAGEMAKER_HUGGINGFACE_MODEL }}
+      POSTGRES_USER: ${{ secrets.INTEG_POSTGRES_USER }}
+      POSTGRES_PASSWORD: ${{ secrets.INTEG_POSTGRES_PASSWORD }}
+      POSTGRES_DB: ${{ secrets.INTEG_POSTGRES_DB }}
+      POSTGRES_HOST: ${{ secrets.INTEG_POSTGRES_HOST }}
+      POSTGRES_PORT: ${{ secrets.INTEG_POSTGRES_PORT }}
 
     steps:
       - name: Checkout actions

diff --git a/.github/workflows/pull-request-links.yml b/.github/workflows/pull-request-links.yml
@@ -17,3 +17,4 @@ jobs:
         with:
           project-slug: ${{ secrets.READTHEDOCS_PROJECT_SLUG }} 
           single-version: 'false'
+          project-language: ""
diff --git a/.readthedocs.yml b/.readthedocs.yml
@@ -10,7 +10,7 @@ build:
       - poetry config virtualenvs.create false
       - git clone -b $([ $READTHEDOCS_VERSION_NAME = "latest" ] && echo "main" || echo "dev") https://github.com/griptape-ai/griptape.git
     post_install:
-      - poetry install --with docs
+      - VIRTUAL_ENV=$READTHEDOCS_VIRTUALENV_PATH poetry install --with docs
 
 mkdocs:
   configuration: mkdocs.yml

diff --git a/README.md b/README.md
@@ -3,3 +3,5 @@
 [![Docs](https://readthedocs.org/projects/griptape/badge/)](https://griptape.readthedocs.io)
 
 Docs for [Griptape](https://github.com/griptape-ai/griptape).
+
+Terms of Service was converted from .rtf file format using [Pandoc](https://pandoc.org/) with the following command: `pandoc -f rtf -t markdown terms-of-service.rtf > terms-of-service2.md`
diff --git a/docs/assets/css/extra.css b/docs/assets/css/extra.css
@@ -5,3 +5,8 @@
 .md-typeset table:not([class]) {
     display: table;
 }
+
+/* Hide the code block title since we're using it for other purposes.*/
+.filename {
+    display: none !important
+}
diff --git a/docs/examples/load-query-and-chat-marqo.md b/docs/examples/load-query-and-chat-marqo.md
@@ -6,7 +6,7 @@ from griptape.engines import VectorQueryEngine
 from griptape.loaders import WebLoader
 from griptape.structures import Agent
 from griptape.tools import VectorStoreClient
-from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver
+from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver, OpenAiChatPromptDriver
 
 # Define the namespace
 namespace = "griptape-ai"
@@ -19,13 +19,14 @@ vector_store = MarqoVectorStoreDriver(
     embedding_driver=OpenAiEmbeddingDriver()
 )
 # Initialize the query engine
-query_engine = VectorQueryEngine(vector_store_driver=vector_store)
+query_engine = VectorQueryEngine(vector_store_driver=vector_store, prompt_driver=OpenAiChatPromptDriver(model="gpt-3.5-turbo"))
 
 # Initialize the knowledge base tool
 vector_store_tool = VectorStoreClient(
     description="Contains information about the Griptape Framework from www.griptape.ai",
     query_engine=query_engine,
     namespace=namespace,
+    off_prompt=False
 )
 
 # Load artifacts from the web

diff --git a/docs/examples/multiple-agent-shared-memory.md b/docs/examples/multiple-agent-shared-memory.md
@@ -0,0 +1,77 @@
+This example shows how to use one `Agent` to load content into `TaskMemory` and get that content from another `Agent` using `TaskMemoryClient`.
+
+The first `Agent` uses a remote vector store (`MongoDbAtlasVectorStoreDriver` in this example) to handle memory operations. The second `Agent` uses the same instance of `TaskMemory` and the `TaskMemoryClient` with the same `MongoDbAtlasVectorStoreDriver` to get the data.
+
+The `MongoDbAtlasVectorStoreDriver` assumes that you have a vector index configured where the path to the content is called `vector`, and the number of dimensions set on the index is `1536` (this is a commonly used number of dimensions for embedding models).
+
+`asker` uses the same instance of `TaskMemory` as `loader` so that `asker` has access to the `namespace_storages` that `loader` has set.
+
+```python
+import os
+from griptape.tools import WebScraper, VectorStoreClient, TaskMemoryClient
+from griptape.structures import Agent
+from griptape.drivers import AzureOpenAiChatPromptDriver, AzureOpenAiEmbeddingDriver, AzureMongoDbVectorStoreDriver
+from griptape.engines import VectorQueryEngine, PromptSummaryEngine, CsvExtractionEngine, JsonExtractionEngine
+from griptape.memory import TaskMemory 
+from griptape.artifacts import TextArtifact
+from griptape.memory.task.storage import TextArtifactStorage
+from griptape.config import StructureConfig, StructureGlobalDriversConfig
+
+
+AZURE_OPENAI_ENDPOINT = os.getenv("AZURE_OPENAI_API_BASE")
+
+MONGODB_HOST = os.getenv("MONGODB_HOST")
+MONGODB_USERNAME = os.getenv("MONGODB_USERNAME")
+MONGODB_PASSWORD = os.getenv("MONGODB_PASSWORD")
+MONGODB_DATABASE_NAME = os.getenv("MONGODB_DATABASE_NAME")
+MONGODB_COLLECTION_NAME = os.getenv("MONGODB_COLLECTION_NAME")
+MONGODB_INDEX_NAME = os.getenv("MONGODB_INDEX_NAME")
+MONGODB_VECTOR_PATH = os.getenv("MONGODB_VECTOR_PATH")
+MONGODB_CONNECTION_STRING = f"mongodb+srv://{MONGODB_USERNAME}:{MONGODB_PASSWORD}@{MONGODB_HOST}/{MONGODB_DATABASE_NAME}?tls=true&authMechanism=SCRAM-SHA-256&retrywrites=false&maxIdleTimeMS=120000"
+
+
+azure_embedding_driver = AzureOpenAiEmbeddingDriver(
+    model='text-embedding-ada-002',
+    azure_endpoint=AZURE_OPENAI_ENDPOINT,
+    azure_deployment='text-embedding-ada-002'
+)
+
+azure_prompt_driver = AzureOpenAiChatPromptDriver(
+    model='gpt-4',
+    azure_endpoint=AZURE_OPENAI_ENDPOINT,
+    azure_deployment='gpt-4'
+)
+
+mongo_driver = AzureMongoDbVectorStoreDriver(
+    connection_string=MONGODB_CONNECTION_STRING,
+    database_name=MONGODB_DATABASE_NAME,
+    collection_name=MONGODB_COLLECTION_NAME,
+    embedding_driver=azure_embedding_driver,
+    index_name=MONGODB_INDEX_NAME,
+    vector_path=MONGODB_VECTOR_PATH
+)
+
+loader = Agent(
+    tools=[
+        WebScraper()
+    ],
+    config=StructureConfig(
+        global_drivers=StructureGlobalDriversConfig(
+            prompt_driver=azure_prompt_driver,
+            vector_store_driver=mongo_driver,
+            embedding_driver=azure_embedding_driver
+        )
+    ),
+)
+asker = Agent(
+    tools=[
+        TaskMemoryClient(off_prompt=False),
+    ],
+    meta_memory=loader.meta_memory,
+    task_memory=loader.task_memory,
+)
+
+if __name__ == "__main__":
+    loader.run("Load https://medium.com/enterprise-rag/a-first-intro-to-complex-rag-retrieval-augmented-generation-a8624d70090f")
+    asker.run("why is retrieval augmented generation useful?")
+```
diff --git a/docs/examples/talk-to-a-pdf.md b/docs/examples/talk-to-a-pdf.md
@@ -9,14 +9,17 @@ from griptape.loaders import PdfLoader
 from griptape.structures import Agent
 from griptape.tools import VectorStoreClient
 from griptape.utils import Chat
-from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver
+from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver, OpenAiChatPromptDriver
 
 
 namespace = "attention"
 
 response = requests.get("https://arxiv.org/pdf/1706.03762.pdf")
 
 engine = VectorQueryEngine(
+    prompt_driver=OpenAiChatPromptDriver(
+        model="gpt-3.5-turbo",
+    ),
     vector_store_driver=LocalVectorStoreDriver(
         embedding_driver=OpenAiEmbeddingDriver(
             api_key=os.getenv("OPENAI_API_KEY")

diff --git a/docs/examples/talk-to-a-webpage.md b/docs/examples/talk-to-a-webpage.md
@@ -8,12 +8,15 @@ from griptape.rules import Ruleset, Rule
 from griptape.structures import Agent
 from griptape.tools import VectorStoreClient
 from griptape.utils import Chat
-from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver
+from griptape.drivers import LocalVectorStoreDriver, OpenAiEmbeddingDriver, OpenAiChatPromptDriver
 
 
 namespace = "physics-wiki"
 
 engine = VectorQueryEngine(
+    prompt_driver=OpenAiChatPromptDriver(
+        model="gpt-3.5-turbo",
+    ),
     vector_store_driver=LocalVectorStoreDriver(
         embedding_driver=OpenAiEmbeddingDriver(
             api_key=os.getenv("OPENAI_API_KEY")
Original file line number	Diff line number	Diff line change
Expand Up		@@ -3,3 +3,5 @@
		[![Docs](https://readthedocs.org/projects/griptape/badge/)](https://griptape.readthedocs.io)

		Docs for [Griptape](https://github.com/griptape-ai/griptape).

		Terms of Service was converted from .rtf file format using [Pandoc](https://pandoc.org/) with the following command: `pandoc -f rtf -t markdown terms-of-service.rtf > terms-of-service2.md`