-
Notifications
You must be signed in to change notification settings - Fork 80
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'master' into griffin/frontend-objects-deletino
- Loading branch information
Showing
157 changed files
with
9,861 additions
and
1,904 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,176 @@ | ||
| import Tabs from '@theme/Tabs'; | ||
| import TabItem from '@theme/TabItem'; | ||
|
|
||
| # NVIDIA NIM | ||
|
|
||
| Weave automatically tracks and logs LLM calls made via the [ChatNVIDIA](https://python.langchain.com/docs/integrations/chat/nvidia_ai_endpoints/) library, after `weave.init()` is called. | ||
|
|
||
| ## Tracing | ||
|
|
||
| It’s important to store traces of LLM applications in a central database, both during development and in production. You’ll use these traces for debugging and to help build a dataset of tricky examples to evaluate against while improving your application. | ||
|
|
||
| <Tabs groupId="programming-language"> | ||
| <TabItem value="python" label="Python" default> | ||
| Weave can automatically capture traces for the [ChatNVIDIA python library](https://python.langchain.com/docs/integrations/chat/nvidia_ai_endpoints/). | ||
|
|
||
| Start capturing by calling `weave.init(<project-name>)` with a project name your choice. | ||
|
|
||
| ```python | ||
| from langchain_nvidia_ai_endpoints import ChatNVIDIA | ||
| import weave | ||
| client = ChatNVIDIA(model="mistralai/mixtral-8x7b-instruct-v0.1", temperature=0.8, max_tokens=64, top_p=1) | ||
| # highlight-next-line | ||
| weave.init('emoji-bot') | ||
|
|
||
| messages=[ | ||
| { | ||
| "role": "system", | ||
| "content": "You are AGI. You will be provided with a message, and your task is to respond using emojis only." | ||
| }] | ||
|
|
||
| response = client.invoke(messages) | ||
| ``` | ||
|
|
||
| </TabItem> | ||
| <TabItem value="typescript" label="TypeScript"> | ||
| ```plaintext | ||
| This feature is not available in TypeScript yet since this library is only in Python. | ||
| ``` | ||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
|  | ||
|
|
||
| ## Track your own ops | ||
|
|
||
| <Tabs groupId="programming-language"> | ||
| <TabItem value="python" label="Python" default> | ||
| Wrapping a function with `@weave.op` starts capturing inputs, outputs and app logic so you can debug how data flows through your app. You can deeply nest ops and build a tree of functions that you want to track. This also starts automatically versioning code as you experiment to capture ad-hoc details that haven't been committed to git. | ||
|
|
||
| Simply create a function decorated with [`@weave.op`](/guides/tracking/ops) that calls into [ChatNVIDIA python library](https://python.langchain.com/docs/integrations/chat/nvidia_ai_endpoints/). | ||
|
|
||
| In the example below, we have 2 functions wrapped with op. This helps us see how intermediate steps, like the retrieval step in a RAG app, are affecting how our app behaves. | ||
|
|
||
| ```python | ||
| # highlight-next-line | ||
| import weave | ||
| from langchain_nvidia_ai_endpoints import ChatNVIDIA | ||
| import requests, random | ||
| PROMPT="""Emulate the Pokedex from early Pokémon episodes. State the name of the Pokemon and then describe it. | ||
| Your tone is informative yet sassy, blending factual details with a touch of dry humor. Be concise, no more than 3 sentences. """ | ||
| POKEMON = ['pikachu', 'charmander', 'squirtle', 'bulbasaur', 'jigglypuff', 'meowth', 'eevee'] | ||
| client = ChatNVIDIA(model="mistralai/mixtral-8x7b-instruct-v0.1", temperature=0.7, max_tokens=100, top_p=1) | ||
|
|
||
| # highlight-next-line | ||
| @weave.op | ||
| def get_pokemon_data(pokemon_name): | ||
| # highlight-next-line | ||
| # This is a step within your application, like the retrieval step within a RAG app | ||
| url = f"https://pokeapi.co/api/v2/pokemon/{pokemon_name}" | ||
| response = requests.get(url) | ||
| if response.status_code == 200: | ||
| data = response.json() | ||
| name = data["name"] | ||
| types = [t["type"]["name"] for t in data["types"]] | ||
| species_url = data["species"]["url"] | ||
| species_response = requests.get(species_url) | ||
| evolved_from = "Unknown" | ||
| if species_response.status_code == 200: | ||
| species_data = species_response.json() | ||
| if species_data["evolves_from_species"]: | ||
| evolved_from = species_data["evolves_from_species"]["name"] | ||
| return {"name": name, "types": types, "evolved_from": evolved_from} | ||
| else: | ||
| return None | ||
|
|
||
| # highlight-next-line | ||
| @weave.op | ||
| def pokedex(name: str, prompt: str) -> str: | ||
| # highlight-next-line | ||
| # This is your root op that calls out to other ops | ||
| # highlight-next-line | ||
| data = get_pokemon_data(name) | ||
| if not data: return "Error: Unable to fetch data" | ||
|
|
||
| messages=[ | ||
| {"role": "system","content": prompt}, | ||
| {"role": "user", "content": str(data)} | ||
| ] | ||
|
|
||
| response = client.invoke(messages) | ||
| return response.content | ||
|
|
||
| # highlight-next-line | ||
| weave.init('pokedex-nvidia') | ||
| # Get data for a specific Pokémon | ||
| pokemon_data = pokedex(random.choice(POKEMON), PROMPT) | ||
| ``` | ||
|
|
||
| Navigate to Weave and you can click `get_pokemon_data` in the UI to see the inputs & outputs of that step. | ||
| </TabItem> | ||
| <TabItem value="typescript" label="TypeScript"> | ||
| ```plaintext | ||
| This feature is not available in TypeScript yet since this library is only in Python. | ||
| ``` | ||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
|  | ||
|
|
||
| ## Create a `Model` for easier experimentation | ||
|
|
||
| <Tabs groupId="programming-language"> | ||
| <TabItem value="python" label="Python" default> | ||
| Organizing experimentation is difficult when there are many moving pieces. By using the [`Model`](/guides/core-types/models) class, you can capture and organize the experimental details of your app like your system prompt or the model you're using. This helps organize and compare different iterations of your app. | ||
|
|
||
| In addition to versioning code and capturing inputs/outputs, [`Model`](/guides/core-types/models)s capture structured parameters that control your application’s behavior, making it easy to find what parameters worked best. You can also use Weave Models with `serve`, and [`Evaluation`](/guides/core-types/evaluations)s. | ||
|
|
||
| In the example below, you can experiment with `model` and `system_message`. Every time you change one of these, you'll get a new _version_ of `GrammarCorrectorModel`. | ||
|
|
||
| ```python | ||
| import weave | ||
| from langchain_nvidia_ai_endpoints import ChatNVIDIA | ||
|
|
||
| weave.init('grammar-nvidia') | ||
|
|
||
| class GrammarCorrectorModel(weave.Model): # Change to `weave.Model` | ||
| system_message: str | ||
|
|
||
| @weave.op() | ||
| def predict(self, user_input): # Change to `predict` | ||
| client = ChatNVIDIA(model="mistralai/mixtral-8x7b-instruct-v0.1", temperature=0, max_tokens=100, top_p=1) | ||
|
|
||
| messages=[ | ||
| { | ||
| "role": "system", | ||
| "content": self.system_message | ||
| }, | ||
| { | ||
| "role": "user", | ||
| "content": user_input | ||
| } | ||
| ] | ||
|
|
||
| response = client.invoke(messages) | ||
| return response.content | ||
|
|
||
|
|
||
| corrector = GrammarCorrectorModel( | ||
| system_message = "You are a grammar checker, correct the following user input.") | ||
| result = corrector.predict("That was so easy, it was a piece of pie!") | ||
| print(result) | ||
| ``` | ||
| </TabItem> | ||
| <TabItem value="typescript" label="TypeScript"> | ||
| ```plaintext | ||
| This feature is not available in TypeScript yet since this library is only in Python. | ||
| ``` | ||
| </TabItem> | ||
| </Tabs> | ||
|
|
||
|  | ||
|
|
||
| ## Usage Info | ||
|
|
||
| The ChatNVIDIA integration supports `invoke`, `stream` and their async variants. It also supports tool use. | ||
| As ChatNVIDIA is meant to be used with many types of models, it does not have function calling support. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,31 +1,44 @@ | ||
| # Platform & Security | ||
|
|
||
| Weave is available on [W&B SaaS Cloud](https://docs.wandb.ai/guides/hosting/hosting-options/saas_cloud) which is a multi-tenant, fully-managed platform deployed in W&B's Google Cloud Platform (GCP) account in a North America region. | ||
| Weave is available on the following deployment options: | ||
|
|
||
| :::info | ||
| It's coming soon on [W&B Dedicated Cloud](https://docs.wandb.ai/guides/hosting/hosting-options/dedicated_cloud). Reach out to your W&B team if that would be of interest in your organization. | ||
| ::: | ||
| - **[W&B SaaS Cloud](https://docs.wandb.ai/guides/hosting/hosting-options/saas_cloud):** A multi-tenant, fully-managed platform deployed in W&B's Google Cloud Platform (GCP) account in a North America region. | ||
| - **[W&B Dedicated Cloud](https://docs.wandb.ai/guides/hosting/hosting-options/dedicated_cloud):** Generally available on AWS and in preview on GCP and Azure. | ||
| - **Self-managed instances:** For teams that prefer to host Weave independently, guidance is available from your W&B team to evaluate deployment options. | ||
|
|
||
| ## Identity & Access Management | ||
| ## Identity and Access Management | ||
|
|
||
| Use the identity and access management capabilities for secure authentication and effective authorization in your [W&B Organization](https://docs.wandb.ai/guides/hosting/iam/org_team_struct#organization). The following capabilities are available for Weave users in W&B SaaS Cloud: | ||
| Use the identity and access management capabilities for secure authentication and effective authorization in your [W&B Organization](https://docs.wandb.ai/guides/hosting/iam/org_team_struct#organization). The following capabilities are available for Weave users depending on your deployment option and [pricing plan](https://wandb.ai/site/pricing/): | ||
|
|
||
| * Authenticate using Single-Sign On (SSO), with available options being Google, Github, Microsoft, and [OIDC providers](https://docs.wandb.ai/guides/technical-faq/general#does-wb-support-sso-for-saas) | ||
| * [Team-based access control](https://docs.wandb.ai/guides/hosting/iam/manage-users#manage-a-team), where each team may correspond to a business unit / function, department, or a project team in your company | ||
| * Use W&B projects to organize different initiatives within a team, and configure the required [visibility scope](https://docs.wandb.ai/guides/hosting/restricted-projects) for each project | ||
| - **Authenticate using Single-Sign On (SSO):** Options include public identity providers like Google and Github, as well as enterprise providers such as Okta, Azure Active Directory, and others, [using OIDC](https://docs.wandb.ai/guides/technical-faq/general#does-wb-support-sso-for-saas). | ||
| - **[Team-based logical separation](https://docs.wandb.ai/guides/hosting/iam/manage-organization/#add-and-manage-teams):** Each team may correspond to a business unit, department, or project team within your organization. | ||
| - **Use W&B projects to organize initiatives:** Organize initiatives within teams and configure the required [visibility scope](https://docs.wandb.ai/guides/hosting/restricted-projects), including the `restricted` scope for sensitive collaborations. | ||
| - **Role-based access control:** Configure access at the [team](https://docs.wandb.ai/guides/hosting/iam/manage-organization#assign-or-update-a-team-members-role) or [project](https://docs.wandb.ai/guides/hosting/iam/restricted-projects#project-level-roles) level to ensure users access data on a need-to-know basis. | ||
| - **Scoped service accounts:** Automate Gen AI workflows using service accounts scoped to your organization or team. | ||
| - **[SCIM API and Python SDK](https://docs.wandb.ai/guides/hosting/iam/automate_iam):** Manage users and teams efficiently with SCIM API and Python SDK. | ||
|
|
||
| ## Data Security | ||
|
|
||
| In the W&B SaaS Cloud, data of all Weave users is stored in a shared cloud storage and is processed using shared compute services. The shared cloud storage is encrypted using the cloud-native encryption mechanism. When reading or writing data on behalf of a user, a security context comprising of the user's W&B organization, team and project is utilized to ensure data path isolation. | ||
| - **SaaS Cloud:** Data for all Weave users is stored in a shared Clickhouse Cloud cluster, encrypted using cloud-native encryption. Shared compute services process the data, ensuring isolation through a security context comprising your W&B organization, team, and project. | ||
|
|
||
| - **Dedicated Cloud:** Data is stored in a unique Clickhouse Cloud cluster in the cloud and region of your choice. A unique compute environment processes the data, with the following additional protections: | ||
| - **[IP allowlisting](https://docs.wandb.ai/guides/hosting/data-security/ip-allowlisting):** Authorize access to your instance from specific IP addresses. This is an optional capability. | ||
| - **[Private connectivity](https://docs.wandb.ai/guides/hosting/data-security/private-connectivity):** Route data securely through the cloud provider's private network. This is an optional capability. | ||
| - **[Data encryption](https://docs.wandb.ai/guides/hosting/data-security/data-encryption):** W&B encrypts data at rest using a unique W&B-managed encryption key. | ||
| - **Clickhouse cluster security:** W&B connects to the unique Clickhouse Cloud cluster for your Dedicated Cloud instance over the cloud provider's private network. W&B also encrypts the cluster using a unique W&B-managed encryption key, while leveraging Clickhouse's file level encryption. | ||
|
|
||
| :::note | ||
| [Secure storage connector](https://docs.wandb.ai/guides/hosting/secure-storage-connector) is not applicable to Weave. | ||
| :::important | ||
| [The W&B Platform secure storage connector or BYOB](https://docs.wandb.ai/guides/hosting/data-security/secure-storage-connector) is not available for Weave. | ||
| ::: | ||
|
|
||
| ## Maintenance | ||
| ## Maintenance | ||
|
|
||
| If you're using Weave on W&B SaaS Cloud, you do not incur the overhead and costs of provisioning and maintaining the W&B platform. It's all fully managed for you. | ||
| If you're using Weave on SaaS Cloud or Dedicated Cloud, you avoid the overhead and costs of provisioning, operating, and maintaining the W&B platform, as it is fully managed for you. | ||
|
|
||
| ## Compliance | ||
|
|
||
| Security controls for W&B SaaS Cloud are periodically audited internally and externally. Refer to the [W&B Security Portal](https://security.wandb.ai/) to request the SOC2 report and other security and compliance documents. | ||
| :::tip | ||
| To request SOC 2 reports and other security and compliance documents, refer to the [W&B Security Portal](https://security.wandb.ai/) or contact your W&B team for more information. | ||
| ::: | ||
|
|
||
| Security controls for both SaaS Cloud and Dedicated Cloud are periodically audited internally and externally. Both platforms are SOC 2 Type II compliant. Additionally, Dedicated Cloud is HIPAA-compliant for organizations managing PHI data while building Generative AI applications. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.