Merge branch 'master' into ericakdiaz/data-test-toggle-button-group

docs/docs/guides/core-types/env-vars.md

@@ -20,6 +20,7 @@ os.environ["WEAVE_PRINT_CALL_LINK"] = "false"
 
 | Variable | Type | Default | Description |
 |----------|------|---------|-------------|
+| `WANDB_API_KEY` | `string` | `None` | If set, automatically log into W&B Weave without being prompted for your API key. To generate an API key, log in to your W&B account and go to [https://wandb.ai/authorize](https://wandb.ai/authorize). |
 | `WEAVE_DISABLED` | `bool` | `false` | When set to `true`, disables all Weave tracing. Weave ops will behave like regular functions. |
 | `WEAVE_PRINT_CALL_LINK` | `bool` | `true` | Controls whether to print a link to the Weave UI when calling a Weave op. |
 | `WEAVE_CAPTURE_CODE` | `bool` | `true` | Controls whether to save code for ops so they can be reloaded for later use. |

docs/docs/guides/evaluation/scorers.md

@@ -23,11 +23,12 @@ In Weave, Scorers are used to evaluate AI outputs and return evaluation metrics.
 ## Create your own Scorers
 
 :::tip[Ready-to-Use Scorers]
-While this guide shows you how to create custom scorers, Weave comes with a variety of [predefined scorers](./builtin_scorers.mdx) that you can use right away, including:
+While this guide shows you how to create custom scorers, Weave comes with a variety of [predefined scorers](./builtin_scorers.mdx) and [local SLM scorers](./weave_local_scorers.md) that you can use right away, including:
 - [Hallucination detection](./builtin_scorers.mdx#hallucinationfreescorer)
 - [Summarization quality](./builtin_scorers.mdx#summarizationscorer)
 - [Embedding similarity](./builtin_scorers.mdx#embeddingsimilarityscorer)
-- [Relevancy evaluation](./builtin_scorers.mdx#ragas---contextrelevancyscorer)
+- [Toxicity detection (local)](./weave_local_scorers.md#weavetoxicityscorerv1)
+- [Context Relevance scoring (local)](./weave_local_scorers.md#weavecontextrelevancescorerv1)
 - And more!
 :::
 

docs/docs/guides/integrations/azure.md

@@ -1,30 +1,36 @@
 # Microsoft Azure
 
-Weights & Biases integrates with Microsoft Azure OpenAI services, helping teams to manage, debug, and optimize their Azure AI workflows at scale. This guide introduces the W&B integration, what it means for Weave users, its key features, and how to get started.
+Weights & Biases (W&B) Weave integrates with Microsoft Azure OpenAI services, helping teams to optimize their Azure AI applications. Using W&B, you can 
 
 :::tip
 For the latest tutorials, visit [Weights & Biases on Microsoft Azure](https://wandb.ai/site/partners/azure).
 :::
 
-## Key features
-
-- **LLM evaluations**: Evaluate and monitor LLM-powered applications using Weave, optimized for Azure infrastructure.  
-- **Seamless integration**: Deploy W&B Models on a dedicated Azure tenant with built-in integrations for Azure AI Studio, Azure ML, Azure OpenAI Service, and other Azure AI services.  
-- **Enhanced performance**: Use Azure’s infrastructure to train and deploy models faster, with auto-scaling clusters and optimized resources.  
-- **Scalable experiment tracking**: Automatically log hyperparameters, metrics, and artifacts for Azure AI Studio and Azure ML runs.  
-- **LLM fine-tuning**: Fine-tune models with W&B Models.
-- **Central repository for models and datasets**: Manage and version models and datasets with W&B Registry and Azure AI Studio.  
-- **Collaborative workspaces**: Support teamwork with shared workspaces, experiment commenting, and Microsoft Teams integration.  
-- **Governance framework**: Ensure security with fine-grained access controls, audit trails, and Microsoft Entra ID integration.  
-
 ## Getting started
 
-To use W&B with Azure, add the W&B integration via the [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/weightsandbiasesinc1641502883483.weights_biases_for_azure?tab=Overview).
+To get started using Azure with Weave, simply decorate the function(s) you want to track with `weave.op`.
 
-For a detailed guide describing how to integrate Azure OpenAI fine-tuning with W&B, see [Integrating Weights & Biases with Azure AI Services](https://learn.microsoft.com/en-us/azure/ai-services/openai/how-to/weights-and-biases-integration).
+```python
+@weave.op()
+def call_azure_chat(model_id: str, messages: list, max_tokens: int = 1000, temperature: float = 0.5):
+    response = client.chat.completions.create(
+        model=model_id,
+        messages=messages,
+        max_tokens=max_tokens,
+        temperature=temperature
+    )
+    return {"status": "success", "response": response.choices[0].message.content}
+
+```
 
 ## Learn more
 
-- [Weights & Biases + Microsoft Azure Overview](https://wandb.ai/site/partners/azure)
-- [How W&B and Microsoft Azure Are Empowering Enterprises](https://techcommunity.microsoft.com/blog/azure-ai-services-blog/how-weights--biases-and-microsoft-azure-are-empowering-enterprises-to-fine-tune-/4303716)
-- [Microsoft Azure OpenAI Service Documentation](https://learn.microsoft.com/en-us/azure/ai-services/openai/)
+Learn more about advanced Azure with Weave topics using the resources below.
+
+### Use the Azure AI Model Inference API with Weave
+
+Learn how to use the [Azure AI Model Inference API] with Weave to gain insights into Azure models in [this guide](https://wandb.ai/byyoung3/ML-NEWS2/reports/A-guide-to-using-the-Azure-AI-model-inference-API--Vmlldzo4OTY1MjEy#tutorial:-implementing-azure-ai-model-inference-api-with-w&b-weave-).
+
+### Trace Azure OpenAI models with Weave
+
+Learn how to trace Azure OpenAI models using Weave in [this guide](https://wandb.ai/a-sh0ts/azure-weave-cookbook/reports/How-to-use-Azure-OpenAI-and-Azure-AI-Studio-with-W-B-Weave--Vmlldzo4MTI0NDgy).  

docs/docs/guides/integrations/bedrock.md

@@ -2,14 +2,12 @@
 
 Weave automatically tracks and logs LLM calls made via Amazon Bedrock, AWS's fully managed service that offers foundation models from leading AI companies through a unified API.
 
+There are multiple ways to log LLM calls to Weave from Amazon Bedrock. You can use `weave.op` to create reusable operations for tracking any calls to a Bedrock model. Optionally, if you're using Anthropic models, you can use Weave’s built-in integration with Anthropic. 
+
 :::tip
 For the latest tutorials, visit [Weights & Biases on Amazon Web Services](https://wandb.ai/site/partners/aws/).
 :::
 
-:::note
-Do you want to experiment with Amazon Bedrock models on Weave without any set up? Try the [LLM Playground](../tools/playground.md).
-:::
-
 ## Traces
 
 Weave will automatically capture traces for Bedrock API calls. You can use the Bedrock client as usual after initializing Weave and patching the client:
@@ -143,3 +141,15 @@ print(result)
 ```
 
 This approach allows you to version your experiments and easily track different configurations of your Bedrock-based application.
+
+## Learn more
+
+Learn more about using Amazon Bedrock with Weave
+
+### Try Bedrock in the Weave Playground
+
+Do you want to experiment with Amazon Bedrock models in the Weave UI without any set up? Try the [LLM Playground](../tools/playground.md).
+
+### Report: Compare LLMs on Bedrock for text summarization with Weave
+
+The [Compare LLMs on Bedrock for text summarization with Weave](https://wandb.ai/byyoung3/ML_NEWS3/reports/Compare-LLMs-on-Amazon-Bedrock-for-text-summarization-with-W-B-Weave--VmlldzoxMDI1MTIzNw) report explains how to use Bedrock in combination with Weave to evaluate and compare LLMs for summarization tasks, code samples included.

docs/docs/guides/integrations/google-gemini.md → docs/docs/guides/integrations/google.md

@@ -1,21 +1,21 @@
-# Google Gemini
+# Google 
 
 :::tip
 For the latest tutorials, visit [Weights & Biases on Google Cloud](https://wandb.ai/site/partners/googlecloud/).
 :::
 
 :::note
-Do you want to experiment with Google Gemini models on Weave without any set up? Try the [LLM Playground](../tools/playground.md).
+Do you want to experiment with Google AI models on Weave without any set up? Try the [LLM Playground](../tools/playground.md).
 :::
 
-Google offers two ways of calling Gemini via API:
+This page describes how to use W&B Weave with the Google Vertex AI API and the Google Gemini API.
 
-1. Via the [Vertex APIs](https://cloud.google.com/vertex-ai/docs).
-2. Via the [Gemini API SDK](https://ai.google.dev/gemini-api/docs/quickstart?lang=python).
+You can use Weave to evaluate, monitor, and iterate on your Google GenAI applications. Weave automatically captures traces for the:
 
-## Tracing
+1. [Google Vertex AI API](https://cloud.google.com/vertex-ai/docs), which provides access to Google’s Gemini models and [various partner models](https://cloud.google.com/vertex-ai/generative-ai/docs/partner-models/use-partner-models).
+2. [Google Gemini API](https://ai.google.dev/gemini-api/docs/quickstart?lang=python), which is accessible via Python SDK, Node.js SDK, Go SDK, and REST.
 
-It’s important to store traces of language model applications in a central location, both during development and in production. These traces can be useful for debugging, and as a dataset that will help you improve your application.
+## Get started
 
 Weave will automatically capture traces for [Gemini API SDK](https://ai.google.dev/gemini-api/docs/quickstart?lang=python). To start tracking, calling `weave.init(project_name="<YOUR-WANDB-PROJECT-NAME>")` and use the library as normal.
 
@@ -120,3 +120,4 @@ Given a weave reference to any `weave.Model` object, you can spin up a fastapi s
 ```shell
 weave serve weave:///your_entity/project-name/YourModel:<hash>
 ```
+

docs/docs/guides/integrations/index.md

@@ -14,7 +14,7 @@ LLM providers are the vendors that offer access to large language models for gen
 - **[Anthropic](/guides/integrations/anthropic)**
 - **[Cerebras](/guides/integrations/cerebras)**
 - **[Cohere](/guides/integrations/cohere)**
-- **[Google Gemini](/guides/integrations/google-gemini)**
+- **[Google](/guides/integrations/google)**
 - **[Groq](/guides/integrations/groq)**
 - **[Hugging Face Hub](/guides/integrations/huggingface)**
 - **[LiteLLM](/guides/integrations/litellm)**

docs/docs/guides/tools/playground.md

@@ -2,7 +2,7 @@
 
 > **The LLM Playground is currently in preview.**
 
-Evaluating LLM prompts and responses is challenging. The Weave Playground is designed to simplify the process of iterating on LLM prompts and responses, making it easier to experiment with different models and prompts. With features like prompt editing, message retrying, and model comparison, Playground helps you to quickly test and improve your LLM applications. Playground currently supports OpenAI, Anthropic, Google Gemini, Groq, and Amazon Bedrock models.
+Evaluating LLM prompts and responses is challenging. The Weave Playground is designed to simplify the process of iterating on LLM prompts and responses, making it easier to experiment with different models and prompts. With features like prompt editing, message retrying, and model comparison, Playground helps you to quickly test and improve your LLM applications. Playground currently supports models from OpenAI, Anthropic, Google, Groq, Amazon Bedrock, and Microsoft Azure.
 
 ## Features
 
@@ -32,12 +32,11 @@ Before you can use Playground, you must [add provider credentials](#add-provider
 
 ### Add provider credentials and information
 
-Playground currently supports OpenAI, Anthropic, Gemini, Groq, Amazon Bedrock, and Azure models.
-To use one of the available models, add the appropriate information to your team secrets in W&B settings.
+Playground currently supports models from OpenAI, Anthropic, Google, Groq, Amazon Bedrock, and Microsoft Azure. To use one of the available models, add the appropriate information to your team secrets in W&B settings.
 
 - OpenAI: `OPENAI_API_KEY`
 - Anthropic: `ANTHROPIC_API_KEY`
-- Google Gemini: `GOOGLE_API_KEY`
+- Google: `GOOGLE_API_KEY`
 - Groq: `GEMMA_API_KEY`
 - Amazon Bedrock:
   - `AWS_ACCESS_KEY_ID`
@@ -67,7 +66,7 @@ You can switch the LLM using the dropdown menu in the top left. The available mo
 - [Amazon Bedrock](#amazon-bedrock)
 - [Anthropic](#anthropic)
 - [Azure](#azure)
-- [Google Gemini](#gemini)
+- [Google](#google)
 - [Groq](#groq)
 - [OpenAI](#openai)
 - [X.AI](#xai)
@@ -126,7 +125,7 @@ You can switch the LLM using the dropdown menu in the top left. The available mo
 - azure/gpt-4o-mini
 - azure/gpt-4o-mini-2024-07-18
 
-### [Google Gemini](../integrations/google-gemini.md)
+### [Google](../integrations/google.md)
 
 - gemini/gemini-1.5-flash-001
 - gemini/gemini-1.5-flash-002

docs/docs/guides/tracking/faqs.md

@@ -0,0 +1,53 @@
+# FAQs
+
+The following page provides answers to common questions about Weave tracing.
+
+## What information does Weave capture for a function?
+
+A function can be designated as a Weave [Op](/guides/tracking/ops) either manually through a decorator or automatically as part of an enabled integration. When an Op executes, Weave captures detailed information to support your analysis. Weave provides you with fine grained control over what is logged in case you would like something different than the default; see below for configuration examples.
+
+- **Code capture** - Weave captures a representation of the Op's source code. This includes inline comments as well as recursively capturing the value of variables or the source of non-Op functions that were called. Code capture allows you to see what your function was doing even if the change was not saved to your source control system. Code capture is used as part of Op versioning, allowing you to understand the evaluation of your code over time. If code capture is disabled, a hash value will be used instead.
+
+- **Function name, inputs, and outputs** - The name of the function will be captured but can be [overridden](/guides/tracking/tracing/#call-display-name). A JSON-based representation of the inputs and outputs will be captured. For inputs, argument name will be capture in addition to value. Weave lets you [customize the logging](/guides/tracking/ops#customize-logged-inputs-and-outputs) of inputs and outputs - you can specify a function to add/remove/modify what is logged.
+
+- **Op call hierarchy** - When an Op is called within the context of another Op executing, this relationship is captured, even in cases
+  where there is an intermediate non-Op function executing. This relationship between Op calls is used to provide a "Trace tree".
+
+- **Execution status and exceptions** - Weave tracks whether a function is executing, finished, or errored. If an exception occurs during execution the error message and a stack track is recorded.
+
+- **System information** - Weave may capture information about which operating system the client is running on including detailed version information.
+
+- **Client information** - Weave may capture information about the Weave client itself, such as the programming language in use and detailed version information for that language and the Weave client library.
+
+- **Timing** - The execution start and end time is captured and also used for latency calculations.
+
+- **Token usage** - In some [integrations](/guides/integrations/) LLM token usage counts may be automatically logged.
+
+- **User and run context** - Logging is associated with a W&B user account. That will be captured along with any wandb Run context.
+
+- **Derived information** - Weave may compute derived information from the raw information logged, for example a cost estimate may be calculated based on token usage and knowledge of the model used. Weave also aggregates some information over calls.
+
+- **Additional information you choose** - You can choose to log [custom attributes](/guides/core-types/models#track-production-calls) as part of your call or attach [feedback](/guides/tracking/feedback#add-feedback-to-a-call) to a call.
+
+## How can I disable code capture?
+
+You can disable code capture during Weave client initialization: `weave.init("entity/project", settings={"capture_code": False})`.
+You can also use the [environment variable](/guides/core-types/env-vars) `WEAVE_CAPTURE_CODE=false`.
+
+## How can I disable system information capture?
+
+You can disable system information capture during Weave client initialization: `weave.init("entity/project", settings={"capture_system_info": False})`.
+
+## How can I disable client information capture?
+
+You can disable client information capture during Weave client initialization: `weave.init("entity/project", settings={"capture_client_info": False})`.
+
+## Will Weave affect my function's execution speed?
+
+The overhead of Weave logging is typically negligible compared to making a call to an LLM.
+To minimize Weave's impact on the speed of your Op's execution, its network activity happens on a background thread.
+When your program is exiting it may appear to pause while any remaining enqueued data is logged.
+
+## How is Weave data ingestion calculated?
+
+We define ingested bytes as bytes that we receive, process, and store on your behalf. This includes trace metadata, LLM inputs/outputs, and any other information you explicitly log to Weave, but does not include communication overhead (e.g., HTTP headers) or any other data that is not placed in long-term storage. We count bytes as "ingested" only once at the time they are received and stored.