Skip to content

Expanded the _validate_anthropic_response function to include comprehensive checks #10505

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

S1LV3RJ1NX
Copy link
Collaborator

✅ Test

Changes

  • Updated unit tests to utilize the new validation function for verifying response integrity in both non-streaming and streaming scenarios.

…ehensive checks for response structure, ensuring all required fields are validated.

- Updated unit tests to utilize the new validation function for verifying response integrity in both non-streaming and streaming scenarios.
Copy link

vercel bot commented May 2, 2025

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
litellm ✅ Ready (Inspect) Visit Preview 💬 Add feedback May 3, 2025 9:45am

Copy link
Contributor

@ishaan-jaff ishaan-jaff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

mostly good, missing validation for streaming chunks. please add that

if isinstance(response, AsyncIterator):
async for chunk in response:
print("chunk=", chunk)
collected_chunks.append(chunk)
if "content" in chunk and len(chunk["content"]) > 0:
for content_block in chunk["content"]:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please can you validate the streaming chunks received

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, also using Anthropic SDK for non-streaming validation.

@krrishdholakia krrishdholakia marked this pull request as draft May 2, 2025 17:31
@krrishdholakia
Copy link
Contributor

krrishdholakia commented May 2, 2025

One approach - use anthropic sdk's pydantic object and use .model_validate() to confirm output matches expected format

For streaming, try to validate each chunk

- Upgraded Poetry version in poetry.lock to 2.0.1.
- Added the Anthropic library (version 0.50.0) to pyproject.toml.
- Enhanced unit tests for Anthropic messages to utilize the new validation method and updated model references for consistency.
- Improved response validation in tests to ensure compliance with the Anthropic Messages API structure.
@S1LV3RJ1NX
Copy link
Collaborator Author

Testcase:
image

@S1LV3RJ1NX S1LV3RJ1NX marked this pull request as ready for review May 3, 2025 09:45
@S1LV3RJ1NX S1LV3RJ1NX requested a review from ishaan-jaff May 3, 2025 09:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants