Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

chore(weave): Add more robust AsyncBatchProcessor #3795

Draft
wants to merge 12 commits into
base: master
Choose a base branch
from
Draft

chore(weave): Add more robust AsyncBatchProcessor #3795

wants to merge 12 commits into from
+1,113 −79

Conversation

andrewtruong
Copy link
Collaborator

@andrewtruong andrewtruong commented Feb 26, 2025 •
edited by coderabbitai bot
Loading

  • test
  • add thread death recovery
  • add retry
  • tests
  • test
  • test
  • test

Summary by CodeRabbit

  • New Features
    • Introduced an enhanced asynchronous batch processing engine that improves performance, stability, and error recovery.
  • Refactor
    • Restructured the processing module for improved efficiency and maintainability.
  • Tests
    • Added a comprehensive suite of tests to verify robust batch handling, error management, and recovery mechanisms.

@circle-job-mirror
Copy link

Preview this PR with FeatureBee: https://beta.wandb.ai/?betaVersion=d6fbf5fe5ac412ecee2d460be3a54dd5b46c09dd

@coderabbitai coderabbitai
Copy link

coderabbitai bot commented Feb 26, 2025 •
edited
Loading

Walkthrough

The changes refactor the asynchronous batch processing functionality by removing the old implementation from weave/trace_server/async_batch_processor.py and introducing a new version in weave/trace_server_bindings/async_batch_processor.py with enhanced features such as retry logic, timeouts, and thread recovery. Associated updates include a shift in the import path in weave/trace_server_bindings/remote_http_trace_server.py and the addition of a comprehensive test suite in tests/trace_server_bindings/test_async_batch_processor.py.

Changes

File(s) Change Summary
weave/trace_server/async_batch_processor.py
weave/trace_server_bindings/async_batch_processor.py		 Removed the old AsyncBatchProcessor class and introduced a new implementation with extended parameters (max_retries, process_timeout), enhanced error handling, thread safety, and recovery mechanisms. Added a new RetryTracker class.
weave/trace_server_bindings/remote_http_trace_server.py Updated the import statement for AsyncBatchProcessor to reference the new module path.
tests/trace_server_bindings/test_async_batch_processor.py Added a new suite of unit tests to verify functionality, covering batch processing, error handling, retries, timeouts, and thread recovery.

Sequence Diagram(s)

sequenceDiagram
    participant C as Client
    participant AP as AsyncBatchProcessor
    participant T as Processing Thread
    participant PF as Processor Function

    C->>AP: Enqueue items
    AP->>T: Check thread health / Start processing loop
    T->>AP: Retrieve batch of items
    AP->>PF: Process batch (with retries & timeout)
    PF-->>AP: Return result (success or error)
    alt Error occurs
        AP->>T: Retry or re-enqueue batch (if retries remain)
    end
    T->>AP: Continue processing next batch
Loading

Poem

I'm a rabbit, quick on the hop,
New code changes make my heart go "pop!"
Batches now dance with retries in line,
Threads recover—oh, how divine!
In this code garden, I happily play,
Hoping and coding all through the day! 🐰✨

✨ Finishing Touches
  • 📝 Generate Docstrings

Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

❤️ Share
🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>, please review it.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (Invoked using PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai generate docstrings to generate docstrings for this PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Other keywords and placeholders

  • Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
  • Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
  • Add @coderabbitai anywhere in the PR title to generate the title automatically.

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

coderabbitai[bot]
coderabbitai bot reviewed Feb 26, 2025
View reviewed changes
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (5)
weave/trace_server_bindings/async_batch_processor.py (4)

49-49: Be cautious with atexit.register for thread shutdown.

Using atexit.register(self.wait_until_all_processed) ensures a graceful shutdown in normal exit scenarios, but might not be triggered in some environments (e.g., when the process is forcibly killed). Consider adding a separate explicit shutdown path if reliability is crucial.

57-101: Re-check concurrency logic in _ensure_processing_thread_alive.

The "double-check" lock approach is sound for ensuring minimal lock contention. However, if an unexpected shard of code kills the thread repeatedly, re-enqueuing the batch inside the lock might starve other enqueues. Keep in mind the potential risk of consistently failing batches clogging the queue if they always end up back in line.

124-125: Validate the loop’s exit condition to prevent unnecessary busy waiting.

While the loop checks stop_event or a non-empty queue condition, the current approach might repeatedly spin over an empty queue if items arrive slowly. If lower CPU utilization is desired, consider using blocking queue operations or a condition notification.

226-260: Expose max_wait_time setting for better configurability.

The hard-coded max_wait_time = 10.0 in wait_until_all_processed might not fit all use cases, especially if downstream processing is time-intensive. Exposing a parameter or adjusting this limit dynamically could improve flexibility.

Would you like me to propose a refactor to allow a configurable wait time?

tests/trace_server/test_async_batch_processor.py (1)

54-54: Consider reducing reliance on fixed sleep durations in tests.

Using time.sleep(0.1) might introduce flaky behavior if system load varies. For more robust tests, consider signaling from the processor when items are dequeued or processed, eliminating the need for artificial sleeps.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between fe4d85c and feaca54.

📒 Files selected for processing (4)
  • tests/trace_server/test_async_batch_processor.py (1 hunks)
  • weave/trace_server/async_batch_processor.py (0 hunks)
  • weave/trace_server_bindings/async_batch_processor.py (1 hunks)
  • weave/trace_server_bindings/remote_http_trace_server.py (1 hunks)
💤 Files with no reviewable changes (1)
  • weave/trace_server/async_batch_processor.py
✅ Files skipped from review due to trivial changes (1)
  • weave/trace_server_bindings/remote_http_trace_server.py
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Focus on pythonic code patterns. Check for proper...

**/*.py: Focus on pythonic code patterns.
Check for proper exception handling.
Verify type hints usage where applicable.
Look for potential performance improvements.
Don't comment on formatting if black/isort is configured.
Check for proper dependency injection patterns.
Verify proper async handling if applicable.

  • weave/trace_server_bindings/async_batch_processor.py
  • tests/trace_server/test_async_batch_processor.py
⏰ Context from checks skipped due to timeout of 90000ms (40)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, trace)
🔇 Additional comments (4)
weave/trace_server_bindings/async_batch_processor.py (1)

8-8: Consider potential issues with importing test-only utilities in production code.

Importing get_raise_on_captured_errors from weave.trace.context.tests_context might inadvertently introduce test-only behavior or dependencies into production code. Verify that this is intentional and won't cause problems outside of tests.

tests/trace_server/test_async_batch_processor.py (3)

129-131: Good use of a blocking processor to test queue growth.

This test effectively demonstrates how the queue can grow while the processor is blocked. The scenario is well-covered, ensuring the system can handle temporary stalls.

373-377: Appreciate the thorough retry coverage in timeouts.

These parameters (process_timeout=0.1, max_retries=2) accurately test the handling of items that hang or need re-attempts. The test is both comprehensive and clear, ensuring that partial batch failures and timeouts are properly validated.

385-425: Nicely detailed checks for completed and retried items.

The logic verifying the final state—especially ensuring that always-hanging items never complete—is a solid approach to guarantee that the timeout and retries logic is behaving as intended.

coderabbitai[bot]
coderabbitai bot reviewed Feb 27, 2025
View reviewed changes
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (9)
tests/trace_server_bindings/test_async_batch_processor.py (5)

42-60: Consider avoiding time.sleep(0.1) for synchronization.
Using an event or a condition variable can reduce the risk of flakiness on slower or overloaded environments.

112-174: Consider using condition variables instead of time.sleep(0.2).
Sleeping for a fixed duration can introduce test flakiness in slower CI pipelines; event-based synchronization is more robust.

176-237: Be aware of using SystemExit to kill the thread.
This approach works but can cause discrepancies if the runtime environment changes or intercepts SystemExit differently.

239-300: Thread death with retries looks sound.
However, using time-based sleeps can lead to sporadic failures in resource-constrained environments.

303-425: Test covers many scenarios in one place.
Consider splitting or parameterizing to simplify future maintenance and debugging.

weave/trace_server_bindings/async_batch_processor.py (4)

26-63: Revisit atexit.register(self.wait_until_all_processed).
Depending on atexit may not gracefully handle abrupt interpreter shutdowns. Consider offering a more explicit shutdown hook.

65-106: Enhance debugging details.
When restarting the thread after failure, consider logging the exception that caused the thread to die, aiding troubleshooting.

119-227: Consider an event-driven loop instead of time.sleep(min_batch_interval).
An event-based approach can reduce idle waiting and improve responsiveness for high-throughput scenarios.

228-263: Hardcoded shutdown timeout could be user-configurable.
Allowing customization of max_wait_time may help in diverse environments.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between feaca54 and 7cbc1dd.

📒 Files selected for processing (2)
  • tests/trace_server_bindings/test_async_batch_processor.py (1 hunks)
  • weave/trace_server_bindings/async_batch_processor.py (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Focus on pythonic code patterns. Check for proper...

**/*.py: Focus on pythonic code patterns.
Check for proper exception handling.
Verify type hints usage where applicable.
Look for potential performance improvements.
Don't comment on formatting if black/isort is configured.
Check for proper dependency injection patterns.
Verify proper async handling if applicable.

  • weave/trace_server_bindings/async_batch_processor.py
  • tests/trace_server_bindings/test_async_batch_processor.py
⏰ Context from checks skipped due to timeout of 90000ms (767)
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, notdiamond)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, huggingface)
  • GitHub Check: Trace nox tests (3, 11, groq)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, cerebras)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, anthropic)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, pandas-test)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, vertexai)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, mistral0)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, groq)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cohere)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, anthropic)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, notdiamond)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, huggingface)
  • GitHub Check: Trace nox tests (3, 11, groq)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, anthropic)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, pandas-test)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, vertexai)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, mistral0)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, groq)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cohere)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, anthropic)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, notdiamond)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, huggingface)
  • GitHub Check: Trace nox tests (3, 11, groq)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, anthropic)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, pandas-test)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, vertexai)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, mistral0)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, groq)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cohere)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, notdiamond)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, pandas-test)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, vertexai)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, groq)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cohere)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, notdiamond)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, pandas-test)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cohere)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, notdiamond)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, pandas-test)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cohere)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
  • GitHub Check: notify-wandb-core
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, notdiamond)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, pandas-test)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cohere)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
  • GitHub Check: notify-wandb-core
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, notdiamond)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
  • GitHub Check: notify-wandb-core
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
  • GitHub Check: notify-wandb-core
  • GitHub Check: Legacy (Query Service) Python unit tests (1)
  • GitHub Check: Legacy (Query Service) Python unit tests (0)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, litellm)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 9, langchain)
🔇 Additional comments (7)
tests/trace_server_bindings/test_async_batch_processor.py (4)

12-24: Well-structured test.
Validates basic enqueuing and processing in a single batch.

26-40: Proper batch size testing.
Confirms logic for splitting items across multiple batches is correct.

62-73: No concerns with empty batch handling.
The test properly confirms no unnecessary batch processing is triggered.

75-110: Good error-handling coverage.
Ensures that subsequent batches continue processing even after a batch failure.

weave/trace_server_bindings/async_batch_processor.py (3)

17-21: RetryTracker is clear and concise.
No issues found with this straightforward dataclass.

107-118: Enqueue method is straightforward.
Gracefully ignores empty lists and ensures the thread is alive.

264-273: Batch retrieval logic is clean.
The _safely_get_batch function is concise and effectively handles empty queues.

coderabbitai[bot]
coderabbitai bot reviewed Feb 27, 2025
View reviewed changes
Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (7)
weave/trace_server_bindings/async_batch_processor.py (2)

65-106: Consider adding more detailed error logging for thread death recovery

The thread recovery logic is well-designed with clear comments outlining the recovery strategy. However, it would be helpful to add more context to the log messages when a thread dies.

Consider enhancing logging with more details:

-            logger.info("Processing thread died, restarting...")
+            logger.info("Processing thread died, restarting... Thread name: %s, Thread ID: %s", 
+                       self.processing_thread.name, self.processing_thread.ident)

Similar improvements could be made to the retry and drop log messages to include more context about the items.

227-261: Consider making shutdown timeout configurable

The current hard-coded timeout of 10.0 seconds during shutdown might not be enough for large queues or slow processor functions.

Make the shutdown timeout configurable:

-        max_wait_time = 10.0  # Hard timeout to prevent indefinite hanging
+        max_wait_time = getattr(self, 'shutdown_timeout', 10.0)  # Hard timeout to prevent indefinite hanging

Then add a parameter to the __init__ method:

def __init__(
    self,
    processor_fn: Callable[[list[T]], None],
    max_batch_size: int = 100,
    min_batch_interval: float = 1.0,
    max_retries: int = 3,
    process_timeout: float = 30.0,
    shutdown_timeout: float = 10.0,
) -> None:
    # ...
    self.shutdown_timeout = shutdown_timeout
tests/trace_server_bindings/test_async_batch_processor.py (5)

43-61: Consider using an event instead of sleep for test determinism

While the test works as is, using time.sleep() can make tests flaky depending on system load.

Replace the sleep with an event-based approach for better test determinism:

-    # Sleep briefly to ensure items are all enqueued before processing
-    time.sleep(0.1)
+    # Use an event to ensure items are all processed together
+    processing_started = threading.Event()
+    original_processor_fn = processor_fn
+    
+    def instrumented_processor_fn(items):
+        processing_started.set()
+        return original_processor_fn(items)
+    
+    processor.processor_fn = instrumented_processor_fn
+    
+    # Wait for processing to start
+    processing_started.wait(timeout=1.0)

333-456: Consider optimizing the timeout test for faster execution

The timeout test is comprehensive but might run slower than necessary due to fixed timeouts.

Consider reducing the timeout values for faster test execution:

-        process_timeout=0.1,
+        process_timeout=0.05,

This would still validate the timeout behavior while allowing the test to complete faster.

525-647: Consider adding a test for extreme load conditions

While the current shutdown test is good, it would be beneficial to test with an extremely high number of items or very slow processing to validate behavior under extreme conditions.

Consider adding a test that:

  1. Simulates an extremely slow processor (e.g., several seconds per batch)
  2. Enqueues thousands of items
  3. Validates that shutdown behavior is still predictable, even if some items are dropped due to timeout

This would help ensure the processor behaves predictably under extreme load.

649-704: Improve memory pressure test with actual memory measurements

The current memory pressure test doesn't actually measure memory usage.

Use psutil to get actual memory measurements:

-    # Capture memory stats before
-    gc.collect()
-    memory_before = (
-        0  # This is a placeholder as we can't reliably measure memory in Python
-    )
+    # Capture memory stats before
+    gc.collect()
+    try:
+        import psutil
+        memory_before = psutil.Process().memory_info().rss
+    except (ImportError, AttributeError):
+        # Fallback if psutil isn't available
+        memory_before = 0

# ... later in the test

-    # Capture memory stats after
-    gc.collect()
-    memory_after = (
-        0  # This is a placeholder as we can't reliably measure memory in Python
-    )
+    # Capture memory stats after
+    gc.collect()
+    try:
+        import psutil
+        memory_after = psutil.Process().memory_info().rss
+        assert memory_after - memory_before < large_item_size * 3, "Memory usage grew more than expected"
+    except (ImportError, AttributeError):
+        # Fallback
+        memory_after = 0

This would provide actual memory measurements if psutil is available.

762-841: Consider adding thread name validation in resource cleanup test

The resource cleanup test verifies that no thread leaks occur, but it could be more precise.

Add validation that specifically checks for the absence of threads created by the processor:

+    # Get names of all threads before test
+    thread_names_before = {t.name for t in threading.enumerate()}
+
     # Count active threads before test
     thread_count_before = threading.active_count()

# ... (existing test code)

     # Wait a short time for threads to clean up
     time.sleep(0.2)
+
+    # Get names of all threads after test
+    thread_names_after = {t.name for t in threading.enumerate()}
+    # Look for any new threads with the processor's class name in them
+    new_processor_threads = {
+        name for name in thread_names_after - thread_names_before
+        if "AsyncBatchProcessor" in name
+    }
+    assert not new_processor_threads, f"Processor threads weren't cleaned up: {new_processor_threads}"

     # Count active threads after test
     thread_count_after = threading.active_count()

This would provide a more precise check that specifically validates that no AsyncBatchProcessor threads remain.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 7cbc1dd and 50a96bd.

📒 Files selected for processing (2)
  • tests/trace_server_bindings/test_async_batch_processor.py (1 hunks)
  • weave/trace_server_bindings/async_batch_processor.py (1 hunks)
🧰 Additional context used
📓 Path-based instructions (1)
`**/*.py`: Focus on pythonic code patterns. Check for proper...

**/*.py: Focus on pythonic code patterns.
Check for proper exception handling.
Verify type hints usage where applicable.
Look for potential performance improvements.
Don't comment on formatting if black/isort is configured.
Check for proper dependency injection patterns.
Verify proper async handling if applicable.

  • tests/trace_server_bindings/test_async_batch_processor.py
  • weave/trace_server_bindings/async_batch_processor.py
⏰ Context from checks skipped due to timeout of 90000ms (684)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, groq)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, cerebras)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, pandas-test)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, vertexai)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, mistral0)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, groq)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cohere)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, bedrock)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, groq)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, cerebras)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, pandas-test)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, vertexai)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, groq)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cohere)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, bedrock)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, groq)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, cerebras)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, vertexai)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, groq)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, bedrock)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: test-query-service-matrix-check
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, pandas-test)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, cerebras)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, bedrock)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: test-query-service-matrix-check
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, google_ai_studio)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral1)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cohere)
  • GitHub Check: Trace nox tests (3, 11, cerebras)
  • GitHub Check: Trace nox tests (3, 11, bedrock)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, instructor)
  • GitHub Check: Trace nox tests (3, 10, huggingface)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, bedrock)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, anthropic)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cerebras)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, cerebras)
  • GitHub Check: Trace nox tests (3, 10, bedrock)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, instructor)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cerebras)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, mistral0)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, litellm)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, cerebras)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, dspy)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, vertexai)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
  • GitHub Check: Trace nox tests (3, 13, pandas-test)
  • GitHub Check: Trace nox tests (3, 13, vertexai)
  • GitHub Check: Trace nox tests (3, 13, openai)
  • GitHub Check: Trace nox tests (3, 13, mistral1)
  • GitHub Check: Trace nox tests (3, 13, mistral0)
  • GitHub Check: Trace nox tests (3, 13, llamaindex)
  • GitHub Check: Trace nox tests (3, 13, instructor)
  • GitHub Check: Trace nox tests (3, 13, huggingface)
  • GitHub Check: Trace nox tests (3, 13, groq)
  • GitHub Check: Trace nox tests (3, 13, cerebras)
  • GitHub Check: Trace nox tests (3, 13, trace_server)
  • GitHub Check: Trace nox tests (3, 13, trace)
  • GitHub Check: Trace nox tests (3, 12, pandas-test)
  • GitHub Check: Trace nox tests (3, 12, scorers)
  • GitHub Check: Trace nox tests (3, 12, vertexai)
  • GitHub Check: Trace nox tests (3, 12, openai)
  • GitHub Check: Trace nox tests (3, 12, notdiamond)
  • GitHub Check: Trace nox tests (3, 12, mistral1)
  • GitHub Check: Trace nox tests (3, 12, llamaindex)
  • GitHub Check: Trace nox tests (3, 12, litellm)
  • GitHub Check: Trace nox tests (3, 12, langchain)
  • GitHub Check: Trace nox tests (3, 12, instructor)
  • GitHub Check: Trace nox tests (3, 12, huggingface)
  • GitHub Check: Trace nox tests (3, 12, groq)
  • GitHub Check: Trace nox tests (3, 12, dspy)
  • GitHub Check: Trace nox tests (3, 12, cohere)
  • GitHub Check: Trace nox tests (3, 12, cerebras)
  • GitHub Check: Trace nox tests (3, 12, bedrock)
  • GitHub Check: Trace nox tests (3, 12, trace_server)
  • GitHub Check: Trace nox tests (3, 12, trace)
  • GitHub Check: Trace nox tests (3, 11, scorers)
  • GitHub Check: Trace nox tests (3, 11, openai)
  • GitHub Check: Trace nox tests (3, 11, notdiamond)
  • GitHub Check: Trace nox tests (3, 11, mistral0)
  • GitHub Check: Trace nox tests (3, 11, llamaindex)
  • GitHub Check: Trace nox tests (3, 11, langchain)
  • GitHub Check: Trace nox tests (3, 11, trace_server)
  • GitHub Check: Trace nox tests (3, 11, trace)
  • GitHub Check: Trace nox tests (3, 10, scorers)
  • GitHub Check: Trace nox tests (3, 10, openai)
  • GitHub Check: Trace nox tests (3, 10, notdiamond)
  • GitHub Check: Trace nox tests (3, 10, mistral1)
  • GitHub Check: Trace nox tests (3, 10, llamaindex)
  • GitHub Check: Trace nox tests (3, 10, litellm)
  • GitHub Check: Trace nox tests (3, 10, langchain)
  • GitHub Check: Trace nox tests (3, 10, dspy)
  • GitHub Check: Trace nox tests (3, 10, trace_server)
  • GitHub Check: Trace nox tests (3, 10, trace)
  • GitHub Check: Trace nox tests (3, 9, pandas-test)
  • GitHub Check: Trace nox tests (3, 9, scorers)
  • GitHub Check: Trace nox tests (3, 9, openai)
  • GitHub Check: Trace nox tests (3, 9, notdiamond)
  • GitHub Check: Trace nox tests (3, 9, mistral1)
  • GitHub Check: Trace nox tests (3, 9, mistral0)
  • GitHub Check: Trace nox tests (3, 9, llamaindex)
  • GitHub Check: Trace nox tests (3, 9, litellm)
🔇 Additional comments (11)
weave/trace_server_bindings/async_batch_processor.py (4)

1-58: Good initialization approach with configurable parameters

The class has a well-documented initialization with good default values. The use of atexit.register ensures proper cleanup even if the user forgets to call wait_until_all_processed.

I particularly like the clear docstrings explaining the purpose of each parameter and the proper use of type hints.

59-64: LGTM - Thread initialization looks good

The thread creation logic is concise and correctly sets the thread as a daemon thread, which is important for preventing the application from hanging on exit.

107-117: LGTM - Proper thread safety in enqueue method

The enqueue method correctly checks for thread health and uses proper locking to ensure thread safety when adding items to the queue.

263-271: LGTM - Efficient batch collection from queue

The _safely_get_batch helper function is well-implemented and efficiently collects items from the queue without blocking unnecessarily.

tests/trace_server_bindings/test_async_batch_processor.py (7)

13-25: LGTM - Good basic test coverage

The basic functionality test appropriately verifies that items are properly enqueued and processed.

27-41: LGTM - Thorough batch size limit testing

This test effectively verifies the batch size limit functionality by checking that items are correctly split into batches of the specified size.

76-138: LGTM - Comprehensive error handling test

This test covers multiple important error handling scenarios, including proper continuation of processing after errors and error logging. The test is well-structured with clear comments explaining the test logic.

140-202: LGTM - Well-designed blocking test

The test for processor blocking effectively demonstrates the queue behavior when the processor function blocks. It's well-structured with distinct test phases and clear verifications at each step.

204-268: LGTM - Thread death recovery tests

Both thread death recovery tests (with and without retries) thoroughly validate the recovery mechanism. The use of events makes the tests deterministic, which is important for threading tests.

Also applies to: 270-331

458-523: LGTM - Concurrent modification test handles race conditions

This test effectively validates the processor's ability to handle items being enqueued while processing is ongoing, including concurrent enqueueing from multiple threads.

707-760: LGTM - Good cancellation handling test

The test effectively validates that cancellation during processing allows in-progress items to complete while ensuring no new processing is initiated after cancellation.

weave/trace_server_bindings/async_batch_processor.py
Comment on lines +118 to +226
"""
while not self.stop_event.is_set() or not self.queue.empty():
try:
# Safely collect a batch of items from the queue
current_batch: list[RetryTracker[T]] = _safely_get_batch(
self.queue, self.max_batch_size
)
if not current_batch:
continue

# Keep track of the current batch in case of thread death
with self.lock:
self.current_batch = current_batch

# Extract the actual items from the trackers for processing
items_to_process = [tracker.item for tracker in current_batch]

processed = False # Flag to track if batch was processed

if self.process_timeout > 0:
# If a timeout is set, use a separate thread to enforce the timeout.
# This is necessary because the processing function may block indefinitely.
processing_completed = Event()
processing_error = None

def process_with_timeout() -> None:
nonlocal processing_error

try:
self.processor_fn(items_to_process)
except Exception as e:
processing_error = e
processing_completed.set()
else:
processing_completed.set()

# Start and wait for processing to complete or timeout
processing_thread = Thread(target=process_with_timeout)
processing_thread.daemon = True
processing_thread.start()
processing_success = processing_completed.wait(self.process_timeout)

# Case 1: Processing timed out
if not processing_success:
logger.info(
f"Processing batch of {len(current_batch)} items timed out after {self.process_timeout}s. "
f"Moving to next batch. Items will be retried if retry attempts remain."
)
# Queue items back up for retry if they haven't exceeded max retries
with self.lock:
for tracker in current_batch:
if tracker.retry_count < self.max_retries:
tracker.retry_count += 1
self.queue.put(tracker)
else:
logger.error(
f"Item processing timed out and exceeded max retries ({self.max_retries}). "
f"Dropping item."
)
self.current_batch = []

# Case 2: Processing completed with an error
elif processing_error is not None:
with self.lock:
self.current_batch = []
if get_raise_on_captured_errors():
raise processing_error
logger.exception(f"Error processing batch: {processing_error}")

# Case 3: Processing completed successfully
else:
with self.lock:
self.current_batch = []

processed = True # Mark as processed regardless of outcome

# Only process if not already processed with timeout mechanism.
if not processed:
try:
self.processor_fn(items_to_process)
except Exception as e:
# Clear the current batch even on error to avoid retrying indefinitely
with self.lock:
self.current_batch = []
if get_raise_on_captured_errors():
raise
logger.exception(f"Error processing batch: {e}")
else:
# If we succeed, then clear it out ahead of the next one
with self.lock:
self.current_batch = []

# Rate limiting to prevent CPU overuse on empty/small queues
if not self.stop_event.is_set():
time.sleep(self.min_batch_interval)
except Exception as e:
# SystemExit, KeyboardInterrupt, etc. will still kill the thread
logger.exception(f"Unexpected error in processing thread: {e}")
break # Thread death will trigger recovery on next operation

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Potential race condition in batch processing and timeout handling

The timeout handling implementation is thorough, but there's a subtle issue with the two separate execution paths.

If an exception occurs during processing in the timeout thread but after the timeout, it might not be properly handled. Consider restructuring the code to have a single execution path:

-                processed = False  # Flag to track if batch was processed
-
-                if self.process_timeout > 0:
-                    # If a timeout is set, use a separate thread to enforce the timeout.
-                    # This is necessary because the processing function may block indefinitely.
-                    processing_completed = Event()
-                    processing_error = None
-
-                    def process_with_timeout() -> None:
-                        nonlocal processing_error
-
-                        try:
-                            self.processor_fn(items_to_process)
-                        except Exception as e:
-                            processing_error = e
-                            processing_completed.set()
-                        else:
-                            processing_completed.set()
-
-                    # Start and wait for processing to complete or timeout
-                    processing_thread = Thread(target=process_with_timeout)
-                    processing_thread.daemon = True
-                    processing_thread.start()
-                    processing_success = processing_completed.wait(self.process_timeout)
-
-                    # Case 1: Processing timed out
-                    if not processing_success:
-                        logger.info(
-                            f"Processing batch of {len(current_batch)} items timed out after {self.process_timeout}s. "
-                            f"Moving to next batch. Items will be retried if retry attempts remain."
-                        )
-                        # Queue items back up for retry if they haven't exceeded max retries
-                        with self.lock:
-                            for tracker in current_batch:
-                                if tracker.retry_count < self.max_retries:
-                                    tracker.retry_count += 1
-                                    self.queue.put(tracker)
-                                else:
-                                    logger.error(
-                                        f"Item processing timed out and exceeded max retries ({self.max_retries}). "
-                                        f"Dropping item."
-                                    )
-                            self.current_batch = []
-
-                    # Case 2: Processing completed with an error
-                    elif processing_error is not None:
-                        with self.lock:
-                            self.current_batch = []
-                        if get_raise_on_captured_errors():
-                            raise processing_error
-                        logger.exception(f"Error processing batch: {processing_error}")
-
-                    # Case 3: Processing completed successfully
-                    else:
-                        with self.lock:
-                            self.current_batch = []
-
-                    processed = True  # Mark as processed regardless of outcome
-
-                # Only process if not already processed with timeout mechanism.
-                if not processed:
-                    try:
-                        self.processor_fn(items_to_process)
-                    except Exception as e:
-                        # Clear the current batch even on error to avoid retrying indefinitely
-                        with self.lock:
-                            self.current_batch = []
-                        if get_raise_on_captured_errors():
-                            raise
-                        logger.exception(f"Error processing batch: {e}")
-                    else:
-                        # If we succeed, then clear it out ahead of the next one
-                        with self.lock:
-                            self.current_batch = []
+                # Unified processing approach with or without timeout
+                try:
+                    if self.process_timeout > 0:
+                        # If a timeout is set, use a separate thread to enforce the timeout.
+                        processing_completed = Event()
+                        processing_error = None
+
+                        def process_with_timeout() -> None:
+                            nonlocal processing_error
+
+                            try:
+                                self.processor_fn(items_to_process)
+                            except Exception as e:
+                                processing_error = e
+                            finally:
+                                processing_completed.set()
+
+                        # Start and wait for processing to complete or timeout
+                        processing_thread = Thread(target=process_with_timeout)
+                        processing_thread.daemon = True
+                        processing_thread.start()
+                        processing_success = processing_completed.wait(self.process_timeout)
+
+                        # Case 1: Processing timed out
+                        if not processing_success:
+                            logger.info(
+                                f"Processing batch of {len(current_batch)} items timed out after {self.process_timeout}s. "
+                                f"Moving to next batch. Items will be retried if retry attempts remain."
+                            )
+                            # Queue items back up for retry if they haven't exceeded max retries
+                            with self.lock:
+                                for tracker in current_batch:
+                                    if tracker.retry_count < self.max_retries:
+                                        tracker.retry_count += 1
+                                        self.queue.put(tracker)
+                                    else:
+                                        logger.error(
+                                            f"Item processing timed out and exceeded max retries ({self.max_retries}). "
+                                            f"Dropping item."
+                                        )
+                                self.current_batch = []
+                            # Return early to skip cleanup since we've already handled it
+                            continue
+
+                        # Case 2: Processing completed with an error
+                        if processing_error is not None:
+                            raise processing_error
+                    else:
+                        # No timeout, process directly
+                        self.processor_fn(items_to_process)
+                except Exception as e:
+                    # Clear the current batch even on error to avoid retrying indefinitely
+                    with self.lock:
+                        self.current_batch = []
+                    if get_raise_on_captured_errors():
+                        raise
+                    logger.exception(f"Error processing batch: {e}")
+                else:
+                    # If we succeed, then clear it out ahead of the next one
+                    with self.lock:
+                        self.current_batch = []

This unified approach eliminates the duplicate code paths and ensures consistent error handling.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def _process_batches(self) -> None:
"""Main thread loop that processes batches from the queue.
Thread safety approach:
1. Collect batch items outside lock when possible
2. Update shared state (current_batch) under lock
3. Process batch outside lock to avoid blocking other operations
4. Only clear batch tracking on success
5. Implement timeout mechanism to prevent indefinite blocking
"""
while not self.stop_event.is_set() or not self.queue.empty():
try:
# Safely collect a batch of items from the queue
current_batch: list[RetryTracker[T]] = _safely_get_batch(
self.queue, self.max_batch_size
)
if not current_batch:
continue
# Keep track of the current batch in case of thread death
with self.lock:
self.current_batch = current_batch
# Extract the actual items from the trackers for processing
items_to_process = [tracker.item for tracker in current_batch]
processed = False # Flag to track if batch was processed
if self.process_timeout > 0:
# If a timeout is set, use a separate thread to enforce the timeout.
# This is necessary because the processing function may block indefinitely.
processing_completed = Event()
processing_error = None
def process_with_timeout() -> None:
nonlocal processing_error
try:
self.processor_fn(items_to_process)
except Exception as e:
processing_error = e
processing_completed.set()
else:
processing_completed.set()
# Start and wait for processing to complete or timeout
processing_thread = Thread(target=process_with_timeout)
processing_thread.daemon = True
processing_thread.start()
processing_success = processing_completed.wait(self.process_timeout)
# Case 1: Processing timed out
if not processing_success:
logger.info(
f"Processing batch of {len(current_batch)} items timed out after {self.process_timeout}s. "
f"Moving to next batch. Items will be retried if retry attempts remain."
)
# Queue items back up for retry if they haven't exceeded max retries
with self.lock:
for tracker in current_batch:
if tracker.retry_count < self.max_retries:
tracker.retry_count += 1
self.queue.put(tracker)
else:
logger.error(
f"Item processing timed out and exceeded max retries ({self.max_retries}). "
f"Dropping item."
)
self.current_batch = []
# Case 2: Processing completed with an error
elif processing_error is not None:
with self.lock:
self.current_batch = []
if get_raise_on_captured_errors():
raise processing_error
logger.exception(f"Error processing batch: {processing_error}")
# Case 3: Processing completed successfully
else:
with self.lock:
self.current_batch = []
processed = True # Mark as processed regardless of outcome
# Only process if not already processed with timeout mechanism.
if not processed:
try:
self.processor_fn(items_to_process)
except Exception as e:
# Clear the current batch even on error to avoid retrying indefinitely
with self.lock:
self.current_batch = []
if get_raise_on_captured_errors():
raise
logger.exception(f"Error processing batch: {e}")
else:
# If we succeed, then clear it out ahead of the next one
with self.lock:
self.current_batch = []
# Rate limiting to prevent CPU overuse on empty/small queues
if not self.stop_event.is_set():
time.sleep(self.min_batch_interval)
except Exception as e:
# SystemExit, KeyboardInterrupt, etc. will still kill the thread
logger.exception(f"Unexpected error in processing thread: {e}")
break # Thread death will trigger recovery on next operation
def _process_batches(self) -> None:
"""Main thread loop that processes batches from the queue.
Thread safety approach:
1. Collect batch items outside lock when possible
2. Update shared state (current_batch) under lock
3. Process batch outside lock to avoid blocking other operations
4. Only clear batch tracking on success
5. Implement timeout mechanism to prevent indefinite blocking
"""
while not self.stop_event.is_set() or not self.queue.empty():
try:
# Safely collect a batch of items from the queue
current_batch: list[RetryTracker[T]] = _safely_get_batch(
self.queue, self.max_batch_size
)
if not current_batch:
continue
# Keep track of the current batch in case of thread death
with self.lock:
self.current_batch = current_batch
# Extract the actual items from the trackers for processing
items_to_process = [tracker.item for tracker in current_batch]
# Unified processing approach with or without timeout
try:
if self.process_timeout > 0:
# If a timeout is set, use a separate thread to enforce the timeout.
processing_completed = Event()
processing_error = None
def process_with_timeout() -> None:
nonlocal processing_error
try:
self.processor_fn(items_to_process)
except Exception as e:
processing_error = e
finally:
processing_completed.set()
# Start and wait for processing to complete or timeout
processing_thread = Thread(target=process_with_timeout)
processing_thread.daemon = True
processing_thread.start()
processing_success = processing_completed.wait(self.process_timeout)
# Case 1: Processing timed out
if not processing_success:
logger.info(
f"Processing batch of {len(current_batch)} items timed out after {self.process_timeout}s. "
f"Moving to next batch. Items will be retried if retry attempts remain."
)
# Queue items back up for retry if they haven't exceeded max retries
with self.lock:
for tracker in current_batch:
if tracker.retry_count < self.max_retries:
tracker.retry_count += 1
self.queue.put(tracker)
else:
logger.error(
f"Item processing timed out and exceeded max retries ({self.max_retries}). "
f"Dropping item."
)
self.current_batch = []
# Return early to skip cleanup since we've already handled it
continue
# Case 2: Processing completed with an error
if processing_error is not None:
raise processing_error
else:
# No timeout, process directly
self.processor_fn(items_to_process)
except Exception as e:
# Clear the current batch even on error to avoid retrying indefinitely
with self.lock:
self.current_batch = []
if get_raise_on_captured_errors():
raise
logger.exception(f"Error processing batch: {e}")
else:
# If we succeed, then clear it out ahead of the next one
with self.lock:
self.current_batch = []
# Rate limiting to prevent CPU overuse on empty/small queues
if not self.stop_event.is_set():
time.sleep(self.min_batch_interval)
except Exception as e:
# SystemExit, KeyboardInterrupt, etc. will still kill the thread
logger.exception(f"Unexpected error in processing thread: {e}")
break # Thread death will trigger recovery on next operation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@andrewtruong