Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Create and Document Comprehensive Comparison on available VectorDBs in Llama Stack. #1824

Open
franciscojavierarceo opened this issue Mar 28, 2025 · 0 comments
Assignees
Labels
enhancement New feature or request RAG Relates to RAG functionality of the agents API

Comments

@franciscojavierarceo
Copy link
Collaborator

franciscojavierarceo commented Mar 28, 2025

🚀 Describe the new functionality needed

As mentioned in #1165 (comment), there's interest in creating a comprehensive benchmark of VectorDBs for Llama Stack.

The folks at @zilliztech have created https://github.com/zilliztech/VectorDBBench which is an extensive comparison.

In #1665 we also discussed evaluation of retrieval and potentially using the CISI dataset for Information Retrieval for benchmarking but the Issue was scoped down to focus on the read and write performance to highlight the tradeoffs to users.

This ticket is more comprehensive and should have subtickets to coordinate the complexity involved.

💡 Why is this needed? What if we don't build it?

This is needed to give more information to users when using VectorDB providers, as it'll influence default choices for inline and what providers we recommend to users for remote.

Other thoughts

N/A

@franciscojavierarceo franciscojavierarceo added enhancement New feature or request RAG Relates to RAG functionality of the agents API labels Mar 28, 2025
@franciscojavierarceo franciscojavierarceo self-assigned this Mar 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request RAG Relates to RAG functionality of the agents API
Projects
None yet
Development

No branches or pull requests

1 participant