Skip to content

Latest commit

 

History

History
50 lines (38 loc) · 2.83 KB

best-practices.md

File metadata and controls

50 lines (38 loc) · 2.83 KB

Optimizing Vector Databases: Tricks and Best Practices

Querying in Vector Databases

  • Understanding the Querying Process

These are the steps that you need to follow to query a vector database:

  1. Generating Embeddings: Convert your raw data, text or media, into vectors using ML algorithms or models.
  2. Indexing: Upsert the vectors into the database and build to build your collection. All vectors of a collection must have the same dimensionality.
  3. Querying: Convert your raw query, text or media, into a vector using the same ML model. Then pass it to the database along with any metadata filters you want to apply to limit the search space and get the results.
  • Types of Queries: Similarity Search, k-NN, Range Search e.g. Grouped Documents, Metadata filtering
  1. Similarity Search: It is performed by approximation algorithms like HNSW, NMSLIB, etc and is also called k-NN search.
  2. Metadata Filtering: This is traditional search on the metadata that you ingest along with the embeddings. You can do all kinds of filtering on the metadata: range queries, exact match, full text search, etc dependending on what the database supports.
  3. Recommendations: You can do a simple recommendation search based on this formula: average_vector = avg(liked_vectors) + ( avg(liked_vectors) - avg(disliked_vectors) ) and then search for the average vector. Qdrant even provides an API for this out of the box.
  • Query Optimization: Choosing the Right Index, Balancing Precision and Speed, Cost and throughput

TODO

  • Real-World Query Examples and How to Handle Them

TODO

Performance Optimization

  • Understanding Performance Metrics in Vector Space
  • Storage Optimization: Efficient Disk Usage, Reducing I/O Overhead
  • Load Balancing and Sharding: Distributing Data and Workload
  • Hardware Considerations: Effect of Memory and CPU on Performance
  • Benchmarking and Monitoring: Tools and Techniques for Performance Tracking

Security Considerations

  • Data Encryption: Ensuring Data At Rest Security
  • Access Control: Managing User Permissions and Roles
  • Audit Logging: Tracking Data Access and Modifications
  • Securing Data Transfers: SSL/TLS, gRPC Security Features

Maintaining and Scaling Vector Databases

  • Routine Maintenance: Cleanup, Defragmentation, Periodic Re-indexing
  • Troubleshooting Common Issues: Performance Degradation, Failed Queries
  • Scalability Strategies: Scaling Up vs Scaling Out
  • Monitoring System Health: Keeping Track of System Metrics
  • Backup and Disaster Recovery: Best Practices for Data Safety

Additional Best Practices

  • Effective Use of gRPC: Optimizing Data Exchange and Communication
  • Advanced Search Techniques: Multimodal, Hybrid Search
  • Utilizing Payloads and Filters: Advanced Data Manipulation Techniques
  • Indexing Techniques: Choosing the Right Algorithm, Delayed Indexing
  • Understanding and Mitigating the "Curse of Dimensionality"