Add hub integration #58

davidberenstein1957 · 2025-01-28T21:04:31Z

I added a basic integration for the Hub.

import numpy as np

from vicinity import Backend, Metric, Vicinity

# Create some dummy data
items = [
    {"name": "triforce", "id": 0},
    {"name": "master sword", "id": 1},
    {"name": "hylian shield", "id": 2},
    {"name": "boomerang", "id": 3},
    {"name": "hookshot", "id": 4},
]
vectors = np.random.rand(len(items), 128)

# Initialize the Vicinity instance (using the basic backend and cosine metric)
vicinity = Vicinity.from_vectors_and_items(vectors=vectors, items=items)
vicinity.push_to_hub("davidberenstein1957/my-vicinity-repo")
vicinity = Vicinity.load_from_hub("davidberenstein1957/my-vicinity-repo")

Resulting in https://huggingface.co/datasets/davidberenstein1957/my-vicinity-repo
You can also find datasets here: https://huggingface.co/datasets?other=vicinity

- Updated README.md to clarify that items can be strings or other serializable objects. - Modified the Vicinity class to accept a broader range of item types by changing type hints from `str` to `Any` in several methods. - Enhanced the insert and delete methods to handle non-string tokens appropriately, ensuring that items can be checked and managed regardless of their type.

… and evaluating backends

- Simplified the logic for checking and appending tokens in the insert method, ensuring that duplicate tokens are properly managed.

- Updated the `items` fixture to return a mix of dictionaries and strings based on index parity. - Modified `test_vicinity_insert_duplicate` to use the updated `items` fixture for inserting items. - Adjusted `test_vicinity_delete_and_query` to reference items by their indices instead of hardcoded values. - Enhanced the Vicinity class to streamline token management, ensuring proper handling of duplicates and improving error messaging for token deletions.

Co-authored-by: Stephan Tulkens <stephantul@gmail.com>

…ling - Replaced the nested loop for checking duplicates with a single extend operation for tokens. - Improved efficiency by directly appending tokens to the items list, ensuring proper management of duplicates.

…ling - Replaced the nested loop for token matching with a more efficient list comprehension. - Enhanced error messaging to specify which tokens were not found in the vector space.

- Added a try-except block around the JSON serialization process to catch JSONEncodeError.

- Introduced a new pytest fixture `non_serializable_items` that generates a list of non-serializable objects for testing. - Added a test case `test_vicinity_save_and_load_non_serializable_items` to verify that saving a Vicinity instance with non-serializable items raises a JSONEncodeError. - Updated the Vicinity class documentation to specify that JSONEncodeError may be raised if items are not serializable.

- Introduced HuggingFaceMixin to enable saving and loading Vicinity instances to/from Hugging Face Hub - Added optional import of HuggingFaceMixin based on huggingface_hub and datasets library availability - Implemented methods for pushing Vicinity instances to the Hub, including dataset and metadata upload - Created a method to load Vicinity instances from Hugging Face repositories

codecov-commenter · 2025-01-28T21:08:25Z

Codecov Report

Attention: Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
vicinity/vicinity.py	80.00%	1 Missing ⚠️

Files with missing lines	Coverage Δ
vicinity/vicinity.py	`97.98% <80.00%> (+0.31%)`	⬆️

Pringled

Really nice integration, would be awesome to have this in Vicinity 🎉 Left some minor feedback

Pringled · 2025-02-15T08:20:26Z