You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Index builders allow to set the input vector dataset (to index) as a float[][]. This requires that all vectors be reified in the Java heap as a single contiguous block of memory. This can be expensive and may exhaust the java heap. Usage example:
For the Lucene use case, we write the raw vectors to disk and mmap them. To index them, then we need to copy them back into that Java heap as a float[][]. This is not ideal. It would be better to allow the index builder to set the dataset as an opaque pointer + length.
The text was updated successfully, but these errors were encountered:
Index builders allow to set the input vector dataset (to index) as a
float[][]
. This requires that all vectors be reified in the Java heap as a single contiguous block of memory. This can be expensive and may exhaust the java heap. Usage example:For the Lucene use case, we write the raw vectors to disk and mmap them. To index them, then we need to copy them back into that Java heap as a
float[][]
. This is not ideal. It would be better to allow the index builder to set the dataset as an opaque pointer + length.The text was updated successfully, but these errors were encountered: