Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: variable block size & elias-fano encoding #18

Closed
wants to merge 7 commits into from

Conversation

silver-ymz
Copy link
Member

@silver-ymz silver-ymz commented Dec 19, 2024

bench

  • machine: 8 vCPU, 32 GB, AMD Ryzen 7 7700, hyper-v Virtualization

dataset: trec-covid

before the PR:
Index: 4.5458s (37689.88/s)
Query: top1000 - 15.9687s (31.31/s), top10 - 18.0994s (138.13/s)
Index disk size: 384 MB

fixed block partition + delta bitpack:
Index: 4.7356s (36179.51/s)
Query: top1000 - 12.9719s (38.54/s), top10 - 13.9670s (178.99/s)
Index disk size: 508 MB

fixed block partition + elias fano:
Index: 4.8131s (35597.16/s)
Query: top1000 - 13.1613s (37.99/s), top10 - 13.3764s (186.90/s)
Index disk size: 508 MB

variable block partition + delta bitpack:
Index: 9.2832s (18456.21/s)
Query: top1000 - 13.1917s (37.90/s), top10 - 13.4416s (185.99/s)
Index disk size: 508 MB

variable block partition + elias fano:
Index: 9.3289s (18365.81/s)
Query: top1000 - 12.9883s (38.50/s), top10 - 12.4730s (200.43/s)
Index disk size: 513 MB

dataset: arguana

before the PR:
Index: 2.1008s (4128.82/s)
Query: top1000 - 47.9059s (29.35/s), top10 - 35.7517s (39.33/s)
Index disk size: 260 MB

fixed block partition + delta bitpack:
Index: 2.5946s (3343.05/s)
Query: top1000 - 36.7708s (38.24/s), top10 - 19.0610s (73.76/s)
Index disk size: 347 MB

fixed block partition + elias fano:
Index: 2.6600s (3260.88/s)
Query: top1000 - 36.0452s (39.01/s), top10 - 18.4784s (76.09/s)
Index disk size: 347 MB

variable block partition + delta bitpack:
Index: 2.7843s (3115.37/s)
Query: top1000 - 33.9380s (41.43/s), top10 - 18.7480s (74.99/s)
Index disk size: 347 MB

variable block partition + elias fano:
Index: 2.7671s (3134.68/s)
Query: top1000 - 33.4785s (42.00/s), top10 - 17.4946s (80.37/s)
Index disk size: 347 MB

Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
@silver-ymz silver-ymz changed the title [WIP] feat: variable block size [WIP] feat: variable block size & elias-fano encoding Jan 7, 2025
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
Signed-off-by: Mingzhuo Yin <yinmingzhuo@gmail.com>
@silver-ymz silver-ymz marked this pull request as ready for review January 7, 2025 14:57
@silver-ymz silver-ymz changed the title [WIP] feat: variable block size & elias-fano encoding feat: variable block size & elias-fano encoding Jan 7, 2025
@VoVAllen
Copy link
Member

VoVAllen commented Feb 6, 2025

Closed due to limited performance improvement

@VoVAllen VoVAllen closed this Feb 6, 2025
@silver-ymz silver-ymz deleted the feat/variable-block-size branch February 6, 2025 07:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants