[Neurips23] ParlayANN Submission for OOD track

[magdalendobson]

This PR contains our submission for the OOD track. We tested on an AWS c6i.2xlarge and found that at 90% recall, our entry was averaging around 7000-7500 QPS, while the baseline was around 3500. The build took about 4 hours. I found that on this machine, when building and then querying, the main memory did not seem to flush out between saving the files and then reloading them and thus overran the memory limit. This means that it was necessary to build the index and then load and query it on a separate run. I observed the same problem with the DiskANN baseline. Just wanted to note that in case it comes up during evaluation.

I have edited the neurips23.yml file to test our build using the random-xs dataset. Please let me know if there is anything else needed.

[harsha-simhadri]

Thanks for the heads up on memory. Will try out your PR.

[magdalendobson]

I think my latest commit fixes those three issues.

[harsha-simhadri]

I am unable to run this since the time out for docker is set to 30 minutes in your branch. You may want to rebase with the latest commit on this repo's main to get the latest runner config.

[magdalendobson]

I am unable to run this since the time out for docker is set to 30 minutes in your branch. You may want to rebase with the latest commit on this repo's main to get the latest runner config.

I just pulled from main and rebased.

[maumueller]

@magdalendobson Do you think the memory issue is happening because of some detail in the eval framework? It seems to me that https://github.com/harsha-simhadri/big-ann-benchmarks/pull/186/files#diff-923b8101f105761e2d658a67e990261dd3658cc86f6482d9231316d184cb101aR100-R108 might be the culprit because build_vamana_index might not free the memory after writing the index to disk.

[magdalendobson]

Hmm, yeah, I think it's possible. Let me experiment with modifying the delete constructors of some of my objects and I'll get back to you.

[magdalendobson]

@maumueller thanks for the suggestion. I took a closer look and it did turn out that some of my data structures were not freeing memory properly. You should now be able to re-install and then run my submission without memory problems.

[harsha-simhadri]

I see the following results after restarting from a crash post index build

vamana,vamana,text2image-10M,10,8568.28073422188,0.0,1000000.0,13098176.0,1528.6819382196045,0,0,ood,0.8773770000000001
vamana,vamana,text2image-10M,10,7777.599618424573,0.0,1000000.0,13098176.0,1684.089776101532,0,0,ood,0.88975
vamana,vamana,text2image-10M,10,7128.303182660356,0.0,1000000.0,13098176.0,1837.488623079529,0,0,ood,0.8999010000000001
vamana,vamana,text2image-10M,10,6873.170646655227,0.0,1000000.0,13098176.0,1905.6963188269044,0,0,ood,0.9041650000000001
vamana,vamana,text2image-10M,10,6607.938732179394,0.0,1000000.0,13098176.0,1982.1878699047852,0,0,ood,0.908165
vamana,vamana,text2image-10M,10,6376.8978543202775,0.0,1000000.0,13098176.0,2054.0043606196596,0,0,ood,0.911952
vamana,vamana,text2image-10M,10,6148.750960929249,0.0,1000000.0,13098176.0,2130.2173536103824,0,0,ood,0.9153420000000001
vamana,vamana,text2image-10M,10,5739.398208736937,0.0,1000000.0,13098176.0,2282.1514597229,0,0,ood,0.92154
vamana,vamana,text2image-10M,10,5551.22053214029,0.0,1000000.0,13098176.0,2359.512817796478,0,0,ood,0.924339
vamana,vamana,text2image-10M,10,4789.167993183752,0.0,1000000.0,13098176.0,2734.958560368347,0,0,ood,0.935529

[magdalendobson]

That seems about right. Are you sure you pulled the latest version if you're still having the crash issue? My latest push fixed it on the 16G machine I tested on. You might have to increment the CACHEBUST argument in the Dockerfile. Also, are there any outstanding edits needed? I think I fixed everything you requested earlier.

[harsha-simhadri]

Let me merge this PR since its functional. We will do a final run later and it would have your latest commit.