- ./build_sa : Build suffix array
- Save suffix array data with safetensors format.
- ./exact_dedup : Do exact dedup with built suffix array
Files
exact_dedup_at_scale
Folders and files
Name | Name | Last commit date | ||
---|---|---|---|---|
parent directory.. | ||||
Name | Name | Last commit date | ||
---|---|---|---|---|
parent directory.. | ||||