Skip to content

Files

Latest commit

 

History

History

exact_dedup_at_scale

  • ./build_sa : Build suffix array
  • Save suffix array data with safetensors format.
  • ./exact_dedup : Do exact dedup with built suffix array