Skip to content

Files

Latest commit

 

History

History
11 lines (9 loc) · 412 Bytes

README.md

File metadata and controls

11 lines (9 loc) · 412 Bytes

How many tokens per word using given tokenizer and given dataset?

$ pip install .

$ cat raw-sentences.txt | tokens-per-word -t "north/t5_base_scand3M"
# or (if you have csvkit/csvtools)
$ csvcut -c "text-column" tabular-dataset.csv | tokens-per-word -t "north/t5_base_scand3M"
# or (if you jave jq)
$ cat line-formatted-dataset.jsonl | jq .text-field | tokens-per-word -t "north/t5_base_scand3M"