program stuck (when ”Loading cached shuffled indices for dataset at ...“)

[ccx06]

When running the training code, the loading cached shuffled indexes program is stuck? How to solve it?

[xszheng2020]

Same issue here, have you solved it? @ccx06 @sangmichaelxie

[sangmichaelxie]

Maybe there is some kind of OOM issue or num_workers is set too high? Does this happen every time and on all datasets?

[xszheng2020]

I set the num_workers as 1. I tried to reduce the RANDOM_BATCH_SIZE in dataloader.py but it does not work.

It seems the issue is related to caching

06/25/2024 03:50:31 - INFO - datasets.arrow_dataset - Caching indices mapping at ~/doremi/preprocessed/train/Pile-CC/66/cache-abce09a69c09a6c6.arrow