Reproduce the training process at the right positions in corpora

[PC91]

Hello,
Several weeks ago I updated this PR to properly resume a training at the right corpus positions.
How do you guys think about this ? I could create a new PR and migrate my code to the Eole project.

[francoishernandez]

Hello,
I don't think this is very efficient, especially when datasets are bigger and bigger.
A more future proof approach might be to try and support some things like sharded datasets (e.g. with parquet), which might alleviate a bit the need of reading a lot of data.
Also, an issue which might remain will be the "bucketing" mechanism, which makes it quite hard to know exactly at which point we were. (Might be doable if we have a way to "replay" buckets with proper seeding etc.)