Allow huggingface_hub<1.0

[Wauplin]

Follow-up PR after #1383 and especially #1383 (comment). I still believe that we should not block huggingface_hub from introducing breaking changes. However for the most used and less prone to change APIs (like the upload/download ones), I'm more than fine to consider them as "fixed until at least v1.0".

I've looked into tokenizers implementation and given that it uses only hf_hub_download, let's set huggingface_hub<1.0 as dependency. This will reduce the friction for users wanting to install the latest huggingface_hub version. If at any point tokenizers starts to use other methods, let's reassess and chose a solution described in #1383 (comment). In the meantime, I think we should merge this PR.

(also related: once merged and released you should be able to delete #1377 @clefourrier)

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ArthurZucker]

Alright with me 😉 Let's just wait for @Narsil

[julien-c]

asked by theBloke too

[TheBloke]

Oh thank god! 😁

 [pytorch2] tomj@MC:/workspace/process ᐅ history | grep huggingface-hub
 8761  3.10.2023 13:29  pip3 install huggingface-hub'>=0.17'
 8954  13.10.2023 08:07  pip3 freeze | grep huggingface-hub
 9189  22.10.2023 10:47  pip3 install -U huggingface-hub
 9349  27.10.2023 11:20  pip3 install huggingface-hub'>=0.18'
 9628  29.10.2023 17:14  pip3 install huggingface-hub'>=0.18.0'
10045* 2.11.2023 17:17  pip3 install huggingface-hub'>=0.18.0'
10054* 2.11.2023 17:33  pip3 install huggingface-hub'>=0.18.0'
10359* 3.11.2023 18:30  pip3 install huggingface-hub'>=0.18.0'
10666* 5.11.2023 02:22  pip3 install huggingface-hub'>=0.18.0'
11513* 8.11.2023 16:54  pip3 install --upgrade huggingface-hub
11845* 10.11.2023 08:42  pip3 install --upgrade huggingface-hub
11847* 10.11.2023 08:46  pip3 install --upgrade huggingface-hub
11916* 10.11.2023 11:52  pip3 install --upgrade huggingface-hub

Thank you!

[clefourrier]

@ArthurZucker Atm, transformers depends on tokenizers<0.15 and >=0.14 - do you know for which version it will change?

[Wauplin]

@clefourrier this PR has been merged this morning to unpin tokenizers: huggingface/transformers#27494. I guess starting from next release then :)

[clefourrier]

Missed it, ty!

[ArthurZucker]

I'll do a patch today to support this version as well 😉