Upload / save model as a package and re-use in other repos

[darioprencipe]

Hey there!

first of all, thanks for the amazing work you keep doing. Impressive.

With our own private data, I trained a NER component starting from a blank sheet Language class (nlp). Now, I would like to use this model in a production Python service / application. This means this model has to be somehow uploaded in a repo, shared across teammates, productionized, etc.

When I try to push my project - containing the model - to Github, the push fails. I always get the - very understandable - message:
"
this exceeds GitHub's file size limit of 100.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
"

Same thing occurs when I package my model using python -m spacy package CLI command. When I try to upload the tar.gz file, I get exactly the same issue.

Must be me, but I can't find anywhere in your docs how to actually get these nice models downloadable and usable without them being necessarily publicly hosted on PyPi.

How can we do that? What's the best practice for uploading and re-using models to / across git repositories? What am I missing here in the package step?

Thanks a ton!
Dario

[adrianeboyd]

In the end the pipeline packages are just large files, so you can host them pretty much anywhere you can upload a large file with the permissions and download options you need. As you've seen, github repos and pypi.org aren't good options due to file size limits.

One option that many spacy users use is to host their models on huggingface hub, with easy uploads for spacy using spacy-huggingface-hub. We also have mirrors of the trained pipelines in our organization there: https://huggingface.co/spacy. I think you can make individual models private for a user or an organization.

We host the trained pipelines like en_core_web_sm as assets attached to github releases (which are separate from the git repo itself). You can expand one of the "Assets" sections here to see the attached models: https://github.com/explosion/spacy-models/releases .

Although pypi.org isn't an option for hosting models due to package size limits, you can host models using your own internal PyPI server. I wouldn't recommend setting up a pypi server just for this, but if your organization already has one, then that could be a reasonable choice.