PEX documentation and Makefile is unclear on supported Python versions

[Cagrosso]

I am unable to build a PEX file as described in the documentation.

I am attempting to build the PEX file in a docker container for my version of Python (3.6). According to the Makefile, that version of Python is the default $PYVER. I come across multiple errors from the default $SPACY_EXTRAS and final packaging that point to Python 3.6 being incompatible.

My docker-compose.yml

version: '3.3'

services:
  build:
    image: python:3.6-bullseye
    volumes: 
      - ./spaCy:/spacy
    working_dir: /spacy
    command: tail -f /dev/null

The docker file exists in a directory with the SpaCy repo cloned into a folder spaCy ex:

- docker-compose.yml
- spaCy
| --- ... SpaCy repo root

After launching through a docker compose up -d I have to install the rust compiler following their instructions here.

Once the rust compiler is installed, run make in the working directory. This will result in the $SPACY_EXTRAS failing to build properly. For example, when attempting to build sudachipy, it fails with error: the configured Python interpreter version (3.6) is lower than PyO3's minimum supported version (3.7)

Even dropping the extras that I don't need still results in a failed build at the 'packaging' stage saying that Python3.6 is not supported.

If there is a different version of SpaCy that supports python3.6, or a different method of packaging, I would greatly appreciate being pointed in the correct direction.

Which page or section is this issue related to?

documentation source

[adrianeboyd]

The idea with the Makefile is that you can set PYVER and SPACY_EXTRAS to the versions/packages that you need by setting env variables (or by editing the Makefile).

A word of warning about pex on linux with torch>=1.12: the default pypi torch 1.12+ (with nvidia libraries) does not appear to work at all in the pex file and in any case, the final pex file with spacy requirements will go beyond pex's hard 2GB filesize limit.

For our own internal use (again, linux-only), we recently switched from pex to python-appimage. The setup is a little clunky because it assumes you need a desktop icon, etc., but otherwise I thought it was easy to get working. Put any models as requirements with github URLs into your requirements.txt.

[Cagrosso]

I will give the python-appimage a try, thank you!

However, I believe that as written, the Makefile will not produce a pex that will work. Unless I am doing something wrong, Python3.6 seems to be unable to build the pex at all, let alone with any SPACY_EXTRAS.

[adrianeboyd]

Hi, we've updated the defaults in #12832.

[adrianeboyd]

Let me move this to the discussion board...

[Cagrosso]

For anyone interested in what I ended up doing:

I'm not a Python Dev, so my knowledge of the ecosystem is pretty limited. I ended up finding Miniconda3 and using that to package up spacy for deployment. It allowed me to bundle a newer Python version as well as all the required SpaCy dependencies and models that I needed.

[adrianeboyd]

Thanks for the note about your solution!