SpaCy 3.4.1 & Anaconda3-2022.05 install fail (package conflicts?)

[hpcpony]

The install instructions (https://spacy.io/usage#conda) kinda indicate it should work but I can't install a recent spaCy in a recent Anaconda3.

Specifically: Anaconda3-2022.05-Linux-x86_64.sh, spaCy 3.4.1

[~]$ date ; conda install -c conda-forge spacy=3.4.1 ; date
Sun Sep 18 15:06:53 EDT 2022
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Solving environment: / 
Found conflicts! Looking for incompatible packages.
This can take several minutes.  Press CTRL-C to abort.
Examining conflict for _openmp_mutex importlib_metadata gmpy2 parso alabaster google-resumable-media python-snappy cytoolz twisted diff-| ^
Examining conflict for sortedcollections distributed sortedcontainers anaconda spyder attrs anyio dask intervaltree: : 449it [5:59:16, 15.4
    .
    .
    .
Examining conflict for pytables conda-package-handling anaconda python-libarchive-c conda-build libarchive lzo: : 487it [6:08:34,  7.41s/it
Examining conflict for libarchive python-libarchive-c conda-build: : 487it [6:08:50,  7.41s/it]                                            failed

UnsatisfiableError: The following specifications were found
to be incompatible with the existing python installation in your environment:

Specifications:

  - alabaster -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|3.4.*|>=3.7,<3.8.0a0']
  - appdirs -> python[version='2.7.*|3.5.*|3.6.*|3.4.*|>=3.5,<3.6.0a0|>=2.7,<2.8.0a0|>=3.7,<3.8.0a0|>=3.6,<3.7.0a0']
  - argon2-cffi -> python[version='2.7.*|3.5.*|3.6.*|3.6.12|3.6.12|3.7.10|3.7.10|3.7.12|3.8.12|3.9.10|3.7.9|3.6.9|3.6.9|3.6.9|3.6.9|>=3.5,<3.6.0a0|3.4.*|3.7.*|3.8.*|3.9.*|3.10.*',build='0_73_pypy|3_73_pypy|4_73_pypy|5_73_pypy|1_73_pypy|0_73_pypy|0_73_pypy|5_73_pypy|2_73_pypy|1_73_pypy']
  - argon2-cffi-bindings -> python[version='2.7.*|3.5.*|3.6.*|3.6.12|3.6.12|3.7.10|3.7.10|3.7.12|3.8.12|3.9.10|>=3.6,<3.7.0a0|3.7.9|3.6.9|3.6.9|3.6.9|>=2.7,<2.8.0a0|3.6.9|>=3.5,<3.6.0a0|3.4.*|3.7.*|3.10.*|3.9.*|3.8.*',build='0_73_pypy|3_73_pypy|4_73_pypy|5_73_pypy|1_73_pypy|0_73_pypy|0_73_pypy|5_73_pypy|2_73_pypy|1_73_pypy']
  - async-timeout -> python[version='>=3.10,<3.11.0a0']
  - automat -> python[version='2.7.*|3.5.*|3.6.*|>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.8,<3.9.0a0|>=3.7,<3.8.0a0']
  - backcall -> python[version='>=2.7,<2.8.0a0|>=3.5,<3.6.0a0|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.8,<3.9.0a0']

   .
   .
   .
   .  (15782 lines deleted)
   .
   .
   .

Your installed version is: 2.17

Note that strict channel priority may have removed packages required for satisfiability.

Sun Sep 18 23:06:47 EDT 2022

I also tried (think I noted these correctly):

Anaconda3-2022.05-Linux-x86_64.sh, spaCy 3.3.1 : FAIL
Anaconda3-2021.11-Linux-x86_64.sh, spaCy 3.3.1 : FAIL
Anaconda3-2022.05-Linux-x86_64.sh, spaCy 3.2.1 : Works
Anaconda3-2021.11-Linux-x86_64.sh, spaCy 3.2.1 : Works

Miniconda3-py39_4.12.0-Linux-x86_64.sh, spaCy 3.4.1: Works

  Collecting package metadata (repodata.json): done
  Solving environment: failed with initial frozen solve. Retrying with flexible solve.
  Solving environment: done

I also tried playing with different conda config setting for "channel_priority" and "allow_conda_downgrades" but not in a systematic way.

So, at least for me, it looks like recent spaCy packages are fundamentally incompatible with recent Anaconda3 distributions.

Any idea what I'm doing wrong? Is anyone else able to make this work? Which version of Anaconda should work with which versions of spaCy? Any help appreciated.

(in all cases the Anaconda3 distributions were fresh, virgin, installs)

[adrianeboyd]

Hmm, I wasn't able to replicate this locally so I'm not entirely sure what's going on. As far as I'm aware we haven't had any other reports of particular versions of anaconda being incompatible with the conda-forge packages that we maintain.

Can you try temporarily removing any custom channel-related settings in your .condarc and installing again? Because of the message at the bottom that "strict channel priority may have removed packages required for satisfiability" it does sound like the problem might be related to the channel order or channel priority settings?

[hpcpony]

Will try some other experiments thanks. (Pretty sure i tried wo strict but will check)

…

On Fri, Sep 23, 2022, 04:58 Adriane Boyd ***@***.***> wrote: Hmm, I wasn't able to replicate this locally so I'm not entirely sure what's going on. As far as I'm aware we haven't had any other reports of particular versions of anaconda being incompatible with the conda-forge packages that we maintain. Can you try temporarily removing any custom channel-related settings in your .condarc and installing again? Because of the message at the bottom that "strict channel priority may have removed packages required for satisfiability" it does sound like the problem might be related to the channel order or channel priority settings? — Reply to this email directly, view it on GitHub <#11520 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALWTX46URFRDBPSUKPEFVETV7VWMFANCNFSM6AAAAAAQQFE56M> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[hpcpony]

I just tried again with the latest Anaconda (2022.10) and the latest spacy (3.4.3) and still can't get it to work. I was able to start with Miniconda and build up from there. I tried so many things, but I think I was also able to start with a full Anaconda and install into a environment (but that's such a kludge...) I think starting with Anaconda it is just too bloated and can't figure out a compatible set of packages (spacy isn't the only place I run into this), so maybe it's time to just give up on Anaconda.

If anyone can unpack a full Anaconda (2022.10?) and then "conda install spacy=3.4.3" and get it to work (linux) I'd be curious to see what version of Anaconda you use, "conda info", "conda config --show", your .condarc, etc...

(If there's a way to mark this as not so much "answered" but as resolved/hopeless... feel free ;^))

[pmbaumgartner]

I've just installed spaCy 3.4.1 in a new conda environment successfully. While it did take a while to solve for the environment, eventually it did work.

conda create -n spacy-discussion-11520 python=3.8
conda activate spacy-discussion-11520
conda install -c conda-forge spacy=3.4.1  # this took a while to solve

# validate
python -c "import spacy; print(spacy.__version__)"
3.4.1

Here's my output from conda info:

     active environment : spacy-discussion-11520
    active env location : /home/peter/miniconda3/envs/spacy-discussion-11520
            shell level : 2
       user config file : /home/peter/.condarc
 populated config files : 
          conda version : 4.11.0
    conda-build version : not installed
         python version : 3.8.12.final.0
       virtual packages : __cuda=11.6=0
                          __linux=5.15.0=0
                          __glibc=2.31=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /home/peter/miniconda3  (writable)
      conda av data dir : /home/peter/miniconda3/etc/conda
  conda av metadata url : None
           channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /home/peter/miniconda3/pkgs
                          /home/peter/.conda/pkgs
       envs directories : /home/peter/miniconda3/envs
                          /home/peter/.conda/envs
               platform : linux-64
             user-agent : conda/4.11.0 requests/2.27.1 CPython/3.8.12 Linux/5.15.0-50-generic ubuntu/20.04.5 glibc/2.31
                UID:GID : 1000:1000
             netrc file : None
           offline mode : False

[hpcpony]

Thanks. There are a couple things in your example that I think make things "easier". First off, you install into a conda environment. I've had that work too and I believe it's consistent with the idea that a full conda install (from the .sh file) has way more stuff and is easier to "resolve". Secondly, I see mentions to "miniconda" in your logs. miniconda has way fewer packages installed (a couple dozen vs. 400+ for anaconda) so that should be much easier to "resolve".

I will note another experiment that I tried which was to install the full anaconda (400+ packages) and then deleted the package "zfp" (no idea what it does) which in turn deleted about 300 (I think) other packages. After that I was able to conda install spacy without issue.

Thanks for the feedback. With so many different platforms, all the different configurations, and Anaconda's/Python's approach to management it's really hard to reproduce and figure out what's going on.

[pmbaumgartner]

@hpcpony I would recommend always creating and installing packages into a new conda environment when you have a new project. I would recommend leaving your base environment alone. When you create a new conda envrionment, there won't be any of the 400 anaconda packages in it, regardless of whether you're using anaconda or miniconda. There's a helpful article on using conda environments here.

I know all the *conda terms are confusing, as I've been confused by them before, but just to set things straight:

• anaconda installs python, conda and ~400 common data science packages.
• miniconda installs python and conda
• conda is an environment manager and package manager

Finally, you can always use conda only for the environment management functionality and not the package management, which is what I do. This means I create new environments with conda, but then use pip to install packages. If you're having trouble installing something with conda install, be sure you've tried pip first. Never mix the two package installation methods in an environment though, it will lead to redundancies in packages and possibly "confuse" your environment about which package should be used.

[hpcpony]

Thanks for the information. We tend to stay away from conda environments and instead favor linux environment modules. There are a number of advantages (at least for us). I think going forward we'll just give up on (full) Anaconda and instead start with Miniconda and build up from there; that'll be "like" a conda environment, just without all the conda environment clunkyness.

Sorry this strayed into more Anaconda vs. spaCy, but hopefully it will save some future installer-person some aggravation.