Explicitly specifying mappings between project.dependencies and tool.pixi.dependencies at pyproject.toml

[anjos]

In the context of pixi manifests in pyproject.toml files, I wonder to what extent one can smartly map Python dependencies that are automatically injected from project.dependencies as pypi-dependencies, into conda dependencies.

I'm raising this question because, In some cases, Python dependencies and conda dependencies have slightly different names (e.g. torch in PyPI vs pytorch in conda-forge). However, when pixi installing the project, one may want to prefer conda dependencies and avoid altogether the installation of PyPI equivalents.

As of now, one has to repeat conda dependencies in pyproject.toml manifests, what looks like this:

[project]
name = "test"
requires-python = ">=3.11"
dependencies = [ "dep1", "dep2", "dep3" ]

[tool.pixi.dependencies]
# python ~=3.11.0 is auto-injected here
dep1 = "*"
dep2 = "*"
dependency3 = "*"  # e.g. different conda package name 

If the project has many dependencies, the above process becomes tedious and error-prone. If one of the dependencies is forgotten on tool.pixi.dependencies, then it is automatically injected into tool.pixi.pypi-dependencies, which may inadvertently trigger issues such as #1051 or #1046.

I wonder if it would not be possible, instead to have something like:

name = "test"
requires-python = ">=3.11"
dependencies = [ "dep1", "dep2", "dep3" ]

[tool.pixi.project]
# auto injects all pypi-dependencies as `dependencies` with matching name
# defaults to `false`, falling-back on the current behaviour
convert-pypi-to-conda-package-names = true

[tool.pixi.pypi-to-conda-name-mapping]
# except for this one package, inject name "dependency3" instead of "dep3"
# nevertheless, respect any pinnings from `project.dependencies`
dep3 = "dependency3"

A feature such as this would allow further control over the packaging-type of dependencies to be installed.

Possibly connected to #532 and #1198.

[ruben-arts]

He @anjos thanks for the write-up.

The name mapping is something we fix with conda-pypi-mapping so that should be avoidable with those settings.

This was one of the first requests around the pyproject.toml integration and was even in code in the first iteration by @olivier-lacroix. We've not merged it as it might also give unwanted behavior if it was the default.

So adding it as an "opt-in" option sound like a logical next step!

Currently the project.dependencies in a pyproject.toml are always "seen" as tool.pixi.pypi-dependencies so would you expect this mapping logic to only work on the project.dependencies or also on the tool.pixi.pypi-dependencies variant? Or should that be part of the configuration itself?

There was an effort for a design in : #1094 (comment) which is part of this story.

That would result in something like:

[pypi-options]
# When not used, will use uv default which is just pypi
primary-index = "pypi.org/simple"
extra-indices = ["foo", "bar"]
flat-indices = ["path/to/index"]
convert-to-conda = true
# Or
# convert-to-conda = { dep3 = "dependency3" }
# or a different name

# Or only per feature if you want that, as conda is safer.
[feature.prod.pypi-options]
convert-to-conda = true

What I'm trying to say is, thanks for the write-up, this is most definitely coming, stay connected or contribute if you can 😃.

[anjos]

So adding it as an "opt-in" option sound like a logical next step!

Right, looks like the right thing to do.

Currently the project.dependencies in a pyproject.toml are always "seen" as tool.pixi.pypi-dependencies so would you expect this mapping logic to only work on the project.dependencies or also on the tool.pixi.pypi-dependencies variant? Or should that be part of the configuration itself?

I may be overlooking something, however I think the mapping should be only one way: converting pypi-dependencies to dependencies so that one uses conda packages as much as possible. In this context, you only need a flag somewhere to indicate that, plus a mechanism to specify a mapping if necessary (ie. in cases package names do not match). Of course, that could be combined into a single variable as I understand you suggested.

Thinking about it a bit deeper, you also may want to "exclude" packages from auto-conversion. This could be useful in cases where there are PyPI packages that simply do not exist as conda packages. Some examples borrowing from your nomenclature:

[pypi-options]
# brute force convert all pypi-dependencies to dependencies (default is false)
convert-to-conda = true

# brute force convert all pypi-dependencies to dependencies, except dep3
convert-to-conda = {force-conda = true, mappings = {dep3 = "dependency3"}}

# converts only dep1 and dep3
convert-to-conda = {mappings = {dep1 = "dep1", dep3 = "dependency3"}}

# converts dep1 and dep2, not dep3 (leave it in pypi-dependencies)
convert-to-conda = {force-conda = true, except = ["dep3"]}
# or
convert-to-conda = ["dep1", "dep2"]

# Or only per feature if you want that, as conda is safer.
[feature.prod.pypi-options]
convert-to-conda = true

The per-feature add-on would be a plus IMO.

What I'm trying to say is, thanks for the write-up, this is most definitely coming, stay connected or contribute if you can 😃.

I'm not sure how to contribute as my rust skills are zero presently - please let me know how to be useful (e.g. I can open an issue with this suggestion, or contribute to an open issue if that is useful).