poc: isolated project plugins & tools

[abn]

This is a proof of concept for discussions and feedback only.

Project Level Plugins

Demo: https://asciinema.org/a/bFC2lNUV8K8IJurJz0RbJEJ1s

Rationale

In a lot of case, a project might require a plugin that is otherwise not required anywhere else for a user. This can in theory be defined within the pyproject.toml as opposed to being manual instructions for the user.

Project Tools

Demo: https://asciinema.org/a/uIKtlMM8nv8f1ECLRJCUzQUsC

Rationale

Projects usually require additional tools like pre-commit and tox. Today, these tools are part of the project's dependency graph. While this is useful for cases like pytest, it is not so much the case for tox and/or pre-commit that are used as cli tools and not used within the project code.

Implementation Details

In this poc, when poetry install is run, we

1. create a .poetry directory if either tools or plugins are defined.
2. create a plugins directory (.poetry/plugins) to which plugin wheels will be installed (without polluting the project venv or the global system site)
3. create a tools/<name> virtual environment with only the tool and any additional dependencies specified.
4. create a .poetry/config.toml to store hashes, sytem python version and poetry version to detect changes.

Example Configuration

In this poc we use overlay section to avoid validation errors.

[overlay]
plugins = [
    "poethepoet"
]

[overlay.tools]
tox = [
    "tox",
    "tox-venv",
]
pre-commit = [
    "pre-commit"
]

This can use used as follows.

poetry install
poetry poe --help
poetry run tox --version
poetry run pre-commit --version

Limitations

1. We use pip install to demonstrate capability. This can be replaced with Poetry's internals.
2. Plugins are all installed to a single directory (this can be changed)
3. Implementation adds a lot of overhead to commands (this can be seen in the pytest runs in CI)

[Kobold]

Looking forward to seeing how the Poetry plugin concept evolves! This PR makes a lot of sense, although the implementation details are pretty fancy. I made a comment on issue #5729 on how a blessed npm run equivalent may be a 70% solution to this with a much smaller implementation surface area commitment from Poetry:

#5729 (comment)

[aucampia]

Would really love to have this, for now I'm using tox for this (ref) but it is quite hacky.

[aldencolerain]

I really like the idea of having plugins on a per-project basis specified in pyproject.toml or poetry.toml. The plugin system is fantastic, however we also commonly have projects that require additional functionality: ci, tasks, env, etc. It quickly becomes complicated when these need to be documented for every project and setup globally.

[kubijo]

Is there any idea of further movement / next steps on this?

[DaStoll]

Would absolutely love this feature!

[radoering]

tl;dr: I split off the plugins part so that you can find a full blown implementation for project plugins in #9547.

Long story:

I took a closer look at this and I think there are some not so small TODOs to bring this over the finish line:

• In the PoC, plugins and tools are only strings that are passed to pip. However, I think it makes more sense to allow the same syntax as for tool.poetry.dependencies when installing via poetry instead of pip. (In my opinion, we definitively need the option to specify sources!)
• What about locking? In the PoC, there is no locking. I think for plugins that is ok. For tools, I'm not that sure. Without the PoC, you put tools in some dependency group so that they are locked. In this PoC, you trade locked tools in a common project venv for unlocked tools in isolated venvs.
• Further, I think there is some overlap between tools and isolated groups as described in (🎁) Support isolated dependencies that install like pyprojectx/pipx #5037. Using isolated groups for tools combined with Lock markers and groups #9427 would have the nice side effect that we could put locked depenencies for the tools (resp. isolated groups) into poetry.lock without the risk of mixing a tool's dependencies with other dependencies.

I think we should split this in two parts: project plugins on the one side and tools on the other for the following reasons:

• Although both are similar enough in this PoC, they are different enough that concentrating on one of both reduces the necessary work to bring it over the finish line a lot.
• Considering that we want to remove poetry-plugin-export from poetry's dependencies and project plugins may mitigate the consequences, some of us consider project plugins as a blocker for 2.0 so it has a higher priority for me. (Tools can still be added in a minor release later.)

[MrLoh]

@radoering is there any indication when the new tool.poetry.requires-plugins functionality will be released? It seems like it is not available in 1.8.4 yet.

[radoering]

See #9448, it should be only a matter of weeks.

[abn]

@radoering thanks for running with it btw 💜 and I agree with your assessment, the PoC was done around 2 years ago - a lot has also changed under the hood that makes more sense to split the objectives. I will close this once we have something on the tools front.