Local path dependencies during development

[finswimmer]

What's the task?

I'm developing a tool Tool which relies on a library Lib which is under heavy development too. So I would like to add this library as a path dependency in editable mode during developing Tool. When publishing my Tool, Lib should be handled as a normal dependency.

A support by poetry for the above task is asked regularly, e.g. in #1168 (comment).

Why does poetry not support it (yet)?

poetry makes sure, that all defined dependencies are resolvable for the range of python versions defined in the [tool.poetry.dependencies] section. The result is stored in the poetry.lock, which makes it possible to create identical environments on different systems.

Defining different sources for a package makes it necessary to resolve dependencies for all sources, because even if the name is equal, the defined dependencies can differ. This can lead to conflicts in dependency resolution.

Even one can assume that dependencies are equal, there's still the problem how to tell poetry from which source it should install the package?

Ideas about how to solve this problem are very welcome 😄

Is there a workaround?
Yes, it is! (But maybe it's not suitable for all use cases.)

To get it work, one have to publish a minimal release of Lib to the place, where Tool can fetch it in production (pypi, private repository, git repository, ...). This minimal release must have defined all necessary dependencies.

Once this is done, add Lib as a dependency to Tool in the normal way. poetry can resolve dependencies and install Lib.

Now we can add the path to Lib to the environment variable PYTHONPATH . Doing this, python will look at this place for module files when running a script.

How you set PYTHONPATH depends on your workflow. E.g. if you are using PyCharm you can attach the Lib project to Tool project and PyCharm will do that for you.

If you are working within a terminal, it might be a good idea to create an .env file containing:

PYTHONPATH=/path/to/lib:$PYTHONPATH

and run source .env once when starting the terminal.

Note: The dependency resolution, getting package metadata and installing scripts given in the [tool.poetry.scripts] are made based on your published minimal release. So if these things changes you have publish a new release, so poetry can fetch up these changes.

Update 2024-07-09: In the meantime an alternative is to simply run pip install -e /path/to/project to temporary install the dependency in development mode.

[absassi]

Defining different sources for a package makes it necessary to resolve dependencies for all sources, because even if the name is equal, the defined dependencies can differ. This can lead to conflicts in dependency resolution.

This only leads to a conflict if Poetry considers possible that both sources can be used at the same time, but that is not possible.
It is not too much different than a package having two versions (each version has its own set of dependencies, and may conflict with the other). The critical difference is that for versions, only one version is locked in the end, while to support this feature, Poetry will have to resolve and lock both cases of dev and non-dev environments.
The trivial solution is to just resolve everything twice if any dependency is specified differently in non-dev and dev sections. I believe the slowest part of resolution is gathering all the dependencies for each package, so with proper in-memory caching resolving twice shouldn't be a big performance penalty.
Obviously, there would be the need to either change the lock file format to support two separate resolved sets of dependencies (perhaps a new key for each package to say if the locked data refers to dev, non-dev or both), or simply create two lock files. Using a single file is much better, as normally it would be almost all the same for both, so things could be reused.

Even one can assume that dependencies are equal, there's still the problem how to tell poetry from which source it should install the package?

I'd say poetry install selects the dev source, poetry install --no-dev selects the non-dev source.

[TBBle]

I think that dev/non-dev is just the right-now state of more general 'extras' support, so it probably does not make sense to have one-file-per-X or boolean states, as those "2-state" situations are really just a specific case of "n-state" (where n is still likely to be small, but is combinational, so perhaps will go as high 16 distinct possible combinations of 4 extras).

Extras tend to add dependencies, like 'dev' does now, but similarly to 'dev', they may flow through into modifying dependencies, e.g., I create a Poetry package with a 'markdown' extra, and that extra requires the 'markdown' extra of some other package I used.

[finswimmer]

Hello @absassi,

The critical difference is that for versions, only one version is locked in the end, while to support this feature, Poetry will have to resolve and lock both cases of dev and non-dev environments.

this is something I thought about too. Beside the performance issue you've mentioned, there is another problem: The locked version for the packages can be then different between dev and no-dev. Is this something users can live with? Does this irritate users? Or is this actually a no-go? I don't know ...

[absassi]

@finswimmer, they can be different, and the difference should be locked correctly.
Suppose Tool depends on Lib-1.0.0 for non-dev, which in turn depends on ThirdParty ^2.3.0 (thus some version like 2.3.4 will be locked for Tool non-dev). Now, the local version of Lib is being upgraded to support the shiny new API of ThirdParty-3.0.0 and therefore now depends on ThirdParty ^3.0.0. I would expect that Tool locks 2.3.4 for non-dev (which is compatible with Lib-1.0.0) and locks 3.0.0 for dev (which is compatible with the currently in-development version of Lib).

[absassi]

In my answer, I missed an important point that @jonapich said in reply to the other answer:

There's no point in locking the published version of libA because end users will obtain the constraint version and not the locked version. Developers don't want to use the constrained version, they want the one in the repository and poetry should basically ignore the version here.

Translating to example in the comment above, it means that Poetry would create only one lock file for Tool with Lib path and ThirdParty-3.0.0. There would be no lock file for non-dev with Lib-1.0.0 and ThirdParty-2.3.4, because nobody needs that.

[abn]

An option here might to talk about a net-new feature rather than coerce existing capabilities. For example, we could maybe introduce a new command pair attach/detach (note this can be a plugin now too).

This might function as follows.

# add foobar as a dependency
poetry add foobar@1.0.0

# attach the local project into the active environment
poetry attach foobar /path/to/foobar

One option, if we want to keep things isolated, might be to create the following.

$ tree .poetry/
.poetry/
├── lock
└── override.toml

0 directories, 2 file
$ cat .poetry/override.toml 
[tool.poetry.dependencies]
foobar = { path = "/path/to/foobar", develop = true }

The .poetry/lock file might contain the "local" lock file. The directory .poetry might be recommended ignored perhaps to avoid accidental commits.

# detach from active environment; remove temporary files
poetry detach foobar

All that said, I am not convinced that this really brings much to the table than a workflow where you do something like this;

poetry add foobar@1.0.0
poetry add -e /path/to/foobar
# do development what not
poetry add foobar@1.0.0

Is the ask in general that these "local" development dependencies be preserved in the pyproject.toml file, even in your vcs?

Alternatively attach/detach could simply rely on something a local file like .poetry/override.toml to sort of alias the adds easier from the last example.

[TBBle]

The use-case as I understand it is not a short-term temporary setup to debug the other library, but when actively working on the other library, in order to test it in-place in its use-case. So it's local to the user, but not "just for 20 minutes". The original poster sounded like they would generally leave it like that locally (as they tend to work on the two projects interweavedly), but don't want to publish the path-dependency by accident.

[jonapich]

Since pip doesn't have a --dev option when installing packages, it should be fine for the dev-dependencies section to refer to local repository folders, the same way you would list them in the packages sections of the pyproject.toml.

Also, there's no point in locking the published version of a module if it's local. For instance:

[deps]
libA = "*"

[dev-deps]
libA = { path = "../libA", develop = True }

There's no point in locking the published version of libA because end users will obtain the constraint version and not the locked version. Developers don't want to use the constrained version, they want the one in the repository and poetry should basically ignore the version here. In the off chance that you need a published version (for instance, to tackle some bug involving old versions), you can simply comment out the dev-deps line in your pyproject.toml (or pin it to an old published version) temporarily.

The above example already works but it's not officially documented by Poetry. #2693 has a link to a public repository that uses the dep + dev-deps pattern successfully. I think Poetry should at least officialize this behavior, because it's very useful and doesn't seem too far off... and I don't want my whole repo to explode with some new poetry upgrade 😭

[TBBle]

That seems like a different (monorepo) use-case... where anyone who is developing on that package also has ../libA available by definition. (That's actually the use-case I have)

I'm also not clear how this works for non-developers, who would now be able to resolve any libA version. Does this work?

[deps]
libA = ">3"

[dev-deps]
libA = { path = "../libA", develop = True }

I would expect, that if the above worked, I would get an error from poetry install if I accidentally had ../libA checked out at an old revision, e.g., a mistake in the use of the git submodules command, or a user in a revision control system like Perforce or SVN where it's possible to update some part of the workspace and not another.

That said, I think the above would break pip install mylib[dev], if Poetry was able to successfully export from that pyproject.toml, as I'm not sure where that ../libA would point, but I'm fairly sure it wouldn't be right. (I don't know off-hand what Poetry currently does if you poetry build with pathed dependencies.)

[jonapich]

That seems like a different (monorepo) use-case... where anyone who is developing on that package also has ../libA available by definition. (That's actually the use-case I have)

Yes, and in my case each library is published on its own even though they refer to the local version during development.

I'm also not clear how this works for non-developers, who would now be able to resolve any libA version. Does this work?

[deps]
libA = ">3"

[dev-deps]
libA = { path = "../libA", develop = True }

Yes, your example works as long as the local libA's version matches the constraint (based on its pyproject.toml local file). Non-developers using pip will end up using libA>3 from pypi.org.

If the local version doesn't match the constraint, poetry lock will complain.

That said, I think the above would break pip install mylib[dev], if Poetry was able to successfully export from that pyproject.toml, as I'm not sure where that ../libA would point

Indeed, it cannot work if one of the extra points to a local-only, dev dependency. But I could live with that limitation. To lift the limitation, we would need a new syntax for dependencies, something like libA = { version = ">3", develop = "../libA" }. For all user installs, use the versions. For dev installs (i.e. poetry install), use the one from the path.

(I don't know off-hand what Poetry currently does if you poetry build with pathed dependencies.)

It works. poetry build doesn't include dependencies, it will package the sdist and wheel and include the pyproject.toml file. Then when you install it from pip, it will use the constraints from the dependencies section, and download libA>3.

[TBBle]

Excellent, thank you. We are currently using a monorepo with relative pathed dependencies, but hadn't yet started testing out how to make constraints and publishing work. As you mentioned, it's not really documented how this would behave, and we haven't needed to publish yet (all our current use-cases are in the same monorepo for now), so it's good to know that we're not going to hit an unresolvable brick wall the moment we do start pushing thing to our local pypi index.

[MadJlzz]

Not sure if this can help the discussion here but on other langages I am used of having my dependencies loading first from the local cache of my package manager.

That being said, we could imagine that:

poetry publish --build --local-registry

builds and store the generated artifact in the local filesystem.

Running just behind

poetry lock && poetry install

would fetch this new package locally and install in in the venv

Working that way would also mean that the pyproject.toml dependency section would not change even if your are developing on a project that uses your lib changes locally.

[tool.poetry.dependencies]
# directory
my-package = "0.0.1-snapshot"

VS

[tool.poetry.dependencies]
# directory
my-package = { path = "../my-package/", develop = true }

From the answer here: #1135 (comment)

[jonapich]

This is a good idea and solves a number of use cases, but it doesn't seem to play well with develop=true where you can edit the source of multiple local projects without extra steps.

In practice, all you need is a my_package.pth with the path to the local source of the package. This file is already created by poetry when develop=true is set, and placed in the sites-package folder.