How to manage and publish multiple libraries that might depend on each other

[inoc603]

I've had some great time using polylith with poetry, but as the monorepo grows larger, some libraries we wrote as components for the applications in one monorepo are needed in other repo as well. So I need to publish these libraries as pypi packages, but I couldn't figure out a reasonable way to achieve this. I've read the documentation on building libraries, but I'm still kind of confused.

Let's say the structure of the repo is something like this:

• bases
  • apps
    • app1
    • app2
• components
  • apps
    • app_lib1
  • libs
    • lib1
    • lib2

lib2 depends on lib1, and in the pyproject.toml of app1:

[tool.poetry]
packages = [
    {from = "../../bases", include = "apps/app1"},
    {from = "../../components", include = "libs/lib1"},
    {from = "../../components", include = "libs/lib2"},
]

What should I do if I want to publish lib1 and lib2 as 2 pypi packages, and avoiding shipping the code of lib1 in the wheel package built for lib2?

[DavidVujic]

Hi @inoc603!

If you are going to publish libraries from your Polylith repo, you might want to look into the code-generation features of the build-project command (i.e. --with-top-namespace). This is useful if a user app will install more than one library originated from the monorepo. This, because they share the same top namespace and since Python installs libraries in a shared & flat structure it might cause unwanted problems (such as when uninstalling or updating one of those packages).

If I understand your question about lib1 and lib2: if the second library depends on the first one I think it makes sense to include it. Otherwise the second library would be broken, right?

An alternative approach: In this case, maybe the libraries lib1 and/or lib2 doesn't fit the use case of Polylith? Both, or one of them, could live in a separate repo outside of the monorepo. The code in the Polylith workspace - and the external repos - could then add them as third-party libraries.

[DavidVujic]

You can have the code in the Monorepo, and publish separate libraries that the projects use internally as third-party dependencies. The main use case for Polylith is to avoid those kinds of roundtrips via libraries, but instead use the code directly as bricks.

If there are others, outside of the monorepo, that would have use for the code: then I think libraries is a great idea! I don't think it should be a problem with the duplicated code in the virtual environment, because that would assert that the third-party packages are installed in isolation.

Otherwise, you could have the lib2 depend on lib1 as a third-party package (the roundtrip to pypi/artifact repository as mentioned above) if you want the to have libraries with sub-dependencies.

I think your use case is very interesting, please keep me updated in how you will proceed!

[inoc603]

I wrote a poetry plugin that generates pyproject.toml for all sub projects in a polylith monorepo, setting tool.poetry.packages and add dependencies according to the code included. It works well and keeps third party dependencies in sync. I was thinking about identifying the components which should be published as pypi packages and generate pyproject.toml differently.

But that will require a lot of assumption like where the package is going to be published and making sure that all dependencies are published together. Should be doable but probably too much effort for my tasks at hand. I'm just going to extract these libraries out to separate repos for now. 😞

[DavidVujic]

This was also interesting, thank you for sharing! 😄

I am curious about your plugin, what does it do that the polylith (and multiproject) plugin doesn't do?

[DavidVujic]

Oh, does it add/sync third-party deps maybe?

[inoc603]

Yes, it adds third-party deps and make sure all sub projects are using the same version of libraries as the root pyproject.toml. Our workflow of managing dependencies is running poetry add at the root project and let the plugin filter and sync dependencies to sub projects.