Automatic [compat] bounds added on add breaks some workflows

[jonniediegelman]

The new functionality that automatically adds [compat] bounds whenever a dependency is added makes it more difficult to work with a monorepo workflow. By "monorepo workflow" here, I mean specifically workflows that:

• don't use a private registry for internal packages in the repo; internal packages are all deved by path
• check in Manifest.toml files so exact dependencies are pinned for every git commit SHA

In these workflows, compat bounds are only useful in cases where certain dependency versions need to be explicitly restricted from updating due to known issues. It's unnecessary to add them for every dependency, as things will already be pinned by the Manifest files. Beyond that, it makes it difficult to tell which compat bounds were actually added for good reason when all dependencies have compat bounds.

Making everything automatically add compat bounds also forces the use of CompatHelper. This is more difficult for monorepos, since you need to explicitly pass in the package subdirectory paths in order for them to be included. Any time someone creates a new package, they need to know to add to this list (for large organizations, this isn't easy to enforce). Writing a wrapper around CompatHelper.main that automatically crawls the repo and searches for Julia packages isn't a good solution either, since it will regularly change multiple different teams’ projects and require sign-off from each of them each time its run. There’s a lot of extra infrastructure you need to manually build around Pkg if you want it to work with a monorepo workflow (in the very least, you need to write something that resolves the package Manifest files in order whenever a dependency is added to one of them), so little things like this–while they aren’t that hard to implement–just end up making things a lot more complex/brittle with no added benefit.

I'd also argue there is limited benefit to standard registry workflows, as this only adds compat bounds the first time you add a package, it doesn't set up CompatHelper for the repo. Anyone creating a new package that doesn't know about CompatHelper won't receive any messages telling them they need to set it up, they'll just run into unresolvable dependency issues a few weeks/months down the road.

[giordano]

check in Manifest.toml files so exact dependencies are pinned for every git commit SHA

I don't think that's a good reason for not adding compat bounds. I often want to update my environment, whether it has a checked in manifest or not, in a "compatible way", and having compat bounds would enable that. If you don't, you start getting upgrades to breaking releases of packages and good luck dealing with that

[visr]

Here are the config settings pixi offers for pinning: https://pixi.sh/latest/reference/pixi_configuration/#pinning-strategy. For the same reason I use no-pin there, also since there is no dependabot support yet. The default is similar to Julia. So I just run a monthly update action that conveniently highlights breaking changes so they can be reviewed: Deltares/Ribasim#1992.

[jonniediegelman]

If you don't, you start getting upgrades to breaking releases of packages and good luck dealing with that

It's pretty easy actually. Every so often, you try to update everything in your environment. If things break, they fail CI and you either add a compat bound to restrict the version or fix what broke. It's the exact same thing you'd do with CompatHelper.

[giordano]

I was speaking from the experience of helping newbies who don't really do any of that and expect that updating an environment after a while everything will magically work. If you have an advanced CI system to deal with breakage, probably you can also deal with compat bounds. People who don't know that compat bounds can save them frustration are left hopeless.

[jonniediegelman]

People who don't know that compat bounds can save them frustration are left hopeless.

I would imagine those people would be equally frustrated when they start getting upstream unsatisfiable requirements errors because they didn't know they needed to add CompatHelper to their package repo to keep the compat bounds up-to-date.

If you have an advanced CI system to deal with breakage

That's just the regular function of any CI system?

probably you can also deal with compat bounds

Probably. But there is already a lot of friction to using Julia in a monorepo organization (which is a fairly popular setup these days) and little things like this really do add up.

[lassepe]

How about a slight UI change to ]dev path/to/Foo that offers to remove the compat bound for Foo if there is one. Then offer to add it back on ]free and ]add

[Cvikli]

I want to emphasize that this issue is very dangerous. Actually I think lot of us who already working in julia for ages run into problems due to these automatical package version locks. And as it was pointed out by @giordano every newbie will face problems. And lot of them will end up losing their hair to understand the extreme situation that can come out with these package locks.

I think this should be an optional feature (anybody who is knowledgeabel can turn this on if he needs). Till 1.10 everything was magically working. It was totally developer friendly.
We need it back! :D

[StefanKarpinski]

FWIW, these are not package locks. Foo = "1.2.3" means "any version that is compatible with 1.2.3 according to SemVer," which roughly means ≥ 1.2.3 and < 2.0.0 (ignoring prereleases and such). This is can more explicitly be written as Foo = "^1.2.3" which I think would be better. Yes, these mean the same thing, but it's confusing and based on various comments about "locking", people seem to think that the latter syntax locks to a single version.

[StefanKarpinski]

To be clear, I think that at the very least, we should explicitly put Foo = "^1.2.3" in the autogenerated compat entries to avoid this confusion.

[StefanKarpinski]

It's pretty easy actually. Every so often, you try to update everything in your environment. If things break, they fail CI and you either add a compat bound to restrict the version or fix what broke. It's the exact same thing you'd do with CompatHelper.

This sounds great in theory. However, lack of lower bounds ends up being a significant problem for resolving package versions when there are conflicts between the latest versions of packages that someone needs to use at the same time. Why? Because when there aren't any lower compat bounds the resolver thinks "great, I can use this ancient version because it's compatible with everything!". Of course, that ancient version probably doesn't actually work with everything, it's just that no one bothered to put a lower bound on what versions of that package they actually work with so it looks like it should work to the resolver.