fix: support package pinning via http+tar

[maiste]

This PR reduces the gap between opam and dune package management functionnalities. It allows users to ping a tar file (gz or bz) using http (and https) protocol.

Closes #10121

[Leonidas-from-XIV]

Thanks. The ZIP file changes can be added in a separate PR if you want, but I think that it would be useful to support it given it is a rather common format, especially on Windows.

[Leonidas-from-XIV]

@maiste I've looked into the describe.t failure and when I run it locally the issue does not happen, so it seems like that might be another CI failure.

[rgrinberg]

Do we plan to support tarballs on the file system?

[maiste]

Do we plan to support tarballs on the file system?

No, we expect people to untar directory themselves on the file system and point to the path. I don't think it is worth adding the machinery for this as there is already one for file://.

What if the tar file is updated remotely?

If the file is updated remotely:

• you will have to re-run dune pkg lock if you clean the _build or,
• it will keep using the version in the _build without updating it.

I don't think we have a proper when to address it. If we checked on every run, people would have to download the tar file every time they run dune build. In the current configuration, people will have a checksum error if they do dune clean && dune build which is the equivalent of the behavior (good or bad, idk) with opam: you have to run opam reinstall to get the changes.

I would expect people to want to be aware when the checksum change, but having to explicitly state when they want to synchronize the checksum. It would prevent them from downloading tar they don't expect to change.

[rgrinberg]

I think we should re-download the tarball and construct the checksum every time we run dune pkg lock. This is consistent with how we treat all other sources that lack a checksum. Anything else will be bound to create confusion that will have to be solved by reminding everyone to do the equivalent of opam update.

You could still avoid re-downloading if you use curl with the correct e-tag incantations.

[maiste]

I agree with you, and this is what happens in the current situation. If you run dune pkg lock, it will download the tar and update the checksum. What I meant is that it is not going to change if you run dune build and the file changed remotely.

Regarding the etag, @Leonidas-from-XIV suggested this. However, many of the packages on opam-repository live in GitHub Release and from the requests I tried with curl, it does not seem to provide an ETag header. It is still an optimization we can do later.

[rgrinberg]

Okay, thanks. Is there a test that demonstrates that subsequent dune pkg lock re-download it? We should have one.

[maiste]

I'm rewriting my test to show the behavior. It should be ready for a next review round by the end of the day 👍

[Leonidas-from-XIV]

I think this is the wrong error message to show: this shows that the checksum is invalid as a checksum, but we want an error message that the checksum doesn't match.

You can probably just use any valid MD5, like 92449184682b45b5f07e811fdd61d35f.

[maiste]

It seems that having caching efficient is not really easy for the pinning with http and tar. More specifically, it is hard to cache the fetch for when you need the opam file and later when you will need the code. They live in the same directory but of_opam_rule needs the tar to present, and it will generate new rules using the opam file.

To reduce the amount of tar executed, I can "cache" the file somewhere. This way, if the file is still the same, it won't try to unpack it again. I would like to avoid storing it inside the ~/.cache/dune dir because there is a risk it grows exponentially in function of the user usage. I was thinking about storing it under _build/_archive but I'm really not confident with this.

Maybe I'm missing something, but it seems there is no way to avoid the double download without changing the logic.

My proposition is to stick to the behavior where it stores the tar into the _build and let the engine do the fetch when parsing the *.pkg file. It is not optimal but at least the change is not big, and it brings the feature to Dune.

WDTY?

[rgrinberg]

My proposition is to stick to the behavior where it stores the tar into the _build and let the engine do the fetch when parsing the *.pkg file. It is not optimal but at least the change is not big, and it brings the feature to Dune.

IMO you should just write to a temp dir and leave a comment about future improvements. I think placing it in _build just creates a false expectation. You could just fix the re-downloading issue in subsequent PR's. In any case, I don't think this need to block the PR.