Restore previous builds of JLLs which (currently) depend on JLLWrappers 1.7 #122129

giordano · 2024-12-29T11:04:36Z

Bash script used to produce this PR:

#!/bin/bash

set -euo pipefail

GENERAL_REPO="${HOME}/repo/General"

JLLS=(
    OpenSSL_jll
    Hwloc_jll
    JpegTurbo_jll
    libpng_jll
    libsixel_jll
    Bzip2_jll
    FFTW_jll
    Giflib_jll
    LERC_jll
    Expat_jll
    Libuuid_jll
    Libmount_jll
    OpenSpecFun_jll
    Lz4_jll
    LZO_jll
    Pixman_jll
    Libffi_jll
    Qhull_jll
    Zstd_jll
    SQLite_jll
    XZ_jll
    libaec_jll
    boost_jll
    brotli_jll
    pugixml_jll
    yaml_cpp_jll
    snappy_jll
    MPItrampoline_jll
    Blosc_jll
    Blosc2_jll
    libzip_jll
    libsodium_jll
    ZeroMQ_jll
    Libgpg_error_jll
    libusb_jll
    Nettle_jll
    Xorg_libXau_jll
    Xorg_xcb_proto_jll
    Xorg_libpthread_stubs_jll
    Xorg_xtrans_jll
    Xorg_libXdmcp_jll
    Xorg_libXext_jll
    Xorg_libX11_jll
    Xorg_libxcb_jll
)

for jll in "${JLLS[@]}"; do
    dir="jll/$(echo ${jll:0:1} | tr [:lower:] [:upper:])/${jll}"
    compat="${dir}/Compat.toml"
    versions="${dir}/Versions.toml"
    last_hash=$(git -C "${GENERAL_REPO}" log -1 --oneline --format=%H -- "${compat}")
    # Restore Compat.toml to previous state
    git -C "${GENERAL_REPO}" reset "${last_hash}^" -- "${compat}"
    git -C "${GENERAL_REPO}" restore -- "${compat}"
    # Yank last version
    if [[ $(tail -n 1 "${GENERAL_REPO}/${versions}") != "yanked = true" ]]; then
        echo "yanked = true" >> "${GENERAL_REPO}/${versions}"
    fi
    git -C "${GENERAL_REPO}" add "${versions}"
done

With

% find jll -name 'Compat.toml' -exec grep -H 'JLLWrappers.*1\.7\.0' '{}' \;
jll/R/Reactant_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/N/nghttp2_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/G/GMP_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/Z/Zlib_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/S/SCIP_PaPILO_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/F/Fmt_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/O/OpenLDAPClient_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/O/OpenLibm_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/H/HelloWorldC_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/H/HELICS_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/M/MbedTLS_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/D/dSFMT_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/E/Enzyme_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/L/LibCURL_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/L/LibSSH2_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/L/LibUV_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/L/libblastrampoline_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/L/LibGit2_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/P/PCRE2_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/P/PDAL_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/P/p7zip_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/W/Wayland_protocols_jll/Compat.toml:JLLWrappers = "1.7.0-1"
jll/W/Wasmtime_jll/Compat.toml:JLLWrappers = "1.7.0-1"

you can then see that all packages which still refer to JLLWrappers 1.7.0 are either stdlibs or packages which which had a brand new version (Reactant, Enzyme, SCIP_PaPILO, Fmt, OpenLDAPClient_jll, HelloWorldC, HELICS, Wayland_protocols, PDAL, Wasmtime).

giordano · 2024-12-29T14:51:11Z

New approach: duplicate previous build, which didn't depend on JLLWrappers. New version of the script:

#!/bin/bash

set -euo pipefail

GENERAL_REPO="${HOME}/repo/General"

JLLS=(
    OpenSSL_jll
    Hwloc_jll
    JpegTurbo_jll
    libpng_jll
    libsixel_jll
    Bzip2_jll
    FFTW_jll
    Giflib_jll
    LERC_jll
    Expat_jll
    Libuuid_jll
    Libmount_jll
    OpenSpecFun_jll
    Lz4_jll
    LZO_jll
    Pixman_jll
    Libffi_jll
    Qhull_jll
    Zstd_jll
    SQLite_jll
    XZ_jll
    libaec_jll
    boost_jll
    brotli_jll
    pugixml_jll
    yaml_cpp_jll
    snappy_jll
    MPItrampoline_jll
    Blosc_jll
    Blosc2_jll
    libzip_jll
    libsodium_jll
    ZeroMQ_jll
    Libgpg_error_jll
    libusb_jll
    Nettle_jll
    Xorg_libXau_jll
    Xorg_xcb_proto_jll
    Xorg_libpthread_stubs_jll
    Xorg_xtrans_jll
    Xorg_libXdmcp_jll
    Xorg_libXext_jll
    Xorg_libX11_jll
    Xorg_libxcb_jll
)

for jll in "${JLLS[@]}"; do
    dir="jll/$(echo ${jll:0:1} | tr [:lower:] [:upper:])/${jll}"
    compat="${dir}/Compat.toml"
    versions="${dir}/Versions.toml"
    last_hash=$(git -C "${GENERAL_REPO}" log -1 --oneline --format=%H -- "${compat}")
    # Restore Compat.toml to previous state
    git -C "${GENERAL_REPO}" reset "${last_hash}^" -- "${compat}"
    git -C "${GENERAL_REPO}" restore -- "${compat}"
    # Clean up any local changes to Versions.toml
    git -C "${GENERAL_REPO}" restore --staged -- "${versions}"
    git -C "${GENERAL_REPO}" checkout -- "${versions}"
    # Find second to last version and bump build number by 2
    old_version_entry=$(tail -n 5 "${GENERAL_REPO}/${versions}"|head -n 2)
    new_build_version=$(($(echo "${old_version_entry}"|perl -lne 'print $1 if /[0-9.]+\+([0-9]+)/;') + 2))
    # Append new entry, duplicate of the second to last one but change the build number.
    echo >> "${GENERAL_REPO}/${versions}"
    echo "$(echo "${old_version_entry}" | sed -E "s/([0-9.]+\+)[0-9]+/\1${new_build_version}/")" >> "${GENERAL_REPO}/${versions}"
    git -C "${GENERAL_REPO}" add "${versions}"
done

* New version: FFTW_jll v3.3.10+2 UUID: f5851436-0d7a-5f13-b9de-f02708fd171a Repo: https://github.com/JuliaBinaryWrappers/FFTW_jll.jl.git Tree: 5cf2433259aa3985046792e2afc01fcec076b549 Registrator tree SHA: 191228b6dd8b9d0e2965ae3e705fe54c51dcfee8 * Update Compat.toml --------- Co-authored-by: Mosè Giordano <765740+giordano@users.noreply.github.com>

DilumAluthge · 2024-12-30T13:38:30Z

@giordano Should the PR title say "Restore previous builds of JLLs which depend on JLLWrappers 1.2" instead?

giordano · 2024-12-30T13:42:22Z

it's not necessarily 1.2, could also be 1.4 for some (didn't check)
the "which" refers to the JLLs which currently depend on JLLWrappers 1.7

DilumAluthge · 2024-12-30T13:43:39Z

2. the "which" refers to the JLLs which currently depend on JLLWrappers 1.7

Ah, yeah that was my confusion. Maybe let's update the PR title to clarify that.

DilumAluthge · 2024-12-30T13:47:23Z

it's not necessarily 1.2, could also be 1.4 for some (didn't check)

Yeah, I should've written "1.2 through 1.6".

giordano · 2024-12-30T13:49:44Z

Any comment on the content of the PR besides the title?

DilumAluthge · 2024-12-30T13:54:20Z

Yeah. I'm a little confused by the PR changes. Give me a minute, let me find an example.

DilumAluthge · 2024-12-30T14:13:47Z

Okay, so using OpenSpecFun_jll as an example.

So, originally, we had OpenSpecFun_jll v0.5.5+0, which depended on JLLWrappers ^1.2.0.

And then OpenSpecFun_jll v0.5.5+1 was released, which depended on JLLWrappers ^1.7.0. This unfortunately changed the compat entries for all OpenSpecFun_jll v0.5.5+* to be JLLWrappers ^1.7.0.

So now this PR makes two changes to OpenSpecFun_jll:

It creates a new version OpenSpecFun_jll v0.5.5+2, which has the exact same git-tree-sha1 as OpenSpecFun_jll v0.5.5+0.
It edits the compat entry for all three OpenSpecFun_jll v0.5.5+* versions (OpenSpecFun_jll v0.5.5+0, OpenSpecFun_jll v0.5.5+1, and OpenSpecFun_jll v0.5.5+2) to be JLLWrappers ^1.2.0.
- This is fine for OpenSpecFun_jll v0.5.5+0 and OpenSpecFun_jll v0.5.5+2.
- **But isn't this technically incorrect for OpenSpecFun_jll v0.5.5+2? The registry now says that OpenSpecFun_jll v0.5.5+2 is fine with any JLLWrappers ^1.2.0. But in reality, OpenSpecFun_jll v0.5.5+2 needs JLLWrappers ^1.7.0.

So this PR technically breaks OpenSpecFun_jll v0.5.5+1, but is the idea of this PR that this breakage is intentional, because we prioritize fixing OpenSpecFun_jll v0.5.5+{0,2} over breaking OpenSpecFun_jll v0.5.5+1?

giordano · 2024-12-30T14:17:51Z

So this PR technically breaks OpenSpecFun_jll v0.5.5+1, but is the idea of this PR that this breakage is intentional, because we prioritize fixing OpenSpecFun_jll v0.5.5+{0,2} over breaking OpenSpecFun_jll v0.5.5+1?

That's what @IanButterworth suggested, yes.

IanButterworth · 2024-12-30T14:19:33Z

The premise being far fewer manifests will have +1 than +0

giordano · 2024-12-30T14:27:04Z

Also, this is unlikely to effectively break the build versions we're overriding: a re-resolution of the package manager would also need to downgrade JLLWrappers to v1.6-, besides not updating the jll packages, which I don't think would happen?

DilumAluthge · 2024-12-30T14:29:09Z

Ah, okay, yeah this PR makes sense to me. Do we want to get any other eyes on it before merging?

@fredrikekre @KristofferC @StefanKarpinski @ericphanson?

DilumAluthge · 2024-12-30T14:38:12Z

Also, I haven't personally had the chance to test this PR to make sure it fixes the problem in existing manifests (although by glancing at the PR diff, and based on the discussion above, it does sound like this PR should fix the problem).

Has anyone tested this PR branch, to make sure it fixes the problem in existing manifests?

DilumAluthge

SGTM based on the above discussion.

DilumAluthge · 2024-12-30T17:04:54Z

Passing along feedback from a colleague: We can verify that this PR fixed the issue for one of our use cases.

giordano added the yank label Dec 29, 2024

giordano requested review from IanButterworth, DilumAluthge and ericphanson December 29, 2024 11:04

giordano temporarily deployed to stopwatch December 29, 2024 11:04 — with GitHub Actions Inactive

Restore previous builds of JLLs which depend on JLLWrappers 1.7.0

0d1d178

giordano force-pushed the mg/yank-jllwrappers branch from f02e0d2 to 0d1d178 Compare December 29, 2024 14:49

giordano temporarily deployed to stopwatch December 29, 2024 14:49 — with GitHub Actions Inactive

giordano removed the yank label Dec 29, 2024

giordano changed the title ~~Yank versions of packages which changed compat bound for JLLWrappers~~ Restore previous builds of JLLs which depend on JLLWrappers 1.7 Dec 29, 2024

DilumAluthge changed the title ~~Restore previous builds of JLLs which depend on JLLWrappers 1.7~~ Restore previous builds of JLLs which (currently) depend on JLLWrappers 1.7 Dec 30, 2024

DilumAluthge approved these changes Dec 30, 2024

View reviewed changes

giordano merged commit 70074d2 into master Dec 30, 2024
8 checks passed

giordano deleted the mg/yank-jllwrappers branch December 30, 2024 15:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restore previous builds of JLLs which (currently) depend on JLLWrappers 1.7 #122129

Restore previous builds of JLLs which (currently) depend on JLLWrappers 1.7 #122129

giordano commented Dec 29, 2024

giordano commented Dec 29, 2024 •

edited

Loading

DilumAluthge commented Dec 30, 2024

giordano commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

giordano commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

giordano commented Dec 30, 2024

IanButterworth commented Dec 30, 2024

giordano commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

DilumAluthge left a comment

DilumAluthge commented Dec 30, 2024

Restore previous builds of JLLs which (currently) depend on JLLWrappers 1.7 #122129

Restore previous builds of JLLs which (currently) depend on JLLWrappers 1.7 #122129

Conversation

giordano commented Dec 29, 2024

giordano commented Dec 29, 2024 • edited Loading

DilumAluthge commented Dec 30, 2024

giordano commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

giordano commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

giordano commented Dec 30, 2024

IanButterworth commented Dec 30, 2024

giordano commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

DilumAluthge commented Dec 30, 2024

DilumAluthge left a comment

Choose a reason for hiding this comment

DilumAluthge commented Dec 30, 2024

giordano commented Dec 29, 2024 •

edited

Loading