julia-bin: 1.7.3 -> 1.8.0

[wentasah]

Description of changes

• https://julialang.org/blog/2022/08/julia-1.8-highlights/
• https://github.com/JuliaLang/julia/blob/release-1.8/NEWS.md

This is a quick attempt to make Julia 1.8.0 available in Nixpkgs. I tested it on a few of my projects and it seems to work. I only get the following warning from time to time when using Pkg:

┌ Warning: NetworkOptions could only find OpenSSL-specific TLS certificate files which cannot be used by MbedTLS. Please open an issue at https://github.com/JuliaLang/NetworkOptions.jl/issues with details about your system, especially where generic non-OpenSSL certificates can be found. See https://stackoverflow.com/questions/55447752/what-does-begin-trusted-certificate-in-a-certificate-mean for more details.
└ @ NetworkOptions /cache/build/default-amdci4-3/julialang/julia-release-1-dot-8/usr/share/julia/stdlib/v1.8/NetworkOptions/src/ca_roots.jl:104

I'll try to fix it later. For now, I'm marking this as a draft, because I'm not sure how severe the warning is.

I'm keeping version 1.7 in place. There are no big differences in the expression of 1.7 and 1.8, as can be seen from the diff below:

diff --git a/1.7-bin.nix b/1.8-bin.nix
index 5352d38a221..0dbdb3713d2 100644
--- a/1.7-bin.nix
+++ b/1.8-bin.nix
@@ -2,12 +2,12 @@
 
 stdenv.mkDerivation rec {
   pname = "julia-bin";
-  version = "1.7.3";
+  version = "1.8.0";
 
   src = {
     x86_64-linux = fetchurl {
       url = "https://julialang-s3.julialang.org/bin/linux/x64/${lib.versions.majorMinor version}/julia-${version}-linux-x86_64.tar.gz";
-      sha256 = "0ff7ypr76xf99h3dmy1xdnkq2xn432qnzihxs72xrd4j5nhlybwv";
+      sha256 = "sha256-6A1zLMt/eeAA15jLi2Vtw2QatZUW1uTlLhZ2UBeJKgA=";
     };
   }.${stdenv.hostPlatform.system} or (throw "Unsupported system: ${stdenv.hostPlatform.system}");
 
@@ -19,7 +19,7 @@ stdenv.mkDerivation rec {
   '';
   patches = [
     # Source release Nix patch(es) relevant for binary releases as well.
-    ./patches/1.7-bin/0005-nix-Enable-parallel-unit-tests-for-sandbox.patch
+    # (currently we have no patches)
   ];
   postPatch = ''
     # Revert symlink hack.
@@ -48,6 +48,8 @@ stdenv.mkDerivation rec {
   preInstallCheck = ''
     # Some tests require read/write access to $HOME.
     export HOME="$TMPDIR"
+    # Run tests in parallel
+    export JULIA_TEST_USE_MULTIPLE_WORKERS=true
   '';
   installCheckPhase = ''
     runHook preInstallCheck

Closes #189444

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.11 Release Notes (or backporting 22.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[ghost]

On Sat 03 Sep 2022, ofborg[bot] wrote: @ofborg[bot] requested your review on: NixOS/nixpkgs#189656 julia-bin: 1.7.3 -> 1.8.0.

Looks like a great start wentasah! Quick comments: 1. Deprecate 1.7.x in `aliases.nix`, it is no longer supported upstream 2. Remove `patches` and the symlink hack for patches in the expression, there is no need to have a placeholders 3. You can rename `1.7-bin.nix` to `1.8-bin.nix` and delete `patches/1.7-bin` In terms of the certificates, I would have to read the Julia code and think about it more deeply than I have time for right now. It does *sound* serious though.

[rikhuijzer]

Thanks for working on this, @wentasah! Much appreciated. I'm currently running binaries from Julia by just downloading them and running those. I did it by adding a symlink for /lib64/ld-linux-x86-64.so.2 on my system, see https://discourse.nixos.org/t/making-lib64-ld-linux-x86-64-so-2-available/19679/3 for details. I say this for people who run into this issue later and want to run Julia but do not have the right binary available. It's still good to get it in nixpkgs!

I only get the following warning from time to time when using Pkg:

I'm running with that error for months already. Everything works fine. The risk is probably that sophisticated attackers could alter code sent from the package servers. We probably need to teach NetworkOptions.jl where the certificates are on NixOS or symlink the certificates to one of the paths that NetworkOptions.jl knows. I would say that this is outside the scope of this PR.

You're halfway a patch already I see. Would be nice to fix it, but I do think that it's outside the scope of this PR. I fear that it will take long to fix and thus cause Julia 1.8 to not become available as a nixpkg

[ghost]

On Sun 04 Sep 2022, Rik Huijzer wrote: Thanks for working on this, @wentasah! Much appreciated. I'm currently running binaries from Julia by just downloading them and running those. I did it by adding a symlink for `/lib64/ld-linux-x86-64.so.2` on my system, see https://discourse.nixos.org/t/making-lib64-ld-linux-x86-64-so-2-available/19679/3 for details. I say this for people who run into this issue later and want to run Julia but do not have the right binary available. It's still good to get it in nixpkgs!

I have said this before, but it is worth repeating. This hack is *not* compatible with the general philosophy of NixOS and can break both Julia and other software in unexpected ways. My personal position is that if you have to resort to these kinds of hacks, perhaps one should consider not running NixOS and resort to a “standard” Linux distribution and install Nix? Alternatively, run Julia in a VM with a different distribution on NixOS.

> I only get the following warning from time to time when using Pkg: I'm running with that error for months already. Everything works fine. The risk is probably that sophisticated attackers could alter code sent from the package servers. We probably need to teach [`NetworkOptions.jl`](https://github.com/JuliaLang/NetworkOptions.jl) where the certificates are on NixOS or symlink the certificates to one of the paths that `NetworkOptions.jl` knows.

The current code [1] looks safe to me at a glance. If I am correct, Julia simply falls back on using a certificate that is shipped with the installation. Thus the fear of an attacker is unwarranted. [1]: https://github.com/JuliaLang/NetworkOptions.jl/blob/4d3df6445bbca0556a7a9f2abb0e48ba7d774f5a/src/ca_roots.jl

[wentasah]

@ninjin I addressed your three points and pushed an updated version.

I agree with your analysis of the TLS certificate warnings and therefore, I'm removing the draft flag from this PR. I dug even deeper and figured out, that NixOS-provided certificates are probably OK, the issue is in how Julia detects their incompatibility. I submitted the issue, as requested in the warning.

With respect to the dynamic linker, I'm using nix-ld. Formerly, it was necessary to use it in order to have packages like Plots.jl work. It was because their binary artifacts needed libraries like qt and freetype from the host system. Nowadays, these are no longer necessary. My wrapper currently looks this way. It only makes OpenGL libraries available for Makie.jl and similar. I was thinking about upstreaming the wrapper as a NixOS module. Do you know about some packages, which depend on host libraries (besides OpenGL/Vulkan) these days?

[ghost]

On Sun 04 Sep 2022, Michal Sojka wrote: @ninjin I addressed your three points and pushed an updated version. I agree with your analysis of the TLS certificate warnings and therefore, I'm removing the draft flag from this PR. I dug even deeper and figured out, that NixOS-provided certificates are probably OK, the issue is in how Julia detects their incompatibility. I [submitted the issue](JuliaLang/NetworkOptions.jl#24), as requested in the warning.

Thanks, looks good to me! Only minor comment I would have is that you could remove “# Run tests in parallel” as it is kind of obvious from the name of the variable what it does. Other than that, I think this looks good to merge. Thank you for creating that upstream NetworkOptions.jl issue, I have subscribed to it.

With respect to the dynamic linker, I'm using [nix-ld](https://github.com/Mic92/nix-ld). Formerly, it was necessary to use it in order to have packages like Plots.jl work. It was because their binary artifacts needed libraries like qt and freetype from the host system. Nowadays, these are no longer necessary. My wrapper currently [looks this way](https://github.com/wentasah/nix-conf/blob/9981b24de36a320bbb344b795230e14f9ccae016/pkgs/julia-wrapper/default.nix). It only makes OpenGL libraries available for Makie.jl and similar. I was thinking about upstreaming the wrapper as a NixOS module. Do you know about some packages, which depend on host libraries (besides OpenGL/Vulkan) these days?

I have very limited experience with wrappers as it feels like “not the right way” (tm) for at least the issues I have encountered with binary artifacts and Julia. Rather, after discussing with several of the people responsible for Julia packaging and members of the Nix community I think that what we ideally need to do is patch Pkg.jl along the lines of what rustup does and rebuild the Julia standard library. I have code working for the latter, but have not had the time/motivation to work on the latter for over a year. Not sure if this would work with OpenGL/Vulkan since I a do not know how they are packaged for Julia, but it should resolve pretty much every binary artifact out there as they are statically compiled and only need to have their interpreter fixed. In terms of what the interpreter patching would look like, it is fairly simple [1]. What is tricky is how to cleanly do this upon installation and preferably with overlays (was that the correct term?) so that we do not alter the binary artifacts managed by Pkg.jl and break their hashes. It is probably something that can be done in a few days if you know what you are doing. [1]: https://git.sr.ht/~ninjin/julia-nix/commit/1980696738dcca0963de0423c966f4ae8c13432a

[wentasah]

On Sun, Sep 04 2022, ninjin wrote: Thanks, looks good to me! Only minor comment I would have is that you could remove “# Run tests in parallel” as it is kind of obvious from the name of the variable what it does. Other than that, I think this looks good to merge.

Done.

I have very limited experience with wrappers as it feels like “not the right way” (tm) for at least the issues I have encountered with binary artifacts and Julia.

Actually, wrappers are used at many places in Nixpkgs. Basically, all programs that use some kind of plugins (e.g. all Gtk and Qt apps) usually need wrappers, which tell the apps via environment variables, in which store path to find the plugins.

Rather, after discussing with several of the people responsible for Julia packaging and members of the Nix community I think that what we ideally need to do is patch Pkg.jl along the lines of what rustup does and rebuild the Julia standard library.

I agree that this sounds like more correct way. I was afraid of digging into Pkg.jl myself.

I have code working for the latter, but have not had the time/motivation to work on the latter for over a year. Not sure if this would work with OpenGL/Vulkan since I a do not know how they are packaged for Julia,

The problem with OpenGL and Vulkan libraries is that their vendor and usually also version should match the in-kernel drivers. So that they cannot be a part of Julia closure but must be provided by the system. In NixOS, they are available in /run/opengl-driver and binary artefacts don't tell the linker to look there. But if we're able to patch the interpreter, it should be also easy to add /run/opengl-driver to RPATH. However, this would be correct only on NixOS. So maybe, we would need a NixOS wrapper anyway.

but it should resolve pretty much every binary artifact out there as they are statically compiled and only need to have their interpreter fixed. In terms of what the interpreter patching would look like, it is fairly simple [1]. What is tricky is how to cleanly do this upon installation and preferably with overlays (was that the correct term?) so that we do not alter the binary artifacts managed by Pkg.jl and break their hashes. It is probably something that can be done in a few days if you know what you are doing. [1]: https://git.sr.ht/~ninjin/julia-nix/commit/1980696738dcca0963de0423c966f4ae8c13432a

Great I'll definitely look at that.

[ghost]

On Mon 05 Sep 2022, Michal Sojka wrote: On Sun, Sep 04 2022, ninjin wrote: > Thanks, looks good to me! Only minor comment I would have is that you > could remove “# Run tests in parallel” as it is kind of obvious from > the name of the variable what it does. Other than that, I think this > looks good to merge. Done.

You have my mark of approval. @7c6f434c, are you happy to have a last sanity check and merge?

> I have very limited experience with wrappers as it feels like “not the > right way” (tm) for at least the issues I have encountered with binary > artifacts and Julia. Actually, wrappers are used at many places in Nixpkgs. Basically, all programs that use some kind of plugins (e.g. all Gtk and Qt apps) usually need wrappers, which tell the apps via environment variables, in which store path to find the plugins.

No disagreement there, just that it is, from my perspective, generally a last resort when “the true way” (tm) fails. On that note, this entire package is a result of me failing to package Julia “properly” by building it from source and then being hammered with test failure after test failure over two months. '^^

> Rather, after discussing with several of the people responsible for > Julia packaging and members of the Nix community I think that what we > ideally need to do is patch Pkg.jl along the lines of what rustup does > and rebuild the Julia standard library. I agree that this sounds like more correct way. I was afraid of digging into Pkg.jl myself.

The code is not that scary, do have a look if you want to.

> I have code working for the latter, but have not had the > time/motivation to work on the latter for over a year. Not sure if > this would work with OpenGL/Vulkan since I a do not know how they are > packaged for Julia, The problem with OpenGL and Vulkan libraries is that their vendor and usually also version should match the in-kernel drivers. So that they cannot be a part of Julia closure but must be provided by the system. In NixOS, they are available in /run/opengl-driver and binary artefacts don't tell the linker to look there. But if we're able to patch the interpreter, it should be also easy to add /run/opengl-driver to RPATH. However, this would be correct only on NixOS. So maybe, we would need a NixOS wrapper anyway.

Roger that, it sounds sensible and sorry for not being able to provide guidance.

> but it should resolve pretty much every binary artifact out there as > they are statically compiled and only need to have their interpreter > fixed. > > In terms of what the interpreter patching would look like, it is > fairly simple [1]. What is tricky is how to cleanly do this upon > installation and preferably with overlays (was that the correct term?) > so that we do not alter the binary artifacts managed by Pkg.jl and > break their hashes. It is probably something that can be done in a few > days if you know what you are doing. > > [1]: https://git.sr.ht/~ninjin/julia-nix/commit/1980696738dcca0963de0423c966f4ae8c13432a Great I'll definitely look at that.

Please do. Feel free to send me an e-mail if you take it on if you think I can help and know that the people in #releases over on the Julia Slack are very helpful.