Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

enhance generic Bundle easyblock to transfer module requirements of components, but do not create logfile in components #3509

Open
wants to merge 6 commits into
base: develop
Choose a base branch
from
Open

enhance generic Bundle easyblock to transfer module requirements of components, but do not create logfile in components #3509

wants to merge 6 commits into from
+41 −10

Conversation

Thyre
Copy link
Contributor

@Thyre Thyre commented Nov 15, 2024 •
edited by boegel
Loading

This fixes the increased disk usage when running the EasyConfig test suite as log files would stay open.
See easybuilders/easybuild-easyconfigs#21841 for more information.

Requires:

Related:

See #3472 for the initial review and #3504 for the occurring issue, causing a revert of the PR.

@Thyre
Copy link
Contributor Author

Thyre commented Nov 15, 2024

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS GCC-13.3.0.eb
  • SUCCESS Autotools-20231222-GCCcore-13.3.0.eb

Build succeeded for 2 out of 2 (2 easyconfigs in total)
Linux - Linux EndeavourOS UNKNOWN, x86_64, AMD Ryzen 7 7800X3D 8-Core Processor, 1 x NVIDIA NVIDIA GeForce RTX 3070, 565.57.01, Python 3.12.7
See https://gist.github.com/Thyre/91835076bbb9bad8921c1b8608402863 for a full test report.

@Thyre Thyre changed the title Reintroduce #3472 (transfer Bundle requirements to module), but do not transfer logfile Reintroduce #3472 (transfer Bundle requirements to module) but do not create logfile in components Nov 15, 2024
@Thyre Thyre marked this pull request as ready for review November 15, 2024 08:45
@Thyre
Copy link
Contributor Author

Thyre commented Nov 15, 2024

Test report by @Thyre

Overview of tested easyconfigs (in order)

Build succeeded for 9 out of 19 (10 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/1f2efb9ca0b89f91030fc316828e19d2 for a full test report.

@Thyre
Copy link
Contributor Author

Thyre commented Nov 15, 2024 •
edited
Loading

Test report by @Thyre

Overview of tested easyconfigs (in order)

[...]

Build succeeded for 9 out of 19 (10 easyconfigs in total) datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3 See https://gist.github.com/Thyre/1f2efb9ca0b89f91030fc316828e19d2 for a full test report.

Failures due to #3402 missing.

== 2024-11-15 09:07:43,949 build_log.py:171 ERROR EasyBuild crashed with an error (at easybuild/tools/build_log.py:111 in caller_info): No viable license specifications found; specify 'license_file', or define $INTEL_LICENSE_FILE or $LM_LICENSE_FILE (at easybuild/easyblocks/generic/intelbase.py:308 in prepare_step)

@Thyre Thyre mentioned this pull request Nov 18, 2024
Open
4 tasks
@boegel boegel changed the title Reintroduce #3472 (transfer Bundle requirements to module) but do not create logfile in components enhance generic Bundle easyblock to transfer module requirements of components, but do not create logfile in components Dec 4, 2024
@boegel boegel added this to the release after 4.9.4 milestone Dec 4, 2024
boegel
boegel requested changes Dec 4, 2024
View reviewed changes
easybuild/easyblocks/generic/bundle.py Outdated Show resolved Hide resolved
easybuild/easyblocks/generic/bundle.py Outdated Show resolved Hide resolved
easybuild/easyblocks/generic/bundle.py Outdated Show resolved Hide resolved
easybuild/easyblocks/generic/bundle.py Outdated Show resolved Hide resolved
easybuild/easyblocks/generic/bundle.py Outdated Show resolved Hide resolved
easybuild/easyblocks/generic/bundle.py Show resolved Hide resolved
@boegel boegel mentioned this pull request Dec 4, 2024
Open
Thyre added 4 commits December 4, 2024 15:33
@Thyre
Reintroduce easybuilders#3472 after framework changes
14c889f 
Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre
Update clang_aomp for bundle changes
fd3cd65 
Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre
Transfer logfile from bundle to components
43ec355 
This fixes the increased memory usage when running the EasyConfig test suite
as log files would stay open.

Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre
Bundle: Address review comments
bce45f5 
Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre Thyre force-pushed the generic-bundle-include-module-requirements branch from 995cf1b to 16c5dd6 Compare December 4, 2024 14:33
@Thyre
Bundle: Add comment to try-except block in make_module_req_guess()
4ebf4a3 
Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre Thyre force-pushed the generic-bundle-include-module-requirements branch from 16c5dd6 to 4ebf4a3 Compare December 4, 2024 15:35
@boegel boegel changed the title enhance generic Bundle easyblock to transfer module requirements of components, but do not create logfile in components enhance generic Bundle easyblock to transfer module requirements of components, but do not create logfile in components Dec 4, 2024
@boegel
Copy link
Member

boegel commented Dec 4, 2024

Test report by @boegel

Overview of tested easyconfigs (in order)

  • SUCCESS matplotlib-3.8.2-gfbf-2023b.eb
  • **FAIL (unhandled exception: expected str, bytes or os.PathLike object, not list)Traceback (most recent call last):
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/main.py", line 137, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 4282, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 4161, in run_all_steps
    self.run_step(step_name, step_methods)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3996, in run_step
    step_method(self)()
    File "/tmp/eb-6kii0jqq/included-easyblocks-yrkr96bg/easybuild/easyblocks/generic/bundle.py", line 378, in sanity_check_step
    fake_mod_data = self.load_fake_module(purge=True)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 1721, in load_fake_module
    fake_mod_path = self.make_module_step(fake=True)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3763, in make_module_step
    txt += self.make_module_req()
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 1603, in make_module_req
    paths = sum((glob.glob(path) if path else [path] for path in reqs), []) # sum flattens to list
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 1603, in
    paths = sum((glob.glob(path) if path else [path] for path in reqs), []) # sum flattens to list
    File "/usr/lib64/python3.9/glob.py", line 22, in glob
    return list(iglob(pathname, recursive=recursive))
    File "/usr/lib64/python3.9/glob.py", line 43, in _iglob
    dirname, basename = os.path.split(pathname)
    File "/usr/lib64/python3.9/posixpath.py", line 103, in split
    p = os.fspath(p)
    TypeError: expected str, bytes or os.PathLike object, not list
    ** FlexiBLAS-3.3.1-GCC-12.3.0.eb
  • **FAIL (unhandled exception: expected str, bytes or os.PathLike object, not list)Traceback (most recent call last):
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/main.py", line 137, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 4282, in build_and_install_one
    result = app.run_all_steps(run_test_cases=run_test_cases)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 4161, in run_all_steps
    self.run_step(step_name, step_methods)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3996, in run_step
    step_method(self)()
    File "/tmp/eb-6kii0jqq/included-easyblocks-yrkr96bg/easybuild/easyblocks/generic/bundle.py", line 375, in sanity_check_step
    super(Bundle, self).sanity_check_step(*args, **kwargs)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3089, in sanity_check_step
    self._sanity_check_step(*args, **kwargs)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3622, in _sanity_check_step
    self.fake_mod_data = self.sanity_check_load_module(extension=extension, extra_modules=extra_modules)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3549, in sanity_check_load_module
    self.fake_mod_data = self.load_fake_module(purge=True, extra_modules=extra_modules, verbose=True)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 1721, in load_fake_module
    fake_mod_path = self.make_module_step(fake=True)
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 3763, in make_module_step
    txt += self.make_module_req()
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 1603, in make_module_req
    paths = sum((glob.glob(path) if path else [path] for path in reqs), []) # sum flattens to list
    File "/arcanine/scratch/gent/vo/000/gvo00002/vsc40023/easybuild/easybuild-framework/easybuild/framework/easyblock.py", line 1603, in
    paths = sum((glob.glob(path) if path else [path] for path in reqs), []) # sum flattens to list
    File "/usr/lib64/python3.9/glob.py", line 22, in glob
    return list(iglob(pathname, recursive=recursive))
    File "/usr/lib64/python3.9/glob.py", line 43, in _iglob
    dirname, basename = os.path.split(pathname)
    File "/usr/lib64/python3.9/posixpath.py", line 103, in split
    p = os.fspath(p)
    TypeError: expected str, bytes or os.PathLike object, not list
    ** X11-20231019-GCCcore-13.2.0.eb

Build succeeded for 1 out of 3 (3 easyconfigs in total)
node3141.skitty.os - Linux RHEL 9.4, x86_64, Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz (skylake_avx512), Python 3.9.18
See https://gist.github.com/boegel/a09672728e55530c7cda76daab6ee7bc for a full test report.

@Thyre
Bundle: Replace append with extend when copying reqs
4b0522a 
Signed-off-by: Jan André Reuter <j.reuter@fz-juelich.de>
@Thyre
Copy link
Contributor Author

Thyre commented Dec 4, 2024

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS FlexiBLAS-3.3.1-GCC-12.3.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/d3a7d60b320f3bcb317d4c55d0b79166 for a full test report.

@Thyre
Copy link
Contributor Author

Thyre commented Dec 4, 2024

Test report by @Thyre

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS gfbf-2024a.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
  • SUCCESS X11-20240607-GCCcore-13.3.0.eb

Build succeeded for 8 out of 8 (8 easyconfigs in total)
datenlager - Linux Ubuntu 24.04, x86_64, AMD Ryzen 7 3700X 8-Core Processor, Python 3.12.3
See https://gist.github.com/Thyre/3e6e109c6a568293be8b285416d628af for a full test report.

@Thyre
Copy link
Contributor Author

Thyre commented Dec 4, 2024

@boegelbot please test @ jsc-zen3
EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb Clang-AOMP-4.5.0-GCCcore-11.2.0.eb"

@boegelbot
Copy link

@Thyre: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de

PR test command 'if [[ develop != 'develop' ]]; then EB_BRANCH=develop ./easybuild_develop.sh 2> /dev/null 1>&2; EB_PREFIX=/home/boegelbot/easybuild/develop source init_env_easybuild_develop.sh; fi; EB_PR=3509 EB_ARGS="buildenv-default-foss-2023b.eb OpenSSL-3.eb Circuitscape-5.12.3-Julia-1.9.2.eb Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb gfbf-2024a.eb crypt4gh-1.7-GCC-12.3.0.eb jiter-0.4.1-GCCcore-12.3.0.eb X11-20240607-GCCcore-13.3.0.eb Clang-AOMP-4.5.0-GCCcore-11.2.0.eb" EB_REPO=easybuild-easyblocks EB_BRANCH=develop /opt/software/slurm/bin/sbatch --job-name test_PR_3509 --ntasks=8 ~/boegelbot/eb_from_pr_upload_jsc-zen3.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 5384

Test results coming soon (I hope)...

- notification for comment with ID 2518605340 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS buildenv-default-foss-2023b.eb
  • SUCCESS OpenSSL-3.eb
  • SUCCESS Circuitscape-5.12.3-Julia-1.9.2.eb
  • SUCCESS Perl-bundle-CPAN-5.38.2-GCCcore-13.3.0.eb
  • SUCCESS gfbf-2024a.eb
  • SUCCESS crypt4gh-1.7-GCC-12.3.0.eb
  • SUCCESS jiter-0.4.1-GCCcore-12.3.0.eb
  • SUCCESS X11-20240607-GCCcore-13.3.0.eb
  • SUCCESS ROCR-Runtime-4.5.0-GCCcore-11.2.0.eb
  • FAIL (build issue) Clang-AOMP-4.5.0-GCCcore-11.2.0.eb (partial log available at https://gist.github.com/boegelbot/d82fb4858e7f8509d04629748780597a)

Build succeeded for 9 out of 10 (9 easyconfigs in total)
jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.19
See https://gist.github.com/boegelbot/6f439391fecc47536c2a2c3e0be99993 for a full test report.

@Thyre
Copy link
Contributor Author

Thyre commented Dec 5, 2024 •
edited
Loading

FAIL (build issue) Clang-AOMP-4.5.0-GCCcore-11.2.0.eb (partial log available at https://gist.github.com/boegelbot/d82fb4858e7f8509d04629748780597a)

Build succeeded for 9 out of 10 (9 easyconfigs in total) jsczen3c1.int.jsc-zen3.fz-juelich.de - Linux Rocky Linux 9.5, x86_64, AMD EPYC-Milan Processor (zen3), Python 3.9.19 See https://gist.github.com/boegelbot/6f439391fecc47536c2a2c3e0be99993 for a full test report.

The gist unfortunately did not include the error. Checking on jsc-zen3 directly, I see the following error message:

/dev/shm/reuter1/ClangAOMP/4.5.0/GCCcore-11.2.0/llvm-project-rocm-4.5.0/compiler-rt/lib/sanitizer_common/sanitizer_platform_limits_posix.cpp:146:10: fatal error: linux/cyclades.h: No such file or directory
  146 | #include <linux/cyclades.h>
      |          ^~~~~~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.x86_64.dir/build.make:345: projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.x86_64.dir/sanitizer_platform_limits_posix.cpp.o] Error 1
make[2]: Leaving directory '/dev/shm/reuter1/ClangAOMP/4.5.0/GCCcore-11.2.0/easybuild_obj'
make[1]: *** [CMakeFiles/Makefile2:21910: projects/compiler-rt/lib/sanitizer_common/CMakeFiles/RTSanitizerCommon.x86_64.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

This looks unrelated to the changes in this PR.

Well, looking into it further, this is a header being removed in newer Linux kernels. Not much we can do about that aside from porting patches, if they exist.

@boegel
Copy link
Member

boegel commented Dec 18, 2024

@boegelbot please test @ generoso
EB_ARGS="Clang-AOMP-4.5.0-GCCcore-11.2.0.eb "

@boegelbot
Copy link

@boegel: Request for testing this PR well received on login1

PR test command 'EB_PR=3509 EB_ARGS="Clang-AOMP-4.5.0-GCCcore-11.2.0.eb " EB_CONTAINER= EB_REPO=easybuild-easyblocks /opt/software/slurm/bin/sbatch --job-name test_PR_3509 --ntasks=4 ~/boegelbot/eb_from_pr_upload_generoso.sh' executed!

  • exit code: 0
  • output:
Submitted batch job 14883

Test results coming soon (I hope)...

- notification for comment with ID 2551801188 processed

Message to humans: this is just bookkeeping information for me,
it is of no use to you (unless you think I have a bug, which I don't).

@boegelbot
Copy link

Test report by @boegelbot

Overview of tested easyconfigs (in order)

  • SUCCESS Clang-AOMP-4.5.0-GCCcore-11.2.0.eb

Build succeeded for 1 out of 1 (1 easyconfigs in total)
cns1 - Linux Rocky Linux 8.9, x86_64, Intel(R) Xeon(R) CPU E5-2667 v3 @ 3.20GHz (haswell), Python 3.6.8
See https://gist.github.com/boegelbot/bcce16a46456242fbcbb9efbd856830d for a full test report.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@boegel boegel boegel requested changes

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
enhancement
Projects
None yet
Milestone
release after 4.9.4
Development

Successfully merging this pull request may close these issues.
4 participants
@Thyre @boegel @boegelbot @bartoldeman