Support intel-compiler-llvm (oneapi) compilers

[harshula]

https://forum.access-hive.org.au/t/how-to-build-access-om2-on-gadi-using-intel-oneapi-compilers/1574/1

[access-hive-bot]

This issue has been mentioned on ACCESS Hive Community Forum. There might be relevant details there:

https://forum.access-hive.org.au/t/how-to-build-access-om2-on-gadi-using-intel-oneapi-compilers/1574/3

[harshula]

Notes

https://www.intel.com/content/www/us/en/developer/articles/guide/porting-guide-for-ifort-to-ifx.html

[harshula]

Status: IFX internal compiler errors occur. e.g. #101 (comment)

[penguian]

Intel has discontinued the ifort compiler as at Intel OneAPI 2025.
See the Release Notes, Developer Guide and Reference and the updated Porting Guide.

[harshula]

Thanks @penguian , @manodeep had a chat with me before the holidays and said he was going to look into the state of IFX. I opened ACCESS-NRI/ACCESS-OM2#93.

[manodeep]

Thanks Harshula. Yes, I have been looking into compiling the packages with with oneAPI compilers - so far without success, at least within spack. I am currently building from git source directly (oasis3-mct right now) but running into strange configure error with ifx. Building with the latest classic intel compiler (2021.10.0), with all of the optimisation flags turned on, works fine.

[manodeep]

Thanks for the Release Notes link Paul - that shows the autoconf error that I am seeing when swapping out the classic with the oneAPI compilers in the section "Configure Script Reports: "linking to Fortran libraries from C fails".

(Unfortunately, the suggested fix did not work for me, even after loading the latest autoconf module on NCI)

[manodeep]

Success - I managed to circumvent the ifx ICE by splitting up the offending line into two lines, and assigned to an intermediate variable.

[~/codes/oneapi_compilation_dir/COSIMA-libaccessom2/oasis3-mct @gadi07] git diff lib/psmile/
diff --git a/lib/psmile/src/mod_oasis_method.F90 b/lib/psmile/src/mod_oasis_method.F90
index 57b6bc9..5684c57 100644
--- a/lib/psmile/src/mod_oasis_method.F90
+++ b/lib/psmile/src/mod_oasis_method.F90
@@ -614,6 +614,10 @@ CONTAINS
    character(len=*),parameter :: subname = 'oasis_get_intercomm'
 !  ---------------------------------------------------------
 
+!  Added by MS to try to circumvent ifx ICE
+   CHARACTER(len=:), ALLOCATABLE :: trim_compnm
+
+
    call oasis_debug_enter(subname)
    if (present(kinfo)) then
       kinfo = OASIS_OK
@@ -649,7 +653,9 @@ CONTAINS
        CALL oasis_flush(nulprt)
    ENDIF
 
-   tag=ICHAR(TRIM(compnm))+ICHAR(TRIM(cdnam))
+   trim_compnm=TRIM(compnm)
+   tag=ICHAR(trim_compnm)
+   tag=tag+ICHAR(TRIM(cdnam))
    CALL mpi_intercomm_create(mpi_comm_local, 0, MPI_COMM_WORLD, &
                              mpi_root_global(il), tag, new_comm, ierr)

I don't think this allocatable string requires an explicit deallocate but may be a Fortran expert can chime in here. @micaeljtoliveira ?

[micaeljtoliveira]

I don't think this allocatable string requires an explicit deallocate but may be a Fortran expert can chime in here.

No, here it doesn't. The memory will be automatically deallocated when the string goes out of scope.

[manodeep]

Thanks @micaeljtoliveira !

[manodeep]

While trying to compile ACCESS-ESM1.6 with the oneapi compilers, I keep getting an error with gcc-runtime:

[esm1p6-dev] [~/codes/access-nri/ACCESS-ESM1.6 @gadi08] time spack concretize -f --fresh
==> Error: concretization failed for the following reasons:

   1. cannot satisfy a requirement for package 'gcc-runtime'.

real    2m18.731s
user    1m18.631s
sys     0m31.596s

• list of compilers from spack:

[esm1p6-dev] [~/codes/access-nri/ACCESS-ESM1.6 @gadi08] spack compilers
==> Available compilers
-- clang rocky8-x86_64 ------------------------------------------
clang@18.1.8  clang@17.0.6

-- gcc rocky8-x86_64 --------------------------------------------
gcc@8.5.0  gcc@14.1.0  gcc@13.2.0  gcc@12.2.0  gcc@11.1.0  gcc@10.3.0

-- intel rocky8-x86_64 ------------------------------------------
intel@2021.8.0  intel@2021.6.0  intel@2021.4.0  intel@2021.2.0   intel@2021.10.0  intel@19.1.3.304  intel@19.1.1.217  intel@19.0.5.281  intel@19.0.3.199
intel@2021.7.0  intel@2021.5.0  intel@2021.3.0  intel@2021.11.1  intel@2021.1     intel@19.1.2.254  intel@19.1.0.166  intel@19.0.4.243

-- oneapi rocky8-x86_64 -----------------------------------------
oneapi@2024.2.1  oneapi@2024.2.0  oneapi@2024.0.2  oneapi@2023.2.0  oneapi@2023.0.0  oneapi@2022.2.0  oneapi@2022.1.0  oneapi@2022.0.0  oneapi@2021.4.0  oneapi@2021.3.0  oneapi@2021.2.0  oneapi@2021.1

• changes to the spack.yaml file (main branch)

[esm1p6-dev] [~/codes/access-nri/ACCESS-ESM1.6 @gadi08] git diff
diff --git a/spack.yaml b/spack.yaml
index a04cbc5..de55196 100644
--- a/spack.yaml
+++ b/spack.yaml
@@ -58,11 +58,11 @@ spack:
     # Preferences for all packages
     all:
       require:
-        - '%intel@19.0.3.199'
+        - '%oneapi@2024.2.1'
         - 'target=x86_64_v4'
   view: true
   concretizer:
-    unify: true
+    unify: false
   modules:
     default:
       tcl:

@harshula Do you have thoughts on what I should be doing here?

[harshula]

Hi @manodeep , Is that error triggered without your changes to the spack.yaml?

[manodeep]

@harshula Yup - my attempts at changing to oneapi lead to this error.

[harshula]

@manodeep , Can you please clarify. Does that error occur without your changes to spack.yaml?

[manodeep]

Didn't check without the changes but presumably that worked.

In any case, I have managed to resolve the error by adding the following:

gcc-runtime:
    require:
      - '%gcc@14.1.0'

[manodeep]

@harshula I added the four sub-issues, which are part of my ESM1.6 work, but I now notice that the parent issue is on the ACCESS-OM2 project board. Did you want me to remove the sub-issues from here and attach to a different parent issue within the ESM1.6 project board?

[harshula]

Hi @manodeep , This issue was opened for ACCESS-OM2 components. Do you have a summary of your findings of IFX with the ACCESS-OM2 components?

For your ACCESS-ESM1.6 work, you should create a parent issue in the ACCESS-ESM1.6 repository.

Furthermore, try to open an issue against spack-packages when the issue is with the Spack Package Recipe (SPR). If the issue is with the package's source code, open the issue against the respective repository.

[manodeep]

Hi @manodeep , This issue was opened for ACCESS-OM2 components. Do you have a summary of your findings of IFX with the ACCESS-OM2 components?

@harshula Since ESM1.6 is what is being planned to run - that's what I am looking at currently. Some previously captured details about compiling with oneAPI are here: https://hackmd.io/@manodeep/access-nri-oneapi-compile-status

For your ACCESS-ESM1.6 work, you should create a parent issue in the ACCESS-ESM1.6 repository.

Will do. But the sub-issues can only have one parent - so I will point these to the ESM1.6 project board instead of here.

Furthermore, try to open an issue against spack-packages when the issue is with the Spack Package Recipe (SPR). If the issue is with the package's source code, open the issue against the respective repository.

If I wanted to change the source code or the code in the build system, then sure thing - I would open the issue in the respective repo. But based on y'all and specifically Aidan for using spack ("eating our own dog food") - I would have thought that we want to incorporate changes into the spack-packages. Otherwise, we will end up fixing the make/cmake/autotools once, and then within the spack recipe again.

As a specific example, Micael has successfully built access-fms with the latest oneAPI compiler.

[harshula]

Hi @manodeep ,

I would have thought that we want to incorporate changes into the spack-packages. Otherwise, we will end up fixing the make/cmake/autotools once, and then within the spack recipe again.

Spack uses the package's make/cmake/autotools to build the package. I suspect the confusion is due to some ACCESS-OM2 packages containing Makefile snippets in the SPR that are included by the packages's Makefile. That is not the case for all SPRs. You need to read the SPR first and check whether the build failure is due to code in the SPR or whether the build failure is due to the package's build system. Then open the issue in the appropriate repository. I'm happy to have a virtual chat next week, if you want me to show you some examples.

[manodeep]

Let me give an example - perhaps that will make it clearer what I mean about duplicating the work. The UM7 spack package contains these lines which effectively write out the build config and then invokes fcm to build the package. Now, if I needed to fix/add something - I would need to first fix in the bld-hadgmem3-mct.cfg file within the source repo, test that fix, and then re-implement that fix within the spack package recipe. Is that not correct?

[harshula]

Hi @manodeep , Broadly speaking, there are two types of SPRs. Let's call them Type A SPRs and Type B SPRs. You seem to be solely focused on Type A SPRs, perhaps, from when you started looking at ACCESS-OM2. You need to be able to identify Type B SPRs that do not contain compiler flags. e.g. access-fms.

[manodeep]

Let me see if I understand - do you mean that for those packages where spack is acting as a wrapper (e.g., access-fms), I should open oneAPI build failure issue in the source repo (rather than spack-packages)?

For the other kind, where spack is re-implementing bits of the package build system (e.g., UM7) - where would you like me to open the issue - the source repo or the spack-packages? And how should we go about fixing the build failures - fix in source first and then apply fix to spack recipe?

(Since the atmosphere and ocean components take majority of the runtime, I am focussing on those two first. Both those spack recipes are of "Type A" and contain compiler flags)

[manodeep]

As a specific example, Micael has successfully built access-fms with the latest oneAPI compiler.

Correction: @micaeljtoliveira built the upstream version of FMS with oneAPI and not access-fms

[harshula]

Hi @manodeep ,

do you mean that for those packages where spack is acting as a wrapper (e.g., access-fms), I should open oneAPI build failure issue in the source repo (rather than spack-packages)?

For Type B SPRs, yes, please. The content of the SPR should provide guidance on where to open the issue. e.g. Issue #198 , that you opened, is unlikely to be a problem with the SPR.

For Type A SPRs, if you've identified that the problem is with the SPR, please continue to open issues in spack-packages. Perhaps @micaeljtoliveira and @aidanheerdegen could provide guidance on the question of whether to also open an issue in the package's source repo?

[aidanheerdegen]

Happy to put our heads together if this still needs clarification.

[manodeep]

We might want to get together and sort out the oneAPI building issues.

Looks like oneAPI 2025 (intel-compiler-llvm/2025.0.4) has been installed on gadi

[harshula]

Repo	Branch	Tag with oneapi support
cice4	access-esm1.5	access-esm1.5-2025.03.001
cice5	master	2025.03.001
cice5	acesss-esm1.6	access-esm1.6-2025.03.001
mom5	master	2025.03.001
mom5	access-esm1.5	access-esm1.5-2025.03.001
mom5	development	dev-2025.03.001
oasis3-mct	master	2025.03.001
oasis3-mct	access-esm1.5	access-esm1.5-2025.03.001

[manodeep]

@harshula Are you updating the spack package recipes to build with oneAPI? If so, could we build with oneAPI + the latest netcdf + openmpi modules (netcdf/4.9.2p, openmpi/5.0.5 as of now)?