Fixes esmf-org/esmf#136, fix cygwin build

[harmenwierenga]

At Deltares we rely on a Cygwin build of ESMF. There was an attempt at updating ESMF two few years ago, see #136. I have now managed to create a working build of ESMF 8.8.0, and it required me to make a few changes to build everything (including PIO) for Windows.

I installed the following packages in cygwin (not 100% sure that this list is exhaustive, since some other packages were already installed):

• gcc-core 12.4.0-3
• gcc-fortran 12.4.0-3
• gcc-g++ 12.4.0-3
• make 4.4.1-2
• libnetcdf-devel 4.9.2-2
• libnetcdf-fortran-devel 4.5.4-1
• zlib-devel 1.3.1-1
• libhwloc-devel 2.10.0-1
• libevent-devel 2.1.12-1
• cmake 3.28.3-1

set -o igncr
export ESMF_DIR=/cygdrive/c/<ESMF_CHECKOUT_DIR>
export ESMF_COMM=mpiuni
export ESMF_OPTLEVEL=2
export ESMF_NETCDF=split
export ESMF_NETCDF_INCLUDE=/usr/include
export ESMF_NETCDF_LIBPATH=/usr/lib
export ESMF_INSTALL_PREFIX=install
export ESMF_CXXSTD=sysdefault # some C++ libraries require gcc posix extensions, specifically the sigaction definition
export ESMF_RANLIB=true # ranlib does not work during the install stage, use this no-op

The issues that I encountered:

1. Some POSIX headers are used, and the default C++ standard -std=c++11 turns off GNU extensions of the compiler. I had to export ESMF_CXXSTD=sysdefault to ensure that the standard used is -std=gnu++17 so that GNU extensions were turned on. Going to C++17 did not give any issues.
2. In the PIO CMake (src/Infrastructure/IO/PIO/ParallelIO/src/clib/CMakeLists.txt), there is a check that long long and size_t are the same size. On cygwin, these sizes cannot be retrieved, which makes them both empty strings. This causes the check to fail, because empty strings cannot be parsed as numbers so ${SIZEOF_SIZE_T} EQUAL ${SIZEOF_LONG_LONG} evaluates to false in CMake. I have added a small fix for this that does nothing if either returns an empty string.
3. In the PIO C library (src/Infrastructure/IO/PIO/ParallelIO/src/clib/pioc_support.c) there is an include of execinfo.h. This header is missing on many platforms, including Cygwin. I have added a (I believe portable) check to see if the header is available, and I have excluded the include and the use of the backtrace functionality from that file when execinfo.h is not available. The backtrace functionality appears to be added as some useful logging for debugging purposes, but not as a necessary feature of the library, so I hope I was correct that it could simply be turned off. I have tried to suppress any warnings of unused variables.
4. The order of linker flags of NetCDF and PIOC was wrong: it was -lnetcdff -lnetcdf -lpioc, but since pioc requires NetCDF, it will add a bunch of extra missing symbols that then cannot be found any more. It could be fixed through a bit of a hack (I set export ESMF_NETCDF_LIBS="-lpioc -lnetcdff -lnetcdf", in hindsight I should have probably export ESMF_PIO_LIBS="-lpioc -lnetcdf"). Instead, I now reordered PIO and NetCDF in the makefile for third party components, so that the -lpioc flag gets added before netcdf.
5. During the install, ranlib kept complaining that it did not understand the binaries. Since it appears to be an optimization for the linker, but does not appear necessary in this case, I export ESMF_RANLIB=true, since true is a no-op command in bash. This was also a bit of a hack.

This PR addresses points 2, 3, and 4. Please let me know if I can do more to get this merged!

[danrosen25]

@harmenwierenga
Hello and thank you for your contributions.

The PIO code change suggestions should be requested to the PIO repository here. We can then upgrade the PIO version within ESMF. The code maintainer of PIO @jedwards4b will review your suggestions once a PR is opened.

In the common.mk file the PIO library information should follow the NetCDF library information because there are (sort of) checks related to NetCDF. I would suggest prepending the -lpio flag to the third party library flags ESMF_CXXLINKLIBSTHIRD ESMF_F90LINKLIBSTHIRD. See example. Variable expansion is a little tricky in make but it should work. We can also think rewriting the entire PIO block to be within NETCDF

ifdef ESMF_PIO_LIBS
ESMF_CXXLINKLIBSTHIRD     := $(ESMF_PIO_LIBS) $(ESMF_CXXLINKLIBSTHIRD)
ESMF_CXXLINKRPATHSTHIRD   += $(addprefix $(ESMF_CXXRPATHPREFIX),$(subst -L,,$(filter -L%,$(ESMF_PIO_LIBS))))
ESMF_F90LINKLIBSTHIRD     := $(ESMF_PIO_LIBS) $(ESMF_F90LINKLIBSTHIRD)
ESMF_F90LINKRPATHSTHIRD   += $(addprefix $(ESMF_F90RPATHPREFIX),$(subst -L,,$(filter -L%,$(ESMF_PIO_LIBS))))
endif

or maybe

ifdef ESMF_PIO_LIBS
ESMF_CXXLINKLIBSTHIRD     := $(addprefix $(ESMF_PIO_LIBS) ,$(ESMF_CXXLINKLIBSTHIRD))
ESMF_CXXLINKRPATHSTHIRD   += $(addprefix $(ESMF_CXXRPATHPREFIX),$(subst -L,,$(filter -L%,$(ESMF_PIO_LIBS))))
ESMF_F90LINKLIBSTHIRD     := $(addprefix $(ESMF_PIO_LIBS) ,$(ESMF_F90LINKLIBSTHIRD))
ESMF_F90LINKRPATHSTHIRD   += $(addprefix $(ESMF_F90RPATHPREFIX),$(subst -L,,$(filter -L%,$(ESMF_PIO_LIBS))))
endif

[danrosen25]

• Prepend to make variable instead of moving PIO block.
• Send PIO contributions to PIO repository.

[DWesl]

From a CI setup I did a while back in #202, you could add to the list of Cygwin packages:

• openmpi
• perl
• make
• bash
• dash
• liblapack-devel
• python39-devel
• python39-numpy

I also managed to get it working with ESMF_RANLIB=ranlib, but things might have changed since then. I may have just not gotten PIO working, instead focusing on getting ESMPy to build.

Your idea of changing the C++ standard is more elegant than my method of just defining visibility macros until ESMF compiled

[harmenwierenga]

@harmenwierenga Hello and thank you for your contributions.

The PIO code change suggestions should be requested to the PIO repository here. We can then upgrade the PIO version within ESMF. The code maintainer of PIO @jedwards4b will review your suggestions once a PR is opened.

In the common.mk file the PIO library information should follow the NetCDF library information because there are (sort of) checks related to NetCDF. I would suggest prepending the -lpio flag to the third party library flags ESMF_CXXLINKLIBSTHIRD ESMF_F90LINKLIBSTHIRD. See example. Variable expansion is a little tricky in make but it should work. We can also think rewriting the entire PIO block to be within NETCDF

ifdef ESMF_PIO_LIBS
ESMF_CXXLINKLIBSTHIRD     := $(ESMF_PIO_LIBS) $(ESMF_CXXLINKLIBSTHIRD)
ESMF_CXXLINKRPATHSTHIRD   += $(addprefix $(ESMF_CXXRPATHPREFIX),$(subst -L,,$(filter -L%,$(ESMF_PIO_LIBS))))
ESMF_F90LINKLIBSTHIRD     := $(ESMF_PIO_LIBS) $(ESMF_F90LINKLIBSTHIRD)
ESMF_F90LINKRPATHSTHIRD   += $(addprefix $(ESMF_F90RPATHPREFIX),$(subst -L,,$(filter -L%,$(ESMF_PIO_LIBS))))
endif

or maybe

ifdef ESMF_PIO_LIBS
ESMF_CXXLINKLIBSTHIRD     := $(addprefix $(ESMF_PIO_LIBS) ,$(ESMF_CXXLINKLIBSTHIRD))
ESMF_CXXLINKRPATHSTHIRD   += $(addprefix $(ESMF_CXXRPATHPREFIX),$(subst -L,,$(filter -L%,$(ESMF_PIO_LIBS))))
ESMF_F90LINKLIBSTHIRD     := $(addprefix $(ESMF_PIO_LIBS) ,$(ESMF_F90LINKLIBSTHIRD))
ESMF_F90LINKRPATHSTHIRD   += $(addprefix $(ESMF_F90RPATHPREFIX),$(subst -L,,$(filter -L%,$(ESMF_PIO_LIBS))))
endif

@danrosen25 Thank you for the suggestion. I have taken it over, and I will try to send the other contributions to PIO.

[harmenwierenga]

From a CI setup I did a while back in #202, you could add to the list of Cygwin packages:

• openmpi
• perl
• make
• bash
• dash
• liblapack-devel
• python39-devel
• python39-numpy

I also managed to get it working with ESMF_RANLIB=ranlib, but things might have changed since then. I may have just not gotten PIO working, instead focusing on getting ESMPy to build.

Your idea of changing the C++ standard is more elegant than my method of just defining visibility macros until ESMF compiled

Thank you for the response. In my case, I used mpiuni, so I do not need openmpi, and the default compilation does not require lapack either. make was also in my list indeed. I also did not try to run the tests, so that is probably why I did not require the others.

ranlib is needed during the install step, I could not get it to work with any version of ranlib that I found in cygwin.

It would be great if an automatic test could be set up after this!

[harmenwierenga]

I have also created a PR to PIO: NCAR/ParallelIO#2002

Edit: These changes were merged to PIO. Their last official release is from August 2023, would ESMF want to wait for a new PIO release to update it?

[DWesl]

I have also created a PR to PIO: NCAR/ParallelIO#2002

Edit: These changes were merged to PIO. Their last official release is from August 2023, would ESMF want to wait for a new PIO release to update it?

I suspect a reply would be more successful than an edit in getting ESMF maintainer attention on this thread

[harmenwierenga]

I have also created a PR to PIO: NCAR/ParallelIO#2002
Edit: These changes were merged to PIO. Their last official release is from August 2023, would ESMF want to wait for a new PIO release to update it?

I suspect a reply would be more successful than an edit in getting maintainer attention

Yes, I have replied to and tagged @danrosen25 5 days ago.

[jedwards4b]

https://github.com/NCAR/ParallelIO/releases. New release pio2_6_5