MOAB Build Configuration - Dependency Inclusion for Wheel

[ahnaf-tahmid-chowdhury]

I have successfully built the MOAB wheels with HDF5 support enabled for PyMOAB. However, I am facing some decisions regarding which dependencies to include for optimal functionality and size reduction.

Current Setup:

• HDF5: Enabled (required by PyMOAB).
• Testing: Disabled.
• gfortran and LAPACK: Enabled by default.
• Eigen: Disabled by default.

Default Configuration:

• Shared libraries are enabled.
• Support for CGNS, MPI, Zlib, szip, NetCDF, Metis, ParMetis, and Zoltan is disabled by default.
• BLAS/LAPACK and Fortran support are enabled.
• Testing is enabled.

Proposed Changes:

1. Enable Eigen: Turning on Eigen while disabling gfortran and LAPACK could reduce the wheel size by 60%.
2. Disable gfortran and LAPACK: These libraries are currently enabled but may not be necessary, especially if Eigen is used.

Request for Input:

• Should I enable additional support for features such as NetCDF, Metis, or ParMetis for broader functionality, or focus on keeping the wheel lightweight with minimal dependencies?
• Is there a recommended configuration for MOAB that balances performance, functionality, and size when building Python wheels?

You can find the successful builds here.

Tagging: @vijaysm, @gonuke, @pshriwise

[vijaysm]

The reasoning for including LAPACK by default is because most machines have some flavor of BLAS/LAPACK available and if not, it's easy for admin to do so. Eigen3 is almost never available and an admin/user always has to install it. This was more about the ease of dependency.

I am refactoring Matrix3 to eliminate some of these hard constraints. Will submit a PR for review this week.

[vijaysm]

Until my mpi4py branch works with latest master, we may not need any other additional dependency. I was going to suggest Netcdf but in a serial build, not sure how much value that adds.

[gonuke]

The reasoning for including LAPACK by default is because most machines have some flavor of BLAS/LAPACK available and if not, it's easy for admin to do so. Eigen3 is almost never available and an admin/user always has to install it. This was more about the ease of dependency.

I am refactoring Matrix3 to eliminate some of these hard constraints. Will submit a PR for review this week.

But Eigen3 is a header only library and doesn't need an admin to install.

LAPACK requires FORTRAN, and can be the only reason for FORTRAN, thus adding a constraint.

[gonuke]

I recommend reviewing the conda-forge build configurations, since we discussed some of these things there a few years ago.

[ahnaf-tahmid-chowdhury]

Here is the related issue in the conda-forge: conda-forge/moab-feedstock#27

[shimwell]

Link to the dependencies used in conda moab
https://github.com/conda-forge/moab-feedstock/blob/cbcb576742e51eebbb0b73acf5f8b289048ed62b/recipe/meta.yaml#L60-L92

[gonuke]

Here is the related issue in the conda-forge: conda-forge/moab-feedstock#27

I think that's out of date, though, since we do depend on LAPACK in the current build

[gonuke]

They became dependencies again 3 years ago: conda-forge/moab-feedstock#50

Many years ago, removing BLAS/LAPACK dependency was valuable for building on Windows with native compilers. (i.e. no FORTRAN!). I was definitely pushing for it then for local user use cases other than shared HPC use cases.

[ahnaf-tahmid-chowdhury]

Upon reviewing the build.sh script, it seems Eigen is enabled while Fortran is disabled.

[ahnaf-tahmid-chowdhury]

It seems MOAB two versions: a general version and one with MPI support. Therefore, I think, it might be beneficial to implement a similar approach with pip.

[ahnaf-tahmid-chowdhury]

To address the issue of multiple variants, I created an issue on scikit-build-core to explore potential solutions. Based on the feedback, it seems we may need to create separate packages for each variant. Please advise on the next steps for proceeding with this approach.

[vijaysm]

The PR https://bitbucket.org/fathomteam/moab/pull-requests/700 should fix all of these issues. You don't need Eigen3 or LAPACK now as a requirement. The only reason why any of these were a requirement was due to the eigen decomposition code. Now, I have a native implementation that works as well as LAPACK for cases tested. Review the PR and comment there.

[vijaysm]

LAPACK requires FORTRAN, and can be the only reason for FORTRAN, thus adding a constraint.

@gonuke Not really. I removed this requirement ages ago. You can specify the mangling format directly and it will be used without needing Fortran anywhere. At least, I have this working and used with autotools workflows. Can't remember if I did the same in CMake previously.

[gonuke]

Good to hear @vijaysm - I guess my motivations are outdated.