HPCpy integration

[jo-basevi]

HPCpy is a Python package for interacting with HPC scheduling systems. Using this in payu, may reduce the need for scheduler specific code. Currently, there is some development on a SLURM client: ACCESS-NRI/hpcpy#28. See related payu issue for supporting different schedulers - #182

Payu currently runs qsub with executable commands directly rather than a script, (e.g., qsub <directives> -- /g/data/vk83/prerelease/apps/payu/dev/bin/python /g/data/vk83/prerelease/apps/payu/dev/bin/payu-run). See related issue in HPCpy to support this: ACCESS-NRI/hpcpy#24.

However, HPCpy supports script jinja templates which can be used to generate a script. So it could be possible to generate a job script that contains {{python}} {{payu-cmd}}. This could potentially help:

• module initialisation issues which requires a shell to be created (see User-scripts can't run module use/load #522). This will allow modules to be used in bash user-scripts.
• maybe support different compiler/mpi combinations with different script templates (see Error using intel-mpi #336 (comment))

There is also some support for dependency strategy in HPCpy, this could be useful for running sync jobs - after a user-defined postscript (see some related discussion in this issue: #463 (comment))