Check if bootstrap from ESMF attribute rather than regional max value

[lizziel]

Your name

Lizzie Lundgren

Your affiliation

Harvard University

What happened? What did you expect to happen?

GMAO recently encountered a GEOS issue where layout reproducibility tests fail. Atanas Trayanov (GMAO) traced the issue to a test in HEMCO for whether internal state fields are bootstrapped. Bootstrapping is when MAPL assigns a constant value to an internal state field since it missing from the restart file. Rather than check MAPL for whether the field was bootstrapped HEMCO checks if max value is less than a threshold. GMAO is using an older version of HEMCO but similar behavior persists in the currently released version:

HEMCO/src/Core/hco_restart_mod.F90

Lines 487 to 505 in 0ae25d2

	! ------------------------------------------------------------------
	! Try to get from ESMF internal state
	! ------------------------------------------------------------------
	#if defined(ESMF_)
	CALL HCO_CopyFromIntnal_ESMF( HcoState, TRIM(Name), &
	1, FLD, RC, Arr2D=Arr2D )
	IF ( RC /= HCO_SUCCESS ) THEN
	CALL HCO_ERROR( 'ERROR 4', RC, THISLOC=LOC )
	RETURN
	ENDIF
	! Determine if field has been filled
	IF ( FLD ) THEN
	#if defined(MODEL_GEOS)
	IF ( MAXVAL(Arr2D) < 0.0 ) FLD = .FALSE.
	#else
	IF ( MAXVAL(Arr2D) <= 0.0 ) FLD = .FALSE.
	#endif
	ENDIF

This handling is not ideal and may introduce silent bugs. The restart fields could be any values depending on what they physically represent. It is totally legitimate that a field could potentially be all zero, or all negative, in a sub-region of the globe. If a field is bootstrapped then the test for whether it is bootstrapped should return all true or all false across all cores, but using this method does not ensure that. If different cores give different answers on whether the field was bootstrapped, as is the case for GEOS in the layout test, then the handling of the field is wrong.

A better way is to get the bootstrap info from MAPL. We do this in the GEOS-Chem code repository for GCHP such as here, checking the restart attribute. We should do the same thing in HEMCO.

What are the steps to reproduce the bug?

Layout reproducibility tests pass in GCHP but fail in GEOS.

Please attach any relevant configuration and log files.

No response

What HEMCO version were you using?

Pre-3.10

What environment were you running HEMCO on?

Local cluster

What compiler and version were you using?

Not sure

Will you be addressing this bug yourself?

Yes

In what configuration were you running HEMCO?

Other (please explain in additional information section below)

As what resolution were you running HEMCO?

Not sure

What meterology fields did you use?

Other (please explain in additional information section below)

Additional information

No response