more aerosol metrics in e3sm_diags

[mahf708]

Description

Add preliminary code for aerosol-related diagnostics from the v3 atmosphere subgroup "Aerosol Working Group" led by Susannah Burrows. This work is partially from a collaborative effort by Naser Mahfouz, Johannes Mülmenstädt, and Susannah Burrows in 2023, and we picked a subset of diagnostics we think will be helpful to a wider audience when evaluating aerosols. This PR adds aerosol masses at different levels, aerosol integrated burdens, aerosol--cloud interaction metrics (e.g., cdnc, lwp), as well as aerosol effective radiative forcing. The PR also exposes all 2D "AeroCom" variables to the derived parameter list (thus enabling easier independent usage by users and/or extending e3sm_diags in future releases).

Caveats:

• The specs in cfg files are not final and will be edited based on feedback from users in upcoming releases.
• Note that the code added may result in non-fatal (harmless) runtime errors (which will appear in e3sm_diags logs) due to potential unavailability of all specified parameters in the h0 tape.

Closes #760

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• My changes generate no new warnings (⚠️ the added code will generate non-fatal runtime errors due to potentially unavailability of some diagnostics in the h0 tapes ⚠️)
• Any dependent changes have been merged and published in downstream modules

If applicable:

• New and existing unit tests pass with my changes (locally and CI/CD build)
• I have added tests that prove my fix is effective or that my feature works
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have noted that this is a breaking change for a major release (fix or feature that would cause existing functionality to not work as expected)

[mahf708]

Hi @chengzhuzhang, I added basic support for more aerosol variables through the derived variables in acme.py. I will try to refine some aspects of the code (e.g., the cdnc/lwp functions could be improved). Please have a look and let me know if you'd like any additions or fixes. I largely collected this code from previous efforts that occurred over the summer, so I may have done messed something up. The tests pass and figures produced can be found here: https://web.lcrc.anl.gov/public/e3sm/diagnostic_output/ac.ngmahfouz/E3SMv3_dev/test-aero-metrics-in-sets/

Some notes:

1. Not everything added is used immediately; I don't have much time to test figures for all variables now, but hopefully this preliminary support is helpful; I am hoping users will give us feedback about figures and we can fix them as we move forward
2. In the future, if we have more funding for this type of effort, we can bring to life a new set (call it awg metrics or something); for now, we are just plugging things into the easiest and most straightforward interface
3. See the (mostly Susannah's) solution for correcting mass/burden units (without the unit-change hack, numbers won't even show up on these figures); hope it is acceptable, but in the future, we likely can do better than that

For reviews, I cannot request review from anyone, so please invite Johannes (@jmuelmen) and Susannah (@susburrows) to review this PR. They contributed to this work throughout the year (and I added them as co-authors in two important commits above where their contributions were especially important).

[chengzhuzhang]

I can't seem to find codes to support 3D masses..

[mahf708]

Addressed in 095af28

[chengzhuzhang]

hey @mahf708, I should have be clearer, for 3D mass, I was going to suggest to have the function to [create column integrated](#

e3sm_diags/e3sm_diags/driver/aerosol_budget_driver.py

Line 29 in 633b52c

def calc_column_integral(data, aerosol, season):

) value from 3D aerosol mass concentration in derived variable list under the entries of ABURDENXX, the entry in derived_variable would be something like.

"ABURDENBC": OrderedDict(
        [
            (("ABURDENBC",), aero_burden_fxn),
            # If ABURDENBC is not exist, detect if Mass_bc is available, and do column integral.
            (("Mass_bc","hyai", "hybi", "PS"), calc_column_integral),              
        ]
    ), 

This could be helpful when re-run e3sm_diags with simulations prior to v3.

[mahf708]

Okay, I will add this. Let me think about implementing it. If not before the 2.10.0 release, then I will get in for the 2.10.1 or 2.11.0 release.

[chengzhuzhang]

Sounds good. Other changes in this PR looks good to me!

[chengzhuzhang]

Thank you @mahf708 for the PR. It looks great. This provides important update for supporting aerosol variables adjusted for upcoming v3 simulation. Also it is intriguing to see the use case with the model vs model run for ERF analysis, nice work! I only have a comment regarding supporting 3d mass for backward compatibility. I saw comments about it, but I think it hasn't been implemented?

I just tagged Susannah and Johannes for review. Appreciate this nice contribution from the aerosol working group!

[jmuelmen]

These diagnostics will be extremely useful for ACI analysis. Great to have them in e3sm_diags!

[susburrows]

Thanks, @mahf708, for implementing the changes in this PR and following through on the steps required to submit them to the e3sm_diags repository. These diagnostics are extremely valuable for understanding and tracking how the aerosol forcing and understanding the 3D distribution of aerosols change across model versions, while keeping the I/O burden manageable (e.g., by using 2D fields such as cross-sections of mass concentrations at several pressure levels).

Thanks for providing the opportunity for these contributions from the AWG to be included into e3sm_diags, @chengzhuzhang .

I think this looks good and am approving, based on examination of the test output. If further changes are required in the future, I trust that those can be addressed in separate PRs.

[tomvothecoder]

I just had some minor docstring updates. Other than that it looks good to me.

---

On a side-note, I started refactoring acme.py in another branch. I split it into derivations.py, formulas.py, and utils.py. These modules operate on Xarray objects instead of cdms2 objects. Also the derived_variables dictionary will eventually be cleaned up because it is massive and hard to follow.

[mahf708]

On a side-note, I started refactoring acme.py in another branch. I split it into derivations.py, formulas.py, and utils.py. These modules operate on Xarray objects instead of cdms2 objects. Also the derived_variables dictionary will eventually be cleaned up because it is massive and hard to follow.

This is great! I intentionally put my additions at the end because I knew this was going to be refactored. I also think we may end up revisiting my haphazard implementation soon. Once, you're ready with your branch, I can push my additions to it, so that you don't have to bother rewriting this code ;)

Also, great to have this done via xarray (I know xarray, but I know nothing about cmds2). I actually was wondering how I could do a better cutoff for these quantities divided by lcc/icc (obviously these can be zero, so we need a cutoff; turns out it worked well without a cutoff for cdnc, but not so well for lwp). Instead of dwelling on it, I thought I would revisit it later :)

[tomvothecoder]

This is great! I intentionally put my additions at the end because I knew this was going to be refactored. I also think we may end up revisiting my haphazard implementation soon. Once, you're ready with your branch, I can push my additions to it, so that you don't have to bother rewriting this code ;)

Also, great to have this done via xarray (I know xarray, but I know nothing about cmds2).

Awesome! Whenever you're ready, you can open a PR that includes the Xarray version of the code. You just need to cdat-migration-fy24 as the base branch.

This PR looks good to me.

[mahf708]

Okay, @tomvothecoder feel free to merge and initiate a release candidate for @xylar to start testing the emergency e3sm-unified release. I just rebased this PR.