Add GFSv17 IAU restart tests

[junwang-noaa]

Description

GFSv17 is running with IAU turned on, the model should be able to restart with that configuration. A GFSv17 IAU restart test (cpld_restart_control_gfsv17_iau) based on cpld_control_gfsv17_iau needs to be added in RT.

Solution

Alternatives

Related to

[junwang-noaa]

The RT test cpld_control_gfsv17_iau in ufs-weather-model generates consistent restart files for all the components. The test starts ar 2021032206 with fhrot=3 and total forecast length 30hr, the model produces forecast results from 2021032212 to 2021032312 (2021-03-23-43200). Below is the list of restart files:

ufs.cpld.ww3.r.2021-03-23-43200

RESTART/
20210323.120000.MOM.res.nc  
         
20210323.120000.fv_core.res.tile6.nc     20210323.120000.phy_data.tile1.nc
20210323.120000.ca_data.tile1.nc      20210323.120000.fv_srf_wnd.res.tile1.nc  20210323.120000.phy_data.tile2.nc
20210323.120000.ca_data.tile2.nc      20210323.120000.fv_srf_wnd.res.tile2.nc  20210323.120000.phy_data.tile3.nc
20210323.120000.ca_data.tile3.nc      20210323.120000.fv_srf_wnd.res.tile3.nc  20210323.120000.phy_data.tile4.nc
20210323.120000.ca_data.tile4.nc      20210323.120000.fv_srf_wnd.res.tile4.nc  20210323.120000.phy_data.tile5.nc
20210323.120000.ca_data.tile5.nc      20210323.120000.fv_srf_wnd.res.tile5.nc  20210323.120000.phy_data.tile6.nc
20210323.120000.ca_data.tile6.nc      20210323.120000.fv_srf_wnd.res.tile6.nc  20210323.120000.sfc_data.tile1.nc
20210323.120000.fv_core.res.nc        20210323.120000.fv_tracer.res.tile1.nc   20210323.120000.sfc_data.tile2.nc
20210323.120000.fv_core.res.tile1.nc  20210323.120000.fv_tracer.res.tile2.nc   20210323.120000.sfc_data.tile3.nc
20210323.120000.fv_core.res.tile2.nc  20210323.120000.fv_tracer.res.tile3.nc   20210323.120000.sfc_data.tile4.nc
20210323.120000.fv_core.res.tile3.nc  20210323.120000.fv_tracer.res.tile4.nc   20210323.120000.sfc_data.tile5.nc
20210323.120000.fv_core.res.tile4.nc  20210323.120000.fv_tracer.res.tile5.nc   20210323.120000.sfc_data.tile6.nc
20210323.120000.fv_core.res.tile5.nc  20210323.120000.fv_tracer.res.tile6.nc

ufs.cpld.cpl.r.2021-03-23-43200.nc

iced.2021-03-23-43200.nc

A test run directory can be seen on hera at:
/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_906282/cpld_control_gfsv17_iau_intel

[junwang-noaa]

@aerorahul FYI.

[junwang-noaa]

I checked the IAU test generated from global-workflow, the test case /scratch1/NCEPDEV/climate/Jessica.Meixner/IAUcheckpointrestarts/iau01/TMP/RUNDIRS/testiau/gfsfcst.2021032418/fcst.1642156

In the run directory we have the following in restart/FV3_RESTART:

[Jun.Wang@hfe01 RESTART]$ ls 20210329.180000*
20210329.180000.ca_data.tile1.nc      20210329.180000.fv_core.res.tile6.nc     20210329.180000.phy_data.tile1.nc
20210329.180000.ca_data.tile2.nc      20210329.180000.fv_srf_wnd.res.tile1.nc  20210329.180000.phy_data.tile2.nc
20210329.180000.ca_data.tile3.nc      20210329.180000.fv_srf_wnd.res.tile2.nc  20210329.180000.phy_data.tile3.nc
20210329.180000.ca_data.tile4.nc      20210329.180000.fv_srf_wnd.res.tile3.nc  20210329.180000.phy_data.tile4.nc
20210329.180000.ca_data.tile5.nc      20210329.180000.fv_srf_wnd.res.tile4.nc  20210329.180000.phy_data.tile5.nc
20210329.180000.ca_data.tile6.nc      20210329.180000.fv_srf_wnd.res.tile5.nc  20210329.180000.phy_data.tile6.nc
20210329.180000.coupler.res           20210329.180000.fv_srf_wnd.res.tile6.nc  20210329.180000.sfc_data.tile1.nc
20210329.180000.fv_core.res.nc        20210329.180000.fv_tracer.res.tile1.nc   20210329.180000.sfc_data.tile2.nc
20210329.180000.fv_core.res.tile1.nc  20210329.180000.fv_tracer.res.tile2.nc   20210329.180000.sfc_data.tile3.nc
20210329.180000.fv_core.res.tile2.nc  20210329.180000.fv_tracer.res.tile3.nc   20210329.180000.sfc_data.tile4.nc
20210329.180000.fv_core.res.tile3.nc  20210329.180000.fv_tracer.res.tile4.nc   20210329.180000.sfc_data.tile5.nc
20210329.180000.fv_core.res.tile4.nc  20210329.180000.fv_tracer.res.tile5.nc   20210329.180000.sfc_data.tile6.nc
20210329.180000.fv_core.res.tile5.nc  20210329.180000.fv_tracer.res.tile6.nc

in restart/MOM6_RESTART:

20210329.150000.MOM.res.nc

in restart/CICE_RESTART

cice_model.res.2021-03-29-54000.nc

in restart/CMEPS_RESTART

 ufs.cpld.cpl.r.2021-03-29-54000.nc

To make consistent restart files, we need to change the following line from:
in ufs.configure:

restart_n = 12

in model_configure:

restart_interval:        12 24 36 48 60 72 84 96 108 120

to:
in ufs.configure:

restart_n = 15

in model_configure:

restart_interval:        12 27 42 57 72 87 102 117

At this time, the atmosphere history files may not reproduce when the model restarts at fh=27, 57,87 and 117. New updates are required to let marine/CMEPS components to have flexible restart file output time, that will completely resolve the issue.

[junwang-noaa]

@aerorahul FYI.

[aerorahul]

@junwang-noaa Thanks.
I think this is straightforward and do-able.
I might have questions about what those should be when doing a restart from failure or a second segment. I can ask them elsewhere.

[junwang-noaa]

A GFSv17 IAU restart test cpld_restart_gfsv17_iau is added. Unfortunately the test does not reproduce the results from cpld_control_gfsv17_iau control test. Further debugging shows that the test reproduces in atmonly only mode and in s2s (atm-ocn-ice), but in coupled gfsv17 mode (atm-ocn-ice-wav). In the coupled gfsv17 mode, the coupling fields (e.g. wavImp_Sw_pstokes, wavImp_Sw_z0) that WW3 sends to ocn and atm do not reproduce in control run and restart run.

[Jun.Wang@hfe12 cpld_control_gfsv17_iau_intel_rst]$ nccmp -d ufs.cpld.cpl.hi.atm.2021-03-22-68400.nc ../cpld_control_gfsv17_iau_intel/ufs.cpld.cpl.hi.atm.2021-03-22-68400.nc
[Jun.Wang@hfe12 cpld_control_gfsv17_iau_intel_rst]$ nccmp -d ufs.cpld.cpl.hi.ice.2021-03-22-68400.nc ../cpld_control_gfsv17_iau_intel/ufs.cpld.cpl.hi.ice.2021-03-22-68400.nc
[Jun.Wang@hfe12 cpld_control_gfsv17_iau_intel_rst]$ nccmp -d ufs.cpld.cpl.hi.ocn.2021-03-22-68400.nc ../cpld_control_gfsv17_iau_intel/ufs.cpld.cpl.hi.ocn.2021-03-22-68400.nc
[Jun.Wang@hfe12 cpld_control_gfsv17_iau_intel_rst]$ nccmp -d ufs.cpld.cpl.hi.wav.2021-03-22-68400.nc ../cpld_control_gfsv17_iau_intel/ufs.cpld.cpl.hi.wav.2021-03-22-68400.nc
DIFFER : VARIABLE : wavImp_Sw_pstokes_x1 : POSITION : [0,0,0] : VALUES : -1.37339e-16 <> -1.37338e-16

for field wavImp_Sw_z0 that ww3 is sent to atm, it shows:

108c108
<     1.97232438949868e-05, 1.04722093965393e-05, 9.44941621128237e-06,
---
>     1.97232475329656e-05, 1.04722093965393e-05, 9.44941621128237e-06,
110,118c110,118
<     2.25347548621357e-06, 1.74139472619572e-06, 1.11799658952805e-06,
<     4.90223101223819e-05, 2.61735185631551e-05, 5.37803043698659e-06,
---
>     2.25347548621357e-06, 1.74139449882205e-06, 1.11799658952805e-06,
>     4.90223101223819e-05, 2.61735185631551e-05, 5.37802816324984e-06,
...

[junwang-noaa]

The two run directory is at:
/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_3917402/cpld_control_gfsv17_iau_intel
/scratch1/NCEPDEV/stmp2/Jun.Wang/FV3_RT/rt_3917402/cpld_control_gfsv17_iau_intel_rst

The coupling fields in ufs.cpld.cpl.hi.wav.2021-03-22-68400.nc show that the import fields going to ww3 are identical in the control and restart run, but the coupled fields coming out of ww3 are different.

@MatthewMasarik-NOAA would you please take a look? Thanks

[MatthewMasarik-NOAA]

Hi @junwang-noaa, thank you for letting us know about this issue. We've added this to our task list and are in the process of prioritizing tasking for our group given our staffing for the next few months. I'll post back again shortly.

[MatthewMasarik-NOAA]

Hi @junwang-noaa, I spoke with @sbanihash last week, who is now on leave through August. She advised to wait until she returned from leave for one of us to look into it.

[junwang-noaa]

@MatthewMasarik-NOAA Thanks for letting me know.

[sbanihash]

@junwang-noaa just wanted to let you know that we still haven't had time to dig into this but it is on our to-do list. Thank you for your patience.

[sbanihash]

@junwang-noaa Wanted to let you know that I haven't had time to dig into this yet. I know you were also asking for any updates in the Monday meeting, so wanted to let you know that it is on my list, but haven't had time to get to it yet. Thanks for your patience.

[JessicaMeixner-NOAA]

Trying to assess the issues here. I've tried using #2404 but am having trouble running the tests after trying to update. Is there something else I should be using for addressing this? i can use the workflow or something else if needed.