Skip to content

Commit

Permalink
hopefully clarify snapshot documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@peverwhee
peverwhee committed Dec 9, 2024
1 parent 7acca22 commit 3d5ef6f
Showing 1 changed file with 109 additions and 53 deletions.
162 changes: 109 additions & 53 deletions docs/conversion/create-snapshots.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# 6 - Create snapshots of CAM

## CAM snapshot run tips

1. Try to use the lowest resolution grid possible in order to keep the snapshot file size small (e.g. `ne3pg3_ne3pg3_mg37`).
1. Run with debug flags on (`./xmlchange DEBUG=True`), as this way errors that can occur when you bring the CCPP-ized code back into CAM are easier to catch. Make sure ALL CAM and CAM-SIMA runs have debug on.
1. You should generate at least one set of snapshot files using GNU on derecho with DEBUG=TRUE, as these will be saved for use in the CAM-SIMA physics testbed to prevent unexpected answer changes.

## Configure and set-up CAM snapshot
Make a normal **CAM** (not CAM-SIMA) run using cam_snapshot to capture before and after files using a compset which exercises the parameterization being converted.

Expand All @@ -9,53 +15,8 @@ Make a normal **CAM** (not CAM-SIMA) run using cam_snapshot to capture before an
- To find the snapshot name, look in your source code checkout in `bld/namelist_files/namelist_definition.xml`.
- Inside this file, find the “cam_take_snapshot_before” definition. In this definition, it lists all possible parameterizations which can have a snapshot file created around.
- If your parameterization is there, that will be your `cam_take_snapshot_before` and `_after` values in your `user_nl_cam`
- If it's not there, you'll add `user_set` snapshot calls to `physpkg.F90` in the following way:

For `tphysbc` in `physics/cam7/physpkg.F90`:
```
if (trim(cam_take_snapshot_before) == "user_set") then
call cam_snapshot_all_outfld_tphysbc(cam_snapshot_before_num, state, tend, cam_in, cam_out, pbuf, &
flx_heat, cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, det_s, det_ice, net_flx)
end if
<call to parameterization run phase>
if ( (trim(cam_take_snapshot_after) == "user_set") .and. &
(trim(cam_take_snapshot_before) == trim(cam_take_snapshot_after))) then
call cam_snapshot_ptend_outfld(ptend, lchnk)
end if
<call to physics_update and/or ptend outfld calls>
if (trim(cam_take_snapshot_after) == "user_set") then
call cam_snapshot_all_outfld_tphysbc(cam_snapshot_after_num, state, tend, cam_in, cam_out, pbuf, &
flx_heat, cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, det_s, det_ice, net_flx)
end if
```

For `tphysac`:
```
if (trim(cam_take_snapshot_before) == "user_set") then
call cam_snapshot_all_outfld_tphysac(cam_snapshot_before_num, state, tend, cam_in, cam_out, pbuf, &
flx_heat, cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, det_s, det_ice, net_flx)
end if
<call to parameterization run phase>
- If it's not there, you'll add `user_set` snapshot calls to `physpkg.F90` per the instructions [here](#user-set-snapshot-calls)

if ( (trim(cam_take_snapshot_after) == "user_set") .and. &
(trim(cam_take_snapshot_before) == trim(cam_take_snapshot_after))) then
call cam_snapshot_ptend_outfld(ptend, lchnk)
end if
<call to physics_update and/or ptend outfld calls>
if (trim(cam_take_snapshot_after) == "user_set") then
call cam_snapshot_all_outfld_tphysac(cam_snapshot_after_num, state, tend, cam_in, cam_out, pbuf, &
flx_heat, cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, det_s, det_ice, net_flx)
end if
```
!!! Note
The above calls may vary if your physpkg.F90 exists in a different directory within `$CAM/src/physics` (not `cam7`)


- To enable cam_snapshot, add the following to your user_nl_cam file:
Expand All @@ -78,7 +39,7 @@ ndens = 2,2,2,2,2,1,1
- The run must be at least 2 time steps in order to work in subsequent steps. We recommend 3 to at most 9 time steps. (Note a 3 timestep run using a 2 degree FV grid can produce snapshot files with sizes over 4 Gb each).
- To change the number of timesteps to 3 use `./xmlchange STOP_OPTION=nsteps` and `./xmlchange STOP_N=3`

!!! Warning
!!! Note "Coupling frequency"
With short NUOPC runs, you may need to change ROF_NCPL and/or GLC_NCPL to the same size at ATM_NCPL to get them to run properly (`./xmlchange ROF_NCPL=\$ATM_NCPL` and `./xmlchange GLC_NCPL=\$ATM_NCPL`)

## Create CAM snapshots
Expand Down Expand Up @@ -113,11 +74,106 @@ On **izumi**, save your converted snapshot files in:
/project/amp02/cam_snapshot_files/<parameterization>
```

## CAM snapshot run tips
Once you have created your snapshot files, proceed to [7 - Run CAM-SIMA](run-cam-sima.md)

1. Try to use the lowest resolution grid possible in order to keep the snapshot file size small (e.g. `ne3pg3_ne3pg3_mg37`).
1. We generally recommend running with debug flags on (`./xmlchange DEBUG=True`), as this way errors that can occur when you bring the CCPP-ized code back into CAM are easier to catch.
- However, if you decide not to, then do make sure that all of your other CAM runs also have debug flags off, as otherwise you can get differences in the results that are solely due to compiler settings, and not your code changes.
1. You should generate at least one set of snapshot files using GNU on derecho with DEBUG=TRUE, as these will be saved for use in the CAM-SIMA physics testbed to prevent unexpected answer changes.
## "user set" snapshot calls
Depending on where your scheme is called within CAM physics (in CAM6 or CAM7 physpkg and in tphysac or tphysbc), choose a code snippet below.

Once you have created your snapshot files, proceed to [7 - Run CAM-SIMA](run-cam-sima.md)
!!! Warning "Snapshot call changes"
If the snapshot calls change in CAM, this section may not be updated immediately. If you're running into issues, mirror an existing set of snapshot calls in the physpkg.F90 routine you are working in.

### CAM7 tphysbc

In `$CAM/src/physics/cam7/physpkg.F90`:

```
if (trim(cam_take_snapshot_before) == "user_set") then
call cam_snapshot_all_outfld_tphysbc(cam_snapshot_before_num, state, tend, cam_in, cam_out, pbuf, &
cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, net_flx)
end if
<call to parameterization run phase>
if ( (trim(cam_take_snapshot_after) == "user_set") .and. &
(trim(cam_take_snapshot_before) == trim(cam_take_snapshot_after))) then
call cam_snapshot_ptend_outfld(ptend, lchnk)
end if
<call to physics_update and/or ptend outfld calls>
if (trim(cam_take_snapshot_after) == "user_set") then
call cam_snapshot_all_outfld_tphysbc(cam_snapshot_after_num, state, tend, cam_in, cam_out, pbuf, &
cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, net_flx)
end if
```

### CAM7 tphysac
In `$CAM/src/physics/cam7/physpkg.F90`

```
if (trim(cam_take_snapshot_before) == "user_set") then
call cam_snapshot_all_outfld_tphysac(cam_snapshot_before_num, state, tend, cam_in, cam_out, pbuf,&
fh2o, surfric, obklen, flx_heat, cmfmc, dlf, det_s, det_ice, net_flx)
end if
<call to parameterization run phase>
if ( (trim(cam_take_snapshot_after) == "user_set") .and. &
(trim(cam_take_snapshot_before) == trim(cam_take_snapshot_after))) then
call cam_snapshot_ptend_outfld(ptend, lchnk)
end if
<call to physics_update and/or ptend outfld calls>
if (trim(cam_take_snapshot_after) == "user_set") then
call cam_snapshot_all_outfld_tphysac(cam_snapshot_after_num, state, tend, cam_in, cam_out, pbuf,&
fh2o, surfric, obklen, flx_heat, cmfmc, dlf, det_s, det_ice, net_flx)
end if
```

### CAM6 tphysbc
In `$CAM/src/physics/cam/physpkg.F90`:
```
if (trim(cam_take_snapshot_before) == "user_set") then
call cam_snapshot_all_outfld_tphysbc(cam_snapshot_before_num, state, tend, cam_in, cam_out, pbuf, &
flx_heat, cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, det_s, det_ice, net_flx)
end if
<call to parameterization run phase>
if ( (trim(cam_take_snapshot_after) == "user_set") .and. &
(trim(cam_take_snapshot_before) == trim(cam_take_snapshot_after))) then
call cam_snapshot_ptend_outfld(ptend, lchnk)
end if
<call to physics update and/or ptend outfld calls>
if (trim(cam_take_snapshot_after) == "user_set") then
call cam_snapshot_all_outfld_tphysbc(cam_snapshot_after_num, state, tend, cam_in, cam_out, pbuf, &
flx_heat, cmfmc, cmfcme, zdu, rliq, rice, dlf, dlf2, rliq2, det_s, det_ice, net_flx)
end if
```

### CAM6 tphysac
In `$CAM/src/physics/cam/physpkg.F90`:

```
if (trim(cam_take_snapshot_before) == "user_set") then
call cam_snapshot_all_outfld_tphysac(cam_snapshot_before_num, state, tend, cam_in, cam_out, pbuf,&
fh2o, surfric, obklen, flx_heat)
end if
<call to parameterization run phase>
if ( (trim(cam_take_snapshot_after) == "user_set") .and. &
(trim(cam_take_snapshot_before) == trim(cam_take_snapshot_after))) then
call cam_snapshot_ptend_outfld(ptend, lchnk)
end if
<call to physics update and/or ptend outfld calls>
if (trim(cam_take_snapshot_after) == "user_set") then
call cam_snapshot_all_outfld_tphysac(cam_snapshot_after_num, state, tend, cam_in, cam_out, pbuf,&
fh2o, surfric, obklen, flx_heat)
end if
```

0 comments on commit 3d5ef6f

Please sign in to comment.