Skip to content

Protocols on updating FATES within CTSM

Jump to bottom
Erik Kluzek edited this page Aug 21, 2024 · 39 revisions

[Introduction]:

The FATES code is an submodule to CTSM and an optional configuration. The version of FATES that CTSM points to is a version under the standard NGEET repository. FATES users and developers however, may have need to update the default version of FATES pointed to in CTSM tags and will need to update their FATES submodule pointed to in CTSM to a different tag or branch, as well as their own fork of FATES. Some of the core CTSM developers are also responsible to update the version of FATES pointed to by default CTSM as well as do the needed testing with the new version of FATES for CTSM. So this guide is divided into those two functions. Updating the FATES version for one's own use, and update FATES for CTSM tags.

NOTE REGARDING git-fleximod: The philosophy of the git-fleximod tool is to provide an easy way to checkout all of the submodules needed that go along with a CTSM tag, and to checkout specific versions that a given tag was tested with. It's expected that your real version control work will be done using git. With that philosophy git-fleximod tries to stay out of your way and not do too much. It does exactly what you ask it to do, and then stops, and that means pointing to a known frozen version. So it might not do what you want it to do, and in that case you should do what you are actually trying to do in git rather than with git-fleximod. In general when you run git-fleximod, you want to have "clean" directories in your clone, where any changes you have made have been committed. And you want the submodules that are pointed to in .gitmodules for git-fleximod to be the repository and tag or branch to be in agreement with what the FATES git clone is pointing to. You don't want git-fleximod to needlessly clobber over your work.

FATES user

What we will show here treats FATES differently from other submodules in CTSM, because in this case FATES users are very likely to mostly do all of their work within FATES. You could use git-fleximod to point to the FATES branch you are working on, but in general this workflow is easier and more intuitive to most FATES users.

Updating FATES for your own use

In general we are keeping the version of FATES that CTSM points to close to the latest on FATES main-development. But, sometimes you will need to point to a different version or to a branch on your own fork.

When a FATES user wants to point to a different version of FATES than CTSM is pointing to. The following steps are taken. The first list are the steps you do to update your CTSM clone to point to a different NGEET FATES version.

  1. ./bin/git-fleximod status # Just to check that your submodules are updated (Should show that submodules listed are correct without a "s", "M" or "e" prefix in front of them, nor does it call the submodules "dirty".) (If submodules are NOT correct run ./bin/git-fleximod update to get the correct versions of everything)
  2. cd src/fates
  3. git checkout <fates_tag>

(Optionally you could edit the .gitmodules file so that git-fleximod will know about the update. We show the above since it's simpler and useful for most FATES developers doing work on just FATES.)

Creating a branch on your fork

These next steps are what you do to create a new FATES branch from your fork, to checkin changes that you are working on. It does assume you did all of the previous steps. This includes the step of creating a new branch, if you already have a branch on your fork, you can skip that step. To create a branch to work on...

  1. cd src/fates
  2. git remote add $GITUSER git@github.com:$GITUSER/fates.git # Add a remote for your fork
  3. Create branch from origin remote (NGEET) branch or tag: i.e. git checkout -b <newbranch> origin/master
  4. Send the new branch to your fork i.e. git push -u $GITUSER <newbranch> # Note, assumes step 2 above
  5. cd ../..
  6. ./bin/git-fleximod status # Run status on git-fleximod and it will show that fates is modified (i.e. M src/fates)

Then make changes and commit to your clone as normal.

To push commits on your clone to your fork.

  1. cd src/fates
  2. git push -u $GITUSER <newbranch>

To update to a newer CTSM version when you've done the above to change your fates version

  1. cd src/fates
  2. Make sure everything you have in your fates branch is checked in i.e. git status # make sure the branch is up to date and there are no modified files or files to be committed
  3. cd ../..
  4. git fetch --all
  5. git checkout <ctsm-tag>
  6. Check if submodules were updated: i.e. ./bin/git-fleximod status You will only need to proceed if submodules outside of FATES were updated If FATES was updated you'll need to check if your branch is compatible with the new version of FATES that CTSM is using. If not you'll need to either update your branch or go back to a previous version of CTSM.
  7. ./bin/git-fleximod update
  8. cd src/fates
  9. git checkout -b <yourbranch> $GITUSER/<yourbranch> NOTE: In some cases you might have to delete your src/fates version (with \rm -rf src/fates) in order to make ./bin/git-fleximod update happy. In that case do the above steps and then repeat the steps above to point to your branch.

CTSM/FATES Developer

When new FATES tags have gone into NGEET and we want to bring those changes into CTSM, the following steps need to happen. For most of this the same process for creating CTSM tags is done, mostly below are outlined the additional steps that need to happen. Note, also that in contrast to FATES Users git-fleximod is used to point to the FATES version.

General notes about FATES testing

The fates test list is run on every FATES tag as it is merged to the main FATES branch in NGEET. This testing is done by checking out the new FATES branch in the context of the last CTSM tag where there was a FATES update, and comparing against baselines that used that same CTSM tag.

For example, if ctsm5.1.dev061 involved a FATES update to fates-sci.1.50.0_api.17.0.0, then the fates test suite would have been run with this tag, creating baselines fates-sci.1.50.0_api.17.0.0-ctsm5.1.dev061. Then, for subsequent FATES changes, each new FATES tag will be tested in the context of ctsm5.1.dev061. So the next few baselines might be fates-sci.1.51.0_api.17.0.0-ctsm5.1.dev061, fates-sci.1.52.0_api.17.0.0-ctsm5.1.dev061, fates-sci.1.52.0_api.17.0.0-ctsm5.1.dev061, etc. (noting that all of them list ctsm5.1.dev061).

FATES Updates that don't include API changes

  1. Create a branch for CTSM that includes the FATES update
  2. Update .gitmodules to point to the latest FATES tag (see README_GITFLEXIMOD.rst at the top level of CTSM for more details), then commit that change to the CTSM branch (NOTE: If the FATES tag is not made because it needs to be coordinated with the CTSM testing see the section below)
  3. Run the aux_clm testing (https://github.com/ESCOMP/CTSM/wiki/System-Testing-Guide#pre-merge-system-testing ), comparing against the previous CTSM tag
  4. Run the fates test list, comparing against the latest existing FATES baselines (these should have been created with the FATES tag you are pulling in, but in the context of an earlier CTSM tag). These should generate baselines with the FATES suite baseline naming convention (e.g., fates-sci.1.50.0_api.17.0.0-ctsm5.1.dev061)
  5. The changes are presented as a Pull Request to CTSM and approved as a normal CTSM tag (see CTSM checklist)

FATES Updates that include API changes

In this case the FATES tag isn't complete, because it has to be coordinated with the CTSM PR and tag. Most of the above steps still apply, but there are some changes or additional steps that are added.

  1. Create a branch in FATES that includes the updates
  2. Create a branch in CTSM that includes any updates within the CTSM repository
  3. Update .gitmodules to point to the FATES branch (or latest hash on that branch), then commit that change to the CTSM branch
    • Using a hash ensures that you will use the same, fixed version for all clones; to do this, change the tag = ... line to hash = ....
    • Using a branch allows you to fetch the latest version of the branch each time you run ./bin/git-fleximod update; to do this, change the tag = ... line to branch = ...
    • In either case, change the repo_url to point to your fork
    • See README_GITFLEXIMOD.rst at the top level of CTSM for more details
  4. Create a PR in FATES for the update
  5. Create a PR in CTSM for the update
  6. Get approvals for the FATES PR and get it scheduled for next in line
  7. Get the CTSM PR scheduled for next in line
  8. Merge the latest version of FATES master into your FATES branch
  9. Merge the latest version of CTSM master into your CTSM branch
  10. Create the baselines for the fates test lists for the previous CTSM tag
    • To create the baselines, check out the baseline CTSM tag (i.e., the latest CTSM tag, which your CTSM branch should now be up-to-date with) and update the FATES submodule to the baseline FATES tag (i.e., the latest FATES tag, which your FATES branch should now be up-to-date with); create the baselines from there.
    • Note that this step is optional, but recommended. You can compare against the existing FATES baselines from the last run of the FATES test suite (i.e., with the latest FATES tag in the context of (generally) some earlier CTSM tag). There are two situations where this can make sense:
      • If you expect answer changes in all FATES tests anyway (so there's no point in making new baselines since you'll have baseline failures in any case)
      • If you are confident enough that there have only been bit-for-bit changes on the CTSM side since the last time FATES baselines were generated (so new baselines, i.e., in the context of the latest CTSM tag, would be bit-for-bit with already-existing baselines).
  11. As in the above section (step 3.) run the standard testing for CTSM
  12. As in the above section (step 4.) also run the fates test list, but now compare against the baselines you just created
  13. Once the testing is complete and satisfactory, have the FATES admin create the new FATES tag
  14. Update the .gitmodules to point to the new tag, commit and push that change to your branch in the CTSM PR.
  15. Run ./bin/git-fleximod update to get the new FATES tag
  16. Make sure the version of FATES you actually ran with (in src/fates) and the new tag are identical (by doing a git diff of the new tag against the hash you used for testing)

Integrator

If integrator didn't do the testing:

  1. Review the PR, if needed
  2. Make sure .gitmodules is pointing to a NGEET FATES tag and not a personal branch/hash
  3. Double check that the fates version used for the baselines is the same as in .gitmodules for the PR
  4. Make sure the testing baselines were run and have standard names and are on: izumi, and Derecho for both aux_clm and fates tests
  5. OPTIONAL: Double check test results are as expected (looking at the their test cases, found either by them giving the directory names in the PR or by looking at the tail of CaseDocs/lnd_in for the directory name in the a test for the baseline)
  6. Review the ChangeLog (maybe do simple updates for clarity or have author update if really needed)
  7. Update the date in the ChangeLog if a day or more has passed (commit and push it to the PR branch)

Finally, complete the CTSM tag as for other CTSM tags (see CTSM checklist)

Clone this wiki locally