Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Utilize jobid variable for non-DA tasks #512

Merged
merged 2 commits into from
Oct 7, 2024

Conversation

BenjaminBlake-NOAA
Copy link
Contributor

@BenjaminBlake-NOAA BenjaminBlake-NOAA commented Oct 4, 2024

DESCRIPTION OF CHANGES:

  • In the NCO implementation standards, $jobid needs to be used when setting the run directory names. $jobid is defined at the job card level as $job.$PBS_JOBID. $job is the <jobname> tag in Rocoto (and whatever the equivalent is in ecflow), and $PBS_JOBID is a sequence of letters/numbers that is specific to each individual task. In this PR, we introduce a temporary variable called $taskid which is defined at the job card level in the Rocoto xml, and is passed into the J-jobs where $jobid is set and used to define the run directory ($DATA=$DATAROOT/$jobid).
  • Note: At the moment we shouldn't switch entirely to using $jobid = $job.$PBS_JOBID because then other jobs in the workflow cannot access the run directories for other tasks. Once workflow changes are made to utilize the umbrella data directory structure described in issue directory structures: com, umbrella, $DATA #498 then we can shift to using $jobid = $job.$PBS_JOBID everywhere, and the temporary $taskid variable will be removed.
  • If these changes look good for the non-DA tasks I will submit a follow-up PR for the rest of the tasks in the workflow.

TESTS CONDUCTED:

Completed successful non-DA engineering and fire weather tests.

Machines/Platforms:

  • WCOSS2
    • Cactus/Dogwood
    • Acorn
  • RDHPCS
    • Hera
    • Jet
    • Orion
    • Hercules

Test cases:

  • Engineering tests
    • Non-DA engineering test
    • DA engineering test
      • Retro
      • Ensemble
      • Parallel
  • RRFS fire weather
  • RRFS_A:
  • RRFS_B:
  • RTMA:
  • Others:

ISSUE:

Copy link
Contributor

@MatthewPyle-NOAA MatthewPyle-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made the same comment on a couple of files, then realized it applied to more (but didn't keep adding the same comment). Just making sure that some of the conditional definitions of jobid being removed either isn't needed, or is covered in other ways.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to double check, the taskid coming in will cover all of the flavors of runs (spinup, ensemble) being defined in the jobid defining block being removed?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The taskid (and eventually $job) variable that is defined at the job card level will be able to handle all different scenarios (spinup, ensemble, etc.). I started doing this in FV3LAM_wflow.xml for the DA tasks which will run for the different configurations.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as for JRRFS_FORECAST - assuming the taskid definition will handle the DO_ENSEMBLE true/false cases okay?

Copy link
Contributor

@MatthewPyle-NOAA MatthewPyle-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving after getting confirmation that the removed script logic won't cause any harm.

@MatthewPyle-NOAA MatthewPyle-NOAA merged commit e247cb8 into NOAA-EMC:main Oct 7, 2024
2 checks passed
@BenjaminBlake-NOAA BenjaminBlake-NOAA deleted the feature/jobid branch October 8, 2024 12:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants