Merge pull request #1366 from cmclaug2/fix/public-testcase-walkthroug…

…h-update

Fix/public testcase walkthrough update

docs/public_testcase_walkthrough/public_testcase_walkthrough.adoc

@@ -1,6 +1,6 @@
 = LIS Public Testcase Walkthrough
-:revnumber: 1.0
-:revdate: September 23, 2022
+:revnumber: 1.1
+:revdate: June 26, 2023
 :revremark: Updated for LISF Public 7.4 release
 :imagesdir: ./images
 :title-page:
@@ -12,15 +12,16 @@
 :figure-caption!:                       // disable figure captions
 
 // github commit known to be compatible with this walkthrough
-:compatible-lisf-branch: support/lisf-public-7.3
+:compatible-lisf-branch: support/lisf-public-7.4
 
 // urls
 :url-lis-website: https://lis.gsfc.nasa.gov
 :url-lis-testcases-page: {url-lis-website}/tests/lis
 :url-lis-testcases-data: https://portal.nccs.nasa.gov/lisdata_pub/Tutorials/Web_Version/
 :url-lisf-docs: https://github.com/NASA-LIS/LISF/tree/master/docs
 :url-lisf-github: https://github.com/NASA-LIS/LISF
-:ssh-lisf-github: git@github.com:NASA-LIS/LISF.git
+:url-nccs-discover-jobs-page: https://www.nccs.nasa.gov/nccs-users/instructional/using-slurm/submit-jobs
+:ssh-lisf-github: github:NASA-LIS/LISF.git
 
 // begin document
 

docs/public_testcase_walkthrough/running_ldt.adoc

@@ -8,7 +8,9 @@ You're now ready to run LDT:
 % ./LDT {ldt-config-filename}
 ----
 
-The run should take a few minutes to complete. If the run aborts, troubleshoot the issue by reviewing any errors printed to the terminal and by viewing the contents of `ldtlog.0000`. If no errors print to the terminal, verify that the run completed successfully by checking for the following confirmation at the end of the `ldtlog.0000`:
+The run should take a few minutes to complete. If the run aborts, troubleshoot the issue by reviewing any errors printed to the terminal and by viewing the contents of `ldtlog.0000`. If no errors print to the terminal, verify that the run completed successfully by checking for the following confirmation at the end of the `ldtlog.0000`. 
+
+NOTE: If the terminal reports the error "./LDT: symbol lookup error:", this may be due extraneous modules loaded into your environement that do not allow lis to run properly. View your modules using the command `module list`.  If extraneous modules are loaded, run `module purge` to clear your environment and then `module load [lisf module file]` to load an environment suitable to running LISF.  See the "discover_quick_start" document in the {url-lisf-docs}[LISF Users' Guides,window=_blank] for more information.  
 
 [source,shell]
 ----
@@ -17,4 +19,4 @@ The run should take a few minutes to complete. If the run aborts, troubleshoot t
  --------------------------------
   Finished LDT run
  --------------------------------
-----
+----

docs/public_testcase_walkthrough/setup.adoc

@@ -37,12 +37,14 @@ IMPORTANT: The `{compatible-lisf-branch}` branch is known to be compatible with
 
 For information about configuration settings and detailed compilation instructions, see the {url-lisf-docs}[LISF Users' Guides,window=_blank]. In general, the compilation process for each component is as follows:
 
+NOTE: As you run these testcases, you will have the opportunity to compare your output to TARGET_OUTPUT files or "solutions".  The TARGET_OUTPUT files were created by running ldt, lis, and lvt compiled with *default* compile configuration settings.  For this walkthrough, it is recommended that you also use default compiler configuration settings.   
+
 [#compile-example]
 [source,shell,subs="attributes"]
 ----
 % cd ldt
 % ./configure
-> # Select compile configuration settings (see Docs)...
+> # Select compile configuration settings (Default settings recommended.)
 % ./compile
 > # Compilation output...
 % mv LDT ../../            # move executable up into $WORKING_DIR
@@ -52,3 +54,5 @@ For information about configuration settings and detailed compilation instructio
 Repeat the above process in the `lis/` and `lvt/` directories to generate the _LIS_ and _LVT_ executables.
 
 Change directories back into your `$WORKING_DIR` which should now contain three executable files: `LDT`, `LIS`, `LVT`.
+
+If you ever wish to generate new executable files, `cd` into `/LISF/l??/make` and run `gmake realclean`.  This will clear all dependency files and allow you to cleanly define new compile configuration settings and recompile. 

docs/public_testcase_walkthrough/step_1.adoc

@@ -1,9 +1,10 @@
 = Step 1: LSM Parameter Processing Run (LDT)
 
 :step-label: Step 1
-:download-filename: testcase1_ldt_parms_2020.tar.gz
+:download-filename: testcase1_ldt_parms_2023.tar.gz
+:download-batch-scripts-filename: testcase_batch_scripts.tar.gz
 :input-filesize-compressed: 350MB
-:input-filesize-unpacked: 4.1GB
+:input-filesize-unpacked: 4.2GB
 :ldt-run-dir: 
 :ldt-config-filename: ldt.config.noah36_params
 
@@ -81,7 +82,7 @@ Landcover spatial transform:       tile
 
 [NOTE]
 --
-LISF configuration files use relative paths to describe file locations. Keep this in mind if you encounter any file not found errors.
+LISF configuration files use relative paths to describe file locations. Keep this in mind if you encounter any "file not found" errors.
 --
 
 === Noah Land Surface Model Parameters
@@ -114,6 +115,21 @@ image::step_1_domain_radar.jpeg[align=center]
 
 include::running_ldt.adoc[]
 
+[NOTE]
+--
+ldt, lis, and lvt can also be run by submitting batch jobs using SLURM (**S**imple **L**inux **U**tility for **R**esource **M**anagement).  If you are running this tutorial on Discover, you can download example batch scripts for all steps from the {url-lis-testcases-page}[LIS Testcase website] using the following command:
+
+[source,shell,subs="attributes"]
+----
+% curl -O {url-lis-testcases-data}{download-batch-scripts-filename}
+% tar -xzf {download-batch-scripts-filename} -C .
+----
+
+The batch scripts (`step*.job`) will unpack into the appropriate directories. To submit a job to discover using a batch script, use the command `sbatch [BATCH SCRIPT]` from the same directory as the job you wish to run.  The command `squeue -u [YOUR USER ID]` will provide updates on your job. If the job crashes, troubleshoot using any `slurm.out` files. 
+
+See the {url-nccs-discover-jobs-page}[NCCS Discover Jobs Users' Guide] for more information.
+--
+
 == LDT Output Files
 
 The following files are typically generated by an LDT parameter processing run:
@@ -122,15 +138,15 @@ The following files are typically generated by an LDT parameter processing run:
 [%autowidth]
 |===
 |`lis_input.nldas.noah36.d01.nc` |The NetCDF-formatted model parameter file which LIS reads at runtime. `d01` refers to the indexed domain, typically used as "nests" for NASA-Unified WRF simulations.
-|`ldtlog.0000` |The runtime configuration file read by LDT in this stepThe output diagnostic file that provides runtime messages, including warnings and error messages. This file is useful for verifying successful run completion and troubleshooting unsuccessful runs.
-|`MaskParamFill.log` |The "target" NetCDF file generated by LDT in this step. This diagnostic file informs of any disagreements between an LSM-based parameter and the landmask, and whether any parameter gridcells were "filled" to agree with the landmask.
+|`ldtlog.0000` |The output diagnostic file that provides runtime messages, including warnings and error messages. This file is useful for verifying successful run completion and troubleshooting unsuccessful runs.
+|`MaskParamFill.log` |This diagnostic file informs of any disagreements between an LSM-based parameter and the landmask, and whether any parameter gridcells were "filled" to agree with the landmask.
 |===
 
 NOTE: The output filenames for `lis_input.nldas.noah36.d01.nc` and `ldtlog.0000` can be defined in the `ldt.config` file, but the filename of `MaskParamFill.log` cannot be modified.
 
 === The LIS Parameter File
 
-The contents of the LIS parameter input file produced by LDT can be examined visually using a command-line program such as http://meteora.ucsd.edu/~pierce/ncview_home_page.html[`ncview`,window=_blank] or a desktop application like NASA's https://www.giss.nasa.gov/tools/panoply/[Panoply,window=_blank]. For this walkthrough we will be using `ncview`.
+The contents of the LIS parameter input file produced by LDT can be examined visually using a command-line program such as http://meteora.ucsd.edu/~pierce/ncview_home_page.html[`ncview`,window=_blank] or a desktop application like NASA's https://www.giss.nasa.gov/tools/panoply/[Panoply,window=_blank]. For this walkthrough we will be using `ncview`. If needed, add `ncview` to your environment by running, `module load ncview`.
 
 Open the LIS input file using `ncview`:
 
@@ -249,7 +265,7 @@ Where the flags `d`, `f`, and `s` enable the following options:
 * `f`: force comparison to continue after differences found
 * `s`: report identical files to terminal (by default `nccmp` runs silently if files are identical)
 
-NOTE: The `nccmp` command will be used in future steps. Going forward, some files may not be identical, but any differences reported _should_ be small.
+NOTE: The `nccmp` command will be used in future steps. Going forward, some files may not be identical, but any differences reported _should_ be small. Also note: in this step, the target lis input file was generated using `Soil texture spatial transform: mode`
 
 == Wrap-up
 

docs/public_testcase_walkthrough/step_2.adoc

@@ -1,9 +1,9 @@
 = Step 2: LSM "Open-loop" (OL) Experiment (LIS)
 
 :step-label: Step 2
-:download-filename: testcase2_lis_ol_2020.tar.gz
-:input-filesize-compressed: 67MB
-:input-filesize-unpacked: 250MB
+:download-filename: testcase2_lis_ol_2023.tar.gz
+:input-filesize-compressed: 65MB
+:input-filesize-unpacked: 343MB
 
 == Overview
 
@@ -53,9 +53,9 @@ Jump to the <<The LIS Configuration File,next step>>.
 ====
 Before running the download script:
 
-. If needed, install `wget`.
-. Create a link:https://urs.earthdata.nasa.gov/users/new[NASA Earthdata] account if you do not already have one.
-. Follow steps 1-4 in link:https://disc.gsfc.nasa.gov/data-access#mac_linux_wget[these instructions] to save your NASA Earthdata login credentials locally.
+. If needed, install `wget`
+. Create a link:https://urs.earthdata.nasa.gov/users/new[NASA Earthdata] account if you do not already have one
+. Follow steps 1-3 in link:https://disc.gsfc.nasa.gov/data-access#mac_linux_wget[these instructions] to save your NASA Earthdata login credentials locally
 ====
 
 When you are ready to run the script, change directories into `INPUT/` and run the following command:
@@ -98,7 +98,7 @@ In your `$WORKING_DIR`, execute the following command:
 LIS can also be run in parallel. The examples in this walkthrough, however, demonstrate how to run LIS on a single processor. See Chapter 6 in link:{url-lisf-docs}[the LIS Users' Guide] for instructions for running LIS in parallel.
 ====
 
-With a single processor the run should take approximately 20 minutes to complete. If the run fails, diagnose the issue by reviewing any errors that have printed to the terminal and by viewing the contents of `lislog` files (if any are present). If no errors appear and the run appears to have completed successfully, examine the end of any `lislog` files present to check for a confirmation message:
+With a single processor the run should take approximately 20 minutes to complete. If the run fails, diagnose the issue by reviewing any errors that have printed to the terminal and by viewing the contents of `lislog` files (located in the `log` directory as defined by line 42 of `lis.config_noah36_ol`). If no errors appear and the run appears to have completed successfully, examine the end of any `lislog` files present to check for a confirmation message:
 
 [source,shell]
 ----

docs/public_testcase_walkthrough/step_3.adoc

@@ -1,9 +1,9 @@
 = Step 3: Ensemble Restart File Generation (LDT)
 
 :step-label: Step 3
-:download-filename: testcase3_ldt_ensrst_2020.tar.gz
-:input-filesize-compressed: 77KB
-:input-filesize-unpacked: 180KB
+:download-filename: testcase3_ldt_ensrst_2023.tar.gz
+:input-filesize-compressed: 97KB
+:input-filesize-unpacked: 100KB
 
 :ldt-run-dir: DA_ensrst
 :ldt-config-filename: ldt.config
@@ -85,6 +85,6 @@ include::running_ldt.adoc[]
 
 During this run, LDT produced one output file named `LIS_EnRST_NOAH36_201801010000.d01.nc` and placed it into the `$WORKING_DIR/DA_ensrst` directory.
 
-Use `ncview`, Panoply, Matlab or any other viewing package that supports NetCDF files. Compare what you see with the _target_ version of this output file, also located in `$WORKING_DIR/D_ensrst`. Additionally, you can use `nccmp` to directly compare the contents of the file created by your LDT run and the target.
+Use `ncview`, Panoply, Matlab or any other viewing package that supports NetCDF files. Compare what you see with the _target_ version of this output file, also located in `$WORKING_DIR/DA_ensrst`. Additionally, you can use `nccmp` to directly compare the contents of the file created by your LDT run and the target. If there are differences between your file and the target file, you can use `ncdiff` to create a new NetCDF file to visualize the differences. 
 
 // TODO: ADD EXAMPLES HERE

docs/public_testcase_walkthrough/step_4.adoc

@@ -1,9 +1,9 @@
 = Step 4: Generate LSM OL Cumulative Density Function-based Files (LDT)
 
 :step-label: Step 4
-:download-filename: testcase4_ldt_lsmcdf_2020.tar.gz
-:input-filesize-compressed: 362KB
-:input-filesize-unpacked: 815KB
+:download-filename: testcase4_ldt_lsmcdf_2023.tar.gz
+:input-filesize-compressed: 384KB
+:input-filesize-unpacked: 803KB
 :ldt-run-dir: DA_proc_LSM
 :ldt-config-filename: ldt.config.noah36_cdf
 

docs/public_testcase_walkthrough/step_5.adoc

@@ -1,9 +1,9 @@
 = Step 5: Generate Observations CDF-based Files (LDT)
 
 :step-label: Step 5
-:download-filename: testcase5_ldt_obscdf_2020.tar.gz
-:input-filesize-compressed: 4.5GB
-:input-filesize-unpacked: 5.3GB
+:download-filename: testcase5_ldt_obscdf_2023.tar.gz
+:input-filesize-compressed: 4.6GB
+:input-filesize-unpacked: 9.1GB
 :ldt-run-dir: DA_proc_SMAP
 :ldt-config-filename: ldt.config.smapobs_cdf
 
@@ -83,7 +83,7 @@ SMAP(NASA) soil moisture Composite Release ID (e.g., R16): R16 <8>
 <1> _DA preprocessing_ mode is used to generate the observation domain and scaling parameters.
 <2> Prefix of output files (e.g., `cdf_smapobs_domain.nc`).
 <3> Temporal resolution of resultant CDF. Specifies whether to generate lumped (i.e., considering all years and all seasons) CDFs or to stratify CDFs for each calendar month.
-<4> Averaging interval used while computing CDF. In this case, one day.
+<4> Averaging interval used while computing the CDF. In this case, one day.
 <5> Relative path to the directory containing SMAP observation data.
 <6> Specifies which SMAP data product is being used.
 <7> Specifies the radius in which LDT searches to detect open water. Then removes all pixels within the radius in the CDF calculations.

docs/public_testcase_walkthrough/step_6.adoc

@@ -1,9 +1,9 @@
 = Step 6: LSM Data Assimilation (DA) Experiment (LIS)
 
 :step-label: Step 6
-:download-filename: testcase6_lis_da_2020.tar.gz
+:download-filename: testcase6_lis_da_2023.tar.gz
 :input-filesize-compressed: 80MB
-:input-filesize-unpacked: 220MB
+:input-filesize-unpacked: 327MB
 
 == Overview
 
@@ -61,7 +61,7 @@ This step relies on files downloaded or generated in the previous steps. In addi
 
 Review the contents of `lis.config_noah36_smapda` to view the configuration settings used for the DA run:
 
-.Restart options
+.Restart options:
 [source,shell]
 ----
 ...
@@ -72,11 +72,11 @@ Number of ensembles per tile:              12 <2>
 Noah.3.6 restart file:                    ./DA_ensrst/LIS_EnRST_NOAH36_201801010000.d01.nc <3>
 ...
 ----
-<1> Select "restart" mode to use a restart file (compare with coldstart mode).
+<1> Select "restart" mode to use a restart file (as opposed to "coldstart" mode).
 <2> # of ensemble members to use (matches ensemble restart file).
 <3> Filepath of the 12 member ensemble restart file generated in Step 3.
 
-.Data assimilation options
+.Data assimilation options:
 [source,shell]
 ----
 ...
@@ -96,7 +96,7 @@ Data assimilation output innovations:                1
 ...
 ----
 
-.Scaling strategy options
+.Scaling strategy options:
 [source,shell]
 ----
 ...
@@ -111,7 +111,7 @@ Bias estimation restart file:
 ...
 ----
 
-.Perturbation options
+.Perturbation options:
 [source,shell]
 ----
 ...
@@ -139,7 +139,7 @@ Observation perturbation attributes file: ./DA_INPUT/smap_pertattribs.txt
 <2> State perturbation options
 <3> Observation perturbation options
 
-.Observation data options
+.Observation data options:
 [source,shell]
 ----
 SMAP(NASA) soil moisture data designation:       SPL3SMP
@@ -283,6 +283,7 @@ View the output of the run using `GrADS`, `ncview`, Matlab, or other software. C
 
 // TODO: ADD VISUALIZATION OF OUTPUT
 
+.`SoilMoist_tavg` variable in `LIS_HIST_201701010600.d01.nc`
 image::step_6_output_example.png[pdfwidth=50%,align="center"]
 
 == Wrap Up

docs/public_testcase_walkthrough/step_7.adoc

@@ -1,9 +1,9 @@
 = Step 7: Comparison of OL and DA Experiments (LVT)
 
 :step-label: Step 7
-:download-filename: testcase7_lvt_expcomp_2020.tar.gz
-:input-filesize-compressed: 150KB
-:input-filesize-unpacked: 700KB
+:download-filename: testcase7_lvt_expcomp_2023.tar.gz
+:input-filesize-compressed: 481KB
+:input-filesize-unpacked: 7.2MB
 
 == Overview