Non-UTF8 characters are output when a scalar changes between files

[gdicker1]

When I was working on new input files for ESCOMP/CAM #1029 I was getting unexpected failures of the BASELINE for tests using the new files. Instead of finding a "DIFF" the tests were reporting "Exception during BASELINE:".

Example TestStatus.log output:

 2024-09-11 20:22:32: Exception during BASELINE:
 'utf-8' codec can't decode byte 0xf5 in position 6813: invalid start byte
 Traceback (most recent call last):
   File "/glade/derecho/scratch/gdicker/2024Jun04_CAMMPASv8Grids_2/CAM_prtest/cime/CIME/SystemTests/system_tests_common.py", line 618, in               _phase_modifying_call
     function()
   File "/glade/derecho/scratch/gdicker/2024Jun04_CAMMPASv8Grids_2/CAM_prtest/cime/CIME/SystemTests/system_tests_common.py", line 758, in               _compare_baseline
     self._log_cprnc_output_tail(pattern)
   File "/glade/derecho/scratch/gdicker/2024Jun04_CAMMPASv8Grids_2/CAM_prtest/cime/CIME/SystemTests/system_tests_common.py", line 540, in               _log_cprnc_output_tail
     cprnc_log_tail = fin.readlines()[-20:]
   File "/glade/u/apps/derecho/23.09/opt/._view/ki3gspoeflu225iuymcch2cddghxcain/lib/python3.10/codecs.py", line 322, in decode
     (result, consumed) = self._buffer_decode(data, self.errors, final)
 UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf5 in position 6813: invalid start byte

This is because the following text is in the *.cprnc.out file that was being read and causing the python read or readlines functions to fail:

  cf1
                    1   1.346970915794373E+00õÃ5X1<8d>õ?
  RMS cf1                              2.9020E-08            NORMALIZED  0.0000E+00
 
  cf2
                    1  -5.392833948135376E-01Ù^U(TÏAá¿
  RMS cf2                              3.7545E-08            NORMALIZED  0.0000E+00
 
  cf3
                    1   1.923125386238098E-01»7ò<8e>²<9d>È?
  RMS cf3                              6.9598E-09            NORMALIZED  0.0000E+00

The dimensionless cf1, cf2, and cf3 variables changed between files (as expected), but apparently were not handled well when CPRNC printed results.

Example to reproduce:

Using version after tag cam6_4_033 from ESCOMP/CAM, run a ERS_Ln9_P36x1.mpasa480_mpasa480.F2000climo.derecho_intel.cam-outfrq9s_mpasa480 test and compare against the cam6_4_032 baseline.

# From within a fully checked out CAM clone, assuming Derecho
./cime/scripts/create_test \
  --compare "/glade/campaign/cesm/community/amwg/cam_baselines/cam6_4_032_intel" \ 
  ERS_Ln9_P36x1.mpasa480_mpasa480.F2000climo.derecho_intel.cam-outfrq9s_mpasa480