More time-efficient order in timeslice file creation

[JurgenZach-NOAA]

Changed the order of timeslice file import and creation of dataframes. Previously, a pandas dataframe was created for each timeslice, and then the dataframes were concatenated into a combined dataframe for 15-minute resampling. Now, only numpy arrays are imported, which are then combined into one large dataframe.

Additions

Removals

Changes

• For reading individual timeslices, added a routine _read_timeslice_file_numpy in nhd_io that only returns numpy arrays
• Rewrote get_obs_from_timeslices: streamlined creation of the observation_df in one step

Testing

1. Running any example that reads usgs timeslice files
2. If you want to benchmark the time, the pyinstruments profiler is suggested, which will create a report with 1 millisecond time resolution:

from pyinstrument import Profiler
profiler = Profiler()
profiler.start()

 [CODE YOU WANT TO BENCHMARK]

profiler.stop()
profiler.print()

Screenshots

For Lower Colorado example [test_AnA.yaml]: benchmark for get_obs_from_timeslices shows 2-fold speedup:

BEFORE:

AFTER:

Notes

Todos

• Suggest considering to delete the "quality" dataframe, which will result in further speedup

Checklist

• PR has an informative and human-readable title
• Changes are limited to a single goal (no scope creep)
• Code can be automatically merged (no conflicts)
• Code follows project standards (link if applicable)
• Passes all existing automated tests
• Any change in functionality is tested
• New functions are documented (with a description, list of inputs, and expected output)
• Placeholder code is flagged / future todos are captured in comments
• Visually tested in supported browsers and devices (see checklist below 👇)
• Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
• Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

• Windows
• Linux
• Browser

Accessibility

• Keyboard friendly
• Screen reader friendly

Other

• Is useable without CSS
• Is useable without JS
• Flexible from small to large screens
• No linting errors or warnings
• JavaScript tests are passing

[AminTorabi-NOAA]

I test it on an example I had on vpu-17 and on this line timeslice_obs_df = pd.concat(dfList, axis = 1)
It gives error that *** pandas.errors.InvalidIndexError: Reindexing only valid with uniquely valued Index objects
Probably because you try to concatenate dataframes that have non-unique index values. Also when I checked there was a difference in length between dfList[0] and dfList[1] and others.

[JurgenZach-NOAA]

Archived. End of Project.