Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

speeding up Parquet_output function by vectorizing #816

Merged
merged 3 commits into from
Aug 1, 2024

Conversation

AminTorabi-NOAA
Copy link
Contributor

@AminTorabi-NOAA AminTorabi-NOAA commented Jul 31, 2024

Reworking a _parquet_output_format_converter function to speeding it up by vectorizing and using numpy array. Previously it was taking around 50 sec. Can see in below picture.
image

But by vectorizing it. It improved to 1 second.
image

The result is also checked its the same.
Original function output:
image

Updated function output:
image

Additions

Removals

Changes

Testing

Screenshots

Notes

Todos

Checklist

  • PR has an informative and human-readable title
  • Changes are limited to a single goal (no scope creep)
  • Code can be automatically merged (no conflicts)
  • Code follows project standards (link if applicable)
  • Passes all existing automated tests
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future todos are captured in comments
  • Visually tested in supported browsers and devices (see checklist below 👇)
  • Project documentation has been updated (including the "Unreleased" section of the CHANGELOG)
  • Reviewers requested with the Reviewers tool ➡️

Testing checklist

Target Environment support

  • Windows
  • Linux
  • Browser

Accessibility

  • Keyboard friendly
  • Screen reader friendly

Other

  • Is useable without CSS
  • Is useable without JS
  • Flexible from small to large screens
  • No linting errors or warnings
  • JavaScript tests are passing

Copy link
Contributor

@shorvath-noaa shorvath-noaa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks good, I'm glad this could be sped up. See my one comment about aggregating values by nexus point ID.

src/troute-nwm/src/nwm_routing/output.py Show resolved Hide resolved
@shorvath-noaa shorvath-noaa dismissed their stale review August 1, 2024 21:28

will be done in another PR

@AminTorabi-NOAA AminTorabi-NOAA merged commit c2cd1b4 into NOAA-OWP:master Aug 1, 2024
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants