Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fetch feeds in batches for export_csv #907

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

sylvansson
Copy link
Contributor

@sylvansson sylvansson commented Feb 8, 2025

Summary:

This PR updates the export CSV function to fetch data in batches to reduce its memory usage (#898). My first approach used yield_per which resulted in a large number of queries due to eager loading not being allowed (see fe236b6). Instead we get a full list of feed ids and then fetch data in batches using the existing queries.

Expected behavior:

The export CSV function should use less memory but produce the same output.

Testing tips:

Test against all three environments and check the memory usage.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@@ -0,0 +1,13 @@
from itertools import islice
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Initially wanted to put this file under api/src/utils but ran into import issues when running the tests so I chose the path of least resistance. Happy to move it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant