Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce join operation memory consumption #223

Open
d33bs opened this issue Sep 1, 2024 · 0 comments
Open

Reduce join operation memory consumption #223

d33bs opened this issue Sep 1, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@d33bs
Copy link
Member

d33bs commented Sep 1, 2024

After merging #220 pre-join operations appear to require far less memory. Join operations are also reduced but could be further enhanced through SQL-based filtering (perhaps through where clauses in sub-queries), custom page selection (reading limited number of intermediate Parquet files), both those methods, or something else to limit the overall memory consumption for joins.

@d33bs d33bs added the enhancement New feature or request label Sep 1, 2024
@d33bs d33bs moved this from Todo to In Progress in No image left behind Dec 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant