Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use chunks="auto" by default in xarray_open_kwargs #633

Closed
wants to merge 2 commits into from

Conversation

dougiesquire
Copy link
Contributor

Change Summary

Very small change to default to chunks="auto" rather than chunks={} when opening files with xarray. This is motivated by recent changes in xarray to the behaviour of chunks={}.

Related issue number

Closes #632

Checklist

  • Tests pass on CI

@dougiesquire dougiesquire requested a review from mgrover1 as a code owner October 9, 2023 22:10
@dougiesquire
Copy link
Contributor Author

Ah, unfortunately auto rechunking does not work with object dtype. This is probably too common for it to make sense to use chunks="auto" by default. Some options:

  • try to use chunks="auto" and revert to chunks={} if that fails;
  • use chunks=-1, which gives the same behaviour as pre-v2023.09.0 chunks={} (prior to v2023.09.0, these did the same thing);
  • ditch this and stick with chunks={}

Thoughts @mgrover1, @dcherian?

@dougiesquire dougiesquire marked this pull request as draft October 9, 2023 23:18
@mgrover1
Copy link
Collaborator

Ah, unfortunately auto rechunking does not work with object dtype. This is probably too common for it to make sense to use chunks="auto" by default. Some options:

  • try to use chunks="auto" and revert to chunks={} if that fails;
  • use chunks=-1, which gives the same behaviour as pre-v2023.09.0 chunks={} (prior to v2023.09.0, these did the same thing);
  • ditch this and stick with chunks={}

Thoughts @mgrover1, @dcherian?

I think sticking with chunks={} by default makes the most sense here still... any thoughts here @dcherian or @andersy005 ?

@andersy005
Copy link
Member

Ah, unfortunately auto rechunking does not work with object dtype. This is probably too common for it to make sense to use chunks="auto" by default. Some options:

  • try to use chunks="auto" and revert to chunks={} if that fails;
  • use chunks=-1, which gives the same behaviour as pre-v2023.09.0 chunks={} (prior to v2023.09.0, these did the same thing);
  • ditch this and stick with chunks={}

Thoughts @mgrover1, @dcherian?

I think sticking with chunks={} by default makes the most sense here still... any thoughts here @dcherian or @andersy005 ?

i vote in favor of sticking with chunks={} unless implementing the work around above ( * try to use chunks="auto" and revert to chunks={} if that fails;) doesn't result in complicated code :(

@dougiesquire
Copy link
Contributor Author

Thanks @mgrover1 and @andersy005. I agree with you both so I'll close this PR

@dougiesquire dougiesquire deleted the 632_chucks_auto branch October 27, 2023 06:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Change default xarray_open_kwargs to use chunks="auto"
3 participants