zarr-python v3 compatibility
#516
Conversation
| @@ -496,6 +516,8 @@ def _translator( | |||
| if h5obj.fletcher32: | |||
| logging.info("Discarding fletcher32 checksum") | |||
| v["size"] -= 4 | |||
| key = str.removeprefix(h5obj.name, "/") + "/" + ".".join(map(str, k)) | |||
There was a problem hiding this comment.
This is the same as what _chunk_key did? Maybe make it a function with a comment saying it's a copy/reimplementation.
By the way, is h5obj.name not actually a string, so you could have done h5obj.name.removeprefix()?
| shape=h5obj.shape, | ||
| dtype=dt or h5obj.dtype, | ||
| chunks=h5obj.chunks or False, | ||
| fill_value=fill, | ||
| compression=None, | ||
| compressor=None, |
There was a problem hiding this comment.
So here, you could reintroduce the compressor
filters = filters[:-1]
compressor = filters[-1]
but obviously it depends on whether there are indeed any filters at all.
It would still need back compat, since filters-only datasts definitely exist.
There was a problem hiding this comment.
yeah the big issue is that v3 cares about what type of operation it is, and v2w doesnt so moving them around doesnt necessarily fix that bug
There was a problem hiding this comment.
So there needs to be a change upstream?
There was a problem hiding this comment.
Yes this: zarr-developers/zarr-python#2325
kerchunk/hdf.py
Outdated
| for k, v in self.store_dict.items(): | ||
| if isinstance(v, zarr.core.buffer.cpu.Buffer): | ||
| key = str.removeprefix(k, "/") | ||
| new_keys[key] = v.to_bytes() | ||
| keys_to_remove.append(k) | ||
| for k in keys_to_remove: | ||
| del self.store_dict[k] |
There was a problem hiding this comment.
This is the hacky bit and could use some explanations. Even when requesting "v2", zarr makes Buffer objects, and the keys are also wrong?
There was a problem hiding this comment.
Yeah so two issues here:
- the keys we get from hdf are for example
/depth/.zarraywhen then need to bedepth/.zarray - we cant jsonify buffers, which is how the internal
MemoryStorein v3 stores its data. So we need to convert the buffers to bytes to be serialized
There was a problem hiding this comment.
OK - would appreciate comments on the code saying this.
|
Also worth noting... zarr 3 doesn't support numcodecs codecs out of the box. There is a pr to help this zarr-developers/numcodecs#524 (see also here for updated version zarr-developers/numcodecs#597) but it would mean a change to codec names which causes an incompatibility. For the initial icechunk examples we handle this in virtualizarr but long term it probably belongs here to work standalone. |
So far the only tested file type is HDF5. Thats the only module that currently works with new zarr python in some way