Can I convert my full filepaths to partially templated paths?

[rsignell-usgs]

I used SingleHdf5toZarr to generate references on a local filesystem for some remotely mounted filesystem NetCDF files, and then used MultiZarrToZarr to combine them. The resulting JSON ended up full of long paths, the prefix of which is all the same, for example:

  'latitude/0.0': ['W:data/CMIP6/HadGEM3-GC31-HH_10km/r1i1p1f1/highres-future/thetao_Omon_HadGEM3-GC31-HH_highres-future_r1i1p1f1_gn_201501-201501.nc',
   100602,
   18929],
  'latitude/0.1': ['W:data/CMIP6/HadGEM3-GC31-HH_10km/r1i1p1f1/highres-future/thetao_Omon_HadGEM3-GC31-HH_highres-future_r1i1p1f1_gn_201501-201501.nc',
   119531,
   153765],
  'latitude/0.2': ['W:data/CMIP6/HadGEM3-GC31-HH_10km/r1i1p1f1/highres-future/thetao_Omon_HadGEM3-GC31-HH_highres-future_r1i1p1f1_gn_201501-201501.nc',
   273296,
   94817],

I know in other references I've created with kerchunk, the templates specify the entire file path, like:

templates:
    u:"s3://sciencebase/609bf69ed34ea221ce39b261/breach_wilderness_veg.nc"

In my use case all the netcdf files are prefixed by: W:data/CMIP6/HadGEM3-GC31-HH_10km/r1i1p1f1/highres-future and it would be nice to be able to specify that as a template.

Is it possible to use the templates capability to specify the common part of a path? Like:

templates:
    u:"W:data/CMIP6/HadGEM3-GC31-HH_10km/r1i1p1f1/highres-future"

This would reduce the reference file size, but more importantly make it easy to switch the prefix, which is important in this use case since the data files appear differently to different users, and so they would need to switch the references accordingly!

[martindurant]

You maybe want to use function kerchunk.utils.templateize, but you'll need to push the references back into the reference set yourself. Since you already know the likely best prefix, a little loop with path.replace(prefix, "{u}")would do.
However, I think with any decent compression like Zstd, it won't make as much of a difference in size as you imagine.

Another more exotic solution would be to wrap the "target" filesystem with a "DirFileSystem", so that you always specify the prefix path at runtime.

[rsignell-usgs]

The more exotic solution sounds interesting, but I'm not sure how to make that work.
Would I create a reference file system fs that does not contain the root_path, and then somehow use

from fsspec.implementations.dirfs import DirFileSystem

fs_new = DirFileSystem(root_path, fs)

The specific use case:

We created the reference file system on Windows where the climate data is mounted as:

W:data/CMIP6

But we would like to use the reference file system also on linux machines where the data is mounted as:

/data/climate_data/CMIP6

[martindurant]

DirFileSystem will only add prefix paths, so it wouldn't get rid of the "W:" stuff. But otherwise, yes the idea would be to create a chain of filesystems like

fs_target = fsspec.filesystem("file")  # or s3 whatever
fs_prefix = DirFileSystem(root_path, fs_target)  # it is not yet in the registry, so fsspec.filesystem doesn't find it
fs = fsspec.filesystem("reference", fo=path_to_refs, fs=fs_prefix)

[martindurant]

I wouldn't be averse to DirFileSystem being able to replace path bits in general (more like "mount"), or this functionality could be in ReferenceFS itself, which already has a lot of options and complexity and is in the generally business of transforming paths from one system to another.

[rsignell-usgs]

Okay, thanks @martindurant.
For now we solved it the old fashioned way:

import fsspec
import ujson as json

combined_reference_file  = './HadGEM3/combined.json'

data_dir = 'W:data/CMIP6'
new_data_dir = '/home/richard.signell/data-CMIP_climate_data/data/CMIP6'

with open(combined_reference_file) as f:
    refs_dict=json.loads(f.read().replace(data_dir.replace('/','\\/'),  new_data_dir.replace('/','\\/')))
                                   
fs = fsspec.filesystem('reference', fo=refs_dict, skip_instance_cache=True)   

[martindurant]

That works too