[Feature request]: flepimop sync CLI Tool

[TimothyWillard]

Label

enhancement

Priority Label

low priority

Is your feature request related to a problem? Please describe.

flepiMoP in general, but in particular inference runs, produce a large number of outputs. Both in number of files and size of files. This presents a challenge when trying to use the files and in particular move files from HPC environments to local for development/analysis or to AWS for long-term storage.

Is your feature request related to a new application, scenario round, pathogen? Please describe.

No response

Describe the solution you'd like

A general utility that can handle moving files locally on the filesystem (directory to directory), moving files between computers directly (local to HPC or visa versa), and moving files between cloud storage (local to AWS or visa versa). A (rough) outline of the man page:

Usage: flepimop sync [OPTIONS] [LOCATION]...

  Sync configuration, manifest, and model input/output files across destinations.

Options:
  --content [all|config|manifest|input|output] The type of files to sync. [default: all]
  --pull                                       A flag indicating if this sync should be a 
                                               pull instead of a push.
  -v, --verbose                                The verbosity level to use for this command.
  --dry-run                                    Should this command be run using dry run?
  --help                                       Show this message and exit.

This is very loose as there are likely options required for pointing to credentials if stored in non-standard locations as well as for the flepi/project paths. There are some open questions about this utility:

1. How should destination be interpreted, assuming that the order of operations would be to check for s3:// prefix for AWS, <cluster name>:// for HPCs, and then otherwise default to local filesystem?
2. I've structured this such that it's push, location=destination, by default instead of pull, location=source, but worth think about if that's the preferred usage or if that's even the right framing (maybe source/destination should always be required so there's no confusion)?
3. I think it would be extremely useful if there was a way to filter the content being transferred. For example moving a subset of a large inference run from an HPC to a local machine for development/testing purposes. What would be the semantics of filtering? By subpopulation, chain, iterations, etc.? This one is complicated enough that it might warrant splitting into it's own issue after a first pass at this.

In general, can think of this as the next step beyond GH-192/GH-296, but with more functionality. I think that PR provides some of the infrastructure needed for this issue. I also think this becomes much easier to do after standardizing inference outputs.

[TimothyWillard]

GH-501