Remote Ray Trials

[17zhangw]

Note that if we are running trials on ray on different machines, we will need to setup some proper cloud/remote storage that is visible to all machines: ray-project/ray#37177.

This behavior is as-of ray 2.5.

[lmwnshn]

Are we planning experiments that require multi-machine (i.e., we cannot shard experiments manually ourselves)?

I was looking at cloudlab yesterday, but I'm not sure if it is worth the effort to also take on the responsibility for network setup.