Performance: slow metadata service

[steven-varga]

What happened:
copying linux kernel tree show few KB/sec throughput at best, mdtest showed 5 - 10 transactions per second
What you expected to happen:
10MB/sec throughput for similar class file systems for linux kernel tree copy, 1000 - 10000 transaction per second based on redis performance
How to reproduce it (as minimally and precisely as possible):
on decent/non-virtual AWS EC2 instance setup/run redis, 22ms latency away run juiceFS mounted as a local directory, copy a recent linux kernel tree using {rsync, cp, midnight commander, ...}
Anything else we need to know?:
Great project/undertaking!
When the mounted directory is much 'closer' to redis, the metadata service exhibits fair behaviour. IO ops for larger files are OK, and for multi GB files JuiceFS has excellent performance; although this has not been verified for multi host mixed read/write scenerio.
Environment:

• JuiceFS version (use ./juicefs --version): 0.9.3-211 (2021-01-28 9cdfa8a)
• Cloud provider or hardware configuration running JuiceFS: a1.2xlarge
• OS (e.g: cat /etc/os-release): ubuntu 20.04
• Kernel (e.g. uname -a): 0.9.3-211 (2021-01-28 9cdfa8a)
• Object storage (cloud provider and region): aws s3 us-east-1
• Redis info (version, cloud provider managed or self maintained): 6.0.10
• Network connectivity (JuiceFS to Redis, JuiceFS to object storage): 500Mbit/sec bell fiber to redis, same for s3
• Others:

[davies]

@steven-varga The latency of meta operation is critical to JuiceFS, that's why we choose to use Redis here. It's usually expected that the latency to Redis server is less than 2ms. JuiceFS will be unusable if the latency goes above 10ms, and there is not much we can do for now.

So I'd say the current performance is expected under your network condition.

What's the use case there?

[steven-varga]

I am working on a distributed computing service, a rental supercomputer so to speak. The file system requirements are three fold:

1. for data service able to partial read and write a single but massive data file, in HDF5 format -- into S3 buckets; juiceFS does pretty well on this side; in few weeks will share with the results. In this case the metadata server is not stressed much, as HDF5 stores metadata within its container.

2. able to host SPACK; this is where it gets harder on the metadata service -- when you compile large software systems the files are opened/read rapidly many times.

3. possibly to share the SPACK with on premise workstations to provide consistent work experience-- and this is where I am having problems: an HDF5 compile takes too long because the meta data service has inadequate latency when redis server is 'far' in terms of packet round trip time.

[davies]

For 2), we will use the Redis client cache to speedup read, and enable writeback_cache to speedup random write. The redis cache may also help 3). Stay tuned!

[davies]

Move it into discussion.