juicefs with redis and minio is very slow in macos, linux is normal

[vklonghml]

What happened:
juicefs with redis and minio is very slow in macos

What you expected to happen:
macos performs well

How to reproduce it (as minimally and precisely as possible):
1、juicefs mount --buffer-size 600M --max-uploads 40 redis://:xxx@xx.xx.xx.xx:6379/1 /jfs
2、open mount point ， there are 40 file in mount point, range from 10MB to 1GB
3、each file operation with 25 second delay, like: choose one file, open a file, copy a file

Anything else we need to know?

Environment:

• JuiceFS version (use juicefs --version) or Hadoop Java SDK version: juicefs version 1.2.0+2024-06-18.873c47b
• Cloud provider or hardware configuration running JuiceFS:
• OS (e.g cat /etc/os-release): sonama14.3, M1
• Kernel (e.g. uname -a): Darwin Kernel Version 23.3.0: Wed Dec 20 21:30:44 PST 2023
• Object storage (cloud provider and region, or self maintained): minio with 4 node, 16 disks
• Metadata engine info (version, cloud provider managed or self maintained): redis 5.0.7
• Network connectivity (JuiceFS to metadata engine, JuiceFS to object storage): 20Mb
• Others:

[jiefenghuang]

Can you see any slow requests in the logs? The 'stats' and 'profile' commands should help you analyze the specific problem and identify where the issue is occurring.

You can refer to the documentation below:

• https://juicefs.com/docs/zh/community/performance_evaluation_guide/
• https://juicefs.com/docs/zh/community/fault_diagnosis_and_analysis

[prologic]

Came here to report the same experience. Performance is orders of magnitude slower on macOS with a Redis metadata and MinIO object store :/ is this normal?

[prologic]

I ran juicefs stats as suggested whilst running a juicefs bench and this is what I'm seeing.

[prologic]

I upgraded a few components, including JuiceFS and macFUSE and this is now my results:

JuiceFS (running redis and minio locally):

+------------------+------------------+--------------+
|       ITEM       |       VALUE      |     COST     |
+------------------+------------------+--------------+
|   Write big file |    1282.10 MiB/s |  0.80 s/file |
|    Read big file |    1335.41 MiB/s |  0.77 s/file |
| Write small file |    152.8 files/s | 6.54 ms/file |
|  Read small file |    896.6 files/s | 1.12 ms/file |
|        Stat file |   2010.2 files/s | 0.50 ms/file |
|   FUSE operation | 21176 operations |   0.06 ms/op |
|      Update meta |   725 operations |   0.28 ms/op |
|       Put object |   457 operations |  15.51 ms/op |
|       Get object |   256 operations |  17.31 ms/op |
|    Delete object |     0 operations |   0.00 ms/op |
| Write into cache |   424 operations |   1.73 ms/op |
|  Read from cache |   100 operations |   0.34 ms/op |
+------------------+------------------+--------------+

JuiceFS against local filesystem:

+------------------+-----------------+--------------+
|       ITEM       |      VALUE      |     COST     |
+------------------+-----------------+--------------+
|   Write big file |   2495.86 MiB/s |  0.41 s/file |
|    Read big file |   5887.86 MiB/s |  0.17 s/file |
| Write small file |  2462.3 files/s | 0.41 ms/file |
|  Read small file |  3799.9 files/s | 0.26 ms/file |
|        Stat file | 21746.8 files/s | 0.05 ms/file |
+------------------+-----------------+--------------+

[prologic]

And with JuiseFS with Redis and Minio running in Docker Desktop:

+------------------+------------------+--------------+
|       ITEM       |       VALUE      |     COST     |
+------------------+------------------+--------------+
|   Write big file |     169.66 MiB/s |  6.04 s/file |
|    Read big file |     321.74 MiB/s |  3.18 s/file |
| Write small file |    100.5 files/s | 9.95 ms/file |
|  Read small file |    455.6 files/s | 2.19 ms/file |
|        Stat file |    885.1 files/s | 1.13 ms/file |
|   FUSE operation | 21062 operations |   0.47 ms/op |
|      Update meta |   726 operations |   1.09 ms/op |
|       Put object |   457 operations | 257.66 ms/op |
|       Get object |   256 operations |  93.95 ms/op |
|    Delete object |     0 operations |   0.00 ms/op |
| Write into cache |   457 operations |   1.76 ms/op |
|  Read from cache |   100 operations |   0.32 ms/op |
+------------------+------------------+--------------+

[prologic]

I think there's an inherent cost to macFUSE over the more performant fuse implementation on Linux. I also think there's an additional overhead cost to running Redis/MinIO in Docker for Desktop on macOS.

[sedlund]

Minio's implementation stores each files metadata in a separate file. this causes read/write amplifications.
Users of slow disks and or many small files will notice the poor performance of this design on a native installation. When further layering on fuse and JuiceFS on top of this, this will further exacerbate this design and be a poor performer.

IMO Juicefs is designed for real object storage with low latency network connectivity. if you are looking for a S3 implementation for local use on one machine that is simple to setup and use I would use seaweedfs.