Juicefs 0.16.2 is not able to sustain IO

[ayush-san]

Hi,

We are using juicefs as persistent storage for jupyter notebooks. I have noticed that juicefs is not able to sustain IO for a long duration and after a while, it just slows down. PFA screenshots of juicefs metrics attached when the process is running

If I run this script multiple time, I face this issue

import joblib
import numpy as np
import pandas as pd

def get_parallel_func(df, i):
    df = df.reset_index(drop=True)
    
    temp_arr = np.zeros((df.shape[0], 10, 17), np.float32)
    for j in range(df.shape[0]):
        temp_arr[j, :, :] = np.full((10, 17), 4.556789)*np.full((10, 17), 14.556792)
    
    np.save(f'test/item_{i}.npy', temp_arr)

def get_item_level_embeddings(df):
    m = (df.shape[0]//1000000)+1
    print('df shape: ', df.shape[0], ' and m is: ', m)
    print('Applying a function in df in a parallel manner..')
    joblib.Parallel(n_jobs=16)(joblib.delayed(get_parallel_func)(df.loc[i*1000000:((i+1)*1000000)-1], i) for i in range(m))
    
    for i in range(m):
        if i==0:
            arr = np.load(f'test/item_{i}.npy')
        else:
            temp_arr = np.load(f'test/item_{i}.npy')
            arr = np.vstack([arr, temp_arr])
    
    print(f'array shape: {arr.shape}')
    np.save('test/arr.npy', arr)

df = pd.DataFrame({'col_a': range(20000000)})
df['col_b'] = list(range(20000000))
get_item_level_embeddings(df)

Command use to mount jfs on the host machine

sudo /usr/local/bin/juicefs mount --metrics 0.0.0.0:9567 --no-usage-report --writeback -d -o allow_other,writeback_cache redis://REDIS_DNS:6379/1 /mnt/jfs

Output of /mnt/jfs/.config

{
 "Meta": {
  "Strict": true,
  "Retries": 10,
  "CaseInsensi": false,
  "ReadOnly": false,
  "OpenCache": 0,
  "MountPoint": "/mnt/jfs",
  "Subdir": ""
 },
 "Format": {
  "Name": "jfs",
  "UUID": "fc206360-e9b2-4172-b125-705964b242cd",
  "Storage": "s3",
  "Bucket": "https://s3.ap-southeast-1.amazonaws.com/BUCKET_NAME",
  "AccessKey": "",
  "BlockSize": 4096,
  "Compression": "none",
  "Shards": 0,
  "Partitions": 0,
  "Capacity": 0,
  "Inodes": 0
 },
 "Chunk": {
  "CacheDir": "/var/jfsCache/fc206360-e9b2-4172-b125-705964b242cd",
  "CacheMode": 384,
  "CacheSize": 1024,
  "FreeSpace": 0.1,
  "AutoCreate": true,
  "Compress": "none",
  "MaxUpload": 20,
  "Writeback": true,
  "Partitions": 0,
  "BlockSize": 4194304,
  "GetTimeout": 60000000000,
  "PutTimeout": 60000000000,
  "CacheFullBlock": true,
  "BufferSize": 314572800,
  "Readahead": 0,
  "Prefetch": 1
 },
 "Version": "0.16.2 (2021-08-25T04:01:15Z 29d6fee)",
 "Mountpoint": "/mnt/jfs"
}

JuicefS logs /var/log/juicefs.log

2022/01/14 04:56:47.675019 juicefs[6684] <INFO>: Meta address: redis://REDIS_DNS:6379/1
2022/01/14 04:56:47.678436 juicefs[6684] <WARNING>: AOF is not enabled, you may lose data if Redis is not shutdown properly.
2022/01/14 04:56:47.679342 juicefs[6684] <INFO>: Ping redis: 879.072µs
2022/01/14 04:56:47.680393 juicefs[6684] <INFO>: Data use s3://BUCKET_NAME/jfs/
2022/01/14 04:56:47.680524 juicefs[6684] <INFO>: Disk cache (/var/jfsCache/fc206360-e9b2-4172-b125-705964b242cd/): capacity (1024 MB), free ratio (10%), max pending pages (15)
2022/01/14 04:56:47.719117 juicefs[6711] <INFO>: Meta address: redis://REDIS_DNS:6379/1
2022/01/14 04:56:47.722290 juicefs[6711] <WARNING>: AOF is not enabled, you may lose data if Redis is not shutdown properly.
2022/01/14 04:56:47.723099 juicefs[6711] <INFO>: Ping redis: 767.63µs
2022/01/14 04:56:47.724065 juicefs[6711] <INFO>: Data use s3://BUCKET_NAME/jfs/
2022/01/14 04:56:47.724220 juicefs[6711] <INFO>: Disk cache (/var/jfsCache/fc206360-e9b2-4172-b125-705964b242cd/): capacity (1024 MB), free ratio (10%), max pending pages (15)
2022/01/14 04:56:47.729044 juicefs[6711] <INFO>: Mounting volume jfs at /mnt/jfs ...

[xiaogaozi]

Could you try the latest version (1.0.0-beta1) to run the test again?

[ayush-san]

@xiaogaozi Is there any documentation regarding any breaking changes while updating the the juicefs version from 0.16.2 to 1.0.0-beta1

Also any timeline for the 1.0.0 release?

[xiaogaozi]

For differences between versions, please refer to the release notes: https://github.com/juicedata/juicefs/releases.

The 1.0.0 release has no confirmed release date yet.

[ayush-san]

ok thanks

Why would EBS perform better than juicefs? Is there any property tuning I should do? Since my use case is mostly analytical(reading orc/parquet files) what should be my chunk size?

[ayush-san]

@xiaogaozi Can we backport the issue causing IO throttling to 0.16 or 0.17 releases? Assuming 1.0.0 release is gonna take time

[xiaogaozi]

@ayush-san Could you try v0.17.5 and run again?