juicefs client out of memory

[deelaaay]

Out of memory: Kill process 2523920 (juicefs) score 444 or sacrifice child
Killed process 2523920 (juicefs) total-vm:15826948kB, anon-rss:14931252kB, file-rss:0kB, shmem-rss:0kB
oom_reaper: reaped process 2523920 (juicefs), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

exec dmesg ,display the above content。 how do I control the JuiceFS client physical memory usage?

[davies]

There is a option (--buffer-size) to control the memory used for read/write, it's 300MB by default, the actual used memory could be higher than that, but should not be as high as 15GB.

What's the workload were you running? Which version?

[deelaaay]

juicefs stats /mount-point --verbosity 1

juicefs --version
juicefs version 1.0.0-beta1 (2022-01-13T15:45:45Z 28a42cf)

workload were videos and pictures，as shown above picture，The buffer size is always less than 300M。 but the juicefs client process used memory keeps growing until oom 。 how do I control the JuiceFS client physical memory usage?

[davies]

There could be a bug causing memory leaking, can you upload the heap of the process:

wget localhost:6060/debug/pprof/heap

If the above command fail, please find the correct port of debug agent using lsof

[deelaaay]

[root@node165 ~]# lsof -i -nP | grep LISTEN | grep juicefs
juicefs 133460 root 11u IPv4 410700 0t0 TCP 127.0.0.1:6062 (LISTEN)
juicefs 133460 root 13u IPv4 408878 0t0 TCP 127.0.0.1:6072 (LISTEN)
juicefs 133460 root 17u IPv4 405295 0t0 TCP 127.0.0.1:9567 (LISTEN)

[root@node165 ~]# wget localhost:6062/debug/pprof/heap

heap.zip

thanks

[davies]

Here are the graph generated by go tool pprof

We can see there are lots of goroutines (2.7 millions) waiting to upload the blocks, and lots of objects used by goroutines are not freed. So it's caused by slow upload of objects in writeback mode, we will optimized this use case and terminate these goroutines to save memory.

[SandyXSD]

How many files in your volume (df -i)? And what's the metadata engine? It may be caused by the backup of metadata.
Mount with -v enables debug log for the start time of backup.