Redis vs TiKV as JuiceFS metadata engine

[myc724]

We’re using Redis (configured with 8 io-threads) for metadata engine for a while. We want to see whether JuiceFS’s recent support of TiKV can facilitate the current solution. Even though a September report (https://pingcap.com/blog/get-started-with-juicefs-using-tikv) suggested that JuiceFS+Redis is 2~3x slower than JuiceFS+TiKV using TiUP playground mode (ie. over localhost), we decide to deploy TiKV in production mode (https://tikv.org/docs/5.1/deploy/install/production). Using the following cluster topology for TiKV, we found JuiceFS+TiKV is much slower than JuiceFS+Redis. For example, we had 100 worker processes each …

• mount the same volume locally, eg. /tmp/jfs.
• write ten 1GB to the mount point sequentially and sleep 10 microseconds between each write.

The distributed process finished less than 10 seconds on JuiceFS+Redis (8 io-threads and AOF persistence) but 448 seconds on JuiceFS+TiKV. I understand TiKV is better than Redis in terms of reliability and scalability. But, does this performance difference expected? Thanks! Please let me know if other TiKV cluster topology can be tried (for dealing with JuiceFS metadata).

global:
  user: "an_user_id_with_nopasswd_sudo_privilege"
  ssh_port: 22
  deploy_dir: "/tmp/tikv-deploy"
  data_dir:   "/tmp/tikv-data"

server_configs: {}

pd_servers:
  - host: 10.xxx.xxx.161
  - host: 10.xxx.xxx.162
  - host: 10.xxx.xxx.163

tikv_servers:
  - host: 10.xxx.xxx.164
  - host: 10.xxx.xxx.165
  - host: 10.xxx.xxx.166

monitoring_servers:
  - host: 10.xxx.xxx.167

grafana_servers:
  - host: 10.xxx.xxx.168

[davies]

A few questions on the environment and workload:

1. Which kind of disk is used by TiKV? SSD is required.
2. For each worker process, did it write into single file or multiple files? What's the size of each write operation?

juicefs profile and juicefs stats is useful to understand what happened during the benchmark, can you post the screenshot of these two tools?

[myc724]

The /tmp on all nodes are not SSD. Why SSD is required? Is TiKV file-based (like MinIO)? Not an apple-to-apple comparison, but, If I use the same /tmp for MinIO (for object-store), the same experiment just got slightly slower (to 11 seconds).

Each worker process wrote 10 files where each write look like "dd if=/dev/zero of=<mount_point><unique_file_name> bs=1G count=1 oflag=direct". I will dig further using juicefs stats and profile (as you suggested). Thanks!

[davies]

Are you also use the same TiKV cluster as object store?

TiKV is designed for small KV, even RawKV interface used as object store. It has a feature called Titan [1], which is used for bigger KV. It's recommended to use separate cluster for object store and enable Titan in it.

[1] https://docs.pingcap.com/tidb/stable/titan-overview

[davies]

TiKV has lots of small read/write and was not tuned for HDD, and the performance could be very poor with HDD.

[myc724]

I see. Once I get SSD, I will continue the experiment.

[SandyXSD]

No, it's not expected to be that huge, especially for workloads like big sequential writes. Could you try these two tools to get more detailed information?

• stats: shows statistics of FUSE, meta and object storage. Try adding --verbosity 1 argument to get more internal latencies
• profile: shows statistics of all operations

[myc724]

Hmm ... If this performance difference is not expected, I will dig further using juicefs stats and profile (as you suggested). Thanks!