ErrnoException: Cannot read from file

[Sallery-X]

[Sallery-X]

strerror: Input/output error

[davies]

Can you check the syslog?

IOError usually means it failed to talk to Redis or object storage, there should be detailed logging in syslog.

[Sallery-X]

where could I find the syslog?

[SandyXSD]

Please take a look at this doc

[Sallery-X]

Can you check the syslog?

IOError usually means it failed to talk to Redis or object storage, there should be detailed logging in syslog.

[davies]

can you post all the logs, not just the ERROR?

[Sallery-X]

can you post all the logs, not just the ERROR?
Apr 29 11:01:50 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : read file 443: input/output error
Apr 29 11:01:50 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : slow operation: read (443,4096,0): input/output error (0) <20.815054>
Apr 29 11:02:32 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:33 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:34 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:36 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:38 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:40 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:42 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:45 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:48 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:52 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2476 (off:0, size:1, clen: 1): get chunks/0/2/2476_0_1: open /var/jfs/clickhouse/chunks/0/2/2476_0_1: no such file or directory
Apr 29 11:02:52 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : read file 432: input/output error
Apr 29 11:02:52 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : slow operation: read (432,4096,0): input/output error (0) <20.815511>
Apr 29 11:02:53 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:02:54 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:02:55 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:02:56 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:02:58 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:03:01 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:03:03 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:03:06 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:03:09 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:03:12 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : fail to read chunkid 2450 (off:0, size:78, clen: 78): get chunks/0/2/2450_0_78: open /var/jfs/clickhouse/chunks/0/2/2450_0_78: no such file or directory
Apr 29 11:03:12 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : read file 443: input/output error
Apr 29 11:03:12 A06-R08-I135-39-5000418 juicefs[8812]: juicefs[8812] : slow operation: read (443,4096,0): input/output error (0) <20.815153>

[davies]

It looks like some chunks are missing from /var/jfs/clickhouse, can you reproduce this with a new volume?

[davies]

What's the version of JuiceFS and when do you created this volume?

[Sallery-X]

What's the version of JuiceFS and when do you created this volume?

0.12.1, yesterday

[Sallery-X]

two node operate one file

[Sallery-X]

I'm trying to make clickhouse to share data among replicas

[davies]

@Sallery-X You are using local disk as the object storage, which is only used for test purpose, because it can't be accessed from multiple machines.

In order to share JuiceFS across multiple machines you need to use other object store, for example, S3.