Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Number of digest items must match that calculated #4588

Closed
2 tasks done
ehsan6sha opened this issue May 26, 2024 · 4 comments
Closed
2 tasks done

Number of digest items must match that calculated #4588

ehsan6sha opened this issue May 26, 2024 · 4 comments
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known.

Comments

@ehsan6sha
Copy link

ehsan6sha commented May 26, 2024 •
edited
Loading

Is there an existing issue?

  • I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

  • This is not a support question.

Description of bug

I updated the runtime code by following these instructions: https://docs.substrate.io/tutorials/build-a-blockchain/upgrade-a-running-network/
(functionland/sugarfunge-node@b073df0)

And now when syncing from 0, the node gives an error on the block that update happened:

2024-05-25 14:21:54 Substrate Node
2024-05-25 14:21:54 ✌️  version 0.1.0-fab981dafd8
2024-05-25 14:21:54 ❤️  by SugarFunge Developers, 2017-2024
2024-05-25 14:21:54 📋 Chain specification: Functionyard
2024-05-25 14:21:54 🏷  Node name: FulaNode
2024-05-25 14:21:54 👤 Role: AUTHORITY
2024-05-25 14:21:54 💾 Database: RocksDb at /uniondrive/chain/chains/functionyard/db/full
2024-05-25 14:21:54 ⛓  Native runtime: sugarfunge-node-101 (sugarfunge-node-1.tx1.au1)
2024-05-25 14:23:14 Using default protocol ID "sup" because none is configured in the chain specs
2024-05-25 14:23:14 🏷  Local node identity is: 12D3KooWPgLZ3Y4zPuvtDf81VfsAtvy6JWo3
egXp6Jg6WC2Nkkrd
2024-05-25 14:23:14 💻 Operating system: linux
2024-05-25 14:23:14 💻 CPU architecture: aarch64
2024-05-25 14:23:14 💻 Target environment: gnu
2024-05-25 14:23:14 💻 Memory: 7810MB
2024-05-25 14:23:14 💻 Kernel: 6.6.20+rpt-rpi-v8
2024-05-25 14:23:14 💻 Linux distribution: Ubuntu 24.04 LTS
2024-05-25 14:23:14 💻 Virtual machine: no
2024-05-25 14:23:14 📦 Highest known block at #531910
2024-05-25 14:23:14 〽️ Prometheus exporter started at 127.0.0.1:9615
2024-05-25 14:23:14 Running JSON-RPC server: addr=0.0.0.0:9945, allowed origins=["*"]
Connection to 127.0.0.1 9945 port [tcp/*] succeeded!
2024-05-25 14:23:14 Accepting new connection 1/100
2024-05-25 14:23:14 Accepting new connection 1/100
[2024-05-25T14:23:14Z INFO  actix_server::builder] starting 4 workers
[2024-05-25T14:23:14Z INFO  actix_server::server] Actix runtime found; starting in Actix runtime
2024-05-25 14:23:15 🔍 Discovered new external address for our node: /ip4/86.242.11.169/tcp/30335/p2p/12D3KooWPgLZ3Y4zPuvtDf81VfsAtvy6JWo3egXp6Jg6WC2Nkkrd
Connection to 127.0.0.1 4000 port [tcp/*] succeeded!
Connection to 127.0.0.1 5001 port [tcp/*] succeeded!
[2024-05-25T14:23:16Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.013675
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:19 panicked at /home/ubuntu/.cargo/git/checkouts/substrate-7e08433d4c370a21/5e49f6e/frame/executive/src/lib.rs:588:9:
assertion `left == right` failed: Number of digest items must match that calculated.
  left: 2
 right: 1
2024-05-25 14:23:19 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x219d0 - <unknown>!rust_begin_unwind
    1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
    4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
    5: 0xb1d2e - <unknown>!Core_execute_block
2024-05-25 14:23:19 💔 Error importing block 0x44433e9e7f3ea86dc1911f5f4042ce389ec2443f40927c22ba6d9f11699926db: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x219d0 - <unknown>!rust_begin_unwind
    1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
    4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
    5: 0xb1d2e - <unknown>!Core_execute_block
2024-05-25 14:23:19 ⚙️  Syncing, target=#694235 (1 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 82.0kiB/s ⬆ 1.3kiB/s
2024-05-25 14:23:20 panicked at /home/ubuntu/.cargo/git/checkouts/substrate-7e08433d4c370a21/5e49f6e/frame/executive/src/lib.rs:588:9:
assertion `left == right` failed: Number of digest items must match that calculated.
  left: 2
 right: 1
2024-05-25 14:23:20 Block prepare storage changes error: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x219d0 - <unknown>!rust_begin_unwind
    1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
    4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
    5: 0xb1d2e - <unknown>!Core_execute_block
2024-05-25 14:23:20 💔 Error importing block 0x44433e9e7f3ea86dc1911f5f4042ce389ec2443f40927c22ba6d9f11699926db: consensus error: Import failed: Import failed: Error at calling runtime api: Execution failed: Execution aborted due to trap: wasm trap: wasm `unreachable` instruction executed
WASM backtrace:
error while executing at wasm backtrace:
    0: 0x219d0 - <unknown>!rust_begin_unwind
    1: 0x2d4d - <unknown>!core::panicking::panic_fmt::h7d22643b0becf577
    2: 0x384b - <unknown>!core::panicking::assert_failed_inner::hf53be49d983c07ca
    3: 0x1034be - <unknown>!core::panicking::assert_failed::h81267043375e33f4
    4: 0x13ee2d - <unknown>!frame_executive::Executive<System,Block,Context,UnsignedValidator,AllPalletsWithSystem,COnRuntimeUpgrade>::execute_block::h92071aebb140cbc0
    5: 0xb1d2e - <unknown>!Core_execute_block
[2024-05-25T14:23:21Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.001207
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:24 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 80.4kiB/s ⬆ 0.2kiB/s
[2024-05-25T14:23:26Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.003389
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:29 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 21 B/s ⬆ 35 B/s
2024-05-25 14:23:30 🔍 Discovered new external address for our node: /ip6/2a01:cb00:12a6:6500:2664:12f7:d79d:7bd2/tcp/30335/p2p/12D3KooWPgLZ3Y4zPuvtDf81VfsAtvy6JWo3egXp6Jg6WC2Nkkrd
[2024-05-25T14:23:31Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.001223
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:34 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 1.9kiB/s ⬆ 0.9kiB/s
[2024-05-25T14:23:36Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 54 "-" "curl/8.5.0" 0.001243
Waiting for Node API to have at least 3 peers...
2024-05-25 14:23:39 💤 Idle (0 peers), best: #531910 (0x897f…6fa2), finalized #531910 (0x897f…6fa2), ⬇ 0.5kiB/s ⬆ 0.2kiB/s
[2024-05-25T14:23:41Z INFO  actix_web::middleware::logger] 127.0.0.1 "POST /health HTTP/1.1" 200 55 "-" "curl/8.5.0" 0.001217
Waiting for Node API to have at least 3 peers...

Any ideas of how to troubleshoot further?
https://functionyard.fula.network/#/explorer/query/0x44433e9e7f3ea86dc1911f5f4042ce389ec2443f40927c22ba6d9f11699926db

Steps to reproduce

  1. upgrade a running node
  2. sync from scratch

However, strangely it happens on some nodes and not all
@ehsan6sha ehsan6sha added I10-unconfirmed Issue might be valid, but it's not yet known. I2-bug The node fails to follow expected behavior. labels May 26, 2024
@ehsan6sha
Copy link
Author

I tried this one by setting state cache size to 0 but it did not solve the issue:
paritytech/substrate#9697

@bkchr
Copy link
Member

bkchr commented May 26, 2024

Did you increment the spec version?

@ehsan6sha
Copy link
Author

ehsan6sha commented May 26, 2024 •
edited
Loading

Did you increment the spec version?

Yes, both spec_version and EXISTENTIAL_DEPOSIT
https://github.com/functionland/sugarfunge-node/blob/main/runtime/src/lib.rs

@ehsan6sha
Copy link
Author

any ideas on what is causing it or how to troubleshoot further?

@mustermeiszer mustermeiszer mentioned this issue Jun 17, 2024
Closed
2 tasks
@bkchr bkchr closed this as completed Sep 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bkchr @ehsan6sha