Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Database corruption during warp sync #5383

Open
2 tasks done
koenmtb1 opened this issue Aug 16, 2024 · 0 comments
Open
2 tasks done

Database corruption during warp sync #5383

koenmtb1 opened this issue Aug 16, 2024 · 0 comments
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known.

Comments

@koenmtb1
Copy link

Is there an existing issue?

  • I have searched the existing issues

Experiencing problems? Have you tried our Stack Exchange first?

  • This is not a support question.

Description of bug

I've tried using warp sync on a new node a few times to get the node up and running quickly.
But twice now have I run into issues with database corruption half way through the sync. The state sync goes fine and it will also successfully import the state; but it will eventually crash when downloading block history.

When I start the node it will throw an error that looks like this:

Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.
Run with RUST_BACKTRACE=full to include source snippets.
2024-08-16 12:23:20 Parity Polkadot    
2024-08-16 12:23:20 ✌️  version 1.15.0-743dc632fd6    
2024-08-16 12:23:20 ❤️  by Parity Technologies <admin@parity.io>, 2017-2024    
2024-08-16 12:23:20 📋 Chain specification: Polkadot    
2024-08-16 12:23:20 🏷  Node name:     
2024-08-16 12:23:20 👤 Role: AUTHORITY    
2024-08-16 12:23:20 💾 Database: RocksDb at /parity/chains/polkadot/db/full    
Error: 
   0: Database error: Corruption: block checksum mismatch: stored = 750900125, computed = 2049841005, type = 4  in /parity/chains/polkadot/db/full/140831.sst offset 1297885 size 27162
   1: Corruption: block checksum mismatch: stored = 750900125, computed = 2049841005, type = 4  in /parity/chains/polkadot/db/full/140831.sst offset 1297885 size 27162

  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
                                 ⋮ 1 frame hidden ⋮                               
   2: polkadot::main::hd30df8ede3b815fa
      at <unknown source file>:<unknown line>
   3: std::sys_common::backtrace::__rust_begin_short_backtrace::h9e678cfc7f31d6ed
      at <unknown source file>:<unknown line>
   4: main<unknown>
      at <unknown source file>:<unknown line>
   5: __libc_start_main<unknown>
      at <unknown source file>:<unknown line>
   6: _start<unknown>
      at <unknown source file>:<unknown line>

I also have logs from when the node crashed for the first time if that's useful. Let me know if any more information is helpful.
Just as a side note; I have successfully warp synced other nodes perfectly fine so I haven't ruled out hardware issues. From a software perspective these nodes are the exact same (same OS installed with the same ISO and configured exactly the same with Ansible).

Steps to reproduce

Start a node with the following paramaters with the docker image parity/polkadot:stable2407

      --base-path=/parity
      --chain polkadot
      --name ""
      --rpc-methods=unsafe
      --rpc-external
      --prometheus-external
      --rpc-cors all
      --state-pruning 256
      --blocks-pruning 256
      --validator
      --no-private-ip
      --node-key-file /parity/node-key.json
      --sync warp
@koenmtb1 koenmtb1 added I10-unconfirmed Issue might be valid, but it's not yet known. I2-bug The node fails to follow expected behavior. labels Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
I2-bug The node fails to follow expected behavior. I10-unconfirmed Issue might be valid, but it's not yet known.
Projects
None yet
Development

No branches or pull requests

1 participant