dispute-coordinator: peers banned due to failure to import statements

[sandreim]

Seeing a burst of these warnings on Kusama:

WARN tokio-runtime-worker sub-libp2p::peerstore: PeerId("12D3KooWPCuh53rEcvLjE7P5CHfstEPi5Bmgj4wgnPVMHDqwCu4K") banned, disconnecting, reason: Import was deemed invalid by dispute-coordinator.

https://grafana.teleport.parity.io/goto/YjeuLB9Ig?orgId=1

It seems that the import fails because spam slots are full:
DEBUG tokio-runtime-worker parachain::dispute-coordinator: Rejecting import because of full spam slots. candidate_hash=0xad6b8ee78cdc61ba21770d5681ac882829ad6b72c3daa7c41220256c3c570a0a session=41088 invalid_voters=[ValidatorIndex(190)] traceID=230514917482117258459231744050202445864

https://grafana.teleport.parity.io/goto/ARV1EBrSg?orgId=1

CC @tdimitrov @eskimor

[tdimitrov]

A vote from a disabled validator is treated as spam too:

polkadot-sdk/polkadot/node/core/dispute-coordinator/src/lib.rs

Line 478 in 002d926

let all_invalid_votes_disabled = vote_state.invalid_votes_all_disabled(is_disabled);

It would be nice to have a TRACE log showing why exactly a vote is considered spam -> #5252

We do have 3 disabled validators at this point in time (n_disabled_validators=3):

	2024-08-05 21:42:31.856	
2024-08-05 18:42:31.830 DEBUG tokio-runtime-worker parachain::dispute-coordinator: Rejecting import because of full spam slots. candidate_hash=0xad6b8ee78cdc61ba21770d5681ac882829ad6b72c3daa7c41220256c3c570a0a session=41088 invalid_voters=[ValidatorIndex(190)] traceID=230514917482117258459231744050202445864
	
	
2024-08-05 21:42:31.856	
2024-08-05 18:42:31.830 TRACE tokio-runtime-worker parachain::dispute-coordinator: Is spam? own_vote_missing=false potential_spam=true is_included=false candidate_hash=0xad6b8ee78cdc61ba21770d5681ac882829ad6b72c3daa7c41220256c3c570a0a confirmed=false has_invalid_voters=true n_disabled_validators=3 traceID=230514917482117258459231744050202445864
	
	
2024-08-05 21:42:31.856	
2024-08-05 18:42:31.830 TRACE tokio-runtime-worker parachain::dispute-coordinator: Import result ready candidate_hash=0xad6b8ee78cdc61ba21770d5681ac882829ad6b72c3daa7c41220256c3c570a0a session=41088 n_validators=500 traceID=230514917482117258459231744050202445864

My guess at this point is that a validator has raised disputes, was disabled and warnings were generated. Looking further to confirm this theory.

[tdimitrov]

I've checked that there are no onchain disabled validator at the end of session 41088 (last block from it is 24,354,639) so I believe the warnings were caused by offchain disabling. Maybe we should reconsider the severity of the failed import log message. Just saw that the warn was caused by banning the peer not import failure.

Meanwhile I've extended the logging PR with a few more traces. I believe it will make debugging such issues easier in the future.

[tdimitrov]

Logs are merged. @sandreim can we close this?

[eskimor]

Should go away with runtime disabling enabled ... pun not intended.

[sandreim]

Would they really go away ? AFAIU the on-chain disablement woll not cover small ofenders like voting against valid as we can't safely slash them until we implement re-enabling.

[sandreim]

alleviated by #5392