catchup: fetchAndWrite/fetchRound quit early on errNoBlockForRound

[algorandskiy]

Summary

Checking this test failure (three nodes, one has 99%+ of the stake) I found frequent retries with after errNoBlockForRound could lead to punishing nodes. In particular in this test the relay node punished node1 and failed to catchup.

Two functions are modified in slightly different way:

1. fetchAndWrite simply checks for errNoBlockForRoundThreshold number of errors before giving up.
2. fetchRound sleeps and recreate the local peer selector. This flow is for a case when agreement saw the cert but not the block itself, and asking the block immediately could fail since the other side has not added it into the ledger yet. According to the method contract it must try indefinitely. To fix a rare situation with punishing one of two nodes more the peer selector gets recreated if too many errors.

Additionally, modified how peerRankDownloadFailed is handled in rankPeer ->> historicStats.push.

1. Added peerRankNoBlockForRound to handle "no block error" a bit differently: the penalty function is p*2^(0.1*x) instead of p*2^x for this case to punish no block errors less aggressive.

Test Plan

1. Added a unit test
2. Removed ReadyCheck from TestBasicCatchup. The test is too fast now, from start to finish catchup on 3-10 blocks takes only 20 ms. ReadyCheck still covered by TestReadyEndpoint.

[codecov]

Codecov Report

Merging #5809 (f70f0aa) into master (92af372) will decrease coverage by 0.05%.
The diff coverage is 68.75%.

@@            Coverage Diff             @@
##           master    #5809      +/-   ##
==========================================
- Coverage   55.64%   55.60%   -0.05%     
==========================================
  Files         475      475              
  Lines       66869    66892      +23     
==========================================
- Hits        37209    37193      -16     
- Misses      27146    27178      +32     
- Partials     2514     2521       +7     

Files	Coverage Δ
catchup/peerSelector.go	98.71% <100.00%> (-1.29%)	⬇️
catchup/service.go	70.41% <50.00%> (-1.95%)	⬇️

... and 7 files with indirect coverage changes

📣 Codecov offers a browser extension for seamless coverage viewing on GitHub. Try it in Chrome or Firefox today!