Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add individual by_range sync requests #6497

Merged
merged 9 commits into from
Feb 5, 2025
Merged

Conversation

dapplion
Copy link
Collaborator

Issue Addressed

Part of

To address PeerDAS sync issues we need to make individual by_range requests within a batch retriable. We should adopt the same pattern for lookup sync where each request (block/blobs/columns) is tracked individually within a "meta" request that group them all and handles retry logic.

Proposed Changes

second step is to add individual request accumulators for blocks_by_range, blobs_by_range, and data_columns_by_range. This will allow each request to progress independently and be retried separately.

Most of the logic is just piping, excuse the large diff. This PR does not change the logic of how requests are handled or retried. This will be done in a future PR changing the logic of RangeBlockComponentsRequest.

Before

  • Sync manager receives block with SyncRequestId::RangeBlockAndBlobs
  • Insert block into SyncNetworkContext::range_block_components_requests
  • (If received stream terminators of all requests)
    • Return Vec<RpcBlock>, and insert into range_sync

Now

  • Sync manager receives block with SyncRequestId::RangeBlockAndBlobs
  • Insert block into SyncNetworkContext:: blocks_by_range_requests
  • (If received stream terminator of this request)
    • Return Vec<SignedBlock>, and insert into SyncNetworkContext::components_by_range_requests
    • (If received a result for all requests)
      • Return Vec<RpcBlock>, and insert into range_sync

@dapplion dapplion added ready-for-review The code is ready for review syncing labels Oct 16, 2024
Copy link
Member

@realbigsean realbigsean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really good overall!

beacon_node/network/src/sync/network_context.rs Outdated Show resolved Hide resolved
"epoch" => epoch,
"peer" => %peer_id,
);
let _blocks_req_id = self.send_blocks_by_range_request(peer_id, request.clone(), id)?;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The _blocks_req_id here is unused and blobs_req_id is only used to check if a blobs request was made. I don't see these send_ methods used anywhere else so I think we can return Result<(), ..> rather than Result<BlocksByRangeRequestId, ..> for blocks. Maybe a bool for blobs. Looks like for custody columns, we care about the count requested but not particular ids

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In a future PR RangeBlockComponentsRequest will track individual requests by ID. Left this variable named but unused as a hint that it should be used.

beacon_node/network/src/sync/network_context.rs Outdated Show resolved Hide resolved
beacon_node/network/src/sync/network_context.rs Outdated Show resolved Hide resolved
Base automatically changed from sync-active-request-generalize to unstable October 17, 2024 18:14
@dapplion dapplion force-pushed the sync-active-request-byrange branch from df1a6c7 to 5fa2912 Compare October 18, 2024 14:28
@jimmygchen jimmygchen self-assigned this Oct 24, 2024
@dapplion dapplion requested a review from realbigsean November 18, 2024 08:41
@dapplion dapplion force-pushed the sync-active-request-byrange branch from 5fa2912 to a2d1d69 Compare January 13, 2025 11:02
@dapplion dapplion requested a review from jxs as a code owner January 13, 2025 11:02
@dapplion dapplion force-pushed the sync-active-request-byrange branch 2 times, most recently from afe72d8 to 7f71079 Compare January 13, 2025 11:18
@dapplion dapplion requested a review from jimmygchen January 13, 2025 11:19
@dapplion
Copy link
Collaborator Author

Rebased on latest unstable, @jimmygchen ready for another review

@dapplion dapplion force-pushed the sync-active-request-byrange branch from 7f71079 to 9f85064 Compare January 17, 2025 03:09
let Entry::Occupied(mut entry) = self.range_block_components_requests.entry(request_id)
else {
id: ComponentsByRangeRequestId,
block_or_blob: RangeBlockComponent<T::EthSpec>,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we rename this to block_component or range_block_component?

Copy link
Collaborator Author

@dapplion dapplion Jan 25, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah! 000bc5512

Copy link
Member

@jimmygchen jimmygchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

The id naming was a bit confusing to me at first, but overall the logic is actually simpler than I thought 👍

@jimmygchen jimmygchen added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Jan 22, 2025
@dapplion dapplion force-pushed the sync-active-request-byrange branch from 9f85064 to d72b61d Compare January 25, 2025 10:12
@dapplion dapplion requested a review from jimmygchen January 25, 2025 10:14
@dapplion dapplion added ready-for-review The code is ready for review and removed waiting-on-author The reviewer has suggested changes and awaits thier implementation. labels Jan 25, 2025
Copy link
Member

@realbigsean realbigsean left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, couldn't find any issues

Copy link
Member

@jimmygchen jimmygchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks

@jimmygchen jimmygchen added ready-for-merge This PR is ready to merge. and removed ready-for-review The code is ready for review labels Feb 5, 2025
mergify bot added a commit that referenced this pull request Feb 5, 2025
@mergify mergify bot merged commit 2193f6a into unstable Feb 5, 2025
31 checks passed
@mergify mergify bot deleted the sync-active-request-byrange branch February 5, 2025 07:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ready-for-merge This PR is ready to merge. syncing
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants