Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix treatment of not-found nodes and update provisioning semantics #2677

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

AhmedSoliman
Copy link
Contributor

@AhmedSoliman AhmedSoliman commented Feb 8, 2025

This fixes mishandling of deleted and unknown nodes in the config in f-majority checks. Integration tests were misconfigured where the nodeset was [N2..N4] where N4 didn't actually exist in config. In this case we should not accept f-majority seal if only one node is sealed (replication=2)
Although this bug wouldn't impact us immediately, it's best to fix this condition and I took it as an opportunity to update the semantics of provisioning state to match the latest design direction. Documentation has also been updated to reflect the correct semantics.

Summary:

  • Nodes observed in node-set but not in nodes-config is treated as "provisioning" rather than "disabled"
  • Nodes that are "deleted" in config (tombstone exists) are treated as "disabled"
  • Nodes in provisioning are fully authoritative, but are automatically excluded from new nodesets (already filtered by candidacy filter in nodeset selector)
  • If provisioning nodes were added to the nodeset, they are treated as fully authoritative and are required to participate in f-majority.
// intentionally empty

Stack created with Sapling. Best reviewed with ReviewStack.

Copy link

github-actions bot commented Feb 8, 2025

Test Results

  7 files  ±0    7 suites  ±0   2m 49s ⏱️ - 1m 28s
 45 tests  - 2   44 ✅  - 2  1 💤 ±0  0 ❌ ±0 
174 runs   - 8  171 ✅  - 8  3 💤 ±0  0 ❌ ±0 

Results for commit ecf966a. ± Comparison against base commit 7dc8864.

This pull request removes 2 tests.
dev.restate.sdktesting.tests.AwaitTimeout ‑ timeout(Client)
dev.restate.sdktesting.tests.RawHandler ‑ rawHandler(Client)

♻️ This comment has been updated with latest results.

@AhmedSoliman AhmedSoliman marked this pull request as ready for review February 8, 2025 17:58
This also adds a little bit more stress to the tests to help them fail more often if there is an issue

```
// intentionally empty
```
This fixes mishandling of deleted and unknown nodes in the config in f-majority checks. Integration tests were misconfigured where the nodeset was [N2..N4] where N4 didn't actually exist in config. In this case we should not accept f-majority seal if only one node is sealed (replication=2)
Although this bug wouldn't impact us immediately, it's best to fix this condition and I took it as an opportunity to update the semantics of provisioning state to match the latest design direction. Documentation has also been updated to reflect the correct semantics.

Summary:
- Nodes observed in node-set but not in nodes-config is treated as "provisioning" rather than "disabled"
- Nodes that are "deleted" in config (tombstone exists) are treated as "disabled"
- Nodes in provisioning are fully authoritative, but are automatically excluded from new nodesets (already filtered by candidacy filter in nodeset selector)
- If provisioning nodes were added to the nodeset, they are treated as fully authoritative and are required to participate in f-majority.

```
// intentionally empty
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant