Avoid admin roles in local cluster runner #2026
Conversation
For the time being we only want to run admin on the metadata node, which will also be allowed to bootstrap.
| @@ -25,7 +25,7 @@ async fn main() { | |||
| let nodes = Node::new_test_nodes_with_metadata( | |||
| base_config, | |||
| BinarySource::CargoTest, | |||
| enum_set!(Role::Admin | Role::Worker), | |||
| enum_set!(Role::Worker), | |||
There was a problem hiding this comment.
Do you think we should have something in the cluster builder to specify which node is admin?
There was a problem hiding this comment.
currently my thinking is that its easier to have one singleton node that has all the singleton roles (metadata, admin), and then the other N nodes can all be more similar. its always possible to specify nodes in whatever setup you like, but the goal of new_test_nodes_with_metadata is to create a list of nodes with some sensibl defaults
There was a problem hiding this comment.
hmm, but i can see now that non-bootstrap nodes will fail to start if there is no nodes config yet, which is a slightly unpleasant race condition. i wonder if indeed the cluster construct needs to know about who is admin and make sure its started and healthy before moving on
There was a problem hiding this comment.
Would a slightly longer timeout help in preventing this situation? I could see someone who is deploying Restate might run into the same situation that some nodes start a bit earlier than others creating a race condition.
There was a problem hiding this comment.
only if your environment can restart the process. currently the non bootstrap nodes will shut down if they reach the metadata service and find no nodes config. this is fine with systemd or kubernetes which will restart on failure. but my local cluster runner does not do this
There was a problem hiding this comment.
We could not immediately fail but wait a bit in order to mitigate the race condition at start-up. Alternatively, all nodes could be started with the bootstrap option, assuming they have identical configuration.
There was a problem hiding this comment.
my current thinking for the runner is to wait for admins to be ready on port 9070 before progressing to other nodes. but i agree, it would be good if non admin nodes would wait a bit instead of bailing
jackkleeman
force-pushed
the
avoid-admin
branch
from
cca482d to
19cfad6
Compare
jackkleeman
force-pushed
the
avoid-admin
branch
from
032c0b5 to
1dcf712
Compare
For the time being we only want to run admin on the metadata node, which will also be allowed to bootstrap.