Replies: 14 comments 24 replies
-
|
Prototype implementation #2339 |
Beta Was this translation helpful? Give feedback.
-
|
For repro support, I understand with our current repro architecture (spin up a node based on the task/pool config) it wouldn't be possible for the reason you mention that the image might not exist in azure. But, an idea was brought up in #2399 of reimplementing the script generation for repros to be on VM rather than server side. By letting the agent generate the scripts, I think we could unblock repro support for unmanaged nodes. It might be out of scope for this first iteration of this feature, but it could be worth calling out as a possibility for future iterations if we decide to prioritize it. |
Beta Was this translation helpful? Give feedback.
-
|
Taking it even a step further, repro could be implemented as a task. |
Beta Was this translation helpful? Give feedback.
-
|
Agreed. This can be revisited if we move the repro script creation on the machine. Making it a task would actually make the integration seamless with the unmanaged nodes. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
|
No it is not possible for an unmanaged node to be assigned to an unmanaged pool. This is already enforced |
Beta Was this translation helpful? Give feedback.
-
|
Right now pool is defined by Hardware and OS In unmanaged case Pool is defined only by OS ? If that is the case - can we have consistent Pool design across Managed/Unmanaged so there are no "special cases" ? |
Beta Was this translation helpful? Give feedback.
-
|
The pool definition is not changing. I am just calling out the additional check we will make when the new node connect to the machine. |
Beta Was this translation helpful? Give feedback.
-
Is there any protection provided to users of unmanaged pool from other users ? Or I am free to add my |
Beta Was this translation helpful? Give feedback.
-
|
No there is no protection currently. We only verify the claim in the request. |
Beta Was this translation helpful? Give feedback.
-
|
I updated the proposal to include the ClientID check |
Beta Was this translation helpful? Give feedback.
-
If I deploy a Mac as unmanaged node, or a linux flavor that OneFuzz agent does not work on, do I build my own ? |
Beta Was this translation helpful? Give feedback.
-
|
Yes. The only download file available will be the one that we currently build and deploy |
Beta Was this translation helpful? Give feedback.
-
|
Can I run several OneFuzz agents on a machine ? Or is it one agent per machine ? |
Beta Was this translation helpful? Give feedback.
-
|
With the current design we are limited to one per machine because the machine identifier is unique per machine. We can change this by including the machine id in the configuration. |
Beta Was this translation helpful? Give feedback.
-
Any protection from other users of OneFuzz to schedule to my unmanaged pool ? or DOS my pool ? |
Beta Was this translation helpful? Give feedback.
-
|
No protection. The pool works the same as the managed one. |
Beta Was this translation helpful? Give feedback.
-
|
What's the task management situation if I trip on a cord and unplug my "unmanaged node" ? Do we fail the task or let it resume when node is back online ? If user wants to delete the node altogether, do they run CLI command to delete the node ? |
Beta Was this translation helpful? Give feedback.
-
|
We will not fail the tsk based on the node state. The task might eventually die because of the timeout. |
Beta Was this translation helpful? Give feedback.
-
|
Can an unmanged node be in several unamanged pools ? |
Beta Was this translation helpful? Give feedback.
-
|
actually the node will only exist under the last pool it was registered to |
Beta Was this translation helpful? Give feedback.
-
|
Having unmanaged node means that moving to file shares becomes impossible (my internet provider does not let me mount azure file share on my machine at home). |
Beta Was this translation helpful? Give feedback.
-
|
There are no restrictions to where the unmanaged node lives. If the node cannot exist locally you can create a VM in azure instead |
Beta Was this translation helpful? Give feedback.
-
|
My point here is that long term plan of moving to file shares might need rethinking because of unmanaged nodes... |
Beta Was this translation helpful? Give feedback.
-
|
Would be nice to have Testing that adds/improves on top of existing testing rather than just using existing tests that are not very good in the first place.... |
Beta Was this translation helpful? Give feedback.
-
|
Can I move a node between unmanaged pools ? Move node from one Pool to another ? |
Beta Was this translation helpful? Give feedback.
-
|
So if I assign a node to a new pool, I have to restart the node ? Is it with CLI ? or do I need to reboot the machine or just restart the agent ? |
Beta Was this translation helpful? Give feedback.
-
|
restart the agent |
Beta Was this translation helpful? Give feedback.
-
|
in this scenario:
After node finishes the task, will it pick up the task from Pool1 ? and keep picking up tasks from Pool1 until I restart the agent ? |
Beta Was this translation helpful? Give feedback.
-
|
The node assignment does not happen on the cli. |
Beta Was this translation helpful? Give feedback.
-
|
makes sense |
Beta Was this translation helpful? Give feedback.
-
|
What's agent upgrade story ? If we release a newer version of agent and it is different in major version number - the agent will be rejected from OneFuzz. Is it up to the user to manually upgrade all agents they have ? If they have 10,000,000 nodes ? |
Beta Was this translation helpful? Give feedback.
-
|
We don't manage the upgrade. The user will be responsible for updating their machines no matter the number |
Beta Was this translation helpful? Give feedback.
-
|
Sounds like if I start a job before I go on vacation and node with version 1 picks up a task, then next day service is upgraded to version 2 (which |
Beta Was this translation helpful? Give feedback.
-
|
Isn't this the situation we have already? We release new versions of the agents with every release. Existing agents run to completion; the node isn't "broken" - at least until a new job needs to run. The difference is in managed nodes we "manage" new agents on the nodes. So, this is a downside of unmanaged nodes, the user has to manage the agent version. Unless we relax enforcement of the version. |
Beta Was this translation helpful? Give feedback.
-
|
We should start with versioning at the API level for the agent. |
Beta Was this translation helpful? Give feedback.
-
|
|
Beta Was this translation helpful? Give feedback.
-
An unmanaged node is an external compute resource created outside of the service and used for scheduling tasks in the OneFuzz.
This new type of node is expected to behave the same as the existing nodes except for the following.
Node management
As the name indicates, the creation and lifetime of these nodes will not be managed by the service and will be the responsibility of the user.
The unmanaged node can only be assigned to an unmanaged pool. It will not be involved in the cleanup process that reimages nodes based on their version or their heartbeat status.
Node Registration
The node will report their OS when registering to the service.
The service will check the OS against the one configured in the pool.
Authentication/Authorization
The nodes will authenticate by using the client credential flow from the machine.
The user will create an application registration and generate a secret for the authentication.
The application registration will be assigned the approle "Unmanaged node" via a helper script.
AAD will add this role as a claim to each authenticated request from the machine.
The service will authorize the request coming from the machine by verifying the presence of the "Unmanaged node" claim.
The service will also check the ClientId of the request against the one configured in the pool
Proxy and Repro
proxy will be disabled because the user already has access to the node and the service has no control over the location of that node
repro will be disabled because the nodes might use a sku or os not available in azure.
Workflow
These are the steps to use an unmanaged node in OneFuzz
Create a new application registration in azure to serve as the authentication
Register that application to the OneFuzz instance to allow the authentication of the nodes (with the helper script)
Create a new unmanaged pool and assign the clientId of the application registration
Download the configuration file to connect to the pool from the OneFuzz instance
Download the agent from the OneFuzz instance
Create a login (client_secret) from the application registration and adds it to the configuration downloaded earlier
Run the agent on a machine with the configuration files
The nodes receive works scheduled to the pool.
Testing
We can leverage our existing functional tests in check-pr by running them on an unmanaged pool.
The process will be as follow:
Note: Since we expect the same behavior from the agent after the registration we can limit our testing to a single test for each os.
Beta Was this translation helpful? Give feedback.
All reactions