Add "retry" command to spackbot. #60
Conversation
josephsnyder
force-pushed
the
add_pipeline_retry
branch
6 times, most recently
from
12c0441 to
2921e4c
Compare
There was a problem hiding this comment.
Thanks @josephsnyder this is looking pretty good. I think we probably want to debug the sender (instead of info) or else pick out the user name from that dictionary.
I'm curious how you're testing this change, or if it's just too much of a pain. I had to do a lot of gymnastics to test #58, which is why I'm asking.
|
@scottwittenburg, I've been testing this using a combination of the docker-compose system and a non-spack repository (josephsnyder/spack_docs#1) I told my local instance of spackbot to listen on this repository. I haven't gotten one to officially "work" via spackbot yet since the branches there don't match any that would be on SPACK's GitLab but they work when manually hitting the API endpoint for the development GitLab instance I run. |
|
I've added a few new parts: it now grabs all pipelines and wont try to restart a pipeline if the 0th one is a success. It also tries to find the last one of I can't quite test the processing as I still am receiving a |
josephsnyder
force-pushed
the
add_pipeline_retry
branch
from
2921e4c to
7c3020c
Compare
There was a problem hiding this comment.
Your changes querying all pipelines for the branch make sense, thanks @josephsnyder! Just update the comment to reflect that if you don't mind 😉
Also, maybe your attempts to hit the endpoint on the gitlab/spack project failed because you weren't a member of that project? 🤷 I invited you as a maintainer so you could try again.
In general, without deploying this change for real (or fake, as I did with #58) in the cluster, we probably can't hope to test everything. So I'm fine with you just testing the parts you can piecemeal, and then ironing out any issues once we deploy.
spackbot/handlers/pipelines.py
Outdated
| async with session.post(url, headers=headers) as response: | ||
| result = await response.json() | ||
| else: | ||
| # Get a list of failed pipelines |
There was a problem hiding this comment.
| # Get a list of failed pipelines | |
| # Get a list of all pipelines |
Utilize the "retry" API endpoint for pipelines to re-run the jobs that had failed during the most recent failed run of a pipeline. It does require a new comment to listen to and a call to acquire the ID of the most recently failed pipeline. Use the PR ref to capture the pipelines for that specific instance.
josephsnyder
force-pushed
the
add_pipeline_retry
branch
from
7c3020c to
94740c2
Compare
|
Not sure/don't recall why this hasn't been merged but if it is active it needs resolution of the conflicts. |
|
This is a year and half old, and I don't really recall why we wanted it. Right now it seems to me to be kind of risky to expose this functionality since we know that when you use the UI to retry jobs in gitlab, the job runs with the environment variables given to it when the job was created. This effectively means you can't retry jobs created more than 12 hrs ago, or the aws credentials will have expired. Maybe we hadn't implemented credential rotation yet when this was done? I'm not sure. At any rate, I think we should close this unless someone has a good argument otherwise. |
Utilize the "retry" API endpoint for pipelines to re-run the jobs that
had failed during the most recent failed run of a pipeline.
It does require a new comment to listen to and a call to acquire the ID
of the most recently failed pipeline before triggering the jobs to be retried.