New heartbeat automation marks running flow as Crashed #16459
Labels
bug
Something isn't working
Comments
|
Thanks for the bug report @bkempe! It looks like there are several heartbeat events clustered together in your log. Did you have multiple flow runs or running at the same time? |
|
@desertaxle Sorry, something went wrong the first time I copy&pasted the event log. I fixed it above. Do any events still look clustered? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug summary
I've set up the new automation to detect and respond to zombie flows.
However, the automation marked a running flow run as Crashed despite a recent heartbeat event.
The flow run continued to run, sent more heartbeat events, and got into Complete state.
Version info
Additional context
PREFECT_RUNNER_HEARTBEAT_FREQUENCYset to31.Automation trigger:
{ "type": "event", "match": { "prefect.resource.id": [ "prefect.flow-run.*" ] }, "match_related": {}, "after": [ "prefect.flow-run.heartbeat" ], "expect": [ "prefect.flow-run.Crashed", "prefect.flow-run.Failed", "prefect.flow-run.heartbeat", "prefect.flow-run.Completed", "prefect.flow-run.Cancelled" ], "for_each": [ "prefect.resource.id" ], "posture": "Proactive", "threshold": 1, "within": 90 }Logs
Details for event 06:43:52 PM prefect-cloud.automation.action.triggered:
{ "id": "c72e5b22-fc0b-4eca-9704-4fa53b118b6d", "account": "f230f7a2-7128-44f5-ba98-3ac534612f5d", "event": "prefect-cloud.automation.action.triggered", "occurred": "2024-12-19T17:43:53.962Z", "payload": { "action_index": 0, "action_type": "change-flow-run-state", "invocation": "32e4bae6-19a4-4977-8fe8-206b4de2fd36" }, "received": "2024-12-19T17:43:54.203Z", "related": [ { "prefect.resource.id": "prefect.flow-run.117fa91f-8d45-4019-815b-32c0221713c3", "prefect.resource.role": "target" } ], "resource": { "prefect.resource.id": "prefect-cloud.automation.905058bc-c58b-4659-b04b-6885a0567043", "prefect.resource.name": "Crash zombie flows", "prefect-cloud.trigger-type": "event", "prefect-cloud.posture": "Proactive" }, "workspace": "d212164e-16e9-4fd0-9fce-34f9fb0f953d" }Details for event 06:43:54 PM prefect-cloud.automation.action.failed:
{ "id": "d2be71dc-c61a-49cd-baa1-dde7c2456d77", "account": "f230f7a2-7128-44f5-ba98-3ac534612f5d", "event": "prefect-cloud.automation.action.failed", "occurred": "2024-12-19T17:43:54.264Z", "payload": { "action_index": 0, "action_type": "change-flow-run-state", "invocation": "14560e8a-090c-4533-8ac1-2faf42378353", "reason": "Failed to set state: Run is already in terminal state CRASHED.", "status_code": 200 }, "received": "2024-12-19T17:43:54.264Z", "related": [ { "prefect.resource.id": "prefect.flow-run.117fa91f-8d45-4019-815b-32c0221713c3", "prefect.resource.role": "target" } ], "resource": { "prefect.resource.id": "prefect-cloud.automation.905058bc-c58b-4659-b04b-6885a0567043", "prefect.resource.name": "Crash zombie flows", "prefect-cloud.trigger-type": "event", "prefect-cloud.posture": "Proactive" }, "workspace": "d212164e-16e9-4fd0-9fce-34f9fb0f953d" }The text was updated successfully, but these errors were encountered: