How to terminate hanging runs after dagster restart

[hbruch]

When redeploying dagster (using dagster community edition, running via docker compose), we encounter hanging runs which were started before dagster shutdown and apparently could not be (marked) terminated on shutdown.

I'm trying to cancel such hanging runs right after startup, by retrieving all running tasks and calling DagsterGraphQLClient().terminate_runs(run_ids) as soon as dagster is available at port 3000. However, this fails for all tasks without a helpful cause message:

dagster_graphql.client.utils.DagsterGraphQLClientError: ('TerminateRunsError', "All run terminations failed: [
  ('TerminateRunFailure', 'Unable to terminate run dc5a4b65-3587-4798-a509-32d757ba9b37'), 
  ('TerminateRunFailure', 'Unable to terminate run 5790b062-295e-4916-b805-74dc9f69d106'), 
  ('TerminateRunFailure', 'Unable to terminate run 8a584806-2047-4078-a2b0-97060f3b5330'), 
  ('TerminateRunFailure', 'Run fe9dbbf2-770a-46f1-bc36-48f2a0ea54d2 could not be terminated due to having status STARTING.'), 
  ('TerminateRunFailure', 'Unable to terminate run 1bf0cb61-097a-4e20-b1b0-32826c81aad9')]")

In the UI such a task is marked RUN_CANCELING afterwards:

The UI allows to force a termination sending a GraphQL mutation with "terminatePolicy":"MARK_AS_CANCELED_IMMEDIATELY", while DagsterGraphQLClient.terminate_runs does not provide this option.

Do you have any recommendations
a) to avoid hanging jobs on shutdown in the first place
b) run a cleanup script on startup in a kind of hook after the API is available but before any new runs are started

In any case, I think it would be helpful if DagsterGraphQLClient.terminate_run(s) would support an optional terminatePolicy parameter.