Add ability in task spec to wait for a lock before proceeding

[m4dcoder]

There are use cases where users want to synchronize access to a resource (i.e. server) across multiple and different workflow executions. Currently, StackStorm offers concurrency policies to synchronize executions for one given workflow.

The following proposal allows specification of the lock requirement across different workflow definition so synchronization is possible against different workflow executions on the same resource(s). In the proposal below, a new wait attribute is introduced to the task spec. The wait attribute takes the name of the lock, which can be a unique name or name of the resource. The task will sleep/delay until the lock is acquired. On exit from the task scope, the lock will be released.

An additional wait delay can be specified. If the lock is not acquired within the wait period, the task will abandon the wait and exit with failure. On failure, the task transition will be traversed as normal so that clean up or retry is possible.

tasks:
  task1:
    wait: <% ctx().resource_lock_name %>
    action: ...

tasks:
  task1:
    wait:
      lock: <% ctx().resource_lock_name %>
      delay: 600
    action: ...

The mechanism will require on external platform such as redis which is currently already being used for various use cases in StackStorm (required for concurrency policies and running complex branching workflows, etc.).

[nzlosh]

Small question, would it make sense to support a list of locks rather than a single lock? Could people want to acquire locks on different resources with different timeouts but still want the executing workflow to depend one or both of them?

[nmaludy]

List of locks could potentially be dangerous and cause deadlock scenarios, in my opinion.

Classic deadlock scenario is:

# Thread A

Lock X
Lock Y

Unlock Y
Unlock X


# Thread B
Lock X
Lock Y

Unlock X
Unlock Y

If locks are not obtained and released in the exact same order in all code paths, you risk the chance of a deadlock scenario. It's much safer to just have one single lock and ensure it is always released in the face of an exception or return from your code path (In this case a workflow success or failure).

[nzlosh]

Fair enough, I was coming from a specific use case where I needed a lock for a group of servers and another lock for individual server within that group when I asked if it made sense to support a list of locks.

If Orquesta would suffer from the same deadlock/livelock issues as OS threads, then I agree it's a dangerous feature to allow.

[m4dcoder]

This is a good point on deadlock. I think for the time being we will keep this simple. The engine should detect that a lock is already acquired and will not allow additional lock requests in subworkflow(s) since that is possible.