Understanding resources

[SultanOrazbayev]

I am trying to understand the correct use of resources (https://distributed.dask.org/en/latest/resources.html) with custom priorities.

My workflow involves tasks that have somewhat unpredictable memory requirements (multiplication of very large sparse matrices) and to avoid worker crashes I want to restrict the number of tasks per worker. This can be achieved by specifying some arbitrary resource constraint foo and requesting foo for each task.

from dask.distributed import Client, LocalCluster
cluster = LocalCluster(n_workers=1, resources={'foo': 1})
client = Client(cluster)

def test_func(i):
    import time
    time.sleep(1+i)
    return i

# executed in the order of submission, one task at a time
futures = [client.submit(test_func, i, resources={'foo': 1}) for i in range(5)]

Now, further in the pipeline, results of this task (task A) are consumed by other tasks (task B) which have lower memory constraints (element-wise multiplication of sparse matrices), so to reduce the overall memory burden I want to give priority to task B. This can be achieved by specifying the priority of each task with priority (higher value = higher priority).

from dask.distributed import Client, LocalCluster
cluster = LocalCluster(n_workers=1, resources={'foo': 1})
client = Client(cluster)

def test_func(i):
    import time
    time.sleep(1+i)
    return i

# executed in the order of priority, last task first
futures = [client.submit(test_func, i, priority=i) for i in range(5)]

Combining the resources and task priority does not seem to work for me. And I am not sure if this is a bug or a feature.

from dask.distributed import Client, LocalCluster
cluster = LocalCluster(n_workers=1, resources={'foo': 1})
client = Client(cluster)

def test_func(i):
    import time
    time.sleep(1+i)
    return i

# executed in the order of submission, ignoring priority
futures = [client.submit(test_func, i, priority=i, resources={'foo': 1}) for i in range(5)]

[phildong]

I have the exact same question, and I have a relevant issue #4491 where if you compare Approach 2 and Approach 3 you can definitely see that the resource annotation affects execution order. It's good to know that this effect cannot be avoided either with priority...