Does scheduler mandatory for control job number?

[pee8379]

I am trying to running dpgen in my local desktop machine. (one 12-core cpu, 2 gpu)

As far I figured out, number of task in each job is controlled by "group_size" in machine json file.
However, I keep failing to limit number of job submitted to run.
It is predictable during "train", since "numb_models" is number of process submitted, but during "model_devi", hundreds of new processes try to run in same time, breaks entire system and make practically impossible to run dpgen in my desktop.

I try large "group_size" after read "Note that group_size should be set as large as possible to ensure there is only one job and avoid running multiple jobs at the same time." In doc/examples/shell.md , but it didn't help.

I found some local shell machine examples in this github, (discussions #666 , machine-local-4GPU.json , machine-local.json ) , but all use old dispatcher style.
Does using old dispatcher will solve this problem?

If controlling number of running processes is impossible without scheduler, I considering to learning how to use slurm. This might be most common, easy option, right?

Thanks.

[AnguseZhang]

I think that settinggroup_size to be a large number can solve your problem. It's probable that there's wrong with your machine.json. Could you post your machine.json?

[pee8379]

machine_local_shell_pwscf.txt

Yes, you are right. large group_size can was solution. Mixing old and new dpdispatcher style confuses me.

However, It seems number of POSCAR in sys_configs seems also affect number of submitting jobs.
Is there a equation or other methods to predict number of maximum processes using group_size?

Anyway, I upload working machine.json file for local, single node, shell, pwscf, with newest dpgen/deepmd/dpdispatcher releases. (0.10.5, 2.1.1, 0.4.9)
I hope this help others who need example for this specific configuration.

[njzjz]

Added support to infinite group_size in deepmodeling/dpdispatcher#194.