Failure in setting up machine.json for multiple GPU training

[shuaijiang-ustc]

Hi, dear all,
I installed dpgen v0.10.3 and dpmd 2.1.0 and have more than four 3090 GPUs available on one node, so I would like to use four GPUs or more to train four models simultaneously, but it didn't work. There are four python PIDs running on one GPU only instead of four GPUs.
Here are parts of machine.json related to GPU settings:
"train_resources": {
"numb_node": 1,
"numb_gpu": 4,
"task_per_node": 12,
"manual_cuda_devices": 4,
"manual_cuda_multiplicity":1,
"cuda_multi_task": true,
"group_size": 4
},
Here are the messages about GPU running:

Thanks!

[AnguseZhang]

Could you please provide the complete machine.json and the training scripts (xxx.sub) generated by DP-GEN ?

[shuaijiang-ustc]

Sure.
Here is machine.json:
{
"_comment": "training on localhost ",
"python_path": "/public/jiangshuai/software/dpmd_210_gpu/bin/python",
"train_machine": {
"batch": "shell",
"_comment" : "If lazy_local is true, calculations are done directly in current folders.",
"lazy_local" : true
},
"train_resources": {
"numb_node": 1,
"numb_gpu": 4,
"task_per_node": 12,
"manual_cuda_devices": 4,
"manual_cuda_multiplicity":1,
"cuda_multi_task": true,
"group_size": 4
},

"_comment":		"model_devi on localhost ",
"model_devi_command":	"/public/jiangshuai/software/dpmd_210_gpu/bin/lmp",
"model_devi_group_size": 5,
"model_devi_machine":	{
"batch":	"shell",
"_comment" : "If lazy_local is true, calculations are done directly in current folders.",
"lazy_local" : true
},	
"model_devi_resources":	{

"numb_node": 1,
"task_per_node": 5,
"with_mpi": false,
"module_list" : ["mpirun"],
"_comment" : "that's all"
},

"_comment":		"fp on localhost ",
"fp_command":	"g16 < input",
"fp_group_size":	50,
"fp_machine":	{
"batch":	"shell",
"_comment" : "If lazy_local is true, calculations are done directly in current folders.",
"lazy_local" : true
},	
"fp_resources":	{
"module_list":  [],

"numb_node": 1,
"task_per_node": 8,
"with_mpi": false,
"_comment": "that's all"
},

"_comment":		" that's all "

}

Here are the sub file for the first model:
#!/bin/bash

cd 000
test $? -ne 0 && exit 1

export CUDA_VISIBLE_DEVICES=0
{ if [ ! -f tag_0_finished ] ;then
/bin/sh -c '{ if [ ! -f model.ckpt.index ]; then dp train input.json; else dp train input.json --restart model.ckpt; fi }' 1>> train.log 2>> train.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }&

cd /home/jiangshuai/ML/TensorMol/TensorMol/structures_sampl/dpgen/test_dpgen_dpmd_multiple_GPUs/iter.000000/00.train
test $? -ne 0 && exit 1

wait

cd 000
test $? -ne 0 && exit 1

export CUDA_VISIBLE_DEVICES=1
{ if [ ! -f tag_1_finished ] ;then
dp freeze 1>> train.log 2>> train.log
if test $? -ne 0; then exit 1; else touch tag_1_finished; fi
fi }&

cd /home/jiangshuai/ML/TensorMol/TensorMol/structures_sampl/dpgen/test_dpgen_dpmd_multiple_GPUs/iter.000000/00.train
test $? -ne 0 && exit 1

wait

touch 29f2dc03-9281-4da1-b6a3-89a96c37b3e6_tag_finished

Other sub files are basically the same except the directory and file name. For instance, here is the sub file for the second model:
#!/bin/bash

cd 001
test $? -ne 0 && exit 1

export CUDA_VISIBLE_DEVICES=0
{ if [ ! -f tag_0_finished ] ;then
/bin/sh -c '{ if [ ! -f model.ckpt.index ]; then dp train input.json; else dp train input.json --restart model.ckpt; fi }' 1>> train.log 2>> train.log
if test $? -ne 0; then exit 1; else touch tag_0_finished; fi
fi }&

cd /home/jiangshuai/ML/TensorMol/TensorMol/structures_sampl/dpgen/test_dpgen_dpmd_multiple_GPUs/iter.000000/00.train
test $? -ne 0 && exit 1

wait

cd 001
test $? -ne 0 && exit 1

export CUDA_VISIBLE_DEVICES=1
{ if [ ! -f tag_1_finished ] ;then
dp freeze 1>> train.log 2>> train.log
if test $? -ne 0; then exit 1; else touch tag_1_finished; fi
fi }&

cd /home/jiangshuai/ML/TensorMol/TensorMol/structures_sampl/dpgen/test_dpgen_dpmd_multiple_GPUs/iter.000000/00.train
test $? -ne 0 && exit 1

wait

touch 6d8ab71b-0b71-4f3a-8879-cfe313a1ea3b_tag_finished

[shuaijiang-ustc]

Yes, as you said, now it's working! thank you so much!
Here is the correct training part in machine.json for other to refer to:
"train": [
{
"machine": {
"batch": "shell",
"_comment" : "If lazy_local is true, calculations are done directly in current folders.",
"lazy_local" : true
},
"resources": {
"numb_node": 1,
"numb_gpu": 4,
"task_per_node": 12,
"manual_cuda_devices": 4,
"manual_cuda_multiplicity":1,
"cuda_multi_task": true,
"train_group_size": 4
},
"group_size": 4
}
],

[AnguseZhang]

Still, I suggest configure a batch system, like Slurm, PBS, etc. to help you monitor the machine resource allocations. Though this may take some time.

[shuaijiang-ustc]

Indeed, thanks again!

[llsnnn]

Do not just replace "group_size", you should move "group_size" out of "train_resources". Please verify this. Or you can provide the modifiedmachine.json.

but in my computer，if i use the key of “train_group_size”，it will give me a error information：key "group size" is required in arguments but not found.

[th-nic]

Please move "group_size" out of "train_resources". Specify "train_group_size" : 4 . If you want to train four models simultaneously, one script is expected to be generated.

Please provide a similar solution for DPGEN v 0.12.1? Although I specify "gpu_per_node": 2, in the training resources section, only one GPU is utilized.