running dpgen on 2 gpu nodes

[poyi81]

Hi everyone,
I want to run dpgen model_devi on 2 GPU at same time. I have questions regarding using "batch_type": "Slurm"
Here is my machine.json

     "model_devi":[
            {
                    "command": "mpirun lmp -i input.lammps",
                    "machine": {
                            "batch_type": "Slurm",
                            "context_type": "local",
                            "local_root" : "./",
                            "remote_root": "./work"
                    },
         "resources": {
                            "number_node": 2,
                            "cpu_per_node": 48,
                            "gpu_per_node": 4,
                            "group_size": 0,
                            "para_deg": 12,
                            "queue_name": "a100",
                            "custom_flags": ["#SBATCH --nodelist=ioc-dl-teslaa[01,03]"],
                            "source_list": ["/old_ceph/sharedfs/users/l/litimmy/deepmd/bin/activate  /old_ceph/sharedfs/users/l/litimmy/deepmd"],
                            "module_list": ["nvhpc_sdk/20.11"],
                            "envs": {"CUDA_VESIBLE_DEVICES":"0,1,2,3","LD_LIBRARY_PATH=$LD_LIBRARY_PATH":"/old_ceph/sharedfs/ioc_backup/groups/davidwu2022/package/hdf5/lib"},
                            "strategy":{
                                    "if_cuda_multi_devices": false
                            },
                            "prepend_script": [],
                            "envs": {
                                    "OMP_NUM_THREADS": 1,
                                    "TF_INTRA_OP_PARALLELISM_THREADS": 1,
                                    "TF_INTER_OP_PARALLELISM_THREADS": 1
                                    }
                    }
            }
    ]

and here is the error i got.

Error: node list format not recognized. Try using '-hosts='.
/bin/sh: line 1: 20501 已經終止 mpirun lmp -i input.lammps -i input.lammps -v restart 0
Error: node list format not recognized. Try using '-hosts='.
/bin/sh: line 1: 22275 已經終止 mpirun lmp -i input.lammps -i input.lammps -v restart 0
Error: node list format not recognized. Try using '-hosts='.
/bin/sh: line 1: 24067 已經終止 (core dumped) mpirun lmp -i input.lammps -i input.lammps -v restart 0

Because different lammps tasks are run parallelly. So I think I might should use slurm to assign half of tasks to node 1 and half of tasks to node2. And I try this command:

     "model_devi":[
            {
                    "command": "mpirun -np 1 -f hostfile.$SLURM_JOB_ID lmp -i input.lammps",
                    "machine": {
                            "batch_type": "Slurm",
                            "context_type": "local",
                            "local_root" : "./",
                            "remote_root": "./work"
                    },
         "resources": {
                            "number_node": 2,
                            "cpu_per_node": 48,
                            "gpu_per_node": 4,
                            "group_size": 0,
                            "para_deg": 12,
                            "queue_name": "a100",
                            "custom_flags": ["#SBATCH --nodelist=ioc-dl-teslaa[01,03]"],
                            "source_list": ["/old_ceph/sharedfs/users/l/litimmy/deepmd/bin/activate  /old_ceph/sharedfs/users/l/litimmy/deepmd"],
                            "module_list": ["nvhpc_sdk/20.11"],
                            "envs": {"CUDA_VESIBLE_DEVICES":"0,1,2,3","LD_LIBRARY_PATH=$LD_LIBRARY_PATH":"/old_ceph/sharedfs/ioc_backup/groups/davidwu2022/package/hdf5/lib"},
                            "strategy":{
                                    "if_cuda_multi_devices": false
                            },
                            "prepend_script": ["scontrol show hostnames $SLURM_JOB_NODELIST > hostfile.$SLURM_JOB_ID"],
                            "envs": {
                                    "OMP_NUM_THREADS": 1,
                                    "TF_INTRA_OP_PARALLELISM_THREADS": 1,
                                    "TF_INTER_OP_PARALLELISM_THREADS": 1
                                    }
                    }
            }
   ]

Here is the errors i got

main (ui/mpich/mpiexec.c:148): error parsing parameters
HYDU_parse_hostfile (utils/args/args.c:319): unable to open host file: hostfile.6247
mfile_fn (ui/mpich/utils.c:336): error parsing hostfile
match_arg (utils/args/args.c:156): match handler returned error
HYDU_parse_array (utils/args/args.c:178): argument matching returned error
parse_args (ui/mpich/utils.c:1642): error parsing input array
HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1694): unable to parse user arguments
main (ui/mpich/mpiexec.c:148): error parsing parameters
HYDU_parse_hostfile (utils/args/args.c:319): unable to open host file: hostfile.6248
mfile_fn (ui/mpich/utils.c:336): error parsing hostfile
match_arg (utils/args/args.c:156): match handler returned error
HYDU_parse_array (utils/args/args.c:178): argument matching returned error
parse_args (ui/mpich/utils.c:1642): error parsing input array
HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1694): unable to parse user arguments
main (ui/mpich/mpiexec.c:148): error parsing parameters

I am wondering if anyone ever uses 2 nodes at same time. Please kindly give me some suggestions. Thank you so much!

[njzjz]

The submission directory and the task directory are different. You create the host file in the submission directory, so when you in the task directory, the absolute path should be used:

$SLURM_SUBMIT_DIR/hostfile.$SLURM_JOB_ID