I got an error when running in_bulk

[manbujin]

Hello, I have the following problem when I run in_bulk, I would like to ask how to solve it,.Thank you very much for your help.

2022-09-21 18:44:34,389 - INFO : info:check_all_finished: False
2022-09-21 18:44:34,390 - INFO : job: 7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 unsubmitted; submit it
2022-09-21 18:44:34,410 - INFO : job: 7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 submit; job_id is 33898
2022-09-21 18:45:05,517 - INFO : job: 7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 33898 terminated;fail_cout is 1; resubmitting job
2022-09-21 18:45:05,541 - INFO : job:7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 re-submit after terminated; new job_id is 33911
2022-09-21 18:45:05,613 - INFO : job:7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 job_id:33911 after re-submitting; the state now is 4
2022-09-21 18:45:35,717 - INFO : job: 7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 33911 terminated;fail_cout is 2; resubmitting job
2022-09-21 18:45:35,739 - INFO : job:7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 re-submit after terminated; new job_id is 33934
2022-09-21 18:45:35,815 - INFO : job:7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 job_id:33934 after re-submitting; the state now is 4
2022-09-21 18:46:05,919 - INFO : job: 7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 33934 terminated;fail_cout is 3; resubmitting job
Traceback (most recent call last):
File "/root/data/dpgen/lib/python3.8/site-packages/dpdispatcher/submission.py", line 215, in handle_unexpected_submission_state
job.handle_unexpected_job_state()
File "/root/data/dpgen/lib/python3.8/site-packages/dpdispatcher/submission.py", line 532, in handle_unexpected_job_state
raise RuntimeError(f"job:{self.job_hash} {self.job_id} failed {self.fail_count} times.job_detail:{self}")
RuntimeError: job:7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43 33934 failed 3 times.job_detail:{'7648d4bdfa62b5f62c3b9ea3794a2c7a97215c43': {'job_task_list': [{'command': 'mpirun -n 4 vasplws', 'task_work_path': 'sys-0108-0108', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR'], 'backward_files': ['OUTCAR', 'CONTCAR'], 'outlog': 'fp.log', 'errlog': 'fp.log'}], 'resources': {'number_node': 1, 'cpu_per_node': 4, 'gpu_per_node': 0, 'queue_name': 'CPU', 'group_size': 125, 'custom_flags': [], 'strategy': {'if_cuda_multi_devices': False}, 'para_deg': 1, 'module_unload_list': [], 'module_list': [], 'source_list': [], 'envs': {}, 'kwargs': {'gpu_usage': False}}, 'job_state': <JobStatus.terminated: 4>, 'job_id': 33934, 'fail_count': 3}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/root/data/dpgen/bin/dpgen", line 10, in
sys.exit(main())
File "/root/data/dpgen/lib/python3.8/site-packages/dpgen/main.py", line 175, in main
args.func(args)
File "/root/data/dpgen/lib/python3.8/site-packages/dpgen/data/gen.py", line 778, in gen_init_bulk
run_vasp_relax(jdata, mdata)
File "/root/data/dpgen/lib/python3.8/site-packages/dpgen/data/gen.py", line 637, in run_vasp_relax
submission.run_submission()
File "/root/data/dpgen/lib/python3.8/site-packages/dpdispatcher/submission.py", line 182, in run_submission
self.handle_unexpected_submission_state()
File "/root/data/dpgen/lib/python3.8/site-packages/dpdispatcher/submission.py", line 219, in handle_unexpected_submission_state
f"Meet errors will handle unexpected submission state.\n"
AttributeError: 'Submission' object has no attribute 'remote_root'

[manbujin]

After I modified the machine.json file, the problem became like this:
Traceback (most recent call last):
File "/root/data/dpgen/bin/dpgen", line 10, in
sys.exit(main())
File "/root/data/dpgen/lib/python3.8/site-packages/dpgen/main.py", line 175, in main
args.func(args)
File "/root/data/dpgen/lib/python3.8/site-packages/dpgen/data/gen.py", line 724, in gen_init_bulk
mdata = convert_mdata(mdata, ["fp"])
File "/root/data/dpgen/lib/python3.8/site-packages/dpgen/remote/decide_machine.py", line 39, in convert_mdata
for key, item in mdata[task_type][0].items():
KeyError: 0

[AnguseZhang]

You're suggested discuss this problem in https://github.com/deepmodeling/dpgen/discussions. At the same time, please provide your software version, input files as clear as possible to help locate the problem.

[manbujin]

Thanks for your reply, I will pay attention to ask the question to the correct place next time.
DeePMD-kit v2.0.4.dev1+g96e623d5
dpgen 0.10.2
numpy 1.20.3
dpdata 0.2.1
pymatgen 2021.3.5
monty 2021.8.17
ase 3.22.0
paramiko 2.7.2
custodian 2022.5.26
python 3.9.13
machine.json
{
"fp": {
"machine": {
"batch_type": "Shell",
"context_type": "local",
"_remote_profile": {
"_hostname": "xxx.xxx.xxx.xxx",
"_port": 22,
"_username": "USERNAME"
},
"remote_root": "/root/data/rootlwy/vasp/tic",
"local_root": "./"
},
"resources": {
"source_list": ["/opt/intel/bin/compilervars.sh intel64",
"/opt/intel/impi/2019.0.117/intel64/bin/mpivars.sh intel64"],
"_module_list": [
"mpich/3.2.1-intel-2017.1"
],
"batch_type": "Shell",
"group_size": 1,
"cpu_per_node": 8,
"gpu_per_node": 0,
"_queue_name": "C032M0128G",
"custom_flags": [
"#SBATCH -t 120:0:0"
],
"_kwargs": {},
"number_node": 1
},
"command": "mpirun -np 4 vasp_std"
},
"api_version": "1.0"
}
param.json
{
"stages" : [1,2,3,4],
"elements": ["Ti","C"],
"cell_type": "diamond",
"latt": 10.0,
"super_cell": [1, 1, 1],
"from_poscar": true,
"from_poscar_path": "./TiC.POSCAR",
"potcars": ["./POTCAR_Ti","./POTCAR_C"],
"relax_incar": "./INCAR_rlx",
"md_incar" : "./INCAR_md",
"skip_relax": false,
"scale": [1.00],
"pert_numb": 10,
"pert_box": 0.03,
"pert_atom": 0.01,
"md_nstep" : 10,
"coll_ndata": 5000,
"type_map" : ["Ti","C"],
"_comment": "that's all"
}

[manbujin]

This prompt appeared in dpgen.log

• INFO : GromacsWrapper>=0.8.0 is needed for DP-GEN + Gromacs.

[HuangJiameng]

What is the version of dpdispatcher?

[HuangJiameng]

Though only "fp" part is used in init, "train" and "model_devi" should also exist in machine.json. Otherwise error will be raised. You can refer to https://github.com/deepmodeling/dpgen/blob/master/examples/machine/DeePMD-kit-2.x/lebesgue_v2_machine.json. DPGUI is also recommended to generate a machine.json.

[manbujin]

Thank you very much for your suggestion, but after I changed machine.json, the problem still occurs.
{
"api_version": "1.0",
"deepmd_version": "2.0.4",
"train" :
{
"command": "dp",
"machine": {
"batch_type": "Shell",
"context_type": "local",
"local_root" : "./",
"_remote_profile":{
"_email": "your lebesgue register email",
"_password": "your lebesgue password",
"_program_id": 0,
"_input_data":{
"_api_version":2,
"_job_type": "indicate",
"_log_file": "00*/train.log",
"_grouped":true,
"_job_name": "dpgen_train_job",
"_disk_size": 100,
"_scass_type":"c8_m60_1 * NVIDIA P100",
"_platform": "ali",
"_image_name":"deepmd-kit:2.0.1",
"_on_demand":0
}
}
},
"resources": {
"number_node": 1,
"local_root":"./",
"cpu_per_node": 4,
"gpu_per_node": 0,
"queue_name": "CPU",
"group_size": 1
}
},
"model_devi":
{
"command": "lmp -i input.lammps -v restart 0",
"machine": {
"batch_type": "Shell",
"context_type": "local",
"_local_root" : "./",
"_remote_profile":{
"_email": "your lebesgue register email",
"_password": "your lebesgue password",
"_program_id": 0,
"_input_data":{
"_api_version":2,
"_job_type": "indicate",
"_log_file": "/model_devi.log",
"_grouped":true,
"_job_name": "dpgen_model_devi_job",
"_disk_size": 200,
"_scass_type":"c8_m60_1 * NVIDIA P100",
"_platform": "ali",
"_image_name":"deepmd-kit:2.0.1",
"_on_demand":0
}
}
},
"resources": {
"number_node": 1,
"local_root":"./",
"cpu_per_node": 4,
"gpu_per_node": 0,
"queue_name": "CPU",
"group_size": 5
}
},
"fp":
{
"command": "mpirun -n 16 vasp_std",
"machine": {
"batch_type": "Shell",
"context_type": "local",
"local_root" : "./",
"_remote_profile":{
"_email": "your lebesgue register email",
"_password": "your lebesgue password",
"_program_id": 0,
"_input_data":{
"_api_version":2,
"_job_type": "indicate",
"_log_file": "task/fp.log",
"_grouped":true,
"_job_name": "dpgen_fp_job",
"_disk_size": 100,
"_scass_type":"c8_m8_cpu",
"_platform": "ali",
"_image_name":"vasp:5.4.4",
"_on_demand":0
}
}
},
"resources": {
"number_node": 1,
"cpu_per_node": 32,
"gpu_per_node": 0,
"queue_name": "CPU",
"group_size": 5,
"local_root":"./",
"source_list": [ "/opt/intel/bin/compilervars.sh intel64",
"/opt/intel/impi/2019.0.117/intel64/bin/mpivars.sh intel64"]
}
}
}

Description

Traceback (most recent call last):
File "/root/data/dpgen/bin/dpgen", line 10, in
sys.exit(main())
File "/root/data/dpgen/lib/python3.8/site-packages/dpgen/main.py", line 175, in main
args.func(args)
File "/root/data/dpgen/lib/python3.8/site-packages/dpgen/data/gen.py", line 724, in gen_init_bulk
mdata = convert_mdata(mdata, ["fp"])
File "/root/data/dpgen/lib/python3.8/site-packages/dpgen/remote/decide_machine.py", line 39, in convert_mdata
for key, item in mdata[task_type][0].items():
KeyError: 0

[manbujin]

After installation, it shows that several dependencies are not found. After I installed it with pip or conda, it did not show. Do I need to put the installed package in the dpgen directory? I don't know if it failed because of the lack of the package.
Dependency

 numpy     1.20.3   /root/data/dpgen/lib/python3.8/site-packages/numpy
dpdata      0.2.1   /root/data/dpgen/lib/python3.8/site-packages/dpdata

pymatgen unknown version or path
monty 2021.8.17 /root/data/dpgen/lib/python3.8/site-packages/monty
ase 3.22.0 /root/data/dpgen/lib/python3.8/site-packages/ase
paramiko 2.7.2 /root/data/dpgen/lib/python3.8/site-packages/paramiko
custodian Not Found

[HuangJiameng]

I tried to reproduce this problem with dpgen 0.10.6, but it did not occur. Could you update dpgen?

[manbujin]

I've upgraded to version 0.10.6, but a new problem has occurred, it doesn't seem to be running.
Show “ nohup: ignoring input “in err file
Show" 2022-09-23 16:07:32,672 - INFO : info:check_all_finished: False
2022-09-23 16:07:32,679 - INFO : job: a451c7733205b41135efbdc92d432482724589aa submit; job_id is 53360" in dpdispatcher.log

dpgen 0.10.6
DeePMD-kit v2.1.1
numpy 1.22.3 /root/data/anaconda3/envs/deepmd/lib/python3.10/site-packages/numpy
dpdata 0.2.8 /root/.local/lib/python3.10/site-packages/dpdata
pymatgen unknown version or path
monty 2022.9.9 /root/.local/lib/python3.10/site-packages/monty
ase 3.22.1 /root/.local/lib/python3.10/site-packages/ase
paramiko 2.11.0 /root/.local/lib/python3.10/site-packages/paramiko
custodian 2022.5.26
python 3.10.4
local+CPU
this is my machine.json
{
"api_version": "1.0",
"deepmd_version": "2.1.1",
"train" :
{
"command": "dp",
"machine": {
"batch_type": "Shell",
"context_type": "local",
"local_root" : "./",
"remote_root": "/root/data/rootlwy/vasp/ceshi/1",
"_remote_profile":{
"_email": "your lebesgue register email",
"_password": "your lebesgue password",
"_program_id": 0,
"_input_data":{
"_api_version":2,
"_job_type": "indicate",
"_log_file": "00*/train.log",
"_grouped":true,
"_job_name": "dpgen_train_job",
"_disk_size": 100,
"_scass_type":"c8_m60_1 * NVIDIA P100",
"_platform": "ali",
"_image_name":"deepmd-kit:2.0.1",
"_on_demand":0
}
}
},
"resources": {
"number_node": 1,
"local_root":"./",
"cpu_per_node": 4,
"gpu_per_node": 0,
"queue_name": "CPU",
"group_size": 1
}
},
"model_devi":
{
"command": "lmp -i input.lammps -v restart 0",
"machine": {
"batch_type": "Shell",
"context_type": "local",
"_local_root" : "./",
"remote_root": "/root/data/rootlwy/vasp/ceshi/1",
"_remote_profile":{
"_email": "your lebesgue register email",
"_password": "your lebesgue password",
"_program_id": 0,
"_input_data":{
"_api_version":2,
"_job_type": "indicate",
"_log_file": "/model_devi.log",
"_grouped":true,
"_job_name": "dpgen_model_devi_job",
"_disk_size": 200,
"_scass_type":"c8_m60_1 * NVIDIA P100",
"_platform": "ali",
"_image_name":"deepmd-kit:2.0.1",
"_on_demand":0
}
}
},
"resources": {
"number_node": 1,
"local_root":"./",
"cpu_per_node": 4,
"gpu_per_node": 0,
"queue_name": "CPU",
"group_size": 5
}
},
"fp":
{
"command": "mpirun -n 16 vasp_std",
"machine": {
"batch_type": "Shell",
"context_type": "local",
"local_root" : "./",
"remote_root": "/root/data/rootlwy/vasp/ceshi/1",
"_remote_profile":{
"_email": "your lebesgue register email",
"_password": "your lebesgue password",
"_program_id": 0,
"_input_data":{
"_api_version":2,
"_job_type": "indicate",
"_log_file": "task/fp.log",
"_grouped":true,
"_job_name": "dpgen_fp_job",
"_disk_size": 100,
"_scass_type":"c8_m8_cpu",
"_platform": "ali",
"_image_name":"vasp:5.4.4",
"_on_demand":0
}
}
},
"resources": {
"number_node": 1,
"cpu_per_node": 32,
"gpu_per_node": 0,
"queue_name": "CPU",
"group_size": 5,
"local_root":"./",
"source_list": [ "/opt/intel/bin/compilervars.sh intel64",
"/opt/intel/impi/2019.0.117/intel64/bin/mpivars.sh intel64"]
}
}
}

[HuangJiameng]

2022-09-23 16:07:32,679 - INFO : job: a451c7733205b41135efbdc92d432482724589aa submit; job_id is 53360" in dpdispatcher.log It seems that it is running normally? So what is the error message?

[manbujin]

I'm running the LiSi example and it's pretty simple, but it keeps getting stuck at 00.place_ele.
err: nohup: ignoring input

[HuangJiameng]

err: nohup: ignoring input has nothing wrong. It will occur whenever you use nohup.

[manbujin]

Oh！Thank you very much for your help, it has worked successfully