dpgen, tutorials running problem

[JasonGyy]

Hello, I'm a rookie in study dpgen. When I ran the tutorial "Hands-on tutorial for DP-GEN (v0.10.3)", some error occured, and cannot find the solution. please give me some help, that would be very thankful. I'm running the tutorial on a centOS+VMware player, and deepmd(including lmp), dpgen and vasp have been installed.

dpgen run param.json machine.json
\NOTE: Some configuration directories are not set up yet:
/root/.gromacswrapper
/root/.gromacswrapper/qscripts
/root/.gromacswrapper/templates
NOTE: You can create the configuration file and directories with:
>>> import gromacs
>>> gromacs.config.setup()
/root/.local/lib/python3.10/site-packages/gromacs/init.py:286: GromacsImportWarning: Some Gromacs commands were NOT found; maybe source GMXRC first? The following are missing:
['release']

warnings.warn("Some Gromacs commands were NOT found; "
DeepModeling

Version: 0.10.6
Date: Sep-12-2022
Path: /root/.local/lib/python3.10/site-packages/dpgen

Dependency

 numpy     1.22.3   /root/anaconda3/envs/deepmd/lib/python3.10/site-packages/numpy
dpdata      0.2.8   /root/anaconda3/envs/deepmd/lib/python3.10/site-packages/dpdata

pymatgen unknown version or path
monty 2022.9.9 /root/anaconda3/envs/deepmd/lib/python3.10/site-packages/monty
ase 3.22.1 /root/.local/lib/python3.10/site-packages/ase
paramiko 2.11.0 /root/.local/lib/python3.10/site-packages/paramiko
custodian 2022.5.26 /root/.local/lib/python3.10/site-packages/custodian

Reference

Please cite:
Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and Weinan E,
DP-GEN: A concurrent learning platform for the generation of reliable deep learning
based potential energy models, Computer Physics Communications, 2020, 107206.

Description

INFO:dpgen:-------------------------iter.000000 task 01--------------------------
2022-09-21 21:10:47,393 - INFO : info:check_all_finished: False
2022-09-21 21:10:47,419 - INFO : job: 38a5a0b3bfb0ec3ce5ad02b4c334e1a1bd5e385f submit; job_id is 3711
2022-09-21 21:10:47,421 - INFO : job: 92dae58cb1c251ea2e4f3365db0af66d6c616cd8 submit; job_id is 3713
2022-09-21 21:10:47,459 - INFO : job: 5d146339791d27d9185aff7670b06529905268f5 submit; job_id is 3721
2022-09-21 21:10:47,465 - INFO : job: df2fc03c3312034f6c844ce119db3c159ef75352 submit; job_id is 3723
2022-09-21 21:13:18,873 - INFO : job: 5d146339791d27d9185aff7670b06529905268f5 3721 finished
2022-09-21 21:13:18,880 - INFO : job: df2fc03c3312034f6c844ce119db3c159ef75352 3723 finished
2022-09-21 21:13:48,927 - INFO : job: 38a5a0b3bfb0ec3ce5ad02b4c334e1a1bd5e385f 3711 finished
2022-09-21 21:13:48,934 - INFO : job: 92dae58cb1c251ea2e4f3365db0af66d6c616cd8 3713 finished
INFO:dpgen:-------------------------iter.000000 task 02--------------------------
INFO:dpgen:-------------------------iter.000000 task 03--------------------------
INFO:dpgen:-------------------------iter.000000 task 04--------------------------
2022-09-21 21:13:49,386 - INFO : info:check_all_finished: False
2022-09-21 21:13:49,393 - INFO : job: 1320d04e42aa87fd4e95c78d64e290ae74aa31f5 submit; job_id is 3939
2022-09-21 21:13:49,395 - INFO : job: e501d8a55c702e407f9a257b70ac4ef882c39c91 submit; job_id is 3941
2022-09-21 21:17:51,895 - INFO : job: 1320d04e42aa87fd4e95c78d64e290ae74aa31f5 3939 finished
2022-09-21 21:17:51,904 - INFO : job: e501d8a55c702e407f9a257b70ac4ef882c39c91 3941 finished
INFO:dpgen:-------------------------iter.000000 task 05--------------------------
INFO:dpgen:-------------------------iter.000000 task 06--------------------------
INFO:dpgen:system 000 candidate : 242 in 310 78.06 %
INFO:dpgen:system 000 failed : 19 in 310 6.13 %
INFO:dpgen:system 000 accurate : 49 in 310 15.81 %
INFO:dpgen:system 000 accurate_ratio: 0.1581 thresholds: 1.0000 and 1.0000 eff. task min and max -1 20 number of fp tasks: 20
INFO:dpgen:-------------------------iter.000000 task 07--------------------------
2022-09-21 21:17:51,953 - INFO : info:check_all_finished: False
2022-09-21 21:17:51,960 - INFO : job: 01543ff0cb9edbba72a6d562494033e87c690a64 submit; job_id is 4667
2022-09-21 21:18:23,017 - INFO : job: 01543ff0cb9edbba72a6d562494033e87c690a64 4667 terminated;fail_cout is 1; resubmitting job
2022-09-21 21:18:23,019 - INFO : job:01543ff0cb9edbba72a6d562494033e87c690a64 re-submit after terminated; new job_id is 4822
2022-09-21 21:18:23,231 - INFO : job:01543ff0cb9edbba72a6d562494033e87c690a64 job_id:4822 after re-submitting; the state now is <JobStatus.running: 3>
2022-09-21 21:18:53,259 - INFO : job: 01543ff0cb9edbba72a6d562494033e87c690a64 4822 terminated;fail_cout is 2; resubmitting job
2022-09-21 21:18:53,261 - INFO : job:01543ff0cb9edbba72a6d562494033e87c690a64 re-submit after terminated; new job_id is 4970
2022-09-21 21:18:53,479 - INFO : job:01543ff0cb9edbba72a6d562494033e87c690a64 job_id:4970 after re-submitting; the state now is <JobStatus.running: 3>
2022-09-21 21:19:23,524 - INFO : job: 01543ff0cb9edbba72a6d562494033e87c690a64 4970 terminated;fail_cout is 3; resubmitting job
Traceback (most recent call last):
File "/root/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 245, in handle_unexpected_submission_state
job.handle_unexpected_job_state()
File "/root/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 609, in handle_unexpected_job_state
raise RuntimeError(f"job:{self.job_hash} {self.job_id} failed {self.fail_count} times.job_detail:{self}")
RuntimeError: job:01543ff0cb9edbba72a6d562494033e87c690a64 4970 failed 3 times.job_detail:{'01543ff0cb9edbba72a6d562494033e87c690a64': {'job_task_list': [{'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000019', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000005', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000014', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000004', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000009', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000013', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000015', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000018', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000006', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000012', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000017', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000010', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000001', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000011', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000002', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000016', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000007', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000008', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000000', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}, {'command': 'mpirun -n 4 vasp_std', 'task_work_path': 'task.000.000003', 'forward_files': ['POSCAR', 'INCAR', 'POTCAR', 'KPOINTS'], 'backward_files': ['fp.log', 'OUTCAR', 'vasprun.xml'], 'outlog': 'fp.log', 'errlog': 'fp.log'}], 'resources': {'number_node': 1, 'cpu_per_node': 4, 'gpu_per_node': 1, 'queue_name': '', 'group_size': 125, 'custom_flags': [], 'strategy': {'if_cuda_multi_devices': False, 'ratio_unfinished': 0.0}, 'para_deg': 1, 'module_purge': False, 'module_unload_list': [], 'module_list': [], 'source_list': [], 'envs': {}, 'wait_time': 0, 'kwargs': {}}, 'job_state': <JobStatus.terminated: 4>, 'job_id': 4970, 'fail_count': 3}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/root/.local/bin/dpgen", line 8, in
sys.exit(main())
File "/root/.local/lib/python3.10/site-packages/dpgen/main.py", line 185, in main
args.func(args)
File "/root/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 3642, in gen_run
run_iter (args.PARAM, args.MACHINE)
File "/root/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 3628, in run_iter
run_fp (ii, jdata, mdata)
File "/root/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 3018, in run_fp
run_fp_inner(iter_index, jdata, mdata, forward_files, backward_files, _vasp_check_fin,
File "/root/.local/lib/python3.10/site-packages/dpgen/generator/run.py", line 2997, in run_fp_inner
submission.run_submission()
File "/root/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 212, in run_submission
self.handle_unexpected_submission_state()
File "/root/.local/lib/python3.10/site-packages/dpdispatcher/submission.py", line 248, in handle_unexpected_submission_state
raise RuntimeError(
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/home/work/DeePMD/dpg_tutorial/dpgen_example/run/new/82ea5da2808459e9d080f8ecfbd5f57031b4a7b5.
Debug information: submission_hash==82ea5da2808459e9d080f8ecfbd5f57031b4a7b5.
Please check the dirs and scripts in remote_root. The job information mentioned above may help.

Contents of dpdispatcher.log

2022-09-21 21:10:47,393 - INFO : info:check_all_finished: False
2022-09-21 21:10:47,419 - INFO : job: 38a5a0b3bfb0ec3ce5ad02b4c334e1a1bd5e385f submit; job_id is 3711
2022-09-21 21:10:47,421 - INFO : job: 92dae58cb1c251ea2e4f3365db0af66d6c616cd8 submit; job_id is 3713
2022-09-21 21:10:47,459 - INFO : job: 5d146339791d27d9185aff7670b06529905268f5 submit; job_id is 3721
2022-09-21 21:10:47,465 - INFO : job: df2fc03c3312034f6c844ce119db3c159ef75352 submit; job_id is 3723
2022-09-21 21:13:18,873 - INFO : job: 5d146339791d27d9185aff7670b06529905268f5 3721 finished
2022-09-21 21:13:18,880 - INFO : job: df2fc03c3312034f6c844ce119db3c159ef75352 3723 finished
2022-09-21 21:13:48,927 - INFO : job: 38a5a0b3bfb0ec3ce5ad02b4c334e1a1bd5e385f 3711 finished
2022-09-21 21:13:48,934 - INFO : job: 92dae58cb1c251ea2e4f3365db0af66d6c616cd8 3713 finished
2022-09-21 21:13:49,386 - INFO : info:check_all_finished: False
2022-09-21 21:13:49,393 - INFO : job: 1320d04e42aa87fd4e95c78d64e290ae74aa31f5 submit; job_id is 3939
2022-09-21 21:13:49,395 - INFO : job: e501d8a55c702e407f9a257b70ac4ef882c39c91 submit; job_id is 3941
2022-09-21 21:17:51,895 - INFO : job: 1320d04e42aa87fd4e95c78d64e290ae74aa31f5 3939 finished
2022-09-21 21:17:51,904 - INFO : job: e501d8a55c702e407f9a257b70ac4ef882c39c91 3941 finished
2022-09-21 21:17:51,953 - INFO : info:check_all_finished: False
2022-09-21 21:17:51,960 - INFO : job: 01543ff0cb9edbba72a6d562494033e87c690a64 submit; job_id is 4667
2022-09-21 21:18:23,017 - INFO : job: 01543ff0cb9edbba72a6d562494033e87c690a64 4667 terminated;fail_cout is 1; resubmitting job
2022-09-21 21:18:23,019 - INFO : job:01543ff0cb9edbba72a6d562494033e87c690a64 re-submit after terminated; new job_id is 4822
2022-09-21 21:18:23,231 - INFO : job:01543ff0cb9edbba72a6d562494033e87c690a64 job_id:4822 after re-submitting; the state now is <JobStatus.running: 3>
2022-09-21 21:18:53,259 - INFO : job: 01543ff0cb9edbba72a6d562494033e87c690a64 4822 terminated;fail_cout is 2; resubmitting job
2022-09-21 21:18:53,261 - INFO : job:01543ff0cb9edbba72a6d562494033e87c690a64 re-submit after terminated; new job_id is 4970
2022-09-21 21:18:53,479 - INFO : job:01543ff0cb9edbba72a6d562494033e87c690a64 job_id:4970 after re-submitting; the state now is <JobStatus.running: 3>
2022-09-21 21:19:23,524 - INFO : job: 01543ff0cb9edbba72a6d562494033e87c690a64 4970 terminated;fail_cout is 3; resubmitting job

Contents of dpgen.log

2022-09-21 21:10:47,285 - INFO : start running
2022-09-21 21:10:47,300 - INFO : =============================iter.000000==============================
2022-09-21 21:10:47,300 - INFO : -------------------------iter.000000 task 00--------------------------
2022-09-21 21:10:47,390 - INFO : -------------------------iter.000000 task 01--------------------------
2022-09-21 21:13:49,047 - INFO : -------------------------iter.000000 task 02--------------------------
2022-09-21 21:13:49,047 - INFO : -------------------------iter.000000 task 03--------------------------
2022-09-21 21:13:49,382 - INFO : -------------------------iter.000000 task 04--------------------------
2022-09-21 21:17:51,907 - INFO : -------------------------iter.000000 task 05--------------------------
2022-09-21 21:17:51,907 - INFO : -------------------------iter.000000 task 06--------------------------
2022-09-21 21:17:51,911 - INFO : system 000 candidate : 242 in 310 78.06 %
2022-09-21 21:17:51,911 - INFO : system 000 failed : 19 in 310 6.13 %
2022-09-21 21:17:51,911 - INFO : system 000 accurate : 49 in 310 15.81 %
2022-09-21 21:17:51,911 - INFO : system 000 accurate_ratio: 0.1581 thresholds: 1.0000 and 1.0000 eff. task min and max -1 20 number of fp tasks: 20
2022-09-21 21:17:51,952 - INFO : -------------------------iter.000000 task 07--------------------------

dpgen.zip

[HuangJiameng]

Could you check fp.log on remote_root? /home/work/DeePMD/dpg_tutorial/dpgen_example/run/new/82ea5da2808459e9d080f8ecfbd5f57031b4a7b5 .

[JasonGyy]

Thank you for your reply.
I did not find fp.log in /home/work/DeePMD/dpg_tutorial/dpgen_example/run/new/82ea5da2808459e9d080f8ecfbd5f57031b4a7b5, the attached file is the zip of 82ea5da2808459e9d080f8ecfbd5f57031b4a7b5.
Thank you.
82ea5da2808459e9d080f8ecfbd5f57031b4a7b5.zip

[HuangJiameng]

fp.log
I find it in the zip package. They are in task.xxx folders.

[JasonGyy]

Thank you very much for your patient help. fp.log indicates that the vasp calculation was not performed due to memory limitation. And it's been solved using "ulimit -s unlimited".
Thank you.

[HuangJiameng]

https://docs.deepmodeling.com/projects/dpgen/en/devel/user-guide/common-errors.html#runtimeerror-job-xxxxxxx-failed-3-times FYI