f"Meet errors will handle unexpected submission state.\n"

[WenyeXuan]

Hello everyone

I got the following errors at iter1 model deviation. I checked the remote file and all the lammps calculation were finished. Can anyone help with that?

Thank you for your help in advance.

INFO:dpgen:-------------------------iter.000001 task 03--------------------------
INFO:dpgen:-------------------------iter.000001 task 04--------------------------
2021-09-21 11:01:26,704 - INFO : info:check_all_finished: False
2021-09-21 11:01:26,705 - INFO : remote path: /users/xuan1996/sharedscratch/dp/20water-dpgen/2762841c045a29aff66f9c65e226bca9cc48633d
2021-09-21 11:01:32,850 - INFO : job: 24041aafcd6cb3b926d584bac1a9991ccd5b563c unsubmitted; submit it
2021-09-21 11:01:33,478 - INFO : job: 24041aafcd6cb3b926d584bac1a9991ccd5b563c submit; job_id is 38181092
2021-09-21 11:01:33,479 - INFO : job: b3748c8b4b57391e4f68c48ee5f109b9685e709c unsubmitted; submit it
2021-09-21 11:01:34,111 - INFO : job: b3748c8b4b57391e4f68c48ee5f109b9685e709c submit; job_id is 38181093
2021-09-21 11:01:34,112 - INFO : job: 6a89fd475c284e8c125a04906545fc707cb64571 unsubmitted; submit it
2021-09-21 11:01:34,722 - INFO : job: 6a89fd475c284e8c125a04906545fc707cb64571 submit; job_id is 38181094
2021-09-21 11:01:34,722 - INFO : job: 97b221c0a76c52832ff4320246513c72715cc40e unsubmitted; submit it
2021-09-21 11:01:35,364 - INFO : job: 97b221c0a76c52832ff4320246513c72715cc40e submit; job_id is 38181095
2021-09-21 11:01:35,364 - INFO : job: c86e80def61985990aa04534c70680f9bec82bac unsubmitted; submit it
2021-09-21 11:01:35,941 - INFO : job: c86e80def61985990aa04534c70680f9bec82bac submit; job_id is 38181096
2021-09-21 11:01:35,941 - INFO : job: 6b392f5028bd8e28f6526f90032d4a6bbf380af9 unsubmitted; submit it
2021-09-21 11:01:36,555 - INFO : job: 6b392f5028bd8e28f6526f90032d4a6bbf380af9 submit; job_id is 38181097
2021-09-21 11:10:25,195 - INFO : job: 97b221c0a76c52832ff4320246513c72715cc40e 38181095 terminated;fail_cout is 1; resubmitting job
2021-09-21 11:10:25,785 - INFO : job:97b221c0a76c52832ff4320246513c72715cc40e re-submit after terminated; new job_id is 38181105
2021-09-21 11:10:26,260 - INFO : job:97b221c0a76c52832ff4320246513c72715cc40e job_id:38181105 after re-submitting; the state now is 3
2021-09-21 11:10:29,312 - INFO : job: 6b392f5028bd8e28f6526f90032d4a6bbf380af9 38181097 finished
2021-09-21 11:10:59,445 - INFO : job: c86e80def61985990aa04534c70680f9bec82bac 38181096 terminated;fail_cout is 1; resubmitting job
2021-09-21 11:11:00,178 - INFO : job:c86e80def61985990aa04534c70680f9bec82bac re-submit after terminated; new job_id is 38181108
2021-09-21 11:11:00,609 - INFO : job:c86e80def61985990aa04534c70680f9bec82bac job_id:38181108 after re-submitting; the state now is 2
2021-09-21 11:12:07,598 - INFO : job: 24041aafcd6cb3b926d584bac1a9991ccd5b563c 38181092 terminated;fail_cout is 1; resubmitting job
2021-09-21 11:12:08,198 - INFO : job:24041aafcd6cb3b926d584bac1a9991ccd5b563c re-submit after terminated; new job_id is 38181111
2021-09-21 11:12:08,716 - INFO : job:24041aafcd6cb3b926d584bac1a9991ccd5b563c job_id:38181111 after re-submitting; the state now is 3
2021-09-21 11:12:08,716 - INFO : job: 6a89fd475c284e8c125a04906545fc707cb64571 38181094 terminated;fail_cout is 1; resubmitting job
2021-09-21 11:12:09,325 - INFO : job:6a89fd475c284e8c125a04906545fc707cb64571 re-submit after terminated; new job_id is 38181112
2021-09-21 11:12:09,860 - INFO : job:6a89fd475c284e8c125a04906545fc707cb64571 job_id:38181112 after re-submitting; the state now is 2
2021-09-21 11:12:42,981 - INFO : job: b3748c8b4b57391e4f68c48ee5f109b9685e709c 38181093 terminated;fail_cout is 1; resubmitting job
2021-09-21 11:12:43,646 - INFO : job:b3748c8b4b57391e4f68c48ee5f109b9685e709c re-submit after terminated; new job_id is 38181113
2021-09-21 11:12:44,155 - INFO : job:b3748c8b4b57391e4f68c48ee5f109b9685e709c job_id:38181113 after re-submitting; the state now is 3
2021-09-21 11:12:44,155 - INFO : job: 97b221c0a76c52832ff4320246513c72715cc40e 38181105 terminated;fail_cout is 2; resubmitting job
2021-09-21 11:12:44,808 - INFO : job:97b221c0a76c52832ff4320246513c72715cc40e re-submit after terminated; new job_id is 38181114
2021-09-21 11:12:45,300 - INFO : job:97b221c0a76c52832ff4320246513c72715cc40e job_id:38181114 after re-submitting; the state now is 2
2021-09-21 11:13:17,901 - INFO : job: c86e80def61985990aa04534c70680f9bec82bac 38181108 terminated;fail_cout is 2; resubmitting job
2021-09-21 11:13:18,517 - INFO : job:c86e80def61985990aa04534c70680f9bec82bac re-submit after terminated; new job_id is 38181115
2021-09-21 11:13:19,025 - INFO : job:c86e80def61985990aa04534c70680f9bec82bac job_id:38181115 after re-submitting; the state now is 2
2021-09-21 11:13:51,712 - INFO : job: 24041aafcd6cb3b926d584bac1a9991ccd5b563c 38181111 terminated;fail_cout is 2; resubmitting job
2021-09-21 11:13:52,433 - INFO : job:24041aafcd6cb3b926d584bac1a9991ccd5b563c re-submit after terminated; new job_id is 38181119
2021-09-21 11:13:52,977 - INFO : job:24041aafcd6cb3b926d584bac1a9991ccd5b563c job_id:38181119 after re-submitting; the state now is 3
2021-09-21 11:14:25,912 - INFO : job: b3748c8b4b57391e4f68c48ee5f109b9685e709c 38181113 terminated;fail_cout is 2; resubmitting job
2021-09-21 11:14:26,549 - INFO : job:b3748c8b4b57391e4f68c48ee5f109b9685e709c re-submit after terminated; new job_id is 38181120
2021-09-21 11:14:27,118 - INFO : job:b3748c8b4b57391e4f68c48ee5f109b9685e709c job_id:38181120 after re-submitting; the state now is 3
2021-09-21 11:14:27,118 - INFO : job: 6a89fd475c284e8c125a04906545fc707cb64571 38181112 terminated;fail_cout is 2; resubmitting job
2021-09-21 11:14:27,788 - INFO : job:6a89fd475c284e8c125a04906545fc707cb64571 re-submit after terminated; new job_id is 38181121
2021-09-21 11:14:28,262 - INFO : job:6a89fd475c284e8c125a04906545fc707cb64571 job_id:38181121 after re-submitting; the state now is 2
2021-09-21 11:15:00,886 - INFO : job: 97b221c0a76c52832ff4320246513c72715cc40e 38181114 terminated;fail_cout is 3; resubmitting job
Traceback (most recent call last):
File "/home/wenye/.local/lib/python3.8/site-packages/dpdispatcher/submission.py", line 215, in handle_unexpected_submission_state
job.handle_unexpected_job_state()
File "/home/wenye/.local/lib/python3.8/site-packages/dpdispatcher/submission.py", line 532, in handle_unexpected_job_state
raise RuntimeError(f"job:{self.job_hash} {self.job_id} failed {self.fail_count} times.job_detail:{self}")
RuntimeError: job:97b221c0a76c52832ff4320246513c72715cc40e 38181114 failed 3 times.job_detail:{'97b221c0a76c52832ff4320246513c72715cc40e': {'job_task_list': [{'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000006', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000024', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000005', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000092', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000068', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000083', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000019', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000072', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000043', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000087', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000020', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000000', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000096', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000057', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000097', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000053', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000090', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000025', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000071', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}, {'command': "/bin/sh -c '{ if [ ! -f dpgen.restart.10000 ]; then lmp -i input.lammps -v restart 0; else lmp -i input.lammps -v restart 1; fi }'", 'task_work_path': 'task.000.000085', 'forward_files': ['conf.lmp', 'input.lammps', 'traj'], 'backward_files': ['model_devi.out', 'model_devi.log', 'traj'], 'outlog': 'model_devi.log', 'errlog': 'model_devi.log'}], 'resources': {'number_node': 1, 'cpu_per_node': 6, 'gpu_per_node': 1, 'queue_name': 'gpu,gpuc', 'group_size': 20, 'custom_flags': ['#!/bin/bash -l', '#SBATCH -D ./', '#SBATCH --export=ALL', '#SBATCH -J H2OIN', '#SBATCH -o output01.txt', '#SBATCH -e error01.txt', '#SBATCH -p gpu,gpuc', '#SBATCH -N 1', '#SBATCH --gres=gpu:1', '#SBATCH -n 6', '#SBATCH -t 0-02:30:00', 'module load apps/anaconda3/2020.07', 'conda activate /users/xuan1996/sharedscratch/deepmd'], 'strategy': {'if_cuda_multi_devices': False}, 'para_deg': 1, 'module_unload_list': [], 'module_list': [], 'source_list': [], 'envs': {}, 'kwargs': {}}, 'job_state': <JobStatus.terminated: 4>, 'job_id': '38181114', 'fail_count': 3}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/wenye/.local/bin/dpgen", line 8, in
sys.exit(main())
File "/home/wenye/.local/lib/python3.8/site-packages/dpgen/main.py", line 175, in main
args.func(args)
File "/home/wenye/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 2944, in gen_run
run_iter (args.PARAM, args.MACHINE)
File "/home/wenye/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 2920, in run_iter
run_model_devi (ii, jdata, mdata)
File "/home/wenye/.local/lib/python3.8/site-packages/dpgen/generator/run.py", line 1297, in run_model_devi
submission.run_submission()
File "/home/wenye/.local/lib/python3.8/site-packages/dpdispatcher/submission.py", line 182, in run_submission
self.handle_unexpected_submission_state()
File "/home/wenye/.local/lib/python3.8/site-packages/dpdispatcher/submission.py", line 219, in handle_unexpected_submission_state
f"Meet errors will handle unexpected submission state.\n"
AttributeError: 'Submission' object has no attribute 'remote_root'

[AnguseZhang]

@felix5572 Please help have a look. Thanks!

[AnguseZhang]

Could you please post as much as information about job 97b221c0a76c52832ff4320246513c72715cc40e to help us locate the problem ? Such as its running log, and how you think all calculations were finished.

[WenyeXuan]

Thank you for your reply. Apology first if I have any misunderstanding of lammps running in dpgen. I check all the task subdirectories and found that all the tasks had been run. But not all the tasks had the "task_tag_finished" file, because a few structures had lammps lost atoms error. I think the errors were caused by not well trained deep potential. Does dpgen can not have failed lammps running in model deviation step?

[njzjz]

The second error has been fixed in deepmodeling/dpdispatcher#125; the first one is expected.

[WenyeXuan]

Thank you for your reply. Yes, I installed the new vision and the second error now have debug information saying its a direct result of the first error. But this did not fix my problem.

[WenyeXuan]

Finally, I fixed this problem after being stuck here for a few weeks :). Just in case someone meets the same problem, I would write down my solution here. The problem is that some lammps calculations failed because of atoms lost. This is because the deep potential is not trained well at this stage. Dpgen used "ratio_failed" in the parameter file to control the highest acceptable fail rate of calculation. If you are confident with what you are doing, just increase the value of ratio_failed to let dpgen continue to work. If the developer can see this reply, please provide a more detailed debug information of this error in the next release. Thank you very much.

[warriors1523]

I set ratio_failed in this position of my paramter.json file ，but it didn't seem to work. Can you tell me where to set this keyword?

Thanks you very mach!