There is always an error when executing "dpgen run param.json machine. Json", I don't know if there is any error in setting file "machine. #811

yuanzuiqaq · 2022-07-19T09:42:26Z

yuanzuiqaq
Jul 19, 2022

First of all, thank you very much for taking the time to read my question. When I execute "dpgen run param.json machine. Json" I always get errors that confuse me.
I used PBS to submit tasks in the remote supercomputer center, and we did not have gpu available for use, so my "machine. Json" file and the errors are as follows.

my error:
(deepmd) [nano006@ln02 run]$ dpgen run param.json machine.json
DeepModeling

Version: 0.10.6
Date: Jul-13-2022
Path: /public/nano006/.local/lib/python3.9/site-packages/dpgen

Dependency

 numpy     1.23.1   /public/nano006/miniconda3/envs/deepmd/lib/python3.9/site-packages/numpy
dpdata      0.2.7   /public/nano006/miniconda3/envs/deepmd/lib/python3.9/site-packages/dpdata

pymatgen unknown version or path
monty 2022.4.26 /public/nano006/miniconda3/envs/deepmd/lib/python3.9/site-packages/monty
ase 3.22.1 /public/nano006/.local/lib/python3.9/site-packages/ase
paramiko 2.11.0 /public/nano006/.local/lib/python3.9/site-packages/paramiko
custodian 2022.5.26 /public/nano006/.local/lib/python3.9/site-packages/custodian

Reference

Please cite:
Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and Weinan E,
DP-GEN: A concurrent learning platform for the generation of reliable deep learning
based potential energy models, Computer Physics Communications, 2020, 107206.

Description

INFO:dpgen:-------------------------iter.000000 task 01--------------------------
2022-07-19 13:38:24,137 - INFO : info:check_all_finished: False
2022-07-19 13:38:24,138 - INFO : remote path: /public/nano006/cxh/deepmd/dpgen_example/run_path/872e78444e69466ca3d6fccef767c7bf6e00649d
2022-07-19 13:38:30,915 - INFO : job: 7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61 submit; job_id is 425938.mu01
2022-07-19 13:38:35,350 - INFO : job: 7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61 425938.mu01 terminated;fail_cout is 1; resubmitting job
2022-07-19 13:38:38,729 - INFO : job:7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61 re-submit after terminated; new job_id is 425939.mu01
2022-07-19 13:38:42,170 - INFO : job:7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61 job_id:425939.mu01 after re-submitting; the state now is <JobStatus.terminated: 4>
2022-07-19 13:38:42,170 - INFO : job: 7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61 425939.mu01 terminated;fail_cout is 2; resubmitting job
2022-07-19 13:38:45,600 - INFO : job:7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61 re-submit after terminated; new job_id is 425940.mu01
2022-07-19 13:38:49,237 - INFO : job:7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61 job_id:425940.mu01 after re-submitting; the state now is <JobStatus.terminated: 4>
2022-07-19 13:38:49,237 - INFO : job: 7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61 425940.mu01 terminated;fail_cout is 3; resubmitting job
Traceback (most recent call last):
File "/public/nano006/.local/lib/python3.9/site-packages/dpdispatcher/submission.py", line 241, in handle_unexpected_submission_state
job.handle_unexpected_job_state()
File "/public/nano006/.local/lib/python3.9/site-packages/dpdispatcher/submission.py", line 612, in handle_unexpected_job_state
self.handle_unexpected_job_state()
File "/public/nano006/.local/lib/python3.9/site-packages/dpdispatcher/submission.py", line 612, in handle_unexpected_job_state
self.handle_unexpected_job_state()
File "/public/nano006/.local/lib/python3.9/site-packages/dpdispatcher/submission.py", line 605, in handle_unexpected_job_state
raise RuntimeError(f"job:{self.job_hash} {self.job_id} failed {self.fail_count} times.job_detail:{self}")
RuntimeError: job:7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61 425940.mu01 failed 3 times.job_detail:{'7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61': {'job_task_list': [{'command': "/bin/sh -c '{ if [ ! -f model.ckpt.index ]; then dp train input.json; else dp train input.json --restart model.ckpt; fi }'&&dp freeze", 'task_work_path': '002', 'forward_files': ['input.json'], 'backward_files': ['frozen_model.pb', 'lcurve.out', 'train.log', 'model.ckpt.meta', 'model.ckpt.index', 'model.ckpt.data-00000-of-00001', 'checkpoint'], 'outlog': 'train.log', 'errlog': 'train.log'}, {'command': "/bin/sh -c '{ if [ ! -f model.ckpt.index ]; then dp train input.json; else dp train input.json --restart model.ckpt; fi }'&&dp freeze", 'task_work_path': '001', 'forward_files': ['input.json'], 'backward_files': ['frozen_model.pb', 'lcurve.out', 'train.log', 'model.ckpt.meta', 'model.ckpt.index', 'model.ckpt.data-00000-of-00001', 'checkpoint'], 'outlog': 'train.log', 'errlog': 'train.log'}, {'command': "/bin/sh -c '{ if [ ! -f model.ckpt.index ]; then dp train input.json; else dp train input.json --restart model.ckpt; fi }'&&dp freeze", 'task_work_path': '003', 'forward_files': ['input.json'], 'backward_files': ['frozen_model.pb', 'lcurve.out', 'train.log', 'model.ckpt.meta', 'model.ckpt.index', 'model.ckpt.data-00000-of-00001', 'checkpoint'], 'outlog': 'train.log', 'errlog': 'train.log'}, {'command': "/bin/sh -c '{ if [ ! -f model.ckpt.index ]; then dp train input.json; else dp train input.json --restart model.ckpt; fi }'&&dp freeze", 'task_work_path': '000', 'forward_files': ['input.json'], 'backward_files': ['frozen_model.pb', 'lcurve.out', 'train.log', 'model.ckpt.meta', 'model.ckpt.index', 'model.ckpt.data-00000-of-00001', 'checkpoint'], 'outlog': 'train.log', 'errlog': 'train.log'}], 'resources': {'number_node': 1, 'cpu_per_node': 32, 'gpu_per_node': 0, 'queue_name': 'thr', 'group_size': 4, 'custom_flags': [], 'strategy': {'if_cuda_multi_devices': False, 'ratio_unfinished': 0.0}, 'para_deg': 1, 'module_purge': False, 'module_unload_list': [], 'module_list': [], 'source_list': ['~/miniconda3/envs/deepmd/bin/dp'], 'envs': {}, 'wait_time': 0, 'kwargs': {}}, 'job_state': <JobStatus.terminated: 4>, 'job_id': '425940.mu01', 'fail_count': 3}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/public/nano006/.local/bin/dpgen", line 8, in
sys.exit(main())
File "/public/nano006/.local/lib/python3.9/site-packages/dpgen/main.py", line 185, in main
args.func(args)
File "/public/nano006/.local/lib/python3.9/site-packages/dpgen/generator/run.py", line 3642, in gen_run
run_iter (args.PARAM, args.MACHINE)
File "/public/nano006/.local/lib/python3.9/site-packages/dpgen/generator/run.py", line 3607, in run_iter
run_train (ii, jdata, mdata)
File "/public/nano006/.local/lib/python3.9/site-packages/dpgen/generator/run.py", line 610, in run_train
submission.run_submission()
File "/public/nano006/.local/lib/python3.9/site-packages/dpdispatcher/submission.py", line 185, in run_submission
self.handle_unexpected_submission_state()
File "/public/nano006/.local/lib/python3.9/site-packages/dpdispatcher/submission.py", line 244, in handle_unexpected_submission_state
raise RuntimeError(
RuntimeError: Meet errors will handle unexpected submission state.
Debug information: remote_root==/public/nano006/cxh/deepmd/dpgen_example/run_path/872e78444e69466ca3d6fccef767c7bf6e00649d.
Debug information: submission_hash==872e78444e69466ca3d6fccef767c7bf6e00649d.
Please check the dirs and scripts in remote_root. The job information mentioned above may help.

my "machine.json":
{
"api_version": "1.0",
"deepmd_version": "2.1.3",
"train" :[
{

		"machine": {
			"batch_type": "PBS",
			"context_type": "SSHContext",
			"local_root" : "./",
			"remote_root": "/public/nano006/cxh/deepmd/dpgen_example/run_path",
                            "remote_profile": {
         				 "hostname": "172.16.0.167",
         				 "username": "nano006",
         				 "password": "******",
        				 "port": 22,
         				 "_key_filename": "/public/nano006/.ssh/id_rsa",
				 "_comment": "key filename used by ssh connection. if left none, find key in ~/.ssh or use password for login"
         				}
		},
		"resources": {
			"number_node": 1,
			"cpu_per_node": 32,
			"gpu_per_node": 0,
			"queue_name": "thr",
			"group_size": 4,
			"custom_flags":[],
      			"_comment": "The extra lines pass to job submitting script header, for example '#PBS --mem=32G'",
      			"_strategy": {"if_cuda_multi_devices": false},
			"_para_deg":2,
      			"source_list": ["~/miniconda3/envs/deepmd/bin/dp"],
			"mem_limit":32,
			"time_limit":"23:0:0"
			

		},
		"command": "dp"
	}
],
"model_devi":[
	{
		
		"machine": {
			"batch_type": "PBS",
			"context_type": "SSHContext",
			"local_root" : "./",
			"remote_root": "/public/nano006/cxh/deepmd/dpgen_example/run_path",
			"remote_profile": {
       			 	"hostname": "172.16.0.167",
        				"username": "nano006",
        				"password": "******,
       			 	"port": 22,
        				"_key_filename": "/public/nano006/.ssh/id_rsa",
        				"_comment": "key filename used by ssh connection. If left None, find key in ~/.ssh or use password for login"
     			 }

		},
		"resources": {
			"number_node": 1,
			"cpu_per_node": 32,
			"gpu_per_node": 0,
			"queue_name": "thr",
			"group_size": 5,
			"source_list": ["~/miniconda3/envs/deepmd/bin/lmp"],
			"mem_limit":32,
			"time_limit":"23:0:0"
		},
		"command": "lmp"

	}
],
"fp":[
	{
		
		"machine": {
			"batch_type": "PBS",
			"context_type": "SSHContext",
			"local_root" : "./",
			"remote_root": "/public/nano006/cxh/deepmd/dpgen_example/run_path",
    			"remote_profile": {
     		 		"hostname": "172.16.0.167",
      				"username": "nano006",
				"password": "******",
  				"port": 22,
        				"_key_filename": "/public/nano006/.ssh/id_rsa",
        				"_comment": "key filename used by ssh connection. If left None, find key in ~/.ssh or use password for login"
     			}

		},
		"resources": {
			"number_node": 1,
			"cpu_per_node": 32,
			"gpu_per_node": 0,
			"queue_name": "thr",
			"group_size": 5,
			"source_list": ["/opt/software/vasp/vasp_std"],
			"mem_limit":32,
			"time_limit":"23:0:0"
		},
		"command": "vasp_std"


	}
]

}

Thank you for taking time out of your busy schedule to answer my question!!!

HuangJiameng · 2022-07-19T09:58:17Z

HuangJiameng
Jul 19, 2022
Collaborator

please provide log files under /public/nano006/cxh/deepmd/dpgen_example/run_path/

5 replies

yuanzuiqaq Jul 20, 2022
Author

7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61.sub.txt
7ced0e2aa93d65c2cdbe53b2335ef0d78e226d61.json.txt
out.txt

HuangJiameng Jul 20, 2022
Collaborator

could you provide train.log

yuanzuiqaq Jul 20, 2022
Author

Sorry, I can't find the train. Log file in the run_path directory
dpdispatcher.log
dpgen.log
.

HuangJiameng Jul 20, 2022
Collaborator

There should be train.log. Could you check the files on the remote server carefully? The information you provide only shows that your job has failed in train step, but has not shown the reason.

yuanzuiqaq Jul 20, 2022
Author

Ok, thank you very much. I need to check my documents.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

There is always an error when executing "dpgen run param.json machine. Json", I don't know if there is any error in setting file "machine. #811

{{title}}

Replies: 1 comment 5 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

There is always an error when executing "dpgen run param.json machine. Json", I don't know if there is any error in setting file "machine. #811

yuanzuiqaq Jul 19, 2022

my error: (deepmd) [nano006@ln02 run]$ dpgen run param.json machine.json DeepModeling

Dependency

Reference

Please cite: Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and Weinan E, DP-GEN: A concurrent learning platform for the generation of reliable deep learning based potential energy models, Computer Physics Communications, 2020, 107206.

Description

Replies: 1 comment · 5 replies

HuangJiameng Jul 19, 2022 Collaborator

yuanzuiqaq Jul 20, 2022 Author

HuangJiameng Jul 20, 2022 Collaborator

yuanzuiqaq Jul 20, 2022 Author

HuangJiameng Jul 20, 2022 Collaborator

yuanzuiqaq Jul 20, 2022 Author

yuanzuiqaq
Jul 19, 2022

my error:
(deepmd) [nano006@ln02 run]$ dpgen run param.json machine.json
DeepModeling

Please cite:
Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and Weinan E,
DP-GEN: A concurrent learning platform for the generation of reliable deep learning
based potential energy models, Computer Physics Communications, 2020, 107206.

Replies: 1 comment 5 replies

HuangJiameng
Jul 19, 2022
Collaborator

yuanzuiqaq Jul 20, 2022
Author

HuangJiameng Jul 20, 2022
Collaborator

yuanzuiqaq Jul 20, 2022
Author

HuangJiameng Jul 20, 2022
Collaborator

yuanzuiqaq Jul 20, 2022
Author