Fail to run 01model_devi after 00.train

[hahha2022]

dpgen version:0.10.6

After finished the 00.train task in the second dpgen iterate, the program suddenly stops and no more error is showed. I want to continue the task of 01.model_devi, but it does not work whatever I setting. The corresponding input file and output file are attached. I don't know why, I hope you can help me.

Description

2022-09-06 09:55:35,475 - INFO : info:check_all_finished: False
2022-09-06 09:55:35,506 - INFO : job: 455d98445bd7dcbc7b183f1851eeec3b4a15c701 submit; job_id is 3750.host
2022-09-06 09:55:35,522 - INFO : job: b5ccc423742b9945929f2fe06374c34ddc63524a submit; job_id is 3751.host
2022-09-06 09:55:35,537 - INFO : job: 310b9fecf609e8052653d0f0264aedf8f3175209 submit; job_id is 3752.host
2022-09-06 09:55:35,551 - INFO : job: dbb77a777f3c553a51036ea1d401165c18865359 submit; job_id is 3753.host
2022-09-06 18:08:54,679 - INFO : job: 310b9fecf609e8052653d0f0264aedf8f3175209 3752.host finished
2022-09-06 18:08:54,695 - INFO : job: dbb77a777f3c553a51036ea1d401165c18865359 3753.host finished
2022-09-06 20:21:39,281 - INFO : job: 455d98445bd7dcbc7b183f1851eeec3b4a15c701 3750.host finished
2022-09-06 20:22:09,343 - INFO : job: b5ccc423742b9945929f2fe06374c34ddc63524a 3751.host finished
INFO:dpgen:-------------------------iter.000001 task 02--------------------------
INFO:dpgen:-------------------------iter.000001 task 03--------------------------
INFO:dpgen:finished

Here is my param.json
{
"type_map": ["O", "Fe", "Co", "Nb", "Ba", "Ta"],
"mass_map": [16, 55.847, 58.933, 92.910, 137.327, 180.948],

"init_data_prefix": "/home/liu103/workplace/gongjian/train/dpgen2/",
"init_data_sys": [
                 "init"
],

"sys_configs_prefix": "/home/liu103/workplace/gongjian/train/dpgen2/MD_POSCAR/",
"sys_configs": [
             ["POSCAR"]
],

"_comment": " 00.train ",
 
"numb_models": 4,

"default_training_param": {
    "model": {
          "type_map": ["O", "Fe", "Co", "Nb", "Ba", "Ta"],
          "descriptor": {
              "type": "se_e2_a",
              "sel": [50, 6, 22, 2, 32, 2],
              "rcut_smth": 1.0,
              "rcut": 8.0,
              "neuron":[20, 40, 80],
              "resnet_dt": false,
              "type_one_side": true,
              "axis_neuron": 12,
              "seed": 1
                         },
          "fitting_net": {
              "neuron": [100, 100, 100],
              "resnet_dt": true,
              "seed": 1
                         }
           },
     "learning_rate": {
          "type": "exp",
          "start_lr": 0.001,
          "decay_steps": 100,
          "start_lr":     0.001,
          "stop_lr":      3.51e-8
                   },
     "loss": {
          "start_pref_e": 0.02,
          "limit_pref_e": 1,
          "start_pref_f": 1000,
          "limit_pref_f": 2,
          "start_pref_v": 0.0,
          "limit_pref_v": 0.0
             },
      "training": {
          "stop_batch": 100000,
          "batch_size":1,
          "seed":1,
          "disp_file": "lcurve.out",
          "disp_freq": 1000,
          "numb_test": 4,
          "save_freq": 1000,
          "save_ckpt": "model.ckpt",
          "disp_training": true,
          "time_training": true,
          "profiling": false,
          "profiling_file": "timeline.json",
          "_comment": "that's all"
                   }
                     },
  
"_comment": " 01.model_devi ",
"model_devi_dt": 0.001,
"model_devi_skip": 0,
"model_devi_f_trust_lo": 0.05,
"model_devi_f_trust_hi": 0.30,
"model_devi_clean_traj": false,
"model_devi_jobs": [
{"sys_idx": [0],"temps": [1273],"press": [1], "trj_freq": 10, "nsteps":   1000, "ensemble": "nvt", "_idx":"00"}
],
  
"_comment": " 02.fp ",
"fp_style": "vasp",
"fp_accurate_threshold": 0.999,
"fp_accurate_soft_threshold": 0.9,
"shuffle_poscar": false,
"fp_task_max": 32,
"fp_task_min": 8,
"ratio_failed": 0.25,
"fp_pp_path": "./POTCAR",
"fp_pp_files": [
  "POTCAR_O","POTCAR_Fe", "POTCAR_Co", "POTCAR_Nb", "POTCAR_Ba", "POTCAR_Ta"
],
"fp_incar": "./INCAR"

}

here is my lcurve.out file in iter.000001/000

step rmse_trn rmse_e_trn rmse_f_trn lr
91000 3.17e-01 4.92e-03 2.15e-01 8.8e-08
92000 3.14e-01 4.32e-03 2.15e-01 8.0e-08
93000 2.92e-01 3.41e-03 2.01e-01 7.2e-08
94000 2.94e-01 3.54e-03 2.03e-01 6.5e-08
95000 3.35e-01 2.69e-03 2.32e-01 5.9e-08
96000 2.95e-01 1.29e-03 2.05e-01 5.3e-08
97000 3.29e-01 3.70e-03 2.27e-01 4.8e-08
98000 2.85e-01 1.32e-03 1.99e-01 4.3e-08
99000 3.03e-01 3.38e-03 2.10e-01 3.9e-08
100000 3.42e-01 3.59e-03 2.38e-01 3.5e-08

[HuangJiameng]

You have just set one model_devi job, so it successfully ended after one iteration. You can refer to https://tutorials.deepmodeling.com/en/devel/Tutorials/DP-GEN/learnDoc/DP-GEN_handson.html to get start with dpgen.

[hahha2022]

Thank you for your answer