some errors encountered during the lmp calculations

[phyoung123]

dear all, i'm a beginner of dpgen, i want to train a dp model for mgsio3 in high T and P, i'm seriously confused with the bad error information during the exploration step, some of the randomly produced work files can be finished while i meet some errors, i think the main reason is the dp model is not trained well but i cannot figure out what is improper in my input files by myself, any advices provided here are appreciated. the following is the model_devi.log

The 1th log file
LAMMPS (29 Sep 2021 - Update 2)
using 2 OpenMP thread(s) per MPI task
Loading plugin: deepmd pair style v2.0 by Han Wang
Loading plugin: compute deeptensor/atom v2.0 by Han Wang
Loading plugin: fix dplr v2.0 by Han Wang
Reading data file ...
triclinic box = (0.0000000 0.0000000 0.0000000) to (9.7101201 9.0263589 4.9050493) with tilt (0.14515478 -1.0237750 0.15806425)
1 by 1 by 1 MPI processor grid
reading atoms ...
40 atoms
read_data CPU = 0.293 seconds
Changing box ...
triclinic box = (0.0000000 0.0000000 0.0000000) to (9.7101201 9.0263589 4.9050493) with tilt (0.14515478 -1.0237750 0.15806425)
Summary of lammps deepmd module ...

Info of deepmd-kit:
installed to: /apps/users/caep-lzg/package/anaconda/envs/deepmd
source: v2.0.2
source branch: HEAD
source commit: 5a847966
source commit at: 2021-09-29 16:48:37 +0800
surpport model ver.:1.0
build float prec: double
build with tf inc: /apps/users/caep-lzg/package/anaconda/envs/deepmd/include;/apps/users/caep-lzg/package/anaconda/envs/deepmd/include
build with tf lib: /apps/users/caep-lzg/package/anaconda/envs/deepmd/lib/libtensorflow_cc.so;/apps/users/caep-lzg/package/anaconda/envs/deepmd/lib/libtensorflow_framework.so
set tf intra_op_parallelism_threads: 52
set tf inter_op_parallelism_threads: 52
Info of lammps module:
use deepmd-kit at: /apps/users/caep-lzg/package/anaconda/envs/deepmd
source: v2.0.2
source branch: HEAD
source commit: 5a847966
source commit at: 2021-09-29 16:48:37 +0800
build float prec: double
build with tf inc: /apps/users/caep-lzg/package/anaconda/envs/deepmd/include;/apps/users/caep-lzg/package/anaconda/envs/deepmd/include
build with tf lib: /apps/users/caep-lzg/package/anaconda/envs/deepmd/lib/libtensorflow_cc.so;/apps/users/caep-lzg/package/anaconda/envs/deepmd/lib/libtensorflow_framework.so
Info of model(s):
using 4 model(s): ../graph.000.pb ../graph.001.pb ../graph.002.pb ../graph.003.pb
rcut in model: 5
ntypes in model: 3

CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE

Your simulation uses code contributions which should be cited:

• USER-DEEPMD package:
  The log file lists these citations in BibTeX format.

CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE-CITE

Neighbor list info ...
update every 1 steps, delay 10 steps, check yes
max neighbors/atom: 2000, page size: 100000
master list distance cutoff = 6
ghost atom cutoff = 6
binsize = 3, bins = 4 4 2
1 neighbor lists, perpetual/occasional/extra = 1 0 0
(1) pair deepmd, perpetual
attributes: full, newton on
pair build: full/bin/atomonly
stencil: full/bin/3d
bin: standard
Setting up Verlet run ...
Unit style : metal
Current step : 0
Time step : 0.0005
Per MPI rank memory allocation (min/avg/max) = 5.266 | 5.266 | 5.266 Mbytes
Step Temp PotEng KinEng TotEng Press Volume Lx Ly Lz Xy Xz Yz
0 200 -286.45485 1.0082291 -285.44662 -43295.07 429.913 9.7101201 9.0263589 4.9050493 0.14515478 -1.023775 0.15806425
10 205.68808 -284.40081 1.0369036 -283.36391 222959.06 384.93153 9.3589165 8.699886 4.7276395 0.1399047 -0.9867463 0.15234725
20 609.7783 -253.74513 3.0739812 -250.67114 1238881.3 306.98479 8.6790068 8.0678539 4.3841844 0.12974086 -0.91506083 0.14127948
...........
1200 1551.4581 -336.31148 7.8211261 -328.49035 1660589.2 19.680924 3.4735981 3.2289964 1.7546817 0.051926173 -0.36623471 0.056544274
1210 1503.2753 -335.26405 7.5782296 -327.68582 3526092.3 19.31399 3.4518751 3.2088031 1.7437084 0.05160144 -0.36394437 0.05619066
1220 1387.5878 -336.31737 6.995032 -329.32234 987723.97 19.092742 3.4386437 3.1965034 1.7370245 0.051403645 -0.36254933 0.055975275
1230 1782.7641 -337.33536 8.9871734 -328.34819 4211981.9 18.991511 3.4325557 3.1908441 1.7339492 0.051312636 -0.36190745 0.055876172
1240 1887.5695 -339.26734 9.5155126 -329.75183 720493.2 18.990278 3.4324813 3.190775 1.7339116 0.051311525 -0.36189961 0.055874962
1250 1818.2176 -340.82726 9.1658998 -331.66136 1560789.7 18.894348 3.4266918 3.1853932 1.7309871 0.051224979 -0.3612892 0.055780719
1260 1644.1309 -340.55506 8.2883035 -332.26676 582526.1 18.722873 3.416294 3.1757275 1.7257346 0.051069544 -0.36019292 0.055611461
1270 1399.2023 -340.75492 7.0535828 -333.70134 1580757.4 18.537234 3.4049656 3.1651968 1.7200121 0.050900197 -0.35899852 0.055427052
ERROR on proc 0: Neighbor list overflow, boost neigh_modify one (src/npair_full_bin_atomonly.cpp:89)
Last command: run ${NSTEPS} upto
application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0
[unset]: write_line error; fd=-1 buf=:cmd=abort exitcode=1
:
system msg for write_line failure : Bad file descriptor

The 2th log file:
1430 3878.2392 -326.63735 19.550769 -307.08658 7727148.9 20.412765 1.9075663 3.2284673 3.3145593 0.058305087 -0.83610498 0.031699876
1440 3480.4961 -326.99899 17.545688 -309.4533 -70516.972 20.718017 1.9170279 3.2444806 3.3309996 0.058594281 -0.84025208 0.031857109
1450 3713.719 -328.60566 18.721398 -309.88427 1862683.1 20.916577 1.9231327 3.2548126 3.3416071 0.058780874 -0.84292786 0.031958557
1460 4200.7709 -336.73118 21.176698 -315.55448 3418357.8 21.079181 1.9281032 3.263225 3.3502439 0.0589328 -0.84510651 0.032041158
1470 3898.8807 -336.2864 19.654825 -316.63157 -361469.24 21.334928 1.9358696 3.2763692 3.3637386 0.059170181 -0.84851058 0.032170219
1480 2879.1748 -329.06812 14.514339 -314.55378 -100758.06 21.531662 1.9418018 3.2864092 3.3740463 0.059351498 -0.85111071 0.0322688
1490 345002.7 -324.50231 1739.2089 1414.7066 90991807 22.445768 1.9689009 3.3322732 3.4211334 0.060179789 -0.86298854 0.032719133
ERROR: Lost atoms: original 40 current 39 (src/thermo.cpp:439)
Last command: run ${NSTEPS} upto

The param.json:
{
"type_map": [
"Mg",
"Si",
"O"
],
"mass_map": [
24,
28,
16
],
"init_data_prefix": "/apps/users/caep-lzg/xufy/deepmd/MgSiO3-restart/init",
"init_data_sys": [
"POSCAR_C2_c.01x01x01/02.md/sys-0008-0008-0024/deepmd",
"POSCAR_P21_c.01x01x01/02.md/sys-0008-0008-0024/deepmd",
"POSCAR_Pbcn.01x01x01/02.md/sys-0008-0008-0024/deepmd",
"POSCAR_Pbca.01x01x01/02.md/sys-0016-0016-0048/deepmd"
],
"sys_configs_prefix": "/apps/users/caep-lzg/xufy/deepmd/MgSiO3-restart/init",
"sys_configs": [
["POSCAR_C2_c.01x01x01/01.scale_pert/sys-0008-0008-0024/scale-1.000/00000*/POSCAR"],
["POSCAR_C2_c.01x01x01/01.scale_pert/sys-0008-0008-0024/scale-1.000/00001*/POSCAR"],
["POSCAR_C2_c.01x01x01/01.scale_pert/sys-0008-0008-0024/scale-1.000/00002*/POSCAR"],

    ["POSCAR_P21_c.01x01x01/01.scale_pert/sys-0008-0008-0024/scale-1.000/00000*/POSCAR"],
    ["POSCAR_P21_c.01x01x01/01.scale_pert/sys-0008-0008-0024/scale-1.000/00001*/POSCAR"],
    ["POSCAR_P21_c.01x01x01/01.scale_pert/sys-0008-0008-0024/scale-1.000/00002*/POSCAR"],

    ["POSCAR_Pbcn.01x01x01/01.scale_pert/sys-0008-0008-0024/scale-1.000/00000*/POSCAR"],
    ["POSCAR_Pbcn.01x01x01/01.scale_pert/sys-0008-0008-0024/scale-1.000/00001*/POSCAR"],
    ["POSCAR_Pbcn.01x01x01/01.scale_pert/sys-0008-0008-0024/scale-1.000/00002*/POSCAR"],

    ["POSCAR_Pbca.01x01x01/01.scale_pert/sys-0016-0016-0048/scale-1.000/00000*/POSCAR"],
    ["POSCAR_Pbca.01x01x01/01.scale_pert/sys-0016-0016-0048/scale-1.000/00001*/POSCAR"],
    ["POSCAR_Pbca.01x01x01/01.scale_pert/sys-0016-0016-0048/scale-1.000/00002*/POSCAR"]
],

"_comment": " that's all ",
"_comment": "01.model_devi",
"numb_models": 4,
"_ratio_failed": 1,
"default_training_param": {
    "model": {
        "type_map": [
            "Mg",
            "Si",
            "O"
        ],
        "descriptor": {
            "type": "se_a",
            "sel": "auto",
            "rcut_smth": 0.5,
            "rcut": 5.0,
            "neuron": [
                25,
                50,
                100
            ],
            "resnet_dt": false,
            "axis_neuron": 12,
            "seed": 1
        },
        "fitting_net": {
            "neuron": [
                240,
                240,
                240
            ],
            "resnet_dt": true,
            "seed": 1
        }
    },
    "learning_rate": {
        "type": "exp",
        "start_lr": 0.001,
        "decay_steps": 2000,
        "decay_rate":0.95
    },
    "loss": {
        "start_pref_e": 0.02,
        "limit_pref_e": 1,
        "start_pref_f": 1000,
        "limit_pref_f": 1,
        "start_pref_v": 0.0,
        "limit_pref_v": 0.0
    },
    "training": {
        "set_prefix": "set",
        "stop_batch": 400000,
        "batch_size": 1,
        "disp_file": "lcurve.out",
        "disp_freq": 1000,
        "numb_test": 4,
        "save_freq": 5000,
        "save_ckpt": "model.ckpt",
        "disp_training": true,
        "time_training": true,
        "profiling": false,
        "profiling_file": "timeline.json",
        "_comment": "that's all"
    }
},
"model_devi_dt": 0.0005,
"model_devi_skip": 0,
"model_devi_f_trust_lo": 0.05,
"model_devi_f_trust_hi": 0.15,
"model_devi_e_trust_lo": 10000000000.0,
"model_devi_e_trust_hi": 10000000000.0,
"model_devi_clean_traj": true,
"model_devi_jobs": [
    {
        "sys_idx": [
          0, 3, 6, 9
        ],
        "temps": [
            100.0, 200.0, 300.0, 400.0, 500, 600, 700
        ],
        "press": [
            1.0
        ],
        "trj_freq": 10,
        "nsteps": 3000,
        "ensemble": "nvt",
        "_idx": "00"
    },
    {
        "sys_idx": [
            1, 4, 7, 10
        ],
        "temps": [
            100.0, 200.0, 300.0, 400.0, 500, 600, 700
        ],
        "press": [
            1.0
        ],
        "trj_freq": 10,
        "nsteps": 3000,
        "ensemble": "nvt",
        "_idx": "01"
    },
    {
        "sys_idx": [
            2, 5, 8, 11
        ],
        "temps": [
            100.0, 200, 300, 400, 500, 600, 700
        ],
        "press": [
            1.0
        ],
        "trj_freq": 10,
        "nsteps": 3000,
        "ensemble": "nvt",
        "_idx": "02"
    },
    {
        "sys_idx": [
            0, 3, 6, 9
        ],
        "temps": [
            100.0, 200.0, 300.0, 400.0
        ],
        "press": [
            1.0, 1000.0, 100000.0, 500000.0, 1000000.0, 1500000, 2000000
        ],
        "trj_freq": 10,
        "nsteps": 2000,
        "ensemble": "npt",
        "_idx": "03"
    },
    {
        "sys_idx": [
            1, 4, 7, 10
        ],
        "temps": [
            100.0, 200.0, 300.0, 400.0
        ],
        "press": [
            1.0, 1000.0, 100000.0, 500000.0, 1000000.0, 1500000, 2000000
        ],
        "trj_freq": 10,
        "nsteps": 3000,
        "ensemble": "npt",
        "_idx": "04"
    },
    {
        "sys_idx": [
            2, 5, 8, 11
        ],
        "temps": [
            100.0, 200.0, 300.0, 400.0
        ],
        "press": [
            1.0, 1000.0, 100000.0, 500000.0, 1000000.0, 1500000, 2000000
        ],
        "trj_freq": 10,
        "nsteps": 3000,
        "ensemble": "npt",
        "_idx": "05"
    },
    {
        "sys_idx": [
            0, 3, 6, 9
        ],
        "temps": [
            500.0, 600.0, 700.0, 800.0
        ],
        "press": [
            1.0, 1000.0, 100000.0, 500000.0, 1000000.0, 1500000, 2000000
        ],
        "trj_freq": 10,
        "nsteps": 2000,
        "ensemble": "npt",
        "_idx": "06"
    },
    {
        "sys_idx": [
            1, 4, 7, 10
        ],
        "temps": [
            500.0, 600.0, 700.0, 800.0
        ],
        "press": [
            1.0, 1000.0, 100000.0, 500000.0, 1000000.0, 1500000, 2000000
        ],
        "trj_freq": 10,
        "nsteps": 3000,
        "ensemble": "npt",
        "_idx": "07"
    },
    {
        "sys_idx": [
            2, 5, 8, 11
        ],
        "temps": [
            500.0, 600.0, 700.0, 800.0
        ],
        "press": [
            1.0, 1000.0, 100000.0, 500000.0, 1000000.0, 1500000, 2000000
        ],
        "trj_freq": 10,
        "nsteps": 3000,
        "ensemble": "npt",
        "_idx": "08"
    },
    {
        "sys_idx": [
          0, 3, 6, 9
        ],
        "temps": [
            900.0, 1000.0, 1100.0, 1200.0
        ],
        "press": [
            1.0, 1000.0, 100000.0, 500000.0, 1000000.0, 1500000, 2000000
        ],
        "trj_freq": 10,
        "nsteps": 6000,
        "ensemble": "npt",
        "_idx": "09"
    },
    {
        "sys_idx": [
            1, 4, 7, 10
        ],
        "temps": [
            900.0, 1000.0, 1100.0, 1200.0
        ],
        "press": [
            1.0, 1000.0, 100000.0, 500000.0, 1000000.0, 1500000, 2000000
        ],
        "trj_freq": 10,
        "nsteps": 6000,
        "ensemble": "npt",
        "_idx": "010"
    },
    {
        "sys_idx": [
            2, 5, 8, 11
        ],
        "temps": [
            900.0, 1000.0, 1100.0, 1200.0
        ],
        "press": [
            1.0, 1000.0, 100000.0, 500000.0, 1000000.0, 1500000, 2000000
        ],
        "trj_freq": 10,
        "nsteps": 6000,
        "ensemble": "npt",
        "_idx": "011"
    }
],
"fp_style": "vasp",
"shuffle_poscar": false,
"fp_task_max": 30,
"fp_task_min": 5,
"fp_pp_path": "./",
"fp_pp_files": [
    "POTCAR_Mg",
    "POTCAR_Si",
    "POTCAR_O"
],
"fp_incar": "./INCAR"

}

The 'nvt' ensemble can be done while the errors appear when the iteration enters 'npt'.
Thank you in advance.

[phyoung123]

Alright, i have solved this problem after trapped several days. its derived from the parameter 'sel', which i set it to 'auto' in my input param.json file, when the program starts to train a model there is a copyfile, out.json, of train section of my param.json file, I found that the 'sel' of the out.json file generated by the program automatically is not consistent with the ratio of atomic number, when i manually change the 'sel', it works well now.