Skip to content

Commit 93b833a

Browse files
amcadmusamcadmus
authored
Merge pull request #613 from amcadmus/master
Merge recent devel into master
2 parents 02037ad + eb6449c commit 93b833a

39 files changed

+669
-238
lines changed

‎README.md

+7-5
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@
2222

2323
[![GitHub release](https://img.shields.io/github/release/deepmodeling/dpgen.svg?maxAge=86400)](https://github.com/deepmodeling/dpgen/releases/)
2424
[![doi:10.1016/j.cpc.2020.107206](https://img.shields.io/badge/DOI-10.1016%2Fj.cpc.2020.107206-blue)](https://doi.org/10.1016/j.cpc.2020.107206)
25+
![Citations](https://citations.njzjz.win/10.1016/j.cpc.2020.107206)
2526
[![conda install](https://img.shields.io/conda/dn/conda-forge/dpgen?label=conda%20install)](https://anaconda.org/conda-forge/dpgen)
2627
[![pip install](https://img.shields.io/pypi/dm/dpgen?label=pip%20install)](https://pypi.org/project/dpgen)
2728

@@ -544,20 +545,21 @@ The bold notation of key (such aas **type_map**) means that it's a necessary key
544545
| training_iter0_model_path | list of string | ["/path/to/model0_ckpt/", ...] | The model used to init the first iter training. Number of element should be equal to `numb_models` |
545546
| training_init_model | bool | False | Iteration > 0, the model parameters will be initilized from the model trained at the previous iteration. Iteration == 0, the model parameters will be initialized from `training_iter0_model_path`. |
546547
| **default_training_param** | Dict | | Training parameters for `deepmd-kit` in `00.train`. <br /> You can find instructions from here: (https://github.com/deepmodeling/deepmd-kit)..<br /> |
548+
| dp_compress | bool | false | Use `dp compress` to compress the model. Default is false. |
547549
| *#Exploration*
548550
| **model_devi_dt** | Float | 0.002 (recommend) | Timestep for MD |
549551
| **model_devi_skip** | Integer | 0 | Number of structures skipped for fp in each MD
550-
| **model_devi_f_trust_lo** | Float | 0.05 | Lower bound of forces for the selection.
551-
| **model_devi_f_trust_hi** | Float | 0.15 | Upper bound of forces for the selection
552-
| **model_devi_v_trust_lo** | Float | 1e10 | Lower bound of virial for the selection. Should be used with DeePMD-kit v2.x |
553-
| **model_devi_v_trust_hi** | Float | 1e10 | Upper bound of virial for the selection. Should be used with DeePMD-kit v2.x |
552+
| **model_devi_f_trust_lo** | Float or List of float | 0.05 | Lower bound of forces for the selection. If List, should be set for each index in `sys_configs`, respectively. |
553+
| **model_devi_f_trust_hi** | Float or List of float | 0.15 | Upper bound of forces for the selection. If List, should be set for each index in `sys_configs`, respectively. |
554+
| **model_devi_v_trust_lo** | Float or List of float | 1e10 | Lower bound of virial for the selection. If List, should be set for each index in `sys_configs`, respectively. Should be used with DeePMD-kit v2.x. |
555+
| **model_devi_v_trust_hi** | Float or List of float | 1e10 | Upper bound of virial for the selection. If List, should be set for each index in `sys_configs`, respectively. Should be used with DeePMD-kit v2.x. |
554556
| model_devi_adapt_trust_lo | Boolean | False | Adaptively determines the lower trust levels of force and virial. This option should be used together with `model_devi_numb_candi_f`, `model_devi_numb_candi_v` and optionally with `model_devi_perc_candi_f` and `model_devi_perc_candi_v`. `dpgen` will make two sets: 1. From the frames with force model deviation lower than `model_devi_f_trust_hi`, select `max(model_devi_numb_candi_f, model_devi_perc_candi_f*n_frames)` frames with largest force model deviation. 2. From the frames with virial model deviation lower than `model_devi_v_trust_hi`, select `max(model_devi_numb_candi_v, model_devi_perc_candi_v*n_frames)` frames with largest virial model deviation. The union of the two sets is made as candidate dataset|
555557
| model_devi_numb_candi_f | Int | 10 | See `model_devi_adapt_trust_lo`.|
556558
| model_devi_numb_candi_v | Int | 0 | See `model_devi_adapt_trust_lo`.|
557559
| model_devi_perc_candi_f | Float | 0.0 | See `model_devi_adapt_trust_lo`.|
558560
| model_devi_perc_candi_v | Float | 0.0 | See `model_devi_adapt_trust_lo`.|
559561
| model_devi_f_avg_relative | Boolean | False | Normalized the force model deviations by the RMS force magnitude along the trajectory. This key should not be used with `use_relative`. |
560-
| **model_devi_clean_traj** | Boolean | true | Deciding whether to clean traj folders in MD since they are too large. |
562+
| **model_devi_clean_traj** | Boolean or Int | true | If type of model_devi_clean_traj is boolean type then it denote whether to clean traj folders in MD since they are too large. If it is Int type, then the most recent n iterations of traj folders will be retained, others will be removed. |
561563
| **model_devi_nopbc** | Boolean | False | Assume open boundary condition in MD simulations. |
562564
| model_devi_activation_func | List of list of string | [["tanh","tanh"],["tanh","gelu"],["gelu","tanh"],["gelu","gelu"]] | Set activation functions for models, length of the List should be the same as `numb_models`, and two elements in the list of string respectively assign activation functions to the embedding and fitting nets within each model. *Backward compatibility*: the orginal "List of String" format is still supported, where embedding and fitting nets of one model use the same activation function, and the length of the List should be the same as `numb_models`|
563565
| **model_devi_jobs** | [<br/>{<br/>"sys_idx": [0], <br/>"temps": <br/>[100],<br/>"press":<br/>[1],<br/>"trj_freq":<br/>10,<br/>"nsteps":<br/> 1000,<br/> "ensembles": <br/> "nvt" <br />},<br />...<br />] | List of dict | Settings for exploration in `01.model_devi`. Each dict in the list corresponds to one iteration. The index of `model_devi_jobs` exactly accord with index of iterations |

‎conda/meta.yaml

+6
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,9 @@ requirements:
2525
- requests
2626
- dpdata
2727
- dpdispatcher
28+
- ase
29+
- GromacsWrapper
30+
- custodian
2831

2932
run:
3033
- python >=3.6
@@ -34,6 +37,9 @@ requirements:
3437
- dpdata
3538
- dpdispatcher
3639
- pymatgen
40+
- ase
41+
- GromacsWrapper
42+
- custodian
3743

3844
test:
3945
imports:

‎dpgen/auto_test/EOS.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -169,8 +169,8 @@ def _compute_lower(self,
169169
# vol = self.vol_start + ii * self.vol_step
170170
vol = loadfn(os.path.join(all_tasks[ii], 'eos.json'))['volume']
171171
task_result = loadfn(all_res[ii])
172-
res_data[vol] = task_result['energies'][-1] / task_result['atom_numbs'][0]
173-
ptr_data += '%7.3f %8.4f \n' % (vol, task_result['energies'][-1] / task_result['atom_numbs'][0])
172+
res_data[vol] = task_result['energies'][-1] / task_result['atom_numbs'].sum()
173+
ptr_data += '%7.3f %8.4f \n' % (vol, task_result['energies'][-1] / task_result['atom_numbs'].sum())
174174
# res_data[vol] = all_res[ii]['energy'] / len(all_res[ii]['force'])
175175
# ptr_data += '%7.3f %8.4f \n' % (vol, all_res[ii]['energy'] / len(all_res[ii]['force']))
176176

‎dpgen/auto_test/VASP.py

+2-2
Original file line numberDiff line numberDiff line change
@@ -140,8 +140,8 @@ def make_input_file(self,
140140
incar['ENCUT'] = cal_setting['encut']
141141

142142
if 'kspacing' in cal_setting:
143-
dlog.info("%s setting KSAPCING to %s" % (self.make_input_file.__name__, cal_setting['kspacing']))
144-
incar['KSAPCING'] = cal_setting['kspacing']
143+
dlog.info("%s setting KSPACING to %s" % (self.make_input_file.__name__, cal_setting['kspacing']))
144+
incar['KSPACING'] = cal_setting['kspacing']
145145

146146
if 'kgamma' in cal_setting:
147147
dlog.info("%s setting KGAMMA to %s" % (self.make_input_file.__name__, cal_setting['kgamma']))

‎dpgen/auto_test/common_equi.py

+69-37
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@
22
import os
33
import warnings
44
from monty.serialization import dumpfn
5+
from multiprocessing import Pool
56

67
import dpgen.auto_test.lib.crys as crys
78
import dpgen.auto_test.lib.util as util
@@ -12,6 +13,8 @@
1213
from distutils.version import LooseVersion
1314
from dpgen.dispatcher.Dispatcher import make_submission
1415
from dpgen.remote.decide_machine import convert_mdata
16+
from dpgen.auto_test.lib.utils import create_path
17+
1518
lammps_task_type = ['deepmd', 'meam', 'eam_fs', 'eam_alloy']
1619

1720

@@ -76,11 +79,11 @@ def make_equi(confs,
7679
poscar = os.path.abspath(os.path.join(ii, 'POSCAR'))
7780
if not os.path.exists(poscar):
7881
raise FileNotFoundError('no configuration for autotest')
82+
if os.path.exists(os.path.join(ii, 'relaxation', 'jr.json')):
83+
os.remove(os.path.join(ii, 'relaxation', 'jr.json'))
84+
7985
relax_dirs = os.path.abspath(os.path.join(ii, 'relaxation', 'relax_task')) # to be consistent with property in make dispatcher
80-
if os.path.exists(relax_dirs):
81-
dlog.warning('%s already exists' % relax_dirs)
82-
else:
83-
os.makedirs(relax_dirs)
86+
create_path(relax_dirs)
8487
task_dirs.append(relax_dirs)
8588
os.chdir(relax_dirs)
8689
# copy POSCARs to mp-xxx/relaxation/relax_task
@@ -111,6 +114,47 @@ def make_equi(confs,
111114
inter.make_input_file(ii, 'relaxation', relax_param)
112115

113116

117+
def worker(work_path,
118+
run_task,
119+
forward_common_files,
120+
forward_files,
121+
backward_files,
122+
mdata,
123+
inter_type):
124+
machine, resources, command, group_size = util.get_machine_info(mdata, inter_type)
125+
disp = make_dispatcher(machine, resources, work_path, [run_task], group_size)
126+
print("%s --> Runing... " % (work_path))
127+
128+
api_version = mdata.get('api_version', '0.9')
129+
if LooseVersion(api_version) < LooseVersion('1.0'):
130+
warnings.warn(f"the dpdispatcher will be updated to new version."
131+
f"And the interface may be changed. Please check the documents for more details")
132+
disp.run_jobs(resources,
133+
command,
134+
work_path,
135+
[run_task],
136+
group_size,
137+
forward_common_files,
138+
forward_files,
139+
backward_files,
140+
outlog='outlog',
141+
errlog='errlog')
142+
elif LooseVersion(api_version) >= LooseVersion('1.0'):
143+
submission = make_submission(
144+
mdata_machine=machine,
145+
mdata_resources=resources,
146+
commands=[command],
147+
work_path=work_path,
148+
run_tasks=run_task,
149+
group_size=group_size,
150+
forward_common_files=forward_common_files,
151+
forward_files=forward_files,
152+
backward_files=backward_files,
153+
outlog='outlog',
154+
errlog='errlog'
155+
)
156+
submission.run_submission()
157+
114158
def run_equi(confs,
115159
inter_param,
116160
mdata):
@@ -120,6 +164,11 @@ def run_equi(confs,
120164
for conf in confs:
121165
conf_dirs.extend(glob.glob(conf))
122166
conf_dirs.sort()
167+
168+
processes = len(conf_dirs)
169+
pool = Pool(processes=processes)
170+
print("Submit job via %d processes" % processes)
171+
123172
# generate a list of task names like mp-xxx/relaxation/relax_task
124173
# ...
125174
work_path_list = []
@@ -150,45 +199,28 @@ def run_equi(confs,
150199
if len(run_tasks) == 0:
151200
return
152201
else:
153-
# if LooseVersion()
154202
run_tasks = [os.path.basename(ii) for ii in all_task]
155203
machine, resources, command, group_size = util.get_machine_info(mdata, inter_type)
156204
print('%d tasks will be submited '%len(run_tasks))
205+
multiple_ret = []
157206
for ii in range(len(work_path_list)):
158207
work_path = work_path_list[ii]
159-
disp = make_dispatcher(machine, resources, work_path, [run_tasks[ii]], group_size)
160-
print("%s --> Runing... "%(work_path))
161-
162-
api_version = mdata.get('api_version', '0.9')
163-
if LooseVersion(api_version) < LooseVersion('1.0'):
164-
warnings.warn(f"the dpdispatcher will be updated to new version."
165-
f"And the interface may be changed. Please check the documents for more details")
166-
disp.run_jobs(resources,
167-
command,
168-
work_path,
169-
[run_tasks[ii]],
170-
group_size,
171-
forward_common_files,
172-
forward_files,
173-
backward_files,
174-
outlog='outlog',
175-
errlog='errlog')
176-
elif LooseVersion(api_version) >= LooseVersion('1.0'):
177-
submission = make_submission(
178-
mdata_machine=machine,
179-
mdata_resource=resources,
180-
commands=[command],
181-
work_path=work_path,
182-
run_tasks=run_tasks,
183-
group_size=group_size,
184-
forward_common_files=forward_common_files,
185-
forward_files=forward_files,
186-
backward_files=backward_files,
187-
outlog = 'outlog',
188-
errlog = 'errlog'
189-
)
190-
submission.run_submission()
191208

209+
ret = pool.apply_async(worker, (work_path,
210+
run_tasks[ii],
211+
forward_common_files,
212+
forward_files,
213+
backward_files,
214+
mdata,
215+
inter_type,
216+
))
217+
multiple_ret.append(ret)
218+
pool.close()
219+
pool.join()
220+
for ii in range(len(multiple_ret)):
221+
if not multiple_ret[ii].successful():
222+
raise RuntimeError("Task %d is not successful! work_path: %s " % (ii, work_path_list[ii]))
223+
print('finished')
192224

193225
def post_equi(confs, inter_param):
194226
# find all POSCARs and their name like mp-xxx

‎dpgen/auto_test/common_prop.py

+10-21
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,7 @@
1515
from dpgen.dispatcher.Dispatcher import make_dispatcher
1616
from dpgen.dispatcher.Dispatcher import make_submission
1717
from dpgen.remote.decide_machine import convert_mdata
18+
from dpgen.auto_test.lib.utils import create_path
1819
lammps_task_type = ['deepmd', 'meam', 'eam_fs', 'eam_alloy']
1920

2021

@@ -73,10 +74,7 @@ def make_property(confs,
7374
path_to_equi = os.path.join(ii, 'relaxation', 'relax_task')
7475
path_to_work = os.path.join(ii, property_type + '_' + suffix)
7576

76-
if os.path.exists(path_to_work):
77-
dlog.warning('%s already exists' % path_to_work)
78-
else:
79-
os.makedirs(path_to_work)
77+
create_path(path_to_work)
8078

8179
prop = make_property_instance(jj)
8280
task_list = prop.make_confs(path_to_work, path_to_equi, do_refine)
@@ -112,6 +110,7 @@ def run_property(confs,
112110
conf_dirs.sort()
113111
task_list = []
114112
work_path_list = []
113+
multiple_ret = []
115114
for ii in conf_dirs:
116115
sepline(ch=ii, screen=True)
117116
for jj in property_list:
@@ -159,7 +158,7 @@ def run_property(confs,
159158
all_task = tmp_task_list
160159
run_tasks = util.collect_task(all_task, inter_type)
161160
if len(run_tasks) == 0:
162-
return
161+
continue
163162
else:
164163
ret = pool.apply_async(worker, (work_path,
165164
all_task,
@@ -169,23 +168,13 @@ def run_property(confs,
169168
mdata,
170169
inter_type,
171170
))
172-
# run_tasks = [os.path.basename(ii) for ii in all_task]
173-
# machine, resources, command, group_size = util.get_machine_info(mdata, inter_type)
174-
# disp = make_dispatcher(machine, resources, work_path, run_tasks, group_size)
175-
# disp.run_jobs(resources,
176-
# command,
177-
# work_path,
178-
# run_tasks,
179-
# group_size,
180-
# forward_common_files,
181-
# forward_files,
182-
# backward_files,
183-
# outlog='outlog',
184-
# errlog='errlog')
171+
multiple_ret.append(ret)
185172
pool.close()
186173
pool.join()
187-
if ret.successful():
188-
print('finished')
174+
for ii in range(len(multiple_ret)):
175+
if not multiple_ret[ii].successful():
176+
raise RuntimeError("Job %d is not successful!" % ii)
177+
print('%d jobs are finished' % len(multiple_ret))
189178

190179

191180
def worker(work_path,
@@ -215,7 +204,7 @@ def worker(work_path,
215204
elif LooseVersion(api_version) >= LooseVersion('1.0'):
216205
submission = make_submission(
217206
mdata_machine=machine,
218-
mdata_resource=resources,
207+
mdata_resources=resources,
219208
commands=[command],
220209
work_path=work_path,
221210
run_tasks=run_tasks,

‎dpgen/data/gen.py

+5-2
Original file line numberDiff line numberDiff line change
@@ -464,7 +464,7 @@ def make_vasp_md(jdata, mdata) :
464464

465465
for ii in sys_ps :
466466
for jj in scale :
467-
for kk in range(pert_numb) :
467+
for kk in range(pert_numb+1) :
468468
path_work = path_md
469469
path_work = os.path.join(path_work, ii)
470470
path_work = os.path.join(path_work, "scale-%.3f" % jj)
@@ -534,6 +534,8 @@ def coll_vasp_md(jdata) :
534534
#dlog.info("md_nstep", md_nstep)
535535
if nforce == md_nstep :
536536
valid_outcars.append(outcar)
537+
elif md_nstep == 0 and nforce == 1 :
538+
valid_outcars.append(outcar)
537539
else:
538540
dlog.info("WARNING : in directory %s nforce in OUTCAR is not equal to settings in INCAR"%(os.getcwd()))
539541
arg_cvt = " "
@@ -767,13 +769,14 @@ def gen_init_bulk(args) :
767769
dlog.info("Current stage is 1, relax")
768770
create_path(out_dir)
769771
shutil.copy2(args.PARAM, os.path.join(out_dir, 'param.json'))
772+
skip_relax = jdata['skip_relax']
770773
if from_poscar :
771774
make_super_cell_poscar(jdata)
772775
else :
773776
make_unit_cell(jdata)
774777
make_super_cell(jdata)
775778
place_element(jdata)
776-
if args.MACHINE is not None:
779+
if args.MACHINE is not None and not skip_relax:
777780
make_vasp_relax(jdata, mdata)
778781
run_vasp_relax(jdata, mdata)
779782
else:

‎dpgen/dispatcher/Dispatcher.py

+9-3
Original file line numberDiff line numberDiff line change
@@ -346,11 +346,17 @@ def make_dispatcher(mdata, mdata_resource=None, work_path=None, run_tasks=None,
346346
def make_submission(mdata_machine, mdata_resources, commands, work_path, run_tasks, group_size,
347347
forward_common_files, forward_files, backward_files, outlog, errlog):
348348

349-
machine = Machine.load_from_dict(mdata_machine)
350-
resources = Resources.load_from_dict(mdata_resources)
351-
352349
if mdata_machine['local_root'] != './':
353350
raise RuntimeError(f"local_root must be './' in dpgen's machine.json.")
351+
352+
abs_local_root = os.path.abspath('./')
353+
354+
abs_mdata_machine = mdata_machine.copy()
355+
abs_mdata_machine['local_root'] = abs_local_root
356+
357+
machine = Machine.load_from_dict(abs_mdata_machine)
358+
resources = Resources.load_from_dict(mdata_resources)
359+
354360

355361
command = "&&".join(commands)
356362

‎dpgen/dispatcher/LocalContext.py

+7-1
Original file line numberDiff line numberDiff line change
@@ -117,7 +117,13 @@ def download(self,
117117
pass
118118
elif (os.path.exists(rfile)) and (not os.path.exists(lfile)) :
119119
# trivial case, download happily
120-
shutil.move(rfile, lfile)
120+
# If the file to be downloaded is a softlink, `cp` should be performed instead of `mv`.
121+
# Otherwise, `lfile` is still a file linked to some original file,
122+
# and when this file's removed, `lfile` will be invalid.
123+
if os.path.islink(rfile):
124+
shutil.copyfile(rfile,lfile)
125+
else:
126+
shutil.move(rfile, lfile)
121127
elif (os.path.exists(rfile)) and (os.path.exists(lfile)) :
122128
# both exists, replace!
123129
dlog.info('find existing %s, replacing by %s' % (lfile, rfile))

0 commit comments

Comments
 (0)