Issue with finding the dependencies among the sub-tasks of a primary calculation using pymatgen API. #3884
Issue with finding the dependencies among the sub-tasks of a primary calculation using pymatgen API.
#3884
-
|
Dear pymatgen Development Team, I want to retrieve task dependencies and their corresponding directories using the pymatgen API. However, I am encountering an issue where the Here is a snippet of the code I am using: from mp_api.client import MPRester
material_id = "mp-1183063"
# Get all task IDs related to the given material ID
with MPRester() as mpr:
tasks = mpr.materials.search(material_ids=[material_id], fields=["task_ids"])
# Print all task IDs
print("All related task IDs:")
task_ids = tasks[0].task_ids
for task_id in task_ids:
print(task_id)
# Analyze the dependencies between different task IDs
print("\nDependencies between task IDs:")
path_to_mp_id = {}
for task_id in task_ids:
with MPRester() as mpr:
selected_task = mpr.materials.tasks.search(task_ids=[task_id], fields=["task_id", "calcs_reversed"])
if selected_task:
task = selected_task[0]
calcs_reversed = getattr(task, "calcs_reversed", [])
if calcs_reversed:
print(f"\nTask ID {task_id} reverse dependencies:")
for calc in calcs_reversed:
reverse_task_dir = calc["dir_name"] if "dir_name" in calc else "dir_name field not found"
print(f" - {reverse_task_dir}")
path_to_mp_id[reverse_task_dir] = None # Initialize dictionary
else:
print(f"\nTask ID {task_id} has no reverse dependencies.")
else:
print(f"\nNo data found for task_id: {task_id}")
# Get detailed information for all related tasks
all_task_details = []
with MPRester() as mpr:
for task_id in task_ids:
task_details = mpr.materials.tasks.search(task_ids=[task_id], fields=["*"])
all_task_details.extend(task_details)
# Print detailed information for all tasks
print("\nDetailed information for all tasks:")
for task_detail in all_task_details:
print(task_detail) # Directly print MPDataDoc object
# Find the mp-id corresponding to each path
print("\nFinding mp-id for each path:")
for path in path_to_mp_id.keys():
found = False
for task_detail in all_task_details:
if hasattr(task_detail, "dir_name") and task_detail.dir_name == path:
path_to_mp_id[path] = task_detail.task_id
found = True
break
if not found:
print(f"Path not found: {path}")
# Print the mapping from path to mp-id
print("\nMapping from path to mp-id:")
for path, mp_id in path_to_mp_id.items():
print(f"Path: {path} -> mp-id: {mp_id}")The output I receive indicates that the Could you please help me understand why the Thank you for your time and assistance. Best regards, |
Beta Was this translation helpful? Give feedback.
Answered by
QuantumChemist
Jun 17, 2024
Replies: 1 comment 12 replies
-
|
Dear @hongyi-zhao , when you print the for task_detail in all_task_details:
print(task_detail.dir_name)You will get Alternatively, you can also print all the entries: for task_detail in all_task_details:
for item in task_detail:
print(item)which will get you ('builder_meta', None)
('nsites', None)
('elements', None)
('nelements', None)
('composition', None)
('composition_reduced', None)
('formula_pretty', None)
('formula_anonymous', None)
('chemsys', None)
('volume', None)
('density', None)
('density_atomic', None)
('symmetry', None)
('tags', None)
('dir_name', None)
('state', None)
('calcs_reversed', None)
('structure', None)
('task_type', None)
('task_id', None)
('orig_inputs', None)
('input', None)
('output', None)
('included_objects', None)
('vasp_objects', None)
('entry', None)
('task_label', None)
('author', None)
('icsd_id', None)
('transformations', None)
('additional_json', None)
('custodian', None)
('analysis', None)
('last_updated', None)
('fields_not_requested', ['builder_meta', 'nsites', 'elements', 'nelements', 'composition', 'composition_reduced', 'formula_pretty', 'formula_anonymous', 'chemsys', 'volume', 'density', 'density_atomic', 'symmetry', 'tags', 'dir_name', 'state', 'calcs_reversed', 'structure', 'task_type', 'task_id', 'orig_inputs', 'input', 'output', 'included_objects', 'vasp_objects', 'entry', 'task_label', 'author', 'icsd_id', 'transformations', 'additional_json', 'custodian', 'analysis', 'last_updated'])Again, every entry is |
Beta Was this translation helpful? Give feedback.
12 replies
-
|
if you look closer, you'll see that the For the case of multiple dependencies: When comparing the IDs and the calculation types they belong to, it appears to me that the order of the IDs in the list has been the sequence of the calculation executions, but honestly I wouldn't know why one would do a R2SCAN structure optimization before a PBEsol structure optimization... |
Beta Was this translation helpful? Give feedback.
-
|
Or in other words, this has been the sequence of the calculation. But I'm not 100% sure. |
Beta Was this translation helpful? Give feedback.
-
|
Now, I construct a dependency graph and then use topological sorting to ensure the dependencies are in the correct order, as shown below: from mp_api.client import MPRester
from collections import defaultdict, deque
def standardize_task_type(task_type):
"""Standardize the task type string"""
return str(task_type).replace('_', ' ').title()
def analyze_dependencies(item):
task_types = item.task_types
calc_types = item.calc_types
run_types = item.run_types # Assuming run_types is available in the item
# Print task types, calculation types, and run types for debugging
print("\nTask Types:")
for task_id, task_type in task_types.items():
print(f"Task ID: {task_id}, Task Type: {task_type}")
print("\nCalculation Types:")
for task_id, calc_type in calc_types.items():
print(f"Task ID: {task_id}, Calculation Type: {calc_type}")
print("\nRun Types:")
for task_id, run_type in run_types.items():
print(f"Task ID: {task_id}, Run Type: {run_type}")
dependencies = defaultdict(list)
for task_id, task_type in task_types.items():
dependencies[task_id] = []
# Determine dependencies based on task type, calculation type, and run type
task_ids = list(task_types.keys())
for i in range(len(task_ids)):
current_task_id = task_ids[i]
current_task_type = standardize_task_type(task_types[current_task_id])
current_calc_type = calc_types.get(current_task_id, None)
current_run_type = run_types.get(current_task_id, None)
for j in range(i):
previous_task_id = task_ids[j]
previous_task_type = standardize_task_type(task_types[previous_task_id])
previous_calc_type = calc_types.get(previous_task_id, None)
previous_run_type = run_types.get(previous_task_id, None)
# Determine dependencies based on task type, calculation type, and run type
if current_task_type in ['Nscf Line', 'Nscf Uniform'] and previous_task_type in ['Structure Optimization', 'Static']:
dependencies[current_task_id].append(previous_task_id)
elif current_task_type == 'Static' and previous_task_type == 'Structure Optimization':
dependencies[current_task_id].append(previous_task_id)
# Function to perform topological sort
def topological_sort(dependencies):
in_degree = {u: 0 for u in dependencies} # Initialize in-degree of each node to 0
for u in dependencies:
for v in dependencies[u]:
in_degree[v] += 1 # Calculate in-degree of each node
queue = deque([u for u in dependencies if in_degree[u] == 0]) # Nodes with in-degree 0
topo_order = []
while queue:
u = queue.popleft()
topo_order.append(u)
for v in dependencies[u]:
in_degree[v] -= 1
if in_degree[v] == 0:
queue.append(v)
return topo_order
# Perform topological sort to get the correct order
sorted_tasks = topological_sort(dependencies)
# Print dependencies in topologically sorted order
print("\nDependencies:")
for task_id in sorted_tasks:
print(f"Task ID: {task_id}")
print(f" Depends on: {dependencies[task_id]}")
material_id = "mp-1183063"
with MPRester() as mpr:
tasks = mpr.materials.search(material_ids=[material_id])
for item in tasks:
analyze_dependencies(item)The result is as follows: What do you think of the above result? |
Beta Was this translation helpful? Give feedback.
-
|
hmm.... you basically produced the same result. At this point, I'm not quite sure if you can extract the sequential order of the jobs using the MP API. The entries of the MPIDs are according to "last_updated": tasks = mpr.materials.search(material_ids=[material_id])
for task in tasks:
for mpid, dft_type in task.calc_types.items():
print(mpid, dft_type, mpr.materials.tasks.search(task_ids=[mpid])[0].last_updated)with the output: But regarding the correct sequence of DFT calculations, why don't you rely on the atomate2 VASP workflows, e.g. the |
Beta Was this translation helpful? Give feedback.
Answer selected by
hongyi-zhao
-
I have a question: Are the MP website databases built via atomate2? Another question: Should I use atomate or atomate2?
RelaxBandStructureMaker will do a one-shot calculation for this type of job. |
Beta Was this translation helpful? Give feedback.
-
|
See https://matsci.org/t/using-atomate-and-atomate2-in-the-same-virtual-environment/56000 for the related discussion. |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
hmm.... you basically produced the same result.
At this point, I'm not quite sure if you can extract the sequential order of the jobs using the MP API.
The entries of the MPIDs are according to "last_updated":
with the output: