file format plugin: how to keep data persistent

[jodyxha]

I have a plugin to import data from our specific HDF format into VisIt so we can visualize the various data sets stored there.
For this to work we collect the paths (i.e. the sequience of HDF subgroups to move through to reache the data set) to all numeric data sets we find in avtQDFFileFormat:::PopulateDatabaseMetaData() and save this information in a map (name => path)member variable, so that we can access the data set in the call to avtQDFFileFormat:::GetVar().

This works fine when i visualize datasets from a single file.

But when i load an entire block of files in order to make an animation, only the first plot is done correctly, while the following plots display some random data.

I located the problem: when i d plot the datasets from single files, avtQDFFileFormat:::PopulateDatabaseMetaData() is called every time i do a plot, so the map with the path information is created and is available.
In the case of the animation, the avtQDFFileFormat:::PopulateDatabaseMetaData() is called once before getting the data, so the first plot is done correctly. Each subsequent call to GetVar() happens in a new instance of avtQDFFileFormat (i printed the this pointer at the beginning of GetVar()), but apparently PopulateDatabaseMetaData() is not called for these instances, so this means the map with the path information is empty in each of the new instances.

I see two possibilities to fix this:

1. somehow attach the map to a vtk data structure that is persistent (i,e, is the same for instances of avtQDFFileFormat ).
2. make sure PopulateDatabaseMetaData() is called in each new instance during an animation.

The first possibility could work - in my pie plot plugin i also used a vtk object to "smuggle" data into a new instance. The question here is, is there such a data structure which i could add my data to in the first instantiation if the class, and retrieve it from this same data structure in the new instance?

As for variant 2: is there a way to find out whether we are currently displaying an animation? if yes, i could call the method PopulateDatabaseMetaData() (or at least the part of it collecting the paths) at the beginning of GetVar()

So my question is, is any of those two variants possible?
Thank you

[jodyxha]

I found a solution:
whenever GetVar() is called and the map with the path information is empty, i call ``PopulateDatabaseMetaData() which collects the information again.

[markcmiller86]

What kind of plugin are you developing. There are four possibilities SDST, SDMT, MDST, MDMT where S=single, M=multi, D=domain, T= time. The difference is in what VisIt delegates to the plugin to know about and handle and what VisIt must know about and handle itself in the way of domains and timesteps.

I worry about calling PopulateDatabaseMetaData() from within a GetVar() call for parallel use cases because I think the former is supposed to be collective while the latter is definitely NOT collective and so could easily lead to parallel deadlock. In other words, when PopulateDatabaseMetaData() is called (from VisIt) it is done so on all MPI ranks simultaneously. However, there is no such guarantee with GetVar() primarily because not all MPI ranks are required to have domains participating in the call to get a variable.

I need to give your initial description some additional thought to provide best guidance. That said, if each file is a different timestep and you need PopulateDatabaseMetaData() to be called for each, then having your plugin return false for HasInvariantMetadata() and HasInvariantSIL() may be sufficient.

[jodyxha]

Thank you for your reply.

My avtQDFFileFormat is derived from avtSTSDFileFormat.

I must admit i did not feel very comfortable wit placing PopulateDatabaseMetaData() in Getvar(), but at the time it seemed to be the easiest way out.

I had noticed that the avtDatabaseMetaData object passed to PopulatDataBaseMetaData is always identical (same pointer) in all instances of avtQDFFileFormat. I thought about somehow attaching a string containing the serialized contents of the map (maybe using SetDatabaseComment() and GetDatabaseComment()).

I will give your suggestions regarding HasInvariantMetadata() and HasInvariantSIL() a try. (What does SIL stand for?)

[jodyxha]

I added the line

virtual bool HasInvariantMetaData(void) const { return false; };

to avtQDFFileFormat.h and now it works perfectly without GetVar() explicitely calling PopulateDatabaseMetaData().

Thanks!

[markcmiller86]

Ok, good. Be aware that you may also need

virtual bool HasInvariantSIL(void) const { return false; };

also.

I am not sure why we chose to differentiate between MetaData and SIL because there is really only one way a SIL is created and that is via information obtained in MetaData so the two sort of go hand in hand.

SIL is an abbreviation for Subset Inclusion Lattice and has to do with how your data is decomposed into subets for things like domains, materials, blocks, etc and if those vary with time, HasInvariantSIL() should return false.