Upstream your frontend?

[ngoldbaum]

Hi!

I'm a yt developer who stumbled on this repo via a github search.

First, can you add a note that this is not the canonical repo for yt? Since we're not on github a naive searcher might get confused.

Second, I'd like to engage with you about upstreaming your frontend. This will not only make it easier for users of openPMD to get access to your frontend, but you will also be able to take advantage of our testing infrastructure and get help maintaining the frontend and avoiding bitrot from upstream changes.

If you're interested, I'd encourage you to contact our development mailing list:

http://lists.spacepope.org/listinfo.cgi/yt-dev-spacepope.org

Or if you're not comfortable with that, feel free to contact me offline (ngoldbau@illinois.edu). I'm very happy to help out with mercurial or yt code issues.

Best,

Nathan Goldbaum

[ax3l]

@ngoldbaum thanks for contacting us, wuhu! ✨

Our plan is definitely to send the final version to upstream, absolutely. This repo is only a development repo until we have a first usable version.

Unfortunately, the repo is progressing a bit slower then I hoped since the work is done by undergraduates and interns that are working at our lab(s) from time to time (readme).

Our current status (#6) is that we can finally read 2D and 3D meshes of arbitrary user-defined size (regularly spaced, Cartesian) but parallel particle reading seems not to work. If you would be willing to help a bit with reviewing (advising) the code this would be fantastic! ☀️ I have to admit the yt frontent documentation is a bit rare :)

I motivated our students to engage with the mailing list (Daniel in 2015) but it is also a learning progress for them.

I know we have new interns starting this month and I might be able to catch some for this task. Nevertheless, I am working this and next week on openPMD related topics anyway at LBNL, so if you are available we could also meet via video chat and push the implementation a bit for a day or so.

Thanks again for getting in contact!
Axel

[ax3l]

CCing @RemiLehe in case he is interested to participate.

[ax3l]

oh on your initial question:

First, can you add a note that this is not the canonical repo for yt? Since we're not on github a naive searcher might get confused

I tried to avoid confusion by giving the repository description from the beginning as

openPMD-yt: [WIP] Reader for yt-project.org (draft)

Additionally, the main branch is called readme where only a single, descriptive README.md is added. I hope that is clear enough since this is the entry point, otherwise I can add more links to the README.md if you prefer. Update: just added an other TL;DR section in the readme :)

[ngoldbaum]

Additionally, the main branch is called readme where only a single, descriptive README.md is added. I hope that is clear enough since this is the entry point, otherwise I can add more links to the README.md if you prefer. Update: just added an other TL;DR section in the readme :)

Thank you, it is much clearer now!

Our current status (#6) is that we can finally read 2D and 3D meshes of arbitrary user-defined size (regularly spaced, Cartesian) but parallel particle reading seems not to work. If you would be willing to help a bit with reviewing (advising) the code this would be fantastic! ☀️ I have to admit the yt frontent documentation is a bit rare :)

Yes, the how-to-write-a-frontend docs is on our list of things to improve. If you can open issues with suggestions for specific improvements or requests for clarification that will also help identifying concrete things to improve. As the person writing a frontend you are in a far better position to tell us how to improve the docs than we are as people who are already familar with yt.

Can you share a little bit more detail about how openPMD stores particles? In principle each process in a domain-decomposed MPI-parallel yt job should only need to read the particles it needs to. You mentioned that your data are uniform resolution --- are you splitting up the data into tiles? We do a similar trick in the FITS frontend to allow reading in large PPV data cubes.

[ax3l]

If you can open issues with suggestions for specific improvements or requests for clarification that will also help identifying concrete things to improve.

Good point, shall I open questions directly in the main bitbucket repo?

Can you share a little bit more detail about how openPMD stores particles? [...] are you splitting up the data into tiles?

Yes we implement tiling to allow parallel writing (e.g., via parallel HDF5) and parallel processing.
Here is how particles are implemented in openPMD files:

• grouped per species in their own directory as "struct of arrays", e.g. all electron attributes will be found under
  • /data/0/particles/electrons/
• within that directory/group we place their records, which are scalar or vector attributes in long, 1D arrays, e.g.,
  • colorful example
  • .../particles/electrons/position/x, .../y, .../z
  • .../particles/electrons/position/charge
• to make them readable and writable in parallel, we introduce a tiling structure (per MPI rank that wrote the species), called particlePatches
• you can imagine each particle patch as a line in the following table:

patch (below) \ record (->)	numParticles	numParticlesOffset	offset (e.g., x,y,z)	extent (e.g., x,y,z)
(first and only)	global num particles	0	x<=min(position.x), y <= min(...), \ <= ...	x >= max(position.x), y >= ..., z >= ...

• particle patches again implemented as array of struct with numParticles and numParticlesOffset as scalar (1D) records and offset and extent being vector records (representation as above example for position/x|y|z and charge: at the end all particle data structures are 1D arrays but they get grouped in a directory if an attribute has several components such as x, y, z...).

[ngoldbaum]

New issues should be opened here: https://bitbucket.org/yt_analysis/yt/issues/new

I guess the key is the teach your frontend about this particlePatch capability. Are the mesh fields decomposed in the same patches? Does your frontend decompose the data into patches or just create one uniform resolution grid object? If it's the latter that might explain your parallelization issues.

It would probably be a good idea to discuss this further on yt-dev, where you will get the attention of more people than just me :)

[ax3l]

I guess the key is the teach your frontend about this particlePatch capability.

Absolutely, maybe we can then leach a bit from your FITS/PPV implementation of tiled particles :)

Are the mesh fields decomposed in the same patches?

yes, we do a spatial domain decomposition in our codes.

Does your frontend decompose the data into patches or just create one uniform resolution grid object?

Nevertheless, it is not guaranteed that a file contains both particles and mesh fields. Luckily, this is also unnecessary since the final fields are "contiguously" stored in HDF5 (at least virtually it looks contiguous due to internal chunking using MPI-I/O) so that a reader can choose the new decomposition on it's own.

We are also implementing the same structure in ADIOS where one can actually query the underlying chunking (in "progress groups") to remove the transparency layer. But this would be an additional implementation we can do after parallel HDF5 is up and working.

If it's the latter that might explain your parallelization issues.

Actually we just don't understand the master concept how to tell our parallelization to yt. Performance-wise our codes scale pretty well, even with I/O (full size Titan/ORNL) :)

It would probably be a good idea to discuss this further on yt-dev, where you will get the attention of more people than just me :)

Okey, let us continue on the list :)
I will first open some issues on the repo about the documentation if that is ok :)

[ax3l]

Frontend documentation questions migrated to
https://bitbucket.org/yt_analysis/yt/issues/1210

[PrometheusPi]

@ax3l What is the status of merging this openPMD-yt fronted to yt?