Reading BP5 VariableBased Encoding must be frictionless

[ax3l]

If we want to ship openPMD BP5 + variable based encoding as new defaults, e.g., in ImpactX, then the read of those files must be friction less.

As of 0.17.1, such output must read with Access.read_linear / read_iterations() and random-access reads on variable-based series miss iterations (openPMD-api itself warns about this). This is super cumbersome for users, and especially our implementations in openPMD-viewer and ParaView, because one cannot seamless switch backends and encodings and has to add further options during read. Even more problematic, information that ones needs to have before even opening the series.

• https://github.com/openPMD/openPMD-viewer/blob/1.11/openpmd_viewer/openpmd_timeseries/data_reader/data_reader.py#L117-L120
• https://gitlab.kitware.com/paraview/paraview/-/blob/c55664abd5949a6790fc4b23da34aef810bbc265/Wrapping/Python/paraview/algorithms/openpmd.py#L388-392

I think the logic we need to implement is that if a user opens a BP5 series that has variable based encoding with the wrong Access. mode, we switch to the right one. Probably even silently, because there is no way for a user to know upfront before opening the a .bp5 file if it uses that encoding.
Then, we need to make sure they can query at runtime if the current access mode supports random-access or only linear access to iterations, so we can branch accordingly when opening iterations.

What do you think? @franzpoeschel @guj @pnorbert

[franzpoeschel]

As of 0.17.1, such output must read with Access.read_linear / read_iterations() and random-access reads on variable-based series miss iterations (openPMD-api itself warns about this)

Can you give me an example, please? That warning is outdated and should have been removed
Is this still the problem with attribute-only Iterations? If yes, that is a fundamental issue with the data model of ADIOS2. Has always been an issue, we are running into it just now, because this is the first time that we actually use ADIOS2 the way it is supposed to be used.
If we need variable-based encoding for attribute-only Iterations, that is an edge case for which we need to find a separate solution, without throwing out the majority of use cases for which variable-based encoding works fine

Ideas:

• Use scalar variables for mutable metadata. Much work, we tried this in the past, would change our entire data layout over an edge case
• Use one single scalar variable to anchor the iterations. We could even do that by default, turning /data/snapshots into a scalar variable.
• Bug the ADIOS2 devs to detect this situation at write time. Engine::Close() would then have a check to verify if the Step would even be visible to readers. This is not an openPMD-specific problem. If you write only attributes, then ADIOS2 knows no steps.

I think the logic we need to implement is that if a user opens a BP5 series that has variable based encoding with the wrong Access. mode, we switch to the right one. Probably even silently

No
The reason why we use variable-based encoding as a default in some places now is that both access modes finally have full support; there is no wrong Access for opening a Series in variable-based encoding
Variable-based encoding is the default now in places where 0.16 would have chosen groupbased encoding, which has far greater issues

Then, we need to make sure they can query at runtime if the current access mode supports random-access or only linear access to iterations, so we can branch accordingly when opening iterations.

This would require tentatively opening the ADIOS2 file in one access mode, detecting that the access mode is not ideal, closing and reopening again. Shenanigans like these are the reason why 0.17 now clearly distinguishes access modes chosen up front by the user, and tries to fully support both of them.

For the moment, file-based encoding is the only production quality encoding for ADIOS2. Variable-based encoding is now full-featured, but still beta; group-encoding is fundamentally bad.

[franzpoeschel]

openPMD-api itself warns about this

Waaaiiiit, that warning HAS been removed. Can you make sure that users actually have recent versions installed in their analysis workflows? With READ_RANDOM_ACCESS in openPMD-api 0.16, a variable-based Series will show only one Iteration. Fixing this is one of the major new features of 0.17.