Repository navigation
Pickle: Mesh points at wrong series after deserializing #1919
Description
Describe the bug
When serializing and deserializing a series object in python and doing the same for a MeshRecord (or MeshRecordComponent ?) both will successfully deserialize. But, although calling load_chunk on the deserialized mesh will work, calling flush on the deserialized series won't actually load the data.
There is no proper error message, but the buffer will contain random data. When serializing with some parallelization libraries like joblib workers can fail with messages like:
Fatal Python error: gilstate_tss_set: failed to set current tstate (TSS)
Python runtime state: finalizing (tstate=0x00000000017f9e18)
To Reproduce
uv run deserialize_bug.py
deserialize_bug.py
Expected behavior
Since the MeshRecordComponent and MeshRecord both have add_pickle both methods in the example should work
Software Environment
See attached file
Activity
pordyna commentedon Jul 22, 2026
@franzpoeschel this is not really urgent, just pinging in case you haven't seen it yet.
franzpoeschel commentedon Jul 22, 2026
Ah thanks, I had forgotten about it
Will check
franzpoeschel commentedon Jul 22, 2026
I see that something is going wrong, but I have not yet understood why
Will continue
franzpoeschel commentedon Jul 22, 2026
Okaayyy, I understand why this happens, but this might get ugly
Our serialization/deserialization strategy is to save only the filename + the location of the object within that file (e.g. /400/meshes/rho), and then upon deserialization init a new Series and point it to the right object again.
Consequence: After deserializing,
series,itandmeshare three entirely different openPMD Series objects. Meaning thatseries.flush()won't do things that you enqueued intomesh.You can work around this by adding a line
mesh.series_flush(), but the pitfall remains.. No clue how to easily solve that at the moment[-]Mesh points at wrong series after deserializing[/-][+]Pickle: Mesh points at wrong series after deserializing[/+]pordyna commentedon Jul 24, 2026
If there is no easy solution maybe you should think about removing pickle capabilities from everything except for series. And just force the user to rebuild everything from series.
franzpoeschel commentedon Aug 3, 2026
That would be the last resort, and we will have to consider it if there is no other solution
As this would break a lot of production use cases, I would like to avoid it
My current idea would be to deserialize additionally an ID for the Series, such that readers will know which Series objects to associate with each other. Will have to try if this can work
test: Add pickle cache tests for multiple Series with multiple refere…