Add device reader api to harp-data - #15
Merged
Merged
Conversation
Member
Author
|
@glopesdev one major thing to solve in the API is where to get the runtime registers from. Currently, they exist in |
bruno-f-cruz
force-pushed
the
feat-add-device-reader
branch
from
July 26, 2026 16:51
d5d7ef5 to
4ff7f94
Compare
bruno-f-cruz
force-pushed
the
feat-runtime-register-binding
branch
from
July 26, 2026 16:57
de70211 to
933071c
Compare
bruno-f-cruz
force-pushed
the
feat-add-device-reader
branch
from
July 26, 2026 16:57
4ff7f94 to
72522ae
Compare
glopesdev
force-pushed
the
feat-add-device-reader
branch
from
August 10, 2026 18:30
72522ae to
94a1c70
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a device-driven dataset reader for
harp.dataCloses #10. Stacked on #13 (runtime generation of the device interface).
What this adds
A
DatasetReaderthat reads a de-multiplexed Harp dataset folder into pandas DataFrames, driven by a generated device. You give it a device class and a folder, then read a register by class or by address:harp.device.Devicesubclass and reads itsREGISTER_MAPand class name on demand — the same devices produced by Add runtime generation of device interface #13.harp-deviceis now a dependency ofharp-data.<DeviceName>_<address>.bin; when a register was logged as several<DeviceName>_<address>_<suffix>.binchunks they are concatenated in filename order.name=overrides the file prefix.resolverargument (FileNameResolver) lets you swap in an alternative on-disk layout;default_file_resolverimplements the Harp file format. This is the seam for future sources (e.g. remote/object storage).read()inspects the first frame's payload-type bit (_TIMESTAMP_FLAG, imported fromharp.protocol) to decide whether to build a time index, so you don't have to know per-register whether logging was timestamped.timestamp=True/Falseforces it.Scope is intentionally limited to reading an already de-multiplexed folder — stream de-multiplexing is out of scope for this PR, but we can add it in the future using the framer from the device class. I just felt the API surface would be unecessarily large.
Epoch /
"Time"index parity with harp-pythonparse_to_dataframegained anepochargument and now places the Harp time on the DataFrame index named"Time", matching harp-python: noepoch→ float seconds (pd.Index);epochgiven (e.g.REFERENCE_EPOCH, exported here) → absoluteDatetimeIndex. The old"timestamp"column is gone in favor of this index.harp-python deprecation and dropping
reader.Register.read()harp-python's reader exposed per-register readers as attributes, so you read data with
reader.Register.read(). This PR deliberately does not reproduce that API. pyharp's pattern is a method on the reader —reader.read(register)— and keeping the attribute-style accessor (via__getattr__) would mean carrying a second, redundant surface just for source compatibility.The plan: harp-python adopts this reader for one final release, emitting a
DeprecationWarningon the oldreader.Register.read()access, and is then archived. There is therefore no compatibility shim here — no__getattr__layer, noreader.Register.read(). New code usesreader.read(...), which follows the pyharp convention rather than the historical harp-python one.Tests
tests/data/test_dataset.pybuilds mock dataset folders intmp_path(cleaned up per test) usingRegisterBase.format_bulkto generate real Harp frames, then exercises: read by class and by address, automatic timestamp detection,read_allkeyed by register name, multi-chunk suffix concatenation, custom resolver, name override, missing-file / unknown-address errors, float-seconds vs absoluteDatetimeIndex(epoch), and an end-to-end "read every register from adevice.yml-generated device".