Skip to content

Add device reader api to harp-data - #15

Merged
glopesdev merged 4 commits into
mainfrom
feat-add-device-reader
Aug 10, 2026
Merged

Add device reader api to harp-data#15
glopesdev merged 4 commits into
mainfrom
feat-add-device-reader

Conversation

@bruno-f-cruz

Copy link
Copy Markdown
Member

Add a device-driven dataset reader for harp.data

Closes #10. Stacked on #13 (runtime generation of the device interface).

What this adds

A DatasetReader that reads a de-multiplexed Harp dataset folder into pandas DataFrames, driven by a generated device. You give it a device class and a folder, then read a register by class or by address:

from harp.data import DatasetReader

reader = DatasetReader(Behavior, "session.harp")
df  = reader.read(AnalogData)    # by register class
df  = reader.read(44)            # by address
all = reader.read_all()          # {register_name: DataFrame}
  • Device-driven. The reader takes a generated harp.device.Device subclass and reads its REGISTER_MAP and class name on demand — the same devices produced by Add runtime generation of device interface #13. harp-device is now a dependency of harp-data.
  • Harp file-format layout by default. Files are resolved as <DeviceName>_<address>.bin; when a register was logged as several <DeviceName>_<address>_<suffix>.bin chunks they are concatenated in filename order. name= overrides the file prefix.
  • Pluggable file resolution. A resolver argument (FileNameResolver) lets you swap in an alternative on-disk layout; default_file_resolver implements the Harp file format. This is the seam for future sources (e.g. remote/object storage).
  • Automatic timestamp detection. read() inspects the first frame's payload-type bit (_TIMESTAMP_FLAG, imported from harp.protocol) to decide whether to build a time index, so you don't have to know per-register whether logging was timestamped. timestamp=True/False forces it.

Scope is intentionally limited to reading an already de-multiplexed folder — stream de-multiplexing is out of scope for this PR, but we can add it in the future using the framer from the device class. I just felt the API surface would be unecessarily large.

Epoch / "Time" index parity with harp-python

parse_to_dataframe gained an epoch argument and now places the Harp time on the DataFrame index named "Time", matching harp-python: no epoch → float seconds (pd.Index); epoch given (e.g. REFERENCE_EPOCH, exported here) → absolute DatetimeIndex. The old "timestamp" column is gone in favor of this index.

harp-python deprecation and dropping reader.Register.read()

harp-python's reader exposed per-register readers as attributes, so you read data with reader.Register.read(). This PR deliberately does not reproduce that API. pyharp's pattern is a method on the reader — reader.read(register) — and keeping the attribute-style accessor (via __getattr__) would mean carrying a second, redundant surface just for source compatibility.

The plan: harp-python adopts this reader for one final release, emitting a DeprecationWarning on the old reader.Register.read() access, and is then archived. There is therefore no compatibility shim here — no __getattr__ layer, no reader.Register.read(). New code uses reader.read(...), which follows the pyharp convention rather than the historical harp-python one.

Tests

tests/data/test_dataset.py builds mock dataset folders in tmp_path (cleaned up per test) using RegisterBase.format_bulk to generate real Harp frames, then exercises: read by class and by address, automatic timestamp detection, read_all keyed by register name, multi-chunk suffix concatenation, custom resolver, name override, missing-file / unknown-address errors, float-seconds vs absolute DatetimeIndex (epoch), and an end-to-end "read every register from a device.yml-generated device".

@bruno-f-cruz
bruno-f-cruz requested a review from glopesdev July 26, 2026 05:17
@bruno-f-cruz bruno-f-cruz linked an issue Jul 26, 2026 that may be closed by this pull request
@bruno-f-cruz

bruno-f-cruz commented Jul 26, 2026

Copy link
Copy Markdown
Member Author

@glopesdev one major thing to solve in the API is where to get the runtime registers from. Currently, they exist in generated_device.REGISTER_MAP as a mapping[int, Register]. This becomes a bit clunky to use since you are required to know the register number, which defeats the purpose. The alternative is to have something like: device.registers.Register in all devices, which i think may be a good decision for all devices' api overal, do you have thoughts? This is important to get right asap, since the generators upstream should target the same api surface for consistency

@bruno-f-cruz
bruno-f-cruz force-pushed the feat-add-device-reader branch from d5d7ef5 to 4ff7f94 Compare July 26, 2026 16:51
@bruno-f-cruz
bruno-f-cruz force-pushed the feat-runtime-register-binding branch from de70211 to 933071c Compare July 26, 2026 16:57
@bruno-f-cruz
bruno-f-cruz force-pushed the feat-add-device-reader branch from 4ff7f94 to 72522ae Compare July 26, 2026 16:57
Base automatically changed from feat-runtime-register-binding to main August 10, 2026 18:17
@glopesdev
glopesdev force-pushed the feat-add-device-reader branch from 72522ae to 94a1c70 Compare August 10, 2026 18:30
@glopesdev
glopesdev merged commit 49672f4 into main Aug 10, 2026
13 checks passed
@glopesdev
glopesdev deleted the feat-add-device-reader branch August 10, 2026 18:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add device-driven reader factory for harp.data

2 participants