Add dataset reader to harp-data (#15)

DatasetReader reads a de-multiplexed dataset folder driven by a device,
by register class or by address, with read_all returning every register
that has a file present. Timestamps are auto-detected per register and
become a Time index, float seconds by default or an absolute
DatetimeIndex given an epoch, matching harp-python.

create_dataset_reader finds device.yml inside the folder and returns a
reader ready to read.

Closes #10

Author: bruno-f-cruz

README.md

@@ -52,42 +52,48 @@ pip install harp-data
 
 ## Quickstart
 
-Have only a device's `device.yml`? `create_device` compiles it into a typed
-`Device` at runtime — no code-generation step — giving you the device's registers
-(keyed by address) and its identity:
+There are two ways you'll typically use `harp`: talking to a **live device** over a
+serial connection, or reading **data recorded to disk**.
+
+**Talk to a live device.** Open a connection and read/write registers by class:
 
 ```python
-from pathlib import Path
-from harp.device import create_device
+from harp.device import Device, WhoAmI, OperationControl, OperationControlPayload, OperationMode
+from harp.serial import open_serial_device
 
-Behavior = create_device(Path("device.yml").read_text())
-Behavior.__whoami__            # device identity from the schema
-AnalogData = Behavior.REGISTER_MAP[44]   # registers are reached by address
+# Use "COMx" on Windows, "/dev/ttyUSBx" on Linux.
+with open_serial_device(Device, port="/dev/ttyUSB0") as device:
+    print("WhoAmI:", device.read(WhoAmI).parsed)
+    device.write(OperationControl, OperationControlPayload(operation_mode=OperationMode.ACTIVE))
 ```
 
-The generated device works like any other. **Talk to hardware** over a serial
-transport — `read`/`write` take a register class:
+**Read a recorded session.** Point a `DatasetReader` at a dataset folder and read
+registers into pandas DataFrames — no hardware required:
 
 ```python
-from harp.serial import open_serial_device
+from harp.data import create_dataset_reader
 
-# Use "COMx" on Windows, "/dev/ttyUSBx" on Linux.
-with open_serial_device(Behavior, port="/dev/ttyUSB0") as device:
-    print(device.read(AnalogData).parsed)
+# Finds device.yml in the folder, builds the device, returns a ready-to-use reader.
+reader = create_dataset_reader("session.harp")
+df = reader.read(44)              # one register, by address (or pass its class)
+everything = reader.read_all()    # {register_name: DataFrame}
 ```
 
-...or use the same register classes to **decode recorded data** into a pandas
-DataFrame:
+Both paths are driven by a device schema. If you have only a `device.yml` and no
+pre-generated package, `create_device` compiles it into a typed `Device` at runtime —
+no code-generation step — which is exactly what `create_dataset_reader` does under
+the hood:
 
 ```python
-from harp.data import parse_to_dataframe
+from pathlib import Path
+from harp.device import create_device
 
-df = parse_to_dataframe(AnalogData, "Behavior_44.bin")
+Behavior = create_device(Path("device.yml").read_text())
+AnalogData = Behavior.REGISTER_MAP[44]   # registers are reached by address
 ```
 
-See the [Examples](https://harp-tech.org/pyharp/examples/) for full walkthroughs,
-including reading device info, subscribing to events, and working with custom
-interface-type converters.
+See the [Examples](https://harp-tech.org/pyharp/examples/) for the full walkthroughs,
+including subscribing to device events and working with custom interface-type converters.
 
 ## Contributing
 

docs/api/data.md

@@ -2,5 +2,11 @@
 
 ---
 
+::: harp.data.create_dataset_reader
+::: harp.data.DatasetReader
+::: harp.data.default_file_resolver
 ::: harp.data.parse_to_dataframe
 ::: harp.data.payload_to_dataframe
+::: harp.data.to_file
+::: harp.data.to_buffer
+::: harp.data.REFERENCE_EPOCH

docs/examples/index.md

@@ -14,4 +14,5 @@ Talking to a device:
 
 Reading recorded data:
 
-- [Reading Data into a DataFrame](./read_data_to_dataframe/read_data_to_dataframe.md) - load a register's binary data file into a pandas DataFrame with `harp.data`.
+- [Reading a Whole Dataset Folder](./read_dataset/read_dataset.md) - load an entire recorded session folder into pandas DataFrames with `DatasetReader`.
+- [Reading Data into a DataFrame](./read_data_to_dataframe/read_data_to_dataframe.md) - decode a single register's binary file into a pandas DataFrame.

docs/examples/read_data_to_dataframe/read_data_to_dataframe.md

@@ -1,8 +1,14 @@
 # Reading Data into a DataFrame
 
-This example demonstrates how to load a Harp register's binary data file into a
-pandas DataFrame using `harp.data`. The register definition tells `parse_to_dataframe`
-how to decode each frame, so you get named columns (and decoded enums) for free.
+This example demonstrates how to load a **single** Harp register's binary data
+file into a pandas DataFrame using `harp.data`. The register definition tells
+`parse_to_dataframe` how to decode each frame, so you get named columns (and
+decoded enums) for free.
+
+!!! tip
+    Have a whole recorded session folder rather than one loose file? Use
+    [`DatasetReader`](../read_dataset/read_dataset.md), which reads every register
+    in a dataset folder driven by the device schema.
 
 <!--codeinclude-->
 ```python

docs/examples/read_data_to_dataframe/read_data_to_dataframe.py

@@ -1,11 +1,20 @@
 from harp.data import parse_to_dataframe
 from harp.device import OperationControl
 
-# Parse a register's binary dump into a pandas DataFrame — one row per frame,
-# one column per field, plus a leading "timestamp" column.
-df = parse_to_dataframe(OperationControl, "OperationControl.bin", timestamp=True)
+# Parse a single register's binary dump into a pandas DataFrame — one row per
+# frame, one column per field. The register class tells `parse_to_dataframe` how
+# to decode each frame, so you get named columns (and decoded enums) for free.
+df = parse_to_dataframe(OperationControl, "OperationControl.bin")
 print(df.head())
 
+# When the frames are timestamped (the default), the Harp time becomes the
+# DataFrame index, named "Time" — float seconds from device start.
+print(df.index.name, df.index[:3].to_list())
+
 # `parse_to_dataframe` also accepts raw bytes or an open binary file object:
 with open("OperationControl.bin", "rb") as f:
     df = parse_to_dataframe(OperationControl, f)
+
+# To read a whole recorded session folder at once (many registers, driven by the
+# device schema) use `harp.data.DatasetReader` — see the "Reading a Whole Dataset
+# Folder" example.

docs/examples/read_dataset/read_dataset.md

@@ -0,0 +1,25 @@
+# Reading a Whole Dataset Folder
+
+A Harp acquisition is usually saved as a **de-multiplexed dataset folder**: one
+binary file per register, named `<DeviceName>_<address>.bin`, next to the device's
+`device.yml` schema. `harp.data.DatasetReader` reads that whole folder into pandas
+DataFrames, driven by a [generated device](../../api/device.md) that describes how
+to decode each register.
+
+This is the recommended entry point when you have a recorded session on disk. To
+decode a single loose `.bin` file instead, see
+[Reading Data into a DataFrame](../read_data_to_dataframe/read_data_to_dataframe.md).
+
+The quickest way in is `create_dataset_reader(folder)`: it finds the `device.yml`
+inside the folder, builds the device for you, and returns a reader ready to go.
+(If you already have a device class — e.g. from a pre-generated package — construct
+`DatasetReader(Device, folder)` directly instead.) You then read a register by
+class or by address, or read every register at once with `read_all()`. Timestamps
+are detected automatically and placed on the `"Time"` index (float seconds, or an
+absolute `DatetimeIndex` when you pass an `epoch`).
+
+<!--codeinclude-->
+```python
+[](./read_dataset.py)
+```
+<!--/codeinclude-->

docs/examples/read_dataset/read_dataset.py

@@ -0,0 +1,46 @@
+from harp.data import REFERENCE_EPOCH, create_dataset_reader
+from harp.device import OperationControl
+
+# A Harp acquisition is usually saved as a de-multiplexed dataset folder — one
+# `.bin` file per register, named "<DeviceName>_<address>.bin", next to the
+# device's `device.yml` schema:
+#
+#   📦 session.harp
+#    ┣ 📜 Behavior_0.bin
+#    ┣ 📜 Behavior_44.bin
+#    ┣ ...
+#    ┗ 📜 device.yml
+#
+# `create_dataset_reader` does the right thing: it finds `device.yml` inside the
+# folder, builds the device that knows how to decode each register, and hands back
+# a reader ready to go.
+reader = create_dataset_reader("session.harp")
+
+# Read one register into a DataFrame — by register class (any register in the
+# device's map, including the common ones like `OperationControl`)...
+df = reader.read(OperationControl)
+
+# ...or by address. Timestamps are auto-detected from the frames, and when present
+# they become the DataFrame index, named "Time" (float seconds from device start).
+df = reader.read(44)
+print(df.head())
+
+# Read every register that has a file on disk at once, keyed by register name.
+everything = reader.read_all()
+print(list(everything))
+
+# Pass an epoch to turn the "Time" index into an absolute `DatetimeIndex` instead
+# of float seconds. `REFERENCE_EPOCH` is time zero of the Harp clock (UTC).
+absolute = reader.read(44, epoch=REFERENCE_EPOCH)
+print(absolute.index[:3])
+
+# --- Already have a device class? -------------------------------------------
+# A pre-generated device package, or one you built yourself with `create_device`,
+# can drive the reader directly — construct `DatasetReader(Device, folder)`:
+#
+#   from harp.data import DatasetReader
+#   from harp.device import create_device
+#   from pathlib import Path
+#
+#   Behavior = create_device((Path("session.harp") / "device.yml").read_text())
+#   reader = DatasetReader(Behavior, "session.harp")

mkdocs.yml

@@ -77,6 +77,7 @@ nav:
     - Getting Device Info: examples/get_info/get_info.md
     - Read and Write from Registers: examples/read_and_write_from_registers/read_and_write_from_registers.md
     - Subscribing to Events: examples/subscribing_to_events/subscribing_to_events.md
+    - Reading a Whole Dataset Folder: examples/read_dataset/read_dataset.md
     - Reading Data into a DataFrame: examples/read_data_to_dataframe/read_data_to_dataframe.md
   - API:
     - Protocol: api/protocol.md

src/packages/harp-data/README.md

@@ -4,7 +4,59 @@ Load Harp register data into pandas DataFrames. This is the package that pulls
 in `pandas` — [`harp-protocol`](../harp-protocol) stays numpy-only and exposes a
 pandas-free `ColumnData` view that this package assembles into a DataFrame.
 
-## Read a register from a file
+There are two ways in, depending on what you have on disk:
+
+- a whole **dataset folder** (many registers) → `DatasetReader`
+- a single **register file** or buffer → `parse_to_dataframe`
+
+## Read a whole dataset folder
+
+A Harp acquisition is usually saved as a de-multiplexed folder — one binary file
+per register, named `<DeviceName>_<address>.bin`, alongside the device's
+`device.yml` schema:
+
+```text
+📦 session.harp
+ ┣ 📜 Behavior_0.bin
+ ┣ 📜 Behavior_44.bin
+ ┣ ...
+ ┗ 📜 device.yml
+```
+
+Reading is driven by a generated
+[`harp.device.Device`](../harp-device) that describes how to decode each register.
+`create_dataset_reader` does that for you — it finds the `device.yml` in the folder,
+builds the device, and returns a ready-to-use reader:
+
+```python
+from harp.data import create_dataset_reader
+
+reader = create_dataset_reader("session.harp")
+df  = reader.read(AnalogData)     # by register class
+df  = reader.read(44)             # by address
+everything = reader.read_all()    # {register_name: DataFrame}
+```
+
+Already have a device class (e.g. a pre-generated package, or one built with
+`create_device`)? Drive `DatasetReader` with it directly:
+
+```python
+from pathlib import Path
+from harp.data import DatasetReader
+from harp.device import create_device
+
+Behavior = create_device((Path("session.harp") / "device.yml").read_text())
+reader = DatasetReader(Behavior, "session.harp")
+```
+
+Timestamps are auto-detected per register and placed on the DataFrame index
+(named `"Time"`): float seconds by default, or an absolute `DatetimeIndex` when
+you pass `epoch=REFERENCE_EPOCH`. Multi-chunk registers logged as
+`<DeviceName>_<address>_<suffix>.bin` are concatenated in filename order; pass a
+`resolver` to support an alternative on-disk layout, or `name=` to override the
+file prefix.
+
+## Read a single register file
 
 `parse_to_dataframe` takes a register and a source (path, bytes, or open binary
 file) and returns one row per frame:
@@ -17,7 +69,10 @@ df = parse_to_dataframe(AnalogData, "AnalogData.bin")
 df = parse_to_dataframe(AnalogData, raw, timestamp=True, message_type=False, decode_enums=True)
 ```
 
-Enum fields decode to `pd.Categorical` (`decode_enums=False` keeps raw codes).
+With `timestamp=True` (the default) the Harp time becomes the DataFrame index,
+named `"Time"` — float seconds, or an absolute `DatetimeIndex` when you also pass
+`epoch=REFERENCE_EPOCH`. Enum fields decode to `pd.Categorical`
+(`decode_enums=False` keeps raw codes).
 
 ## From an already-parsed payload
 
@@ -30,3 +85,14 @@ from harp.data import payload_to_dataframe
 _data, timestamps, _msg, payload = AnalogData.parse_bulk(raw)
 df = payload_to_dataframe(payload)
 ```
+
+## Write data back out
+
+`to_file` / `to_buffer` are the inverse of the readers — encode values as Harp
+frames. Handy for round-tripping data or generating test corpora:
+
+```python
+from harp.data import to_file
+
+to_file(AnalogData, values, "AnalogData.bin", timestamps=seconds)
+```

src/packages/harp-data/pyproject.toml

@@ -5,6 +5,7 @@ description = "Load Harp device data into pandas DataFrames"
 requires-python = ">=3.11"
 dependencies = [
     "harp-protocol",
+    "harp-device",
     "numpy>=1.24",
     "pandas>=2.0",
 ]