Refactor Device API to allow binding device's registers

README.md

@@ -50,6 +50,51 @@ pip install harp-data
 
 `harp-benchmarks` (under `src/packages/`) is internal-only and is never published to PyPI.
 
+## Quickstart
+
+There are two ways you'll typically use `harp`: talking to a **live device** over a
+serial connection, or reading **data recorded to disk**.
+
+**Talk to a live device.** Open a connection and read/write registers by class:
+
+```python
+from harp.device import Device, WhoAmI, OperationControl, OperationControlPayload, OperationMode
+from harp.serial import open_serial_device
+
+# Use "COMx" on Windows, "/dev/ttyUSBx" on Linux.
+with open_serial_device(Device, port="/dev/ttyUSB0") as device:
+    print("WhoAmI:", device.read(WhoAmI).parsed)
+    device.write(OperationControl, OperationControlPayload(operation_mode=OperationMode.ACTIVE))
+```
+
+**Read a recorded session.** Point a `DatasetReader` at a dataset folder and read
+registers into pandas DataFrames — no hardware required:
+
+```python
+from harp.data import create_dataset_reader
+
+# Finds device.yml in the folder, builds the device, returns a ready-to-use reader.
+reader = create_dataset_reader("session.harp")
+df = reader.read(44)  # one register, by address (or pass its class)
+everything = reader.read_all()  # {register_name: DataFrame}
+```
+
+Both paths are driven by a device schema. If you have only a `device.yml` and no
+pre-generated package, `create_device` compiles it into a typed `Device` at runtime —
+no code-generation step — which is exactly what `create_dataset_reader` does under
+the hood:
+
+```python
+from pathlib import Path
+from harp.device import create_device
+
+Behavior = create_device(Path("device.yml").read_text())
+AnalogData = Behavior.registers.AnalogData  # registers are reached by name
+```
+
+See the [Examples](https://harp-tech.org/pyharp/examples/) for the full walkthroughs,
+including subscribing to device events and working with custom interface-type converters.
+
 ## Contributing
 
 harp is a [uv workspace](https://docs.astral.sh/uv/concepts/workspaces/): every package under

docs/api/data.md

@@ -2,5 +2,11 @@
 
 ---
 
+::: harp.data.create_dataset_reader
+::: harp.data.DatasetReader
+::: harp.data.default_file_resolver
 ::: harp.data.parse_to_dataframe
 ::: harp.data.payload_to_dataframe
+::: harp.data.to_file
+::: harp.data.to_buffer
+::: harp.data.REFERENCE_EPOCH

docs/api/device.md

@@ -3,10 +3,14 @@
 ---
 
 ::: harp.device.Device
+::: harp.device.create_device
+::: harp.device.parse_device_schema
+::: harp.device.ConverterContext
+::: harp.device.RegisterMap
+::: harp.device.CoreRegisters
 ::: harp.device.HarpFramer
 ::: harp.device.ITransport
 ::: harp.device.TransportError
-::: harp.device.REGISTER_MAP
 ::: harp.device.OperationControl
 ::: harp.device.OperationMode
 ::: harp.device.ResetDevice

docs/examples/create_device/create_device.md

@@ -0,0 +1,57 @@
+# Generating a Device from a Schema
+
+This example demonstrates how to turn a Harp `device.yml` into a typed
+[`Device`](../../api/device.md) at runtime with `create_device`, without a
+code-generation step. This is the quickest way to get started when you have only a
+device's schema and no pre-generated package for it.
+
+The compiled device exposes its registers by name through `device.registers`
+(e.g. `Behavior.registers.AnalogData`, or the `Behavior.registers.by_address` map)
+and carries the device's `__whoami__` identity. From there it works exactly like a
+pre-generated device class — drive it over a transport to talk to hardware, or use
+its register classes to decode recorded data.
+
+## When to use runtime generation
+
+`create_device` trades statically generated device packages for schema-driven
+convenience. It's worth understanding what that buys you and what it costs.
+
+**You gain:**
+
+- **No build step.** A `device.yml` — even one you just pulled off a device —
+  becomes a working device in a single call. There's nothing to generate, install,
+  or keep in sync with the schema.
+- **Coverage for any device.** You don't need a published package for the device;
+  unreleased, custom, or one-off schemas work immediately.
+- **The schema stays the single source of truth.** Register, field, and enum names
+  come straight from the `device.yml`, verbatim.
+
+**You give up:**
+
+- **Static register types.** A runtime device still exposes its registers by name
+  (`device.registers.AnalogData`), but because the class is built at runtime the
+  editor can't autocomplete those names or check them — you get a generic
+  `type[RegisterBase]`, not the specific register type. A statically generated device
+  declares its registers, so `device.registers.AnalogData` autocompletes and
+  `read`/`write` infer the payload type.
+- **Generator naming conventions.** Identifiers are kept verbatim from the yml
+  (`AnalogInput0`, `DIO0`) rather than the C# generator's snake_case fields and
+  `UPPER_SNAKE` enum members, so code written against a generated package won't line
+  up name-for-name.
+- **Turn-key custom types.** A custom `interfaceType` must be injected yourself via
+  `converters=` (see below), whereas a generated package ships its own converters.
+
+For shipped, widely-used devices a pre-generated package from the
+[Harp C# generator](https://github.com/harp-tech/generators) remains the
+authoritative choice — better editor support and static typing. Reach for
+`create_device` when you want to go from a schema to working code with no
+generation step.
+
+!!! warning
+    Don't forget to change the `SERIAL_PORT` to the one that corresponds to your device! The `SERIAL_PORT` must be denoted as `/dev/ttyUSBx` in Linux and `COMx` in Windows, where `x` is the number of the serial port.
+
+<!--codeinclude-->
+```python
+[](./create_device.py)
+```
+<!--/codeinclude-->

docs/examples/create_device/create_device.py

@@ -0,0 +1,42 @@
+from pathlib import Path
+
+from harp.data import parse_to_dataframe
+from harp.device import create_device
+from harp.serial import open_serial_device
+
+SERIAL_PORT = "/dev/ttyUSB0"  # or "COMx" in Windows ("x" is the number of the serial port)
+
+# `create_device` compiles a Harp `device.yml` into a typed `Device` subclass at
+# runtime — no code-generation step. This is the quickest way to work with a device
+# when you don't have a pre-generated package for it: point it at the schema and you
+# get the device's registers (keyed by address) plus its identity.
+Behavior = create_device(Path("device.yml").read_text())
+
+print("WhoAmI:", Behavior.__whoami__)  # device identity, taken from the schema
+
+# Registers are reached by name through `.registers` — the common Harp registers
+# (like WhoAmI) plus the device's own. Address lookup goes through
+# `Behavior.registers.by_address`.
+AnalogData = Behavior.registers.AnalogData
+
+# The generated device behaves like any other `Device` class. Talk to hardware over
+# a transport — `read`/`write` take a register class:
+with open_serial_device(Behavior, port=SERIAL_PORT) as device:
+    print("AnalogData:", device.read(AnalogData).parsed)
+
+# ...or use the same register classes to decode a recorded binary dump into a
+# pandas DataFrame (see the "Reading Data into a DataFrame" example for more):
+df = parse_to_dataframe(AnalogData, "Behavior_44.bin")
+print(df.head())
+
+
+# --- Custom interface types --------------------------------------------------
+# A register with a custom `interfaceType` needs a converter so its field decodes
+# to the right Python type. Pass it via `converters=`, keyed by "<Name>Converter":
+#
+#   Behavior = create_device(yml_text, converters={"DataConverter": DataConverter()})
+#
+# An unresolved custom type raises `UnknownConverterError`; pass `strict=False` to
+# decode it natively instead. `exclude_private=True` (the default) drops registers
+# marked `private` in the schema. If you only want the parsed schema model rather
+# than a device, `parse_device_schema(yml_text)` returns that directly.

docs/examples/get_info/get_info.py

@@ -1,4 +1,4 @@
-from harp.device import REGISTER_MAP, Device, WhoAmI
+from harp.device import Device, WhoAmI
 from harp.serial import open_serial_device
 
 SERIAL_PORT = "/dev/ttyUSB0"  # or "COMx" in Windows ("x" is the number of the serial port)
@@ -8,7 +8,8 @@
     # Identify the device.
     print("WhoAmI:", device.read(WhoAmI).parsed)
 
-    # Dump every core register.
-    for address, register in sorted(REGISTER_MAP.items()):
+    # Dump every register the device exposes. `device.registers` is name-addressable
+    # (device.registers.WhoAmI); `.by_address` gives the address -> register map.
+    for address, register in sorted(device.registers.by_address.items()):
         reply = device.read(register)
         print(f"{register.__name__:24s} (addr {address:2d}) = {reply.parsed}")

docs/examples/index.md

@@ -2,8 +2,18 @@
 
 This section contains some examples to help you get started with `harp`.
 
-Here's the complete list of available examples:
+Defining a device:
+
+- [Defining a Device Statically](./static_device/static_device.md) - write a device as plain, typed Python classes (the shape code generators emit).
+- [Generating a Device from a Schema](./create_device/create_device.md) - compile a `device.yml` into a typed device at runtime with `create_device`.
+
+Talking to a device:
 
 - [Getting Device Info](./get_info/get_info.md) - connect to a Harp device and read its information.
 - [Read and Write from Registers](./read_and_write_from_registers/read_and_write_from_registers.md) - connect to a Harp device and read and write its registers.
-- [Reading Data into a DataFrame](./read_data_to_dataframe/read_data_to_dataframe.md) - load a register's binary data file into a pandas DataFrame with `harp.data`.
+- [Subscribing to Events](./subscribing_to_events/subscribing_to_events.md) - react to messages pushed by the device without polling.
+
+Reading recorded data:
+
+- [Reading a Whole Dataset Folder](./read_dataset/read_dataset.md) - load an entire recorded session folder into pandas DataFrames with `DatasetReader`.
+- [Reading Data into a DataFrame](./read_data_to_dataframe/read_data_to_dataframe.md) - decode a single register's binary file into a pandas DataFrame.

docs/examples/read_data_to_dataframe/read_data_to_dataframe.md

@@ -1,8 +1,14 @@
 # Reading Data into a DataFrame
 
-This example demonstrates how to load a Harp register's binary data file into a
-pandas DataFrame using `harp.data`. The register definition tells `parse_to_dataframe`
-how to decode each frame, so you get named columns (and decoded enums) for free.
+This example demonstrates how to load a **single** Harp register's binary data
+file into a pandas DataFrame using `harp.data`. The register definition tells
+`parse_to_dataframe` how to decode each frame, so you get named columns (and
+decoded enums) for free.
+
+!!! tip
+    Have a whole recorded session folder rather than one loose file? Use
+    [`DatasetReader`](../read_dataset/read_dataset.md), which reads every register
+    in a dataset folder driven by the device schema.
 
 <!--codeinclude-->
 ```python

docs/examples/read_data_to_dataframe/read_data_to_dataframe.py

@@ -1,11 +1,20 @@
 from harp.data import parse_to_dataframe
 from harp.device import OperationControl
 
-# Parse a register's binary dump into a pandas DataFrame — one row per frame,
-# one column per field, plus a leading "timestamp" column.
-df = parse_to_dataframe(OperationControl, "OperationControl.bin", timestamp=True)
+# Parse a single register's binary dump into a pandas DataFrame — one row per
+# frame, one column per field. The register class tells `parse_to_dataframe` how
+# to decode each frame, so you get named columns (and decoded enums) for free.
+df = parse_to_dataframe(OperationControl, "OperationControl.bin")
 print(df.head())
 
+# When the frames are timestamped (the default), the Harp time becomes the
+# DataFrame index, named "Time" — float seconds from device start.
+print(df.index.name, df.index[:3].to_list())
+
 # `parse_to_dataframe` also accepts raw bytes or an open binary file object:
 with open("OperationControl.bin", "rb") as f:
     df = parse_to_dataframe(OperationControl, f)
+
+# To read a whole recorded session folder at once (many registers, driven by the
+# device schema) use `harp.data.DatasetReader` — see the "Reading a Whole Dataset
+# Folder" example.

docs/examples/read_dataset/read_dataset.md

@@ -0,0 +1,25 @@
+# Reading a Whole Dataset Folder
+
+A Harp acquisition is usually saved as a **de-multiplexed dataset folder**: one
+binary file per register, named `<DeviceName>_<address>.bin`, next to the device's
+`device.yml` schema. `harp.data.DatasetReader` reads that whole folder into pandas
+DataFrames, driven by a [generated device](../../api/device.md) that describes how
+to decode each register.
+
+This is the recommended entry point when you have a recorded session on disk. To
+decode a single loose `.bin` file instead, see
+[Reading Data into a DataFrame](../read_data_to_dataframe/read_data_to_dataframe.md).
+
+The quickest way in is `create_dataset_reader(folder)`: it finds the `device.yml`
+inside the folder, builds the device for you, and returns a reader ready to go.
+(If you already have a device class — e.g. from a pre-generated package — construct
+`DatasetReader(Device, folder)` directly instead.) You then read a register by
+class or by address, or read every register at once with `read_all()`. Timestamps
+are detected automatically and placed on the `"Time"` index (float seconds, or an
+absolute `DatetimeIndex` when you pass an `epoch`).
+
+<!--codeinclude-->
+```python
+[](./read_dataset.py)
+```
+<!--/codeinclude-->

docs/examples/read_dataset/read_dataset.py

@@ -0,0 +1,46 @@
+from harp.data import REFERENCE_EPOCH, create_dataset_reader
+from harp.device import OperationControl
+
+# A Harp acquisition is usually saved as a de-multiplexed dataset folder — one
+# `.bin` file per register, named "<DeviceName>_<address>.bin", next to the
+# device's `device.yml` schema:
+#
+#   📦 session.harp
+#    ┣ 📜 Behavior_0.bin
+#    ┣ 📜 Behavior_44.bin
+#    ┣ ...
+#    ┗ 📜 device.yml
+#
+# `create_dataset_reader` does the right thing: it finds `device.yml` inside the
+# folder, builds the device that knows how to decode each register, and hands back
+# a reader ready to go.
+reader = create_dataset_reader("session.harp")
+
+# Read one register into a DataFrame — by register class (any register in the
+# device's map, including the common ones like `OperationControl`)...
+df = reader.read(OperationControl)
+
+# ...or by address. Timestamps are auto-detected from the frames, and when present
+# they become the DataFrame index, named "Time" (float seconds from device start).
+df = reader.read(44)
+print(df.head())
+
+# Read every register that has a file on disk at once, keyed by register name.
+everything = reader.read_all()
+print(list(everything))
+
+# Pass an epoch to turn the "Time" index into an absolute `DatetimeIndex` instead
+# of float seconds. `REFERENCE_EPOCH` is time zero of the Harp clock (UTC).
+absolute = reader.read(44, epoch=REFERENCE_EPOCH)
+print(absolute.index[:3])
+
+# --- Already have a device class? -------------------------------------------
+# A pre-generated device package, or one you built yourself with `create_device`,
+# can drive the reader directly — construct `DatasetReader(Device, folder)`:
+#
+#   from harp.data import DatasetReader
+#   from harp.device import create_device
+#   from pathlib import Path
+#
+#   Behavior = create_device((Path("session.harp") / "device.yml").read_text())
+#   reader = DatasetReader(Behavior, "session.harp")