Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 48 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
# Auto detect text files and normalize line endings
* text=auto

# Scripts
*.cmd text eol=crlf
*.sh text eol=lf
*.ps1 text

# Config
*.gitignore text
*.gitattributes text
*.gitmodules text eol=lf
*.git-blame-ignore-revs text
*.python-version text
*.editorconfig text
*.toml text
*.sln text
*.proj text
*.props text
*.targets text
*.csproj text
*.wixproj text
*.config text
*.json text
*.xml text
*.yml text

# Code
*.css text
*.py text diff=python
*.manifest text
*.vsixmanifest text
*.vstemplate text
*.resx text
*.cs text
*.bonsai text
*.wxs text

# Documents
LICENSE text
*.md text diff=markdown
*.rtf diff=astextplain

# Graphics
*.png binary
*.ico binary
*.gif binary
*.svg text
87 changes: 38 additions & 49 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,22 +6,17 @@

This project includes four main packages:

- **harp-protocol**: Provides the core protocol definitions and utilities for the Harp protocol.
See [Protocol API Documentation](https://harp-tech.org/pyharp/api/protocol) for details.
- **harp-protocol**: Provides the core protocol definitions and utilities for the Harp protocol. See [Protocol API Documentation](https://harp-tech.org/pyharp/api/protocol) for details.

- **harp-serial**: Implements serial communication functionalities for generic Harp devices.
See [Serial API Documentation](https://harp-tech.org/pyharp/api/serial) for more information.
- **harp-serial**: Implements serial communication functionalities for generic Harp devices. See [Serial API Documentation](https://harp-tech.org/pyharp/api/serial) for more information.

- **harp-device**: Implements the transport-agnostic `Device` interface, the common register map, and the shared registers and enums.
See [Device API Documentation](https://harp-tech.org/pyharp/api/device) for details.
- **harp-device**: Implements the transport-agnostic `Device` interface, the common register map, and the shared registers and enums. See [Device API Documentation](https://harp-tech.org/pyharp/api/device) for details.

- **harp-data**: Parses register binary dumps into pandas DataFrames.
See [Data API Documentation](https://harp-tech.org/pyharp/api/data) for more information.
- **harp-data**: Parses register binary dumps into pandas DataFrames. See [Data API Documentation](https://harp-tech.org/pyharp/api/data) for more information.

## Installation

All packages are published to PyPI. The `harp` package is a metadata package with no code of
its own — it just depends on the four packages above, so it's the easiest way to get everything:
All packages are published to PyPI. The `harp` package is a metadata package with no code of its own. It depends on the four packages above, so it is the easiest way to get everything:

```sh
pip install harp
Expand All @@ -31,14 +26,13 @@ pip install harp
uv add harp
```

If you only need part of the stack (e.g. you're parsing offline data dumps and don't need serial
I/O), install just the packages you need — each one only pulls in what it actually depends on:
To install only part of the stack, for example when parsing offline data dumps with no need for serial I/O, install the individual packages. Each one only pulls in what it actually depends on:

| Package | Provides | Depends on |
| --- | --- | --- |
| `harp-protocol` | Core protocol types: registers, messages, payload parsing | |
| `harp-protocol` | Core protocol types: registers, messages, payload parsing | none |
| `harp-device` | Transport-agnostic `Device` class, common register map | `harp-protocol` |
| `harp-serial` | Serial (COM/tty) transport for `Device` | `harp-protocol`, `harp-device` |
| `harp-serial` | Serial COM or tty transport for `Device` | `harp-protocol`, `harp-device` |
| `harp-data` | Parse register binary dumps into pandas DataFrames | `harp-protocol` |

```sh
Expand All @@ -48,81 +42,76 @@ pip install harp-serial
pip install harp-data
```

`harp-benchmarks` (under `src/packages/`) is internal-only and is never published to PyPI.
`harp-benchmarks`, under `src/packages/`, is internal-only and is never published to PyPI.

## Quickstart

There are two ways you'll typically use `harp`: talking to a **live device** over a
serial connection, or reading **data recorded to disk**.
There are two typical ways to use `harp`: talking to a **live device** over a serial connection, or reading **data recorded to disk**.

**Talk to a live device.** Open a connection and read/write registers by class:

```python
from harp.device.core import OperationControl, OperationControlPayload, OperationMode, WhoAmI
from harp.device.client import Device
from harp.serial import open_serial_device
from harp import serial
from harp.device import behavior, core

# Use "COMx" on Windows, "/dev/ttyUSBx" on Linux.
with open_serial_device(Device, port="/dev/ttyUSB0") as device:
print("WhoAmI:", device.read(WhoAmI).parsed)
device.write(OperationControl, OperationControlPayload(operation_mode=OperationMode.ACTIVE))
with serial.open_serial_device(behavior, port="COM3") as device:
print(device.read(core.WhoAmI).parsed) # a common register
print(device.read(behavior.AnalogData).parsed) # a device register
device.write(
core.OperationControl,
core.OperationControlPayload(operation_mode=core.OperationMode.ACTIVE),
)
```

**Read a recorded session.** Point a `DatasetReader` at a dataset folder and read
registers into pandas DataFrames — no hardware required:
**Read a recorded session.** Point a `DatasetReader` at a dataset folder and read registers into pandas DataFrames, with no hardware required:

```python
from harp.data import create_dataset_reader
from harp import data

# Finds device.yml in the folder, builds the device, returns a ready-to-use reader.
reader = create_dataset_reader("session.harp")
df = reader.read(44) # one register, by address (or pass its class)
everything = reader.read_all() # {register_name: DataFrame}
reader = data.create_dataset_reader("session.harp")
behavior = reader.device_module
df = reader.read(behavior.AnalogData) # by register class
df = reader.read(44) # or by address
everything = reader.read_all() # {register_name: DataFrame}
```

Both paths are driven by a device schema. If you have only a `device.yml` and no
pre-generated package, `create_device_module` compiles it into a module of register classes
at runtime — no code-generation step — which is exactly what `create_dataset_reader`
does under the hood:
Both paths are based on a device schema. Given only a `device.yml` and no pre-generated package, `create_device_module` compiles it into a module of register classes at runtime, with no code-generation step. This is exactly what `create_dataset_reader` does internally:

```python
from pathlib import Path
from harp.device.schema import create_device_module

behavior = create_device_module(Path("device.yml").read_bytes())
AnalogData = behavior.AnalogData # registers are reached by name...
assert behavior.REGISTER_MAP[44] is AnalogData # ...or by address
from harp.device import schema

behavior = schema.create_device_module(Path("device.yml").read_bytes())
AnalogData = behavior.AnalogData # registers are reached by name
assert behavior.REGISTER_MAP[44] is AnalogData # or by address
```

See the [Examples](https://harp-tech.org/pyharp/examples/) for the full walkthroughs,
including subscribing to device events and working with custom interface-type converters.
See the [Examples](https://harp-tech.org/pyharp/examples/) for the full walkthroughs, including subscribing to device events and working with custom interface-type converters.

## Contributing

harp is a [uv workspace](https://docs.astral.sh/uv/concepts/workspaces/): every package under
`src/packages/` is its own distribution, plus the root `harp` metadata package. Contributions are
welcome — please open an issue or PR.
harp is a [uv workspace](https://docs.astral.sh/uv/concepts/workspaces/): every package under `src/packages/` is its own distribution, plus the root `harp` metadata package. Bug reports and contributions are welcome, so please open an issue or pull request.

Clone the repo and install everything (all workspace packages, editable, plus test/lint tooling)
with the `dev` dependency group:
Clone the repository and install everything with the `dev` dependency group: all workspace packages, editable, plus test and lint tooling.

```sh
uv sync --group dev
```

Before opening a PR, run the same checks CI runs:
Before opening a pull request, run the same checks CI runs:

```sh
uv run ruff format --check # formatting
uv run ruff check # lint
uv run ty check # type checking
uv run pyright # type checking
uv run codespell # spelling
uv run pytest --cov harp # tests
```

Adding a new package? Drop it under `src/packages/<name>/` with its own `pyproject.toml`, add it
to `[tool.uv.sources]` in the root `pyproject.toml`, and (if it should ship as part of `harp`) add
it to the root package's `dependencies` too.
To add a new package, place it under `src/packages/<name>/` with its own `pyproject.toml` and add it to `[tool.uv.sources]` in the root `pyproject.toml`. If it should ship as part of `harp`, add it to the dependencies of the root package as well.

## Building the documentation

Expand Down
63 changes: 20 additions & 43 deletions docs/examples/create_device_module/create_device_module.md
Original file line number Diff line number Diff line change
@@ -1,51 +1,28 @@
# Generating Registers from a Schema

This example demonstrates how to turn a Harp `device.yml` into a module of register
classes at runtime with `create_device_module`, without a code-generation step. This is the
quickest way to get started when you have only a device's schema and no
pre-generated package for it.

A generated device package is a module: register classes at module level, with a
`REGISTER_MAP` beside them keyed by address. `create_device_module` builds that same shape
from a schema, so registers are reached the same way either way — by name
(`behavior.AnalogData`) or by address (`behavior.REGISTER_MAP[44]`). From there they
work exactly like a pre-generated package's: drive them over a transport with
[`Device`](../../api/device.md) to talk to hardware, or use them to decode recorded
data.
This example demonstrates how to turn a Harp `device.yml` into a module of register classes at runtime with `create_device_module`, without a code-generation step. This is the quickest way to get started given only the schema of a device and no pre-generated package for it.

A generated device package is a module: register classes at module level, with a `REGISTER_MAP` beside them keyed by address. `create_device_module` builds that same structure from a schema, so registers are reached the same way, either by name as `behavior.AnalogData` or by address as `behavior.REGISTER_MAP[44]`. From there they work exactly like the registers of a pre-generated package. Pass the module to [`Device`](../../api/device.md) to talk to hardware, which validates the device identity on open, or use the registers with [`parse_to_dataframe`](../../api/data.md) to decode recorded data.

## When to use runtime generation

`create_device_module` trades statically generated device packages for schema-driven
convenience. It's worth understanding what that buys you and what it costs.

**You gain:**

- **No build step.** A `device.yml` — even one you just pulled off a device —
becomes a working module in a single call. There's nothing to generate, install,
or keep in sync with the schema.
- **Coverage for any device.** You don't need a published package for the device;
unreleased, custom, or one-off schemas work immediately.
- **The schema stays the single source of truth.** Registers, fields, and enums come
straight from the `device.yml`, under the same naming convention a generated
package uses — so code written against either lines up name for name.

**You give up:**

- **Static typing and autocomplete.** The names exist only once the module is built,
so an editor can't offer them and a type checker can't verify them. A generated
package is a real module on disk, so both work. The module also isn't in
`sys.modules`, so you bind it yourself rather than `import`-ing it.
- **Turn-key custom types.** A custom `interfaceType` must be injected yourself via
`converters=` (see below), whereas a generated package ships its own converters.

For shipped, widely-used devices a pre-generated package from the
[Harp C# generator](https://github.com/harp-tech/generators) remains the
authoritative choice — better editor support and static typing. Reach for
`create_device_module` when you want to go from a schema to working code with no
generation step.

!!! warning
Don't forget to change the `SERIAL_PORT` to the one that corresponds to your device! The `SERIAL_PORT` must be denoted as `/dev/ttyUSBx` in Linux and `COMx` in Windows, where `x` is the number of the serial port.
`create_device_module` trades statically generated device packages for schema-driven convenience. Both sides of that trade-off are worth understanding.

**Benefits:**

- **No build step.** A `device.yml`, even one just pulled off a device, becomes a working module in a single call. There is nothing to generate, install, or keep in sync with the schema.
- **Coverage for any device.** No published package is needed. Unreleased, custom, or one-off schemas work immediately.
- **Names match the generated package.** Registers, fields, and enums come straight from the `device.yml`, under the same naming convention a generated package uses, so code written against either lines up name for name.

**Limitations:**

- **Static typing and autocomplete.** The names exist only once the module is built, so an editor cannot offer them and a type checker cannot verify them. A generated package is a real module on disk, so both work. The module is also not in `sys.modules`, so it has to be bound rather than imported.
- **Reproducibility.** A generated package is a versioned dependency, so it can be pinned in a lock file and every install resolves the same register definitions. A runtime module is built from the `device.yml`, so the same analysis code can see different field names when it changes.
- **Turn-key custom types.** A custom `interfaceType` must be injected via `converters=`, shown below, whereas a generated package ships its own converters.

For widely-used devices a pre-generated package remains the authoritative choice, with better editor support, static typing, and a pinnable version. Reach for `create_device_module` to go from a schema to working code with no code-generation step.

{% include-markdown "includes/serial-port.md" %}

<!--codeinclude-->
```python
Expand Down
51 changes: 22 additions & 29 deletions docs/examples/create_device_module/create_device_module.py
Original file line number Diff line number Diff line change
@@ -1,48 +1,41 @@
from pathlib import Path

from harp.data import parse_to_dataframe
from harp.device.client import Device
from harp.device.schema import create_device_module
from harp.serial import open_serial_device
from harp import data
from harp import serial
from harp.device import schema

SERIAL_PORT = "/dev/ttyUSB0" # or "COMx" in Windows ("x" is the number of the serial port)
SERIAL_PORT = "/dev/ttyUSB0" # or "COMx" in Windows, where "x" is the serial port number

# `create_device_module` compiles a Harp `device.yml` into a module of register classes at
# runtime no code-generation step. This is the quickest way to work with a device
# when you don't have a pre-generated package for it: point it at the schema and you
# get the same shape a generated package has, registers at module level beside a
# runtime, with no code-generation step. This is the quickest way to work with a device
# that has no pre-generated package: point it at the schema and it produces the same
# structure a generated package has, registers at module level beside a
# `REGISTER_MAP`.
behavior = create_device_module(Path("device.yml").read_bytes())
behavior = schema.create_device_module(Path("device.yml").read_bytes())

print("WhoAmI:", behavior.WHO_AM_I) # device identity, taken from the schema
AnalogData = behavior.AnalogData # registers are reached by name...
assert behavior.REGISTER_MAP[44] is AnalogData # ...or by address
AnalogData = behavior.AnalogData # registers are reached by name
assert behavior.REGISTER_MAP[44] is AnalogData # or by address

# Registers are ordinary register classes, so they drive `read`/`write` on any
# `Device` over a transport. The schema carries no Python code, so there is no
# generated device class here: use `Device` itself.
with open_serial_device(Device, port=SERIAL_PORT) as device:
# Registers are ordinary register classes, so they work with `read` and `write` on
# any `Device` over a transport. Passing the module itself validates the device
# identity on open, against its `WHO_AM_I`, which a value of `0` skips.
with serial.open_serial_device(behavior, port=SERIAL_PORT) as device:
print("AnalogData:", device.read(AnalogData).parsed)

# ...or use the same register classes to decode a recorded binary dump into a
# pandas DataFrame (see the "Reading Data into a DataFrame" example for more):
df = parse_to_dataframe(AnalogData, "Behavior_44.bin")
# The same register classes also decode a recorded binary dump into a pandas
# DataFrame. See the "Reading Data into a DataFrame" example for more.
df = data.parse_to_dataframe(AnalogData, "Behavior_44.bin")
print(df.head())

# To have the identity checked on connect, pass the module itself. The check is
# driven by its `WHO_AM_I`, and `0` skips it:
#
# with open_serial_device(behavior, port=SERIAL_PORT) as device:
# print("AnalogData:", device.read(AnalogData).parsed)


# --- Custom interface types --------------------------------------------------
# A register with a custom `interfaceType` needs a converter so its field decodes
# to the right Python type. Pass it via `converters=`, keyed by "<Name>Converter":
#
# behavior = create_device_module(yml_text, converters={"DataConverter": DataConverter()})
# behavior = schema.create_device_module(yml_text, converters={"DataConverter": DataConverter()})
#
# An unresolved custom type raises `UnknownConverterError`; pass `strict=False` to
# decode it natively instead. `exclude_private=True` (the default) drops registers
# marked `private` in the schema. If you only want the parsed schema model rather
# than a module, `parse_device_schema(yml_text)` returns that directly.
# An unresolved custom type raises `UnknownConverterError`. Pass `strict=False` to
# decode it natively instead. `exclude_private=True`, the default, drops registers
# marked `private` in the schema. For the parsed schema model rather than a module,
# `parse_device_schema(yml_text)` returns that directly.
5 changes: 2 additions & 3 deletions docs/examples/get_info/get_info.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
# Getting Device Info

This example demonstrates how to connect to a Harp device, read its info and dump the device's registers.
This example demonstrates how to connect to a Harp device, read its information and dump the device registers.

!!! warning
Don't forget to change the `SERIAL_PORT` to the one that corresponds to your device! The `SERIAL_PORT` must be denoted as `/dev/ttyUSBx` in Linux and `COMx` in Windows, where `x` is the number of the serial port.
{% include-markdown "includes/serial-port.md" %}

<!--codeinclude-->
```python
Expand Down
Loading