Skip to content

Add runtime generation of device interface - #13

Merged
glopesdev merged 8 commits into
mainfrom
feat-runtime-register-binding
Aug 10, 2026
Merged

Add runtime generation of device interface#13
glopesdev merged 8 commits into
mainfrom
feat-runtime-register-binding

Conversation

@bruno-f-cruz

@bruno-f-cruz bruno-f-cruz commented Jul 26, 2026

Copy link
Copy Markdown
Member

Closes #12.

Summary

Adds a runtime compiler that turns a Harp device.yml into typed RegisterBase / Device classes, mirroring the upstream C# generator (harp-tech/generators, Python target in PR #119) but emitting descriptors at runtime instead of generating static source code. A user with only a device.yml gets typed registers and a device, and the C# generator still stays authoritative for shipped device packages.

All device.yml knowledge converges in harp-device (harp.device._schema). harp-protocol gains only reusable primitives (format_bulk); harp-benchmarks is refactored onto them.

Public API

from harp.device import create_device, parse_device_schema, ConverterContext

Behavior = create_device(device_yml_text)   # -> type[Device]
Behavior.REGISTER_MAP                        # {**CORE_REGISTER_MAP, 33: AnalogData, ...}
Behavior.__whoami__                          # from the schema's whoAmI (0 if absent)
  • Registers are reached by address through REGISTER_MAP — they are not attached as Device attributes (something to potentially improve in the future?).

  • Custom interfaceTypes are injected, not stubbed — the runtime analog of the generator's from .converters import <Name> contract:

    create_device(yml, converters={"QuaternionConverter": QuaternionConverter()})

    Missing a converter raises UnknownConverterError when strict=True (default); strict=False falls back to native type decoding (e.g: u16). Values are a Converter instance or a (ConverterContext) -> Converter factory (the ctx carries the resolved element dtype/size, mask, length, name — so a factory builds StringConverter(span) / HarpVersionConverter(element) with the right args).

  • exclude_private=True drops registers whose DSL visibility is private.

Change surface

  • harp-protocol
    • RegisterBase.format_bulk(values, *, timestamps=None, message_type=Event, port=255) — the exact vectorized inverse of parse_bulk (payload encode + header/stride + optional timestamp block + checksum in one pass). Also the typed-corpus generator used by benchmarks and fixtures.
    • GroupMask decode now preserves undefined codes as their raw value, matching the C# generator's unchecked enum cast (Interface.cs). Previously an in-range gap code decoded to -1, a code above the largest member IndexError'd, and the scalar accessor raised. Now — consistently across scalar and batch — a defined code yields the enum member and an undefined code is kept as the raw int (its own category in a dataframe, never NaN). A per-field _lookup_safe flag keeps the bounds-check-free gather for narrow masked sub-fields (the ones in high-rate registers); whole-element masks over sparse enums take a guarded gather.
  • harp-device — new private harp.device._schema subpackage:
    • _model.py — pydantic v2 object model of device.yml (hand-maintained; regeneration command against the pinned upstream registers.json recorded in the module). There is a TODO here in the future to ensure this doesnt drift by using a json-pydantic generator
    • _emit.pyparse_device_schema(text) and create_registers(...), built on a single _INTERFACES converter table (every built-in interfaceType — the 9 primitives plus string/bool/HarpVersion.
    • _emit_device.pycreate_device(...) façade. create_device / parse_device_schema / ConverterContext (+ the converter aliases) are the public exports.
  • harp-benchmarks — corpora are now random per-frame via format_bulk (seeded per register, bytes held to 0..127) instead of formatting one frame and byte-repeating it. BenchmarkedRegister is now (name, register, timestamped); the hand-authored value fixtures are gone.

Divergence from the C# generator

  • Identifiers are kept verbatim from the yml (AnalogInput0, DIO0), whereas the generator snake_cases members and UPPER_SNAKEs enum members. Deliberate: it avoids reimplementing and keeping in sync Bonsai.Harp's FirmwareNamingConvention. The yml is the single source of truth for names.
  • converter: Payload and RawPayload emit identical behavior

Testing

  • Layout/type golden — emitting the vendored device.yml reproduces PR #119's expected_device.py register-for-register; common.yml matches expected_core.py (structural for DeviceName, whose core metadata the generator enriches to a string).
  • Golden bulk round-trip — per device register, emitted vs oracle class are wire- and dataframe-compatible for the same payload bytes (bytes(emitted.format_bulk(r)) == bytes(oracle.format_bulk(r)), and parse_to_dataframe equality after a positional column rename).
  • GroupMask preserve-raw — undefined codes preserved as raw ints, scalar matches batch (test_payload.py, test_register_modeling.py).

Not in this PR

@bruno-f-cruz
bruno-f-cruz marked this pull request as ready for review July 26, 2026 03:02
@bruno-f-cruz
bruno-f-cruz requested a review from glopesdev July 26, 2026 03:03
Narrow the Any-typed inputs of format_bulk / to_buffer / to_file to
PayloadBase | ArrayLike, ArrayLike | None, and MessageType | ArrayLike.
Give Device a base REGISTER_MAP ClassVar default ({}) that generated
subclasses override, and tidy the create_device docstring.
@bruno-f-cruz
bruno-f-cruz force-pushed the feat-runtime-register-binding branch from c1e4787 to e4cb977 Compare July 26, 2026 05:08
Add a "Generating a Device from a Schema" example walking through
`create_device`, including a win/loss discussion of runtime generation
vs. pre-generated device packages. Wire it into the examples index and
nav, surface `create_device`/`parse_device_schema`/`ConverterContext` in
the device API reference, and add a Quickstart to the root README.
@bruno-f-cruz
bruno-f-cruz force-pushed the feat-runtime-register-binding branch from de70211 to 933071c Compare July 26, 2026 16:57
@glopesdev
glopesdev merged commit 94c7ad9 into main Aug 10, 2026
13 checks passed
@glopesdev
glopesdev deleted the feat-runtime-register-binding branch August 10, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Runtime generation of device interfaces from device.yml

2 participants