Add a device writer for the standardized Harp file format - #52
Open
bruno-f-cruz wants to merge 1 commit into
Open
Add a device writer for the standardized Harp file format#52bruno-f-cruz wants to merge 1 commit into
bruno-f-cruz wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a device writer for the standardized Harp file format
harp.datacould read a de-multiplexed dataset folder but nothing in the python client could produce one, so a recording had to come from Bonsai or from a script the user wrote againstDevice.subscribe_all.DeviceWriterandattach_writerclose that loop, writing the layout harp-tech/protocol#69 defines andDatasetReaderalready reads:The writer owns its subscription, so leaving the block detaches from the device and closes the files. The device outlives the recording, and several recordings can be made over one open device.
What a recording contains
Each message is appended to the file of its own register as the complete Harp frame it arrived as, header and checksum included. The specification prose says the payload is what is saved, but the reference C#
MessageWriterwritesinput.MessageBytes, andparse_to_dataframereads frames rather than bare payloads, so the implementation is what this follows. A file is then a run of frames that needs nothing else to be decoded.A register's file is created the first time a message for it arrives, so a folder holds exactly the registers that were seen rather than every register the schema declares.
DatasetReaderalready distinguishes those two cases throughcontents, so a register with no data reads as an empty DataFrame with the right columns either way.All three message types are recorded by default, where
subscribe_alldefaults toEventalone. A device answers reads and writes withReadandWritemessages, and the register dump the specification recommends under "Logging the device's initial configuration" arrives as a burst ofRead. Capturing onlyEventwould drop the configuration the session ran under. Requesting that dump stays the caller's to do, since it writes toOperationControland so changes device state (see example).DEVICE_METADATAA folder that cannot be decoded later is not worth writing, so the
device.ymlgoes in beside the binaries. That means the schema has to travel with the device module rather than with whatever file the caller happened to read.DeviceModuleLikegainsDEVICE_METADATA: bytes, alongsideDEVICE_NAME,WHO_AM_IandREGISTER_MAP, andcreate_device_modulefills it from the text it was given. A generated package declares it the same way, whichtests/device/expected_device.pypins.It is bytes rather than a stream. A
BytesIOon a module is a single shared handle with a read position, so the first reader exhausts it and every later one sees nothing, and every consumer has to remember to seek. Bytes have no position, so reading is not a one-shot andBytesIO(module.DEVICE_METADATA)still gets a file-like object where one is wanted.Because the member is required, a module carrying no metadata is broken rather than a folder to write undescribed, and the writer raises instead of silently producing one. There is no opt-out: the copy is not optional, and neither is the module.
This also means that the static code generator upstream MUST honor this interface and include the
device.ymlNaming files
Only the address varies over a recording, so a formatter is given an address and returns a name relative to the folder:
The device name and anything else a layout needs are bound when the formatter is built, which is what the default does with
DEVICE_NAME.Notable details
Writes are serialized under one lock and may come from any thread, and the frames of one register keep the order they were written in.
writeraises on a closed writer, while the subscription path drops a message already in flight at close, since there is nothing wrong with a recording having ended first and a raise there would surface as a logged handler traceback on every normal close.overwriteis off by default, so the first write to a path already on disk raisesFileExistsErrorrather than half-overwriting an existing recording. The folder and itsdevice.ymlare written eagerly at construction, so a path that cannot be written fails before any message is taken; a register file is opened on first use, so that failure arrives later in the run.attach_writerrefuses a device opened without a module, since the module is what names the files and describes the folder.tests/device/test_writer.pycovers the layout, the schema copy and its failures, overwrite, the lifecycle including concurrent writers,attach_writeragainst a scripted transport, and two round trips back throughDatasetReaderandopen_dataset.docs/examples/record-dataset.mdjoins the examples in the navigation, and shows the register dump in place.Closes #50