Properly determine libusb read size for large reports (Fixes #274) - #728
Properly determine libusb read size for large reports (Fixes #274)#728sudobash1 wants to merge 16 commits into
Conversation
068fd62 to
52c6cd9
Compare
|
PR updated to remove different signedness comparison compiler warning when compiling with |
|
Thank you for finally taking care of this longstanding bug. |
|
BTW, there is a typo here. |
|
I put The InputReport contains:
|
There is something missing, as this is autogenerated by pp_data_dump, I guess there was something in the manufacturer_string , that pp_data_dump couldn't handle. Could you please open a dedicated issue for this, as this is unrelated to this PR. |
|
2708264 to
3710895
Compare
|
Thanks @JoergAtGithub for walking me through that. I was misreading the descriptor. I have added tests for libusb using the same data as the windows tests. And I extended the functionality of the libusb method to be able to calculate the maximum output and feature report sizes as well. This has no functional use currently, but it lets us run three times as many tests since the pp_data files have all three max sizes available. |
Independent of this PR, I think it would generally make sense to store these 3 values in the device structure. On Windows we would have to use the values |
Youw
left a comment
There was a problem hiding this comment.
I don't have a strong opinion on test implementation (I trust @JoergAtGithub on this one).
libusb implementation seem fine.
Lets make sure it runs with CI on Github Actions and we're good to go here.
Lets continue here: #731 |
|
This PR does not seem to work. Test device is the same as the one used in Issue #274. The FW is a mod of Jan Axelson's FX2HID example and codes are included in the following #274 comment. I can reproduce the issue reported in #274 with hidapi git libusb backend, hidraw backend is okay. With this PR, no change in hidraw backend behavior, which is good. But the libusb backend fix is not working. |
|
HID Report Descriptor. hidtest is okay. |
|
If I have time, I'll try to address @JoergAtGithub and @Youw's comments on Monday. @mcuee I tried feeding the report descriptor that you sent into I did notice a typo in your test. When you ran |
|
Let me try again. |
|
Same problem under FreeBSD 14.1 Release. This is with a physical machine, Chuwi mini PC, Intel J4125 CPU, 8GB/256GB configuration. Without this PR, hidapi git has the problem mentioned in #274. But then somehow this PR does not work. |
|
If I use the original fx2hid example (loop back of two bytes output report and input report), it seems to me this PR works fine. But that just means this PR has no regression. |
a4c4da9 to
e19ba07
Compare
|
Just wondering if you have any updates on this one. I think it is pretty close now. |
|
Just update the branch to trigger a new build. It will be good to move this PR forward. May need more testing and reviews. |
|
Just wondering if it is worth asking Github Copilot to review this PR. It seems to be pretty close to be okay. |
There was a problem hiding this comment.
Coming back to this after dropping it for a while — did a spec/kernel cross-check pass and a code re-read. Leaving hardware testing to those with the devices.
Algorithm validation
get_max_report_size matches HID 1.11 (§6.2.2.7, §8 for Report ID semantics) and is structurally identical to Linux's report-descriptor parsing in drivers/hid/hid-core.c:
hid_add_field():report->size += parser->global.report_size * parser->global.report_count;hid_register_report():if (id != 0) report_enum->numbered = 1;- On receive, one byte is consumed as the ID prefix iff
numbered— which is exactly the+ report_id_usedsemantic here.
The FX2 128-byte fix is the subtle one: the hang wasn't the size math, it was the unconditional +1 making the buffer a non-multiple of wMaxPacketSize and suppressing the libusb short-packet terminator. Good catch on making it conditional on the Report ID tag actually being used.
Reusing windows/test/data/*.pp_data for a cross-backend parity test is a nice consistency proof.
Suggestions before merge
- Scope of "Fixes #274" — this resolves the long-report truncation aspect, but #274 also asks for Full-Speed misconfiguration detection / reporting to the app. Consider softening to "Partially fixes" or adding a note.
- Platform length asymmetry that @mcuee observed on Windows (libusb
wrote 130 / read 129vs native HIDwrote 129 / read 128) — worth a sentence in the PR body pointing to #731 so users don't re-file it as a regression. - Inline nits on specific lines below.
Algorithm + spec alignment LGTM on the core change.
Extended version:
Comparison with the spec and other implementations
HID 1.11 spec (§6.2.2.7 & §5.6 / §8): "If a Report ID tag is used anywhere in Report descriptor, all data reports for the device are preceded by a single byte ID field." Report ID = 0 is a placeholder meaning "no report IDs". Reports are sized as Σ(report_count × report_size) bits, rounded up to bytes. The PR's algorithm and the conditional +report_id_used byte match the spec exactly.
Windows (HIDP_CAPS): InputReportByteLength/OutputReportByteLength/FeatureReportByteLength — these always include the Report-ID byte (value 0 when unused). That's the source of the read/write-length platform asymmetry and is exactly what follow-up issue #731 is about. The PR's test case subtracts 1 byte from these values when ReportID == 0x00 to reconcile with the libusb convention (see parse_max_input_report_size in max_input_report_size_test.c).
Linux kernel (drivers/hid/hid-core.c): Uses the same algorithm. hid_add_field() accumulates report->size += parser->global.report_size * parser->global.report_count. In hid_register_report(), if (id != 0) report_enum->numbered = 1;. In hid_report_raw_event, when numbered is true the first data byte is consumed as the ID (cdata++; csize--;). Size is bounded by max_buffer_size - 1 bits when numbered. This is structurally identical to what the PR does — so the PR's approach is directly validated by the reference kernel implementation.
hidapi/linux (hidraw backend): Avoids parsing entirely — delegates to the kernel: read(dev->device_handle, data, length) with user-supplied length, because the kernel has already sized the per-report buffer. That's why hidraw "just works" for the 128-byte FX2 firmware and why the libusb backend is the outlier.
Strengths
Correct algorithm. Matches HID 1.11 and Linux kernel's parsing exactly, including the subtle conditional Report-ID byte.
Good test coverage via reuse of 25 real-world Windows pp_data fixtures — this is a nice cross-backend consistency proof.
Safe fallback to input_ep_max_packet_size when the descriptor is corrupt/unparseable — no regression risk for devices already working.
Fixes the classic "exact multiple of max packet size hangs" libusb gotcha — the conditional +1 was the critical insight; without it devices with payloads that happen to be multiples of 64 would hang forever waiting for a short packet.
Remaining concerns worth flagging on review
Re-read of report descriptor. hidapi_initialize_device now calls hid_get_report_descriptor_libusb during init (new control transfer on every open). This is a second descriptor fetch in addition to any caller-driven hid_get_report_descriptor(). Consider caching.
HID_API_MAX_REPORT_DESCRIPTOR_SIZE buffer on stack (4096 bytes typically). libusb/hid.c:1314 in the diff allocates this on the stack — acceptable, but worth noting for embedded targets.
cur_size reset timing. In the parser, cur_size is reset on each Report ID encounter. For descriptors that interleave INPUT/OUTPUT items between Report ID changes this is correct; just note that the max_size comparison happens only at Report-ID boundaries and at the end — which matches the spec semantics where each Report ID opens a fresh report namespace.
Platform-length asymmetry (mcuee's Windows observation) is intentionally out of scope — tracked in #731. Worth linking from the PR description.
PR description still says "Fixes #274" but #274 has broader asks (detection of USB Full-Speed misconfiguration, speed reporting to app). This PR fixes the "long reports get truncated" aspect; the speed/error-reporting aspect remains open.
Typo in test comment (.dpt_desc → .rpt_desc) in libusb/test/CMakeLists.txt.
Bottom line
The PR is technically sound, aligned with HID 1.11 and the Linux kernel reference, and has been hardware-validated on Linux, FreeBSD and Android. The earlier "doesn't work" reports were a spec-compliance bug in the Report-ID handling that sudobash1 fixed. Primary blockers now are administrative: final Youw approval + merge, plus perhaps addressing the minor concerns above. The broader cross-platform length-consistency question is correctly deferred to #731.
Drafted with Claude Code.
|
This is the result of the review by Claude Opus 4.7 (locally). |
I think this statement is not correct. The USB standard specifies the max read size per speed category. With this PR every read size should work and therefore every USB speed. |
Nice reviews. Just want to check if I read the comments correctly.
In that case, I think this PR can be merged. |
|
Latest test results with hidapi git. This PR is OK. Windows FreeBSD |
|
Latest results under Linux. This PR is okay. |
There is still a number of unresolved comments across the code. Need to either handle or a reason to dismiss, before merge this. (Or move out into separate issue/ticket. |
|
No regression for the following HID device under Linux (with non-Zero Input/Output/Feature reports IDs). |
|
No regression under Windows either. Minor modification needs to be used to get hidapitester to build with hidapi git or this PR, using libusb backend. Using WinUSB driver. |
|
@sudobash1 It would be great if you could address the lost open comments in the code review. |
|
@sudobash1 It would be great if you could address the last remaining comments in this PR. |
Test real report descriptors, distinguish malformed descriptors, cache descriptor data, and run the libusb tests in CI. Assisted-by: codex:gpt-5.6-sol
Bring PR libusb#728 onto the current base and retain the libusb report-size tests alongside the new virtual-device test suite. Assisted-by: codex:gpt-5.6-sol
Preserve endpoint-packet compatibility, bound descriptor-derived buffers, unwind read-thread initialization failures, avoid non-HID descriptor requests, and expand cross-platform parser coverage. Assisted-by: codex:gpt-5.6-sol Assisted-by: claude-code:claude-fable-5
Resolve the libusb CMake conflict by retaining upstream's generalized pkg-config generation together with the report-descriptor parser tests. Assisted-by: codex:gpt-5.6-sol
Youw
left a comment
There was a problem hiding this comment.
After latest fixes from codex - this needs to be tested
Summary
wMaxPacketSizeare delivered whole.wMaxPacketSizeas the minimum buffer size for devices that pad short reports, and fall back to it when a descriptor cannot be read or parsed safely.hid_get_report_descriptor()._real.rpt_descfixtures plus malformed, overflow, Report ID, Push/Pop, long-item, and boundary cases.Verification
-Wall -Wextra -pedantic -Werror: 26 runnable tests passed; hardware-only DeviceIO test skipped.-Wall -Wextra -Werror: 26 runnable tests passed; hardware-only DeviceIO test skipped.HIDAPI_WITH_TESTS=ONand registers all 26 libusb tests.Fixes #274
Related to #796
Related to #833
Assisted-by: codex:gpt-5.6-sol
Assisted-by: claude-code:claude-fable-5