Parse BMF v1: benchmark parameters and named metric values

[epompeii]

Parse BMF v1 payloads: a benchmark name maps to an array of entries, each carrying a parameter set and its measures, with named metric values as scalars inside each measure.

This layer is parsing only. Nothing it produces reaches the database yet; the parameter aware ingest path is the next layer.

The adapter tree

json becomes a node over two new leaves, exactly like rust_gungraun is a node over rust_gungraun_json and rust_gungraun_stdout:

• json_v0 (discriminant 11) requires every benchmark to map to an object.
• json_v1 (discriminant 12) requires every benchmark to map to an array.

Each leaf is all or nothing, so a payload mixing the two shapes fails both leaves and therefore the node. There is no version field and no magic key, and BMF v0 parses byte for byte as it always has: the node tries v0 first, and every existing v0 fixture asserts identically through the node and through the leaf.

The leaves are individually selectable through the API's adapter setting but stay undocumented, which is the established pattern for rust_gungraun_json and rust_gungraun_stdout. The CLI's CliReportAdapter enum is untouched. Json stays in the identity arm of Adapter::normalize(): it is a format node, not a language node.

The leaves live in adapters/json/{mod,v0,v1}.rs rather than as flat siblings, matching every other adapter family in the crate (c_sharp/mod.rs over c_sharp/dot_net.rs). The node/leaf structure is gungraun's.

Internal results restructure

AdapterResults was HashMap<BenchmarkNameId, HashMap<MeasureNameId, JsonNewMetric>>, which can represent neither a grid point nor a named scalar. It is now:

• a benchmark name maps to BTreeMap<JsonParameters, AdapterMetrics>, its grid points keyed by canonical parameter set,
• a measure maps to AdapterMetric, a BTreeMap<MetricName, f64> of named scalars,
• and the results carry the BMF version they were parsed from plus the number of named values the cap dropped.

Every other adapter is unchanged: their constructors go through one helper that resolves to the empty parameter set, and From<JsonNewMetric> for AdapterMetric maps the triple onto exactly value, lower_value, and upper_value. A metric without bounds becomes exactly one name. No adapter outside json_v1 needs to know parameter sets exist, and none of their test assertions changed.

AdapterJson::parse used to deserialize straight into AdapterResults. That shortcut cannot survive two wire shapes, so each leaf now has its own wire type and converts.

bencher mock and the noise runner both emit BMF v0 and used to get it from AdapterResults's derived Serialize. They now build the v0 wire type (JsonV0Results) directly, so what they emit is pinned to the format rather than to an internal shape. AdapterResults no longer derives Serialize/Deserialize. Note that services/cli/src/bencher/sub/mock.rs is the only file touched under services/cli: the CLI's adapter surface is untouched, and the mock change is the one the spec for this layer calls for.

Absent parameters and {} are the same grid point

An entry without parameters resolves to the benchmark's empty parameter set, and an explicit {} canonicalizes to that same set, so the two are one grid point and not two. Two entries that land on the same canonical parameter set merge their measures rather than forking a series, which is what makes key order and number spelling irrelevant. Pinned by test.

Parameter values stay JSON scalars only, canonicalized per RFC 8785 by reusing JsonParameters. There is no second canonicalizer.

The named value cap

A const 8 named values per measure. Excess is dropped, never an error: a benchmarking pipe should not fail CI over excess statistics.

Survival is deterministic, because hash map iteration order is not an acceptable tiebreak. value, lower_value, and upper_value are never dropped; the remainder is kept in lexicographic order up to the cap. The cap fixture places seven names that sort before the conventional trio, so a naive lexicographic cut would drop value and upper_value and the test would catch it.

Deliberate split: the adapter truncates and exposes the drop count on its results. The log line and the otel counter land with the ingest layer, where the providers are in scope; bencher_adapter depends on neither a logger nor otel today and this layer does not add them.

Fold

--iter is untouched. Fold over BMF v0 is unchanged and the fold machinery is retained in full.

Fold is not supported for BMF v1, because the mean of per iteration p99 values is not the p99 of the pooled sample. That refusal is structural rather than a runtime check: combined, Add, Div, Sum, median, and fold moved onto FoldableResults/FoldableResultsArray, which are BMF v0 by construction, and the only way in is AdapterResultsArray::foldable(), which hands the array back untouched if any payload is v1. There is no arithmetic on named scalars anywhere, so a folded v1 result cannot be produced by accident.

Refusal keys on the payload version, not on which names a measure happens to carry: a v1 payload that spells only the conventional trio is still v1 and still refused. One v1 payload poisons the whole array, since fold spans every iteration.

The user-visible warn-and-ingest-unfolded behavior belongs to the fold deprecation layer.

Ingest, temporarily

ReportResults::process consumes the v0 view and returns a 400 for a v1 payload, since v1 does not reach the database yet. Before this change a v1 payload already returned a 400, from the adapter failing to convert it; the user-visible class is unchanged. The parameter aware ingest path removes this guard.

Console

adapterIcon, both adapterCommand switches, adapterName, and ViewCard's two switches take the new variants. validAdapter excludes them by omission, which is exactly how it excludes the gungraun leaves: they are undocumented. services/api/openapi.json and services/console/src/types/bencher.ts are regenerated.

One note for reviewers: adapterName was already a non-exhaustive switch before this change, since it never covered the language level variants (Rust, Cpp, Go, and the rest), so adding the JSON leaves was not forced by a compile error. astro check has a large pre-existing error baseline that includes that non-exhaustiveness and a good deal of ViewCard.tsx. It reports 758 errors both with and without this change, so this change adds none, and none of the errors reference the added variants.

Documentation

BMF v1 documentation ships with the parameters API layer. Deferred, not forgotten.

Tests, red before green

The first commit is the tests and fixtures plus the restructure scaffolding, with AdapterJsonV1::parse returning None and no cap truncation. At that commit cargo nextest run -p bencher_adapter --no-fail-fast reports 158 passed, 20 failed, and every one of the twenty is a v1 assertion: v1 parsing through the leaf, the node, and magic; the absent-versus-{} merge; canonicalization; both cap tests; and all four fold refusals. Every BMF v0 test passes there, which is what the restructure had to preserve.

json_v0's leaf is implemented in that commit because it is a pure refactor of behavior that already existed, and the v0 tests are regression guards rather than new behavior. Three of the new tests also pass at red for the same reason: json_v0 rejecting an array, json_v1 rejecting an object, and the mixed-version payload failing everything, all of which hold trivially while json_v1 parses nothing.

Gates

Run at the head of the branch:

• cargo nextest run: 2058 passed, 0 failed, 3 skipped
• cargo nextest run -p bencher_schema --features plus: 193 passed, 0 failed
• cargo test --doc: no failures
• cargo clippy --no-deps --all-targets --all-features -- -Dwarnings: clean
• cargo fmt --check: clean
• cargo check --no-default-features: clean
• cargo deny check: advisories, bans, licenses, and sources ok
• cargo gen-types: services/api/openapi.json and services/console/src/types/bencher.ts regenerated and committed
• ./scripts/clippy.sh and ./scripts/test.sh --linux-only (obligated by the bencher_noise change): both clean, cross compiled to x86_64-unknown-linux-gnu with zig
• Console: astro check reports 758 errors with and without this change, so it adds none, and none of them reference the added variants; biome check clean on the changed files

[github-actions]

Bencher Report

Project	Bencher
Branch	u/ep/benchmark-parameters/bmf-v1
Testbed	intel-v1

🚨 4 Alerts

Benchmark	Measure Units	View	Benchmark Result (Result Δ%)	Upper Boundary (Limit %)
Adapter::Json	Latency microseconds (µs)	📈 plot 🚷 threshold 🚨 alert (🔔)	5.25 µs (+11.80%) Baseline: 4.69 µs	4.99 µs (105.17%)
Adapter::Magic (JSON)	Latency microseconds (µs)	📈 plot 🚷 threshold 🚨 alert (🔔)	5.10 µs (+11.93%) Baseline: 4.55 µs	4.82 µs (105.77%)
Adapter::Rust	Latency microseconds (µs)	📈 plot 🚷 threshold 🚨 alert (🔔)	4.72 µs (+33.62%) Baseline: 3.53 µs	3.93 µs (120.04%)
Adapter::RustBench	Latency microseconds (µs)	📈 plot 🚷 threshold 🚨 alert (🔔)	4.67 µs (+32.25%) Baseline: 3.53 µs	3.93 µs (118.73%)

Click to view all benchmark results

Benchmark	Latency	Benchmark Result microseconds (µs) (Result Δ%)	Upper Boundary microseconds (µs) (Limit %)
Adapter::Json	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	5.25 µs (+11.80%) Baseline: 4.69 µs	4.99 µs (105.17%)
Adapter::Magic (JSON)	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	5.10 µs (+11.93%) Baseline: 4.55 µs	4.82 µs (105.77%)
Adapter::Magic (Rust)	📈 view plot 🚷 view threshold	27.06 µs (+4.80%) Baseline: 25.82 µs	27.12 µs (99.79%)
Adapter::Rust	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	4.72 µs (+33.62%) Baseline: 3.53 µs	3.93 µs (120.04%)
Adapter::RustBench	📈 view plot 🚷 view threshold 🚨 view alert (🔔)	4.67 µs (+32.25%) Baseline: 3.53 µs	3.93 µs (118.73%)

🐰 View full continuous benchmarking report in Bencher