fix(agentic): give KV cache series a real per-engine identity

[cquil11]

Summary

The KV cache utilization chart on the agentic detail page drew one line per raw metric series rather than one per engine. On DEP points that produced a tangle of duplicate lines over an unreadable legend — the 8-rank DP run at /inference/agentic/439261 drew 32 lines labelled DP 0 DP 0 DP 0 DP 0 DP 1 DP 1 ….

229 of the 436 stored rows that carry a per-engine breakdown were affected, at up to 8× duplication (64 lines).

Root cause

A RawSeries in the server_metrics_export.json blob is one (scrape endpoint × phase block × label set) tuple — not one engine. Three independent duplications were being folded straight into kvCacheUsageByEngine:

1. Phase blocks. CHART_SERIES_VERSION v12 merged the warmup_metrics block into metrics. The aggregate path keys by start_ns so it stayed continuous, but the per-engine path emitted one entry per series — so every engine appeared at least twice, and single-engine points started drawing a spurious two-line "per-engine" overlay that shouldn't exist at all.
2. Mirrored API-server frontends. vLLM run with several API servers exposes the same engine set on every /metrics endpoint, scraped ~176 ms apart. On 439261, ports :8895 and :8896 both report engine="0".."7" with values identical to 4+ decimal places → 8 DP ranks became 16 series.
3. Intra-engine shard ranks. A KV cache is allocated per engine and shared by its TP/PP/EP ranks, so each rank reports the same pool. A TP8 SGLang prefill worker (tp_rank=0..7, all mean≈0.05105) looked like 8 distinct engines.

The same fragmentation corrupted the cluster-average line. aggregateByStart groups on an exact start_ns, and scrapes from different endpoints/workers never share a nanosecond — so each output tick averaged only whichever subset happened to collide. On a disaggregated run that alternates between "prefill only" (≈0.005) and "decode only" (≈0.104): a full-scale sawtooth, not an average.

Secondary: the old engineLabel fell back to String(idx) — the index in the concatenated series array — so the same engine got a different label in each phase (4 in profiling, 9 in warmup).

Fix

Group series by their Prometheus label set (seriesIdentityKey): the label set is the series identity and endpoint_url is transport. Deployments whose endpoints really are distinct engines say so in the labels — Dynamo tags every series with worker_id, dynamo_component, engine_type — so prefill rank 0 and decode rank 0 stay separate. tp_rank / pp_rank / ep_rank / moe_ep_rank are excluded from the identity; engine / engine_idx / dp_rank are not, because those do name distinct pools.

Within an identity the three duplications get three treatments: phase blocks union by scrape instant, shard ranks at the same instant collapse to their mean, and mirrored endpoints (which overlap in time) resolve to the endpoint with the most complete coverage — merging them instead would interleave near-duplicate samples and halve the effective span of the frontend's fixed-width rolling average.

The cluster average becomes a true mean across logical engines on the union of their scrape instants, each engine holding its last sample until its next one and contributing only inside its own observed window (so an engine that starts late or stops early neither drags the mean toward a stale value nor drops it to zero).

Engines are labelled 0..N for bare DP ranks, role-qualified (prefill 0, decode) when the orchestrator gives a role, and worker-qualified (decode 0 (ee1b)) only when two would otherwise collide. The frontend renders bare numerics as DP N and passes self-describing labels through unchanged.

CHART_SERIES_VERSION 12 → 13. Run bun run --cwd packages/db db:backfill-chart-series after merge — until then the API serves these points via the slow recompute path (~70 s for the largest blobs).

Verification against real data

Recomputed from the actual stored blobs and compared to what v12 wrote. Roughness = mean |tick-to-tick delta| ÷ series stddev; a clean 1 Hz single-grid row sits near 0.18, so a value ≥ 0.6 is sawtooth rather than signal.

point	config	engines	avg level	roughness
439261	vllm / mi355x / dsv4 · DEP8 c=64	32 → 8	0.1843 → 0.1869	0.59 → 0.05
439292	dynamo-sglang / h200 / glm5.2 · disagg	64 → 18	0.1027 → 0.1285	1.00 → 0.02
439312	dynamo-sglang / gb300 / dsv4 · disagg	10 → 5	0.0465 → 0.0223	1.29 → 0.07
436497	vllm / b200 / dsv4 c=8	16 → 8	0.0516 → 0.0516	0.18 → 0.18
436433	vllm / b300 / dsv4 · DEP8 c=48	8 → 4	0.5159 → 0.5159	0.17 → 0.17
438866	vllm / h200 / kimik3	4 → 2	0.1411 → 0.1411	0.27 → 0.05
437312	dynamo-vllm / b200 / kimik3	2 → 0	0.0440 → 0.0440	0.13 → 0.13

Single-endpoint rows come out byte-identical on the average line. Disaggregated rows shift level on purpose: the mean now weights every engine once instead of over-weighting whichever subset shared a timestamp (on 439312, one decode engine at ~0.104 was previously carrying half the average against four prefill engines at ~0.006).

Label quality on 439312: [0, 0, 1, 1, 2, 2, 3, 3, 4, 9] → [prefill 0, prefill 1, prefill 2, prefill 3, decode].

Browser-verified against a local dev server on the real read-only DB (so the API took the live recompute path, not a fixture): 439261 renders 8 distinct DP lines + Avg with a readable 9-entry legend, in both the Profiling and Warmup stages; 436433 renders 4. The disagg per-source selector on 439312 still resolves each worker to a single engine (empty overlay, 4369-point average), confirming the recursive metricSources branch.

Scope deliberately not taken

Sum-combined metrics (queue depth, prefill/decode TPS, prefix-hit rate) sit on the same exact-start_ns grouping and show the same fragmentation on disaggregated rows (queue depth on 439312: mean tick delta 2.03 vs stddev 1.41). They are not touched here: their point count feeds running-sum cumulative charts (cumulativeUniqueInputTokens does sum += value, not a time integral), so resampling them onto a denser timeline would silently change every cumulative total. That needs its own change with its own verification.

Review pass

Three independent audits ran over this diff against the full production corpus (449 replay rows, 25 config families). They found one real regression introduced by the first commit and several latent hazards; all are fixed in the second commit.

Regression, proven on replay 654 (dynamo-vllm gb200, 4 decode workers):

	labels
v12	[0, 0, 1, 1, 2, 2, 3, 3]
first commit	[decode 3, decode 1, decode 0, decode 2]
now	[0, 1, 2, 3]

Two causes: the role prefix was applied even when every engine shared one role, and sorting keyed off the composed label so role-qualified names fell back to blob order — scrambling the ranks and, with them, the palette. Ordering is now a tuple over the identity components (role, numeric rank, worker, blob order), and the role is only shown when engines actually differ in role.

Hardening (each with a test): same-label endpoints are only fused when their values agree, so a router-fronted multi-replica topology cannot silently lose an engine — plain sglang emits an identity with no distinguishing field at all, and all 169 of its rows are single-endpoint today, so this was latent rather than live. The surviving mirror is picked by wall-clock coverage rather than raw sample count (on replay 815 the old rule was decided by one sample out of 3865). Carry-forward is capped at 5x an engine's own median scrape gap. engineRoleLabel falls through to dynamo_component instead of stopping at a present-but-unmapped engine_type.

Frontend: the average line's name, color, stroke and scatter are all driven by whether the point has multiple engines, so keying that off the phase-sliced array made the chart change identity between the Warmup and Profiling tabs. It now keys off the unsliced count, and engine colors come from the unsliced position.

Ops: maxDuration = 300 on the route. Every stored row is stale until the backfill drains it, the slow path re-parses blobs up to 448 MB compressed, and the blob cache only populates on success — so without this every visitor to a large point would re-pay a request the platform default had already cut off.

Corrections to claims the audits disproved. The first commit documented warmup and profiling blocks as covering disjoint time ranges. Their bounds do overlap (replay 820 by 67 s) — but measurement shows exactly one profiling sample lands inside the warmup window and there are zero exact-instant collisions, so the union is sound and both the comment and the doc now say that precisely. "TP ranks track each other to four decimal places" was also overstated: they agree exactly at 99.7-99.8% of instants, with whole-run means within 0.25%.

Post-review measurements:

point		engines	roughness
438898	dynamo-vllm/gb200/kimik3	8 → 4	0.734 → 0.009
439292	dynamo-sglang/h200/glm5.2	64 → 18	1.003 → 0.023
439261	vllm/mi355x/dsv4 DEP8	8	0.048

Known, deliberately out of scope

extractServerMetricSamples (packages/db/src/queries/agentic-aggregates.ts:142) is a second, independent implementation of the same KV averaging and still does the naive exact-start_ns grouping. It feeds aggregate_stats.kvCacheUtil on the Aggregates tab, so on a disaggregated point that tab can report a median of 0.0% while the Per-point chart sits at ~12%. Both were wrong before; now one is right. Fixing it means a STATS_VERSION bump and a second full backfill over the same 7.57 GB, and the right fix is probably to derive kvCacheUtil from chart_series rather than duplicate the logic a third time — so it belongs in its own PR. The two numbers are never on screen at the same time.

Separately, the sum-combined metrics (queue depth, prefill/decode TPS) share this bug's root cause and read 0.500x the true value on disaggregated points (measured against an all-worker carry-forward sum; a non-disagg control reads 0.991x). 14 rows affected. Not fixed here because their point count feeds running-sum cumulative charts — cumulativeUniqueInputTokens does sum += value, not a time integral — so correcting the instantaneous lines without also converting those helpers to integrate over dt would inflate every cumulative total.

Tests

• 8 new unit tests in compute-chart-series.test.ts covering mirrored frontends, phase joining, single-engine suppression, disagg worker separation, TP shard collapse, DP-vs-shard disambiguation, collision qualification, and sawtooth-free averaging on unaligned grids.
• 1 new E2E test asserting one legend entry per engine and that a self-describing label (decode) is not prefixed into DP decode.
• Full suite green: 3216 app + 463 db + 35 constants + 25 mcp unit tests, 11/11 in the agentic E2E spec, plus lint / fmt / typecheck.

Overlay (?unofficialrun=) support is not applicable — this path is the agentic per-point detail view, which loads a single stored trace replay and has no overlay rendering branch.

🤖 Generated with Claude Code

---

Note

Medium Risk
Chart-series algorithm changes affect many stored agentic points until db:backfill-chart-series runs; wrong mirror/replica fusion could mislabel engines, though tests and value-agreement guards mitigate this.

Overview
KV cache chart series (v13) no longer treat each raw Prometheus series as its own engine. compute-chart-series.ts groups by label identity (seriesIdentityKey), collapsing warmup/profiling duplicates, mirrored vLLM API-server scrapes, and TP/PP/EP shard ranks; cluster kvCacheUsage is averaged across engines on a union timeline with carry-forward and stale-sample limits instead of exact start_ns grouping that sawtoothed on disaggregated runs.

Display and UI: engines sort by role/rank/worker with smarter labels; the agentic KV card uses unsliced engine count for overlay vs average styling, maps numeric ranks to DP N, and leaves self-describing labels like decode unprefixed. trace-server-metrics sets maxDuration = 300 for slow recompute on pre-backfill rows. Docs add a “Logical Engines vs Raw Series” section; extensive unit and Cypress coverage exercises the new identity rules.

^{Reviewed by Cursor Bugbot for commit 1aedd4d. Bugbot is set up for automated code reviews on this repo. Configure here.}

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
inferencemax-app	Ready	Preview	Aug 10, 2026 12:39am