Live evaluation of demo.fanout.run: 33 findings, P0 correctness first

[vishr]

Findings from a live evaluation pass over https://demo.fanout.run (image 2026.9.3), driven through the browser as a user would, with every numeric claim verified against the API rather than read off the screen.

The agent itself came out well: from a single service-map click it root-caused a 504 (checkout's PlaceOrder hung ~103s while cart, payment, shipping and email each finished under 15ms; the proxy gave up at its 15s budget and returned 504 UT response_timeout on an order that had actually succeeded). Asked about zero ERROR logs it named the telemetry blind spot instead of answering "0". Dashboard-from-chat works end to end. The list below is everything that got in the way.

Ordered by priority. Each item is independent; P0 items should land before the demo is put in front of anyone else.

---

P0 — correctness

1. Stat tiles read the last, partial bucket while the card is labelled "Last 1h"

The frontend service card shows Operations 1,281, P95 48.2ms, coloured green. For the same service and window, /api/observability/topology?window=1h returns 15,283 spans and p95 13,691ms, and the card's own Compare tab prints 13.69s → 10.41s. One card contradicts itself by ~285x on latency and ~12x on volume, and paints a 13.7s-P95 service as healthy.

The tile appears to take the most recent 5-minute bucket, which is also incomplete at read time, so it under-reports on both axes.

Fix: aggregate tiles over the labelled window, or label them explicitly as the latest interval and exclude the in-flight bucket.

2. An empty namespace filter reports "Healthy"

Typing a namespace that does not exist (opentelemetry-demo) renders Health: Healthy · 0 services · Error rate 0.00% · 0 operations. No-data is being rendered as good news.

Fix: distinguish "no data" from "healthy" in the health card and every downstream widget. Add namespace autocomplete sourced from known values so the state is hard to reach in the first place.

3. Log severity is not normalised at ingest

Across a 400-entry sample over 6h: INFO 130, empty string 54, INFORMATION 16. System-wide there were zero ERROR or WARN logs in 6h while the platform reported a 1.77% error rate and 9 unhealthy services. Envoy access logs from frontend-proxy arrive with no severity at all; the .NET cart service emits INFORMATION.

Consequences: every severity filter is unreliable, and error-log views are structurally empty even for a system actively returning 504s.

Fix: map OTLP severity_number on ingest, fold INFORMATION into INFO, and give unlabelled records an explicit unspecified value rather than an empty string.

4. The trace picker ignores service scope

A dashboard scoped to checkout pinned a "Slow PlaceOrder Trace" widget that rendered a flagd.evaluation.v1.Service trace — 2 spans, recommendation ↔ flagd, 600.01s. Neither the service nor the operation matches the scope.

5. Health scoring is dominated by long-lived background spans

P95 sits at exactly 600.00s for flagd and fraud-detection, and 510.01s for product-reviews — consumer and poll spans, not request latency. As a result frontend is marked unhealthy despite a 0% error rate and a 3.9ms p50, and the demo permanently reads "Unhealthy · 9 of 20".

Fix: compute request-path percentiles separately from consumer/internal spans, or exclude long-running background span kinds from the health signal.

---

P1 — gaps that stall an evaluation

6. No ingest path in the UI. Signed in as admin, there is no ingest token, endpoint or copy-paste snippet anywhere. Someone who likes the demo cannot take the next step.
7. No route from insight to action. After the agent names a failing service there is nothing to click: no "create an alert from this", no pin-to-dashboard, no saved query. /api/alerts exists and returns data; nothing in the UI reaches it.
8. No logs explorer. 125,713 logs over 6h are reachable only through the agent or a severity-filtered widget that is empty because of item 3.
9. The dashboard generator pins widgets it has not checked. The generated checkout dashboard shipped "Checkout Error Logs" and "Checkout Warnings", both permanently empty. Probe for rows before pinning, or render an explicit empty state at creation time.
10. No drill-down from a Recent activity row. Rows highlight on hover like links and do nothing. Clicking a service should lead to its traces and logs.
11. Search only matches chat and dashboard titles. Searching checkout returns "No matching chats / No matching dashboards" while a service, its traces and an entire thread about it all exist.
12. State is not in the URL. Time window and namespace live outside it, so a shared dashboard link never reproduces what the sender was looking at. A bad dashboard id silently renders the default dashboard while keeping the bogus URL; an unknown route silently redirects to /chat. Both should be a not-found state.
13. Widget "Remove" deletes immediately with no confirmation and no undo.

---

P2 — number presentation

14. Duration formatting and mixed units. 600.00s should read 10m; 240.02s should read 4m 0s. One column currently prints 30.00s next to 25.0ms, and one axis prints 0.0ms at zero and 100.00s above it. The "Latency and error correlation" chart exposes a scaling hack in its legend as Error rate × 1000 instead of using a right-hand percentage axis.
15. Threshold colouring is noisy. A 0.00% cell renders red; a p50 move of 3.9ms → 4.0ms is flagged as a red 2.2% regression; the errors column flips grey↔red around 1% with no legend. Use explicit thresholds with a key, or a continuous scale.
16. P95 equals P99 on most rows because values land on histogram bucket bounds (2.5 / 5 / 10 / 25 / 50ms). Interpolate, or mark the values as approximate.
17. Downsample long series. The 24h performance chart draws 288 raw points and reads as a hairball.

---

P3 — design system

18. The type scale is not a scale. One screen renders 15 distinct size/weight pairs across 9, 10, 10.4, 11, 12, 14, 15, 16, 20 and 32px. Collapse to roughly six steps and two weights, drop everything below 11px (the 9px DEFAULT badge and 10px service-map labels are the ones users actually need to read), and add a step between 20px and 32px.
19. The dark-mode primary button fails contrast. "New chat", white on violet, measures 3.16:1 — below the 4.5:1 AA threshold for 14px text. The same button in light mode measures 5.54:1.
20. Dark-mode muted text is warm on a cool ground (rgb(191,189,182) on #0b0e14). Match the hue family of the background.
21. Status is encoded by hue alone in the service map, health bar, service pills and errors column. Add a redundant channel — shape, fill or glyph — for colour-vision deficiency.
22. The service map inverts salience. Normal edges are #8b8e99 at 3.5–4.0px width; error edges are #f26d78 at 1.5px, so the failing dependencies are the thinnest and palest lines on the canvas. Labels also collide with nodes (otelcol-contrib over load-generator, fraud-detection over flagd), and the circular layout routes every chord through the centre. A layered or directed layout would show the real request shape for 20 nodes and 23 edges.
23. Two status encodings disagree on one row. ad shows a pink "unhealthy" service pill next to a grey 0.69% errors value.
24. Card layout wastes vertical space. The health tile is half empty because it is height-matched to a sparkline tile; Recent activity scrolls internally at 6 rows while the page below it is blank; the generated Performance widget renders at roughly two-thirds width leaving a gutter.
25. The error-rate sparkline carries no information. No baseline, no min/max, no axis, plus a decorative hatch fill. A number with a delta would say more in less space.
26. The latency heatmap legend overlaps the x-axis tick labels and the colour scale has no unit.

---

P4 — hygiene

27. Ten icon buttons expose no accessible name in the accessibility tree.
28. The chat view jumps to the top of the thread after sending, so the answer arrives off-screen.
29. The theme-toggle tooltip stays visible after the click.
30. A dashboard load double-fetches dashboards, overview, topology and performance.
31. /robots.txt and every unknown path return SPA HTML with a 200. A public demo should serve a real robots file. The Cloudflare beacon also 503s on load.
32. Intermittent multi-second stalls on the default window: one pass measured 9.0s for performance and 12.7s for logs at 1h, where repeat calls run 440–550ms. Consistent with rollup/maintenance contention on the write mutex.
33. Timestamps carry no timezone and there is no unit or timezone preference.

[vishr]

P0 items 1–5 and P1 items 9–13 are merged: #233 and #236. #235 fixes an unrelated ingest crash found while verifying them — a span without a status (optional in OTLP, omitted by most SDKs) panicked the export handler and lost the whole batch.

Review of #233 caught two further defects that are fixed in it: the new served_spans column would have dropped service_rollup on upgrade and never rebuilt it, because the stored watermark survived the drop; and the card's headline P95 could still be decided by a half-window spent holding a subscription open. Both have regression tests.

Still open from the list: P1 items 6–8 (ingest screen, alerts UI, logs explorer — each needs a product decision), P2 14–17 (number presentation), P3 18–26 (design system), P4 27–33 (hygiene). Two additions from the local verification: the trace card's Duration tile truncates to 600.…, and the post-login redirect can land on a chat route that does not exist.

[vishr]

P0 through P4 are merged: #233, #235, #236, #237, #238, #239, #240.

What is left on this list, and why:

32 — intermittent multi-second stalls. Not a hygiene fix. It is the rollup/maintenance contention on the write mutex already characterised in the throughput work: maintenance and rollups serialise against ingest and queries, so a query landing in that window waits. It needs a scheduling change, and it deserves its own issue rather than a line in this one.

7 — alerts have no UI, and 8 — no logs explorer. Both are product decisions rather than defects: what an alert list and a logs surface should be is a design question, not a gap to patch.

24 — the card layout (a half-empty health tile, a table that scrolls at six rows, a generated widget at two-thirds width) is layout work on the dashboard grid, not a defect in any one widget.

Three things worth recording from the work itself, because each was found by running the product rather than by reading it:

• Seeding a local instance to check the P0 latency fix crashed the ingest path on a span with no status — a nil dereference on the first real payload that omitted one (fix(ingest): a span without a status no longer takes the export down #235).
• The first P0 fix did not work. Window p95 was still reporting the histogram's top bound, because the per-bucket fallback was reading CLIENT spans in CLIENT-only buckets and the window figure was a MAX across buckets.
• The contrast fix in fix(ui): status that survives greyscale, and a button that clears AA (#232) #238 shipped twice before it reached the elements it claimed to: a variable that was correct in the theme and never landed on the component, and a badge scale that raised the sizes it was meant to leave alone. Both were invisible to tests that asserted configuration; three tests that render the real components now assert what lands on the element.

[vishr]

Item 32 is now #241, with the measurements and the shape of a fix.

Items 7, 8 and 24 stay here rather than becoming issues of their own: each is a question about what the surface should be, and #154 already holds that conversation for the chat and dashboard workspace.

[vishr]

Correction to item 32 and to my comment above: "contention on the write mutex" was wrong. There is no write mutex — PR #207 removed DuckLake, its catalog and that lock, and no read or ingest path takes the gate that replaced it. The sentence came from a stale model of the storage layer, not from a measurement; item 32 never timed a lock.

#241 has been rewritten: what the locks actually are, four candidate mechanisms (publish-gate queuing, connection-pool exhaustion, DuckDB checkpointing, cold Parquet bind), and the metrics that would name which one — one of which is a reader-side wait histogram the code does not have.