Skip to content

feat(stovepipe): report build failure detection latency - #573

Merged
mnoah1 merged 1 commit into
mainfrom
mnoah1/stovepipe-failure-detection-latency
Aug 12, 2026
Merged

feat(stovepipe): report build failure detection latency#573
mnoah1 merged 1 commit into
mainfrom
mnoah1/stovepipe-failure-detection-latency

Conversation

@mnoah1

@mnoah1 mnoah1 commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

Summary

Emit how long a break went undetected: when a validation build fails, record the elapsed time from the commit timestamp of the base it validated against, as a histogram tagged with the queue and the build strategy. A histogram rather than a gauge because the distribution over failures is the point — what an operator wants is how long a break typically survives, not how long the last one did. The buckets (ChangeAgeBuckets) span minutes to a month, since a break caught in minutes and one that survived a fortnight are both ordinary observations.

The observation lives in buildsignal, the stage that records the failure. A failure is the moment a break becomes known, and an elapsed time is only meaningful against it, so unlike the last-known-green age in #572 there is no later moment to sample it from — it cannot be moved off the delivery path onto the periodic schedule. The cost is a source-control call on that path, so it is confined to failures, made after the outcome is durable, and swallows every fault: a failed observation is counted with the step that failed and never disturbs the outcome already written.

A full build pins no base commit, so its failures have nothing to measure from. That is the ordinary case for the strategy rather than a fault, so those failures are counted as unmeasurable (detection_missing) instead of landing in the error series.

Rebased onto the restructured #572, so the emits follow the same conventions as the last-green observation there: an operation name for what is measured rather than for the stage, detection_errors tagged with the step that failed, a separate counter for "nothing to measure", and failures logged. The source-control factory is a required constructor dependency and wired in service/stovepipe/server/main.go.

Test Plan

  • make test — table-driven unit tests cover the measured path, the no-baseline (full build) case, and every step that can fail to observe: source control not resolving, ChangeInfo failing, an undated change, and a change dated in the future.
  • make lint, make check-gazelle, make check-tidy, make build.
  • Deploy and add a query for build_failure.time_to_detection, confirming detection_errors stays flat and detection_missing tracks only full-build failures.

Issues

Stack

  1. feat(stovepipe): report last-green freshness #572
  2. @ feat(stovepipe): report build failure detection latency #573

@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-lastgreen-freshness branch from b282822 to 029d68a Compare August 11, 2026 22:39
@mnoah1 mnoah1 closed this Aug 11, 2026
@mnoah1 mnoah1 reopened this Aug 11, 2026
@mnoah1 mnoah1 changed the title feat(stovepipe): report build failure detection latency [draft] feat(stovepipe): report build failure detection latency Aug 11, 2026
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-failure-detection-latency branch from 210c04c to 2f0c4d8 Compare August 12, 2026 16:02
@mnoah1 mnoah1 changed the title [draft] feat(stovepipe): report build failure detection latency feat(stovepipe): report build failure detection latency Aug 12, 2026
@mnoah1
mnoah1 marked this pull request as ready for review August 12, 2026 16:06
@mnoah1
mnoah1 requested review from a team, behinddwalls and sbalabanov as code owners August 12, 2026 16:06
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-failure-detection-latency branch from 2f0c4d8 to 55ce850 Compare August 12, 2026 16:06
Emit the elapsed time from the commit a failed build validated against, as a
histogram tagged with the queue and the build strategy. A histogram because the
distribution over failures is the point: how long a break typically survives,
not how long the last one did.

The observation belongs to record, the stage that turns a build outcome into
durable validation state. A break becomes known when its broken fact is written,
and an elapsed time is only meaningful against that moment — unlike the
last-known-green age, there is no later moment to sample it from, so it cannot be
moved off the delivery path onto a clock. It is confined to failures, made once
the fact is durable, and swallows every fault: a failed observation is counted
with the step that failed and never disturbs the fact already written. Only the
writer of a fact reports, so a redelivery that adopts a fact it already wrote
cannot count one break twice.

A full build pins no base commit, so its failures are counted as unmeasurable
rather than timed, which keeps the ordinary case out of the error series.
@mnoah1
mnoah1 force-pushed the mnoah1/stovepipe-failure-detection-latency branch from 55ce850 to ce2a856 Compare August 12, 2026 23:17
@mnoah1
mnoah1 changed the base branch from mnoah1/stovepipe-lastgreen-freshness to main August 12, 2026 23:18
@mnoah1
mnoah1 enabled auto-merge August 12, 2026 23:20
@mnoah1
mnoah1 added this pull request to the merge queue Aug 12, 2026
Merged via the queue into main with commit 00882d5 Aug 12, 2026
15 checks passed
@mnoah1
mnoah1 deployed to stack-rebase August 12, 2026 23:24 — with GitHub Actions Active
@behinddwalls
behinddwalls deleted the mnoah1/stovepipe-failure-detection-latency branch August 12, 2026 23:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants