feat(stovepipe): report build failure detection latency - #573
Merged
Conversation
mnoah1
force-pushed
the
mnoah1/stovepipe-lastgreen-freshness
branch
from
August 11, 2026 22:39
b282822 to
029d68a
Compare
mnoah1
force-pushed
the
mnoah1/stovepipe-failure-detection-latency
branch
from
August 12, 2026 16:02
210c04c to
2f0c4d8
Compare
mnoah1
marked this pull request as ready for review
August 12, 2026 16:06
mnoah1
requested review from
a team,
behinddwalls and
sbalabanov
as code owners
August 12, 2026 16:06
mnoah1
force-pushed
the
mnoah1/stovepipe-failure-detection-latency
branch
from
August 12, 2026 16:06
2f0c4d8 to
55ce850
Compare
behinddwalls
approved these changes
Aug 12, 2026
Emit the elapsed time from the commit a failed build validated against, as a histogram tagged with the queue and the build strategy. A histogram because the distribution over failures is the point: how long a break typically survives, not how long the last one did. The observation belongs to record, the stage that turns a build outcome into durable validation state. A break becomes known when its broken fact is written, and an elapsed time is only meaningful against that moment — unlike the last-known-green age, there is no later moment to sample it from, so it cannot be moved off the delivery path onto a clock. It is confined to failures, made once the fact is durable, and swallows every fault: a failed observation is counted with the step that failed and never disturbs the fact already written. Only the writer of a fact reports, so a redelivery that adopts a fact it already wrote cannot count one break twice. A full build pins no base commit, so its failures are counted as unmeasurable rather than timed, which keeps the ordinary case out of the error series.
mnoah1
force-pushed
the
mnoah1/stovepipe-failure-detection-latency
branch
from
August 12, 2026 23:17
55ce850 to
ce2a856
Compare
mnoah1
changed the base branch from
mnoah1/stovepipe-lastgreen-freshness
to
main
August 12, 2026 23:18
mnoah1
enabled auto-merge
August 12, 2026 23:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Emit how long a break went undetected: when a validation build fails, record the elapsed time from the commit timestamp of the base it validated against, as a histogram tagged with the queue and the build strategy. A histogram rather than a gauge because the distribution over failures is the point — what an operator wants is how long a break typically survives, not how long the last one did. The buckets (
ChangeAgeBuckets) span minutes to a month, since a break caught in minutes and one that survived a fortnight are both ordinary observations.The observation lives in
buildsignal, the stage that records the failure. A failure is the moment a break becomes known, and an elapsed time is only meaningful against it, so unlike the last-known-green age in #572 there is no later moment to sample it from — it cannot be moved off the delivery path onto the periodic schedule. The cost is a source-control call on that path, so it is confined to failures, made after the outcome is durable, and swallows every fault: a failed observation is counted with the step that failed and never disturbs the outcome already written.A full build pins no base commit, so its failures have nothing to measure from. That is the ordinary case for the strategy rather than a fault, so those failures are counted as unmeasurable (
detection_missing) instead of landing in the error series.Rebased onto the restructured #572, so the emits follow the same conventions as the last-green observation there: an operation name for what is measured rather than for the stage,
detection_errorstagged with the step that failed, a separate counter for "nothing to measure", and failures logged. The source-control factory is a required constructor dependency and wired inservice/stovepipe/server/main.go.Test Plan
make test— table-driven unit tests cover the measured path, the no-baseline (full build) case, and every step that can fail to observe: source control not resolving,ChangeInfofailing, an undated change, and a change dated in the future.make lint,make check-gazelle,make check-tidy,make build.build_failure.time_to_detection, confirmingdetection_errorsstays flat anddetection_missingtracks only full-build failures.Issues
Stack