Retry transient Kubernetes API errors in chi kube driver Get calls#2025
Open
aaron276h wants to merge 1 commit into
Open
Retry transient Kubernetes API errors in chi kube driver Get calls#2025aaron276h wants to merge 1 commit into
aaron276h wants to merge 1 commit into
Conversation
e3bd728 to
432c51d
Compare
A single transient apiserver/network error (connection refused/reset, timeout, 5xx) during a CHI reconcile currently propagates up as-is and aborts the entire reconcile, discarding progress on all remaining hosts. On installations with hundreds of hosts a reconcile makes tens of thousands of API calls over a long wall-clock window, so even a sub-second control-plane blip is likely to hit one of them and abort an otherwise healthy rollout, leaving the CR stuck in Aborted state. Wrap all Get calls in the chi kube drivers with a bounded exponential-backoff retry (5 attempts, 500ms..4s doubling, ~7.5s total budget) that retries only transient network/API errors. Terminal errors (NotFound, AlreadyExists, Conflict, Forbidden, Unauthorized, BadRequest, Invalid, MethodNotSupported) and context cancellation surface immediately, and when retries are exhausted the last error is returned unchanged - so callers behave exactly as before for non-transient errors and sustained outages. Signed-off-by: Aaron Harlap <aaron@chronosphere.io>
432c51d to
0b02e31
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2026.
Problem
Any Kubernetes API error during a CHI reconcile propagates up unchanged and the top-level handler aborts the entire reconcile (
ErrCRUDAbort->status: Aborted). Transient network errors (connection refused/reset, timeouts, apiserver 5xx) are treated the same as terminal ones (NotFound, Forbidden, ...).On installations with hundreds of hosts, a reconcile issues tens of thousands of API calls over a long wall-clock window, so even a sub-second control-plane blip is nearly certain to land on one of them. We hit this repeatedly in production on a 387-host CHI:
with the abort landing at 385/387 hosts completed — nothing wrong in ClickHouse, one refused GET discarded the whole rollout. The 0.27 auto-recovery (
reconcile.recovery.from.aborted.onPodReady) does not cover this case: all pods were already Ready, so no NotReady->Ready transition ever fires and the CR staysAborteduntil an operator restart or spec edit.Change
getWithRetryhelper andisTransientAPIErrorclassifier inpkg/controller/chi/kube.Getcalls in the chi kube drivers (ConfigMap, CR, PDB, Pod, PVC, Secret, Service, StatefulSet) retry transient errors with bounded exponential backoff: 5 attempts, 500ms -> 4s doubling (~7.5s worst-case budget).net.Error).Getdominates the reconcile hot path (readiness polling) and is unconditionally safe to retry. Happy to extend toList(equally safe) if you'd like it in this PR.Testing
url.Error->net.OpError->ECONNREFUSEDnesting client-go produces — and for the retry loop (first-try success, retry-then-success, terminal short-circuit, retry exhaustion, context cancellation).Notes for reviewers
V(2); giving up logs atV(1).