Skip to content

Retry transient Kubernetes API errors in chi kube driver Get calls#2025

Open
aaron276h wants to merge 1 commit into
Altinity:0.27.2from
aaron276h:retry-transient-kube-api-errors
Open

Retry transient Kubernetes API errors in chi kube driver Get calls#2025
aaron276h wants to merge 1 commit into
Altinity:0.27.2from
aaron276h:retry-transient-kube-api-errors

Conversation

@aaron276h

@aaron276h aaron276h commented Jul 2, 2026

Copy link
Copy Markdown

Fixes #2026.

Problem

Any Kubernetes API error during a CHI reconcile propagates up unchanged and the top-level handler aborts the entire reconcile (ErrCRUDAbort -> status: Aborted). Transient network errors (connection refused/reset, timeouts, apiserver 5xx) are treated the same as terminal ones (NotFound, Forbidden, ...).

On installations with hundreds of hosts, a reconcile issues tens of thousands of API calls over a long wall-clock window, so even a sub-second control-plane blip is nearly certain to land on one of them. We hit this repeatedly in production on a 387-host CHI:

FAILED to reconcile CR <ns>/<chi>, err: Get ".../configmaps/chi-<chi>-deploy-confd-data-a-87-0":
  dial tcp 10.x.0.1:443: connect: connection refused

with the abort landing at 385/387 hosts completed — nothing wrong in ClickHouse, one refused GET discarded the whole rollout. The 0.27 auto-recovery (reconcile.recovery.from.aborted.onPodReady) does not cover this case: all pods were already Ready, so no NotReady->Ready transition ever fires and the CR stays Aborted until an operator restart or spec edit.

Change

  • New getWithRetry helper and isTransientAPIError classifier in pkg/controller/chi/kube.
  • All Get calls in the chi kube drivers (ConfigMap, CR, PDB, Pod, PVC, Secret, Service, StatefulSet) retry transient errors with bounded exponential backoff: 5 attempts, 500ms -> 4s doubling (~7.5s worst-case budget).
  • Transient = apiserver ServerTimeout/Timeout/TooManyRequests/InternalError/ServiceUnavailable/UnexpectedServerError, plus transport-level errors (ECONNREFUSED, ECONNRESET, EPIPE, ETIMEDOUT, EOF, net.Error).
  • Terminal API errors (NotFound, AlreadyExists, Conflict, Forbidden, Unauthorized, BadRequest, Invalid, MethodNotSupported) and context cancellation return immediately, and exhausted retries return the last error unchanged — callers behave exactly as before for non-transient errors and sustained outages.
  • Scope is deliberately reads-only: Get dominates the reconcile hot path (readiness polling) and is unconditionally safe to retry. Happy to extend to List (equally safe) if you'd like it in this PR.

Testing

  • Unit tests for the classifier — including the exact url.Error -> net.OpError -> ECONNREFUSED nesting client-go produces — and for the retry loop (first-try success, retry-then-success, terminal short-circuit, retry exhaustion, context cancellation).

Notes for reviewers

  • Backoff constants are package vars only so tests can shrink them; treated as read-only at runtime.
  • Per-attempt retries log at V(2); giving up logs at V(1).

@aaron276h aaron276h force-pushed the retry-transient-kube-api-errors branch from e3bd728 to 432c51d Compare July 2, 2026 19:06
@aaron276h aaron276h marked this pull request as ready for review July 2, 2026 20:11
A single transient apiserver/network error (connection refused/reset,
timeout, 5xx) during a CHI reconcile currently propagates up as-is and
aborts the entire reconcile, discarding progress on all remaining
hosts. On installations with hundreds of hosts a reconcile makes tens
of thousands of API calls over a long wall-clock window, so even a
sub-second control-plane blip is likely to hit one of them and abort
an otherwise healthy rollout, leaving the CR stuck in Aborted state.

Wrap all Get calls in the chi kube drivers with a bounded
exponential-backoff retry (5 attempts, 500ms..4s doubling, ~7.5s total
budget) that retries only transient network/API errors. Terminal
errors (NotFound, AlreadyExists, Conflict, Forbidden, Unauthorized,
BadRequest, Invalid, MethodNotSupported) and context cancellation
surface immediately, and when retries are exhausted the last error is
returned unchanged - so callers behave exactly as before for
non-transient errors and sustained outages.

Signed-off-by: Aaron Harlap <aaron@chronosphere.io>
@aaron276h aaron276h force-pushed the retry-transient-kube-api-errors branch from 432c51d to 0b02e31 Compare July 3, 2026 00:50
@sunsingerus sunsingerus added the planned for review This feature is planned for review label Jul 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

planned for review This feature is planned for review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants