Skip to content

fix(cache): match client-go's reconnect backoff algorithm - #3026

Draft
davidgamero wants to merge 1 commit into
kubernetes-client:mainfrom
davidgamero:fix/watch-backoff-jitter
Draft

fix(cache): match client-go's reconnect backoff algorithm#3026
davidgamero wants to merge 1 commit into
kubernetes-client:mainfrom
davidgamero:fix/watch-backoff-jitter

Conversation

@davidgamero

@davidgamero davidgamero commented Aug 30, 2026

Copy link
Copy Markdown
Contributor
  • Swapped the backoff constants for client-go's defaultBackoff*: 800ms base, ×2, 30s cap, full jitter, 2min reset. 30s cap is unchanged.
  • Added full jitter — delays now land in [level, 2*level),
  • Added a 2-minute idle reset, so a recovered client stops paying for old failures. New lastBackoffAt field.
  • Added nowFn/randFn injection for deterministic tests; updated 3 existing tests to 800/1600/3200, added 2 for jitter spread and cap-then-reset

followup on #3022

ai disclosure: used opencode + claude

The reconnect delay doubled from a 1s base with no jitter, so every
client disconnected by the same event (an apiserver rollout, a load
balancer cycling) retried at the same instants and kept colliding.

Adopt client-go's defaultBackoff* values from the reflector's
ExponentialBackoffManager: an 800ms base, a factor of 2, a 30s cap, full
jitter, and a reset once the backoff has gone unused for two minutes.
Full jitter spreads each retry over [level, 2*level), which is what
breaks up the lockstep, and the idle reset means a client that recovered
stops paying for failures it has already outlived.

Only the delay computation changes. When a reconnect backs off at all is
untouched: the first reconnect is still free, and receiving a watch
event or a TimeoutError still clears the delay.

nowFn and randFn join delayFn as injectable options so the backoff can
be asserted deterministically instead of by sleeping.
@kubernetes-prow

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: davidgamero

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubernetes-prow kubernetes-prow Bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 30, 2026
@davidgamero
davidgamero marked this pull request as draft August 30, 2026 02:42
@kubernetes-prow kubernetes-prow Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 30, 2026
@davidgamero

Copy link
Copy Markdown
Contributor Author

/hold waiting for CI fix

@kubernetes-prow kubernetes-prow Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Aug 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant