fix(connectors): draw the retry delay inside the cap instead of clamping - #4129
Merged
hubcio merged 2 commits intoSep 12, 2026
Merged
Conversation
ryankert01
force-pushed
the
feat/connectors-jitter-distribution
branch
from
September 11, 2026 09:10
9860ab3 to
30feb7c
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4129 +/- ##
=============================================
- Coverage 86.64% 67.41% -19.23%
Complexity 1455 1455
=============================================
Files 1267 1267
Lines 211572 175176 -36396
Branches 176797 140377 -36420
=============================================
- Hits 183310 118090 -65220
- Misses 23778 52517 +28739
- Partials 4484 4569 +85
🚀 New features to boost your workflow:
|
ryankert01
marked this pull request as ready for review
September 11, 2026 12:29
`retry_backoff` jittered first and clamped second, so any draw above `max_delay` landed on the bound itself. Once the exponential saturates, `exponential_backoff` returns exactly `max_delay` and half of every symmetric draw clamps: 247 of 512 saturated draws returned `max_delay` to the nanosecond, which puts the instances that waited longest back in step. The same happens, less often, one step below saturation. At a target of 57s against a 60s cap, 36% of draws clamp onto the bound. The delay is now drawn uniformly from the ±20 % window with the upper edge cut off at `max_delay`. The window narrows as the backoff approaches the cap and sits fully below it once the backoff saturates, so the bound holds with no pile-up and one rule covers every case. That removes the separate `jitter` helper, whose only caller was this function. `jitter` also computed its window as `millis / 5`, which rounds to zero below 5 ms, and `exponential_backoff` rounded the same way, so a sub-millisecond `retry_delay` produced a zero delay and retried at once. Both now compute in nanoseconds. A sub-millisecond delay is reachable from configuration. Two tests cover the cases, and both fail against the previous behavior. Relates to apache#3702 Relates to apache#4084
ryankert01
force-pushed
the
feat/connectors-jitter-distribution
branch
from
September 11, 2026 12:30
30feb7c to
341efe2
Compare
hubcio
approved these changes
Sep 12, 2026
spetz
approved these changes
Sep 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR address?
Relates to #3702
Rationale
Follow-up 1 from #4104, kept out of that PR to stay reviewable.
What changed?
jittercomputed its ±20 % window in whole milliseconds, so below 5 ms thewindow rounded to zero and every instance waited the identical delay. It also
drew first and clamped to
max_delayafter, so any draw above the cap landedon the cap exactly: roughly half of all draws once the backoff saturates, and
36 % at a 57 s target against a 60 s cap. Both put retrying instances back in
step, which is what jitter exists to prevent.
retry_backoffnow draws uniformly, in nanoseconds, from a window already cutoff at
max_delay, instead of drawing then clamping. That leavesjitterwith no caller, so it goes.
exponential_backoffmoves to nanoseconds for thesame rounding reason; its other caller (
nack_retry_delay) passes wholemilliseconds and is unaffected.
#4104 narrowed
jittertopub(crate)rather than deleting it. The near-capcase only showed up under measurement afterwards, which is what collapsed both
branches into a single draw.
Local Execution
fmt,clippy -D warnings, 165 SDK tests,cargo docunderRUSTDOCFLAGS=-D warnings. Both new tests confirmed failing against feat(connectors)!: add shared retry_async helper to the connector SDK #4104'sbehaviour first.
prekran and passes, exceptmarkdownlint,license-headers,trailing-whitespace,trailing-newlineandbinary-artifacts, which needbash >= 4.2; ran those directly instead.
AI Usage