Skip to content

fix(api): retry 529 overloaded and gateway-flake 400 responses - #3381

Merged
Yeachan-Heo merged 3 commits into
ultraworkers:mainfrom
hakarrr:fix-retry-529-gateway-400
Nov 9, 2026
Merged

Yeachan-Heo merged 3 commits into
ultraworkers:mainfrom
hakarrr:fix-retry-529-gateway-400

Conversation

@hakarrr

@hakarrr hakarrr commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Retry classification in both API providers missed transient statuses, so a single provider-side blip failed the request immediately instead of backing off:
    • is_retryable_status matched 408/409/429/500/502/503/504 but not 529 (Anthropic's documented overloaded_error status, also emitted by several OpenAI-compatible gateways).
    • is_retryable_400 (gateway-flake HTTP 400 bodies: "no parseable body", "connection reset", "broken pipe", "empty reply from server") was defined with a doc comment saying these should be retried, but was dead code - never called by expect_success in either provider.
  • expect_success in anthropic.rs and openai_compat.rs now computes retryable = is_retryable_status(status) || is_retryable_400(status, &body), and 529 is retryable in both matchers. Real 400s (e.g. invalid request bodies) remain non-retryable.

Anti-slop triage

  • Classification: actionable-fix
  • Evidence: failing tests on unpatched main, then green after the fix:
    • With new tests applied to unpatched sources: cargo test -p api --lib providers:: -> 2 failures (retryable_statuses_are_detected in both providers, on the 529 assertion).
    • With only the 529 matcher fix applied: 2 failures remain (expect_success_marks_gateway_flake_400_retryable in both providers), proving the dead-code wiring gap through the real expect_success path.
    • With the full fix: cargo test -p api --lib -> 180 passed, 0 failed.
  • Non-destructive review result: merge candidate

Verification

  • Targeted tests/docs checks ran, or the gap is explicitly recorded. (cargo test -p api --lib: 180 passed incl. 6 new tests; cargo clippy -p api --lib: clean; cargo fmt --check -p api: clean. Full-workspace cargo test --workspace not run locally - change is confined to the api crate's retry classification.)
  • git diff --check passes.
  • No live secrets, tokens, private logs, or unrelated generated churn are included.

Resolution gate

  • If this PR resolves an issue, the issue number and fix evidence are linked. (No tracking issue found - behavior-anchored fix per CONTRIBUTING "tied to a concrete issue or behavior"; evidence above.)
  • If this PR should not merge, the rejection/defer rationale is evidence-backed and does not rely on vibes.
  • I did not merge/close remote PRs or issues from an automation lane without owner approval.

hakarrr and others added 3 commits November 9, 2026 06:42
Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>
Co-authored-by: generaltony007 <144252170+GeneralTony007@users.noreply.github.com>

@Yeachan-Heo Yeachan-Heo left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 758960d. The 400 retry rule is limited to the four known gateway-flake body phrases; ordinary invalid-request 400s remain non-retryable. HTTP 529 is added to the retryable statuses without changing retry limits or backoff. Both provider test suites exercise expect_success directly, and local formatting, clippy, and workspace tests pass.
—
[repo owner's gaebal-gajae (clawdbot) 🦞]

@Yeachan-Heo
Yeachan-Heo merged commit 333298a into ultraworkers:main Nov 9, 2026
6 checks passed
@Yeachan-Heo Yeachan-Heo mentioned this pull request Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants