ci(engine): let the ADR 17 sluice metadata tests fail the build - #67
Merged
Conversation
engine/tests/test_sluice_request_metadata.py (92 tests) covers the ADR 17
{app, agent} request-metadata contract, including the case where the async
ai.core stack is rerouted to the gateway by OPENAI_BASE_URL / ANTHROPIC_BASE_URL
alone. It lives under tests/, so the only step that ran it was "Run tests with
coverage" — which carries continue-on-error for the 75 pre-existing failures in
that tree, and therefore masked this suite too.
Extend the existing ratchet pattern rather than dropping continue-on-error
wholesale: a sibling enforcing step re-runs the suite (~1 min) so it gates for
real, matching what "Enforce quality_engine tests" already does.
The suite is green at 92 passed. Verified it gates rather than merely runs:
reverting the guard to inspect config only (dropping the SDK env-var arm) fails
10 tests and exits 1.
`if: !cancelled()` so a failure in the quality_engine ratchet reports alongside
this one instead of short-circuiting it.
Refs baton 0525aefd-6903-4da1-9d3c-7541837e2d45
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Promotes
tests/test_sluice_request_metadata.pyout of the maskedcontinue-on-errortest-with-coverage step and into its own enforced step, same pattern already used forquality_engine.Why: the suite guards the Sluice request-metadata contract (ADR 10, raised to MUST by ADR 17) — every Sluice-bound request must carry
{app, agent}, agent stays inside the closed label set, and no provider stack reaches the gateway untagged. Right now it's collected under the broadertests/step, which has 75 pre-existing failures/56 errors and runs withcontinue-on-error: true— so a regression here is masked along with everything else. The asyncai.corestack in particular never passesbase_urlexplicitly, soOPENAI_BASE_URL/ANTHROPIC_BASE_URLalone can reroute it to Sluice with no metadata block and nothing visible in a diff. Sluice's ADR 17 is heading for stage-3 enforcement (untagged requests get HTTP 400), so this becomes a production failure instead of a lint if it regresses silently here first.Verified before opening this PR:
tests/test_sluice_request_metadata.pyexists on current master (path resolves correctly given the job'sworking-directory: enginedefault).Single-file CI diff, no application code changes.