Skip to content

feat(ci): endpoint-audit gate (client vs 1.29.0 server spec + test coverage)#216

Open
VascoSch92 wants to merge 8 commits into
vasco/bump-agent-server-1.29.0from
vasco/endpoint-audit
Open

feat(ci): endpoint-audit gate (client vs 1.29.0 server spec + test coverage)#216
VascoSch92 wants to merge 8 commits into
vasco/bump-agent-server-1.29.0from
vasco/endpoint-audit

Conversation

@VascoSch92

Copy link
Copy Markdown
Member

Stacked on #214 (agent-server image → 1.29.0).

What

Adds an endpoint-audit gate that reconciles the client's HTTP surface against the agent-server OpenAPI spec and against test coverage, reporting four classes of drift:

  1. mismatch — client calls an endpoint the server no longer exposes (gating)
  2. missing API — server exposes an endpoint the client doesn't implement
  3. missing unit test — client endpoint never hit by a unit test
  4. missing integration test — client endpoint never hit by an integration test

How

  • Ground truth = the server's own /openapi.json, fetched live from the running container (committed specs/agent-server.openapi.json for 1.29.0 is the offline fallback). So the audited surface tracks the server version automatically.
  • Coverage is runtime-accurate: HttpClient appends ${METHOD} ${pathname} to $OH_AUDIT_OUT when set (test/CI only — no-op and zero cost in prod). No grep heuristics.
  • Runs in release.yml (release gate) and integration-tests.yml (every PR/push, against the live 1.29.0 server).

Gating (ratchet)

Today's 3 known drifts against 1.29.0 are baselined in endpoint-audit.config.json so the gate is green now and fails only on new drift:

  • POST /api/cloud-proxy — dropped server-side
  • POST /api/conversations/{id}/generate_title — dropped server-side
  • POST /api/skills/installed/{name}/update — server now exposes …/refresh

(The ACP /api/acp/* drift is already resolved by #214.) Coverage gaps are report-only by default (gate.missingUnit/Integ: false) — flip to true to enforce later.

Try it

npm run audit:endpoints   # unit coverage + report (uses committed spec offline)

Files

  • scripts/endpoint-audit.mjs — the analyzer
  • endpoint-audit.config.json — specs, external-backend prefixes, allowlists, gate flags
  • specs/agent-server.openapi.json — 1.29.0 spec (offline fallback)
  • src/client/http-client.ts — runtime recorder hook
  • package.jsonaudit:unit / audit:integration / audit:report / audit:endpoints
  • .github/workflows/{release,integration-tests}.yml — wiring

Adds scripts/endpoint-audit.mjs which diffs the client's HTTP surface against
the agent-server OpenAPI spec (live from the running container, or the committed
specs/agent-server.openapi.json fallback) and reports:
  - mismatch:    client calls an endpoint the server does not expose (gated)
  - missing API: server exposes an endpoint the client does not implement

Runs as a release gate in release.yml (boots the 1.29.0 image for the live
spec). Today's 3 known drifts (cloud-proxy, generate_title, skills /update ->
/refresh) are baselined in endpoint-audit.config.json so the gate fails only on
new drift. Run locally with: npm run audit:endpoints
@VascoSch92 VascoSch92 force-pushed the vasco/endpoint-audit branch from a63b449 to a47c7e7 Compare June 19, 2026 15:19
Run the endpoint audit as a standalone Endpoint Audit workflow on push
and pull_request (main/develop) instead of as a gate inside the release
job. Releases no longer depend on the audit.
Drop the pull_request branch filter so the audit runs on every PR
regardless of target branch, and push only on main (no develop branch).
Define the agent-server image once in package.json (config.agentServerImage)
and have the endpoint-audit and integration-tests workflows resolve it at
runtime into $GITHUB_ENV. Bumping the pinned version is now a single edit.

Update README.md and AGENTS.md to point at package.json as the canonical pin.
- Restore release.yml to its original content (no endpoint-audit gate); the
  audit now runs only as its own workflow.
- upload-artifact@v4 skips dot-directories unless include-hidden-files is set,
  so the .audit/ report was never uploaded ("No files were found"). Enable
  include-hidden-files so the report uploads.
Add a github-script step that renders .audit/endpoint-audit.json into a
single self-updating PR comment (matched by a hidden marker). Runs on
pull_request events even when the audit gate fails, so reviewers see the
mismatches. Grants pull-requests: write for commenting.
@github-actions

Copy link
Copy Markdown

Endpoint audit

✅ No mismatches — server endpoints: 104, client endpoints: 104

❌ Mismatch — client calls an endpoint the server does not expose (0)

none

➕ Missing API — server exposes it, client does not implement (17)

  • GET /
  • GET /api/bash/bash_events
  • GET /api/conversations
  • GET /api/conversations/{}/events
  • GET /api/conversations/{}/workspace
  • GET /api/conversations/{}/workspace/{}
  • GET /api/init
  • GET /api/llm/subscription/openai/models
  • GET /api/llm/subscription/openai/status
  • POST /api/conversations/{}/goal
  • POST /api/conversations/{}/goal/resume
  • POST /api/conversations/{}/goal/stop
  • POST /api/init
  • POST /api/llm/subscription/openai/device/poll
  • POST /api/llm/subscription/openai/device/start
  • POST /api/llm/subscription/openai/logout
  • POST /api/skills/installed/{}/refresh
ℹ️ External backend, not gated (16)
  • DELETE /api/keys/{}
  • DELETE /api/meta-profiles/{}
  • GET /api/keys
  • GET /api/meta-profiles
  • GET /api/meta-profiles/{}
  • GET /api/security/export-trace
  • GET /api/security/policy
  • GET /api/security/settings
  • GET /api/shared-events/search
  • POST /api/accept_tos
  • POST /api/keys
  • POST /api/meta-profiles/{}
  • POST /api/meta-profiles/{}/activate
  • POST /api/security/policy
  • POST /api/security/settings
  • POST /api/unset-provider-tokens

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant