Push gateway-owned desired state to supervisors

[pimlock]

Summary

Replace supervisor polling for gateway-owned configuration with bootstrap and live snapshot delivery over the existing reverse ConnectSupervisor session. Add durable completion tracking so callers can return after commit or wait for the exact sandbox configuration revision to reach a terminal state.

The persistent supervisor session and its SSH/exec relay substrate landed in PR #867. Warm-pool registration in PR #2460 remains a pre-session identity and activation step. Managed inference routing was removed in PR #3195 and is no longer part of this work.

Resolved design

• Gateway and supervisor use a release-matched internal protocol revision. Mismatched peers fail clearly; there is no polling fallback or mixed-protocol compatibility mode.
• Gateway-backed supervisors establish ConnectSupervisor before runtime initialization and receive complete sandbox configuration and provider-environment snapshots in SessionAccepted.
• Supervisors apply the bootstrap before starting gateway-owned runtime state. They report the bootstrap result after workload and relay endpoints are ready, and that result gates the gateway initialized state.
• Live changes use complete, one-component ConfigUpdate snapshots with session-scoped correlation and ordering. Delivery is bounded, coalesces newer desired state, and starts from a fresh bootstrap after reconnect.
• Standalone file-backed operation remains independent. Explicit local policy keeps precedence and returns an observable retained-local-override outcome.
• The database remains authoritative. Local notification is the fast path, while owner reconciliation repairs missed delivery and cross-replica writes.
• Sandbox-scoped policy and settings mutations commit desired state and a durable, non-secret operation atomically. COMMIT_ONLY returns after commit. WAIT_FOR_APPLY waits for an exact correlated result or an authoritative terminal lifecycle state. Idempotency keys make retries stable.
• Pending operations survive client timeout and gateway restart. Reconciliation rebuilds current snapshots and republishes them with bounded backoff. No database transaction remains open during delivery or waiting.
• Keep GetSandboxConfig, ReportPolicyStatus, SubmitPolicyAnalysis, PushSandboxLogs, and RelayStream on their existing roles. Remove obsolete supervisor polling and the supervisor-only provider-environment fetch RPC.

Scope

• Define bootstrap, snapshot, update, result, revision, and durable operation contracts.
• Move gateway-backed initialization and live configuration changes to the supervisor stream.
• Add atomic operation persistence, lookup, idempotency, terminal states, bounded waits, and crash-safe reconciliation for sandbox-scoped policy and settings mutations.
• Preserve local source ownership, last-known-good behavior, heartbeat and relay fairness, bounded queues, and secret-safe logging.
• Update the CLI, SDKs, tests, architecture documentation, operator documentation, and public troubleshooting skill.

Non-goals

• Legacy polling fallback or mixed-protocol operation.
• Persisting configuration snapshots, credentials, or an event-by-event delivery log.
• Moving status, analysis, logs, or relay bytes onto the shared session protocol.
• A gateway-initiated supervisor RPC server, endpoint discovery, inbound auth or mTLS, Kubernetes networking changes, or a generic transport abstraction.
• In-place supervisor replacement or broader multi-replica session ownership.
• Changing supervisor image defaults. Release-matched artifacts remain required and deployment-default cleanup is separate work.

Acceptance criteria

• Gateway-backed supervisors apply a complete bootstrap before the gateway marks them initialized. Missing, failed, oversized, timed-out, or revision-mismatched exchanges fail clearly within a bound.
• Live policy, settings, middleware, and provider changes arrive without supervisor polling. Duplicate, stale, failed, coalesced, timeout, and reconnect cases preserve correct runtime state.
• COMMIT_ONLY and WAIT_FOR_APPLY expose durable outcomes for the exact policy and settings revision tuple without holding database locks across network work.
• Operations resolve as applied, inactive, failed, superseded, or cancelled from correlated results or authoritative lifecycle state. Client cancellation does not erase the operation.
• Writes through another gateway replica eventually reach the session owner through reconciliation.
• Standalone supervisors require no gateway session, and explicit local policy remains authoritative.
• Retained RPCs remain compatible, obsolete supervisor fetch paths are removed, and documentation describes the release boundary, source ownership, reconciliation, and operation lifecycle.

Implementation

• Stage 1: PR feat(supervisor): stage gateway configuration snapshot delivery #3244
• Stage 2: PR feat(supervisor): apply streamed configuration snapshots #3265
• Stage 3: PR feat(config): add durable apply transactions #3273

References

• Canonical implementation plan: issue comment
• Stage 3 transaction and state model: design gist
• Session and relay foundation: PR feat(server,sandbox): supervisor-initiated SSH connect and exec over gRPC-multiplexed relay #867
• Managed inference removal: PR refactor(inference): remove managed inference routes #3195
• Supervisor-session token expiry: feat(supervisor): enforce supervisor session token expiry with proactive reconnect #1954

[pimlock]

🏗️ build-plan

Implementation plan

Issue type: feat
Complexity: High
Confidence: High. The staged implementation defines the contract and verification boundaries.

Goal

Replace supervisor polling with release-matched stream bootstrap and live configuration delivery. Add durable completion semantics for sandbox-scoped policy and settings mutations.

Plan

1. Establish ConnectSupervisor before gateway-backed runtime initialization. Send complete sandbox configuration and provider-environment snapshots, and require the supervisor to report bootstrap completion after runtime endpoints are ready.
2. Require an exact internal protocol revision between gateway and supervisor. Reject mismatched peers instead of falling back to polling.
3. Deliver later changes as complete, one-component snapshots with correlated IDs and per-component ordering. Keep delivery bounded, coalesce newer state, and bootstrap again after reconnect.
4. Preserve standalone file-backed startup and explicit local policy ownership. Keep last-known-good runtime state when a live update fails according to the configured failure mode.
5. Keep the database authoritative. Use immediate local publication for latency and periodic owner reconciliation for missed notifications and cross-replica writes.
6. Commit each sandbox-scoped policy or settings mutation with a durable, non-secret operation in one database transaction. Support commit-only responses, server-side wait-for-apply, idempotent retries, later lookup, and terminal applied, inactive, failed, superseded, or cancelled states.
7. Reconcile pending operations after crashes or delivery failures by rebuilding and republishing current snapshots with bounded backoff. Never hold database locks while delivering or waiting.
8. Remove supervisor configuration polling and the supervisor-only provider-environment fetch RPC. Keep public configuration reads and supervisor-triggered status, analysis, log, and relay RPCs separate.
9. Update the CLI, Go and TypeScript SDKs, tests, architecture and operator documentation, and the public troubleshooting skill.

Pull request sequence

• PR feat(supervisor): stage gateway configuration snapshot delivery #3244 stages snapshot delivery while polling remains authoritative.
• PR feat(supervisor): apply streamed configuration snapshots #3265 applies bootstrap and live streamed snapshots and records correlated results.
• PR feat(config): add durable apply transactions #3273 adds durable completion semantics and removes polling compatibility.

Managed inference routing was removed in PR #3195, so inference snapshots are no longer part of this issue. Supervisor image default cleanup also remains separate.

Verification

• Bootstrap ordering, protocol mismatch, bounded failure, reconnect, and standalone startup.
• Exact result correlation, duplicate and stale inputs, local override, last-known-good behavior, ordering, coalescing, queue bounds, and heartbeat and relay fairness.
• Atomic desired-state and operation persistence, idempotent retries, terminal lifecycle states, timeouts, supersession, restart recovery, and cross-gateway reconciliation.
• Public API compatibility, SDK behavior, removed polling paths, secret-safe persistence and logging, documentation, and Docker-backed end-to-end coverage.

Run mise run pre-commit, mise run ci, SDK CI, and the relevant sandbox end-to-end lanes for each stage.

Out of scope

• Mixed-protocol fallback or a legacy polling path.
• Persisted snapshot payloads, credentials, or an event-by-event delivery log.
• Moving status, analysis, logs, or relay bytes onto ConnectSupervisor.
• Direct supervisor RPC transport, discovery, inbound auth or mTLS, Kubernetes networking, transport abstraction, in-place supervisor replacement, or broader session ownership.
• Supervisor image default changes.

---

Revision 6: aligned the canonical plan with the staged implementation, including exact protocol matching, durable apply operations, stream-only delivery, and the removal of managed inference routing.
Revision 5: consolidated the preparation and protocol work into draft PR #2967; supervisor-image cleanup remained separate.
Revision 4: selected ConfigUpdate, fingerprint-plus-sequence ordering, explicit component outcomes, bounded failure semantics, and a two-PR implementation split.