feat(config): add durable apply transactions

.audit/stage-3-logical-transaction.tsv

@@ -0,0 +1,21 @@
+ts	phase	decision	why	evidence	result
+2026-09-11T04:06:40Z	framing	based the implementation branch on the exact Stage 2 PR head	the stack must contain only Stage 3 changes	PR #3265 head 797c36ebafad2e8b7a4eb59b971e4d27aff6745f	branch 1731-durable-config-operations/pimlock created
+2026-09-11T04:43:47Z	design	used an exact policy-version and settings-revision tuple for each operation	a result must not complete a different desired state after concurrent mutations	crates/openshell-server/src/config_update_operation.rs	exact tuple comparison implemented
+2026-09-11T04:43:47Z	durability	stored desired state and its operation in one database transaction	a crash after commit must leave enough durable state to redispatch	crates/openshell-server/src/persistence/postgres.rs, crates/openshell-server/src/persistence/sqlite.rs	atomic policy and settings paths implemented
+2026-09-11T04:43:47Z	migration	removed runtime polling and required supervisor protocol revision 3	all supported peers now consume complete stream bootstrap and updates	crates/openshell-sandbox/src/lib.rs, crates/openshell-core/src/proto/mod.rs	polling fallback removed and old revisions rejected
+2026-09-11T04:43:47Z	api	made wait consistency server-owned and queryable by operation ID	client deadlines must not determine or erase the durable business outcome	proto/openshell.proto, crates/openshell-cli/src/run.rs	CLI and SDK requests use WAIT_FOR_APPLY
+2026-09-11T22:01:00Z	atomicity	included setting annotations and expected sandbox resource-version CAS in the settings transaction	a post-commit annotation failure could otherwise report failure after desired state and its operation committed	crates/openshell-server/src/persistence/postgres.rs, crates/openshell-server/src/persistence/sqlite.rs	settings, sandbox projection, and operation now commit or roll back together
+2026-09-11T22:04:00Z	verification	validated curated TypeScript and Go SDK contracts	public clients must compile and exercise the new consistency and operation fields	mise run sdk:ts:ci; mise run go:ci	both SDK CI suites passed
+2026-09-11T22:05:00Z	design-artifact	published the state and flow design as a secret gist	the implementation and operational contract need a durable reviewable reference	https://gist.github.com/pimlock/73aa746b48d82d84001cf8e0afa56bf6	gist published
+2026-09-11T22:24:00Z	migration	removed the remaining Stage 2 compatibility branches and stale polling language	protocol revision 3 requires bootstrap and must expose one stream-only configuration path	crates/openshell-server/src/supervisor_session.rs, crates/openshell-sandbox/src/lib.rs, docs/	dead optional-bootstrap and fetch-fallback paths removed
+2026-09-11T22:25:00Z	verification	reran the focused custom-image end-to-end suite after one transient assertion failure	the first run logged the correct runtime rejection but the client observed terminal state before the detailed condition	OPENSHELL_E2E_DOCKER_TEST=custom_image mise run e2e:rust	all 3 focused tests passed without changing the assertion
+2026-09-11T22:26:00Z	self-audit	checked for an available session transcript before final review	the decision-trail protocol calls for comparison when a transcript artifact exists	find /home/pmlocek/.codex -type f -path '*/agent-transcripts/*'	no transcript artifact was available; reviewed the trail against the active request and implementation diff
+2026-09-11T22:32:00Z	verification	isolated a full-CI timeout in an unrelated WebSocket compression test	the test exercises supervisor-network frame handling and has no configuration-operation dependency	mise run ci; cargo test -p openshell-supervisor-network --lib proxy::tests::plaintext_websocket_middleware_inspects_compressed_ws_messages -- --exact --nocapture	full run timed out once; isolated rerun passed 1/1 in 8.44 seconds
+2026-09-11T22:34:00Z	compatibility	made protocol revision 3 an exact-match release boundary before removing polling	Stage 2 already supplies complete bootstrap, live stream payloads, acknowledgements, and owner reconciliation; exact matching excludes old peers instead of silently degrading	PR #3265; proto/sandbox.proto; crates/openshell-server/src/supervisor_session.rs; focused Docker E2E with stream-only runtime	current gateway and supervisor interoperate; older revisions are rejected and require coordinated upgrade
+2026-09-11T22:34:00Z	lifecycle	classified phases from authoritative persisted state rather than session presence	Ready and transitional phases may have or regain a live runtime, while Stopped and Completed do not; Deleting invalidates further application	crates/openshell-server/src/config_update_operation.rs authoritative_phase_classification_is_explicit	Stopped and Completed become inactive; Deleting becomes cancelled; Ready, Provisioning, Starting, Stopping, Error, Unknown, and Unspecified stay pending; the current SandboxPhase enum has no Suspended variant
+2026-09-11T22:34:00Z	completion	mapped exact correlated results without treating newer application as proof of older application	a degraded result means the requested snapshot was installed with declared degraded middleware behavior, while a retained local override means the requested gateway policy did not become effective	crates/openshell-server/src/config_update_operation.rs complete_from_apply_result and target_relation	Applied, duplicate, and degraded complete applied; stale completes superseded; retained override and apply failures complete failed; later desired tuples supersede pending older tuples
+2026-09-11T22:34:00Z	recovery	used durable scans, CAS transition claims, and bounded exponential redispatch backoff	request memory and request-gateway ownership cannot be correctness dependencies, and concurrent reconcilers must not create unbounded duplicate work	crates/openshell-server/src/config_update_operation.rs spawn_reconciler, mutate_record, reconcile_one	pending records survive restart, any gateway sharing the store may claim them, and retries rebuild the current complete snapshot
+2026-09-11T22:34:00Z	api-scope	scoped idempotency to sandbox and key, omitted operations for semantic no-ops, bounded server waits, and kept global mutations commit-only	retries need stable identity without creating artificial revisions; unbounded fleet fan-out needs a separate operation model	proto/openshell.proto; crates/openshell-server/src/grpc/policy.rs; crates/openshell-server/src/config_update_operation.rs	retries resolve the stored operation, no-op calls return without an operation, WAIT_FOR_APPLY is rejected globally, and waits default to 60 seconds with a one-hour cap
+2026-09-11T22:34:00Z	verification-scope	mapped Stage 3 recovery requirements to implemented tests and durable invariants	the review trail must distinguish executable coverage from properties established by shared persistence and CAS	server operation tests, stopped_sandbox_setting_update_commits_with_inactive_operation, supervisor stream tests, SDK CI, Docker E2E	inactive, exact tuple ordering, idempotent retry, atomic persistence, stream bootstrap/apply, and polling removal are directly tested; process-kill crash injection and multi-process cross-gateway completion are not dedicated scenarios and remain coverage follow-ups
+2026-09-11T22:34:00Z	independent-review	addressed the cross-model decision-trail audit	the reviewer identified omitted lifecycle, compatibility, recovery, API-scope, completion, and verification rationale	gpt-6-astra review of .audit/stage-3-logical-transaction.tsv	six missing rationale areas added; no code defect was reported by this trail-only audit
+2026-09-11T22:37:00Z	verification	reran the repository full CI gate after isolating the timing failure	a green complete rerun removes ambiguity about the final tree	mise run ci	full CI passed, including 1492 server tests and all configured SDK, lint, policy, packaging, and documentation checks

architecture/compute-runtimes.md

@@ -433,7 +433,7 @@ process-supervision leaf and launches the user workload after the sidecar
 serves bootstrap state over a local control socket. The network sidecar owns
 gateway credentials and sends policy plus workload-facing provider environment
 state to the process leaf over that socket. It also streams provider
-environment updates after settings polls so future process sessions see
+environment updates from the configuration stream so future process sessions see
 updated provider env without giving the process leaf gateway access. The
 pre-workload process supervisor is the only accepted control client: the
 network sidecar verifies its UID, GID, and PID with peer credentials, removes

architecture/gateway.md

@@ -425,8 +425,12 @@ record; sandbox metadata receives the same annotations only as a convenience
 projection and can retain keys from earlier revisions. Policy revision creation,
 optional first-policy backfill, metadata projection, and superseding older
 revisions commit in one database transaction. SQLite serializes this operation
-with an immediate transaction, while Postgres locks the sandbox row. A failed
-resource-version check or revision insert rolls back the entire operation.
+with an immediate transaction. Postgres first locks a dedicated configuration
+fence keyed by sandbox ID, then locks the sandbox row. Settings mutations take
+the same fence before reading the current policy target. This makes concurrent
+policy and settings commits select targets in one database-owned serial order
+across gateway replicas. A failed resource-version check, desired-state write,
+projection, or operation insert rolls back the entire transaction.
 
 SQLite is the default local store; Postgres is supported for deployments that
 need an external database or multi-replica coordination. Both backends expose
@@ -617,12 +621,13 @@ interleave a profile mutation with a sandbox provider-set mutation that would
 leave an ambiguous final dynamic-token state or a deleted custom profile that is
 still referenced by a sandbox.
 
-Policy and runtime settings are delivered together through the effective sandbox
-config path. A gateway-global policy can override sandbox-scoped policy. The
-gateway pushes complete snapshots to active supervisor sessions and periodically
-rebuilds them to repair missed delivery. Supervisors hot-reload accepted policy
-and acknowledge the exact revision. The legacy poller remains as a mixed-version
-compatibility path during this stage.
+Policy and runtime settings are delivered together through the supervisor
+configuration stream. A gateway-global policy can override sandbox-scoped
+policy. The gateway pushes complete snapshots to active supervisor sessions and
+owner reconciliation rebuilds them to repair missed delivery. Supervisors
+hot-reload accepted policy and acknowledge the exact revision. Gateway and
+supervisor protocol revisions must match; there is no configuration polling
+compatibility path.
 
 External supervisor middleware registration is operator-owned configuration
 under `[[openshell.supervisor.middleware]]`. At startup the gateway connects to
@@ -698,13 +703,36 @@ mutation handlers.
 
 Current supervisors establish the stream before gateway-owned runtime
 initialization, apply bootstrap and live snapshots directly, and persist only
-compact component observations from their results. Previous-revision
-supervisors retain polling as a rollout fallback, and owner reconciliation
+compact component observations from their results. The protocol is
+release-matched and no polling compatibility path remains. Owner reconciliation
 repairs missed or failed delivery from current database state. Snapshot build,
 fanout, or enqueue failure cannot fail a mutation that already committed.
 Provider snapshots may contain credentials and must not be persisted or
 included in logs.
 
+For sandbox-scoped policy and settings mutations, the gateway atomically stores
+the desired state and a durable operation whose target is the exact policy and
+settings revision tuple. Server-side `WAIT_FOR_APPLY` reads this durable record
+until a correlated stream result or authoritative lifecycle transition makes it
+terminal. Pending-operation reconciliation provides crash recovery and can run
+on a gateway other than the request handler; local notifications are wake-up
+hints only. Operations persist revisions, outcome, timestamps, response
+metadata, and bounded sanitized errors, never complete configuration payloads
+or credentials. The SQL status column changes atomically with the encoded
+operation. Result correlation queries only pending operations scoped to the
+reporting sandbox. Recovery claims bounded due batches, commits each retry
+deadline before snapshot construction, groups work by sandbox, and publishes
+each component at most once per sandbox pass.
+
+Operation records and their idempotency keys currently have no automatic
+expiration. The gateway retains both until an explicit deletion contract is
+defined, so an idempotency key cannot be reused merely because time passed.
+On startup, the gateway decodes every existing operation and repairs its SQL
+scope, state, and retry-time projection before selective reconciliation starts.
+The repair is restart-safe and does not change operation resource versions.
+Gateways that share a database must be upgraded together while this projection
+is introduced. An older gateway does not maintain these query columns.
+
 See [sandbox configuration delivery](sandbox.md#supervisor-configuration-delivery)
 for bootstrap, revision, and supervisor application semantics.
 

architecture/sandbox.md

@@ -478,19 +478,14 @@ quickly.
 
 ## Supervisor Configuration Delivery
 
-The current gateway and supervisor use internal supervisor protocol revision 2.
-The gateway accepts Stage 1 revision 1 supervisors through the polling
-compatibility path. Peers built before the handshake report revision zero and
-remain accepted for one release with a warning and counter because sandboxes
-keep their supervisor binary until they are recreated. The gateway includes a
+The current gateway and supervisor use internal supervisor protocol revision 3.
+This revision is release-matched: both peers reject every other revision. The gateway includes a
 configuration bootstrap when it accepts a `ConnectSupervisor` session and can
 send complete component replacements on the same stream after policy, settings,
 or provider state changes.
-Revision 2 supervisors require a complete bootstrap. The gateway uses the same
+Supervisors require a complete bootstrap. The gateway uses the same
 bounded 45-second construction window as other snapshot builds and rejects the
-connection when construction fails. Revision 1 compatibility sessions retain
-the optional one-second bootstrap budget and use polling when it expires.
-The revision 2 supervisor opens the stream and consumes the bootstrap before it
+connection when construction fails. The supervisor opens the stream and consumes the bootstrap before it
 constructs gateway-owned policy, provider state, networking, or the workload.
 It reports bootstrap results after those components, the workload, and relay
 endpoints are ready.
@@ -533,8 +528,9 @@ revisions suppress unchanged delivery, while failed or timed-out delivery is
 retried from current database state. Reconnect discards session delivery state
 and starts with a fresh bootstrap.
 
-Revision 2 does not poll configuration fetch APIs. Polling remains available
-only to revision 1 and revision 0 supervisors during the mixed-version rollout.
+Supervisors do not poll configuration fetch APIs. Provider credentials arrive
+only through the encrypted bootstrap or live stream, and the obsolete
+supervisor-only provider-environment fetch RPC no longer exists.
 The gateway serializes construction per sandbox and component, and coalesces
 repeated mutations into the latest full snapshot. An enqueue result means only
 that the local stream queue accepted the message. A bounded scope fanout scheduler
@@ -544,8 +540,24 @@ builds. Fanout waits for worker capacity before admitting each recipient, so a
 fleet-wide change cannot create a fleet-sized task backlog or saturate the store
 and credential backends. Snapshot construction has a deadline that starts once
 a build holds a permit, and the gateway rejects encoded stream messages that
-approach the transport decoder limit. Durable apply operations and final polling
-removal remain separate follow-up work.
+approach the transport decoder limit.
+
+Sandbox-scoped `UpdateConfig` commits the desired policy or settings record and
+a non-secret update operation in one database transaction. The operation target
+is the exact `(policy_version, settings_revision)` tuple. `COMMIT_ONLY` returns
+after that commit; `WAIT_FOR_APPLY` durably waits for `applied`, `inactive`,
+`failed`, `superseded`, or `cancelled`. A client timeout does not roll back the
+mutation or cancel the operation, and the operation ID is returned in error
+metadata for later lookup. Stopped or completed sandboxes finish as `inactive`;
+session absence alone never does. A background reconciler reads pending
+operations in bounded due batches. It claims retry ownership, groups work by
+sandbox, and admits each requested component to the existing bounded delivery
+queue at most once for that sandbox. The delivery worker builds the latest
+sandbox snapshot once, records its exact revision on every matching pending
+operation, then sends that same snapshot. It skips the send if revision
+association fails. Local sandbox-state notifications run the same scoped
+reconciliation path immediately. The periodic database query remains the
+recovery path when another gateway owns the waiter or a notification is missed.
 
 ## Policy Revision Acknowledgement
 
@@ -561,8 +573,7 @@ policy structure.
 Image-specific policy discovery and baseline enrichment can require one initial
 gateway synchronization. The supervisor commits that repair before runtime
 initialization, discards the mutation response, and reconnects so it installs
-only the fresh authoritative stream bootstrap. Compatibility supervisors retain
-the earlier enrichment and first-poll reconciliation path.
+only the fresh authoritative stream bootstrap.
 
 A newer sandbox-scoped revision can carry the same non-empty effective policy
 hash as the currently loaded revision, for example when provenance changes
@@ -573,23 +584,22 @@ reconciliation succeeds. Global policies, local overrides, equal or older
 versions, and different hashes do not use this shortcut. Success telemetry is
 emitted only after the gateway accepts the resulting loaded-status report.
 
-Revision 2 policy status is recorded from the correlated stream result. The
-retained reporting RPC uses the same domain helper for compatibility
-supervisors. Retryable legacy status delivery uses a FIFO background worker so
-status endpoint outages do not block enforcement.
+Current policy status is recorded from the correlated stream result. The
+retained reporting RPC exists for startup construction failures and uses the
+same exact-revision domain helper.
 
 Only sandbox-scoped revisions (`PolicySource::Sandbox`, version greater than
 zero) are acknowledged. Global policies and local-file development policies do
 not use the sandbox revision API and produce no acknowledgement. When explicit
-local Rego and data files are configured, the supervisor continues polling the
-gateway for settings and provider refreshes only on the compatibility path; a
-revision 2 supervisor receives those components on the stream and never
-replaces the local OPA engine with a gateway policy revision.
+local Rego and data files are configured, the supervisor still receives
+settings and provider refreshes on the stream but never replaces the local OPA
+engine with a gateway policy revision.
 
 ## Failure Behavior
 
-- If a compatibility configuration poll fails, the sandbox keeps its
-  last-known-good policy.
+- If stream delivery or application fails, owner and operation reconciliation
+  republish the latest complete desired state while the sandbox retains its
+  last-known-good policy according to the configured failure mode.
 - If a live policy or middleware-registry update is invalid, the supervisor
   rejects the combined update and keeps the current runtime pair.
 - If an operator-run middleware call fails, the selected config's `on_error`