Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions architecture/gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -342,8 +342,8 @@ public descriptor set generated by `openshell-core`; a fingerprint test in
Compute-driver, credential-driver, gateway-interceptor, and
supervisor-middleware services are compiled contracts for internal extension
boundaries, not public gateway RPCs. The current public inventory has 74
methods, 278 messages, and 12 enums
(`8ac68c71d93e6a5e56406b8df1882ee40c6066270969e03eb99803f0e6396fc1`).
methods, 291 messages, and 15 enums
(`3ba470c3278d0c7a49b7dde7ee735017604afab5054fb7a95665c71a602121f1`).
The removed `NetworkBinary.harness` field remains reserved by number and name,
so protobuf implementations cannot reuse its wire slot or source identifier.
The durable-policy compatibility decoder reads the former boolean before Prost
Expand Down Expand Up @@ -673,6 +673,26 @@ successful create therefore yields an immediately usable provider; failures roll
back the provider record. Service-account JSON and private keys remain gateway-side
refresh bootstrap material; sandboxes receive minted access tokens instead.

## Supervisor configuration routing

Committed configuration mutations publish component and scope identifiers,
never configuration payloads, to a bounded coalescing scheduler. The scheduler
admits a fixed number of delivery workers and builds the latest full snapshot
for each affected active sandbox. Fleet fanout waits for worker capacity before
admitting another recipient. An async router owns session lookup, message
sizing, sequence allocation, and enqueue. Its local implementation uses the
process-local supervisor registry. A future HA implementation can resolve the
gateway that owns a session and forward the same typed message without changing
mutation handlers.

Polling remains authoritative during the first rollout stage. Snapshot build,
fanout, or enqueue failure cannot fail a mutation that already committed.
Provider snapshots may contain credentials and must not be
persisted or included in logs.

See [sandbox configuration delivery](sandbox.md#supervisor-configuration-delivery)
for bootstrap, revision, and supervisor application semantics.

## Supervisor Relay

Sandbox workloads maintain an outbound supervisor session to the gateway. This
Expand Down
46 changes: 46 additions & 0 deletions architecture/sandbox.md
Original file line number Diff line number Diff line change
Expand Up @@ -473,6 +473,52 @@ the structured 403 and authors the narrowest rule. Mechanistically mapping L7
would either over-broaden rules or require path-templating logic that rots
quickly.

## Supervisor Configuration Delivery

The gateway and supervisor must implement the same internal supervisor protocol
revision. Peers built before the handshake existed report revision zero and are
accepted for one release with a warning and a counter, because sandboxes keep
their supervisor binary until they are recreated. The gateway includes a configuration bootstrap when it accepts a
`ConnectSupervisor` session and can send complete component replacements on the
same stream after policy, settings, or provider state changes.
While polling remains authoritative, optional bootstrap construction has a
one-second budget. The gateway accepts the session without a bootstrap when
that budget expires, so slow credential backends do not block relay reconnects.
These payloads describe the latest effective state rather than
the mutation that produced it. The gateway assigns ordering sequences within
each session and component, while each snapshot retains its own content
revision.

Bootstrap components are independent read projections, not one atomic database
snapshot. The sandbox configuration carries the provider-environment revision
it was built against. The gateway retries bootstrap construction when that
revision does not match the provider snapshot. Later component updates and
polling repair changes committed while the other projections were being built.

Configuration delivery goes through a gateway-owned routing boundary rather
than exposing local supervisor channels to mutation handlers. The current
implementation routes only to a supervisor connected to the same gateway
process. The asynchronous router contract can resolve a remote owner later
without changing publishers. Provider payloads can contain
credentials, so the gateway does not persist or render complete stream
messages in logs.

The supervisor currently parses and ignores stream-delivered configuration.
Polling remains the only path that changes runtime state and repairs dropped or
unavailable delivery. The gateway serializes construction per sandbox and
component, and coalesces repeated mutations into the latest full snapshot. An
enqueue result means only that the local stream queue accepted the message. A
bounded scope fanout scheduler coalesces repeated workspace and global changes,
and semaphores sized from the database pool bound delivery workers and snapshot
builds. Fanout waits for worker capacity before admitting each recipient, so a
fleet-wide change cannot create a fleet-sized task backlog or saturate the store
and credential backends. Snapshot construction has a deadline that starts once
a build holds a permit, and the gateway rejects encoded stream messages that
approach the transport decoder limit. A later migration will apply these
payloads directly and acknowledge their exact revisions before removing
supervisor polling. At that point, the gateway will require a valid bootstrap
before marking a session ready.

## Policy Revision Acknowledgement

When the supervisor loads a sandbox-scoped policy from the gateway, it retains
Expand Down
15 changes: 15 additions & 0 deletions crates/openshell-core/src/proto/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -95,6 +95,21 @@ pub fn all_workspaces_selector() -> WorkspaceSelector {
}
}

/// Exact protocol revision required between a gateway and its supervisor.
///
/// The supervisor stream is an internal, version-locked deployment contract.
/// Bump this when either peer can no longer honor the previous stream
/// semantics.
pub const SUPERVISOR_PROTOCOL_REVISION: u32 = 1;

/// Revision implied by peers built before the handshake existed. Proto3 leaves
/// the field unset, so such peers report zero.
///
/// Sandboxes keep their supervisor binary until they are recreated, so a
/// gateway upgrade must keep serving them for one release. Remove this
/// allowance once every supported release sends an explicit revision.
pub const LEGACY_SUPERVISOR_PROTOCOL_REVISION: u32 = 0;

#[cfg(test)]
mod tests {
use std::collections::HashMap;
Expand Down
1 change: 1 addition & 0 deletions crates/openshell-server/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ protoc-bin-vendored = { workspace = true }

[dev-dependencies]
base64 = { workspace = true }
tokio = { workspace = true, features = ["test-util"] }
hyper-rustls = { version = "0.27", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "aws-lc-rs"] }
rcgen = { workspace = true }
rsa = { version = "0.9", features = ["pem"] }
Expand Down
55 changes: 55 additions & 0 deletions crates/openshell-server/src/compute/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -989,6 +989,18 @@ impl ComputeRuntime {
}
})?;

if let Err(status) = Box::pin(crate::grpc::policy::initialize_policy_history(
self.store.as_ref(),
&sandbox,
crate::grpc::policy::InitialPolicyHistoryStatus::Pending,
))
.await
{
let _ = self.store.delete(Sandbox::object_type(), &sandbox_id).await;
self.sandbox_index.remove_sandbox(&sandbox_id);
return Err(status);
}

if let Some(token) = sandbox_token
&& let Some(spec) = driver_sandbox.spec.as_mut()
{
Expand Down Expand Up @@ -1026,6 +1038,10 @@ impl ComputeRuntime {
Ok(sandbox)
}
Err(status) if status.code() == Code::AlreadyExists => {
let _ = self
.store
.delete_by_scope(POLICY_OBJECT_TYPE, sandbox.object_id())
.await;
let _ = self
.store
.delete(Sandbox::object_type(), sandbox.object_id())
Expand All @@ -1034,6 +1050,10 @@ impl ComputeRuntime {
Err(Status::already_exists("sandbox already exists"))
}
Err(status) if status.code() == Code::FailedPrecondition => {
let _ = self
.store
.delete_by_scope(POLICY_OBJECT_TYPE, sandbox.object_id())
.await;
let _ = self
.store
.delete(Sandbox::object_type(), sandbox.object_id())
Expand All @@ -1042,6 +1062,10 @@ impl ComputeRuntime {
Err(Status::failed_precondition(status.message().to_string()))
}
Err(err) => {
let _ = self
.store
.delete_by_scope(POLICY_OBJECT_TYPE, sandbox.object_id())
.await;
let _ = self
.store
.delete(Sandbox::object_type(), sandbox.object_id())
Expand Down Expand Up @@ -4938,6 +4962,7 @@ pub async fn new_test_runtime_with_driver(
#[cfg(test)]
mod tests {
use super::*;
use crate::policy_store::PolicyStoreExt;
use futures::stream;
use openshell_core::proto::compute::v1::{
CreateSandboxResponse, DeleteSandboxResponse, GetCapabilitiesResponse, GetSandboxRequest,
Expand Down Expand Up @@ -11240,6 +11265,36 @@ mod tests {
);
}

#[tokio::test]
async fn create_sandbox_persists_initial_policy_revision() {
let runtime = test_runtime(Arc::new(TestDriver::default())).await;
let mut sandbox = sandbox_record(
"sb-initial-policy",
"initial-policy",
SandboxPhase::Provisioning,
);
let policy = openshell_core::proto::SandboxPolicy::default();
sandbox.spec = Some(SandboxSpec {
policy: Some(policy.clone()),
..Default::default()
});

runtime.create_sandbox(sandbox, None, false).await.unwrap();

let revision = runtime
.store
.get_latest_policy("sb-initial-policy")
.await
.unwrap()
.expect("initial policy revision");
assert_eq!(revision.version, 1);
assert_eq!(
revision.policy_hash,
crate::grpc::policy::deterministic_policy_hash(&policy)
);
assert_eq!(revision.status, "pending");
}

#[tokio::test]
async fn created_sandbox_is_immediately_visible_to_label_selectors() {
let runtime = test_runtime(Arc::new(TestDriver::default())).await;
Expand Down
Loading
Loading