feat: trust additional destination CAs for sandbox egress TLS

[jhjaggars]

Problem Statement

Operators need sandboxes to make egress TLS connections to hosts that present certificates signed by a private or self-signed CA (an internal GitLab, artifact registry, or internal API). Today the sandbox's network supervisor trusts only the bundled public roots plus whatever the sandbox image happens to ship, so the handshake to such a host fails after the policy has already allowed the connection. Operators need a way to say "additionally trust these CA certificates for sandbox destination TLS" without replacing the default public roots and without changing the trust the supervisor uses to authenticate the gateway control plane.

Draft PR #3292 implements one candidate design (a global [openshell.supervisor.network].additional_ca_cert_paths gateway setting delivered by every first-party compute driver). This issue records the problem, the investigation of the current code on main, and the decisions a maintainer must make so that work can be accepted or declined.

User Story

As an operator running OpenShell in an environment with private PKI, I want to configure a set of additional CA certificates once at the gateway so that every sandbox can reach internal HTTPS services, while public sites keep working and the sandbox cannot use those certificates to impersonate the gateway.

Impact / Why This Matters

• A policy-allowed curl https://gitlab.internal fails inside the sandbox with an UnknownIssuer TLS error after the CONNECT succeeds, which is confusing and blocks agents that need internal services.
• The current workaround is to bake the CA into every sandbox image's system bundle. That requires an image rebuild per rotation, is owned by the image author rather than the operator, and does not work in the Kubernetes sidecar topology because the sidecar reads its own image's bundle, not the agent image's.
• The other workaround, tls: skip on the endpoint, disables L7 inspection for that host and still requires the CA in the image because the supervisor overrides SSL_CERT_FILE.
• proxy_ca_bundle exists only for Podman and VM, conflates corporate-proxy trust with destination trust, and refuses to load without an https_proxy URL.

Technical Context

OpenShell keeps three distinct trust stores, and any solution must keep them distinct:

1. Supervisor to gateway mTLS. The gRPC client trusts only the mounted gateway CA and the code explicitly forbids adding native or webpki roots, because the supervisor runs inside a user-selected image. This boundary must not change.
2. Destination TLS (proxy mode). The upstream root store is the webpki-roots bundle overlaid with the system bundle found at one of four fixed paths inside the image the supervisor runs in. In builds without the bundled-ca-roots feature, the native store is used and the system bundle is ignored.
3. Corporate proxy CA. An operator PEM delivered by argv that is appended to the system bundle string, so it reaches both the upstream root store and the child trust files. It is fail-closed and requires an https_proxy pairing.

Every gateway-managed sandbox runs in proxy mode: the supervisor generates an ephemeral CA, terminates TLS from the workload, inspects HTTP, and re-encrypts upstream with the root store above. Child processes get SSL_CERT_FILE, NODE_EXTRA_CA_CERTS, REQUESTS_CA_BUNDLE, and friends pointed at supervisor-written files, unconditionally overriding user values. Block and Allow modes exist only for local file-based policies; in those modes no TLS files or env vars are set.

Affected Components

Component	Key Files	Role
Network supervisor TLS	crates/openshell-supervisor-network/src/l7/tls.rs	Builds the upstream root store and writes the child trust files
Network supervisor wiring	crates/openshell-supervisor-network/src/run.rs	Decides when TLS termination exists and folds the proxy CA in
Corporate proxy module	crates/openshell-supervisor-network/src/upstream_proxy.rs	Closest existing pattern for operator-owned trust material
Shared validators	crates/openshell-core/src/driver_utils.rs	Bounded PEM read, fail-closed validation, mount path constants
Sandbox binary	crates/openshell-sandbox/src/main.rs, src/lib.rs	Argv-only operator flags; passes CA paths to process and SSH
Process supervisor	crates/openshell-supervisor-process/src/child_env.rs, process.rs, ssh.rs	Applies TLS env vars to workload and SSH children
Gateway config	crates/openshell-server/src/config_file.rs, compute/driver_config.rs, lib.rs	[openshell.supervisor] section, driver tables, DriverStartupContext
Driver factories	crates/openshell-gateway/src/lib.rs, src/vm.rs	Inject gateway-owned inputs such as guest TLS into driver configs
Docker / Podman / Kubernetes / VM drivers	crates/openshell-driver-docker/src/lib.rs, crates/openshell-driver-podman/src/container.rs, crates/openshell-driver-kubernetes/src/driver.rs, crates/openshell-driver-vm/src/driver.rs	Stage files into the sandbox and author supervisor argv
Control-plane mTLS	crates/openshell-core/src/grpc_client.rs, src/container_paths.rs	Boundary that must not change
Helm chart	deploy/helm/openshell/	Values, RBAC, and volume rendering for Kubernetes delivery

Technical Investigation

Architecture Overview

Trust material flows gateway → compute driver → supervisor. The gateway reads TOML, merges CLI args, builds a DriverStartupContext (carrying gateway-owned inputs such as GuestTlsPaths), and the selected ComputeDriverFactory turns that into a driver config. Each driver stages files into the sandbox (Docker/Podman bind mounts, Kubernetes volumes, VM overlay files) and writes the supervisor's argv. The supervisor validates its inputs at startup, emits OCSF ConfigStateChange events, and fails closed on present-but-invalid operator material.

In proxy mode, run_networking generates the ephemeral CA, reads the system bundle, appends the corporate proxy CA if configured, builds the upstream ClientConfig, and writes two files: openshell-ca.pem (additive, used by NODE_EXTRA_CA_CERTS and DENO_CERT) and ca-bundle.pem (complete, used by SSL_CERT_FILE, REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, GIT_SSL_CAINFO). The proxy MITMs endpoints with TlsMode::Auto and verifies upstream against the root store; TlsMode::Skip endpoints and all non-proxy traffic are raw tunnels where the workload does its own TLS.

Where the supervisor runs matters for which "system bundle" is consulted: the sandbox image for Docker, Podman, VM, and Kubernetes combined mode, but the supervisor image for Kubernetes sidecar mode.

Code References

Location	Description
crates/openshell-core/src/grpc_client.rs:175-190	Gateway mTLS trusts only the mounted CA; comment forbids broadening. Must not change.
crates/openshell-core/src/container_paths.rs:45-48, 59-60, 77-82	Reserved TLS client dir, child CA file paths, VM guest constants
crates/openshell-supervisor-network/src/l7/tls.rs:26-31	SYSTEM_CA_PATHS probed for the image bundle
crates/openshell-supervisor-network/src/l7/tls.rs:218-249	build_upstream_root_store: webpki plus system overlay; native branch ignores the system bundle
crates/openshell-supervisor-network/src/l7/tls.rs:281-302	write_ca_files: standalone and combined child bundles
crates/openshell-supervisor-network/src/l7/tls.rs:310-327	Lenient PEM loader that silently drops bad blocks
crates/openshell-supervisor-network/src/run.rs:317-402	Proxy-mode-only CA generation, proxy CA fold-in, ca_file_paths is None otherwise
crates/openshell-supervisor-network/src/run.rs:363-379	CA file write failure degrades to "termination disabled" rather than aborting
crates/openshell-supervisor-network/src/upstream_proxy.rs:346-351, 410-446	Proxy CA bundle read, fail-closed rules, https_proxy pairing requirement
crates/openshell-core/src/driver_utils.rs:458, 481-539, 557-604	1 MiB bound, regular-file check, at-least-one-anchor validation
crates/openshell-sandbox/src/main.rs:203-233	Argv-only upstream proxy flags (no env alias)
crates/openshell-supervisor-process/src/child_env.rs:24-39	The six TLS env vars
crates/openshell-supervisor-process/src/process.rs:826-830, 1017	TLS env applied after user env, unconditional override
crates/openshell-server/src/config_file.rs:44-73, 217-224, 236	Sections, deny_unknown_fields, existing "gateway reads a PEM at startup" precedent for middleware TLS CA
crates/openshell-server/src/compute/driver_config.rs:18-115	DriverStartupContext and GuestTlsPaths
crates/openshell-gateway/src/lib.rs:178-230, 511-527	Factory injection of gateway-owned paths (apply_guest_tls)
crates/openshell-driver-docker/src/lib.rs:2785-2830	Bind mounts, always :ro,z
crates/openshell-driver-podman/src/container.rs:29-35, 1334-1345	Conditional SELinux z, proxy CA mount
crates/openshell-driver-podman/src/config.rs:173, crates/openshell-driver-vm/src/driver.rs:270	proxy_ca_bundle exists only for these two drivers
crates/openshell-driver-kubernetes/src/driver.rs:1525, 2709-2713, 3995-4035	Sidecar image selection, shared TLS dir, volumes are Secrets only today
crates/openshell-driver-vm/src/driver.rs:6237, 6304-6372	Protected args file and overlay file staging at 0644
crates/openshell-supervisor-network/src/proxy.rs:3096-3120	Proxy resolves hosts from the workload's /etc/hosts
deploy/helm/openshell/values.yaml:432-435	Existing oidc.caConfigMapName operator-provided ConfigMap pattern

Current Behavior

curl https://gitlab.internal from a gateway-managed sandbox: curl is given HTTPS_PROXY and SSL_CERT_FILE=/etc/openshell-tls/ca-bundle.pem. The CONNECT is policy-evaluated and allowed. The proxy presents an ephemeral leaf that curl trusts, then calls tls_connect_upstream with the root store from build_upstream_root_store. The private CA is in neither webpki nor the image bundle, so rustls fails with UnknownIssuer, the tunnel is torn down, and curl reports a TLS or connection-reset error after a successful CONNECT.

Workarounds today: bake the CA into the image's system bundle (fails in Kubernetes sidecar topology), set tls: skip on the endpoint (loses inspection, still needs the CA in the image because SSL_CERT_FILE is overridden), or on Podman/VM misuse proxy_ca_bundle (requires an https_proxy URL).

In Block/Allow modes there is no termination and no TLS env, so curl uses the image bundle or a user-supplied SSL_CERT_FILE unchanged.

What Would Need to Change

• Network supervisor (tls.rs, run.rs): accept a second PEM source; add its anchors after the built-in or native roots in both feature variants (the native branch currently ignores system_ca_bundle, so the feature would silently no-op there otherwise); include it in both child files; decide whether to write child files in non-proxy modes; abort startup on a present-but-invalid file rather than degrading like the current CA-write failure path.
• Sandbox binary: one new argv-only flag mirroring the upstream proxy flags, threaded through run_sandbox to run_networking.
• Process supervisor: only if precedence changes for direct mode, where OpenShell currently overrides user TLS env unconditionally.
• Gateway: a config field (global [openshell.supervisor.network] or per-driver keys), startup validation (bounded read, only certificate blocks, at least one usable anchor), and a DriverStartupContext field so factories can hand drivers a gateway-owned staged path, following guest_tls. Fail closed when the selected driver cannot propagate the material (remote endpoints, out-of-tree factories).
• Docker/Podman: one read-only bind mount to a reserved guest path plus one argv pair, copying the proxy CA mount code.
• VM: one overlay file at a VM_GUEST_* constant plus an args-file line; remove the file when unset.
• Kubernetes: the hard part. Either a gateway-managed ConfigMap (needs create RBAC that cannot be name-restricted, 1 MiB data limit, ownership checks, forced server-side apply) or an operator-named ConfigMap mounted directly (no new RBAC, mirrors oidc.caConfigMapName). Sidecar topology must mount into the container that runs the supervisor.
• Core: a reserved container path constant under TLS_ROOT or ETC_ROOT so it stays inside CONTROL_ROOTS and the /etc Landlock baseline.

Alternative Approaches Considered

Option	Pros	Cons
(a) Global gateway setting, driver-delivered (PR #3292)	Operator-owned; one contract for all drivers; matches argv and fail-closed precedent	Largest surface (70 files, +6.2k lines); Kubernetes ConfigMap plus RBAC widening; global blast radius; restart to rotate
(b) Per-sandbox policy field carrying inline PEM	No driver work; live-reloads with policy; per-sandbox scope; trivial on Kubernetes	Sandbox creator owns trust; PEM blobs in YAML/proto; must be excluded from agent-proposable fields; interacts with credential injection
(c) Document the image system bundle	Zero code	Rebuild per rotation; fails in Kubernetes sidecar topology; not operator-controlled
(d) Relax proxy_ca_bundle pairing	Cheapest; Podman/VM done	Conflates two trust purposes; Docker and Kubernetes lack the knob; undoes a deliberate fail-closed rule
(e) Per-driver keys	Delivery is naturally driver-specific; no gateway staging; matches proxy_ca_bundle precedent	Four docs entries and validators; global semantics only by convention

Patterns to Follow

• Operator inputs arrive as argv, never env (main.rs:203-207, upstream_proxy.rs:6-13).
• Shared host and guest validation via driver_utils so acceptance never diverges.
• Fail closed on present-but-invalid values; absent means unset (upstream_proxy.rs:410-423).
• deny_unknown_fields on every new TOML struct; per-driver tables stay raw toml::Value.
• Reserved container paths in container_paths.rs; stage a gateway-owned copy under the state dir rather than bind-mounting the operator's source file (Docker's :z relabels the host file).
• OCSF ConfigStateChangeBuilder on load success and failure.
• Redacted Debug for driver configs carrying material.

Proposed Approach

Treat destination trust as an operator-owned, gateway-validated input that is separate from both gateway mTLS and the corporate proxy CA. The gateway reads and strictly normalizes the configured PEM files at startup, stages a gateway-owned artifact, and hands drivers a path rather than the operator's source file. Each driver delivers the artifact read-only to a fixed reserved guest path and passes it to the supervisor by argv. The supervisor re-validates it, adds the anchors after the default roots in both feature variants, includes them in the child trust files, and aborts startup if the staged material is invalid. A regression test must prove a certificate signed by the destination CA cannot authenticate the gateway. Kubernetes delivery and direct-mode env precedence are the two decisions that most affect scope. PR #3292 is a complete candidate implementation of option (a) and should be evaluated against the decisions below.

Scope Assessment

• Complexity: Medium (High if Kubernetes uses a gateway-managed ConfigMap)
• Confidence: High on the supervisor side, Medium on Kubernetes delivery
• Estimated files to change: 20-30 for all four drivers with docs, Helm, and e2e; 12-15 for a Docker/Podman/VM-first cut
• Issue type: feat

Risks & Open Questions

• Who owns destination trust: operator or sandbox creator? With bundled-ca-roots the image bundle is already overlaid and the proxy resolves hosts from the workload's /etc/hosts; whether that combination is an accepted part of the threat model should be settled before choosing option (b).
• Kubernetes delivery: gateway-managed ConfigMap (needs namespace-wide configmaps/create, and the ConfigMap becomes a namespace-writable trust root) versus operator-named ConfigMap (no new RBAC, simpler). The investigation leans toward the operator-named ConfigMap.
• Direct-mode child env: should TLS env vars be set when there is no proxy, and should user-supplied values win there? Today proxy mode overrides unconditionally. Setting REQUESTS_CA_BUNDLE to a combined file that lacks a system bundle (image with certs outside SYSTEM_CA_PATHS) would break public TLS for Python requests.
• Strictness: the proxy CA path accepts a bundle if one anchor parses, while the lenient loader silently drops bad blocks. Pick one rule for both bundles.
• Rotation: restart-only (consistent with middleware and proxy config) or file watch.
• Scope: could a first cut ship Docker/Podman/VM plus documented image-bundle guidance for Kubernetes?
• Non-bundled-ca-roots builds ignore the system bundle today; the new roots must be added there or the feature no-ops silently.
• Bundle size must respect both the Kubernetes 1 MiB ConfigMap limit and the existing 1 MiB read bound.
• TlsMode::Skip endpoints are unaffected by the root store; only the child env helps there.
• HA gateways sharing one gateway_id would race on a managed ConfigMap during rollout.

LSM Impact

Docker relabels bind mounts unconditionally with :z; Podman only when selinuxfs is mounted. z relabels the host file, so bind-mounting the operator's /etc/pki/... source would change its system label. Stage a gateway-owned copy under the state directory instead. Inside the sandbox the read happens before privilege drop and under the /etc Landlock baseline; no /proc/<pid> traversal is involved. VM overlay files need no relabel.

Documentation Impact

• docs/reference/gateway-config.mdx: new section, and clarify the distinction from proxy_ca_bundle.
• deploy/helm/openshell/README.md(.gotmpl) and values.yaml for the Kubernetes value.
• architecture/sandbox.md and architecture/compute-runtimes.md.
• skills/debug-openshell-cluster/SKILL.md (required by AGENTS.md for Helm and driver changes).
• rfc/0003-gateway-configuration if the config schema grows.

Disposition Readiness

• State: state:validated
• Assessment: The failure is reproducible by reading the root-store construction, the workarounds and their gaps are confirmed in code, the affected components and integration points are mapped, and a working candidate implementation (PR feat(network): support additional destination CAs #3292, manually verified against a private-PKI GitLab on rootless Podman) exists. A maintainer has enough to accept or decline and to choose among the design decisions above.
• Missing evidence: None for disposition. Kubernetes and VM e2e results for the candidate PR are pending CI.

Test Considerations

• Unit: root store gains anchors in both feature variants; write_ca_files ordering; config parse and deny_unknown_fields; strictness cases (empty, private-key block, oversized, non-regular file, mismatched labels); per-driver argv and mount tests like the Podman proxy CA tests; Kubernetes pod-spec rendering for both topologies; gateway-mTLS negative test in grpc_client.rs.
• Helm unittest: new suite under deploy/helm/openshell/tests/ for values, RBAC, and volume rendering.
• E2E per driver, modeled on podman_corporate_proxy.rs and vm_corporate_proxy.rs: private-CA HTTPS upstream, curl succeeds with the setting and fails without, public host still works, hostname mismatch still rejected, invalid staged material fails closed, removal plus restart drops trust.
• Existing test infrastructure: the corporate proxy e2e harness already stands up a TLS endpoint and stages operator material per driver; a shared additional-ca suite can reuse that pattern.

---

Created by spike investigation. state:validated means the issue is ready for human disposition; state:needs-info means specific evidence is still required. A human applies state:accepted or places the issue on the roadmap if OpenShell should pursue the work. To queue unattended agent planning, a human applies agent:plan-requested; on a direct request, the agent warns about missing expected workflow labels and continues without changing them. Candidate implementation: #3292.